Abstract:
An apparatus and methodology to acquire and organize measured or modeled statistical data into optimal reports with a performance engineering mode of use and a design mode of use. In a performance engineering mode of use, the engineer may select from a set of performance questions, and guided by the apparatus and largely automated, create well-defined answers to the performance questions of interest. A series of template manipulations whereby report objects that are embedded within templates may be defined, reused, modified and improved upon to optimize reports and to aid in a report building process in a design mode of use. Methods are taught for the automatic selection and population of data tables. Column selection and column header information is optimized for relevance to the report design or system question at hand. The automatic joining of data from a variety of data sources is taught that allows for the rapid construction of specific reports from within multiple data tables of different types, structures and formats.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims priority to U.S. Provisional Application No. 60/579,456, entitled “Method and Apparatus for Acquiring and Organizing Simulation Statistics,” filed Jun. 14, 2004; U.S. Provisional Application No. 60/579,306, entitled “Method and Apparatus for Translating Objects Into Templates,” filed Jun. 14, 2004; U.S. Provisional Application No. 60/579,305, entitled “Method and Apparatus for Automatic Selection of Data and Table Population,” filed Jun. 14, 2004; and U.S. Provisional Application No. 60/579,329, entitled “Method and Apparatus for Joining Data and Building Tables,” filed Jun. 14, 2004. 
    
    
     TECHNICAL FIELD OF THE INVENTION 
     The technical field of this invention is software, namely, software to organize and display performance data from complex computer networks. 
     BACKGROUND OF THE INVENTION 
     The performance of business systems in the global economy is an area of considerable interest as businesses become more disperse and applications become more complex. Decisions must be made rapidly and data systems must remain reliable and available. Reliability and performance can be a considerable issue in the face of rapid system or application scaling such as would be experienced in a merger of two large corporations or in an onset of an IT outsourcing contract. 
     A goal of modern IT performance engineers is to optimize business applications on quite large and complex systems with perhaps many thousands of nodes that are often widely geographically dispersed. In order to meet this goal, a performance engineer might design a test environment with actual equipment running actual business applications to be tested but on a much reduced scale from a “production” environment. The performance within the test environment is carefully measured and scaled and the performance engineer would then like to take that data and project how the business application will perform in the more complex production or projected environment. In other situations, a system may be overly stressed, with such low business application performance that the situation is detrimental to the function of the corporation. To relieve the situation, the performance engineer may be asked to troubleshoot the problem quickly. To accommodate the performance engineer a tool for quickly organizing appropriate and existing test data into a form that will answer key system questions is essential. Furthermore, rapidly visualizing the answer to the key system question in a form that optimizes the performance engineer&#39;s ability to draw conclusions and make decisions has considerable value in the art of the field. 
       FIG. 1  shows an example of a test network to investigate application and network performance. This example includes a network of servers, workstations, business applications, data storage devices, test devices and IP network connections between them shown as LAN  115  and Internet  105 . The network of servers is comprised of application server  125  connected to LAN  115  which runs business, engineering or research applications, database server  120  connected to LAN  115  and which is a local database that organizes information of interest to the business, storage server  135  connected to LAN  115  and which holds data storage  138  that feeds the servers and to which data is backed up from the servers, remote database server  190  connected to LAN  115  via the Internet  105  and remote LAN  117  and which is geographically remote from database server  120  and serves a similar function to the local server but may house different pieces of information from different business units, and remote storage server  150  connected to LAN  115  via the Internet  105  and remote LAN  116  and which is used to keep a synchronous or asynchronous copy of the local data storage  130  to remote storage  155 . A workstation  130  is shown which runs a first application client  101  and a second application client  102 ; workstation  130  is also connected to LAN  115 . Interspersed between the LAN  115  and the various servers are network sniffer devices  140 ,  145 ,  150  and  160 . There is a network sniffer device  170  between LAN  115  and the Internet  105 . Network sniffer devices  175  and  185 , are respectively connected between the remote data storage  155  and remote storage server  150  and between the local data storage  138  and the storage server  135 . There is also a network sniffer device  180  between the Internet  105  and remote database server  190 . The network sniffers function to examine data packets as they traverse the network looking for a match and logging a timestamp for each match. They will also count the number of packets that match in a given time frame and perform other such functions related to network packet timing. 
     There are three interesting classes of test to run on this network. The first class of test, test1, captures a network trace of an instance of a business application to establish the flow of the business process through the network. For example, application client  101  may launch a web application from workstation  130  that will require various unknown network resources. Test1 will ultimately trace the paths that the application will take through the network to find the resources. Reports from test1 will typically list the various network resources and response times. 
     The second class of test, test2, captures resource usage of various components of the network. For example, application client  102  utilizes workstation  130 , application server  125 , database server  120  and storage server  135  and remote storage server  150  to create and store a set of business transactions. Test2 will correlate the usage data on the various devices in the network to the business application run to prepare a set of resource usage reports. For example, CPU utilization on Workstation  130  would be included in that report. Fairly complex reports can be created by test2 where the business function is loaded repeatedly to examine network and resource utilization under scaling. 
     The third class of test, test3 captures resource usage and other correlated information from various components of the network when multiple business applications are running. For example, 3 instances of the application client  101  and 5 instances of application client  102  are run at the same time. Even more complex reports are generated by test3 tests that look at resource usage and scaling in a mixed environment. 
     Measured data from tests like those described can be utilized in simulation and modeling programs to predict network or system performance in different environments than the one on which the measurements were made. The performance engineer with these simulation and modeling programs can generate vast amounts of data about his network or system—modeled data that can be used to rapidly solve performance problems given the right tools to organize the data. 
     Several recurring questions routinely arise in analysis of system performance data and in predictive scenarios. For example, questions that could be asked in such a test environment, such as “What are the bottlenecks?”, “Are the performance objectives being met?”, or, “Will the performance objectives be met when the number of clients on the network scales to 10,000?”. Typically the performance engineer will have to manipulate a large amount of data organized in spreadsheets and text files to arrive at the answers to these and other questions. Therefore, a need exists to overcome the inefficiencies in defining the queries and performing manual manipulation of performance data to arrive at answers to routine system performance questions. 
     A motivation of the present invention is to present the performance engineer with a class of questions and a novel apparatus to automatically organize measured data and modeled data into forms that answer system questions clearly and concisely into a visual form using charts, graphs, and tables saving much time and effort. Additionally the present invention provides the performance engineer with flexible means of manipulating complex reports so that valuable classes of reports may be saved as projects and templates to be recreated later. The ability to conveniently save templates combined with other novel mechanisms of the present invention allows the performance engineer the capability to create new questions or categories of reports that can be optimally tailored to the network under consideration. 
     SUMMARY OF THE INVENTION 
     The present invention teaches processes and apparatus to acquire and organize measured or modeled statistical data into optimal reports. In a design mode of use, a report designer utilizes the apparatus as a tool to create optimal reports from a variety of data sources, translating the reports into templates that can be reused to automate a reporting process to repeatedly solve a class of user defined problems. In a performance engineering mode of use, a performance engineer utilizes a process enabled by the apparatus whereby the engineer may select from a set of performance questions, connect the apparatus to a variety data sources, and through an interaction process enabled and guided by the apparatus and largely automated, create well-defined answers to the performance questions of interest. The performance engineer, may create templates or projects that capture the process and allow it to be repeated in a continual process to make network or system optimizations. 
     One embodiment of the present invention teaches a series of template manipulations whereby report objects that are embedded within templates may be defined, reused, modified and improved upon to optimize reports and the report building process in a design mode of operation or a performance engineering mode of operation. 
     Another embodiment of the invention teaches manipulation and use of data within data Tables for the automatic selection and population of other data tables. In particular, column selection and column header information is optimized for relevance to the report design or system question at hand. A novel mechanism for automatically joining data from a variety of data sources is also described that allows for the rapid construction of specific reports from within multiple data tables of different types, structures and formats. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       For a more complete understanding of the features and advantages of the present invention, reference is now made to the detailed description of the invention along with the accompanying figures in which corresponding numerals in the different figures refer to corresponding parts and in which: 
         FIG. 1  is a schematic illustration of representative system to be optimized 
         FIG. 2A  is a block diagram showing the function of the present invention. 
         FIG. 2B  is a block diagram showing a report object. 
         FIG. 3  is a block diagram showing the control and data flow of the present invention. 
         FIG. 4  is a block diagram showing the project document structure of the preferred embodiment of the present invention. 
         FIG. 5  is a flow diagram of the preferred embodiment of the mode of use of templates within the present invention. 
         FIG. 6  is a flow diagram of the table optimization function of the present invention. 
         FIG. 7  is block diagram of a first embodiment of the table optimization process of the present invention wherein the rules formation is coded in a static manner. 
         FIG. 8  is a block diagram of a second embodiment of the table optimization process of the present invention wherein the rules formation is coded in a dynamic manner. 
         FIG. 9  is a block diagram containing lists showing an example of column optimization rules formation within the table optimization process. 
         FIG. 10  shows a picture of a screen shot of an instance of an optimized output table and a listing of rules from a table optimization process executed by the present invention. 
         FIG. 11  is a block diagram of the automatic joining of multiple tables within the preferred embodiment of the present invention. 
         FIG. 12  is a process flow diagram of the automatic joining of multiple tables within the preferred embodiment of the present invention. 
         FIG. 13  is a block diagram of the virtual database structure within the preferred embodiment of the present invention. 
         FIG. 14  is a listing of an example of join specification match rules generated within the preferred embodiment of the present invention. 
         FIG. 15A-E  is a listing of questions and reports within a preferred embodiment of the present invention. 
         FIG. 16A-C  is a set of pictures showing representative example screen shots of visible reports generated by a preferred embodiment of the present invention. 
         FIG. 17A-J  is an annotated listing of an example XML template file utilized within a preferred embodiment of the present invention. 
     
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     While the making and using of various embodiments of the present invention are discussed in detail below, it should be appreciated that the present invention provides many applicable inventive concepts that can be embodied in a wide variety of specific contexts. The specific embodiments described herein are merely illustrative of specific ways to make and use the invention and do not delimit the scope of the invention. 
     In  FIG. 2A  a report visualizer program  200 , which constructs complex reports for a user  204 , provides the following functions: A question selections process  202  for aiding user  204  in selecting a question from a list of questions to be answered by a report, a report design process  210  for gathering information and populating report designs from user  204 , a report building process  220  for automatically gathering data into reports and preparing them for viewing, a report viewing process  230  for presenting reports visually to user  204  and a project control process  240  for initializing a project and for saving project information for later use by user  204 . Solid arrows in  FIG. 2A  represent process flow between entities, dashed arrows represent information flow between entities. Process flow may also include information. User interfaces to report visualizer are standard windows having standard controls as is known in the art, an example being an Explorer window within the Microsoft Windows™ operating system. 
     There are two primary classes of users for the present invention. The first class of user will employ report visualizer program  200  to design a specialized report  208  that is required for a business customer or situation or for a departmental situation given some data (not shown) supporting the particular nature of the report. The first class of user will likely not require the question selection process  202 , bypassing it in favor of heavy use of the report design process  210 . One function of the present invention is to give user  204  a means of rapidly and easily designing reports that do not contain superfluous data and that can be repeatedly, perhaps automatically generated on, for example, a periodic basis. Within the preferred embodiment, user  204  benefits from generating and saving a set of report templates that correspond to the generated report designs. 
     The second type of user will employ report visualizer  200  to solve a particular set of performance engineering problems  207 . The second type of user utilizes the process of selecting a performance related question in question selection process  202 , accepting and or adding information to report designs  210 , supplying raw data for the report building process  220  and viewing and interpreting the report in report viewing  230  to solve the performance related engineering problem. In the preferred embodiment, the second type of user benefits from a pre-defined set of questions in the questions selections process  202  that are associated with pre-defined report structures used throughout the process to create classes of reports that will aid the user in improving the performance of a system. 
     A set of certain system questions  205 - 1  through  205 - q  are created regarding the specific nature of a network or application performance. In the preferred embodiment, system questions  205  are represented as folders in a question selection software process  202  that allows a user  204  to select questions for further inquiry. The system questions  205 - 1  through  205 - q  comprise folder names that appear to user  204  and may be of the form of an interrogative denoted by a question mark, as in “What are the potential bottlenecks?” or they may be more generally of the form of a statement as in “Application Performance Reports”. In the question selections process  202  there are up to Q questions  205  available for selection and one or more system questions  205  may be selected at a time. A list of 92 “questions” available in one preferred embodiment is shown in  FIG. 15 . Eleven of these “questions” are interrogatives. In other embodiments, questions may be worded differently and there may be fewer or more questions available for selection than shown in  FIG. 15 . In the preferred embodiment, question or report folders are internal to the visualizer program and not separately represented in the file system. 
     A particular system question has associated with it a certain number of report designs. The number of report designs associated with a system question can vary. For example, when system question  205 - 1  is selected in the questions selection process  202 , the report design process  210  functions to prepare a list of report designs associated with the selected system question  205 - 1 . Question-report associations  219  indicate a specific mapping between system question  205 - 1  and report designs  215 - 1  through  215 - t . The report design process  210  allows user  204  to change the available list of report designs  215  by adding, deleting and editing report designs. 
     Once the report design process  210  completes, the report designs  215 - 1  through  215 - t  are saved into computer memory by the report visualizer program  200  as complete report objects  225 - 1  through  225 - t  to be used by the report building process  220 . In the preferred embodiment, the report visualizer program  200  creates empty or default report data structures, with no data or with default data, respectively. The report design process  210  fills in the report data structures with the data or accepts the default data at which time the completed report data structure becomes a report object  225 . The report design process  210  may be assisted by the user, modifying the data or form of the report. Alternatively, the process may include selecting a report design template from a dialog window displaying a preset number of format changes.  FIG. 15  shows examples of  244  default report design templates. 
       FIG. 2B  shows report object  225 . In the preferred embodiment, report object  225  is constructed of various attributes. For example, the attributes include report name  226  (for identifying a report), data sources  227  (that contain data of interest), data tables  228  (pointers or references to data tables within the program), queries  229  (for extracting data), table transformations  231  (defining a set function to be performed on the table), data filters  232  (for manipulating data), table layouts  233  and chart layouts  234  for visualizing data. Report name attribute  226  names the report object and its subsequent charts or tables when they are created. Data sources attribute  227  are references to sources for data tables pointed to by the data tables attribute  228 . Data tables are stored in computer memory in a standard program format in the preferred embodiment whereas the data sources may be in non-standard forms and may be generated by external programs, such as network measurement devices or network simulation programs. Data sources may also be formed from existing report objects. Queries attribute  229  are codes for specific searches within data tables that are designed and utilized to locate specific answers, in the form of numbers or text, to the associated aspect of the system question  205 - 1 . Table transformation attribute  231  manipulates data within data tables referenced by data tables attribute  228 . For example, a pivot transform is one of many functions that may be performed in a table. Data filters attribute  232  functions to remove unwanted information from data tables referenced by data tables attribute  228 . Table layout attribute  233  and chart layout attribute  234  contain properties information required to visually display information from a data table. 
     Referring now to  FIGS. 2A and 2B , the process of taking the information encapsulated in a report object  225 , and converting that information into a form to be viewed is collectively called the report building process  220 . The report building process functions to perform a set of queries  229  on a database and to perform transformations as per table transformation attribute  231  or data filters attribute  232  and place the resulting data into a query result. The report building process also functions to execute the layout of tables  233  or the layout of charts  234  for viewing in report viewing process  230 . Properties information contained in the table layout  233  or chart layout  234  structures define the visual appearance of the report. The report building process  220  functions then to obtain and organize the data from various sources, processing and/or reducing the data so that it may become optimally sensible to a user in relation to a network or system question the user is attempting to answer or for a report design. 
     Continuing with  FIG. 2A , report object  225 - 1  through report object  225 - t  sends appropriate instructions for display to report viewing process  230 . The report viewing process converts report objects  225  into reports  235 . Reports  235  are visual displays. The report object displays a unique report. The reports  235 - 1  through  235 - t  consist of charts and/or tables that are displayed on a computer monitor or similar device. In the preferred embodiment, the reports can also be printed, saved or exported to another program. For example, reports can be saved as image files (html, jpg, png), table reports can saved as a .csv file and viewed in Microsoft Excel. 
     Project control process  240  functions to open or close projects, initialize data structures within a project and to save projects and templates. 
     In an alternative embodiment, another mode of operation is available by connecting report visualizer to other running programs via an operating system interface or similar communications structure. A particular application  209  requiring a report may send a pre-defined report template to report visualizer  200  which automatically inserts the template into the report design process  210 , runs the designs encapsulated in the template, builds the reports using report building process  220  and displays the report using report viewing process  230 . Project control  240  will be used throughout to load files as required for the process without required user intervention. 
     It is seen that the present invention functions to efficiently answer system or network questions by choosing and organizing information from various data sources into data tables and charts and displaying them. The answer to these system or network questions can be used to facilitate decisions about such things as troubleshooting system problems, purchasing network components or deploying new business functions. 
     The structure of a preferred embodiment of report visualizer  200  is shown in the block diagram in  FIG. 3 . The various connections shown between the blocks in  FIG. 3  are logical connections that indicate information flow from one block to another. A dynamic project document structure  300  within report visualizer  200  gathers data from measured results  385 , or modeled results  384 , or both, and organizes that data into reports  235  that provide insight to the system question. Data is collected from various network or system tests to form one or more data sources of measured results  385  for one or more networks or systems. Data is collected from various simulators or computer generated models to form one or more data sources of modeled results  384  for one or more networks or systems that may differ from those networks or systems used to generate measured results  385 . 
     Measured results  385  or modeled results  384  may be stored in one or more physical locations, geographically remote from the machine that is operating the report visualizer  200  program, the information may flow from the data sources into report visualizer  200  using, for example, TCP/IP protocol over the Internet or other networking protocols. 
     Project document  300  also contains report wizard  352  which directs the process of creating or modifying the report definition. Report definition module  325  constructs an internal data representation of a report by querying the virtual database  382  and applying table column layout, filtering, sorting, transformations found in report object  225  attributes, found in memory, in a template or in a project. The report view  330  module constructs a GUI representation of a report from the content of the report definition. Report definition module  325  constructs an internal data representation of a report by querying the virtual database  382  and applying table column layout, filtering, sorting, transformations found in report object attribute  225 , found in memory, in a template or in a project. The report view  330  module constructs a GUI representation of a report from the content of report definition  325 . Tree view control module  320  organizes and displays a tree view and processes user commands from the tree view. The presentation control  362  module provides access to the visual attributes of a report so that the appearance of the report can be altered by the user. 
     Virtual database  382  is an object which is automatically initialized by project document  300  to populate internal data tables that correspond to data required to answer the system question. Virtual database  382  creates and manipulates data structures based on data obtained from modeled results  384  and measured results  385 . Queries are sent to virtual database  382  by report definition module  325  to search its data tables for certain information. In response, virtual database  382  returns relevant subsets of information from its internal data tables to the report definition module. The queries and data tables are then included and referenced in report object  225 , queries  229  and referenced data table references  228 . 
     Tree view control  320  forms an interface with user  204  and with report definition module  325 . Tree view control  320  sends the user&#39;s selected data to the report definition module  325 . Tree view control  320  maintains the data structure for report definitions  325  allowing the user to create, insert, rename, delete or move a folder or report in its data structure. 
     Report view control module  330  uses standard Java GUI interfaces and objects which are accepted by graphics generation programs known in the art to create viewable content, such as report  335 . Report view control module  330  interacts and displays content to a standard display unit, such as a computer graphics display device connected to a computer monitor screen and allows user interaction with presentation control  362 . Tree view control  320  and report view control  330  are coupled and display their views simultaneously. 
     Presentation control  362  allows the user to aid in a report&#39;s visual attributes by modifying visual properties of the content maintained by reports definitions module  325 . Upon exiting presentation control  362 , the visual attributes within reports definitions module  325  and its corresponding report object  225  are updated and stored. 
     Report wizard  352  is a user interface utilized within the report design process  210  to construct or load new report objects  225  and set their attributes in report definitions module  325 . The attributes include the identification of specific data sources  228  contained within measured results  384  or modeled results  385 . 
     Reports definition module  325  is called by tree view control  320  to initiate the data structure corresponding to report object  225  and to run the methods associated with report object  225 . Additionally, reports definition module  325 , constructs report object  225  template fragment for inclusion in a project file or template file. 
     Project document  300  collects the information regarding the question selection and initializes virtual database  382 . The virtual database then connects to the appropriate data sources  228 , informing report definition  325  which of the report objects  225  to include on start-up. Project document  300  organizes the project information into data structures, called project files  372  and templates  275 . Project files  372  are a “snap-shots” of the current state of the project document  300  and capture all of the relevant data to recreate that state. Project document  300  has a file saving and loading means by which project document  300  can save and retrieve project files  372  and report templates  275  to and from computer storage or memory. 
     In the preferred embodiment, a template file  375  is used to externally represent a set of report objects and their organization into system questions. Template file  375  is a text file which contains XML standard instructions sufficient to recreate all the report objects and their associated report definitions and reports. Project document  300  can save and retrieve template files  375  so that a multiplicity of reports and their structures can be reproduced in an automatic way. The use of template files  375  is described more fully below. 
       FIG. 17A-J  is an example of a template created by the preferred embodiment of project document  300  within report visualizer  200 . Examining  FIG. 17A , the template is associated with a particular system question “How does the performance compare to the objectives?” labeled as “Folder:”  1000  in the second line and with a particular chart pertaining to that system question: “Business function response time compared to objective chart”  1001 . The template contains a large number of structures which are annotated throughout. Those skilled in the art will easily comprehend the XML text by reading the annotations included. 
     For example, in  FIG. 17A , a column definition  1002  is made within an XML construct for a column within a table with a header column name of “Business Function” containing the text “Business Function” and column value that is obtained from a specified modeled results table with formatted value specification “ScenarioResults.Statistics.comp_name”. 
     In another example from  FIG. 17E , table sorting and transform functions  1010  are defined and annotated for a pivot type transform on tables defined within the template. 
     In a third example from the template file, a query table “ScenarioResults.Statistics”  1012  is queried in  FIG. 17F  that looks for “BF” in the field “component” and looks for “response_time” in the field “stat_name”. 
     Referring to  FIG. 4 , project document  300  includes a view model and a data model. The data model consists of the set of report definition objects  325 - 1 ,  325 - 2 , . . .  325 - t . The view model consists of a frame object which contains both a tree view  320  and a report view  330 . Frame  310  is a container for the presentation of visual information and corresponds to a viewable window in the report visualizer application  200 . Tree view  320  and report view  330 , which are displayed simultaneously to the user  204 , provide the particular organization and representation for presenting the reports that are in the data model as previously described. Frame object  310  gets data from reports definition modules  325 - 1 ,  325 - 2 , . . .  325 - t  to generate corresponding viewable Reports  235 - 1 , . . . ,  235 - t.    
     Tree view  320  consists of nodes  321 - 1  through  321 - t  organized into a tree of folders and reports within folders as stored in its data structure. The nodes form a representation of available reports that are associated with project document  300 . With each node is associated a node name that encapsulates a particular question selection or a report name that answers a particular aspect of the system question. Nodes  321 - 1 , . . . ,  321 - t  are 1:1 associated with report definitions  325 - 1 , . . . ,  325 - t  (which encapsulate report object  225 - 1 , . . . ,  225 - t  information) so that a particular node draws its name from the report definition module  325  to which it is attached. Each node also contains a memory pointer to the associated report definition module  325  so that the nodes can access the entire report definition module  325  or information contained within the report definition module  325  to pass it to other components within the project document. 
     When a node  321  is selected from tree view  320 , the associated report definition module  325  loads, processes its report, and ultimately displays it via report view  330 . In a similar way, the report view  330  is associated with each report object  325 . When a particular node  321  is selected for viewing within tree view  320 , the report view  330  requests its information from the report definition module  325  associated with the selected node  321  to define a visual image of the associated report. 
     Templates  375  are used in a variety of processes to allow the report visualizer  200  considerable flexibility in its usage. The major manipulative steps  410 ,  420  and  430  within report visualizer  200  are shown in a process in  FIG. 5  with a choice of variations for each step. Each variation involves particular template manipulations. The choice of variations is independent for each step. The steps are:
         Step  410 —Load an initial project document.   Step  420 —Allow the user to make modifications to the project document to complete the creation of concrete report or reports   Step  430  Save the resulting project document as a template
 
Step  410  has the following variations:
       

       410 .A. Create a project document, by automatically choosing a template or templates appropriate to the data files selected. This involves some analysis by the report visualizer  200  of the content of the selected files, including an analysis of what statistics are contained in the selected files. 
       410 .B. Create the project document using the template and data files selected by the user or by the invoking program. 
       410 .C. Create a new empty project document with no reports or questions. The user selects the data files to load. 
       410 .D. Create a new project document using data files selected by the user or by the invoking program and using an existing project document as the basis for the report definitions. That is, use an existing project document as a template. 
       410 .E. Load an existing project document. 
     Step  420  has the following embodiments and can be done repeatedly, mixing various embodiments in each repetition: 
       420 .A. The user can insert, delete, modify, rename and rearrange reports and folders manually by interacting with the tree view, report wizard and presentation control associated with project document. 
       420 .B. The user can insert a template defining a set of folders and reports into the open project document. 
     Step  430  has the following embodiments and can be done repeatedly, mixing various embodiments in each repetition: 
       430 .A. The user can save the entire project document as a template. 
       430 .B. The user can select a node (a folder or report) and save that node (and its associated folders) as a template. 
       FIG. 6  depicts a preferred embodiment of a function of report visualizer  200  for automating a process for producing visual and content optimal data tables for reports. A system of networks, servers and business applications is shown as  1100 . Measured data  1110  and modeled data  1120  are extracted and calculated, respectively, from the system  1100 . The measured data  1110  and modeled data  1120  are used as the data sources to form input data table  1130  which is contained in virtual database  282 . Input data table  1130  contains N columns with distinct column headers, Column Header 1, Column Header 2, . . . Column header N and distinct data in each row of each column, data-11, data-12, data-13 . . . data-NK. Each column has K rows. Statistical information related to a system question is encoded in the data within input data table  1130  and may be scattered across different data positions. Column header information within input data table  1130  is textual, data within input data table  1130  may be text or numerical. 
     In report visualizer  200 , user  1175  with a particular system issue to solve  1165  in relation to the system  1100  will be presented with a choice of system statistics to investigate  1169 . Upon the user selecting a set of system statistics to investigate  1169 , a table optimization and layout process  1170  is performed that automatically optimizes the statistical information and layout a new optimal data table  1180 . During this process, the content is reduced and optimized and the headers are renamed to give relevant visual information. For example, optimal data table  1180  contains M+3 columns, shown with column headers “Identifier  1  New Header” and “Identifier  2  New Header” and statistics data columns shown with column headers “New Descriptive Column Header”, “Associated Statistic Value  1 ”, . . . “Associated Statistic Value M”. The identifier data is shown as ID value  1 . . . ID value  6 , the statistic data is shown as returned data- 11 . . . returned data-M3. Other instances of optimal tables (not shown) may contain more or less than two identifier columns and more or less than the three rows of data and the columns that get renamed will vary widely. New descriptive column headers may appear on identifier columns and Associated statistic headers. 
     The user may utilize the information in optimal data table  1180  to optimize specific business applications, network performances and/or server performances within the system  1100 , or to answer certain performance questions at issue. 
     A more detailed description of the table optimization and layout process  1170  in  FIG. 6  is shown in  FIG. 7 . Table optimization process  1170  is constructed of a tree view  1250  for display and select functions, table properties  1230  containing information and methods for layout, rules  1400  for optimizing, and table layout generator  1270  for producing a visible table. Table optimization and layout process  1170  further interacts with user  1280 , input table object  1210  and output table object  1220 . Input table object  1210  and output table object  1220  are analogous to input data table  1110  and optimal data table  1180  in  FIG. 6 . The various connections between the entities shown in  FIG. 7  indicates interaction and information flow between the entities. 
     Table properties  1230  is a data structure containing data and methods required for successful layout of Output table  1220 . Table properties  1230  contains class names  1231  defined for table object  1210 , class associations  1232  defined for table object  1210 , value associations defined for table object  1210 , rules methods  1234  for table object  1210  and layout methods  1238  for describing the layout of new table object  1220 . 
     Tree view checklist  1250  utilizes class names  1231  and class associations  1232  to display a check list  1255  for user  1280  to select from. Tree view checklist  1250  sends a query or set of relevant queries  1272  to table layout generator  1270 . Table layout generator  1270 , in turn, uses the query along with processing methods from table properties  1230 , and data from input table object  1210  to construct the output table object  1220  which is made visible to user  1280 . User  1280  directs the information in output table object  1220  to optimize the system  1100 . During said process, a version of the output table object  1220  may be displayed to user  1280  without data values so that user  1280  has opportunity to further refine the table properties  1230 . 
     Table properties  1230  data structures relate to the content of input table object  1210  by organizing input table object  1210 &#39;s column headers into class identifier columns with class column names  1231 , statistics that are associated with the class identifier columns known as class association instance columns  1232 , and specific statistic associated value columns  1233  which are columns of values associated with a particular statistic. Examples of associated instance columns and associated value columns include, respectively, an instance name column (such as “Component name”) associated with an instance type column (such as “Component type”) and statistics value columns (such as “mean”, “maximum”, “minimum”) associated with a statistic type column (For the statistic type value “response time”, the associated value columns may be appropriate to display, whereas for other statistic type values, the columns may not be needed). The rules methods  1234  within table properties are used to construct layout processing instructions based on rules  1400 . In the preferred embodiment, the rules methods  1234  may be added by the user into the table properties. The rules  1400  and the process for selecting the rules  1400  to code the rules methods  1234  is explained further below. Layout data  1238  is also contained within table properties  1230  which is a repository for visible layout properties of the output table object  1220 . 
     Tree view checklist  1250  is a visible frame of check list  1255  showing the statistics available in the input data table  1210 . A check box  1252  for selecting statistics from check list  1255  is provided. Check box  1253  is selected as an example. The statistics are organized in a tree view with nodes  1254   a - 1254   x . Only the leaf nodes are selectable and these correspond to particular queries: query input  1272  is generated for leaf node  1254   b  as an illustrative example. All the nodes shown except node  1254   a  and node  1254   g  are leaf nodes as shown in tree view checklist  1250 . The tree view organization is customized for each data table in a way that will help the user find statistics. In a preferred embodiment, the queries generally resolves to a set of values to match the Class Columns for the data table. Statistics differ only in the statistic type value and otherwise share the same Class Column. Value specifications tend to be adjacent to Class columns. 
     Table layout generator  1270  is a process with query input  1272 , properties input  1275 , table data input  1276  and a display output  1278 . Table layout generator examines the queries from query input  1272  to assist in executing table processing rules from properties input  1275 . User  1280  is provided a table layout (not shown) without data from Input data table  1210  to verify the suitability of the new table layout for output table object  1220  and to further edit output table object&#39;s  1220  layout properties  1238  if required. The processed data is then laid out into a visual format according to the layout information also gathered from properties input  1275  and displayed to display output  1278 . 
     Referring back to  FIG. 3  and  FIG. 7 , in the preferred embodiment tree view checklist  1250  exists within the project document  300  and is displayed via the report wizard  352 . Table Properties  1230  exists within the virtual database  382  and is created before and during the input table object  1210  creation. The input table object  1210  also exists within the virtual database  382 . The table layout generator  1270  functions within report wizard  352 . The output table object  1220  is displayed as a table layout in the report wizard  352  and then as a report in report view  330 . 
     Another embodiment of table optimization process  1170  in  FIG. 6  is shown in  FIG. 8 . In this embodiment, table optimization process  1170  is constructed of a tree view checklist  1350  for display and selection, table properties  1330  to contain table information and properties for layout, rules  1400 , dynamic rules Formation processor  1365  for applying Rules  1400  and table layout generator  1370  for displaying a table. Table optimization process  1170  interacts with user  1380 , input table object  1310  for optimizing layout and output table object  1320 . Input table object  1310  and output table object  1320  are analogous to input data table  1130  and optimal data table  1180  in  FIG. 6 . The various connections between the entities shown in  FIG. 8  indicate interaction and information flow between the entities. 
     Table properties  1330  is a data structure containing data required for successful layout of output table object  1320 . Table properties  1330  contains class names  1331  defined for table object  1310 , class associations  1332  defined for table object  1310 , value associations defined for table object  1310 , and layout properties  1338  for describing the layout of new table object  1320 . 
     Tree view checklist  1350  utilizes class names  1331  and class associations  1332  to display a check list  1355  for user  1380  to select from. Tree view checklist  1350  sends a query or set of relevant queries  1372  to table layout generator  1370 . Table layout generator  1370 , in turn, uses the query along with processing methods from table properties  1330 , and data from input table object  1310  to construct the output table object  1320  which is made visible to user  1380 . User  1380  directs the information in output table object  1320  to optimize the system  1100 . During said process, a version of the output table object  1320  may be displayed to user  1380  without data values so that user  1380  has opportunity to further refine the table properties  1330 . 
     Table properties  1330  data structures relate to the content of input table object  1310  by organizing input table object  1310 &#39;s column headers into class identifier columns with class column names  1331 , statistics that are associated with the class identifier columns known as class association instance columns  1332 , and specific statistic associated value columns  1333  which are columns of values associated with a particular statistic. Examples of class association instance columns and associated value columns include, respectively, an instance name column (such as “Component name”) associated with an instance type column (such as “Component type”) and statistics value columns (such as “mean”, “maximum”, “minimum”) associated with a statistic type column (For the statistic type value “response time”, the associated value columns may be appropriate to display, whereas for other statistic type values, the columns may not be needed). Layout properties  1338  is also contained within table properties  1330  which is a repository for visible layout properties of the output table object  1320 . 
     Tree view checklist  1350  is a visible frame of check list  1355  showing the statistics available in the input data table  1310 . A check box  1352  for selecting statistics from check list  1355  is provided. Check box  1353  is selected as an example. The statistics are organized in a tree view with nodes  1354   a - 1354   x . Only the leaf nodes are selectable and these correspond to particular queries; query input  1372  is generated for leaf node  1354   b  as an illustrative example. All the nodes shown except node  1354   a  and node  1354   g  are leaf nodes as shown in tree view  1350 . The tree view organization is customized for each data table in a way that will help the user find statistics. The queries generally resolve to a set of values to match for the class columns for the data table. Statistics differ only in the statistic type value and otherwise share the same class column. Value specifications tend to be adjacent to class columns. 
     Dynamic rules formation processor  1365  has an input  1361  for table property information which is tied to table properties  1330 , an input directly from a table object which is tied to the input table object  1310 , an input  1373  for rules tied to Rules  1400  and an output  1374  for encoded rules tied to table layout generator  1370 . The encoded rules are column formation instructions for the output table object  1320  that are specifically based on the information in the input table object  1310 . 
     Table layout generator  1370  is a processor with query input  1372 , properties input  1375 , rules instructions input  1374 , table data input  1376  and a display output  1378 . Table layout generator executes the queries from query input  1372  to assist in executing table layout processing rules from dynamics rules formation processor  1365 . User  1380  is provided a table layout (not shown) without data from input data table  1310  to verify the suitability of the new table layout intended for output table object  1320  and to further edit output table object&#39;s  1320  layout properties  1338  if required. The processed table is put into a visual format according to the layout information also gathered from properties input  1375  and displayed to display output  1378 . 
     Referring back to  FIG. 3  and  FIG. 8 , tree view checklist  1350  exists within the visualizer project document  300  and is displayed via the report wizard  352 . Table properties  1330  exist within the virtual database  382  and is created before and during the input table object  1310  creation, the input table object  1310  also exists within the virtual database  382 . The table layout generator  1370  functions within report wizard  352 . The dynamic rules formation processor  1365  functions within the report definition module  325  structure and the output table object  1320  is displayed as a table layout in the report wizard module  352  and then as a report in report view  330 . 
     In a preferred embodiment, there are two types of columns assumed in the Rules  1400  within the preferred embodiment of the invention: Class identifier columns which identify the class of entity that one or more columns in a row refer to and associated columns which break down further into associated instance columns and associated value columns. Typically, the association is one of dependence, if the class column value is not available in a particular table, the associated identifier or value columns will also be assumed not to be available. The rules  1400  are typically not executed in a pre-defined order, but follow the table layout process so that if a particular column is laid out first (from left to right across the table) then its associated rules will execute first. 
     The rules  1400  for table optimization process in the preferred embodiment of the present invention are: 
     A. Class Column Behaviors Based on Query 
     
         
         
           
             1. Drop the column if it is only blank or if it does not contain available values allowed by the given query. 
             2. Drop the column if it is unique, that is the given query specifies exactly one value allowed for this column.
 
B. Associated Column Behaviors Based on Query
 
             1. Drop the column if the associated class column is dropped by rule A.1 above. 
             2. Replace the column heading with the class value if the class column value is the only class column value requested in the given query. 
             3. Prepend the column heading with the class value if the class column value is the only class column value requested in the given query. 
             4. Append the column heading to the class value if the class column value is the only class column value requested in the given query. 
             5. Include or exclude a column based upon the queried values in a class column. 
           
         
       
    
     6. Drop the column if it is not used for the associated class column values selected. 
     C. Class Column Behavior Based on Values in the Data Table 
     
         
         
           
             1. Drop a column if the data table only contains a single value for the column.
 
D. Implicit Associated Column Behaviors Based on Query
 
             1. Replace a column heading with a different pre-defined value if the associated class value is in a predefined set of values.
 
E. Some Columns Are Always Dropped
 
             1. Do not include the column in the report table by default. 
           
         
       
    
     The B-rules are not mutually exclusive, except that rule 2 cannot be applied at the same time as rules 3 or 4, and rules 3 and 4 are not generally applied at the same time. 
     The original column names as they appear in an input table object are optionally replaced by other more readable text in a column heading in an optimized output table object. The improved text for column headings presented to the user is encoded in a table properties object associated with the input table object in the preferred embodiment. 
     The substitution of presentation values is not a necessary feature required for optimizing table columns, but is a useful embodiment. Substitution of presentation values is applicable when the set of possible values is known (or partially known) prior to generating the report table layout, so that a set of substitutions can be pre-defined. 
     Within the preferred embodiment, other predefined transforms are allowed. Transforms can include capitalization changes and pluralization changes. For example, when the unique class column value is “Response time” and the associated default column heading is “Mean”, “Mean” becomes “Response time mean” or “Mean response time” using rule 3 or 4 above. 
     Another embodiment of the present inventive technique to modify the table title to reflect a unique class identifier column value selected. 
     The D-rule is closely related to the B-rules. However, in the D-rule, the associated class value is determined not explicitly from a class identifier column but instead is deduced from the values queried in some other column. For example, in a particular data tables there is a component name (instance) column, but there is not a corresponding component type (class) column. Instead, the component type is encoded into the statisistic names in the statistic type column. There will be several statistics for each component type and each statistic name pertains to only a single component type. The statistic type column is examined to see if all selected statistics belong to the set of values pertaining to the same component type if the component name column is to show the component type when all of the selected statistics refer to a single component type. Values for class identifier columns can be replaced with presentation values defined in a table properties file. 
     In one embodiment, column behavior may be specified by code written by the user to implement the behavior in a static rules formation process. In an alternate embodiment, the column behaviors are specified by listing the behaviors in a properties file or other data structure and defined at run-time to execute dynamic rules that are responsive to input data table and rules  1400 . In other embodiments, column behaviors may be specified as some combination of both. 
     To further illustrate the table optimization process, a specific example is given in  FIG. 9  with some associated visible output screens for various queries in  FIG. 10 . 
     In  FIG. 9 , an example of a table object  1710  is shown that has associated with it a list of available column objects and presentation names  1720 , a list of class column associations  1740 , a list of statistic value associations  1730 , and a list of specific rules  1750  that are executed on the table object  1710  to create a new optimized output table. 
     The available column objects and presentation names  1720  for the present example are enumerated by listing the column object on the left (e.g., BFSummary.col.Subsystem.pName) and the presentation name on the right (same e.g. as previous, Subsystem) with an equivalence symbol=between them. All of the column objects available for export are listed in this way; the list resides in the table properties, such as table properties  1230  or  1330  shown in  FIGS. 7 and 8 . 
     The class column associations  1740  for the present example are enumerated showing the available class columns on the left (e.g., statistic) and the associated column names on the right (e.g., Mean). 
     The value associations  1730  for the present example are enumerated by listing the statistic classes on the left (e.g., bfResourceStatsList) and the associated value, in this case statistic value on the right (e.g. WriteCount). 
     Specific rules  1750  are formed from the available column objects and presentation names  1720 , class column associations  1740 , statistic value associations  1730  and rules  1400  which are A-rules, B-rules, C-rules, D-rules, and E-rules defined previously. For example, “Run name” in list  1720  appears as a presentation name in the table object  1710 , so that the C.1 rule must be included in the rules list. A second example is for the column “Maximum”. The B.5 rule for “Maximum” must be included in specific rules  1750  since “Maximum” appears as an associated column in list  1740 . Furthermore, if a query that gets processed later includes any non bfResourceStatsList statistics, then the “maximum” column is kept and placed in the optimized output table (as well the minimum, total, num intervals and duration columns). Taking this example of B.5 rule further, if “CPUUtil” is selected as a part of a subsequent query, then “mean”, “minimum”, “maximum”, “total”, “num intervals” and “duration” columns in the input table will be included in the optimized output table (:“mean” is included because rule B.4 is a part of the specific rules  1750 , but only on the condition that “CPUUtil” is the only associated value in the query). 
     In  FIG. 10  is shown an optimized output table  800  that is automatically generated by applying the specific rules  1750  in the processing of an instance of table object  1710  when the query for “bfResponseTime” statistic is selected from a tree view. The rules  810  to obtain table  800  are shown in  FIG. 10 . The output table  800  is a screen image of a window generated by the report visualizer program  200  within a Microsoft Windows operating environment. Without the table optimization process using rules  810  the output table  800  would have looked like unoptimzed table  820  also shown in  FIG. 10 . However, the unoptimized table  820  would have been even less attractive had the column names in the data table been less readable. 
     Referring to  FIG. 9  and the rules  810  in  FIG. 10  the automatic generation of output table  800  by application of rules is explained. Although the instance of table object  1710  is created in a particular measurement run, there is no run number column in the output table  800  because the  810  C.1 rule eliminated it (since it would be redundant information, i.e., all run numbers would be the same in the output table if they were not eliminated). The  810  D.1 rule applies to the association of “bfResponseTime” with the class identifier “business function” which is an implicit association not called out in table properties for table object  1710 , but known to the table layout generator  1270  shown in statistic value association list  1730 . The bfStatsList classification results in “business function” as the implicit component type (and therefore a potential column heading for component column). There is no subcomponent column. The “bfResourceStatsList” classification results in “business function” as the implicit component type (and therefore a potential column heading for component column). The subsysStatsList classification results in “Computer” as the implicit component type (and therefore a potential column heading for component column). The result of the  810  D.1 rule is the appearance of the “business function” column header and column in output table  800 . The next rule applied is the  810  B.5 rule, which drops the subcomponent information about the system since “subcomponent” does not appear explicitly in the statistic type class column associations  1740 . The process proceeds to execute rule  810  A.2 which eliminates a column of text values, in this case “response time” that describe the statistic for which the “mean”, “maximum”, “total”, etc. statistics are given in each row. As in the run number example, keeping “response time” in every row would be redundant in the output table, so it is better to rename the “mean” column holding the returned response time mean values to “mean business function response time”. This is the task of rule  810  B.4. Finally, rule  810  B.5 executes performing the process of including the “minimum”, “maximum”, “total”, “duration” and “num intervals” columns in the final table.  810  B.5 includes these because according to specific rules  1750  those columns are to be included if a non bfResourceStatsList value is selected for query. Since “bfResponseTime” is a value associated with “bfSTatsList”, the other associated statistic columns defined in class associations  1740  are included and appear in the righthand part of the output table  800 . 
     An example of an output table generated in the previous situation is also shown in  FIG. 10  as table  820 . There are empty columns (subsystem) and several columns with redundant information (statistic, run time) of no value to the user. Also, the column headings do not explain clearly what they pertain to. 
     A description of the virtual database  282  function for table joining within the report visualizer  200  is shown in  FIG. 11 . Virtual database  1550  functions to automatically join a primary data table  1506  to zero or more secondary data tables  1507  to form a single query result data table  1508 . Primary data table  1506  is composed of a multiplicity of columns containing a multiplicity of rows of data cells. Secondary data table  1507  is likewise composed of a multiplicity of columns containing a multiplicity of rows of data cells. Query result data table  1508  is similarly composed of a multiplicity of columns containing a multiplicity of rows of data cells. The number of columns and rows in each data table may differ from one to the other. 
     Query  1509  requesting a particular set of data is sent to virtual database  1550 , and designates primary data table  1506  and zero or more secondary data tables  1507  from which to draw the prescribed set of data. Query  1509  expects a new data table to be returned that is comprised of all the data matching the query specification that can be found among the specified tables. Virtual database  1550  first sets the query  1509  against primary data table  1506  and puts the queried content into a newly constructed Data Table  1508   a . Virtual database  1550  performs a series of join instructions to match content from data tables  1507 ; the matched content  1508   b  is appended to the queried content  1508   a  to form query result data table  1508  which is returned to query  1509  as the processed result. 
     A UML sequence diagram is shown in  FIG. 12  describing a multiple data table query  1500  in the preferred embodiment. The vertical axis of the diagram indicates increasing time going from top to bottom. The horizontal axis of the diagram indicates movement from one program entity to another; the program entities are visible in the diagram as user  1510  for program control, report wizard  1520  for program control, report view  1530  for requesting and displaying reports, report definition  1540  for building report queries, virtual database  1550  for organizing data tables and performing queries on them, primary data table  1560  which holds relevant data and operates on it, secondary data table  1570  which holds relevant data and operates on it and join specification  1580  which holds and executes instructions to join primary data table  1560  to zero or more secondary data table  1570  together. Vertical dashed lines indicate the timeline behavior associated with each program entity. Along each timeline a set of time periods labeled  1511 ,  1521 ,  1531 ,  1541   a ,  1541   b ,  1551 ,  1561 ,  1571  and  1581  are shown, each time period indicating that the program entity above it is operating during that time period, either operating specifically on data or waiting for another process to complete and return. Arrows shown from left to right indicate requests or pieces of information transferred or program control between entities and will be explained below. 
     Beginning at the top left side of the diagram and progressing to the right, user  1510  completes a defining or refining process in report wizard  1520  to finish a report design  1512 . Report Wizard  1520  sends the data and issues a command  1522  to build a report to report definition  1540 . Report definition  1540  during time period  1541   a  assembles all of the report definition information in Report object  225  (queries, transforms, filters, table layout, chart layout) into its internal representation. In particular, the representation of the query  1542   a  against the selected data tables is stored in an appropriate format ready to be sent to Virtual Database  1550 . 
     At a later time, report wizard  1520  signals report view  1530  to create a visible report via signal  1524 . Report View  1530  tells report definition  1540  to get the data corresponding to report definition  1512  and return the report data via signal  1532   a . Report definition  1540  upon receiving the instruction  1532   a  to get the data, sends the previously constructed query  1542   a  to virtual database  1550 . Virtual database  1550  accepts query  1542   a  and executes it as query  1552   a  against the primary data table  1560 . Primary data table  1560  assembles the data corresponding to query  1552   a  during time period  1561  and returns the data  1552   b  to virtual database  1550 . A query is then executed against the secondary table  1570 —this query essentially consisting of a command  1554   a  to send all of the data in secondary data table  1570  back to virtual database  1550 . The data  1554   b  from secondary table  1570  is returned and the virtual database then begins the process of appending the secondary table  1570  data  1554   b  to the returned query data  1552   b.    
     The append or join process is accomplished by virtual database  1550  when it calls join specification  1580  with a join command  1556   a . Join specification  1580  completes the join process and returns the joined data table  1556   b  to virtual database  1550 . 
     The joined data table  1556   b  is sent back to report definition  1540  as table  1542   b ; report definition  1540 , in turn, assembles the report data from the table  1542   b  and other attributes of the report definition  1540  and sends the report data  1532   b  to Report View  1530  for viewing. report view  1530  creates a visible report and displays it on a computer monitor as described previously but not shown. 
     The virtual database  1550  has a structure for managing data tables which is shown in  FIG. 13 . The virtual database  1550  is composed of internal database  1630  which manages a list of the data tables  1620  and provides a central point of access to the data in said data tables  1620 ; a data source manager  1614  which functions to control, load and populate data sources  1618  and has a concrete DSMgr implementation  1615  for specifically working with different program entities; data source  1618  which contains, creates, populates and manages a set of data tables  1620  representing the data from the data source  1618  and implements functionality common to all concrete data sources  1619 ; data tables  1620  that contain a row and column representation of the data and support the accessing of this data as required by the database and certain properties of the data tables as required to support queries, joins and table layout optimization and has a concrete data table  1621 ; a table join specification object  1624  that is a structure for holding and executing concrete join specs  1625 ; and concrete join specs  1625  that are associated with each concrete data table  1621 . 
     Virtual database  1550  exists within a project document  1610  of the same kind as project document  300  previously described. Project document  1610  initializes virtual database  1550 , by specifying and loading the concrete DSMgr  1615  and concrete data sources  1619  and through reports definition modules, queries virtual database  1550  for data from data tables  1620 . 
     The concrete DSMgr implementation  1615  utilized in the context of the preferred embodiment of the present invention is specifically coded to work with the report visualizer  200 . The concrete DSMgr  1614  organizes the data sources and data tables appropriately for project document  1610  and the Data Sources  1618  in turn make their data tables  1620  available to the database  1630 . In the preferred embodiment, the database manager  1614  utilizes a Java plug-in architecture which accepts standard jar files to package the concrete data sources  1619  and concrete DSMgr implementations  1615 . Other embodiments are conceived whereby the database  1550  is used with other programs to perform similar functions to those described; this being accomplished by coding a specific concrete DSMgr implementation for the program of interest. 
     Concrete Data sources  1619  which are loaded into the data source  1618  can be designed to load data from spreadsheet files, text files, binary coded data files, other databases, and data streams such as those with FTP protocol specifications. All of these data sources are encapsulated in a Java .jar package along with information about the structure of the data to be loaded and how that data is to be represented in the data tables  1620  in the virtual database  1550 . Other types of data sources may be conceived in other embodiments of the present invention. 
     Concrete data tables  1621  are constructed from data sources  1618  as instances of the database data tables  1620 . Concrete data tables  1621  are made up of a multiplicity of columns containing a multiplicity of rows containing data within each row. 
     Associated with each concrete data table  1621  is one or more concrete join specs  1625  containing specific instructions pertaining to which data tables  1620  may be joined with it and rules for how those tables are joined together. The specific information is coded into a table join specification for execution by the database  1550  as described previously. The rules for joining will be described further. 
     Table 1 contains a pseudocode implementation of the join process which occurs near the end of the time period  1551  in the process diagram of  FIG. 12 . In Table 1:line 1, the assumptions for the pseudocode to operate are given, namely that a primary table with a set of rows is supplied and secondary table with a set of rows is supplied and that the two will be joined into a resultant table. Note that the pseudocode given will also work in the case when no secondary table is supplied since there will be zero rows from a secondary table. The step numbers that follow correspond to line numbers in Table 1. Step 2 implements a flow control For loop that cycles through all the rows in the given primary table. Step 3 implements a nested flow control For loop that cycles through each row in the given secondary table. Inside the nested For loop on the secondary table, a set of operations occur pending the condition that the row values match. The check for a row value match is step 4; the definition of a row value match will be discussed further in the next paragraph. If the row value does match then the data from the secondary table row is appended onto the matching row in the primary table in step 5 to augment the join result. Step 6 checks if it is possible to drop rows taken from the secondary table for join execution efficiency; if so then the row of the secondary table just used to create a join row is removed from the set of secondary rows being considered as join candidates. Step 7 performs a similar function to step 6: it checks that only one match is allowed per primary row table and if so the outer loop increments to the next row in the primary table and the process moves forward with step 3 at the first row of the secondary table. Steps 8, 9 and 11 terminate the structures of steps 4, 3 and 2, respectively. Step 10 automatically adds any primary row to the join result that does not match the secondary table. Step 12 returns the join result set to the calling program. 
     
       
         
               
               
             
           
               
                 TABLE 1 
               
               
                   
               
             
             
               
                 1 
                 Assumption: We have a set of rows from a primary table and a set 
               
               
                   
                 of rows from a secondary table that we are trying to join into a 
               
               
                   
                 join result set. 
               
               
                 2 
                 For each row in primary table result 
               
               
                 3 
                   For each row in secondary table result 
               
               
                 4 
                     if rows match then 
               
               
                 5 
                       create join result row by appending secondary row 
               
               
                   
                       to primary row. 
               
               
                 6 
                       if only one match allowed for secondary table row, 
               
               
                   
                         then remove row from secondary table result 
               
               
                 7 
                       if only one match allowed for primary table row, 
               
               
                   
                         then exit inner for 
               
               
                 8 
                     end if 
               
               
                 9 
                   end for 
               
               
                 10 
                   if no match found for primary row, 
               
               
                   
                     then add primary row to join result set. 
               
               
                 11 
                 end for 
               
               
                 12 
                 return the join result set. 
               
               
                   
               
             
          
         
       
     
     An Example of the handling of join rules for matching rows is shown in  FIG. 14 . There are two classes of rules shown, those that are general join handling rules  1680  and those that are more specific and complex join handling rules  1690 . The goal of these rules is to determine whether primary table row and a secondary table row match. Lines  1681  through  1687  form the general rules. Lines  1691  through  1694  form specific complex rules. 
     A concrete properties file (not shown) packaged with a concrete data source object  1619  forms a concrete join spec  1625  that can fully specify some joins. This is illustrated in the present example in  FIG. 14 , where three tables, “runs” table, “BFSummary.BFSummary” table, and “ScenerioResults.Runs” table are involved in the specification. The properties within the joinTableList attribute of the table “runs”  1681   a  indicate what tables can be joined to, namely “BFSummary.BFSummary”  1681   c  and “ScenarioResults.Runs”  1681   d  which are listed in the joinTableList attribute  1681   b  in line  1681 . Each joinable table indicates which columns are the key columns that must exactly match values in the other table. For example, in lines  1682  and  1683 , “runNumber” is the name of the key column that is used to join tables “BFSummary.BFSummary” and “runs” table. The “runs” table column name(s) used in the join is specified in property “myJoinColumns” as “runNumber”. The column name(s) for the columns from table “BFSummary.BFSummary” that correspond to the columns specified in “myJoinColumns” is listed in property “matchCol” as “runNumber”. The column names for the join columns from the two tables happens to be the same in both tables (“runNumber”), but need not be. More than one column could have been listed in “myJoinColumns” and “matchCol” to indicate that the values for multiple columns were to be compared with the corresponding column from the other table when trying to join rows. Similarly lines  1685  and  1686  specify the key columns for joining tables “Runs” and “ScenarioResults.Runs”. Reverse roles attribute, line  1684  and line  1687  indicate whether the role of primary and secondary table can be reversed in the join process. The join is specified from the perspective of a secondary table; in this case “runs” is secondary. If “runs” is primary, the reverseRoles attribute is set to true. As shown, “runs” is secondary in the case of joining with “BFSummary.BFSummary” by line  1684  but primary in the case of joining with “ScenarioResults.Runs” by line  1687 . 
     Referring to  FIG. 14 , complex rules example  1690  in the following description. Two tables are involved in a complex join specification  1690 , namely “BFSummary”  1691   a  and “ScenarioResults.Statistics”  1691   c . In particular, “ScenarioResults.Statistics”  1691   a  table is included in the joinTableList attribute  1691   b  of the “BFSummary” table  1691   a . The join of “BFSummary” to “ScenarioResults.Statistics” cannot be fully specified in the concrete properties file&#39;s properties in the current implementation because the join is not a simple test for exact match in the values of one or more columns. To handle this case the concrete properties file (stored within the concrete data source object) identifies the class that implements the join (statement  1692 ); namely: 
     ipsvisualizer.ipsexplorerplugin.bfsummary.BFSummaryJoinSpec. 
     In this example of the preferred embodiment, that class was hand coded and packaged within the data source object. In statement  1693  and statement  1694 , the equivalentValues properties are used to identify equivalences between Statistics column values of the two tables, for example “bfrhruput” in the secondary table with “throughput” in the primary table. These value equivalences are used in the joinSpec class that implements the join (statement  1692 ), namely: 
     ipsvisualizer.ipsexplorerplugin.bfsummary.BFSummaryJoinSpec 
     when checking to see if rows are equivalent, i.e. if they match. 
       FIG. 15(A , E) is a printout of a tabular list  1050  of the standard report templates that are available within the preferred embodiment of the present invention. In the first column  1051  of the list  1050  is a set of row numbers which will be used to refer to information within the list  1050 , for example rows  15 - 1  and  15 - 2  in  FIG. 15A  indicate that there are 92 report folders available,  244  reports and 11 system questions available in the preferred embodiment. Report folders contain a number of report templates. The second column  1052  lists a set of report names available to the user. The third column  1053  identifies if the report name in the second column  1052  is a report folder. The fourth column  1054  identifies if the report name in the second column  1052  is a report template. The fifth column  1055  identifies the report name in the second column as a system question. Examples of information in  FIG. 15  will be explained in conjunction with  FIG. 16 . The templates are the templates  275  in this description. The system questions and report folders are the System Questions  205  in this description. 
     In FIG.  16 (A,C) is an example collection of a visible reports showing several of the types of reports that can be generated by the invention. The set of visible reports has been generated using the preferred embodiment of the invention and in particular correspond to certain report templates selected from the list  1050  in  FIG. 15 . The visible reports are the Reports  235  in this description. The data used to generate  FIG. 16  is indicative of a certain set of tests on a multiplicity of business function performances in a network of systems. 
     In  FIG. 16A  is pie chart report  16 - 51  of the “business function mix” showing the percentage of business function throughputs per business function in a mix of three business functions. The pie chart report  16 - 51  is generated by selecting and running report template “business function throughput chart” shown in  FIG. 15A , row  15 - 51 . 
     In  FIG. 16A  is comparison bar chart report  16 - 145  of the “business function response time compared to objective”. The bar chart report  16 - 145  is generated by selecting and running report template “business function response time evaluation compared to objective chart” shown in  FIG. 15B : row  15 - 145 . Report template  15 - 145  is one of several reports generated by the system question “How does performance compare to the objectives?” shown in  FIG. 15B : row  15 - 143 . It indicates the 90 th  percentile statistical business function response time (solid) in comparison to an objective business function response time. 
     In  FIG. 16B  is table  16 - 146  of the “business function response time evaluation”. It is also one of several reports generated by the system question “How does performance compare to the objectives?” in  FIG. 15C : row  15 - 146 . The table  16 - 146  is also generated by selecting and running report template “business function response time evaluation” shown in  FIG. 15C : row  146 . It indicates a FAIL, PASS, OR CAUTION situation for each business function based on their measures response times. 
     In  FIG. 16B  are dual bar graphs  16 - 181  of the “application profile: network bytes transmitted subsystem details” showing request transmissions and reply transmissions in bytes from sixbusiness functions on three servers, DBserver, Webserver and Appserver. The dual bar graphs  16 - 181  were generated by selecting and running report template “Network bytes transmitted subsystem details” shown in  FIG. 15C : row  181 . 
     In  FIG. 16C  is stacked bar graph  16 - 258  of a “T 1  Run Comparison” showing mean response times in several runs of a business function and their breakdowns on three servers and a client, DBserver, Webserver, Appserver and Client. The bar graph  16 - 258  was generated by selecting and running report template “Response time subsystem details” shown in  FIG. 15D : row  258 . 
     In  FIG. 16C  is dual axis line graph  16 - 316  of a “T 3  Run Comparison” showing business function throughput and CPU utilizations for ten business functions running on three servers, DBserver, Webserver and Appserver. The bar graph  16 - 316  was generated by selecting and running report template “Throughput vs. response time for T 3  Comparison” shown in  FIG. 15E : row  316 . Line graph  16 - 316  indicates one of the motivations for the present invention of taking the complex function of collecting the test data for three different variables from at least 13 data sources and combining that data through the table query and table join mechanisms and report template manipulations taught herein to form a report that is useful for understanding in the present context, for example, which servers need to be upgraded, what business functions consume the most resources and how the system scales with the number of users. 
     The implementation of the processes in the preferred embodiment is accomplished using a set of Java applications in a Java application framework that interact together to produce the overall program. The Java applications code exists in computer memory and runs on the computer&#39;s CPU (or multiple CPUs) utilizing the various resources of the computer, including computer memory, hard disk drives, graphics display units and network interfaces. The Java applications code may also utilize resources attached to a network connected to the computer such as application servers, storage servers or database servers. Other embodiments may use other object oriented programming languages, or structured languages, or hardcoding in firmware, or some combination to implement parts or the whole of the present invention. As is well-known in the art of computer programming, objects generated within an object-oriented language may encapsulate data structures and may contain methods to manipulate those data structures and perform other operations. Objects and modules refer to entities that contain data structures and that may contain executable code to perform operations on those data structures. A “program” refers to certain combinations of objects (or modules), the logical information flow between the objects and the process by which the objects interoperate to perform the functions described. 
     While this invention has been described in reference to a preferred embodiment along with other illustrative embodiments, this description is not intended to be construed in a limiting sense. Various modifications and combinations of the illustrative embodiments, as well as other embodiments of the invention, will be apparent to persons skilled in the art upon reference to the description. It is therefore intended that the appended claims encompass any such modifications or embodiments.