Abstract:
A system and method for software test is disclosed that allows user management and adaptation of test procedures and the resulting test data. In an embodiment, the system and method provide a cross-platform user interface which allows testing to be conducted on a plurality of platforms, i.e. it is not integrated with a single platform or language (e.g., C++, Visual Basic, Java, or the like). The method further allows a user to customize a predetermined set of system characteristics relating to the stored test procedure. It is emphasized that this abstract is provided to comply with the rules requiring an abstract which will allow a searcher or other reader to quickly ascertain the subject matter of the technical disclosure. It is submitted with the understanding that it will not be used to interpret or limit the scope of meaning of the claims.

Description:
RELATION TO OTHER APPLICATIONS  
       [0001]    The present invention claims priority through U.S. Provisional Application No. 60/358,993 filed on Feb. 21, 2002 for “A Software Test Creation, Validation, And Management System.” 
     
    
     
       FIELD OF THE INVENTION  
         [0002]    The present invention relates to automated software application testing.  
         BACKGROUND OF THE INVENTION  
         [0003]    Automated testing comprises automating a previously manual testing process. Typically, automated testing tools require that a formalized manual testing process currently exists, e.g. a set of detailed test cases and test scripts, including a set of results expected for a given test case and test environment which provides an initial, known test bed that allows the test cases to be repeated from a known starting place. Test cases are most often derived from a set of system requirements to be verified as being satisfied by the system under test.  
           [0004]    Commercially developed software testing tools have design constraints, e.g. they are specifically designed to be Internet web testers, performance testers, regression testers, script generators, test data generators, and the like. These tools have limited customization capabilities. Moreover, these tools are often static with respect to tailorability to a given test bed.  
           [0005]    However, application systems and environments differ, e.g. in platform, application requirements, and user tastes and desires. Testing methodologies and procedures also vary greatly and often require adaptability to meet the user&#39;s quality needs and certification goals.  
           [0006]    A need exists for a standardizable format and/or process that allows integration of software application testing tools and the software application testing process with other applications such as office applications, e.g. word processors, spreadsheets, and the like. Test results need to be stored in a manner that supports integration with office tools such as word processors. Such integration would further allow sharing and using test systems and their data by a group of people. Sharing is often needed when a group collaborates in a testing effort.  
           [0007]    There is also a need for flexible reporting, e.g. user tailorable, ad hoc result reports.  
           [0008]    System test tools also need to be able to gather and/or generate test metrics and make these metrics available for post testing analysis, e.g. metrics comprising comparisons of test data results between two iterations of project versions, integration into the test system of defects found later by users or technical support, reliability calculations, and the like.  
           [0009]    Accordingly, a test system which provides for testing to a desired test level and thoroughness but which is adaptable to the system under test is desirable. As opposed to current test systems which most often require the system under test to be architected or otherwise modified to work with a test tool, a test tool should itself be able to be modified and adapted to work with the system under test. Such adaptation should provide for management and customization of the test system, including management and customization of test procedures, resulting test data, and presentation of the resulting test data. Ideally, the user interface that allows access to the test system&#39;s functions should be customizable as well to adapt to the system under test.  
         SUMMARY  
         [0010]    A system and method for software test are described which allow a user to manage and customize a test system comprising test procedures and resulting test data. Customization of the system facilitates a user&#39;s ability to add and/or modify fields, tables, and reports to the system as well as change the look and operation of the user interface and its menus.  
           [0011]    In an embodiment of a method, a first test procedure is stored for later use in user assisted testing of a set of functions of an executable application. A test system user may be allowed to customize a predetermined set of system characteristics relating to the stored test procedure. Test data may be acquired and task steps to be performed by the first test procedure may be identified. An executable second test procedure, corresponding to the first test procedure, may be created to automatically exercise a set of functions selected for the executable application. Result data may also be acquired.  
           [0012]    Given its adaptability, the present invention is adaptable to a business&#39; testing needs and development standards.  
           [0013]    The scope of protection is not limited by the summary of an exemplary embodiment set out above, but is only limited by the claims  
       
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0014]    The features, aspects, and advantages of the present invention will become more fully apparent from the following description, appended claims, and accompanying drawings in which:  
         [0015]    [0015]FIG. 1 is schematic overview of an exemplary system;  
         [0016]    [0016]FIG. 1 a  is an exemplary system requirements user interface;  
         [0017]    [0017]FIG. 1 b  is an exemplary test scenario user interface form;  
         [0018]    [0018]FIG. 2 is a flowchart of an exemplary method of the present invention;  
         [0019]    [0019]FIG. 3 is a flowchart of an exemplary method of creation of an executable test application created for managing a user adaptable test procedure;  
         [0020]    [0020]FIG. 4 is an exemplary user interface form showing a primary user access form;  
         [0021]    [0021]FIG. 5 is an exemplary user interface form to tailor system information;  
         [0022]    [0022]FIG. 6 is an exemplary user interface form to define actions to be undertaken;  
         [0023]    [0023]FIGS. 7 a  and  7   b  are exemplary user interface forms to configure aspects of the test system;  
         [0024]    [0024]FIG. 8 is an exemplary user interface form to relate a test procedure to a data element;  
         [0025]    [0025]FIGS. 9 a  and  9   b  are exemplary user interface forms to expand the system; and  
         [0026]    [0026]FIG. 10 is an exemplary user interface form useful to generate a test script shell. 
     
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT  
       [0027]    In general, throughout this description, if an item is described as implemented in software, it can equally well be implemented as hardware. It is also understood that “data,” as used herein, is either singular or plural as the context requires. As further used herein, “certification” comprises processes used to verify and validate that a desired feature or function of a system under test is present and performs according to a predetermined set of performance criteria, e.g. produces a correct result in the presence of data.  
         [0028]    As used herein, the following terms have the following meanings, as will be understood by those of ordinary skill in the software application testing arts. A “test case” is a set of test inputs, execution conditions, and expected results developed to validate or otherwise verify obtaining a given set of results for a predetermined set of input data and conditions. A “test procedure” is a set of detailed instructions for setting up, executing, and evaluating results obtained from executing one or more test cases. A “test script” is a human or computer readable set of directives and instructions that describe execution of all or a portion of a test procedure. An “automated test script” is a test script that is executable by a computer with little or no manual input required. A “test requirement” is a condition or capability needed by a user, system, or portion of a system to solve a problem or achieve an objective. Test requirements may include a condition or capability to be met or possessed by a system or system component to satisfy a contract, standard, specification, or other formally imposed documents.  
         [0029]    Referring now to FIG. 1, in an exemplary embodiment, system  10  for creating and maintaining test procedures  25   b  comprises user customizable certification database  20  and user interface  30 . System  10  is adapted to allow a user to customize a predetermined set of system characteristics  12  relating to certification database  20 . Executable application  40  is the software and/or system to be tested.  
         [0030]    Certification database  20  further comprises requirement repository  22  which contains requirements  22   a  to be tested for executable application  40 ; test procedure repository  24 , which, in a preferred embodiment, may be linked to requirements repository  22 ; and test results repository  26 . Database  20  may be used to import and export data from other software systems such as a third party&#39;s computer aided software engineering tool or performance testing tool.  
         [0031]    A user may customize system  10  by manipulating fields  21   a , tables  21 , reports  23 , and even the look and feel of user interface  30  and its menus  36 , thus adapting system  10  to a business&#39; or application system&#39;s requirements. As used herein, software application requirement repository  22 , test procedure repository  24 , and test results repository  26  may be tables  21  within database  20 . Accordingly, software application requirement repository  22 , test procedure repository  24 , and test results repository  26  may be manipulated as tables, e.g. by modifications of one or more field  21   a  in a record in software application requirement repository  22 , test procedure repository  24 , and test results repository  26 .  
         [0032]    Manipulatable system characteristics  12  may include user customizable characteristics relating to stored test procedures  25   b . For example, a user may manipulate field  21   a  in table  21  such as by changing its type or other characteristics, manipulate table  21  as a component of customizable database  20  such as by adding new fields  21   a  to existing fields  21   a  in table  21 , manipulate report  23  such as by adding detail or formatting, manipulate user interface  30  such as by altering the appearance of user interface  30 , and the like, or a combination thereof. As used herein, manipulate comprises adding, deleting, modifying, or the like, or combinations thereof.  
         [0033]    Referring now to FIG. 1 a , as illustrated in an exemplary user interface to test requirement repository  22  (FIG. 1), test requirement repository  22  (FIG. 1) contains brief statements  22   b  that may be accessed and used to describe what is to be accomplished in a predetermined phase of testing such as unit testing, integration testing or system testing as well as supporting data for that phase.  
         [0034]    Referring back to FIG. 1, test requirements  22   a  may be stored in tables  21  in database  20 . For example, a test requirement  22   a  might be to verify that the software or system may be installed, to ensure that executable application  40  is operational after it has been installed, or to determine that concurrent users may access and update data without causing conflicts or corrupting data. Supporting data may include, but is not limited to, dates on which testing is to occur, priority of each requirement, owner, instructions as to how to accomplish testing, and the like, or combinations thereof.  
         [0035]    Test procedure repository  24  comprises elements  25 , where elements  25  may comprise pointers to test requirements  25   a , test procedures  25   b , test references  25   c , or the like, or a combination thereof. Test procedures  25   b  are typically step-by-step instructions which may be manual, automated, or partially manual and partially automated. Test procedures  25   b  define or otherwise describe how to perform a specific test, e.g. test procedures  25   b  may comprise one or more steps that describe a specific action to take as well as define the expected result that should occur after the action is taken. Test references  25   c  comprise information that may be used to associate test procedures  25   b  with one or more system objects such as screen names, executable programs, and databases. These references may be used to create a traceability report to show the relationship of various system components to test procedures  25   b . For example, if a programmer changes a screen displayable form called “Main Menu,” a report may be used to show which test procedures  25   b  reference that form. Those test procedures  25   b  may be rerun (or modified) to regression test the changes to the “Main Menu” form.  
         [0036]    In a preferred embodiment, test results repository  26  may be present to contain retrievable test results  26   a , and each element  25  in test procedure repository  24  may be linked to at least one requirement  22   a  in software application requirements repository  22 . A form  34 , an example of which is shown in FIG. 1 b , may be used to define and/or otherwise maintain test scenarios, e.g. actions undertaken in test procedure  25   b  and a test result expected in response to that action.  
         [0037]    User interface  30  is adaptable to allow the user to customize a predetermined set of system characteristics  12  relating to system  10  and its certification database  20 . For example, user interface  30  may comprise user customizable appearance characteristics of graphical interface  32 , e.g. colors, content  34  of graphical interface  32 , appearance characteristics of menu  36 , and content  37  of menu  36 .  
         [0038]    User interface  30  may comprise menus  36 , sub-menus, and various controls that are used to value each of the data elements of the system and to initiate execution of processing options such as report generation. Any of the objects of user interface  30  may be modified to accommodate the capture of different data and/or the exercising of new processing features. This allows system  10  to be adapted to a predetermined look and feel and operation required to integrate smoothly with internal testing procedures and standards of the enterprise.  
         [0039]    In a currently preferred embodiment, system  10  employs the ACCESS™ database management system manufactured by Microsoft® Corp. of Redmond, Wash. In this currently preferred embodiment, system  10  may be integrated into a general office system, e.g. it may import items from and export items into general office applications such as word processors. “Items,” as used herein, include descriptions of application and function test requirements, test results  26   a , and the like.  
         [0040]    However, other equivalent database management systems may be used, e.g. those which support the Open Database Connectivity (ODBC) and/or an application programming interface (API) for database access such as to SQL Server™ marketed by Microsoft Corp.  
         [0041]    In the operation of an exemplary embodiment, in general, in a preferred embodiment, prior to using system  10 , a user may elect to review internal testing policies, test requirements  22   a , test procedures  25   b , and test metrics to be captured to substantiate the testing efforts. As used herein, a “test metric” is a data or statistic that measures some event. For example, test metrics may comprise a measure of hours spent testing, defects encountered by root cause (program error, documentation error, user confusion, etc.), defects encountered by module or application under test, defects encountered by test procedure, defects encountered by their severity (critical, major, minor, etc.), percent of test procedure steps that passed versus failed, pass/fail counts by test procedure or test procedure step at the requirement level (e.g. 98% of test procedure steps for procedures linked to requirement “x” passed), defects encountered by hours or days of testing, defects encountered by tester, and the like, or combinations thereof. Test metrics may also be combined with other metrics gathered outside of a testing effort to arrive at additional metrics such as defects by lines of code (KLOC) or changed lines of code, defects by programming organization/group, defects for this version of the product versus the prior version, defects for this testing effort compared to other testing efforts, or the like, or combinations thereof.  
         [0042]    After review, a user may elect to modify desired elements of system  10  to comply with the desired testing policies and test procedures  25   b , e.g. a data element, user interface  30 , report  23 , and the like. Further modifications may be made as well such as to import/export desired elements using one or more additional supported tools, e.g. requirements repository and automated testing tools.  
         [0043]    If necessary, a user may further assess external testing standardization requirements needed for testing, e.g. the International Standards Organization&#39;s ISO 9000, and make such other modifications to elements of system  10  to ensure that test results  26   a  will satisfy those external testing standardization requirements.  
         [0044]    For a particular testing engagement, a user, having modified system  10  as described above, initializes database tables  21   a  as needed, e.g. creates new tables  21   a , reuse tables  21   a  from prior tests, and the like. Test requirements  22   a  are input and/or imported, as are test procedures  25   b , which may be linked to test requirements  22   a  using data entry form  30  (FIG. 1 a ). A predetermined number of linked test procedures  25   b  may be linked to application specific data elements to be tested, e.g. screens, modules, and the like such as by using the form shown in FIG. 8.  
         [0045]    Once initialized and linked, execution of test procedures  25   b  may be initiated to conduct manual testing. Test results  26   a  and timing data may be captured, e.g. input and/or imported into one or more test result tables  21   a . If required, one or more reports  23  may be created to substantiate the testing efforts.  
         [0046]    If further desired, automated test script shells may be created from the validated and tested manual test procedures  25   b , where “shell,” as will be familiar to those of ordinary skill in the software arts, is a rudimentary prototype of a test script which may be completed at a later time. An exemplary interface used to control the generation of automated test scripts is shown in FIG. 10, including specifying and/or otherwise manipulating test script generation options. Automated scripts may be used at a later time, e.g. for stress, regression, or performance related tests.  
         [0047]    Metrics from manual and automated testing may be used by a quality control system, e.g. data from system  10  may be exported into other systems and analyzed as needed.  
         [0048]    Referring now to FIG. 2 is a flowchart of an exemplary software test method for managing user adaptable test procedures and resulting test data according to system  10 . A user stores, step  200 , first test procedure  25   b  (FIG. 1) for use in user assisted and/or automated testing of a set of components of executable application  40  (FIG. 1). Individual components of executable application  40  (FIG. 1) may comprise an executable procedure used by executable application  40 , a function implemented by executable application  40 , a user interface form used by executable application  40 , a software object employed by executable application  40 , or the like, or a combination thereof. As used herein, “executable application  40 ” is understood to include software, hardware, and systems comprising hardware and software.  
         [0049]    Prior to storing first test procedure  25   b  (FIG. 1), an entry in database  20  (FIG. 1) may be created for a pre-existing requirement  22   a  (FIG. 1) for executable application  40  (FIG. 1) describing or otherwise defining what to test. Test procedure  25   b  (FIG. 1) may be created for test requirement  22   a  (FIG. 1) which describes or otherwise defines how to test the corresponding test requirement  22   a.    
         [0050]    Additionally, first test procedure  25   b  (FIG. 1) may be associated with documentary and other data describing test requirement  22   a  (FIG. 1) of executable application  40  (FIG. 1). Functions of executable application  40  to be exercised may also be associated with, e.g. linked to, one or more corresponding individual test procedures  25   b  of test procedure repository  24  (FIG. 1), e.g. in database  20  (FIG. 1).  
         [0051]    The user may elect to customize, step  210 , one or more predetermined system characteristics  12  (FIG. 1) relating to test procedure  25   b  (FIG. 1), e.g. test procedure  25   b  stored in database  20  (FIG. 1). Using system characteristics  12  (FIG. 1), system  10  may be modified and adapted to meet requirements  22   a  (FIG. 1) of executable application  40  under test (FIG. 1), e.g. operating system and/or target language platform requirements, as well as a business&#39; requirements, e.g. level of testing, level of reporting detail, and the like.  
         [0052]    Task data identifying task steps performed by first test procedure  25   b  (FIG. 1) and the user in user assisted testing of executable application  40  (FIG. 1) are acquired, step  220 , such as by observing and collating user initiated keyboard key strokes and by reviewing detailed functional specifications. Task data may include instructions for acquiring test related timing data, memory usage related data, command sequence data, and the like, or combinations thereof. Data may be entered via user interface  30  (FIG. 1) and stored in various tables  21  (FIG. 1) of database  20  (FIG. 1).  
         [0053]    Execution of first test procedure  25   b  (FIG. 1) may be initiated. Result data  26   a  (FIG. 1) which may result from executing first test procedure  25   b  may also be acquired. Results  26   a  (FIG. 1), e.g. test outcome related information based on the acquired test data, may be recorded such as in database  20  by exercising the appropriate controls on user interface  30 .  
         [0054]    In an alternative embodiment, an executable second test procedure  25   b  (FIG. 1) may be generated, step  230 , corresponding to first test procedure  25   b , for automatically exercising the set of functions of the executable application  40  without user assistance. System  10  may generate an automated test procedure template or shell that may be annotated, e.g. with comments. The automated test procedure template may be supplied and augmented with directives and/or data at a later time with keystrokes and commands from a host automated testing tool to create fully automated scripts which may be executed in an unattended manner.  
         [0055]    As used herein, a host automated testing tool may comprise a host automation language, e.g. a proprietary scripting language used by an automated testing tool such as SILKTEST™ from Segue Software, Inc. of Lexington, Mass. or WinRunner® from Mercury Interactive Corporation of Sunnyvale, Calif. Such automated testing tools comprise systems that automate manual test procedures by providing script programming languages, record and playback capabilities, test data repositories and/or other features that can be used to mimic user interactions and system behavior to drive applications in an unattended manner without human interaction. Such systems are typically used for tasks such as regression, stress, and performance testing.  
         [0056]    Result data  26   a  (FIG. 1) derived from exercising executable application  40  (FIG. 1) by executing second test procedure  25   b  may be acquired, e.g. as shown in step  240 .  
         [0057]    Referring now to FIG. 3, an executable test application may be created for managing user adaptable test procedures  25   b  (FIG. 1) and reporting on user adaptable resulting test data  25   c  (FIG. 1). At step  300 , a cross-platform capable user interface  30  (FIG. 1) to database  20  (FIG. 1) may be provided. User interface  30  may be used when customizing a predetermined set of characteristics  12  (FIG. 1) of database  20  relating to test procedure  25   b , e.g. stored in database  20 , and report  23  (FIG. 1) of test results  26   a  (FIG. 1). Such a cross-platform user interface  30  may include testing abilities which may be conducted on a plurality of platforms, e.g. it is not integrated with a single platform, such as UNIX or Microsoft® Windows®, or language such as C++, Visual Basic, Java, or the like.  
         [0058]    Test procedure  25   b  (FIG. 1) may be stored, step  310 , in test procedure repository  24  (FIG. 1) of database  20  (FIG. 1). Stored manual test procedures  25   b  may be used as a basis for generation of test primers, e.g. written or displayable guidelines and/or test scripts, as well as for detailed elements of such test primers.  
         [0059]    One or more requirements  22   a  (FIG. 1) associatable with test procedure  25   b  (FIG. 1) may be imported, step  320 , from a predetermined source of requirements  22   a , e.g. an application design document or functional requirements document.  
         [0060]    One or more automated procedure stubs, which are placeholders for finalized test procedures, may be created, step  330 , and test procedure  25   b  (FIG. 1) executed, step  340 . The placeholders comprise syntactically correct comment statements that are output for each step of a test procedure  25   b . A host automated testing tool, e.g. SILKTEST™ or WINRUNNER®, may be executed and commented instructions provided to the user who may follow the commented instructions to record and capture application interactions that are detailed in the underlying test procedure  25   b.    
         [0061]    For example, system  10  (FIG. 1) may create an automated test script derived from a manual test script. The automated script may contain program code to capture timing data and to control desired features of system  10 , e.g. record/playback features. As used herein, the generated script is an automated procedure placeholder or stub in that it contains comment lines corresponding to one or more steps of the manual procedure from which the generated script was derived. A user may use an automated testing tool such as SILKTEST™ by opening the automated procedure placeholder and record actual keystrokes and mouse interactions necessary to drive the application under test in the manner specified by each manual test procedure step. This process augments the automated procedure placeholder by creating a fully functioning automated test script that may be used in unattended playback to perform regression, performance, load and other forms of testing.  
         [0062]    Test result  25   c  (FIG. 1) obtained from executing test procedure  25   b  (FIG. 1) may be stored, step  350 , into test results repository  26  (FIG. 1) in database  20  (FIG. 1). Test success and/or failure results at the test procedure and/or test procedure step level may be collated and stored. Associated test step timing information, along with other key metrics such as defect ticket numbers, may also be stored.  
         [0063]    If the user so elects, report  23  (FIG. 1) may be created, step  360 , that reflects test result  25   c  (FIG. 1). As part of test procedure execution, data related to a time of executing test procedure  25   b  (FIG. 1) may be entered and optionally reflected in test report  23  (FIG. 1), e.g. data related to the time of execution and length of test execution. As used herein, report  23  may comprise a collection of reports  23  that are derived from test procedures  25   b  and test requirements  22   a  as well as test results repository  26  (FIG. 1). Information in report  23  may be suitable to support certification by a standards associated organization, e.g. International Standards Organization (ISO), Capability Maturity Model (CMM), the United States of America Food and Drug Administration (FDA), and American National Standards Institute (ANSI). Additionally, users may be provided with options that allow user customization of information in report  23 , e.g. custom reports based on the recorded test outcome related information which may include customer indicators as to whether individual requirements of test requirements  25   a  (FIG. 1) were met.  
         [0064]    Because system  10  (FIG. 1) is customizable, tables  21  (FIG. 1) in database  20  (FIG. 1) may be used to link test requirements  25   a  (FIG. 1) with test procedures  25   b  (FIG. 1) and their results  26   a  (FIG. 1). Test metrics may also be generated, e.g. descriptions of test coverage or test result regression progress.  
         [0065]    Additionally, customization of system  10  (FIG. 1) may include allowing a user to add new test procedures  25   b  (FIG. 1) and new database components that support different types of testing, e.g. unit test, integration test, system test, beta test, and the like, or combinations thereof. Customization of system  10  (FIG. 1) may further include allowing a user to define and implement additional metrics as needed. Additionally, steps in test procedure  25   b  (FIG. 1) and requirements  25   a  (FIG. 1) may be exported to other, more specialized tools such as specialized automated test tools or office applications.  
         [0066]    Further, as system  10  (FIG. 1) is customizable, system  10  may be used to gather disparate testing functionality into a single package with a unified user interface  30  (FIG. 1), e.g. defect tracking, test case drivers, and the like.  
         [0067]    By way of example and not limitation, referring now to FIG. 4, in a preferred embodiment system  10  (FIG. 1) comprises an application written in Microsoft® Access® and is customizable, e.g. a user with appropriate rights may change characteristics of system  10  such as by changing a selected Microsoft object such as a screen, report, table, query, module, or the like. Using a front end such as is provided by a database system such as Microsoft® ACCESS®, a user may customize system  10  by changing various programmatic settings such that key data values and options are not “hard coded,” minimizing the need to make external programming changes by using properties exposed by the database system.  
         [0068]    By way of further example, a user changeable feature of user interface  30  (FIG. 1) may include a user selectable portion, e.g. content  37  (FIG. 1), of a displayable menu  36  (FIG. 1) where selection of that user selectable portion retrieves a utility functions menu  36  or form  34 , an exemplar of which is illustrated in FIG. 4. Buttons  50   a ,  50   b ,  50   c ,  50   d ,  50   e , and  50   f  may be present at form  34  to invoke predetermined functionality.  
         [0069]    Referring now to FIG. 5 and FIG. 6, forms  51  (FIG. 5) and  52  (FIG. 6) may be used to define properties related to system  10  (FIG. 1) as a whole or actions to be undertaken. For example, forms  51  and  52  may be used to establish data values based on installation specific naming conventions which may be used to create standard names for each test procedure  25   b  and test requirement  25   a.    
         [0070]    As illustrated in FIG. 7 a , one or more testing phases may be customized using form  53 , allowing a user to define various phases of testing to be performed that may be associated with one or more test procedures  25   b  (FIG. 1). As further illustrated in FIG. 7 b , form  54  may be used to allow a user to define one or more objects that may be later associated with test procedures  25   b.    
         [0071]    Referring now to FIG. 8, defined objects, e.g. those defined using forms  34  as illustrated in FIGS. 7 b , may be used to relate a test procedure  25   b  (FIG. 1) being worked on to a desired object, e.g. executable module shown at  38   b . This information may later be viewed in a report, enabling users who make changes to a desired module such as at  38   b  to identify and execute test procedures related to that module  38   b.    
         [0072]    Customize button  50   f  and region  34   g  of form  34  (FIG. 4) may be used to provide access to tables  21  (FIG. 1) used by system  10  (FIG. 1). Additionally, customize button  50   f  and region  34   g  of form  34  may be used to allow a user to dynamically change characteristics of a desired table  21 , e.g. to tailor those characteristics to an individual user&#39;s need. For example, as new tables  21  are added, access to new forms  34  (FIG. 1) that provide a user interface to the new tables  21  may be provided such as in a list box at region  34   g  (FIG. 4). For example, users of system  10  may be defined such as illustrated at FIG. 9 a  and explanatory codes for testing results defined such as illustrated at FIG. 9 b.    
         [0073]    Control of automated test script generation including access to test script generation options may be accessed such as illustrated at FIG. 10. Form  58  may be used to allow a user to customize the manner in which test script shells are generated. For example, the checkbox entitled “Generate timing and memory usage statistics” on form  58 , when checked, will cause the application to insert code into generated test scripts to record timing and memory usage data.  
         [0074]    It will be understood that various changes in the details, materials, and arrangements of the parts which have been described and illustrated above in order to explain the nature of this invention may be made by those skilled in the art without departing from the principle and scope of the invention as recited in the following claims.