Abstract:
A test harness that comprises one or more tests for a program in one example is executed. An output from the test harness is received. The output comprises one or more respective test results for the one or more tests. A verification that the one or more respective test results comprise one or more expected test results for the one or more tests is received from a user. The one or more expected test results are stored in a benchmark file. A test harness for benchmark file generation that comprises one or more tests for a program in a further example is created. The test harness comprises one or more calls to one or more subroutines that are employable for one or more of: a definition of one or more expected test results for the one or more tests; and/or a verification of the one or more expected test results.

Description:
TECHNICAL FIELD  
       [0001]     The invention relates generally to software and more particularly to testing of software.  
       BACKGROUND  
       [0002]     A test harness comprises a collection of tests designed to verify proper behavior of a program. A test establishes an environment for the program, invokes one or more subroutines of the program, and compares one or more results of the subroutines to one or more expected results for the subroutines. The one or more expected results are hard-coded into the test. As one shortcoming, the expected results for the subroutines must be known when designing the test. It is desirable for the test harness to be developed independently from the determination of the expected results.  
         [0003]     As another shortcoming, adjustments to the subroutines of the program require modifications to the test harness. For example, a test designer spends time manually computing new expected results for the subroutines based on the adjustments to the subroutines. The test designer must then manually update the expected values of the tests of the test harness with the new expected results. For example, the test designer must hard-code the new results into the test, which adds additional lines of code. The additional lines of code increase a margin of error for the test harness and increase difficulty of debugging the test harness.  
         [0004]     Thus, a need exists for providing a reduction in time and/or effort required to develop and maintain one or more test harnesses for one or more programs. A further need exists for providing a reduction in number of lines of code for test harnesses.  
       SUMMARY  
       [0005]     The invention in one implementation encompasses a method. An execution of a test harness that comprises one or more tests for a program is performed. An output that comprises one or more respective test results for the one or more tests is received from the test harness. A verification that the one or more respective test results comprise one or more expected test results for the one or more tests is received from a user. The one or more expected test results are stored in a benchmark file.  
         [0006]     Another implementation of the invention encompasses a method. One or more benchmark files of a program are generated through employment of one or more results obtained from the program during execution of one or more tests on the program.  
         [0007]     Yet another implementation of the invention encompasses an apparatus. The apparatus comprises a test control component that generates one or more benchmark files of a program through employment of one or more results obtained from the program during execution of one or more tests on the program.  
         [0008]     Another implementation of the invention encompasses a method. A test harness for benchmark file generation that comprises one or more tests for a program is created. The test harness comprises one or more calls to one or more subroutines that are employable for one or more of: a definition of one or more expected test results for the one or more tests; and/or a verification of the one or more expected test results. 
     
    
     DESCRIPTION OF THE DRAWINGS  
       [0009]     Features of exemplary implementations of the invention will become apparent from the description, the claims, and the accompanying drawings in which:  
         [0010]      FIG. 1  is a representation of one implementation of an apparatus that comprises one or more storage devices, one or more processors, one or more interface components, one or more programs, one or more test harnesses, one or more benchmark files, and one or more test control components.  
         [0011]      FIG. 2  is a representation of the test harness developed through employment of the test control component of the apparatus of  FIG. 1 .  
         [0012]      FIG. 3  is an exemplary process flow of generating the benchmark files of the apparatus of  FIG. 1 .  
         [0013]      FIG. 4  is an exemplary process flow of benchmarking the program of the apparatus of  FIG. 1 .  
         [0014]      FIG. 5  is an exemplary process flow of processing the test harness through employment of the test control component of the apparatus of  FIG. 1 . 
     
    
     DETAILED DESCRIPTION  
       [0015]     Turning to  FIG. 1 , an apparatus  100  in one example comprises one or more storage devices  101 , one or more processors  105 , and one or more interface components  115 . The processors  105  in one example comprises a central processor unit (“CPU”). The processor  105  executes one or more instructions of one or more programs  125 , one or more test harnesses  130 , and a test control component  140 . The programs  125  and the test harnesses  130  in one example comprise one or more scripting languages, for example, one or more perl modules. The program  125  in one example comprise one or more subroutines and one or more variables, as will be understood by those skilled in the art. The storage device  101  in one example comprises an instance of a recordable data storage medium, as described herein. The storage device  101  stores the programs  125 , the test harnesses  130 , one or more benchmark files  135 , and one or more test control components  140 .  
         [0016]     The interface component  115  in one example comprises a command line interface. The interface component  115  in one example comprises a graphical user interface (“GUI”). In one example, the interface component  115  allows a user  145  to execute one or more of the test harnesses  130  on the programs  125 . In another example, the interface component  115  allows the user  145  to verify one or more results of one or more variables of the program  125 . In yet another example, the interface component  115  allows the user  145  to benchmark one or more tests  150  of the test harness  140  for the program  125 .  
         [0017]     The test harness  130  comprises one or more tests  150  to verify one or more current results of one or more variables of the program  125 . The test harness  130  in one example establishes a testing environment for the program  125 . The test harness  130  employs the tests  150  to invoke one or more of the subroutines of the program  125  to obtain the current results of the variables of the program  125 . The benchmark files  135  in one example comprise one or more expected results for the variables of the program  125  during execution of one or more tests of the test harness  130 . The benchmark files  135  in one example comprise one or more portions of the one or more tests  150 . Each of the portions of the benchmark files  135  comprises one or more expected results for the variables obtained by the test  150 . The benchmark files  135  in one example comprise one or more snapshots of expected behavior for the program  125 . The benchmark files  135  enable regression testing of the program  125 , as will be understood by those skilled in the art.  
         [0018]     The test control component  140  in one example comprises a perl module. The test control component  140  provides for a reduction in the time required by the user  145  to develop a test harness, for example, the test harness  130 , for a program, for example the program  125 . The test control component  140  comprises one or more subroutines for reducing the time required by the user  145  to develop the test harness  130 . The subroutines in one example obtain results of one or more variables and/or data structures from the program  125 . The subroutines of the test control component  140  in one example employ a perl utility, for example, DataDumper, to obtain a result of any data structure in a textual format.  
         [0019]     The test control component  140  in one example comprises one or more data type subroutines, for example, perl subroutines, to obtain current results for scalar, array, and hash data types. The test control component  140  comprises a benchmark file subroutine. The benchmark file subroutine in one example runs the tests in a harness file and carries out benchmarking if appropriate. The benchmark file subroutine opens an output file generated by the tests in the harness file, reads content of the output file, closes the output file, and carries out benchmarking if appropriate. The test control component  140  comprises a printed message subroutine. The printed message subroutine in one example captures one or more messages printed to the interface component  115  by the tests in the harness file. The printed message subroutine tests the messages and then benchmarks the messages if appropriate. The test control component  140  comprises a date removal subroutine. The date removal subroutine in one example replaces one or more date-formatted strings with ‘z’ characters. The date removal subroutine is useful in situations where the results will be different for every invocation of the test  150 . For example, results stored with a timestamp will vary for each invocation of the test  150 .  
         [0020]     Turning to  FIG. 2 , when designing the tests  150  of the test harness  130 , the user  145  employs the one or more subroutines of the test control component  140  to reduce the number of lines coded in the tests  150 . For example,  FIG. 2  comprises an exemplary prior art test harness  205  and the test harness  130 . The prior art test harness  205  and the test harness  130  comprise three tests: test  1 , test  2 , and test  3 . The prior art test harness  205  comprises command groupings  207 ,  209 ,  211 , and  213 . Command grouping  207  sets up an environment for the prior art test harness  205 . Command grouping  209  comprises one or more lines to define an expected result of a variable of test  1 , for example, scalar value s. For example, the command grouping  209  comprises a hard-coded expected result for the scalar value s. Command grouping  209  comprises one or more lines to obtain a current result of the scalar value s. Command grouping  209  comprises one or more lines to compare the current result of the scalar value s with the expected result of the scalar value s.  
         [0021]     Command grouping  211  comprises one or more lines to define an expected result of a variable of test  2 , for example, array value a. For example, the command grouping  211  comprises a hard-coded expected result for the array value a. Command grouping  211  comprises one or more lines to obtain a current result of the array value a. Command grouping  211  comprises one or more lines to compare the current result of the array value a with the expected result of the array value a. Command grouping  213  functions in a similar manner as command groupings  209  and  211 , for the hash value h.  
         [0022]     The test harness  130  comprises one or more command groupings  215 ,  217 ,  219 ,  221 , and  223 . The command grouping  215  comprises a directive statement to include the test control component  140 . The command grouping  217  sets up an environment for the test harness  130 . Command grouping  219  comprises a line to invoke a first subroutine of the test control component  140 , for example, a subroutine to test and benchmark the scalar value s. Command grouping  221  comprises a line to invoke a second subroutine of the test control component  140 , for example, a subroutine to test and benchmark the array value a. Command grouping  223  comprises a line to invoke a third subroutine of the test control component  140 , for example, a subroutine to test and benchmark the hash value h. The test harness  130  lacks any hard-coded expected results for the variables of the tests  1 ,  2 , and  3 .  
         [0023]     Referring back to  FIG. 1 , the test control component  140  receives one or more command arguments from the user  145  via the interface component  115 . The command arguments in one example comprise one or more execution command arguments, one or more benchmark command arguments, and one or more forcemark command arguments. The test control component  140  provides one or more results of one or more variables of the program  125  to the user  145  through employment of the interface component  115 . The test control component  140  employs the interface component  115  to obtain verification of the results of the variables of the program  125 . For example, the interface component  115  comprises GUI. The GUI comprises a text box for a result and a text box for the variable of the program  125 . The GUI comprises an “Accept” button and a “Decline” button. The text control component  140  provides the result and the name of the variable to the interface component  115 . The user  145  presses the “Accept” button to verify the result of the variable. The user  145  presses the “Decline” button to reject the result of the variable.  
         [0024]     In one example, the test control component  140  cooperates with the processor  105  to execute the test harness  130  on the program  125  upon receipt of an execution command argument. The test control component  140  in one example provides one or more results of one or more variables of the program  125  to the user  145  via the interface component  115  during execution of the test harness  130  on the program  125 . The processor  105  retrieves the test harness  130  from the storage component  101 . The processor  105  executes the instructions of the test harness  130 . The processor  105  executes a subroutine of the test control component  140 . The test control component  140  obtains a result for a variable of the program  125 . The test control component  140  provides the result of the variable of the program  125  to the user  145  through employment of the interface component  115 . The test control component  140  obtains a verification of the result of the variable of the program  125  from the user  145  through employment of the interface component  115 . For example, the test control component  140  provides a result, for example, “hello world” of a variable, for example, a message variable, to the user  145 . The user  145  compares the “hello world” to an expected value of the greeting variable and presses an “Accept” button on the GUI of the interface component  115  to accept the result of the greeting variable.  
         [0025]     In another example, the test control component  140  cooperates with the processor  105  automatically generate the benchmark file  135  for the test harness  130  upon receipt of a benchmark command. The test control component  140  generates one or more portions of the benchmark file  135  for each test  150  within the test harness  130 . Each of the portions of the benchmark file  135  comprise one or more expected results for the variables of the program  125  invoked by the test  150 . The test control component  140  associates the portions of the benchmark file  135  with an identifier of the test  150 . The identifier of the test  150  in one example comprises a test name. The test control component  140  receives the results of the variables of the program  125  from the subroutines of the tests  150 . The test control component  140  incorporates the results of the variables of the program  125  into the portion of the benchmark file  135  associated with the test name. For example, the user  145  employs the interface component  115  to select one or more tests  150  to benchmark in the test harness  140 . The test control component  140  cooperates with the processor  105  to execute the selected one or more tests  150 . The test control component  140  generates one or more portions of the benchmark file  135  for the selected one or more tests  150 .  
         [0026]     The test control component  140  updates one or more portions of the benchmark file  135 . The test control component  140  determines existence of the portions of the benchmark file  135  for the test  150 . Where the portions of the benchmark file  135  for the test  150  exist, the test control component  140  updates one or more results for one or more variables of the one or more portions of the benchmark file  135 .  
         [0027]     The test control component  140  generates one or more versions of the benchmark file  135  for the test harness  130 . For example, the test control component  140  generates a first version of the benchmark file  135  at a first execution time and a second version of the benchmark file  135  at a second execution time. If desired, the user  145  may select to employ a previous version of the benchmark file  130 , for example, the first version of the benchmark file  135  from the first execution time, for use in regression testing of the program  125 . The user  145  in one example performs an interaction with the interface component  115  to select a version of the benchmark file  135  for employment in regression testing of the program  125 .  
         [0028]     In yet another example, the test control component  140  generates the benchmark file  135  for the test harness  130  upon receipt of a forcemark command. The test control component  140  generates the benchmark file  135  for the test harness  130 . The test control component  140  deletes one or more previous versions of the benchmark file  135  for the test harness  130 .  
         [0029]     The test control component  140  in one example employs the benchmark files  135  to perform regression testing on the program  125 . The processor  105  executes the test  150  of test harness  130  on the program  125 . The test control component  140  obtains a current result of a variable invoked by a subroutine of the test control component  140  from the program  125 . The test control component  140  employs an identifier of the test  150  to obtain the portion of the benchmark file  135 . The test control component  140  employs the subroutine to obtain a name of the variable. The test control component  140  employs the name of the variable to identify the expected result of the variable within the portion of the benchmark file  135 . The test control component  140  executes the expected result of the variable to obtain a result of the variable for the current invocation. For example, the test control component  140  evaluates the textual format of the expected result to obtain an expected result for the current invocation of the test harness  130  for the program  125  through employment of the DataDumper. The test control component  140  compares the current result of the variable with the expected result of the variable for the current invocation.  
         [0030]     An illustrative description of exemplary operation of the apparatus  100  is presented, for explanatory purposes.  
         [0031]     Turning to  FIG. 3 , the user  145  in one example creates a test harness  130 . For each test  150  of the test harness  130 , the user  145  proceeds to STEP  305 . Upon verifying and benchmarking all tests  150  within the test harness  130 , the user  145  proceeds to STEP  315 . In STEP  310 , the user  145  modifies and/or adds tests to the test harness  130 . In STEP  320 , the user  145  invokes the execute command argument from the interface component  115 . The processor  105  executes the test harness  130  on the program  125 . In STEP  325 , the test control component  140  provides the results of the variables of the program  125  to the interface component  115 . The interface component  115  provides the results to the user  145 . Where the user  145  accepts the results of the variables of the program  125  during execution of the test harness  130 , the user  145  proceeds to STEP  330 . Where the user  145  rejects the results of the program  125  during execution of the test harness  130 , the user  145  proceeds to STEP  310 . In STEP  330 , the user  145  invokes the benchmark command from the interface component  115 . The test control component  140  executes the test harness  130  on the program  125  and generates the benchmark file  125  for the test harness  130 . STEP  330  proceeds to STEP  305 .  
         [0032]     Turning to  FIG. 4 , the processor  105  executes the test harness  130 . In STEP  405 , the processor  105  obtains a result from the program  125  during execution of the test harness  130 . In STEP  410 , the processor  105  checks if the benchmarking option is set. Where the benchmarking option is set, the processor  105  proceeds to STEP  415 . Where the benchmarking option is not set, the processor  105  proceeds to STEP  420 .  
         [0033]     In STEP  415 , the processor  105  cooperates with the interface component  115  to display the result of the variable of the program to the user  145 . The processor  105  cooperates with the test control component  140  to save the result of the variable of the program in the benchmark file  135 . In STEP  420 , the processor  105  cooperates with the test control component  104  to determine existence of the benchmark file  135 . Where the benchmark file  135  does not exist, the processor  105  proceeds to STEP  425 . Where the benchmark file  135  exists, the processor  105  proceeds to STEP  430 .  
         [0034]     In STEP  425 , the processor  105  cooperates with the interface component  115  to display the result of the variable to the user  145 . In STEP  430 , the test control component  140  searches the benchmark file  135  for the expected result of the variable of the program under execution of the test harness  130 . In STEP  435 , the test control component  140  determines the existence of the expected result within the benchmark file  135 . For example, the test control component  140  determines if the expected result has previously been written to the benchmark file  135 . Where the expected result does not exist, the processor  105  proceeds to STEP  425 . Where the benchmark file  135  exists, the processor  105  proceeds to STEP  440 .  
         [0035]     In STEP  440 , the test control component  140  compares the result for the variable to the expected result for the variable. Where the result for the variable matches the expected result for the variable, the processor  105  proceeds to STEP  445 . Where the result for the variable fails to match the expected result for the variable, the processor  105  proceeds to STEP  450 .  
         [0036]     In STEP  445 , the test control component  140  cooperates with the interface component  115  to provide a message indicating that the test passed to the user  145 . The message in one example comprises a terse message, for example, “Test OK,” or another suitable message that can be quickly read and understood by the user  145 . In STEP  450 , the test control component  140  cooperates with the interface component  115  to provide to the user  145  a message comparing the result for the variable to the expected result for the variable from the benchmark file  135 . The message in one example is a detailed message that describes the test failure to the user  145 , as will be appreciated by those skilled in the art.  
         [0037]     Turning to  FIG. 5 , the user  145  performs a regression test on the program  125 . In STEP  505 , the processor  105  reads the test harness  130  and executes one or more statements of the one or more tests  150  of the test harness  130 . In STEP  510 , the processor  105  evaluates a statement of the test  150  of the test harness  130 . Where the statement of the test harness  130  invokes a subroutine of the test control component  140  on a variable of the program  125 , the processor  105  proceeds to STEP  520 . Where the statement of the test harness  130  fails to invoke a subroutine of the test control component  140  on a variable of the program  125 , the processor  105  proceeds to STEP  515 . In STEP  515 , the processor  105  executes the statement as a usual perl statement.  
         [0038]     In STEP  520 , the processor  105  communicates with the test control component  140  to execute the subroutine on the variable of the program  125 . In STEP  525 , the processor  105  executes the subroutine on the variable of the program  125  and obtains the current result of the variable of the program  125 . In STEP  530 , the test control component  140  obtains one or more expected results from the benchmark file  135 . The test control component  140  compares the current result of the subroutine on the variable of the program  125  with the expected result from the benchmark file  135 . In STEP  535 , the processor  105  makes a determination of the completion of the test harness  130 . Where the test harness  130  comprises additional tests  150 , the processor  105  proceeds to STEP  505 . Where the processor  105  has executed the tests  150  of the test harness  130  to completion, the processor  105  proceeds to STEP  540 . In STEP  540 , the test control component  140  provides a summary of results of the tests  150  of the test harness  130  to the interface component  115 . The interface component  115  provides the summary of results to the user  145 .  
         [0039]     The apparatus  100  in one example comprises a plurality of components such as one or more of electronic components, hardware components, and computer software components. A number of such components can be combined or divided in the apparatus  100 . An exemplary component of the apparatus  100  employs and/or comprises a set and/or series of computer instructions written in or implemented with any of a number of programming languages, as will be appreciated by those skilled in the art. The apparatus  100  in one example comprises any (e.g., horizontal, oblique, or vertical) orientation, with the description and figures herein illustrating one exemplary orientation of the apparatus  100 , for explanatory purposes.  
         [0040]     The apparatus  100  in one example employs one or more computer-readable signal-bearing media. The computer-readable signal-bearing media store software, firmware and/or assembly language for performing one or more portions of one or more embodiments of the invention. Examples of a computer-readable signal-bearing medium for the apparatus  100  comprise the storage component  101 . The computer-readable signal-bearing medium for the apparatus  100  in one example comprise one or more of a magnetic, electrical, optical, biological, and atomic data storage medium. For example, the computer-readable signal-bearing medium comprise floppy disks, magnetic tapes, CD-ROMs, DVD-ROMs, hard disk drives, and electronic memory. In another example, the computer-readable signal-bearing medium comprises a modulated carrier signal transmitted over a network comprising or coupled with the apparatus  100 , for instance, one or more of a telephone network, a local area network (“LAN”), a wide area network (“WAN”), the Internet, and a wireless network.  
         [0041]     The steps or operations described herein are just exemplary. There may be many variations to these steps or operations without departing from the spirit of the invention. For instance, the steps may be performed in a differing order, or steps may be added, deleted, or modified.  
         [0042]     Although exemplary implementations of the invention have been depicted and described in detail herein, it will be apparent to those skilled in the relevant art that various modifications, additions, substitutions, and the like can be made without departing from the spirit of the invention and these are therefore considered to be within the scope of the invention as defined in the following claims.