Abstract:
A computer implemented method of software product documentation review involves importing a description of a structure of the product documentation for use in determining locations in the product documentation from which to extract code and/or command portions of the product documentation for testing and providing pre-defined test case stub files for the product documentation into which are inserted extracted portions of the product documentation for testing. The test case stub files are run with the code and/or command portions inserted to determine whether or not they are runnable which is indicative of whether or not an error is present in the documentation.

Description:
FIELD OF THE INVENTION 
       [0001]    This invention relates to a method of delegating and automating at least part of documentation review to software. 
       DESCRIPTION OF BACKGROUND 
       [0002]    After a software product has been on the market for several years, when the current documentation is reviewed, it is often discovered that the documentation contains errors when compared to the original software. Documentation review is a manual process that relies on the attention and competencies of the reviewers. There are currently a few tools that help the reviewers, but such tools are nothing more than collaboration software that allow simultaneous review of an electronic document. In other words, each reviewer can simultaneously review and add their respective comments and see the other reviewers&#39; comments and eventually discuss those comments. There is nothing available that actually frees the reviewer to manually check the code samples, command line options, and compatibility/support tables. 
       SUMMARY OF THE INVENTION 
       [0003]    The shortcomings of the prior art are overcome and additional advantages are provided through the provision of a method of delegating and automating at least part of documentation review of software. Embodiments of the present invention propose a tool that automatically identifies errors in software documentation when compared to the original software which is running and assumed to be error-free. In embodiments of the invention, the documentation is split into logical parts that can be associated to specific use cases and automatically validated by a software that is able to read the documentation, extract the proper information, code/build the corresponding test application, run the test application and check the result. 
         [0004]    Embodiments of the invention propose a computer implemented method of software product documentation review that involves, for example, importing a description of a structure of the product documentation for use in determining locations in the product documentation from which to extract code and/or command portions of the product documentation for testing and also providing pre-defined test case stub files for the product documentation, which test case stub files have tags indicative of locations for insertion of extracted portions of the product documentation for testing. 
         [0005]    According to embodiments of the invention, the code and/or command portions are extracted from the product documentation at locations based on the documentation structure description by a parser/validator component and inserted into the test case stub files at locations indicated by the tags. The test case stub files are run with the code and/or command portions inserted, and a validating test report for the product documentation is generated if the test case stub files with the code and/or command portions inserted are runnable. On the other hand, an error test report is generated if the test case stub files with the code and/or command portions inserted are not runnable. 
         [0006]    Additional features and advantages are realized through the techniques of the present invention. Other embodiments and aspects of the invention are described in detail herein and are considered a part of the claimed invention. For a better understanding of the invention with advantages and features, refer to the description and to the drawings. 
       TECHNICAL EFFECTS 
       [0007]    As a result of the summarized invention, technically we have achieved a solution for implementing a method of software product documentation review that involves checking if code samples are correct and buildable/runnable by extracting code samples from documentation and placing them into a previously defined test case stub and executing the test cases by an automated test framework. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0008]    The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the claims at the conclusion of the specification. The foregoing and other objects, features, and advantages of the invention are apparent from the following detailed description taken in conjunction with the accompanying drawings in which: 
           [0009]      FIG. 1  illustrates an example of a table showing the platform dependencies of commands for embodiments of the invention; 
           [0010]      FIG. 2  is a schematic diagram that illustrates a high level overview of an example of the architecture for the documentation validation method for embodiments of the invention; 
           [0011]      FIG. 3  is a schematic diagram that illustrates an example of components and flow of information between components in the process of document validation for embodiments of the invention; and 
           [0012]      FIG. 4  is a flow chart that illustrates an example of the process of automated document testing for embodiments of the invention. 
       
    
    
       [0013]    The detailed description explains the preferred embodiments of the invention, together with advantages and features, by way of example with reference to the drawings. 
       DETAILED DESCRIPTION OF THE INVENTION 
       [0014]    According to the method for embodiments of the invention, the documentation is split into logical sections according to the specific software use cases. In particular every use case maps a documentation part. Each use case has an associated XML schema and/or grammar rules for defined features including a description on how to read/parse documentation (e.g., how to understand that a line represents a command and how to understand that some of the options are optional and others mandatory). Code samples are also marked in the documentation and associated to a specific use case. 
         [0015]    The process of automated documentation validation for embodiments of the invention consists of checking various matters for correctness. For example, the automated documentation validation process involves checking that documentation contains all use cases described correctly according to defined grammatical rules and XML schemas. 
         [0016]    The process for embodiments of the invention also involves checking if code samples are correct and buildable/runnable. This is done by extracting code samples from documentation and placing them into a previously defined test case stub. Test cases constructed in that way are executed by an automated test framework (i.e. component verification test suite). 
         [0017]    Further, the automated documentation validation process for embodiments of the invention involves checking if command line commands as described in the documentation are working correctly and all of the expected options are listed and work as designed. This is done in a similar way as for code samples, the only difference being in the implementation of test case stubs. 
         [0018]    In addition, the process for embodiments of the invention involves checking correctness of syntactical/compatibility tables (e.g., tables showing the platform dependencies of commands or the relationships between commands). This is done by parsing the table, building the proper column/row matches and executing the corresponding commands 
         [0019]      FIG. 1  illustrates an example of a table showing the platform dependencies of commands for embodiments of the invention. Referring to  FIG. 1 , in the right hand columns  100  appears a list of supported platforms, and in the rows of the ‘Attribute Name’ column  102  on the left is shown the specific data that must appear in the output of a specific command if the intersection between a specific row under the ‘Attribute Name’ and specific platform column is marked by an ‘X’. 
         [0020]    For example, if we look at the first row of the ‘Attribute Name’ column  102  we find ‘Product’ and at the first column under the platform ‘AIX’, when running the specific command on AIX, we should expect an attribute called ‘Product’ in the output. If the attribute called ‘Product’ is missing from the output, it is a bug. The opposite is true if running, for example, on Linux (S/390) because the ‘X’ is absent from the intersection of the appropriate row and column. 
         [0021]    More specifically, after parsing the table and constructing an XML schema for each platform, it is possible to validate output (eventually converted to XML) for a particular machine. If validation fails, it is possible to detect mismatches between the documentation and the software output. Assuming that the software is stable and sufficiently tested, those mismatches can be considered as documentation problems. 
         [0022]    Embodiments of the invention involve, for example, writing test case stub files (i.e., data files or programs that stand in for the original file), which are pieces of code that will execute on the real software, and taking the substance of the extracted documentation and testing it on the real software. 
         [0023]    Thus, according to embodiments of the invention, code samples are extracted from the documentation and moved into a test case dump file, which is executed later via a test framework on the real software and produces an output that is correct or incorrect, depending on whether or not the code samples are error-free. If the test case fails, it means that the documentation contained an error for this test case of this extracted piece of code from the documentation. 
         [0024]      FIG. 2  is a schematic diagram that illustrates a high level overview of an example of the architecture for the documentation validation method for embodiments of the invention. Referring to  FIG. 2 , a documentation validator  200  receives inputs of documentation  202  and use case descriptions  204  and outputs a set of generated test cases  206  which are in turn input to a test harness  208  (i.e., automated component verification test). 
         [0025]      FIG. 3  is a schematic diagram that illustrates an example of components and flow of information between components in the process of document validation for embodiments of the invention. Referring to  FIG. 3 , the documentation  202  is imported to be tested, and the description of the documentation structure  204  is simply the description of the structure of the documentation, such as an XML file. The documentation parser/validator  200  is a tool for embodiments of the invention that takes the documentation  202  and the description  204  and extracts the pieces of the information that are needed for the test. 
         [0026]    The validation report  210  is an outcome of comparing the documentation  202  to the description of the documentation structure  204 . Thus, for example, the documentation  202  is totally broken if the Chapters do not match what is expected (e.g., a Chapter is missing), and it is immediately reported in the validation report  210 . 
         [0027]    The test case stub file  212  is also important to the documentation process. The data extracted from the documentation are simply inserted into the test case stub file  212  as explained in the foregoing example, and thereafter the stub file  212  becomes a test case object  206  that can be executed by the automatic test framework  208 . The result of the execution of the test is the test report  214  that indicates that a particular test case has passed or failed. 
         [0028]    For an example of the validation process for embodiments of the invention utilizing a “Sample Application”, with reference to  FIG. 3 , the input for the automated documentation tests is a fragment of product documentation  202  for the “Sample Application” in HTML format, such as command line interface (CLI) commands of the software, that can be extracted. In the following example, a fragment of the CLI of an application is extracted from the documentation  202  and thereafter injected into the test case stub file  212 . 
       Sample Application 
       [0029]    Chapter 1: Command Line Installation
       Description of CLI installing method for Sample application       
 
         [0031]    Default Target Path Installation
       Following command set allows user to install “Sample application” in default destination directory.
           1. Unpack installation
               Unzip installer.zip   
               2. Start command line installation
               Install.sh -i -default   
               3. Run the service
               /etc/runSample.sh   
               
               
 
         [0039]    Custom Target Path Installation
       Following command set allows user to install “Sample application” in custom destination directory.
           4. Unpack installation
               Unzip installer.zip   
               5. Start command line installation
               Install.sh -i -path /home/SampleApp   
               6. Run the service
               /home/SampleApp/runSample.sh   
               
               
 
         [0047]    Chapter 2: Custom Functions 
         [0048]    Income Report Generation
       Generation of this report can be triggered in integrated environment by using ReportGeneration class.       
 
         [0050]    Example Usage of ReportGeneration Class: 
         [0000]    
       
         
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 ReportGeneration generator = service.getReportGenerator( ); 
               
               
                   
                   generator.setStartTime(“2007-01-01”); 
               
               
                   
                   generator.setEndTime(“2007-12-31”); 
               
               
                   
                 IncomeReport report = generator.generateIncomeReport( ); 
               
               
                   
                   
               
             
          
         
       
     
         [0051]    The documentation structure description file  204  is a file which describes the structural model with which product documentation  202  must comply, and which is used to find the proper places from which to extract information from the real documentation for test purposes. Referring again to  FIG. 3 , an example of the documentation structure description file  204  for the “Sample Application” is as follows: 
         [0000]    
       
         
               
             
           
               
                   
               
             
             
               
                 &lt;?xml version=“1.0” encoding=“UTF-8”?&gt; 
               
               
                 &lt;Documentetion xmlns:xsi=“http://www.w3.org/2001/XMLSchema- 
               
               
                 instance”&gt; 
               
               
                 &lt;Settings&gt; 
               
               
                    &lt;Define name=“CLICommand” font=“CourierNew” size=“12”/&gt; 
               
               
                    &lt;Define name=“CodeSample” font=“CourierNew” size=“10”/&gt; 
               
               
                 &lt;/Settings&gt; 
               
               
                  &lt;Chapters&gt; 
               
               
                   &lt;Chapter name= “Installation”&gt; 
               
               
                    &lt;UseCaseDescription name=“Command line installation”&gt; 
               
               
                     &lt;CLICommand name= “default target path installation” 
               
               
                 testCase=“CLIInstallTest.stub”/&gt; 
               
               
                     &lt;CLICommand name= “custom target path installation” 
               
               
                 testCase=“CLIInstallTest.stub”/&gt; 
               
               
                    &lt;/UseCaseDescription&gt; 
               
               
                   &lt;/Chapter&gt; 
               
               
                   &lt;Chapter name= “Custom functions”&gt; 
               
               
                    &lt;UseCaseDescription name=“Income report generation”&gt; 
               
               
                     &lt;CodeSample name= “Example usage of ReportGeneretor class” 
               
               
                 testCase=“IncomeReportTest.stub”/&gt; 
               
               
                    &lt;/UseCaseDescription&gt; 
               
               
                   &lt;/Chapter&gt; 
               
               
                  &lt;/Chapters&gt; 
               
               
                 &lt;/Documentetion&gt; 
               
               
                   
               
             
          
         
       
     
         [0052]    The test case stub file  212  indicates the location where the code sample that was extracted from the documentation  202  can be inserted, and after insertion at the proper location in the stub file  212 , in the example, the “Income Reports” test can be executed on the real software for test purposes. Referring further to  FIG. 3 , an example of the test case stub files  212  which contain special tags in places where the information extracted from the documentation code or CLI command will be placed for the “Sample Application” is as follows: 
         [0000]    
       
         
               
               
             
           
               
                   
                   
               
             
             
               
                   
                 public class IncomeReportTest { 
               
               
                   
                   public void testIncomeReportGeneration( ){ 
               
               
                   
                     //environment preparation 
               
               
                   
                     Service service = SampleApp.getService( ); 
               
               
                   
                     @CodeSample 
               
               
                   
                     //simplest test 
               
               
                   
                     assertNull(report); 
               
               
                   
                   } 
               
               
                   
                 } 
               
               
                   
                 public class CLIInstallTest { 
               
               
                   
                   public void testDefaultPath( ){ 
               
               
                   
                     Process proc1 = Runtime.getRuntime( ).exec(@CliCmd1); 
               
               
                   
                     assertEquals(0,proc1.exitValue( )); 
               
               
                   
                     Process proc2 = Runtime.getRuntime( ).exec(@CliCmd2); 
               
               
                   
                     assertEquals(0,proc2.exitValue( )); 
               
               
                   
                     Process proc3 = Runtime.getRuntime( ).exec(@CliCmd3); 
               
               
                   
                     assertEquals(0,proc3.exitValue( )); 
               
               
                   
                   } 
               
               
                   
                 } 
               
               
                   
                   
               
             
          
         
       
     
         [0053]    Also referring to  FIG. 3 , after processing the documentation  202  by the documentation parser/validator  200 , possible inconsistencies between the defined structure of the document and real documentation, such as missing descriptions of some use case, can be found. 
         [0054]    An example of additional test case stubs  212  also filled with the extracted data is as follows: 
         [0000]    
       
         
               
             
           
               
                   
               
             
             
               
                 public class IncomeReportTest { 
               
               
                   public void testIncomeReportGeneration( ){ 
               
               
                     //environment preparation 
               
               
                     Service service = SampleApp.getService( ); 
               
               
                     //***CODE PASTED BY DOCUMENTATION PARSER*** 
               
               
                     ReportGeneration generator = service.getReportGenerator( ); 
               
               
                     generator.setStartTime(“2007-01-01”); 
               
               
                     generator.setEndTime(“2007-12-31”); 
               
               
                     IncomeReport report = generator.generateIncomeReport( ); 
               
               
                     //******************************************* 
               
               
                     //simplest test 
               
               
                     assertNull(report); 
               
               
                   } 
               
               
                 } 
               
               
                   
               
             
          
         
       
     
         [0055]    Referring further to  FIG. 3 , the foregoing sample test class is received by the automated test framework  208  and executed on the live product. If the foregoing test fails, it means that there is a bug in the extracted part of the documentation. It is noted that passing other functional tests covering the particular area of code may indicate that it is not actually corrupted but that failure of the test may relate to API/CLI interface changes that typically occur as new software releases are introduced. 
         [0056]    Automatically testing, according to embodiments of the invention, that examples (e.g. command line commands with parameters or code snippets) in product documentation are correct, and/or that the product documentation complies with a defined product documentation structure, applies primarily to textual documentation samples, targets mainly command-line oriented products, and is most valuable/suitable for example-rich documentation. 
         [0057]      FIG. 4  is a flow chart that illustrates an example of the process of automated document testing for embodiments of the invention. Referring to  FIG. 4 , at  400 , a documentation structure description  204  for product documentation  202  is imported, and at  402 , test case stub files  206  for the product documentation  202  are provided. At  404 , the document parser/validator component  200  extracts code/command portions from existing documentation based on the documentation structure description  204 , and at  406 , inserts the extracted code/command portions into the test case stub files  212 . At  408 , the testing framework (system)  208  runs the filled test case files  206 , and problems that arise in testing are indicative of a likely need to correct the examples in the product documentation  202 . Also, by comparing the product documentation  202  with the documentation structure description  204 , it is possible to check whether the product documentation  202  complies with the defined structure  204 . 
         [0058]    The flow diagrams depicted herein are just examples. There may be many variations to these diagrams or the steps (or operations) described therein without departing from the spirit of the invention. For instance, the steps may be performed in a differing order, or steps may be added, deleted or modified. All of these variations are considered a part of the claimed invention. 
         [0059]    While the preferred embodiment to the invention has been described, it will be understood that those skilled in the art, both now and in the future, may make various improvements and enhancements which fall within the scope of the claims which follow. These claims should be construed to maintain the proper protection for the invention first described.