Abstract:
In a distributed data processing system or network testing a distributed processing program is achieved by a test manager and a test scenario. Functionally, the test manager reads the test scenario to identify processes of the system to be tested. The identified processes then read relevant portions of the test scenario and executes such portions. While the processes are executing the scenario portions, the test manager monitors resulting data from each process. If the resulting data does not match an expected data value, the test manager flags an error regarding the process that produced the errant data.

Description:
TECHNICAL FIELD OF THE INVENTION 
     The present invention relates generally to data processing systems, and more particularly, to a system and method for testing a distributed processing system. 
     BACKGROUND OF THE INVENTION 
     A distributed data processing system typically allows for the sharing among locations or facilities of data processing functions by a collection of computers, and other devices linked together by a communications facility such as a network. Quite often, such distributed processing requires a highly structured environment, which allows hardware and software to communicate, share resources, and freely exchange information. 
     Developing program code for operating such distributed systems is difficult and time consuming. One of the most difficult tasks in the development of a distributed processing program is testing of the program. Many bugs in distributed processing programs either appear or disappear because of the relative timing of the events that occur at various nodes. With debugging programs, it is important to be able to reproduce erroneous behavior consistently. Distributed processing programs are inherently non-deterministic, and therefore make testing and debugging extremely difficult. 
     Typically, testing and debugging distributed processing programs involve running the several components of the program from controlled environments (e.g., debuggers) and causing the events in the system to occur in a controlled manner. Although this approach is effective, it is very time consming for regression testing. That is, making sure that the new &#34;fixes&#34; have not undone the &#34;fixes&#34; for the old bugs is very tedious with such an approach. Thus, such typical testing methods suffer from the lack of a deterministic timing behavior and an automatic means for regression testing. 
     Therefore, there is a need in the art for a testing technique for distributed processing programs that is automatic, deterministic, and allows for regression testing. 
     SUMMARY OF THE INVENTION 
     Thus, it is an object of the present invention to test distributed processing system programming in an efficient, automatic, and deterministic manner. 
     In an attainment of this object, the present invention is implemented within a test managing program that exercises various components of the distributed processing system program according to a pre-defined scenario. Each component of the distributed processing program reads and interprets a portion of the scenario, which may be an ASCII text file that describes the steps to be executed by each component of the distributed processing program. The scenario specifies a sequence of events to happen at any given component of the distributed processing program, verifies whether a component&#39;s response to an event is as expected, synchronizes a component with respect to events in another component, and introduces time delays in the execution of a component. 
     More specifically, the present invention begins two or more separate processes within the distributed system, each process implementing a separate portion of the scenario. Typically, the processes in a distributed system send information to one another. Therefore, the scenario is configured so that the two or more processes are required to transfer information between themselves, and then this information is observed. The transfer of this information is the responsibility of the distributed system being tested. The present invention is illustrated by considering a replication framework as the distributed system under test. 
     An example test that may be implemented through the use of a scenario determines whether or not the replication framework operates adequately to transfer the information from a first process to a second process. The simplest test of the replication framework involves two processes each with a &#34;test&#34; object in it. When one process changes the value of the &#34;test&#34; object, the replication framework carries this change information to the other &#34;test&#34; object (in the second process). To know if the framework is working correctly, a scenario that makes the first process make a change in the test object and then has the second process verify the same change in its test object would be needed. In general, scenarios may be configured in a numerous variety of ways so that various components of the distributed processing system can be tested to determine whether or not they operate as designed. For example, in the case of the replication framework, if a certain value is or is not received at a certain point within the system, or if the value has or has not changed from an expected value, it indicates that the replication framework is not working as expected. 
     A technical advantage of the present invention is that it may be implemented through the use of short test scenarios rather than requiring the writing of a complete test program. 
     Another technical advantage of the present invention is that the test scenarios can be written to create predictable sequences of events for the distributed program components, thereby introducing determinism to allow debugging. 
     Yet another technical advantage of the present invention is that it allows for regression testing since the running of the scenarios can be automated. 
     The foregoing has outlined rather broadly the features and technical advantages of the present invention in order that the detailed description of the invention that follows may be better understood. Additional features and advantages of the invention will be described hereinafter which form the subject of the claims of the invention. It should be appreciated by those skilled in the art that the conception and the specific embodiment disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present invention. It should also be realized by those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the invention as set forth in the appended claims. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     For a more complete understanding of the present invention, and the advantages thereof, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: 
     FIG. 1A illustrates a typical distributed processing system for which the present invention:is designed; 
     FIG. 1B illustrates a data processing system for implementing the present invention; 
     FIG. 2 illustrates a diagram of an exemplary implementation of the present invention; 
     FIG. 3 illustrates a flow diagram illustrating one embodiment of the present invention; 
     FIG. 4 illustrates another example of the present invention; and 
     FIG. 5 illustrates another example of the present invention. 
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
     Referring to FIG. 1A, there is illustrated distributed processing system 100 having data processing systems 108, 110, 112, 114, 116, 118, 120, 122, 124, 126 and 128 connected thereto in a conventional manner. Network 106 may be a local area network, a wide area network, or a nationwide or international data transmission network or the like, such as the Internet. 
     Referring next to FIG. 1B, there is illustrated a representative hardware environment, which illustrates a typical hardware configuration of data processing system 108 (see FIG. 1A) in accordance with the subject invention having a central processing unit 10, such as a conventional microprocessor, and a number of other units interconnected via system bus 12. System 108 shown in FIG. 1B includes random access memory (RAM) 14, read only memory (ROM) 16, I/O adapter 18 for connecting peripheral devices such as disk units 20 and tape drives 40 to bus 12, user interface adapter 22 for connecting keyboard 24, mouse 26, speaker 28, microphone 32, and/or other user interface devices such as a touch screen device (not shown) to bus 12, communications adapter 34 for connecting system 108 to network 106 and display adapter 36 for connecting bus 12 to display device 38. 
     Referring next to FIG. 2, there is illustrated a diagram of an example implementation of the present invention. In this example, replication framework 212, as may be implemented within a distributed processing system provides the means for one of the processes to receive information generated by another process. Replication framework 212 provides for replicated data systems to implement data sharing by providing a replica copy of a data object to each process using that data object. Replication reduces the access time for each processor by eliminating the need to send messages over the network to retrieve and supply the necessary data. A replicated object is a logical unit of data existing in one of the computer systems but physically replicated to multiple distributed computer systems. Replicated copies are typically maintained in the memories of the distributed systems. For further discussion of replication, please refer to U.S. Pat. No. (Ser. No. 08/077,231) assigned to a common assignee, which is hereby incorporated by reference herein. (Note that this illustration is for verifying the workings of a replication framework referenced earlier; similar scenarios can be written for testing other distribution systems.) 
     Test manager 201 and processes 206 and 207 may be implemented within any of the dam processing devices illustrated in FIG. 1A and further detailed in FIG. 1B. Furthermore, scenario 202, which may be in the form of an ASCII text file may be stored within any one of the devices within system 100. 
     Test manager 201 first reads the scenario file 202 as illustrated by arrow 203. Test manager 201 is looking for how many processes there are for testing within distributed system 100, and what machines within network 106 to start these processes on. Thus, test manager 201 scans the whole scenario 202 and determines the processes required. In the example shown, two processes, 1: and 2:, are required by scenario 202. Hereinafter, these processes will be referred to simply as process 1 and process 2. Process 1 and process 2 are similar to test manager 201 in that they will also read scenario 202, as illustrated by arrows 204 and 205, respectively. However, process 1 will ignore all statements within scenario 202 not particularly designated for process 1, while process 2 will likewise only read statements particularly designated for process 2. Each process will then perform operations as designated for that process by scenario 202. Typically, processes 1 and 2 will create objects 208 and 209. 
     Processes 1 and 2 will then begin sending messages to objects 208 and 209, respectively. Each time a message is sent then the process will check to determine :if the return value from the invoked method is as expected. 
     For example, message 216 will invoke a method on object 208, and as is typical within object-oriented design, a value 220 will be returned from object 208, and it will be compared to an expected value. For a further discussion of object-oriented programming, please refer to Object-Oriented Technology; A Manager&#39;s Guide, Taylor, D., Addison-Wesley 1994 and Object-Oriented Analysis and Design, Booth, G., 2nd Edition, The Benjamin Publishing Co. 1994, which are hereby incorporated by reference herein. 
     If at any time a return value is not as expected, test manager 201 will produce an error message. Test manager 201 is able to monitor such information Within processes 1 and 2 via communication &#34;pipelines&#34; 214 and 215, An implementation of pipelines 214 and 215 is well within those skilled in the art. 
     Processes 1 and 2 may be identical but not necessarily so. However, it is preferred if processes 1 and 2 pass information between each other in some manner. 
     For example, object 208 may create a value &#34;R&#34; which through communication means 218 and 219 and replication framework 212 is either retrieved or received by object 209, which includes a method for either retrieving or receiving such value. 
     Process 2 may include a step for observing value &#34;R&#34;(as illustrated by arrow 217). If an &#34;R&#34; is not received, test manager 201, which knows of the receipt of the value &#34;R&#34; via communication pathway 215 (which may be implemented within software), then test manager 201 will indicate that an error has occurred. 
     Such an implementation within scenario 202 tests for stimulus type events within distributed system 100, whereby information is shared among the machines within network 106 (e.g., a network administrator broadcasts a message to all users or an administrator changes routing tables that control message flow through the Internet). 
     Referring next to FIG. 3, there is illustrated a flow diagram of the aforementioned implementation. At step 301, the process of the present invention begins. At step 302, test manager 201 reads scenario 202. Then, at step 303, processes 1 and 2 are started in order to exercise particular components of the distributed processing system code. Thereafter, at step 304, processes 1 and 2 each read particular portions of scenario 202. At step 305, objects 208 and 209 are created by processes 1 and 2 respectively. At step 306, methods are invoked upon objects 208 and 209 by processes 1 and 2, respectively. At step 307, a determination is made whether or not return value 220 is as expected. If not, test manager 201 indicates an error (step 308). 
     However, if a return value is as expected, then at step 309, objects 208 and 209 communicate values. At step 310, a timeout process may be implemented so that if process 2 does not receive value &#34;R&#34; within a specified amount of time, an error is designated by test manager 201 (step 311 ). 
     At step 312, process 2 observes the value, which if the value is not as expected (step 313), then test manager 201 designates an error (step 314). The process then ends at 315. 
     Referring next to FIG. 4, there is illustrated another example of an implementation of the present invention wherein replication framework 212 is utilized. Data processing systems 110, 116 and 122 within network 106 may be utilized for testing the distributed system 100. 
     In this example, a typical spreadsheet program is implemented within machines 110 and 116 and displayed on display 38. For example, it may be assumed that one user on machine 110 may call up a 1994 sales report for display on display 38. A second user on machine 116 may also call up the same 1994 sales report spreadsheet. Simultaneously or not, the two users may begin changing data supported by the spreadsheet, which is replicated to the other machines by replication framework 212. For example, the user on machine 110 may change a cell within the spreadsheet from having a value of two to having a value of three. Asynchronously, replication framework 212 will also copy this change to the 1994 sales report spreadsheet to machine 116 so that the spreadsheet displayed to the second user shows the change to a value three. If replication framework 212 is operating correctly, the object operating within the spreadsheet program on machine 116 should observe the change produced by the object within the spreadsheet program operating on machine 110. This replication may be tested by the utilization of a scenario in accordance with the present invention. 
     A scenario may be thought of as an act or a drama with a script having two or more actors playing their roles according to the script. As long as each actor correctly plays his role, the script will be followed as written, and the test will be successful. However, if one of the actors forgets his lines, an error occurs. For example, if a value of four comes up in machine 116 instead of a value of three, then there is an error in replication framework 212. 
     The script may be stored on any one of the machines within network 106, and is called up by test manager 201 running on one of the machines, in this case machine 122. Test manager 201 then begins playing the script as described above. 
     Referring next to FIG. 5, there is illustrated another simple but contrived example of the utilization of a script for testing distributed system 100 in accordance with the present invention. In this example, replication framework 212 is not utilized, since distributed system 100 does not utilize replication. However, the present invention may still be utilized within such a system. Assume that the following is the expected behavior of the distributed system. In this example, test manager 501 reads the script stored on any one of the machines within network 106. The script may call for process A to begin running on machine 113 whereby it computes and produces a value x. Then, test manager 501 may initiate a transfer of value x to process B running on machine 126, which multiplies x by 3 to produce a value y. Thereafter, process C running on machine 120 may receive the value y and divide it by 6 to produce a value z. Value z is then transferred back to process A on machine 113 wherein, if the distributed system is operating correctly, value x and value z will be equal. An error may be observed if the results of the test run by test manager 501 do not result in x and z being equal. 
     Note, many other various scenarios may be implemented within the present invention in order to exercise various components within the program code utilized to implement distributed system 100. 
     Essentially, a scenario may be written to invoke all sorts of methods on various objects created within system 100, synchronize these various processes, and then check for expected responses. 
     Without the present invention, the requirement would be to start up two or more separate test programs within two or more machines and then simultaneously run the test programs on each machine, while also attempting to synchronize their interactions. The present invention alleviates this problem by implementing a scenario, which may be started within system 100, allowed to proceed, and then the results of the test reviewed after the scenario has run its course. 
     The test manager looks for various errors within the distributed programs utilized to implement distributed system 100, such as the dropping of messages, an error in transmissions between various components of network 106, and the occurrence of events in an expected order. 
     Note, the script may be written in any one of various languages. Furthermore, a script may be written to test various other aspects of distributed processing systems. For example, in the context of testing the replication framework, a scenario could be written to test for recovery when a master process goes down with a write lock. Another one could be written to test recovery when a master process goes down without a write lock. 
     One of the advantages of the present invention is that any time a change is made in the software of the distributed processing system, a scenario can be quickly written to test the new change. 
     Further, the present invention could be implemented on a uniprocessor or multiprocessor system and on multiple processes running on one machine. 
     Although the present invention and its advantages have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.