Patent Publication Number: US-6704883-B1

Title: Event-enabled distributed testing system

Description:
BACKGROUND 
     1. Field of Invention 
     The present invention relates to computer networks, and in particular to distributed testing on computer networks that is performed by test agents under the control of a test controller. 
     2. Related Art 
     In a conventional eventing system model (e.g., publish/subscribe), actions by an entity (e.g., software) are based on the presence of “events.” An event is any occurrence that the entity can sense. For example, an event may be mouse pointer movement into a designated portion of a video display, or may be a failure warning message sent by a hardware component. In some models a particular event has two parts: the event type, defining the category into which the event is placed, and the event data that contains specific information regarding this particular event. 
     A publisher is an entity that publishes (e.g., generates, transmits) an event to event engine. For example, a hard disk drive may publish a message that it is failing. The event engine is an entity that broadcasts (e.g., transmits using a one-to-many relationship) one or more events to event subscribers. An event subscriber is an entity that has requested (subscribed) notification from the event engine when a particular event occurs. Thus when a particular type of event occurs, the publisher publishes the event to the event engine which, in turn, broadcasts the event to one or more subscribers which then receive the event. See U.S. Pat. Nos. 5,187,789, 5,257,369, and 5,339,392, incorporated herein by reference, for examples of publisher/subscriber methodology. 
     Event publishing allows information to be distributed without direct addressing. Subscribers wait to receive selected events to which they subscribe, and will take predetermined action upon sensing the subscribed event has occurred. One predetermined event may be to publish a second event upon sensing the first event. In this way, one component may act as both a publisher and a subscriber. 
     A typical computer network is two or more interconnected computers capable of communicating with each other. Many computer network tests require that testing operations be conducted among two or more computers connected to the network. During some typical network distributed testing operations, one principle entity (e.g., software component stored on one computer) acts as a test controller and one or more other computers act as agents of the test controllers. Thus a test agent is an entity (e.g., software component stored on another computer) that acts at the direction of a test controller. Test agents are typically coded instructions stored in one or more test agent computers for performing the test. 
     Typical commercial test products are designed so that the coded test agent instructions are shipped to each test agent computer location, and then are installed at each location. The installation and configuration procedures are cumbersome when many test agents are involved in a single test. During testing, the test controller is required to maintain one-to-one (e.g., point-to-point) connection with each agent. Thus test configuration and maintenance is complex when many test agents are involved in a test. The test process is further complicated when the network includes different computers (e.g., routers, switches, end systems) configured to operate in two or more architectures, such as in UNIX® or WindowsNT®. Furthermore, the test controller must constantly monitor each agent&#39;s capabilities, and for when individual agents are added or deleted from the test. Test agent changes typically require the test be reconfigured. 
     Along with simplicity, a desirable distributed network test feature is the simultaneous, or near-simultaneous, execution by many test agents of a particular test. Simultaneous and near-simultaneous test execution by many agents allows, for example, testing of a sequencing algorithm at a particular network location when the agents are directed to send data to that one particular location. Such simultaneous or near-simultaneous execution is difficult under existing systems operating using many architectures. 
     What is required is an efficient and flexible distributed testing system that allows test agents operating in many architectures to simultaneously, or near-simultaneously, execute the desired test. 
     SUMMARY 
     In accordance with the present invention, a distributed testing environment includes a test controller, an event engine, and one or more test agents interconnected by a computer network. Before a test begins, each test agent is configured with a test script interpreter designed to operate under the software architecture of the test agent&#39;s computer. The interpreters operate in a conventional manner, reading and executing one coded instruction before reading and executing the next coded instruction. In embodiments of the present invention, the interpreters read and execute instructions contained in a test script. 
     The present invention&#39;s testing environment is event-driven. In accordance with the present invention, the event to which the test agents subscribe is publication of a test script by the test controller. The event type identifies the type of test to be executed and the event data is the actual test script to be run. Thus each test agent initiates test execution upon the receipt of the test script broadcast by the event engine. The test agents use the interpreters to carry out the test script instructions. 
     During test set-up, test agents negotiate with the test controller and are registered with the event engine. The negotiations allow the test controller to determine the test agent&#39;s configuration and capabilities. Registration informs the event engine that the test agent is available for the test. A test agent may be registered, but may not subscribe to all test scripts. 
     Once test agents are registered, the test controller receives a test script from a user or other source. The test controller parses the test script, and based on test script contents publishes the script to the event engine which broadcasts the script to the test agents. Test agents subscribing to the particular type of test script receive the broadcast script. The receipt of the test script is the event that triggers the interpreter to execute the instructions in the test script. 
     After running the test script, each test agent transfers the test results to the test controller. In turn, the test controller transfers all test results to an analysis agent. The analysis agent performs conventional test result analysis and produces a test report that is output to the user. 
    
    
     BRIEF DESCRIPTION OF THE DRAWINGS 
     FIG. 1 is a block diagram showing a hardware embodiment of the present invention. 
     FIG. 2 is a block diagram showing components of, and data flow in, an embodiment of the present invention. 
     FIG. 3 is a flow diagram illustrating acts performed by a test controller. 
     FIG. 4 is a flow diagram illustrating acts performed by a test agent. 
     FIG. 5 is a flow diagram illustrating acts performed by an analysis agent. 
    
    
     DETAILED DESCRIPTION 
     Identical element numbers in the accompanying drawings represent the same element. 
     FIG. 1 is a block diagram illustrating an embodiment of testing environment  100 . As shown, control device  102  is connected to conventional network  104 . Control device  102  is an end system computer (e.g., network server) and includes memory  106 . Memory  106  contains coded instructions (software), described below, to be executed as computer programs in accordance with the invention. 
     Conventional user interface  108  is connected to control device  102 . Interface  108  includes one or more conventional human-machine interface devices (e.g., video display terminal, keyboard, printer, etc.) to allow the human test operator and control device  102  to communicate. 
     Three test devices  110 ,  112 , and  114  are shown connected to network  104 . These three test devices are illustrative; embodiments of the invention may have any number (e.g., thousands) of test devices. As shown, each test device is a computer and may be an end system (e.g., server, client) or intermediate system (e.g., router). Test devices are not restricted to operating under the same operating system. For example, device  110  may be operating under the WindowsNT® operating system, device  112  under the UNIX® operating system, and  114  under the CiscoIOS® operating system. 
     Each device  110 ,  112 , and  114  includes a memory  110   a,    112   a,  and  114   a  respectively. Each memory  110   a,    112   a,  and  114   a  contains coded instructions to act as an interpreter configured to run in that device&#39;s native operating system. The interpreter is described below. Tests as directed by control device  102  are performed by software running on test devices  110 ,  112 , and  114  which embody test agents. 
     FIG. 1 also shows event device  116  and analysis device  118  coupled to network  104 . Devices  116  and  118  include conventional memories  120  and  122 , respectively. Stored in memory  120  is an event engine (e.g., software) that receives one or more events from the test controller in device  102  for broadcast to the test devices. Stored in memory  122  is an analysis agent (e.g., software) that acts to perform analysis of test results at the direction of the test controller. The event engine and the analysis agent are shown in devices distinct from control device  102 , although this configuration is illustrative and many hardware configurations may be used. 
     Control device  102  includes conventional storage medium drive  130 . Conventional storage medium  132  (e.g., one or more CD-ROMs, one or more magnetic disks) is shown positioned in drive  130 . Software associated with this invention may be stored on medium  132 . 
     FIG. 1 depicts one possible hardware configuration embodying the present invention. To more clearly describe the invention, however, FIG. 2 illustrates a more general embodiment of the invention, and shows major components and data flow. 
     Test controller  202  controls the test environment for the test to be performed. User interface  108  and event engine  203  are connected to communicate with test controller  202 . In response to user inputs on interface  108 , one or more test agents are registered with event engine  203  for the test. 
     When the user directs an entity (e.g., a software component operating at a particular network address) be registered, controller  202  acts to control a negotiation and registration process with the entity through event engine  203  by exchanging information  210 . The test agent publishes its registration information (e.g., capabilities, such as native operating system) to test controller  202 . If the entity&#39;s capabilities allow it to act as an agent, the entity is registered by exchanging additional information  210 . When the agent is registered, the agent receives from controller  202  an interpreter written to operate in the agent&#39;s native language (e.g., CiscoIOS®). Each test agent later uses the interpreter to run a test script. Once all required agents are registered, the test is performed. As shown, three test agents  204 ,  206 , and  208  have been registered with controller  202  through the exchange of information  210   a,    210   b,  and  210   c,  respectively. 
     In accordance with the invention, the test event is the publishing of a test script to be run by each subscribing test agent. In one embodiment, controller  202  publishes a test script for event engine  203  to broadcast (transmitted using a one-to-many relationship) to the test agents. In other embodiments controller  202  publishes more than one test script for the test agents. Broadcasting the script is much more efficient than sending the script to each test agent using individual connections between controller  202  and each test agent. 
     In accordance with the invention, receiving the test script is the event that each test agent uses to start the test. Thus near-simultaneous test execution may be accomplished by many agents. This testing method allows efficient testing of, for example, prioritization algorithms in devices receiving many near-simultaneous inputs. One possible test may be to evaluate a network server&#39;s prioritization algorithm when the server simultaneously receives one unique change of name request from many unique test agents. In other embodiments one script may be sent to one test agent, and a second script sent to another test agent. If, for example, test agent  204  is a server and test agents  206  and  208  are clients, then one script testing server  204  software may use inputs from client  206  and  208  software controlled by a second script. 
     In the FIG. 2 embodiment, test agents  204 ,  206 , and  208  each execute test script  214  upon receipt. After agents  204 ,  206 , and  208  complete their individual tests, each agent sends test results back to controller  202  for analysis. As shown, test agent  204  sends test results  216   a,  agent  206  sends results  216   b,  and agent  208  sends results  216   c.    
     Controller  202  stores test results  216  and forwards results  216  to analysis agent  218  when all test agents have reported their results. The analysis agent is registered with the test controller using a procedure similar to test agent registration. The analysis agent publishes its capabilities, thereby informing the controller what test results the analysis agent can process. Analysis agent  218  performs a conventional analysis on the combined test results from each agent and sends test analysis report  220  to controller  202 . Controller  202  then directs the analysis report to interface  108  for output to the user. 
     FIG. 3 is a flow diagram illustrating one embodiment of a method performed in accordance with the present invention. The acts shown and described in relation to FIG. 3 are carried out by a test controller, such as controller  202  (FIG.  2 ), publishing to an event engine that broadcasts the test script. Nevertheless, persons familiar with network software will understand that the acts shown and described may be performed using a single computer, or may be distributed among two or more computers. 
     Block  302  represents that the test controller waits in a conventional wait state for one of four test-related inputs. The test controller receives requests to register test agents, test scripts to be sent to registered test agents, test results from test agents, and test reports from the analysis agent. Inputs may be received, for example, from user interface  108 , one or more test agents  206 - 208 , and analysis agent  218  (FIG.  2 ). 
     Block  304  represents that the test controller determines if the input is a request to register a particular agent as a test agent. If so, the test controller begins to negotiate with the particular agent as represented by block  306 . During negotiations, the test controller determines the requested agent&#39;s capabilities, such as the agent&#39;s operating system and application programs. The test controller determines if the particular agent can act as a test agent, as shown by block  308 . If the particular agent does not qualify to act as a test agent, the test controller returns to block  302  and awaits another test input. If the particular agent qualifies as a test agent, block  310  represents that the test controller registers the particular agent as a test agent for the test to be performed. 
     When an agent is registered as a test agent, the test controller sends a test script interpreter to the newly registered test agent as shown by block  312 . After registering the test agent and sending the test script interpreter, the test controller returns to block  302  and awaits another test input. 
     Persons familiar with computer programming will understand that the interpreter sent is configured to operate in the test agent&#39;s architecture. For example, a test script interpreter sent to a UNIX® test agent is configured to lexically analyze the instructions in a particular test script and execute the instructions in the UNIX® environment. A test script interpreter sent to a CiscoIOS® test agent is configured to lexically analyze and execute the same particular test script, but this time in the CiscoIOS® environment. Thus in this example, when the test begins, the test controller sends the same test script to both the test agent operating in UNIX® and the test agent operating in CiscoIOS®. 
     A second input that the test controller may receive is a test script. This test script may be a single script, or may contain more than one component scripts, each to be run on different test agents as described above. The test script sent to the test controller contains a test type (event type) indicator and one or more component test scripts (event data) associated with that test type. In  320  the controller determines from the user input that the input is a test script. In  322  the controller parses the received test script and associates the test type with the one or more component scripts. In  324  the controller publishes each script(s) and its associated test type for broadcast by the event engine as described above. Thus the test controller publishes all test scripts required to perform the test and the test agents subscribe to the particular scripts they must run to complete the test. 
     The event engine triggers the test agents to run the test script by broadcasting the script itself. Broadcasting allows the test controller to distribute the test script to test agents more quickly than if the test scripts are individually addressed. In some embodiments, however, the script may be individually addressed or multi-cast to designated test agents, depending on the event type to which the test agent subscribes. As described below, the receipt by a test agent of the test script is the event that triggers the test agent to run the test. After the test controller sends the test script to the test agents, the controller returns to block  302  and awaits another test-related input. 
     A third test-related input the test controller receives is the test result from each test agent. After each agent runs the test script, each test agent sends the test result back to the test controller. Block  330  represents that the test controller subscribes to test result event types and recognizes the input as a test result. In  332  the controller stores the test result in memory. The test controller determines if all test agents have returned test results as shown in block  334 . If some test results are missing, the test controller returns to block  302  and waits for additional inputs. Once the test controller has received test results from all test agents, the test controller sends all test results to the analysis agent, as represented by block  336 . 
     The final test-related input the test controller receives is the test report from the analysis agent. Block  340  represents that the test controller determines if the input is a test report. If the input is a test report, the test controller outputs the test report to the user, shown by block  342 . 
     Of importance is that the test execution on each test agent is event enabled. That is, each unique test agent begins to run the test script when each unique test agent recognizes that a common event has occurred. And, in accordance with embodiments of the invention, the event that triggers each test agent to execute the test script is each unique agent&#39;s receipt of the test script itself. The arrival of the test script is the event that triggers each agent to interpret and run the test script. 
     FIG. 4 is a flow diagram illustrating actions in accordance with the invention taken by each test agent. Block  402  represents that the test agent waits in a conventional wait state for the test script(s) to which it subscribes to arrive. When the test script arrives, the arrival is the event that triggers the test agent interpreter to run the test script, shown in block  404 . The test script may direct the test agent to run one test or multiple tests. Or, the agent may run multiple test scripts. Block  406  shows that when the test agent completes running the test script, the test agent reports the test result to the test controller, as described above. 
     FIG. 5 is a flow diagram illustrating actions in accordance with the invention taken by the analysis agent. Block  506  represents that the analysis agent waits in a conventional wait state to receive all test results from the test controller. Block  504  represents that the analysis agent performs one or more conventional analyses of the test results. Then, the analysis agent sends the analysis as a test report back to the test controller, as shown by block  506 . 
     Persons familiar with the field of network testing will appreciate that many variations of the present invention exist. Although the invention has been described above in terms of specific embodiments, the scope of the invention is limited only by the following claims.