Abstract:
The present invention provides a method of optimizing an automated test system during a problem determination phase of said test system in a distributed environment with a plurality of local and remote computers, said test system comprising at least a first test suite, said method comprising the steps of: receiving at least said first test suite by a centralized execution controller, said at least first test suite comprising at least a first and second test fragment; receiving at least one breakpoint description comprising a condition for pausing at least said first test suite by said centralized execution controller; sending at least said first test fragment and an execution command of at least said first test fragment from said centralized execution controller to said distributed environment; receiving a event definition and a hierarchy of event definitions describing said executed command by said execution controller; comparing said at least one breakpoint description with the received event definition by a matching mechanism coupled to said centralized execution controller; pausing said test suite execution if said executed event definition matches said breakpoint definition by said centralized execution controller.

Description:
TECHNICAL FIELD 
       [0001]    The invention relates to an automated test system, to a method and to a computer program product. 
       BACKGROUND 
       [0002]    Development of fully automated tests allows a great cost reduction in the test execution phase. It is essential to have test suites fully or almost fully automated in order to successfully test any large-scale product. However, many test scenarios for distributed environments require a big effort to be developed in fully automated fashion. In many cases the saving could be gained by having a majority of scenarios easily automated and minor percentage of additional tests executed manually. Another common problem with test automation is the problem determination phase. Once the automated tests suites are executed, the only information available to the developer are very often the logs collected during the test execution phase. 
         [0003]    There is therefore a need for a method of optimizing an automated test system during the problem determination phase of the test system, a computer program product and a centralized execution controller adapted to perform the method in accordance with the invention. 
       BRIEF SUMMARY 
       [0004]    The illustrative embodiments optimize an automated test system during a problem determination phase of the test system in a distributed environment with a plurality of local computers and a plurality of remote computers. The illustrative embodiments receive a test suite by a centralized execution controller. In the illustrative embodiments the test suite comprises a first test fragment and a second test fragment. The illustrative embodiments receive at least one breakpoint definition comprising a condition for pausing the test suite by the centralized execution controller. The illustrative embodiments send the first test fragment and an execution command of the first test fragment from the centralized execution controller to a distributed environment. The illustrative embodiments receive at least one event definition from a plurality of event definitions and a hierarchy of event definitions describing a result of the execution command by the centralized execution controller. The illustrative embodiments compare the at least one breakpoint definition with the at least one event definition by a matching mechanism coupled to the centralized execution controller. The illustrative embodiments pause the test suite if the at lease one event definition matches the at least one breakpoint definition by the centralized execution controller. 
         [0005]    In other illustrative embodiments, a computer program product comprising computer executable instructions embodied on a computer readable program is provided. The computer readable program, when executed on a computing device, causes the computing device to perform various ones, and combinations of, the operations outlined above with regard to the method illustrative embodiment. 
         [0006]    In yet another illustrative embodiment, a centralized execution controller provided. The centralized execution controller further comprises means for performing various ones, and combinations of, the operations outlined above with regard to the method illustrative embodiment. 
     
    
     
       BRIEF DESCRIPTION OF THE OF THE SEVERAL VIEWS OF THE DRAWINGS 
         [0007]    In the following preferred embodiments of the invention will be described in greater detail by way of example only making reference to the drawings in which: 
           [0008]      FIG. 1  is a block diagram of an automated test system in accordance with a first preferred embodiment of the invention, 
           [0009]      FIG. 2  is a block diagram illustrating a preferred embodiment of a method of the invention, 
           [0010]      FIG. 3  is a block diagram illustrating a second preferred embodiment of a method of the invention, and 
           [0011]      FIG. 4  is a block diagram illustrating a third preferred embodiment of a method of the invention, 
       
    
    
     DETAILED DESCRIPTION 
       [0012]      FIG. 1  shows a block diagram of an automated test system that includes a centralized execution controller  100 , a distributed environment  101  that includes a plurality of local and remote computers  102 , a first test suite  103  and at least an external computer  104 . The first test suite  103  further comprises a description of test scenarios and test fragments  104 , the centralized execution controller  100  is coupled to a matching mechanism  105  and generates a hierarchy event  106 . 
         [0013]    When starting the test system, the centralized execution controller receives a description of the first test suite  103  that includes a description of test scenarios and test fragments  104 . The centralized execution controller  100  also receives at least one break point definition  107  that describes a condition on pausing the test suite execution. This breakpoint definition  107  may be sent by an external computer and may be an input from a user using an application program interface (API). The breakpoint definition  107  may be a plurality of breakpoints and may be received by the centralized controller  100  at any time of the test suite execution  103 . The controller  100  then starts the execution of the test suite  103  and sends test fragments  108  to the distributed environment and execution commands  109 . During the execution, the controller  100  generates event definitions  110  corresponding to the executed commands  109  and, if required, may send the event definition  110  to at least an external computer  104 . This at least external computer  104  may be the same source that send the breakpoint definitions  107 . The external computer may start a request to receive the stream of event definitions  110  at any time during the test suite execution  103 , so that the controller  100  may first send an stream of all event definitions  110  that were generated in the past and may also receive in real time the current event definitions. 
         [0014]    During the test suite execution, the controller  100  or an external entity coupled to the controller  100  generate a hierarchy or tree of events  106  that may be visualized on application program interface in at least the external computer  104 . The controller  100  or an external entity continuously compare the generated event definitions  110  and the breakpoint definitions  107  and pauses the test suite execution when a match between the events  110  and the breakpoints  107  has been found. A user may start collecting logs of the distributed environment or initiate a deep analysis of the environment or machine state. This step may be called the problem determination phase. When this phase has been completed, the user may send a resume command  111  of the test suite to the execution controller  100 . The command may be sent using a debugger port. 
         [0015]    One of the advantages of the invention is that it allows automatically pausing the system when a failure during the test has been detected, so that a user, a tester or an automated tool can look at the content of all the files and the machine or network status that otherwise would not be available during the log collection phase. This allows a great advantage during the problem determination phase, as it gives a great detail of information of the environment at the moment that the failure has occurred. 
         [0016]    Another advantage of the invention is that automating a distributed environment allows a great saving of time and effort, as a single execution controller sends all the initiation preparation, starting and ending of the fragment of test that requires to be executed in the local or remote computers. This same controller may include a comparison mechanism to analyze the pausing conditions and execute the commands when necessary, without modifying the scripts that execute the tests and allowing flexible introduction of new breakpoints at any time of the test execution. 
         [0017]      FIG. 2  shows a block diagram of an automated test system according to the invention.  FIG. 2  comprises a first test suite  201 , a first and a second test scenario  202 ,  203  in a lower hierarchy as the first test suite  201 . The first test scenario  202  comprises a first and a second test fragment  205 ,  206  and second test scenario  203  comprises a third and a fourth test fragment  207 ,  208 . The first test suite  201  represents a set of one or more test scenarios to be executed and is the highest instance within the automated test system. The test suite scenarios as the first and second test scenario  202 ,  203  describes distributed interactions of one or more software components deployed on one or more physical machines. 
         [0018]    The interaction is provided in the form of a computer program with an output, the fundamental programming element of the test suite scenarios is the test fragment. Each of the plurality of test fragments  204 , as those from  204 - 207 , represent a single unit of functionality to be executed as part of the test scenario. Each of the plurality of test fragments  204  can be designated to run on a particular machine in the distributed environment. The test fragment represents the logical functions that need to be executed to complete the test suite scenario. The test fragment may be described based on the machine and the code to be executed. 
         [0019]      FIG. 3  shows a block diagram of an automated test system and the traversal mode sequence to execute the distributed test suites in a pre-order mode.  FIG. 3  shows the suite execution start of the suite 1   301 , the test suite scenario execution start (TSES) of the scenario  1   302 , belonging to the suite execution start  301 . The TSES of scenario  1  ( 302 ) has a subset of: Environment Preparation of the machine  1  ( 303 ), Environment Preparation of the machine  2  ( 304 ), the Test Fragment Execution Start (TFES) of a function  1  into machine  1  ( 305 ), a Test Fragment Execution Start (TFES) of the function  2  into a machine  2  ( 306 ) and a Test Fragment Execution Start (TFES) of the function  1  into the machine  2  ( 307 ). A further subsets of these Test Fragments are a result of Test Fragment Execution End (TFEE)  308  being a child of TFES  305 . A Test Fragment Execution End  309  is a result of TFES  306  and finally a Test Fragment Execution Failure  310  is a result of TFES  307 . Traversing this tree in pre-order mode reflects the execution chronology, using a traversal mode on a preordered sequence. 
         [0020]      FIG. 3  shows an example of this sequence with an execution order from “a” to “h”. The first step of the Suite Execution start SES on suite 1  ( 301 ) is read and executed. This SES  301  further comprises or contains a Test Suite Scenario Execution start of a scenario 1  ( 302 ). The next sequence to be executed belongs to the next hierarchy of the TFES  302  on the left side, which corresponds to the Execution Preparation EP of the machine 1  ( 303 ). When the execution preparation on the machine 1  has been completed, the second event definition to be executed corresponds to the Execution Preparation EP of the machine 2  ( 304 ). When the execution preparation on machine 2  has been completed and as EP of machine 2  has no child of the event, the next event to be executed corresponds to the Test Fragment Execution start from a function 1  into machine 1  ( 305 ). The child of the result of this test fragment  305  is represented as the Test Fragment Execution End TFEE  308 . End events have no attributes and are always considered children of another event. 
         [0021]    The following event definition to be executed belongs to the start of a test fragment execution TFES of function 2  into machine 2  ( 306 ). The result of this TFES  306  is represented as the child in the end of the test fragment execution TFEE  309 . The start of the test fragment execution TFES into machine 2  and running function 1  ( 307 ) is the next element executed. TFES has as a child the failure of the test fragment execution TFEE ( 310 ). As it can be seen in this figure, the sequence starts from the top of the hierarchy going down to the most left child. Subsequent child event definitions are executed until the definition has no more children. Then, the algorithm returns to the next higher level and executes the next event definition to the right until all the event definitions have been executed. One of the advantages of the invention is that the event definitions are described in a high-level language of the language used to write the test scripts. This language allows defining the breakpoint descriptions based on the location, type of function or identification of the event definitions within the hierarchy of event definitions. 
         [0022]      FIG. 4  represents an optimized test system according to the invention and the break point description based on an event definition condition. The elements in the embodiment shown in  FIG. 4  that corresponds to elements of the embodiment of  FIG. 3  have been designated by the same reference signals. The hierarchy of the event definition further comprises the start of the test suite execution TSES into scenario 2  ( 401 ) as a child of Suite Execution Start SES of suite 1  ( 301 ), the failure of a suite execution SEF  402  as a child of the same SES of suite 1  ( 301 ) and the end of a test suite execution TSEE ( 403 ) as a child of TSES of scen 2  ( 401 ). 
         [0023]    One of the advantages of the is to be able to intercept every single step in the test suite execution with the formulation of a break point in a high level language above any script language written to execute the tests. There is no need to plan the pause of the test execution during the test writing phase and the breakpoints may be defined after a first identification of the problem or failure. The invention allows an easy implementation of breakpoints in a distribute environment and across a plurality of local or remote computers. 
         [0024]    For example, during the problem determination case, a user or an automated tool may consider the pause of a test execution after the failure of a given test scenario fragment TFEF. The user may then go to the machine on which the fragment failed and initiate the problem determination phase or analyze the neighbour&#39;s computers, network traffic or the software environment that may be affecting the system. If the break point defines the failure of a test fragment execution TFEF as the child of any test fragment execution on any machine (noted as (*,*) where the first parameter is the identification of a test scenario fragment, the second is the identification of the machine) of a test fragment execution TFES, then the identification of the event definition may be easily located on  FIG. 4 . To narrow even further the break point description, the TFES (*,*) is a child of the start of the test suite execution TSES of scenario 1 . 
         [0025]    Following the syntax as described in  FIG. 4  and with the hierarchy information of the event definition, the matching mechanism easily identifies the condition that in this case corresponds to the TFEF  310 . This breakpoint description corresponds to the syntax: TFEF childof (TFES(*,*) childof (TSES (scen 1 ))). If the hierarchy is analyzed from the top to the bottom, then the breakpoint is found on a child of TSES of scen 1  that corresponds to any function or machine of a TFES(*,*). In this case there are three possibilities: TFES(f 1 ,machine 1 ), TFES(f 2 ,machine 2 ) or TFES(f 1 ,machine 2 ). The breakpoint is defined as a TFEF that is a child of any of these three TFES. This can only correspond to the TFEF  319 . 
         [0026]    The next break point to be identified  405 , is described by the syntax of the break point description as: TFES (*,*) after (EP(*)) childof (TSES(*) childof (SES(suite 1 ))). One of the matching mechanisms may interpret the syntax in the following manner: The TFES corresponds to any scenario fragment executed on any machine. The TFES comes after the execution of an Environment preparation EP into any machine. The TFES (*,*) is as well a child of a TSES executed into any scenario. This TSES is a child of the SES into suite 1 . In the hierarchy of event definition of  FIG. 4 , this matches with the event definition TFES(f 1 , machine 1 )  305 . This breakpoint description pauses the test system into any event definition that matches these characteristics and therefore represents an advantage of the invention for test systems executed on a distributed environment having a plurality of local and remote computers. 
         [0027]    The general grammar expressed to describe the breakpoint description may use the following syntax:
       BREAKPOINT ::=EVENT_DESC   EVENT_DESC ::=EVENT PREV_EVENT_ELEMENT PARENT_EVENT_LIST       
 
         [0030]    As a general rule, the first element (EVENT) represents the event definition where the matching mechanism compares the breakpoint description to try to find the pausing condition. The event may be any of the suite execution start SES, end SEE or failure SEF; by the test scenario execution start TSES, end TSEE, failure TSEF; by the environment preparation EP, environment clean up EC; or by the test fragment execution start TFES, end TEEE or failure TEEF. Another possible events are: paused test execution and resumed test execution. More events may be defined according to the type of test system to be executed. In the example  405 , the event corresponds to any TFES(*,*). The event definitions may be further defined using a string of alphanumeric characters, a machine or function identification where the test is being executed, or some other parameter to identify the location or type of test. 
         [0031]    The previous event element (PREV_EVENT_ELEMENT) describes the event definition that was executed before the EVENT. The PREV_EVENT_ELEMENT may be then uses the syntax:
       PREV_EVENT_ELEMENT ::=after (EVENT_DESC ). This syntax in the example  405  was described by “after (EP(*))”. This represented all the event definitions that were executed after any Environment Preparation. The third element of the breakpoint description is formed by the PARENT_EVENT_LIST that describes the parent of the EVENT. This may be represented in the syntax as:   PARENT_EVENT_LIST ::=childof (EVENT_DESC ). The syntax describes the event description as a child of a determined event definition. In the example  405 , the TFES (*,*) is a child of TSES(*).       
 
         [0034]    While the foregoing has been with reference to particular embodiments of the invention, it will be appreciated by those skilled in the art that changes in these embodiments may be made without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims.