Method and apparatus for correlating input and output messages of system under test

A method and apparatus for determining correlation between input and output messages in a system under test (SUT) is provided in the present invention. The SUT is provided with preset watch-points, and the running of the SUT is detected by triggering watch-points in a test platform at its run time. The method includes the steps of: upon detecting a message input operation, finding a variable that stores an input message, associating the variable with a tag of the input message, and adding a watch-point for the variable in the test platform; as well as, upon detecting network output operation, finding a variable that stores an output message of the SUT; and determining correlation between the output message and an input message according to a tag associated with the variable that stores the output message.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority under 35 U.S.C. §119 from China Patent Application No. 201210177617.1 filed May 31, 2012 the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to the computer field, and more specifically, to a method and apparatus for determining correlation between input and output messages of a system under test.

2. Description of Related Art

The Internet of Things (IOT) has been considered an important component of new generation information technologies, and is defined as a network connecting things to the internet through information sensing devices, for example, Radio Frequency Identification (RFID), infrared sensors, Global Positioning System (GPS), laser scanners, etc., according to agreed protocols for information exchanging and communication, so as to realize intelligent identification, locating, tracking, monitoring, and management of things.

In distributed networks, such as the IOT, functional test is an important stage in system development and malfunction detection. For example, in a system of collecting temperature information for alert, temperature sensors distributed at different locations sense temperature and send collected temperature information to a certain node, which determines whether to send alert information for alert according to the received temperature information. In such an IOT, whether the designed node meets design requirements, or a certain output event, is dictated by which input event or events at the node can be determined by the functional test.

When a system (it can be a node or a portion thereof in a network, or a part formed by multiple nodes in the network) has been designed, performing a functional test on the system is required to determine whether the system can meet design requirements. For a faulted system it is possible to perform functional test on the system to locate malfunction. Because a system may receive a large number of input messages and send a large number of output messages, conveniently determining correlation between input and output messages to facilitate functional test has become a major demand.

Recently, several methods for testing correlation between input and output messages in a network have emerged. In these methods, correlation between input and output messages is determined through semantic analysis, that is, the specific meaning of data contained in input and output messages needs to be parsed for correlation analysis. In doing so, either a great deal of human effort is involved, or data modeling through a huge amount of statistics is necessary, leading to high complexity.

Currently, there are several code analysis tools. For example, the Wisconsi tool may find out associated variables and/or functions from codes through static code analysis without executing the codes. The JSlice tool relies on special Java Virtual Machine (JVM) which records code execution paths through executing codes and finds out associated variables and/or functions following the code execution paths. The SPYDER tool records code dependency trees based on probes used for monitoring which are embedded into executable machine codes at code compiling time, and thereby finds out associated variables and/or functions according to the code dependency trees. The probes cannot be changed during code execution. The above tools are either inaccurate or rigid or intruding for the codes of systems under tests. They are not generally applicable in determining correlation between input and output messages conveniently and accurately.

SUMMARY OF THE INVENTION

The present invention provides a method and apparatus of determining correlation between input and output messages in a system under test by means of watch-points and tags in a simple and convenient way, without the need for performing semantic analysis on messages and the need for instrumenting codes, recording and analyzing the complete execution paths of codes.

The present invention provides a method for determining correlation between input and output messages in a system under test (SUT) having preset watch-points in a test platform, the method includes the steps of: upon detecting a message input operation, finding a variable that stores an input message, associating the variable with a tag of the input message, and adding a watch-point for the variable in the test platform; as well as, upon detecting network output operation, finding a variable that stores an output message of the SUT; and determining correlation between the output message and an input message according to a tag associated with the variable that stores the output message.

The present invention also provides an apparatus for determining correlation between input and output messages in a system under test (SUT) having preset watch-points in a test platform, the apparatus includes: a process module for associating a variable with a tag of an input message and for adding a watch-point for the variable, in response to detecting the variable associated with the input message; as well as, a correlation determining module for determining correlation between an output message and an input message according to a tag associated with a variable for storing the output message, in response to detecting the variable for storing the output message of the SUT.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In accordance with the present invention, through setting watch-points, variables having relation to input messages can be tracked without paying attention to the specific execution flow inside source codes, and through assigning tags, different variables influenced by the same input message can be associated with the same tag. Thus, correlation between input and output messages can be determined simply and conveniently with watch-points and tags, and semantic analysis from the semantic level can be avoided.

Referring toFIG. 1, an exemplary computer system100is shown, which is applicable to implement the embodiments of the present invention. As shown inFIG. 1, the computer system100may include: CPU (Central Process Unit)101, RAM (Random Access Memory)102, ROM (Read Only Memory)103, System Bus104, Hard Drive Controller105, Keyboard Controller106, Serial Interface Controller107, Parallel Interface Controller108, Display Controller109, Hard Drive110, Keyboard111, Serial Peripheral Equipment112, Parallel Peripheral Equipment113and Display114. Among above devices, CPU101, RAM102, ROM103, Hard Drive Controller105, Keyboard Controller106, Serial Interface Controller107, Parallel Interface Controller108and Display Controller109are coupled to the System Bus104. Hard Drive110is coupled to Hard Drive Controller105. Keyboard111is coupled to Keyboard Controller106. Serial Peripheral Equipment112is coupled to Serial Interface Controller107. Parallel Peripheral Equipment113is coupled to Parallel Interface Controller108. And, Display114is coupled to Display Controller109. It should be understood that the structure as shown inFIG. 1is only for the exemplary purpose rather than any limitation to the present invention. In some cases, some devices may be added to or removed from the computer system100based on specific situations.

With reference herein after toFIG. 2, the method200of determining correlation between input and output messages according to an embodiment of the present invention will be described. Herein, according to embodiments of the present invention, input and output messages refer to input and output messages of a system under test (SUT); the SUT is tested through loading the SUT into a test platform to execute. The test platform may comprise test run time environment, such as, JVM (Java Virtual Machine), JVMTI (Java Virtual Machine Tool Interface), GDB (GNU Project Debugger), etc.

The method200comprises: at step S210, upon detecting a message input operation, executing the following operations: finding the variable that stores the input message, associating the variable with a tag of the input message, and adding a watch-point in the test platform for this variable; at step S220, upon detecting network output operation, finding the variable that stores the output message of the SUT, determining correlation between the output message and an input message according to a tag associated with the variable storing the output message.

In the method200, with the execution of the SUT on the test platform, through setting watch-points for the SUT and associating all variables related to an input message with a tag corresponding to this input message, correlation between input and output messages can be determined according to tags. A variable related to an input message refers to a variable directly or indirectly related to the input message, namely, the input message imposes influences directly or indirectly on the variable. The imposed influence may comprise an influence on the value of the variable, or may comprise an influence on variable control. For example, if information A in an input message directly changes the value of a variable X, or changes the value of the variable X through an intermediate variable, then the variable X is related to the input message. As another example, if information A in an input message causes a variable X to be produced or changes the value of the variable X in the case that information A satisfies a certain condition, then the variable X is also related to the input message.

Further, in step S210, when a variable is determined to be related to an input message according to a watch-point that has been set, the variable is associated with a tag corresponding to the input message (hereinafter also referred to as a tag of the input message), and in the case that there is no watch-point toggled on the variable, a watch-point is set to the variable. In contrast, in the case that there has been a watch-point set to the variable, the watch-point is kept for the variable. The watch-point that has been set may be a watch-point set beforehand or a watch-point newly set with the execution of step S210. Specifically, the watch-point set beforehand may be a watch-point that is set in the test platform after the SUT is loaded into the test platform and before the SUT is executed, or a watch-point that is embedded in the SUT in advance before the SUT is loaded into the test platform.

According to an embodiment of the present invention, a watch-point can be preset for an input function for receiving an input message. Thus, when it is determined based on the set watch-point that the input function is invoked to receive the input message, a variable for storing the input message can be detected.

According to an embodiment of the present invention, a watch-point can be preset to an output function for sending an output message. Thus, when it is determined based on the set watch-point that the output function is invoked to output the output message, since it can be determined which variable of the output function stores the output message according to the output function, correlation can be determined according to a tag associated with the determined variable for storing the output message.

Generally, network interfaces of the SUT perform network input/output through invoking library functions, and different network interfaces may have library functions in the same form in case of supporting the same network protocols and employing the same programming language and operating system. These network interfaces may receive an input message through invoking an input function (for example, the receive( ) function), and send output messages through invoking an output function (for example, the send( ) function). When it is required to determine correlation between input and output messages of the SUT, input and output functions in the SUT can be found out in advance and watch-points can be set to them respectively, so that when the SUT runs to the positions corresponding to the input and output functions, since the watch-points are encountered, the steps in method200are triggered to perform. Herein, input and output functions can also be referred to as input and output methods in some programming languages (such as C++), the specific names of which do not make any limitation to the present invention.

Because different SUTs may support different network protocols and employ different programming languages and operating systems, a test specification of an SUT may be first obtained, and then input and output functions required to be set with watch-points are determined according to the obtained test specification. In this way, for different SUTs, input and output functions can be properly identified and thus the initial watch-points that is utilized for the input and output functions can be properly set. The test specification may comprise network protocols supported by an SUT required to be tested currently and programming languages and operating system of the SUT. Undoubtedly, those skilled in the art may conceive of test specifications in other forms. Through specifying a test specification of an SUT, input and output functions of the SUT can be determined.

For example, in order to determine input and output functions through the test specification, a code pattern can be predefined. The code pattern may be identified through the test specification, and the input and output functions may be identified through the code pattern. The code pattern can be represented in a table with the following fields: a library or package encapsulating the network input/output (I/O) functions, a function (for procedural programming language) implementing the network I/O, a class and method (for object-oriented programming language) implementing the network I/O, an indicator on how the received message is retuned for network input operation (i.e., the SUT receives an input message) (for example, as a returned value of a function/method, or storing in a parameter of a function/method), and an indicator on how message to be sent is nominated in network output operation (i.e., the SUT sends an output message) (for example, storing in a parameter of a function/method). After the code pattern has been determined, not only input and output function names may be known so that watch-points may be set, but also the positions of variables for storing input and output messages may be known either. For example, when the input function is invoked, a watch-point will be set for the variable for storing the input message, and when the output function is invoked, the variable for storing the output message is determined.

After the watch-points have been set for the input and output functions, the SUT starts to run on the test platform. When it is determined based on the set watch-point that the input function is invoked to receive the input message, a watch-point will be set to the variable for storing the input message.

For example, because a watch-point has been set to the input function, it may be monitored that an input message is received through invoking the input function, so that the variable for storing the input message has relation to the input message, and another watch-point needs to be set to it to monitor the variable. Further, a tag corresponding to the input message needs to be assigned to the variable to indicate the variable relating to which input message. The tag corresponding to the input message may be a message ID of the input message, a character string of the input message, or other forms that can be conceived of by those skilled in the art, through which the corresponding input message can be retrieved. Further, those skilled in the art may understand that the variable for storing an input message may be a variable representing an address space for storing the input message or a variable pointing to the address space.

Further, when it is determined based on the set watch-point that an output message is sent through invoking the output function, correlation between the output message and an input message is determined based on a tag associated with the variable for storing the output message.

For example, because a watch-point has been set to the output function, whether an output message is sent through invoking the output function may be monitored. Through viewing a tag associated with the variable for storing the output message, an input message corresponding to the tag can be found, thereby it can be determined which input message or messages has association with the output message.

According to an embodiment of the present invention, a tag associated with a variable for storing an output message may comprise a tag directly associated with the variable for storing the output message, a tag associated with the variable for storing the output message through a branch identifier contained in a stack corresponding to a thread where the variable is located, or the both. As described in detail hereinafter, through a watch-point variable list, a tag directly associated with a variable in the list can be determined. Further, because the invoking of the output function may occur in a conditional branch after a conditional decision statement, a tag that is indirectly associated with a variable in the conditional branch can be determined through a branch identifier pushed into a stack, that is, the tag indirectly associated with the variable is the tag directly associated with the branch identifier of the conditional branch where the variable is located. The tag directly associated with the branch identifier may be a tag that is associated with the branch identifier when it is pushed into the stack, as described hereinafter. Similarly, those skilled in the art may understand that a variable for storing an output message may be a variable representing an address space for storing the output message or a variable pointing to the address space.

During the execution of step S210, there may be a new variable to which a new watch-point is set. Step S210is repeatedly executed due to the appearance of new watch-points, so that watch-points in the test platform can be updated to allow all variables having relations with an input message to be tracked and monitored. When a variable is related to an input message, the input message can be associated with the variable by a tag corresponding to the input message after the tag is assigned to the variable. In this way, variables related to the same input message can be associated with the same tag, and the same variable may be associated with multiple tags due to influences by multiple input messages. At step S220, since an output message is represented by a certain variable in the SUT, the output message can be associated with corresponding input messages through tags, which may include a tag directly associated with the variable and/or a tag indirectly associated with the variable through a branch identifier that is still remained in a stack of a thread in which the variable is located, and thus the correlation between input and output messages can be determined.

According to the method of determining correlation between input and output messages provided in the embodiment of the present invention, through setting watch-points, variables having relation to input messages can be tracked without paying attention to the specific execution flow inside source codes, and through assigning tags, different variables influenced by the same input message can be associated with the same tag. Thus, correlation between input and output messages can be determined simply and conveniently with watch-points and tags, and semantic analysis from the semantic level can be avoided.

Next, an example of the implementation system300of determining correlation between input and output messages shown inFIG. 3will be described, and the method400and the method500of determining correlation between input and output messages will be described in more detail with the system300shown inFIG. 3in conjunction withFIG. 4andFIG. 5.

Referring toFIG. 3, the system300of determining correlation between input and output messages comprises a system310for implementing the method200and a test platform360for loading an SUT370. The SUT370runs in the test run time environment of the test platform360. Although the system310and the test platform360are shown separately inFIG. 3, those skilled in the art may understand that a part or all of the system310can be integrated into the test platform360, becoming a component of the test run time environment.

For example, after a development of a product is completed, it is necessary for a tester to perform functional test on the developed product to determine whether it meets design requirements. At this time, the product needed be tested is the SUT370shown inFIG. 3.

The tester may input concerned network protocols supported by the SUT370(such as, TCP (Transmission Control Protocol), UDP (User Datagram Protocol), etc.), programming languages of the SUT370(such as C++, Java, etc.) and a target operating system of the SUT370(such as, Linux, Unix, etc.) as a test specification305into a code pattern generator320. The code pattern generator320searches a predefined code pattern library315according to the received test specification to find out a code pattern325in conformity to the test specification, wherein the code pattern325can be also referred to as a network I/O pattern. With the code pattern, an input function for receiving input messages and an output function for sending output messages of the SUT370can be determined under the corresponding test specification.

The code pattern generator320sends the code pattern325determined according to the test specification to a watch-point editor330. The watch-point editor330determines input and output functions for which watch-points need to be set according to the code pattern325, and notifies the test platform360of the input and output functions that need to be set with watch-points, to enable the test platform360to set watch-points to the input and output functions of the SUT370after the SUT370is loaded.

In addition to setting watch-points for the input and output functions according to the code pattern325, the watch-point editor330needs to set new watch-points or delete watch-points that have been set according to information fed back by an I/O monitor/analyzer350, which will be described in detail hereinafter. When the watch-point editor330needs to set a watch-point, the watch-point editor330sends a watch-point adding command to the test platform360to instruct the test platform360to add a watch-point at a corresponding location in the SUT370. When the watch-point editor330needs to delete a watch-point that has been set, the watch-point editor330sends a watch-point deleting command to the test platform360to instruct the test platform360to delete a watch-point at a corresponding location.

The so-called “watch-point” refers to a location that needs to be monitored in the SUT370, i.e., in the code of the SUT370. When the SUT370proceeds to a watch-point, the test platform360is triggered to send a notification to the system310, causing the I/O monitor/analyzer350to perform related operations. When the I/O monitor/analyzer350determines that a watch-point is to be added or deleted according to the received notification, the I/O monitor/analyzer350sends information to the watch-point editor330, causing the watch-point editor330to set or delete a watch-point through sending a watch-point adding command or a watch-point deleting command.

Since the watch-point adding command or the watch-point deleting command generated by the watch-point editor330may not be recognized by the test platform360, it is required to provide an adaptor340in such a case, so that these commands can be converted into commands that can be recognized by the test platform360. Similarly, since notifications sent to the system310by the test platform360based on watch-points set in the SUT370may not be recognized by the I/O monitor/analyzer350, in such a case, the adaptor340is also required to convert the notifications sent by the test platform360into the ones that can be recognized by the I/O monitor/analyzer350. Obviously, those skilled in the art may understand that the adaptor340is not necessary when the system310and the test platform360can recognize information communicated there-between.

Logically, there may be two lists provided by the I/O monitor/analyzer350. One list is an input-message list utilized for recording tags that are assigned to input messages (for example, message IDs, or other message identifiers corresponding to input messages), and the other list is a watch-point variable list utilized for recording association between variables having watch-points set thereon and tags (for example, message IDs). For example, when a variable corresponding to a watch-point is influenced by an input message having a message ID “e” directly or indirectly, the tag “e” is assigned to the variable to associate the variable with the input message having the message ID “e”. A variable having a watch-point set thereon are also called as a watch-point variable hereinafter. A watch-point variable is a variable to be monitored, and when the watch-point variable is processed by some operation, the test platform360sends a corresponding notification to the system310.

The I/O monitor/analyzer350receives notifications sent by the test platform360based on watch-points and performs corresponding processes.

When the I/O monitor/analyzer350receives a notification, indicating that an input function is invoked to receive an input message, sent by the test platform360based on a watch-point set on the input function, the I/O monitor/analyzer350assigns a message ID (in this example, it is represented by the message ID “e”) to the input message, notifies the watch-point editor330to set a watch-point for a variable that is utilized for storing the input message, and assigns the message ID “e” to the variable as a tag in the watch-point variable list.

When the I/O monitor/analyzer350receives a notification, indicating that a read operation is performed on a variable “p” in a conditional decision statement, sent by the test platform360based on a watch-point set on the variable “p”, the I/O monitor/analyzer350checks all the variables on which write operations are performed in a conditional branch executed following the conditional decision statement. If there are some written variables having no watch-point set thereon, it is necessary to instruct the watch-point editor330to perform watch-point setting on them, and add them into the watch-point variable list. On the other hand, if there are some written variables already having the watch-points set thereon, the watch-points are kept. Further, the I/O monitor/analyzer350may assign a branch ID for the conditional branch to be executed, associate the branch ID with a tag associated with the variable p, and push the branch ID into a stack corresponding to the current thread. Correlation between the branch ID and the variable “p” can be recorded in the following manner. For example, the branch ID and the tag associated with the variable “p” can be correspondingly recorded in the watch-point variable list or other storage spaces, or the branch ID and the associated tag of the variable “p” can be pushed into the stack together as an entirety. Herein, the “stack” refers to a stack provided in the I/O monitor/analyzer350, and each thread has its own stack in a one-to-one correspondence. In this case, the I/O monitor/analyzer350notifies the watch-point editor330to set a watch-point on the exit point of the branch.

When the I/O monitor/analyzer350receives a notification, indicating that the exit point of the branch is reached, sent by the test platform360based on the watch-point set on the exit point of the branch, the I/O monitor/analyzer350pops the branch ID that has been pushed into the stack of the current thread. Further, the I/O monitor/analyzer350may notify the watch-point editor330to delete the watch-point set on the exit point, and clear the association between the branch ID of the branch and its tags.

When the I/O monitor/analyzer350receives a notification, indicating that a read operation is operated on a variable “p” in a non-conditional decision statement, sent by the test platform360based on the watch-point set on the variable p, if the value of the variable “p” is read into for example a variable “q”, the I/O monitor/analyzer350adds the variable “q” into the watch-point variable list, and assigns the variable “q” with the same tag as that of the variable “p”. In addition, it is required for the watch-point editor330to send information to cause the watch-point editor330to set a watch-point for the variable “q”.

When the I/O monitor/analyzer350receives a notification, indicating that a write operation is operated on a variable “p”, sent by the test platform360based on the watch-point set on the variable “p”, the tag associated with the variable “p” in the watch-point variable list is cleared. If there are some variables that are writing the variable “p” have associated tags in the watch-point variable list, the tags are assigned to the variable “p” and the watch-point on the variable “p” is kept. In the case that there is a branch identifier (for example, a branch ID) contained in the stack of the thread where the variable “p” is located, a tag associated with the branch ID is also assigned to the variable “p”. If all the variables that are writing the variable “p” do not exist in the watch-point variable list, but a branch identifier is contained in the stack where the variable “p” is located, the tag associated with the branch identifier is assigned to the variable “p” and the watch-point on the variable “p” is kept. If all the variables that are writing the variable “p” do not exist in the watch-point variable list, and no branch identifier is contained in the stack of the thread where the variable “p” is located, the variable “p” has no tag associated with it and the variable “p” can be removed from the watch-point variable list. At that time, the watch-point editor330may be instructed to remove the watch-point set on the variable “p” in the SUT370by sending a watch-point deleting command.

When the I/O monitor/analyzer350receives a notification, indicating that an output message is to be sent through the invoking of an output function, sent by the test platform360based on the watch-point set on the output function, referring to the tag the variable for storing the output message is associated with in the watch-point variable list and/or the tag the branch ID still remained in the stack of the thread where the variable is located is associated with, correlation between the output message and input messages may be determined, and the correlation may be output as the I/O event correlation mapping335to the tester. In the I/O event correlation mapping335, a tag corresponding to the output message can be recorded, and the tester may determine the correlation between the output message and input messages by referring to correspondence between tags and input messages recorded in other logs or tables. Which input messages correspond to the output message can also be directly recorded in the I/O event correlation mapping335. At that time, the I/O monitor/analyzer350outputs the correlation result after converting tags into the corresponding input messages according to the input-message list. Obviously, since input messages have been associated with the output message through tags, those skilled in the art may easily conceive of other manners to output the result, allowing the tester to determine the correlation between input and output messages.

Next, a specific operation flow of the system300will be described in more detail in conjunction withFIG. 4andFIG. 5.

Referring toFIG. 4, at step S410, if the programming language and the operation system of the SUT370are given and remain unchanged, the tester may specify the concerned network I/O types (for example, UDP, TCP, HTTP (Hypertext Transfer Protocol), etc.) and input a test specification to the code pattern generator320.

At step S420, the code pattern generator320produces a code pattern325through retrieving the code pattern library315according to the test specification.

At step S430, the SUT370is loaded to the test platform360. Herein, specifically, it means loading the package or code of the SUT370to the test run time environment of the test platform360.

At step S440, the watch-point editor330obtains the initial watch-points (such as, input and output functions) according to the code pattern325sent by the code pattern generator320. For example, a static rule can be preset, in which network input and output are methods of a specific class, and then those methods of the specific class may function as the initial watch-points. Next, the watch-point editor330sends a watch-point adding command to the test platform360through the adapter340to set watch-points for the input and output functions of the SUT370.

At step S450, the test platform360initiates the running of the SUT370.

At step S460, the I/O monitor/analyzer350successively collects notifications that are triggered by watch-points, updates variables in the watch-point variable list and their associated tags that are corresponding to input messages, and instructs the watch-point editor330to update watch-points. When the correlation between output and input messages is obtained, the I/O monitor/analyzer350outputs the result335to the tester.

Although step S430is executed after steps S410and S420in the method shown inFIG. 4, step S430may also be executed before step S410and/or step S420or between step S410and step S420; the time at which step S430is executed is independent to step S410and step S420, so long as it is executed before step S440.

Referring toFIG. 5, the processes for different notifications received by the I/O monitor/analyzer350will be described.

At step S510, the watch-point editor330obtains the code pattern325of the network I/O.

At step S515, the watch-point editor330adds watch-points for concerned network I/O operations (for example, input and output functions under the UDP protocol).

At step S520, the test platform360initiates the running of the SUT370.

At step S530, the I/O monitor/analyzer350monitors a notification sent from the test platform360based on watch-points.

At step S540, the I/O monitor/analyzer350determines whether the notification indicates the network input, i.e., receiving input message. If it is determined that the notification indicates the network input, the flow proceeds to step S542, and otherwise, the flow proceeds to step S550.

At step S542, the I/O monitor/analyzer350utilizes a message ID (for example, “e”) to record the received input message.

At step S544, the I/O monitor/analyzer350instructs the watch-point editor330to set a watch-point for a variable (for example, “r”) that directly stores the message.

At step S546, if the variable “r” has already been associated with a tag in the watch-point variable list, the tag is deleted and a tag “e” is assigned to the variable “r”.

At step S550, the I/O monitor/analyzer350determines whether the notification indicates a read operation is performed on a watch-point variable (for example, “v”) on which a watch-point has been set. If so, the flow proceeds to step S552; otherwise, the flow proceeds to step S560.

At step S552, the I/O monitor/analyzer350judges whether the read operation occurs in the conditional decision statement. If the read operation occurs in a conditional decision statement, the flow proceeds to step S558; otherwise the flow proceeds to step S554.

At step S554, a variable to which the value read by the read operation is written is obtained (for example, “w”).

At step S556, the I/O monitor/analyzer350instructs the watch-point editor330to set a watch-point for the variable “w”, adds the variable “w” into the watch-point variable list, and assigns the same tag as that of the variable “v” to the variable “w”. If there is a branch ID stored in a stack corresponding to the current thread, a tag associated with the branch ID is also assigned to the variable “w”.

At step S558, the I/O monitor/analyzer350obtains a current thread ID of the read operation, finds out a stack corresponding to the thread, associates a branch ID (for example, “p”) of a conditional branch executed after the conditional decision with the tag of the variable “v”, and pushes the branch ID “p” into the stack corresponding to the thread.

At step S559, the I/O monitor/analyzer350determines the exit point of the conditional branch, and instructs the watch-point editor330to set a watch-point at the exit point. In addition, the I/O monitor/analyzer350may instruct the watch-point editor330to set watch-points for the variables that have no watch-points set thereon among all the variables written in the branch, so that all the variables written in the branch have watch-points set thereon.

At step S560, the I/O monitor/analyzer350determines whether the notification indicates that the flow has reached the exit point of the conditional branch. If so, the flow proceeds to step S562; otherwise, the flow proceeds to step S570.

At step S562, the I/O monitor/analyzer350obtains the ID of a current thread, finds out a stack corresponding to the thread, and pops the top element in the stack.

At step S570, the I/O monitor/analyzer350determines whether the notification indicates a write operation is performed on a watch-point variable (for example, “u”) on which a watch-point has been set. If so, the flow proceeds to step S572; otherwise the flow proceeds to step S580.

At step S572, the I/O monitor/analyzer350clears, if any, all the tags associated with the variable “v” in the watch-point variable list.

At step S574, the I/O monitor/analyzer350determines the source variables of the write operation, that is, the variables which are written to the variable “u”, and utilizes tags of those variables as tags of the variable “u”.

At step S576, the I/O monitor/analyzer350finds out branch IDs kept in the stack of the current thread, and utilizes tags associated with those branch IDs as tags of the variable “u” as well.

At step S578, the I/O monitor/analyzer350determines whether the tag of the variable “u” is empty. If so, the flow proceeds to step S579; otherwise the flow returns to step S530.

At step S579, the I/O monitor/analyzer350instructs the watch-point editor330to delete the watch-point set on the variable “u”. Further, the I/O monitor/analyzer350may delete the variable “u” from the watch-point variable list.

At step S580, the I/O monitor/analyzer350determines whether the notification indicates the network output, i.e., sending an output message. If the notification indicates the network output, the flow proceeds to step S582; otherwise the flow returns to the step S530.

At step S582, the I/O monitor/analyzer350determines that the following two types of input messages have correlation with the output message: all the input messages corresponding to a tag associated with a branch ID still remained in the stack of the current thread, and all the input messages corresponding to a tag directly associated with a variable which delivers output data to the current network output operation.

At step S584, the correlation between the input messages and the output message is recorded and outputted to a tester.

Below, an example of how to set a watch-point and assign a tag will be described with a simplified code.

Referring toFIG. 6, watch-points are preset to receive( ) and send( ) of DatagramSocket. When an input message is received through invoking ch.receive( ) in a receiving thread, a message ID “e1” is assigned to the input message and is recorded in a input-message list. Since the input message is stored in a variable “p”, a watch-point is set on variable “p”, which is added into a watch-point variable list and is associated with tag “e1”. When the variable “p” is added into the buffer, since the space of buffer[buffer.size( )] in the buffer is related to the input message, a watch-point is set to the space, which is added into the watch-point variable list and associated with the tag “e1”.

In a sending thread, when the input message is read from the buffer and assigned to a variable “t”, since the variable “t” is related to the input message after such an operation, a watch-point is set on the variable “t”, and the variable “t” is added into the watch-point variable list and is associated with the tag “e1”.

In this example, the variable “t” is manipulated in a conditional decision statement to control whether to execute the corresponding conditional branch, and thus a thread ID is recorded, and a branch identifier “c1” of the conditional branch (which may also be considered as the identifier “c1” corresponding to the conditional decision statement) is associated with the tag “e1” and is pushed into a stack corresponding to the thread. In addition, a watch-point is set on the exit point of the conditional branch. If sock.send( ) is in the same thread as the conditional decision and within the branch identified by the branch identifier, an output message “msg” is related to the tag “e1”. When it proceeds to the exit point of the conditional branch, “c1” is popped from the stack.

The above method for determining the correlation between input and output messages can be employed in a distributed system, for example, to perform functional tests on various nodes in the IOT. The above method can also be utilized in other networks to perform functional tests on different forwarding devices or subsystems formed by different forwarding devices in those networks. Obviously, those skilled in the art may conceive of other situations of employing the above method, through which correlation can be determined without the need of semantic analysis on input and output messages.

It can be seen from the detailed description of the above method for determining correlation between input and output messages that there is very little human intervention in the process. Expansibility for different programming languages and test platforms is realized through the implementation of the adapter. Further, it is merely required to set initial watch-points according to a code pattern at the beginning, and then automatic watch-point update is performed according to notifications returned from the test platform based on watch-points during the running of the SUT, so that the correlation between input and output messages can be determined without the need of learning about particular details of the source code, leading to simple implementation.

Next, structural block diagrams of devices700and800for determining correlation between input and output messages in an SUT according to an embodiment of the present invention will be described with reference toFIG. 7andFIG. 8, in which the SUT is provided with preset watch-points, and when the SUT is running, the running of the SUT is detected through watch-points in the SUT.

Referring toFIG. 7, the device700comprises a process module740and an correlation determining module750. The process module740can be utilized, when it is determined that a variable is related to an input message according to the set watch-point, to associate the variable with a tag corresponding to the input message and make the variable set with a watch-point. The correlation determining module750can be utilized for determining correlation between the output message and an input message according to a tag associated with the variable for storing the output message.

Reference can be made to the detailed description of the above methods200,400and500, and the system300ofFIG. 3for the above and other operations and/or functions of the process module740and the correlation determining module750, which will not be described in detail to avoid repetition.

According to the device for determining correlation between input and output messages provided in an embodiment of the present invention, through setting watch-points, variables having relation to input messages can be tracked without paying attention to the specific execution flow inside source codes, and through assigning tags, different variables influenced by the same input message can be associated with the same tag. Thus, correlation between input and output messages can be determined simply and conveniently with watch-points and tags, and semantic analysis from the semantic level can be avoided.

Referring toFIG. 8, the process module840and the correlation determining module850contained in the device800are substantially the same as the process module740and the correlation determining module750contained in the device700as shown inFIG. 7.

According to an embodiment of the present invention, the device800may further comprise a first setting module830, which can be utilized for presetting the watch-point on an input function for receiving the input message. In this case, the variable associated with the input message includes the variable for storing the input message, which is the variable detected when it is determined based on the preset watch-point that the input function is invoked to receive the input message.

According to an embodiment of the present invention, the device800may further comprise a second setting module835. The second setting module835can be utilized for presetting the watch-point on an output function for sending the output message. In this case, the variable for storing the output message is the variable detected when it is determined based on the preset watch-point that the output function is invoked to send the output message.

In the case of presetting watch-points for input and output functions, the process module840can be utilized for associating the variable for storing the input message with a tag corresponding to the input message when it is determined that the input function is invoked to receive the input message according to the set watch-point, and set a watch-point for the variable. The correlation determining module850can be further utilized for determining correlation between the output message and an input message according to a tag associated with the variable for storing the output message when it is determined that the output function is invoked to send the output message according to the set watch-point. According to an embodiment of the present invention, the device800may further comprise an acquiring module810, a first determining module820, and a second determining module825. The acquiring module810can be utilized for, before the watch-points for the input function for receiving input messages and the output function for sending output messages are preset by the first setting module830and the second setting module835, acquiring a test specification of the SUT. The first determining module820can determine a code pattern according to the test specification. The second determining module825can determine the input and output functions for which watch-points are required to be set according the code pattern. Thus, the setting module830performs watch-point setting on the input and output functions only after the input and output functions have been determined by the second determining module825.

According to an embodiment of the present invention, the process module840may further comprise a first process unit842, such as a processing segment. In the case that the read operation does not occur in a conditional decision statement the first process unit842can, when it is determined based on the set watch-point that a read operation is performed on the variable corresponding to the set watch-point, set a watch-point for another variable to which the variable corresponding to the set watch-point is written, and associate the other variable with a tag associated with the variable corresponding to the set watch-point.

According to an embodiment of the present invention, the process module840may comprise a second process unit844, such as a processing segment. In the case that the read operation occurs in the conditional decision statement the second process unit844can, when it is determined based on the set watch-point that a read operation is performed on the variable corresponding to the set watch-point, make a variable in an executed conditional branch on which a written operation is performed set with a watch-point, and set a watch-point on the exit point of the conditional branch, push a branch identifier of the conditional branch into a stack corresponding to a thread where the conditional branch is located, and associate the branch identifier with the tag associated with the variable corresponding to the set watch-point.

According to an embodiment of the present invention, when the process module840comprises a second process unit844, the process module840may further comprise a third process unit846, such as a processing segment. The third process unit846can, when it is determined based on the watch-point set on the exit point of the conditional branch that the exit point of the conditional branch is reached, pop the branch identifier from the stack.

According to an embodiment of the present invention, the device800may further comprise a sending module860and a first conversion module870. The first conversion module870can convert a watch-point adding command into a command recognizable by the test platform on which the SUT is loaded, to enable the test platform to add the watch-point to the SUT. The sending module860can send the watch-point adding command to the test platform to set the watch-point. Thus, when it is necessary to set a watch-point for an input function, an output function, a variable, or the exit point of a conditional branch, it can be realized through sending a watch-point adding command by the sending module860.

According to an embodiment of the present invention, the process module840may comprise a fourth process unit848, such as a processing segment. In the case that the variable corresponding to the set watch-point is still related to an input message after the write operation the fourth process unit848can, when it is determined based on the set watch-point that a write operation is performed on the variable corresponding to the set watch-point, keep the watch-point set on the variable corresponding to the set watch-point, associate the variable with a tag corresponding to the related input message, and in the case that there is a branch identifier contained in a stack of a thread where the variable is located, associate the variable with a tag associated with the branch identifier; and in the case that the variable corresponding to the set watch-point is no longer to be related to an input message after the write operation, deleting the watch-point corresponding to the variable. For example, the fourth process unit848may change a tag associated with the variable to a tag associated with a source variable of the write operation, so that the variable can be associated with the tag of the related input message.

According to an embodiment of the present invention, the device800may comprise a second conversion module890. The second conversion module890can convert a watch-point deleting command into a command recognizable by the test platform on which the SUT is loaded, to enable the test platform to delete the watch-point from the SUT. In such a case, the fourth process unit848can send the watch-point deleting command to the test platform to delete the watch-point corresponding to the variable.

According to an embodiment of the present invention, the tag associated with the variable for storing the output message may comprise at least one of a tag directly associated with the variable for storing the output message and a tag associated with the variable for storing the output message through a branch identifier contained in a stack of a thread wherein the variable is located.

Reference can be made to the description of the above methods200,400,500and the system300ofFIG. 3for the above and other operations and/or functions of various modules and units, which will not be described in detail to avoid repetition. For example, the acquiring module810, the first determining module820, and the second determining module825can be realized by the code pattern generator320; the first setting module830, the second setting module835, the process module840, the correlation determining module850, and the sending module860can be realized by the watch-point editor330and the I/O monitor/analyzer350; and the first conversion module870and the second conversion module890can be realized by the adapter340. In addition, the process module840may comprise all of the various process units, or may comprise a part of them.

There is very little human intervention in the process of determining correlation between input and output messages with the device800according to embodiments of the present invention. Expansibility for different programming languages and test platforms is realized by utilizing the adapter. Further, it is merely required to set initial watch-points according to a code pattern at the beginning, and then automatic watch-point update is performed according to notifications returned from the test platform based on watch-points during the running of the SUT, so that the correlation between input and output messages can be determined without the need of learning about particular details of the source code, leading to simple implementation.