Testing of distributed systems

In an embodiment, a method is provided for tracking a test. In this method, a test session identifier is transmitted to a test system. The test session identifier identifies a particular test session. A test of a component is triggered at the test system, and this test provides test results, which are received from the test system. The test results include the test session identifier, which allows the tests to be associated with the particular test session.

FIELD

The present disclosure relates generally to tests. In an embodiment, the disclosure relates to testing of distributed systems.

BACKGROUND

Many applications are based in a distributed environment where an assignment of tasks to multiple software components (and hardware components) enables, for example, creation of business applications through composition. An example of a composed business application includes business processes and composite applications based on enterprise compound services, process components, and user-interface applications using core services.

Tests may be executed on such a distributed environment to verify that each component works as expected concerning, for example, functional correctness and performance. To test a particular component, most existing test techniques involve the insertion of specialized test code into production code. However, such test techniques can be inefficient because, for example, every new component requires the creation of new test code. Furthermore, it can be extremely difficult to coordinate between all the different test codes within the distributed environment.

DETAILED DESCRIPTION

The embodiments described herein provide techniques for tracking tests of software and hardware components. In general, a central system (or a master system) is provided to communicate with one or more remote test systems in a test environment. This central system is configured to, for example, manage, track, verify, and/or analyze tests executed at the test systems. In a particular embodiment, each test or group of tests may be tracked by the assignment of identifiers to each test or group of tests. As will be explained in more detail below, the central system generates these identifiers and forwards them to the remote test systems for use in tracking the tests.

FIG. 1depicts a block diagram of a distributed test environment100, in accordance with an illustrative embodiment. The distributed test environment100includes multiple test systems110-112that are in communication with a master system120. Generally, a “test system,” such as test system110,111, or112, is an assemblage of one or more components130-135for testing. Each test system110,111, or112(and the master system120) may be hosted on one or more computing devices. For example, each test system110,111, or112may be hosted on a different computing device. Alternatively, the test systems110and111may be hosted on one computing device while the test system112may be hosted on a different computing device. In another example, all the test systems110,111, and112(and even the master system120) may be hosted on a single computing device.

As used herein, a “component,” such as component130,131,132,133,134, or135, refers to a part or element of one or more software application or hardware apparatus. A software component can, for example, be an element of a software system programmed to provide a predefined service or event. Examples of a software component include a function, a subroutine, an application, and an object. As an example, the components130-135can be a number of components in a distributed software system, such as a distributed enterprise application, or services (e.g., an enterprise resource planning software and enterprise performance management software). The components130-135in such a distributed enterprise application may, for example, track context and apply account relationships, data, and/or rule-based constraints in business applications. On the other hand, a hardware component can, for example, be a variety of elements of a hardware system that has the capability to process test session identifiers and test sequence identifiers, which are described in more detail below. Examples of such a hardware component include dedicated circuitry or logic that is permanently configured (e.g., a processor and a field-programmable gate array (FPGA)) or an application-specific integrated circuit (ASIC)) to perform certain operations. It should further be noted that combinations of software and hardware components may also be tested together. For example, components located at different layers (e.g., software layer, component layer, and chip layer) may be tested together. An example of such a test include an integration test of a graphics card, where the test includes calls to the graphics chip and also calls between the components on the graphics chip.

Still referring toFIG. 1, the master system120generally is a console that is in communication with all the test systems110,111, and112. The master system120can manage, track, verify, and/or analyze tests executed at the test systems110-112. For example, the master system120can control the execution of different tests at one or more of the test systems110-112. Furthermore, the master system120can also track and verify every test being executed at the test systems110-112. As explained in more detail below, the master system120may track the tests by, for example, assigning a unique identifier to a single test or a session of tests.

FIG. 2depicts a block diagram of a master test module203, in accordance with an embodiment, included in a computing device200. It should be appreciated that the computing device200may be deployed in the form of, for example, a personal computer, a laptop computer, a server computer, a tablet personal computer, a personal digital assistant, or other computing devices. The computing device200may be included in a test environment. For example, the computing device200may form a part of the master system120depicted inFIG. 1. Referring toFIG. 2, in various embodiments, the computing device200may be used to implement computer programs, logic, applications, methods, processes, or software to manage, track, verify, and/or analyze tests at remote test systems, as described in more detail below.

In the example depicted inFIG. 2, the computing device200executes an operating system202that manages the other software processes and/or services executing on the computing device200. These software processes and/or other services include a master test module203, which may be composed of an identifier distribution module204, a test trigger module206, and a test result collection module208. The identifier distribution module204is configured to distribute or transmit various identifiers used in the tracking of tests. For example, the identifier distribution module204is configured to distribute test session identifiers, which is explained in more detail below, to all the test systems that are in communication with the master test module203. In contrast, the test trigger module206is configured to control the execution of tests. For example, as explained in more detail below, the test trigger module206can trigger specific tests at one or more test systems.

After the tests have been triggered, the test result collection module208collects or receives test results from one or more test systems. The test results may be stored in the master test module203or at another location. In addition to collecting test results, the test result collection module208may also include tools that are used to verify and analyze the test results, the tools of which are explained in more detail below. It should be appreciated that in other embodiments, the master test module203may include fewer, more, or different modules apart from those shown inFIG. 2. For example, in an alternate embodiment, the master test module203may additionally include an analysis module (not shown) that is configured to analyze and/or identify test results associated with a particular test session.

FIG. 3depicts a flow diagram of a general overview of a method300, consistent with an embodiment, for tracking tests executed at one or more test systems. The method300may be implemented by the master test module203and employed in the computing device200depicted inFIG. 2. As depicted inFIG. 3, the master system transmits a test session identifier to a test system at302. As used herein, a “test session identifier” refers to a value (numeric and/or textual) that uniquely identifies a particular test session, which refers to a technical bracket for each run of a set or group of tests. As an example, a test session identifier can be a globally unique identifier, which is a special type of identifier that provides a unique reference number in any context. It should be noted that in other embodiments, the master system may also transmit more than one test session identifier to more than one test system, and such operations are described in more detail below.

With the test session identifier transmitted to the test system, the master system at304thereafter triggers one or more tests of components at the test system. The master system may trigger the tests by, for example, calling an execution of a specific test program or test script at the test system. In a particular example, the master system can call a particular test program by identifying a report name or name of a particular class.

Once the tests of the components are triggered at the test system, the tests may generate or create various test results. A “test result,” as used herein, refers to any suitable output resulting from a test of one or more components. For example, a component itself may generate or provide test results. In another example, a testing program that tests the components may generate the test results. An example of a test result is a response time of a component to a particular input. In another example, a test result may include a data access time of a component. Such response time and data access time may be used, for example, in analyzing the performance of the components. Other examples of test results include data generated by the components, operations resulting from the components, signals generated by the components, and other test results.

Still referring toFIG. 3, the master system receives the test results from the test system at306. The master system may transmit requests for a particular test result to the test system. Alternatively, the test system may be automatically configured to transmit the test results to the master system. In an embodiment, the test results include the test session identifier, which allows the test results to be associated with a particular test session. As a result, for example, test results associated with a particular test session may be identified. That is, the pairing of a test session identifier with associated test results allows, for example, a user or the master system itself to be able to differentiate test results associated with a particular test session from test results of other test sessions. Such a pairing thereby, for example, allows particular test results to be tracked, verified, and/or analyzed on a per test session basis, which may result in fast isolation of defects in one or more components at the test system.

FIG. 4depicts a block diagram of a test environment400, in accordance with one embodiment, illustrating the tracking of tests executed at a test system110. The test environment400includes a master system120in communication with a single test system110. In this embodiment, the test system110includes components130-131, a test controller module402, and a test result collection module403. In software, it should be noted that within the test system110, the components130-131, in one example, are independent of the test controller module402and the test result collection module403. As a result, the components130-131are not customized or particularly programmed to interface with the test controller module402and the test result collection module403. Rather, the components130-131can be the same components130-131that are used in a production system, which may be configured to interface with other dependent components (not shown).

The test controller module402generally is configured for retrieving settings and test context450used in the execution of tests. As used herein, a “test context,” such as test context450, refers to information and/or data defining or specifying a behavior of one or more components130-131during an execution of a test. As an example, the test context450may include configurations and commands for use in testing a component130or131. The test context450, for example, allows the distribution of a single set of test context to different test systems, such as test system110, thereby creating a single test scenario. Furthermore, in another example, the master system120can simultaneously change the test scenario at different test systems by transmitting the test context450to these test systems. It should be noted that different components130-131may recognize or interface with different languages. The statements defined or described in the test context450may be in a specific language usable with a particular component130or131. However, in another embodiment, the statements may be in a generic format, which is independent of the languages used by the components130-131. An example of such a format is Extensible Markup Language (XML), and the following Table A provides an example of the test context450formatted in XML. Other examples of generic formats may include document formats (e.g., text file format (or .txt format)), graphic file formats (e.g., JPEG), video file formats (e.g., MPEG-4), and other formats. The components130-131can be configured to recognize the generic format of the test context450or configured to translate the statements in generic format into a different language.

It should be appreciated that the test environment400can include a variety of different tests executed at the test system110. For example, the test environment400may be configured to test accuracies of mailing addresses stored in a database. In this example, the master system120can simulate an error by loading a particular test context450that specifies to the components130-131to access the database where the addresses are stored, but not to search for the correct address when a request to retrieve a particular user's address is received. That is, the test context450can instruct the components130-131to provide incorrect addresses of users in response to requests for addresses. This test context450forces an error because the components130-131will access the database of addresses, but the addresses retrieved by the components130-131are incorrect or erroneous.

As depicted inFIG. 4, the master system120initially transmits the test context450to the test system110. In this embodiment, the test session identifier may be included in the test context450. However, in another embodiment, the test session identifier may be transmitted separately from the test context450. The test controller module402receives the test context450, which, in this example, includes the test session identifier. In effect, the test controller module402collects the test context450and stores the test context in, for example, a static class. Whenever the components130-131need information about the current test context450, the components130-131may retrieve the test context450from the test controller module402. Alternatively, the test controller module402may automatically transmit the test context450to the components130-131upon receipt of the test context450from the master system120.

After the transmission of the test session identifier, the master system120then triggers tests of the components130-131at the test system110. The execution of such tests based on the test context450results in the creation of test results453by, for example, the components130-131. The test result collection module403is configured to receive or collect test results453created from the execution of the tests. The test result collection module403receives or collects test results453that are created by the components130-131. In the example ofFIG. 4, the components130-131generate the test results453and transmit them to the test result collection module403. After the tests are completed, the test result collection module403transmits the collected test results453to the master system120. Here, the test results453include the test session identifier, and the master system120or a user may use the test session identifier to track the tests by, for example, identifying the received test results453to be associated with a particular test session. It should be appreciated that the test system110may also be able to process or analyze the collected test results453. As an example, the test result collection module403can compress the collected test results453for transmission to the master system120.

FIG. 5depicts a timing diagram illustrating the tracking of tests at a test system, in accordance with an embodiment. In this example, the timing diagram500shows the interactions between a master system120and a test controller module402, a component130, and/or a test result collection module403at a test system. Initially, at550, the master system120transmits to the test controller module402a test context that includes a test session identifier. After transmitting the test context and the test session identifier, the master system120triggers a test at554of at least one component130at the test system.

If information regarding the current test context is needed, the component130can request or retrieve the test context from the test controller module402at556. With the test context, the test of the component130is then executed at560based on the test context, and during the test or after the test is completed, the component130transmits test results at562to the test result collection module403.

As depicted inFIG. 5, in one embodiment, the master system120may transmit a request for the test results at568to the test result collection module403. In another embodiment, the test result collection module403may automatically be configured to transmit the test results to the master system120. In the embodiment where the master system120makes the request, the request may be for test results associated with a particular test session. Here, the request identifies the particular test session with its test session identifier. Upon receipt of the request with the test session identifier, the test result collection module403can identify or locate all the test results that are associated with the test session identifier because each test result is paired with a test session identifier. The test result collection module403then transmits to the master system120only the portion of all the collected test results that is associated with the particular test session (or test session identifier) as requested by the master system120.

FIG. 6depicts a flow diagram of a detailed method600, in accordance with an alternate embodiment, for tracking tests executed at multiple, distributed test systems. The method600in this example may also be implemented by the master test module203and employed in the computing device200depicted inFIG. 2. As depicted inFIG. 6, a master system at602distributes (or transmits) a particular test session identifier to multiple test systems, each of which is hosted on different computing devices. The master system then triggers a test sequence in one of the test systems at604. As used herein, a “test sequence” refers to one or more tests that are executed within a single test session. That is, a test sequence identifies a grouping of tests within a single test session. The tests can be associated with or assigned to a test sequence based on a variety of relationships. It should be appreciated that a test sequence (or a test itself) can trigger other test sequences on the same test system or on other test systems. For example, a test program testing one component may also request a service from another component that is also to be tested. The relationships between the triggered test sequences can be a hierarchical relationship or parent-child relationship where a test sequence that triggers another test sequence is a “parent” test sequence, and the triggered test sequence is a “child” test sequence.

After the master system triggers a test sequence at one of the test systems, the master system may then transmit requests to all the test systems for test results at606. Depending on whether the parent test sequence triggers any child test sequence, the master system at608may receive test results from one or multiple test systems. For example, if the parent test sequence does not trigger any child test sequence, then the master system receives test results from only the one test system that executed the parent test sequence. However, if the parent test sequence does trigger other child test sequences at other test systems, then the master system may receive test results from these other test systems.

In the example described inFIG. 6, it should be noted that a test sequence identifier may also be assigned to each test sequence. As used herein, a “test sequence identifier,” refers to a value (numeric and/or textual) that uniquely identifies a particular test sequence. The use of test sequence identifiers allow more granularity in tracking the tests by further enabling particular test sequences within a test session to be identified and tracked. In an embodiment, a master system may request particular test results associated with a particular test sequence from the test systems by identifying its test session identifier in a request to the test systems. Upon receipt of the request with the test session identifier, a test result collection module can identify or locate all the test results that are associated with the test sequence identifier because each test result may also be paired with a test sequence identifier. In addition to the test sequence identifier, the test systems may further include the test session identifier in the test results transmitted to the master system.

FIG. 7depicts a block diagram of a distributed test environment700, in accordance with another embodiment, illustrating a test sequence at one test system110triggering a child test sequence at another test system111. The test environment700includes a master system120in communication with two test systems110and111. The test system110includes a test controller module402, components130-131, and a test result collection module403. Similarly, the test system111includes a test controller module702, components132-135, and a test result collection module703.

Again, it should be noted that the test environment400can include a variety of different tests executed at the test systems110and111. For example, the test system110can be an order procurement system that is configured to process orders and the test system111can be a supplier system that receives and places the orders. The test environment400may be configured to test accuracies of an order of goods, such as laptop computers. In this example, an order of 10 laptop computers is placed by the order procurement system (or test system110). In this example, the master system120can simulate an error by loading a particular test context450that specifies an order of 10 laptop computers is inputted into or received by the order procurement system. As a result, the order procurement system transmits an order of 10 laptop computers to the supplier system (or test system111). To simulate an error, the test context450also specifies to the components132-135of the supplier system to provide an inaccurate confirmation of the order from the order procurement system. For example, the components132-135can be configured to transmit a confirmation of 8 laptop computers rather than the original order of 10 laptop computers.

In the example ofFIG. 7, the master system120initially distributes test contexts450to both test systems110and111. The test contexts450include multiple test contexts associated with different test sessions, with each test context paired with or assigned a test session identifier. The master system120then triggers a test sequence at test system110, which executes tests of components130-131at the test system110. This trigger may include a test session identifier that identifies a particular test session to be executed. If the components130-131need the test context450′ associated with this particular test session, the components130-131can retrieve from the test controller module402the test context450′ associated with this particular session.

During testing, the initial (or parent) test sequence triggers756a child test sequence of one component132at the test system111. This component132at the test system111may retrieve the particular test context450′ from its test controller module702based on a test session identifier, which may be included in the triggering756by the parent test sequence at test system110.

In this example, both test sequences at test systems110and111may be executed in parallel, and may end at different times. The test results453and753may not be available to the master system120until all the test sequences have been completed. Before the completion of the test sequences, the master system120may query each test system110or111for its state of testing. In one embodiment, each test system110or111may track a state of its testing, which allows the master system120, for example, to monitor the activities of a test run. Examples of states include triggered, initialized, running, ended, cancelled, and other states. It should be noted that the “triggered” state identifies a tracking of dependent test sequences, and may, for example, be used to identify whether test sequences within a test session are still running.

The master system120may periodically transmit requests to the test systems110and111for their state of testing. In turn, the test system110may transmit a response to the master system120with the state of the test. This state allows the master system120to identify, for example, when tests associated with a particular test session and/or test sequence are completed at each test system110or111. When the master system120has identified that all the tests are completed, the master system120may thereafter transmit requests to the test systems110and111for the test results. In response, the test result collection modules403and703transmit their test results453and753, respectively, to the master system120.

FIG. 8depicts a timing diagram800, consistent with an embodiment, illustrating the tracking of tests executed at multiple, distributed test systems. In this example, the timing diagram800shows the interactions between a master system120and test systems110and111. The test system110includes a test controller module402, a component130, and a test result collection module403. Similarly, the test system111includes a test controller module702, a component132, and a test result collection module703.

Initially, the master system120, as depicted at850and854, transmits a test context, which includes a test session identifier, to both test controller modules402and702. After the test context is distributed, the master system120triggers a test at856of the component130at the test system110. In response, the component130retrieves the test context at858from the test controller module402, and a test is then executed at860based on the retrieved test context.

In this example, the initial or parent test sequence triggers at880a child test sequence of the component132at the test system111. At the test system111, the component132retrieves the test context at882from the test controller module702, and another test sequence is executed at886based on the retrieved test context. After the completion of the test, the component132transmits at888the test results to the test result collection module703.

During tests of the component132at the test system111, the testing of the other component130at the other test system110has completed. The component130then transmits at862the test results to the test result collection module403and returns back to the master system120. It should be noted that the test is started by, for example, calling a method of the component130synchronously, thereby returning back to the master system120after the method has come to an end. However, the component132is called asynchronously, for example, by the component130, and therefore returns to nowhere and simply stops after the component132has done its work.

After the completion of all the tests at the test systems110and111, the master system120may transmit requests, as depicted at868and892, to both the test result collection modules403and703for the test results of, for example, a particular test session. Upon receipt of the request, the test result collection modules403and703retrieve the test results associated with the particular test session based on a test session identifier, and then transmit responses with the test results to the master system120.

It should be noted that test session identifiers and/or the test sequence identifiers may be used to identify particular test results that are associated with a particular test session and/or test sequence. A user, for example, may use this information to identify a particular test session or a test sequence within a test session that is working or has failed. Alternatively, the user may use this information to apply, for example, various analyses to test results of particular test sessions or test sequences. The identification of test results associated with particular test sessions and/or test sequences can be made by a user. However, it should be noted that in an embodiment, the master system120may also have some capability to isolate or identify particular test results. For example, still referring toFIG. 8, the master system120may request from the test result collection modules403and703all test results from different test sessions. In turn, the test result collection modules403and703transfer all their test results with their associated test session identifiers and test sequence identifiers. The master system120may store all the test results from test systems110and111in its non-volatile memory.

When a request is subsequently made to the master system120for particular test results associated with a particular test session, the master system120can compare the test session identifier associated with each test result with a particular test session identifier associated with the requested test session. The master system120can then identify the test results that are associated with the requested test session based on a match of their test session identifiers with the particular test session identifier associated with the requested test session.

FIG. 9depicts a block diagram of a machine in the example form of a computing device200within which may be executed a set of instructions for causing the machine to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client machine in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. Embodiments may also, for example, be deployed by Software-as-a-Service (SaaS), Application Service Provider (ASP), or utility computing providers, in addition to being sold or licensed via traditional channels.

The machine is capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The example of the computing device200includes a processor902(e.g., a central processing unit (CPU), a graphics processing unit (GPU) or both), a main memory904(e.g., random access memory (a type of volatile memory)), and static memory906(e.g., static random access memory (a type of volatile memory)), which communicate with each other via bus908. The computing device200may further include video display unit910(e.g., a plasma display, a liquid crystal display (LCD) or a cathode ray tube (CRT)). The computing device200also includes an alphanumeric input device912(e.g., a keyboard), a user interface (UI) navigation device914(e.g., a mouse), a disk drive unit916, a signal generation device918(e.g., a speaker), and a network interface device920.

The disk drive unit916(a type of non-volatile memory storage) includes a machine-readable medium922on which is stored one or more sets of data structures and instructions924(e.g., software) embodying or utilized by any one or more of the methodologies or functions described herein. The data structures and instructions924may also reside, completely or at least partially, within the main memory904and/or within the processor902during execution thereof by computing device200, with the main memory904and processor902also constituting machine-readable, tangible media.

The data structures and instructions924may further be transmitted or received over a computer network950via network interface device920utilizing any one of a number of well-known transfer protocols (e.g., HyperText Transfer Protocol (HTTP)).

Similarly, the methods described herein may be at least partially processor-implemented. For example, at least some of the operations of a method may be performed by one or more processors902or processor-implemented modules. The performance of certain of the operations may be distributed among the one or more processors902, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the processor or processors902may be located in a single location (e.g., within a home environment, an office environment or as a server farm), while in other embodiments the processors902may be distributed across a number of locations.

While the embodiment(s) is (are) described with reference to various implementations and exploitations, it will be understood that these embodiments are illustrative and that the scope of the embodiment(s) is not limited to them. In general, techniques for tracking tests may be implemented with facilities consistent with any hardware system or hardware systems defined herein. Many variations, modifications, additions, and improvements are possible.