Distributed exception handling testing

A distributed testing system for testing exception handling code paths is provided. The system may include multiple workstations configured to distributively test an executable component for exception handling. Each workstation includes a local data structure with data indicating code paths that have been traversed by a test performed by the workstation. The system includes a central data structure that is accessible by the workstations. By synchronizing with the central data structure, the local data structures can include data about code paths that have been traversed by the workstations in the system. Each workstation may use the synchronized, local data structure to determine previously traversed code paths and use this information to configure further tests.

BACKGROUND

One important type of software testing involves triggering exceptions in code under different conditions. An exception handling test is typically designed to be exhaustive so that all significant exceptions of the code are tested. Particularly, exhaustive exception handling testing is typically configured to simulate faults by executing through all possible paths of the code. The computing power and resources required to run through tens of millions of exception handling code paths in a typical server product can be daunting.

SUMMARY

The present example provides a distributed testing system for testing exception handling code paths. The system may include multiple workstations configured to distributively test an executable component for exception handling. Each workstation includes a local data structure with data indicating code paths that have been traversed by a test performed by the workstation. The system includes a central data structure that is accessible by the workstations. By synchronizing with the central data structure, the local data structures can include data about code paths that have been traversed by the other workstations in the system. Each workstation may use the synchronized, local data structure to determine previously traversed code paths and use this information to configure further tests.

DETAILED DESCRIPTION

Although the present examples are described and illustrated herein as being implemented in a distributed exception handling testing system, the system described is provided as an example and not a limitation. As those skilled in the art will appreciate, the present examples are suitable for application in a variety of different types of distributed computing systems.

FIG. 1shows an example software testing system100with distributed exception handling capabilities. Example system100may include workstations102-105and server150. Workstations102-105are configured to provide an operating environment for testing computer-executable components and can be any type of computing devices, such as servers, personal computers, mobile devices, and the like. As shown inFIG. 1, workstations102-105may include testing tools112-115and local call-stack store122-125.

Testing tools112-115are configured to test executable components, such as applications, operating systems, subroutines, functions, scripts, and the like. In particular, testing tools112-115are configured to determine the exceptions that can occur during the execution of an executable component. These exceptions may include any faults, such as program errors, memory faults, machine crashes, bugs, and the like. Testing tools112-115are also configured to perform distributed testing of a particular executable component. For example, testing tools112-115are typically arranged so that each tool is configured to test particular paths of the code associated with the executable component. As the code paths are traversed during a test, testing tools112-115may represent these traversed code paths by hashed stacks and persist the stacks on local call-stack stores122-125.

Testing tools112-115is configured to interact with server150to provide and receive data associated with the testing of executable components. As shown inFIG. 1, server150may include testing data manager151that is configured to manage central call-stack store152. Specifically, testing data manager151is configured to receive from workstations102-105information about the testing done by each of testing tools122-125. The information typically includes code paths that have been traversed by testing tools122-125. In one implementation, the information is provided by synchronizing data in local call-stack stores122-125with data in central call-stack store152in server150. Local call-stack stores122-125and central call-stack store152may be implemented in any type of data structures. For example, each of the call-stack stores can be implemented as a database or a system that can maintain transactional semantics and synchronize the access and update of data. Maintaining the test data, such as traversed code paths, in central call-stack store152and synchronizing the data with local call-stack stores122-125enables the test data to be shared among workstations102-105.

FIG. 2,3and4show an example process200for synchronizing data in local call-stack stores112-115with central call-stack store152. For this example, a particular executable component is being tested by multiple workstations in a distributive manner by testing system100. For illustrative purposes, only local call-stack stores112-113are shown inFIG. 2,3and4as example. InFIG. 2, workstation102performed a test on the executable component where two code paths of the component were traversed. These code paths were recorded as entries A and B in local call-stack store112as hashed stacks. Workstation102then synchronizes local call-stack store112with central call-stack store152. For example, workstation102may be configured to send entries A and B in local call-stack store112to server150, which updates central call-stack store152with the entries. Server150may then send any new entry in central call-stack store152to workstation102. For the example inFIG. 2, central call-stack store152does not include any new entry.

InFIG. 3, workstation103performed a separate test on the same executable component that was tested by workstation102. The test performed by workstation103traversed three code paths of the executable component. These code paths were recorded as entries B, C and D in local call-stack store113. After the test, workstation103synchronizes local call-stack store113with central call-stack store152. For example, workstation103sends entries B, C, and D to server150for synchronizing with data in central call-stack store152. In this example, central call-stack store152already includes entry B but does not include entries C and D. Thus, entries C and D are added to central call-stack store152. Server150sends entry A to local call-stack store152. Workstation103adds entry A to local call-stack store113. For further testing of the executable component, workstation103can skip the code path represented by stack entry A.

InFIG. 4, workstation102performed another test on the executable component and traversed a code path represented by entry E of local call-stack store112. Workstation102then synchronizes local call-stack store112with central call-stack store152. For example, server150may receive entries from local call-stack store112and add entry E to central call-stack store152. Server150then sends entries C and D to workstation102, which adds the entries to local call-stack store112.

The example stack synchronization process200shown inFIG. 2,3and4enables workstations in a software testing system to test an executable component in a distributed manner. Specifically, the example process200enables the workstations to share information through a central call-stack store about which code paths of the executable component have already been traversed in tests. Thus, the workstations can efficiently perform separate tests, without repeatedly testing the same code paths. The example process200for maintaining a synchronized call-stack store also enables the workstations to execute a test past a particular point of failure to exhaustively test the executable component for exception handling. For example, after a workstation has experienced a failure, such as a machine crash, the workstation (as well as other workstations in the testing system) can recognize the last tested code path from the call-stack store and move on to the next code path, without repeatedly encountering the same failure. This allows the workstations to perform an exhaustive exception handling test of the executable component to completion on a single sweep.

Having synchronized call-stack stores further enables a testing system to be scalable and adaptable. In particular, a synchronized call-stack store allows newly added workstations to receive data regarding the current progress of the testing associated with the executable component and to immediately participate in the distributed testing.

FIG. 5shows an example process500for performing tests in a distributed software testing system. Example process500may be implemented by a workstation to synchronize data for exhaustive handling testing with a central server. At block502, code path data is synchronized with data in the server. The code path data may be maintained by the workstation as hashes of call stacks in a local call-stack store. When synchronized, the code path data enables the workstation to recognize code paths that have been traversed by other workstations in the distributed software testing system and by the workstation itself during tests that have already been performed.

At block506, a code path is identified for testing. Typically, the workstation performs testing on code path that has not been identified by the code path data and, thus, has not been previously traversed. The test can be any type of test-like a regular functional test that exercises certain code paths. At block508, testing is performed on the identified code path. At block510, data about the tested code path is persisted in a local data store. The code path data may be stored as an entry in a call-stack store of the workstation as a hashed call-stack.

At decision block512, a determination is made whether the test has been completed. A typical test covers multiple code paths with each code path having multiple points of potential failure (exception), which in turn translates to multiple exception handling code paths. If the test has not been completed, process500returns to block506. If the test has been completed, the process continues at block514where the code path data of the workstation is re-synchronized with a central data store. For example, the workstation may send the code path data in the local data store to the server that manages code path data for all of the workstations in the distributed software testing system. The workstation may then receive from the server code path data that has been submitted by the other workstations. At decision block516, a determination is made whether to perform another test. If so, process500returns to block508. If not, the process exits at block518.

FIG. 6shows an example process600for synchronizing data in a local call-stack store with data in a central call-stack store for a test. Process600may be implemented by a workstation in the distributed software testing system for performing the test on an executable component. The local call-stack store may be maintained by the workstation. The central call-stack store may be maintained by a central server and may include hashed call-stacks provided by other workstations in the distributed software testing system. At block602, data in the local call-stack store and the central call-stack store is synchronized. At block604, a test of the executable component is started. At block606, a call is made to a common subroutine, which is distributively tested by other workstations in the system. At block608, the workstation looks up call-stack associated with the testing of the subroutine.

At decision block610, a determination is made whether the present call-stack is a new one or not. If not, process600moves to decision block620where a determination is made whether the test has been completed. If not, process600returns to block606. If the test has been completed, the process continues at block624where the new stacks associated with the current test are sent to the central call-stack store. Process600then returns to block602.

Returning to decision block610, if the call stack found is a new one, process600goes to block612where the call stack is inserted into the local store. At block614, the call stack is persisted on a device-readable medium, such as a disk. At block616, an exception is thrown, and the executable component being tested is expected to handle the exception appropriately. Process600then returns to block606.

FIG. 7shows an exemplary computer device700for implementing the described systems and methods. In its most basic configuration, computing device700typically includes at least one central processing unit (CPU)705and memory710.

Depending on the exact configuration and type of computing device, memory710may be volatile (such as RAM), non-volatile (such as ROM, flash memory, etc.) or some combination of the two. Additionally, computing device700may also have additional features/functionality. For example, computing device700may include multiple CPU's. The described methods may be executed in any manner by any processing unit in computing device700. For example, the described process may be executed by multiple CPU's in parallel.

Computing device700may also include additional storage (removable and/or non-removable) including, but not limited to, magnetic or optical disks or tape. Such additional storage is illustrated inFIG. 7by storage715. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Memory710and storage715are all examples of computer storage media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can accessed by computing device700. Any such computer storage media may be part of computing device700.

Computing device700may also contain communications device(s)740that allow the device to communicate with other devices. Communications device(s)740is an example of communication media. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. The term computer-readable media as used herein includes both computer storage media and communication media. The described methods may be encoded in any computer-readable media in any form, such as data, computer-executable instructions, and the like.

Computing device700may also have input device(s)735such as keyboard, mouse, pen, voice input device, touch input device, etc. Output device(s)730such as a display, speakers, printer, etc. may also be included. All these devices are well known in the art and need not be discussed at length.