Identifying a configuration element value as a potential cause of a testing operation failure

Examples disclosed herein relate to identifying a configuration element value as a potential cause of a testing operation failure. Examples include causing a testing operation to be performed approximately in parallel on each of a plurality of instances of an application executed in respective testing environments, acquiring configuration element values from each of the testing environments, and identifying at least one of the configuration element values as a potential cause of a testing operation failure.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a U.S. National Stage Application of and claims priority to International Patent Application No. PCT/US2013/070320, filed on Nov. 15, 2013, and entitled “IDENTIFYING A CONFIGURATION ELEMENT VALUE AS A POTENTIAL CAUSE OF A TESTING OPERATION FAILURE,” the entire contents of which are hereby incorporated in its entirety.

BACKGROUND

In the process of developing a computer application, tests may be performed on the application to determine whether the application functions as designed. For example, several tests may be performed on the application to determine whether there are any defects in the code implementing the application. In some examples, the application may also be tested for multiple different computing environments to determine if the application performs as desired in each of the environments.

DETAILED DESCRIPTION

When a computer application is tested for multiple computing environments with different configurations, it may be determined that the application behaves differently in the different environments. For example, the application may exhibit an undesired behavior in one environment with one configuration, but not in another environment having a different configuration. However, even when such differences in operation are identified, it may be difficult to determine how the different configurations of the environments relate to the differences in application behavior.

To address these issues, examples described herein may cause a testing operation to be performed approximately in parallel on each of a plurality of instances of an application executed in testing environments having different configurations, and acquire configuration element values from each of the testing environments. Examples described herein may further determine that the testing operation failed at a given one of the testing environments and, in response, identify at least one of the configuration element values for the given testing environment as a potential cause of the busting operation failure based on differences between the configuration element values for the different testing environments.

In this manner, examples described herein may compare configurations of different testing environments during mirror testing of an application on the different testing environments, and may identify, in real-time during the mirror testing, potential cause(s) of a testing operation failure in one of the testing environments based on configuration element(s) differing between the testing environments. Such examples described herein may quickly identify configuration issues that may be detrimental to the desired operation of application being tested. In addition, due to the real-time identification of potentially problematic configuration differences during application mirror testing, examples described herein may identify environment configuration issues for an application without first identifying “known good” configurations in which it has previously been determined that the application behaves as desired. As such, examples described herein may, for example, be used to quickly identify configuration element values that may negatively affect a particular code change in an application being tested.

Referring now to the drawings,FIG. 1is a block diagram of an example computing device100to identify a potential cause of a test operation failure. As used herein, a “computing device” may be a desktop computer, notebook computer, workstation, tablet computer, mobile phone, smart device, server, blade enclosure, or any other processing device or equipment. In the example ofFIG. 1, computing device100includes a processing resource110and a machine-readable storage medium120encoded with instructions122,124,126, and128. In some examples, storage medium120may include additional instructions. In some examples, instructions122,124,126, and128, and any other instructions described herein in relation to storage medium120, may be stored on a machine-readable storage medium remote from but accessible to computing device100and processing resource110.

In examples described herein, a processing resource may include, for example, one processor or multiple processors included in a single computing device or distributed across multiple computing devices. As used herein, a “processor” may be at least one of a central processing unit (CPU), a semiconductor-based microprocessor, a graphics processing unit (GPU), a field-programmable gate array (FPGA) configured to retrieve and execute instructions, other electronic circuitry suitable for the retrieval and execution instructions stored on a machine-readable storage medium, or a combination thereof. Processing resource110may fetch, decode, and execute instructions stored on storage medium120to perform the functionalities described below. In other examples, the functionalities of any of the instructions of storage medium120may be implemented in the form of electronic circuitry, in the form of executable instructions encoded on a machine-readable storage medium, or a combination thereof.

As used herein, a “machine-readable storage medium” may be any electronic, magnetic, optical, or other physical storage apparatus to contain or store information such as executable instructions, data, and the like. For example, any machine-readable storage medium described herein may be any of Random Access Memory (RAM), volatile memory, non-volatile memory, flash memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disc (e.g., a compact disc, a DVD, etc.), and the like, or a combination thereof. Further, any machine-readable storage medium described herein may be non-transitory.

In the example ofFIG. 1, computing device100may be in communication with a plurality of testing environments140, each being capable of executing and performing testing operations on an application. In examples described herein, an “application” (which may be referred to herein as a “computer application”) may be any system, component, program, website, web application, or any other type of software implemented in the form of executable instructions encoded on a machine-readable storage medium. As used herein, an “instance” of an application is a copy of an application that may be executed independently of any other copy of the application. Computing device100may communicate with testing environments140via any suitable computer network. As used herein, a computer network may include, for example, a local area network (LAN), a wireless local area network (WLAN), a virtual private network (VPN), the Internet, or the like, or a combination thereof. In some examples, a computer network may include a telephone network (e.g., a cellular telephone network).

In the example ofFIG. 1, instructions122may cause a testing operation to be performed approximately in parallel on each of a plurality of instances of an application, each instance executed in a respective one of the plurality of testing environments140. For example, instructions122may cause the testing operation to be performed at each of testing environments140by providing, to each of the testing environments140, a communication180instructing testing environments140to perform the testing operation.

In some examples, instructions122may perform a mirror testing process on an application utilizing the plurality of testing environments140. In such examples, the testing operation may be a mirror testing operation of the mirror testing process. In examples described herein, a “mirror testing process” (which may be referred to herein as “mirror testing”) is a process by which testing operations performed on a first instance of an application executed in a first testing environment are automatically performed on one or more second instances of the application each executed in a different testing environment approximately in parallel. In examples described herein, testing operations performed as part of a mirror testing process may be referred to herein as “mirror testing operations.” In examples in which instructions122perform a mirror testing process, a testing operation may be invoked on computing device100(e.g., by user input or an automated script, etc.) and, in response, instructions122may provide communications180to each of testing environments140to perform that testing operation on the respective instances of the application under test (AUT) by each of testing environments140. In some examples, each testing operation of a mirror testing process may be performed in this manner and approximately in parallel at each of testing environments140.

In some examples, the testing environments utilized in a mirror testing process may each have different configurations from one another such that the mirror testing process is able to test the application for a plurality of different computing environments approximately in parallel. In examples described herein, performing a given testing operation “in parallel” or “approximately in parallel” on a plurality of testing environments includes performing the testing operation on each of the testing environments before any subsequent testing operation is performed on any of the testing environments.

Each of testing environments140may implement, embody, simulate, or otherwise embody or model a computing environment capable of executing the application. In the example ofFIG. 1, testing environments140may have different configurations from one another. As used herein, a “configuration” of a computing or testing environment may be defined by a collection of configuration element values describing the computing or testing environment. In examples described herein, a “configuration element” is an aspect of a computing or testing environment that may differ or be configured to differ between computing or testing environments. Example configuration elements may include such environment aspect categories as computing device model type, type of computer network the computing device interacts with (i.e., “network type”), cellular carrier the device interacts with, user interface (UI) language (e.g., English, French, etc.), wireless signal strength of the computing device, and the like.

In examples described herein, a “configuration element value” for a given computer or testing environment may be information indicating a member of a given configuration element category that at least partially defines the computer or testing environment. Possible configuration element values for the computing device model type may include, for example, the respective model types of various smartphones, tablets, or the like. As another example, possible configuration element values for the cellular carrier category may include the various companies offering cellular phone and data services, or the like. The configuration element values for a given testing environment indicate that the testing environment is implemented to have, simulate, or otherwise embody or model a computing environment having a configuration described by those configuration element values. As an example, if the configuration element values for a given testing environment include a first smartphone model, a first cellular carrier, and a first UI language, the configuration element values indicate that the given testing environment is implemented to have, simulate, or otherwise embody or model the computing environment of the first model smartphone, having the first UI language, and operating in connection with (e.g., on the network of) the first cellular carrier.

In examples described herein, computing or testing environments having “different configurations” from one another are computing or testing environments having different configuration element values for the same configuration element, for each of at least one configuration element. In examples described herein, configuration element values for testing environments are different than and separate from any features, configurations, or any other aspects of the application being tested in the testing environments, including its source code or any configuration of the source code or application itself. In examples described herein, respective testing environments have different configuration element value(s), while each testing environment executes and tests equivalent instances of the same application (i.e., based on the same source code, compiled in the same way, etc.).

In the example ofFIG. 1, instructions124may acquire, from each of testing environments140, configuration element values for each of testing environments140. For example, instructions124may acquire, from test environments140, configuration element values182-1-182-N for test environments140, respectively, where “N” is an integer greater than one that is equivalent to the number of testing environments from which configuration element values are acquired. Each of configuration element values182-1-182-N may include one or a plurality of configuration element values for a respective one of test environments140, the configuration element values for each testing environment defining the configuration of that testing environment.

Instructions126may determine that the testing operation faded at a given one of testing environments140. For example, instructions126may determine that the that the testing operation failed at the given testing environment in response to receiving input indicating that the testing operation failed at the given testing environment. For example, a user of computing device100may provide input to computing device100as part of a mirror testing process implemented by instructions122. For example, the user may provide input indicating a testing operation for instructions122to trigger at each of test environments140. In such examples, the user of computing device100may detect that the testing operation failed at the given testing environment, and may provide user input to computing device100indicating that the testing operation failed at the given testing environment. For example, the user of computing device100may provide such input in response to detecting that the testing operation, at the given testing environment, faded to complete, produced an application state different than the state produced at other testing environments, failed to validate when the testing operation is a validation operation, or the like.

In some examples, instructions126may determine that a testing operation succeeded at a given testing environment in response to user input or independent of user input. For example, instructions126may determine that the testing operation succeeded at a given testing environment if instructions126do not receive any indication (e.g., from a testing engine in communication with the testing environment) that an error condition or possible error condition occurred at the given testing environment. In some examples, instructions126may determine that the testing operation succeeded when no such error condition was detected, or in response to user input indicating that the test environment succeeded after receiving an indication of a possible error condition for the given testing environment. In such examples, a user of computing device100may be alerted of the possible error condition at the given testing environment, and may check the given testing environment in response. If the user confirms that no error condition exists, the user may provide input indicating that there is no error condition at the given testing environment (e.g., indicating that the testing operation has succeeded at the given testing environment).

In other examples, instructions126may determine that the testing operation failed at the given testing environment independent of user input. For example, the testing operation may be a validation operation to determine whether each instance of the application at each testing environment has an expected state, output, display (e.g., of content, GUI elements, etc.), or the like. In such examples, instructions126may determine that the validation operation failed without user input. In such examples, instructions126may determine that the operation failed in response to receiving an indication of the failure from a computing device implementing the testing environment, which may detect the failure of the operation.

In response to the determination that the testing operation failed at the given testing environments, instructions128may identify at least one of the configuration element values for the given testing environment as a potential cause of the testing operation failure based on differences between the configuration element values for the different testing environments140. For example, instructions128may identify at least one configuration element value for the given testing environment as a potential cause of the testing operation failure based on the at least one configuration element value being different than a value of the same configuration element for at least one other testing environment where the testing operation succeeded.

In some examples, instructions128may identify a configuration element value for the given testing environment as a potential cause of the testing operation failure based on the configuration element value being different than a value of the same configuration element for each of testing environments140where the testing operation succeeded. In such examples, instructions128may identify, as a potential cause of the testing operation failure, a configuration element value for the given testing environment that is not shared by any other testing environment in which the same testing operation succeeded. In other examples, instructions128may identify, as a potential cause of the testing operation failure, a configuration element value shared by at least one of the testing environments where the testing operation succeeded and not shared by at least one other of the testing environments where the testing operation succeeded. For example, instructions128may utilize a tolerance threshold to determine whether to identify a given configuration element value even if it is shared by at least one testing environment where the testing operation succeeded.

In some examples, instructions128may identify a plurality of the configuration element values for the given testing environment as potential causes of the test operation failure, based on each of the plurality of values being different than respective values of the same configuration elements for at least one other of the plurality of testing environments where the testing operation succeeded. In some examples, instructions128may identify, as potential causes of the testing operation failure, a plurality of configuration element values for the given testing environment that are not shared by any other testing environment in which the same testing operation succeeded. In some examples, instructions128may identify configuration element value(s) for the given testing environment as described below in relation toFIGS. 2A and 2B.

In examples in which instructions128identify a plurality of configuration element values, instructions128may rank the identified configuration element values based on, for each of the identified values, the likelihood of the identified value being a cause of the failure. For example, instructions128may rank identified configuration elements values at least in part based on, for each identified configuration element value, the number (e.g., percentage, ratio, etc.) of testing environments of the total having the identified value in which the testing operation failed. In some examples, instructions128may rank the identified configuration element values based on configuration element priorities (or rules, etc.) indicating which configuration elements or configuration element values have a greater likelihood of being the cause of a testing operation failure. Such configuration element priorities may be predetermined or automatically determined by instructions128. As an example, such configuration element priorities may indicate, for example, that a network type configuration element value (e.g., WIFI, 3G, etc.) for a testing environment may be more likely than values for other types of configuration elements to be the cause of a testing operation failure.

In some examples, instructions128may output a listing of the identified configuration element values, the listing indicating the determined rank of each of the identified configuration element values. For example, instructions128may output the listing as a ranked report of the identified configuration values, which may be stored in a machine-readable storage medium of or accessible to computing device100. In other examples, instructions128may output the listing by displaying the listing on a display device, such as a computing device monitor, screen, or the like. In other examples, instructions128may output the listing in any other suitable manner.

As described above, instructions124may acquire, configuration element values182-1-182-N for test environments140, respectively. In some examples, one or more of the testing environments140may be implemented on computing device(s) separate from computing device100each having a test engine to collect configuration element values for the testing environment(s) implemented on the computing device. In such examples, instructions124may acquire configuration element values for such testing environments from the respective test engines. Such test engines are described in more detail below in relation toFIG. 2A. In some examples, at least one of testing environments140may be implemented on computing device100, and instructions124may acquire the configuration element values for the at least one testing environment implemented on computing device100.

In some examples, instructions124may acquire configuration element values182-1-182-N from respective testing environments140in response to the determination by instructions126that the testing operation failed at least at the given testing environment. In other examples, instructions124may acquire configuration elements182-1-182-N continually during the operation of the testing (e.g., mirror testing) process operated by instructions122.

In some examples, instructions122,124,126, and128may be part of an installation package that, when installed, may be executed by processing resource110to implement the functionalities described herein in relation to instructions122,124,126, and128. In such examples, storage medium120may be a portable medium, such as a CD, DVD, or flash drive, or a memory maintained by a server from which the installation package can be downloaded and installed. In other examples, instructions122,124,126, and128may be part of an application, applications, or component already installed on computing device100including processing resource110. In such examples, the storage medium120may include memory such as a hard drive, solid state drive, or the like. In some examples, functionalities described herein in relation toFIG. 1may be provided in combination with functionalities described herein in relation to any ofFIGS. 2A-4.

FIG. 2Ais a block diagram of an example system220to identify a configuration element value as a potential cause of a test operation failure. In the example ofFIG. 2A, a computing device200comprises system220, which includes engines222,224,226,227, and228. In some examples, system220may include additional engines.

Each of the engines of system220may be any combination of hardware and programming to implement the functionalities of the respective engine. Such combinations of hardware and programming may be implemented in a number of different ways. For example, the programming for the engines may be processor executable instructions stored on a non-transitory machine-readable storage medium and the hardware for the engines may include a processing resource to execute those instructions. In such examples, the machine-readable storage medium may store instructions that, when executed by the processing resource, implement system220. The machine-readable storage medium storing the instructions may be integrated in a computing device including the processing resource to execute the instructions, or the machine-readable storage medium may be separate but accessible to the computing device and the processing resource. The processing resource may comprise one processor or multiple processors included in a single computing device or distributed across multiple computing devices, in other examples, the functionalities of any of the engines may be implemented in the form of electronic circuitry.

In some examples, the instructions can be part of an installation package that, when installed, can be executed by the processing resource to implement system220. In such examples, the machine-readable storage medium may be a portable medium, such as a CD, DVD, or flash drive, or a memory maintained by a server from which the installation package can be downloaded and installed. In other examples, the instructions may be part of an application, applications, or component already installed on a computing device including the processing resource. In such examples, the machine-readable storage medium may include memory such as a hard drive, solid state drive, or the like.

System220of computing device200may be in communication with at least one other computing device. In the example ofFIG. 2A, system220is in communication with at least computing devices230-1,230-2, and230-3. System220may communicate with other computing devices, including computing devices230-1-230-3, via any suitable computer network, as described above in relation toFIG. 1.

In the example ofFIG. 2A, computing devices230-1-230-3may each implement respective testing environments having different configurations, and may include respective test engines225-1-225-3in communication with testing environments240-1-240-3and system220. Test engines225-1-225-3may collect configuration element values from the respective testing environments240-1-240-2and provide the collected values to system220. Each of test engines225-1-225-3may be any combination of hardware and programming to implement the functionalities of the test engine, as described above in relation to the engines of system220.

In the example ofFIG. 2A, computing device230-1implements a testing environment240-1having a plurality of configuration element values, including at least a device model type value242-1(e.g., “model 1”), a cellular carrier value244-1(e.g., “carrier 1”), and UI language value246-1(e.g., “language 1”). Testing environment240-1may have one or a plurality of additional configuration element values. Such configuration element values may include values for any of a plurality of different configuration elements such as, for example, network type, screen resolution, operating system (OS) (e.g., type and version), hardware type (e.g., type of camera, or other hardware), or the like. Some of these configuration element values may be static (i.e., remain unchanged) over the course of the testing of an application in the testing environment. The configuration element values for testing environment240-1may also include values for other configuration elements that may change over the course of a testing (e.g., mirror testing) process such as, for example, global positioning system (GPS) coordinates, wireless signal strength, whether a wireless interface is on or off, various hardware usage metrics, or the like.

In the example ofFIG. 2A, a first instance250-1of an application (e.g., an application under test (AUT)) is executed in testing environment240-1. Test engine225-1of computing device230-1may, in response to instructions from system220, cause testing operations of a testing (e.g., mirror testing) process to be performed on application instance250-1executed in testing environment240-1. As noted above, test engine225-1may collect configuration element values for testing environment240-1and provide the collected values to system220.

The example ofFIG. 2Ais described in relation to a mobile application to be executed in a mobile computing environment (e.g., on a smartphone or the like). In other examples, the application under test may be any other type of application, such as an application for a notebook or desktop computer, a web application, or the like. In such examples, the testing environments may be suitable for such application and have suitable configuration elements and configuration element values. For example, when the application under test is a web application, the configuration element values for the testing environments may include values for configuration elements such as OS type, OS version, database type, browser type, and the like.

Computing device230-2implements a testing environment240-2having a plurality of configuration element values, including at least a device model type value242-2(i.e., “model 2”), a cellular carrier value244-2(i.e., “carrier 2”), and UI language value246-2(i.e., “language 2”). Testing environment240-2may have additional configuration element value(s), such as those described above. A second instance250-2of the application under test is executed in testing environment240-2. Test engine225-2of computing device230-2may, in response to instructions from system220, cause testing operations of a testing (e.g., mirror testing) process to be performed on application instance250-2executed in testing environment240-2. As noted above, test engine225-2may collect configuration element values for testing environment240-2and provide the collected values to system220.

Computing device230-3implements a testing environment240-3having a plurality of configuration element values, including at least a device model type value242-3(i.e., “model 3”), a cellular carrier value244-3(i.e., “carrier 3”), and UI language value246-3(i.e., “language 3”). Testing environment240-2may have additional configuration element value(s), such as those described above. A third instance250-3of the application under test is executed in testing environment240-3. Test engine225-3of computing device230-3may, in response to instructions from system220, cause testing operations of a testing (e.g., mirror testing) process to be performed on application instance250-3executed in testing environment240-3. As noted above, test engine225-3may collect configuration element values for testing environment240-3and provide the collected values to system220. In the example ofFIG. 2A, each of testing environments240-1-240-3have different configuration element values for device model type, cellular carrier, and UI language, as indicated by the different numbers (e.g., 1, 2, 3) included in the examples values (e.g., “model 1”, “model 2,” etc.).

In some examples, computing device200, in addition to including system220to operate the testing (e.g., mirror testing) process, may also implement a testing environment240-4having a plurality of configuration element values, including at least a device model type value242-4(i.e., “model 2”), a cellular carrier value244-4(i.e., “carrier 1”), and UI language value246-4(i.e., “language 1”). Testing environment240-4may have additional configuration element value(s), such as those described above. A fourth instance250-4of the application under test is executed in testing environment240-4, which system220may communicate with directly. In the example ofFIG. 2A, testing environment240-4may share configuration element values with various other testing environments. For example, testing environment240-4may share its device model type value242-4(e.g., “model 2”) with testing environment240-2, and may share its cellular carrier value244-4(e.g., “carrier 1”) and UI language value (i.e., “language 1”) with testing environment240-1.

In the example ofFIG. 2A, operation engine222may cause a mirror testing operation of a mirror testing process to be performed approximately in parallel on each of a plurality of instances250-1-250-4of the application under test. As described above, each of the instances250-1-250-4is executed in a respective one of testing environments240-1-240-4having different configurations from one another. In some examples, engine222may receive a mirror testing operation trigger284and, in response to trigger284, may cause the mirror testing operation to be performed on instances250-1-250-4in testing environments240-1-240-4.

In some examples, as part of a mirror testing process of system220, a user of computing device200may provide input to computing device200invoke a testing operation on application instance250-4executed on computing device200. In such examples, this user input may include trigger284, and in response, engine222may cause the same testing operation (i.e., mirror testing operation) invoked by the input to be performed on application instances250-1-250-3by providing, to each of test engines225-1-225-3, a communication180instructing test engines225-1-225-3to cause the same testing operation to be performed on application instances250-1-250-3. In response to communication180, each of test engines225-1-225-3may cause the mirror testing operation to be performed on each of application instances250-1-250-3. In some examples, each mirror testing operation of a mirror testing process of system220may be performed in this manner by performing a testing operation on each application instances in response to a testing operation triggered by user input. In some examples, the triggering user input may include input invoking the testing operation on one of the application instances (e.g., instance250-4executing on computing device200). In such examples, in response to the user input (e.g., including trigger284), engine222may provide a communication281to testing environment240-4to perform the testing operation on instance250-4.

In the example ofFIG. 2A, acquisition engine224may acquire, from testing environments240-1-240-4, configuration element values282-1-282-4for testing environments240-1-240-4, respectively. Configuration element values282-1-282-4may define the respective configurations of testing environments240-1-240-4. In some examples, for at least one of the testing environments240-1-240-3, engine224may acquire the configuration element values for the testing environment from a respective testing engine of a respective computing device implementing the testing environment. For example, in the example ofFIG. 2A, engine224may acquire configuration element values282-1-282-3, for testing environments240-1-240-3, from respective testing agents225-1-225-3.

In the example ofFIG. 2A, determination engine226may determine that the mirror testing operation failed at least at a given one of the testing environments and succeeded at least at another one of the testing environments. For example, engine226may determine that the mirror testing operation failed at testing environment240-1but succeeded at each of testing environments240-2-240-4. For example, as described above in relation to instructions126ofFIG. 1, engine226may determine that the that the mirror testing operation failed at the given testing environment in response to receiving input (e.g., user input) indicating that the testing operation failed at the given testing environment. In other examples, engine226may determine that the testing operation failed at the given testing environment independent of user input, as described above in relation to instructions126ofFIG. 1. For example, engine226may determine that the mirror testing test operation failed at a given testing environment240-1in response to a first indication that the test operation failed, received by the determination engine from testing engine225-1of computing device230-1. In such examples, engine226may determine that the mirror testing operation succeeded at another testing environment, such as testing environment240-2, in response to another indication that the mirror testing operation succeeded received by determination engine226from testing engine225-2of computing device230-2. In some examples, engine226may determine that the mirror testing operation failed at one testing environment and succeeded at another testing environment when the mirror testing operation is a validation operation, as described above in relation toFIG. 1. In some examples, engine226may determine that a testing operation succeeded at a given testing environment in response to user input or independent of user input, as described above in relation to instructions126ofFIG. 1.

In some examples, acquisition engine224may acquire configuration element values282-1-282-4from respective testing environments240-1-240-4in response to the determination by engine226that the testing operation failed at least at the given testing environment, as described above in relation toFIG. 1, in other examples, engine224may acquire configuration elements282-1-282-4continually during the operation of the testing (e.g., mirror testing) process operated by engine222, as described above in relation toFIG. 1.

In some examples, acquisition engine224may acquire the configuration element values282-1-282-4for testing environments240-1-240-4in response to trigger284for the mirror testing operation and prior to operation engine222causing the test operation to be performed on application instances250-1-250-4. In other examples, each time a change occurs in one of the testing environments240-1-240-4, acquisition engine224may receive a notification of the change.

In the example ofFIG. 2A, compare engine227may, in response to the determination of engine226, compare the configuration element values for the given testing environment to the respective configuration element values for at least one testing environment at which the mirror testing operation succeeded. For example, in response to the determination of engine226, compare engine227may compare the configuration element values282-1for testing environment240-1to the respective configuration element values282-2-282-4for each of testing environments240-2-240-4at which the mirror testing operation succeeded.

Identification engine228may identify, as a potential cause of the mirror testing operation failure, one of the configuration element values282-1of the given testing environment that differs from the respective value of the same configuration element for the second testing environment. In some examples, engine228may identify one of the configuration element values282-1of the given testing environment in any manner described above in relation toFIG. 1. For example, engine228may identify a configuration element value for the given testing environment as a potential cause of the testing operation failure based on the configuration element value being different than a value of the same configuration element for each of the testing environments where the testing operation succeeded. For example, engine228may identify, as a potential cause of the testing operation failure, a configuration element value for the given testing environment that is not shared by any other testing environment in which the same testing operation succeeded.

In some examples, engine228may identify a plurality of the configuration element values282-1for the given testing environment as potential causes of the test operation failure, based on each of the plurality of values being different than respective values of the same configuration elements for at least one other of the plurality of testing environments where the testing operation succeeded. In some examples, engine228may identify, as potential causes of the testing operation failure, a plurality of configuration element values for the given testing environment that are not shared by any other testing environment in which the same testing operation succeeded.

In some examples, engine228may output a listing of the identified configuration element values, as described above in relation toFIG. 1. In some examples, engine228may rank the identified configuration elements, as described above in relation toFIG. 1. In such examples, the output listing may indicate the determined rank of each of the identified configuration element values.

As noted above, engine226may determine that the mirror testing operation failed at the given testing environment in response to receiving input (e.g., user input) indicating that the testing operation failed at the given testing environment. In some examples, the input may be a request286to report a defect in the application based on the mirror testing operation failure. For example, a user of computing device200may input the request286to report a defect in response to detecting the mirror testing operation failure at the given testing environment (e.g., testing environment240-1). In such examples, in response to system220receiving request286, acquisition engine224may acquire the configuration element values282-1-282-4from testing environments240-1-240-4, and compare engine227may compare the acquired configuration element values282-1-282-4to identify configuration element value(s) as potential cause(s) of the failure, as described herein.

After identifying configuration element value(s), engine228may generate a listing of the identified configuration element values, which may indicate determined rankings of the identified values. In some examples, in response to the defect request286, engine228may create a defect report290based on the failure of the mirror testing operation at the given testing environment. In such examples, the defect report290may include the listing generated by engine228. Engine228may provide the defect report290to an application lifecycle management (ALM) system270such that the defect report290is associated with the application under test in the ALM system270.

Although examples are described herein in relation to the example ofFIG. 2Ain which four testing environments implemented on four different computing devices are shown, in other examples, the mirror testing process and identification of configuration element values may be performed with more or fewer testing environments, more or fewer computing devices, or a combination thereof. In some examples, functionalities described herein in relation toFIG. 2Amay be provided in combination with functionalities described herein in relation to any ofFIGS. 1 and 2B-4.

FIG. 2Bis a diagram of an example collection of configuration element values for different testing environments.FIG. 2Bwill be described herein in relation to the example ofFIG. 2A. In the example ofFIG. 2B, the collection of configuration element values is presented in the form of a table295for illustrative purposes, and includes at least some of configuration element values282-1-282-4of testing environments240-1-240-4ofFIG. 2A. In table295, column295A indicates various configuration elements of testing environments240-1-240-4. Column295B includes the configuration element values242-1,244-1, and246-1of testing environment240-1, and column295C includes the configuration element values242-2,244-2, and246-2of testing environment240-2. Column295D includes the configuration element values242-3,244-3, and246-3of testing environment240-3, and column295E includes the configuration element values242-4,244-4, and246-4of testing environment240-4.

As described above in relation toFIG. 2A, engine228may identify, as a potential cause of the testing operation failure, a configuration element value for the given testing environment that is not shared by any other testing environment in which the same testing operation succeeded. As an example, in the example ofFIG. 2A, determination engine226may determine that a mirror testing operation has failed at testing environment240-1, and succeeded at each of testing environments240-2-240-4. In such examples, after acquisition engine224acquires the configuration element values shown in table295, compare engine227may compare the configuration element values of testing environment240-1(illustrated in column295B) to the configuration element values of testing environments240-2-240-4(illustrated in columns295C-295D, respectively).

For example, comparison engine227may perform comparisons such that identification engine228may identify configuration element values as potential causes of the failure. In some examples, engine227may first eliminate any configuration element values shared by all of the testing environments, which in the example ofFIGS. 2A and 2B, is none. In some examples, engine227may then eliminate from consideration configuration element values shared by testing environment(s) that failed the testing operation and testing environment(s) that succeeded. In this manner, engine227may eliminate the “carrier 1” value of column295B, as it is shared by the failing testing environment240-1of column295B and the succeeding testing environment240-4of column295E. Engine227may also eliminate the “language 1” value of column295B, as it is shared by the failing testing environment240-1of column295B and the succeeding testing environment240-4of column295E. As such, the “model 1” configuration element value of295B remains, which is not shared by any of the succeeding testing environments. In such examples, identification engine228may identify the configuration element value “model 1” as a possible cause of the mirror testing operation failure, indicating that the model type of testing environment240-1may be the cause of the failure.

As noted above in relation toFIG. 2A, in some examples, engine228may output a listing of the identified configuration element values. In the example described in relation toFIG. 2B, engine228may output “model 1”. In other examples, engine228may output table295with “model 1” in column295B emphasized to indicate that it is identified as a possible cause of the failure (e.g., as illustrated in bold and surrounded by a bold outline). As noted above in relation toFIG. 1, there may be a tolerance threshold when determining whether to identify configuration element values as potential causes of the failure. In such examples, engine228may identify “carrier 1” and “language 1” as potential causes of the failure (e.g., as illustrated in bold but without bold outline), even though they are included in one failing testing environment. In some examples, functionalities described herein in relation toFIGS. 2A-2Bmay be provided in combination with functionalities described herein in relation to any ofFIGS. 1 and 3-4.

FIG. 3is a flowchart of an example method300for identifying configuration element values that differ between testing environments. Although execution of method300is described below with reference to system220ofFIG. 2Adescribed above, other suitable systems for the execution of method300can be utilized (e.g., computing device100). Additionally, implementation of method300is not limited to such examples.

At305of method300, engine222may cause a mirror testing operation to be performed in parallel on each of a plurality of instances of an application, wherein each instance is executed in a respective one of a plurality of testing environments having different configurations from one another, as described above. At310, engine224may acquire, from each of the testing environments, configuration element values for the testing environment. In such examples, for each testing environment, the configuration element values may define the configuration of the testing environment.

At315, engine226may determine that the mirror testing operation has failed at a first testing environment of the plurality of testing environments and has passed at least one second testing environment of the plurality of testing environments. At320, in response to the determination, engine228may identify, as potential causes of the failure, a plurality of configuration element values for the first testing environment that differ from the respective values of the same configuration element for each of at least one of the second testing environments. At325, engine228may rank the identified configuration element values of the first testing environment based on, for each of the identified values, the likelihood of the identified value being a cause of the failure.

Although the flowchart ofFIG. 3shows a specific order of performance of certain functionalities, method300is not limited to that order. For example, the functionalities shown in succession in the flowchart may be performed in a different order, may be executed concurrently or with partial concurrence, or a combination thereof. In some examples, functionalities described herein in relation toFIG. 3may be provided in combination with functionalities described herein in relation to any ofFIGS. 1-2B and 4.

FIG. 4is a flowchart of an example method400for generating a listing of identified configuration element values indicating a rank of each of the identified values. Although execution of method400is described below with reference to system220ofFIG. 2Adescribed above, other suitable systems for the execution of method400can be utilized (e.g., computing device100). Additionally, implementation of method400is not limited to such examples.

At405of method400, engine222may cause a mirror testing operation to be performed in parallel on each of a plurality of instances of an application, wherein each instance is executed in a respective one of a plurality of testing environments having different configurations from one another, as described above. At410, engine224may acquire, from each of the testing environments, configuration element values for the testing environment. In such examples, for each testing environment, the configuration element values may define the configuration of the testing environment.

At415, engine226may determine that the mirror testing operation has failed at a first testing environment of the plurality of testing environments and has passed at least one second testing environment of the plurality of testing environments. At420, in response to the determination, engine228may identify, as potential causes of the failure, a plurality of configuration element values for the first testing environment that differ from the respective values of the same configuration element for each of at least one of the second testing environments. At425, engine228may rank the identified configuration element values of the first testing environment based on, for each of the identified values, the likelihood of the identified value being a cause of the failure. In some examples, engine228may rank the identified configuration element values based on predetermined configuration element priorities, as described above in relation toFIG. 1.

At430, engine228may generate a listing of the identified configuration element values. In some examples, the listing may indicate the determined rank of each of the identified configuration element values. At435, engine228may create a defect report based on the failure of the mirror testing operation at the first testing environment, as described above in relation toFIG. 2A. In some examples, the defect report may include the generated listing. At440, engine228may provide the defect report290to an ALM system270such that the defect report290is associated with the application under test in the ALM system270.

Although the flowchart ofFIG. 4shows a specific order of performance of certain functionalities, method400is not limited to that order. For example, the functionalities shown in succession in the flowchart may be performed in a different order, may be executed concurrently or with partial concurrence, or a combination thereof. In some examples, functionalities described herein in relation toFIG. 4may be provided in combination with functionalities described herein in relation to any ofFIGS. 1-3.