Software testing using test entity

A mechanism for performing dynamic software testing on a computer system using a test entity. A method of embodiments of the invention includes causing a test execution script to run a test based on a first configuration setting as defined in the test execution script. The test execution script is executed in a script execution environment utilizing a software framework of a computer system. The method further includes modifying, in runtime, the first configuration setting into a second configuration setting, and causing, in runtime, the test execution script to rerun the test based on the second configuration setting.

RELATED APPLICATIONS

The present application is related to co-filed U.S. patent application Ser. No. 12/789,228 entitled “Mechanism for Performing Dynamic Software Testing Based on Test Result Information Retrieved in Runtime Using Test Result Entity” and U.S. patent application Ser. No. 12/789,248 entitled “Mechanism for Performing Dynamic Software Testing Based on Grouping of Tests Using Test List entity”, which are assigned to the assignee of the present application.

TECHNICAL FIELD

The embodiments of the invention relate generally to software testing and, more specifically, relate to performing dynamic software testing using test entity.

BACKGROUND

It is commonly known that administrative tasks include software testing various entities (e.g., software components, network protocols, etc.) using various script executing platforms (e.g., Windows PowerShell® platform) that use test execution scripts (e.g., PowerShell script) and are powered by extensible automated engines, such as PowerShell® invocation engine (e.g., System.Management.Automation.dll), and software frameworks (e.g., .NET framework). However, complications occur when testing is to be performed based on various (software and/or hardware) configuration settings, and the testing process becomes even more convoluted, and even unmanageable, when one or more configuration settings are to be modified. This is because today's testing solutions require testing team members (e.g., software developers) to build a test execution platform or infrastructure with multiple rules/regulations and configuration setting variants to accommodate each test and its corresponding hardware and software configuration settings, separately and independently, which, at best, requires constant supervision and rerunning of failed tests and, at worst, becomes overly complex and unmanageable.

DETAILED DESCRIPTION

Embodiments of the invention provide for performing dynamic software testing using a test entity. A method of embodiments of the invention includes causing a test execution script to run a test based on a first configuration setting as defined in the test execution script. The test execution script is executed in a script execution environment utilizing a software framework of a computer system. The method further includes modifying, in runtime, the first configuration setting into a second configuration setting, and causing, in runtime, the test execution script to rerun the test based on the second configuration setting.

The embodiments of the invention are used to improve software testing by employing a test entity that enables a test execution script to perform dynamic software testing based on modifying hardware and/or configuration settings. In one embodiment, the test entity provides the execution script the necessary ability to run or rerun tests by dynamically adapting to changing configuration settings. This dynamic and smart testing is performed in runtime (e.g., modification to configuration setting is made in runtime, the execution script adapts to any configuration modification in runtime, etc.), eliminating the need for creating complex, costly, and often unmanageable test execution infrastructures to hold multiple rules and configuration variants to accommodate different configuration settings and so each test can run independently.

FIG. 1illustrates host machine100for employing a test entity114for enabling a test execution script at a script execution platform112for performing dynamic software testing according to one embodiment of the invention. Host machine100includes a base hardware platform102that comprises a computing platform, which may be capable, for example, of working with a standard operating system108(e.g., Microsoft® Windows®, Linux®, etc.). Operating system108serves as an interface between hardware or physical resources of the host machine100and a software framework110(e.g., Microsoft .NET® Framework). In some embodiments, base hardware platform102may include a processor104, memory devices106, network devices, drivers, and so on. Host machine100includes a server computer system and/or a client computer system (e.g., a client device).

Software framework110may provide features (e.g., user interface, database connectivity, network communication, etc.) allowing to address common programming problems and a virtual machine to manage the execution of software programs written for the software framework110. Script execution platform or environment112(e.g., Windows® PowerShell®) supports a test execution script (e.g., PowerShell® script) and is built on software framework110. Script execution platform112may include an extensible automation engine that includes a command-line shell and a corresponding test execution scripting language, such as PowerShell script. As aforementioned, script execution platform112is built on software framework110to provide an environment for a user (e.g., software developers/administrators) to perform local and remote administrative tasks (including software testing). Using the PowerShell script as a test execution script, these administrative tasks may be performed by execution of command-lets (“cmdlets”) that are specialized classes (e.g., .NET classes) implementing a particular operation and are combined using scripts, which are composition of cmdlets and imperative logic, standalone applications, or by instantiating regular classes. For brevity and simplicity, PowerShell platform, PowerShell Script, .NET Framework, and Microsoft Windows are used as examples throughout this document.

Script execution platform112provides a hosting mechanism for the test execution script that can be used by the user to produce and run software tests. For example, when testing a software component, the user defines certain properties (e.g., rules, regulations, script behavior, testing time limit, alerts, errors and exceptions, etc.) and software and/or hardware configuration settings relating to the test by inserting such properties and settings in the execution script prior to running the test. The user then initiates the test using a script (or Graphical User Interface (GUI)-based) representation and the execution script runs the test according to the defined properties and configuration settings. In one embodiment, script representation logic116of script execution platform112provides logical script representation of testing or testing information (e.g., test scripts, test results, test history, test statistics, configuration settings and information, etc.) shown as an example with reference toFIG. 3B. The user further uses the logical script representation to perform certain administrative tasks, such as amending test scripts, starting or stopping tests, modifying configuration settings, etc. In one embodiment, some or all of the testing information may be provided using a GUI-based screen or representation. Test entity114may serve as a database interface to access a database coupled to the host machine100for relevant testing information (e.g., test scripts, test results, test history, test statistics (such as pass/fail percentage) based on test history, etc.), and serve as a user interface to, in communication with script representation logic116, provide access to a script representation (as shown inFIG. 3B) or a GUI-like representation.

In one embodiment, test entity114is provided as an extension of script execution platform112to enable the test execution script to run and rerun various software tests based on changing configuration settings (e.g., software and/or hardware configuration settings) in runtime. Test entity114, in one embodiment, includes one or more modules that incorporate test entity invocation logic (also referred to as “test logic” or “test invocation logic”) (FIGS. 2A,2B) to enable the execution script to define itself, adjust itself, and decide its own behavior based on runtime-modification of hardware and/or software configuration settings relating to various tests.

Test entity invocation logic, in one embodiment, includes a single logic having a single configuration variant (as opposed to requiring a separate one for each test and/or hardware/software configuration setting) having the ability to dynamically adjust, in runtime, according to various test properties and the changing configuration settings. Test entity114enables the execution script to make decisions on how a test is to be executed by it so that the testing infrastructure may remain simplified and manageable (e.g., the execution script limits itself in focusing on the available resources to satisfy the current resting requirements). Test entity114includes an additional ability to, automatically, create test replications and run them with changing hardware/software configuration settings, in runtime, to provide the ability to compare results of different tests on multiple hardware/software configuration settings.

Throughout this document, the term “in runtime” at least refers to performing a task (e.g., modifying hardware or configuration settings, enabling an execution script at the script execution platform112to run/rerun a test according to the modified configuration setting, etc.) without pausing or stopping the testing system, such as without having to stop the test execution script from running one or more tests that the execution script is in the process of running or build a new test infrastructure to accommodate a newly modified configuration setting, or the like.

FIGS. 2A and 2Billustrate transaction sequences for performing dynamic testing using test entity114according to one embodiment of the invention.FIGS. 2A and 2Brefer to transaction sequences relating to modifying hardware configuration settings A202and B204and software configuration settings A254, B256and C258, respectively. For example, a user, such as a software developer, wishes to test a new network protocol and its performance on multiple hardware settings A202, B204(e.g., computer systems A and B). Initially, the user defines a set of properties for running test206on hardware setting A202and inserts the necessary language in the test execution script208so that the execution script behaves according to these properties on hardware setting A202. These properties, as aforementioned, may include a set of rules, regulations, performance criteria, and the like, to properly and successfully test, in this case, the network protocol and its performance in relation to various hardware settings A202, B204(as inFIG. 2A) and various software settings (as inFIG. 2B).

Continuing with the network protocol testing example, the user runs test206on hardware setting A202to test, for example, the central processing unit (CPU) usage as it relates to the new network protocol that is being tested for the newest (internal release) version of a product (e.g., software package). In one embodiment, test entity invocation logic210triggers test execution logic220by referencing it to cause test execution script208to run test206on hardware setting A202according to the properties as set forth by the user in the test execution script208. Upon completing test206, the user may view its results using script or GUI-based representation228. Let us suppose, the CPU usage on hardware setting A202is determined to be 18%, but the user is seeking the usage to be under 15% and thus, reruns test206on hardware setting B204. For example, the user moves test206and executes the same test206now on hardware setting B204(having, for example, a CPU speed of 3 GHz) which is different from hardware setting A202(having, for example, a CPU speed of 2 GHz). Modifying or using different hardware configuration settings A-B202-204could be of any nature, such as two computer systems having different processors (e.g., using a 3 GHz processor as opposed to using a 2 GHz processor), memory sizes (e.g., using a 256 MB Random Access Memory (RAM) as opposed to using a 512 MB RAM), etc.

Test entity invocation logic210references test execution logic220to (re)run test206on hardware setting B204and although this hardware setting B204is different from the previous hardware setting A202(e.g., two different computer systems with different processor speeds), test206is run using test execution script208without having to require a complicated testing infrastructure. In one embodiment, test entity114is free of hardware (and software) dependencies and does not abstract hardware settings A-B202-204. The abstraction layer for test entity114is achieved by providing a “setup” argument in an execution constructor of the test execution script208. Test entity114may have a number of execution overloads, such as the ones provided as examples in box388ofFIG. 3B. The setup parameter of test entity114is an execution parameter that represents different physical devices (which may have different hardware profiles) on which test execution script208is executed. Further, test entity114is not an execution script; rather, it references the test execution script208that acts as a black-box so none of the script changes influence test entity114. Test entity invocation logic210dictates various new (hardware and/or software) configuration settings to the test execution logic220(that are passed to it) through command-line arguments and/or configuration objects.

Now, let us suppose the results of rerunning the test206shows a satisfactory use of 14% CPU on hardware setting B204and, at this point, the user may choose to terminate further testing. The user may continue to rerun the same test206on any number of modified hardware settings using test execution script208, and the test entity114continues to enable the test execution script208, by referencing test execution logic220, to adjust according to the modified hardware settings.

Similarly, the user may choose to test the new network protocol on different software settings A-C254-256. Examples of modifying or using different software settings A-C254-258include changing virtual machine speed, number of virtual machines in iteration, using different operating systems for the same test, modifying software components of the software product that is being tested or of any software application running on host machine (e.g., hardware setting252), etc. In this case, the user creates tests A-C254-256by setting initial properties as they relate to these tests A-C254-256by inserting proper language in test execution script260. The user starts with running test A254(based on software configuration setting A which includes, for example, ten virtual machines) and modifies, in runtime, software setting A to change to software setting B (which includes, for example, twenty virtual machines) and then to software setting C (which includes, for example, thirty virtual machines). Accordingly, in one embodiment, test entity invocation logic210of test entity114dictates to test execution logic220of the changing software configuration settings254-256and enables the test execution script260, by referencing its test execution logic220, to run tests A-C254-258, consecutively or concurrently using, for example, separate threads (e.g., virtual machines) as requested by the user, without having to stop the system or create a complex testing infrastructure. Test entity114further includes interface logic222to provide a user interface to script representation (or GUI-like representation)228or a database interface to database226. Database226may include relevant test information, such as tests scripts of current and previously-run tests, test history, test statistics based on test history, previously-tested hardware and software configuration settings, etc. Test entity114may include other logic or one or more modules224to perform other relevant tasks, such as listing or grouping of tests, accessing and retrieving testing information relating to previously-run tests from database226, retrieve other testing information from database226, which is provided or accessed in response to queries as discussed with reference toFIG. 3A, etc.

Network protocol testing, CPU performance and the number of virtual machines, etc., are merely used here as examples for simplicity and ease of understanding. Software testing involving test entity114includes software testing of any type, such as ranging from testing modules and components of a software application to testing complex and large-scale benchmarks, network scalability, testing internal releases of products (e.g., software package), antivirus scan loading, time drifting, user performance-related components, etc. Further, these tests can be performed for any number of reasons, such as investigating performance issues, performing load/stress testing, investigating regression issues relating to different product versions, etc. As an example, these testing techniques can be used to test a number of aspects of Red Hat Enterprise Virtualization (RHEV) products.

FIG. 2Cillustrates a system280employing a test entity114for enabling a test execution script208at a script execution platform112for performing dynamic software testing according to one embodiment of the invention. This illustration is a simplified block diagram of the system280employing test entity114and other relevant components to perform various tasks as illustrated inFIGS. 2A and 2B. System280(e.g., server machine and/or client machine) runs on a standard operating system (e.g., Linux, Windows, etc.), and includes a script execution platform or environment112(PowerShell platform), built on a software framework (e.g., .NET Framework), supporting a test execution script208(PowerShell script). Script execution platform112includes script representation logic116to provide a script representation284of testing information (e.g., test scripts, test history, test statics, configuration settings, etc.) for the user to access, analyze, and manipulate. Further, GUI-based representation284may be provided. Script or GUI-based representation284is provided via a host machine and/or a client machine.

Test execution script208includes test execution logic220for execution of tests. Test entity114includes test entity invocation logic210that works in communication with test execution logic220to execute tests regardless of changes made to their software and/or hardware configuration settings. Interface logic222provides a user interface to for accessing script representation (or GUI-based representation)284and a database interface for accessing database282that stores testing information. Other logic or modules224are used for performing other tasks, such as access and retrieve testing information from database282, performing listing or grouping of tests, etc.

FIG. 3Aillustrates a method for performing dynamic testing using test entity according to one embodiment of the invention. Method300may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (such as instructions run on a processing device), or a combination thereof, such as firmware or functional circuitry within hardware devices. In one embodiment, method300is performed by Test Entity114ofFIG. 1.

Method300begins at block305with the selection (e.g., by a user) of a test to be run using an execution script at a script execution platform. The test may be selected using a form of test identification (e.g., test identification number) at block310or a test name at block315. At block320, test entity114is triggered and may be used to query data associated with the selected test from a database. The query is run in runtime and may include recalling history of the selected test (such as its previous runs) or, in some cases, other previously-run tests that relate to the selected test. These historical queries include querying data of the last run325of the selected test if the current selection is a rerun of the test, querying whether the selected run was previously run and failed330, querying whether the selected run was previously run and passed335, or querying all instances340of the selected test if it was previously run. At block345, once such historical queries are performed, the results of the queries are obtained and analyzed using a test result entity370.

At block355, a configuration query is made to determine whether any configuration settings (such as hardware or software configuration settings) are to be modified. This determination may be based on predetermined criteria or on the analyses of the query results (block345) performed in runtime. The developer may have predetermined to run/rerun the selected test with any number of modified configuration settings or determined, in runtime, to modify the default configuration settings based on the information obtained from analyzed query results (block345). In either case, configuration settings are modified, in runtime, using configuration object380, and updated at block360, also in runtime, so that the selected test can run/rerun based on the updated configuration settings. The term “in runtime” refers to performing a task (e.g., modifying configuration settings) without pausing or stopping the testing processor or system, such as modifying configuration settings of a test while that test and/or other tests are being run by the execution script (e.g., PowerShell script).

FIG. 3Billustrates a transaction sequence for performing dynamic testing using test entity according to one embodiment of the invention. Block382(paralleling blocks305-320ofFIG. 3A) illustrates selection of a test (e.g., select-test cmdlet) by choosing a form of test identification and/or a test name. Each test (including the selected test and its previous runs, if any) is a fully qualified test that can be configured and executed and is capable of having relevant execution information (e.g., last-run, last-passed, last-failed, Allinstances), in runtime, as illustrated by block384which parallels blocks325-340ofFIG. 3A. Block386, referencing blocks355-360ofFIG. 3A, illustrates modification of test configuration settings, in runtime, including retrieving and modifying a hardware and/or software test configuration setting and saving (e.g., $conf.update( )) the modified test configuration setting for any number of future runs or simply creating it for a single use to be discarded after one run. Block388shows execution of the selected test, in runtime, with different configuration settings (including modified hardware and/or software configuration settings) as paralleling block350ofFIG. 3A.

FIG. 4illustrates a method for performing dynamic testing using test entity according to one embodiment of the invention. Method400may be performed by processing logic that may comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (such as instructions run on a processing device), or a combination thereof, such as firmware or functional circuitry within hardware devices. In one embodiment, method400is performed by Test Entity114ofFIG. 1.

At block410, a test execution script (e.g., PowerShell script) is identified that includes various properties and hardware/software configuration settings for a test. For example, a user may choose to define any number of test properties and configuration settings by inserting the relevant information and language into the test execution script. At block420, the test is run (e.g., as initiated by the user) using a script execution platform (e.g., PowerShell platform) that is based on a software platform (e.g., .NET framework) being powered by an operating system (e.g., Microsoft Windows). In one embodiment, upon user initiation, test entity enables the execution script to run the test according to the predefined test properties and configuration settings. The user may use a graphical user interface (GUI)-like application to initiate the test run. At block430, a determination is made, in runtime, as to whether any modifications are to be made to the current configuration settings. If no modifications are to be made, the test may continue to run until its run is completed at block440.

At block450, modifications are made to the current configuration settings. In one embodiment, modification to configuration settings are made, in runtime, by a user using a script representation (or GUI-representation) provided at the host machine or client machine to make changes to any number of configuration settings, such as changes to software configuration settings (e.g., changing the number of virtual machines, modifying the operating system, etc.), changes to hardware configuration settings (e.g., switching a slower processor for a faster processor, decreasing the size of RAM, etc.). As aforementioned, the decision to modify could be predetermined or made based on query results. In either case, at block460, test entity detects whether the modification was made to software or hardware or both hardware and software configuration settings. If software, at block470, test entity enables the execution script to rerun the test based on the modified software configuration settings. Similarly, if hardware, at block480, or both hardware and software, at block490, test entity enables the execution script to rerun the test based on modified hardware configuration settings or modified hardware and software configuration settings, respectively.

The exemplary computer system500includes a processing device502, a main memory504(e.g., read-only memory (ROM), flash memory, random access memory (RAM), dynamic RAM (DRAM) such as synchronous DRAM (SDRAM), DRAM (RDRAM), etc.), a static memory506(e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory518(e.g., a data storage device in the form of a drive unit, which may include fixed or removable machine-accessible or computer-readable storage medium), which communicate with each other via a bus530.

The computer system500may further include a network interface device508. The computer system500also may include a video display unit510(e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)) connected to the computer system through a graphics port and graphics chipset, an alphanumeric input device512(e.g., a keyboard), a cursor control device514(e.g., a mouse), and a signal generation device516(e.g., a speaker).

The data storage device518may include a machine-accessible storage medium (or a computer-readable storage medium)528on which is stored one or more sets of instructions522(e.g., test entity) embodying any one or more of the methodologies or functions described herein. The test entity may also reside, completely or at least partially, within the main memory504(e.g., test entity (instructions)522) and/or within the processing device502(e.g., test entity (processing logic)526) during execution thereof by the computer system500, the main memory504and the processing device502also constituting machine-readable storage media. Further, for example, the test entity instructions522may be transmitted or received over a network520via the network interface device508.

Test entity modules532, components and other features described herein (for example in relation toFIG. 1) can be implemented as discrete hardware components or integrated in the functionality of hardware components such as ASICS, FPGAs, DSPs or similar devices. In addition, the modules532can be implemented as firmware or functional circuitry within hardware devices. Further, the modules532can be implemented in any combination hardware devices and software components.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise, as apparent from the following discussion, it is appreciated that throughout the description, discussions utilizing terms such as “causing”, “defining”, “detecting”, “running”, “rerunning”, “ending”, “selecting”, “modifying”, “performing”, “analyzing”, “querying”, “updating”, “enabling”, “providing”, “storing”, “saving” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

The present invention also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a machine readable storage medium, such as, but not limited to, any type of disk including floppy disks, optical disks, magnetic-optical disks, ROMs, compact disk ROMs (CD-ROMs), RAMs, erasable programmable ROMs (EPROMs), electrically EPROMs (EEPROMs), magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computer system bus.