Patent Description:
The invention further relates to a method of automatic testing of a software application while said software application is running.

To prevent regression, it is custom to automatically test every build or release of a software application. These tests typically comprise unit tests, which test the smallest functional units, and integration tests, which test the software application after integration of the functional units. Unit tests are often run before integration of the functional units, but can also be run after integration, as disclosed in <CIT>.

<CIT> discloses a system for automatic testing of a software application while said software application is running, said system comprising at least one processor configured to obtain a test script specification associated with a software component of said software application, said test script specification comprising a plurality of test steps, a test step of said plurality of test steps comprising a reference to a process definition stored in a catalog, said process definition not being implemented by said software component and to control said software application by simulating user input according to said test steps specified in said test script.

When testing packaged software with integration tests, it is custom to execute a plurality of tests scripts and check the results. These test scripts verify the functioning of the software application, possibly in different configurations, but test the same software application with the same interface and the same data structure each time.

When a software application is built from an application-specific subset of a collection of components, such an approach to integration testing is not possible. Since not every component is present in every software application, the interface and the data structure of different software applications are normally different. As a result, it would not be possible to perform the same integration test on different software applications. A different integration test is therefore needed for each different software application. Manually creating all these different integration tests would be a lot of work and would be prone to risk of errors and failures.

It is a first object of the invention to provide a system, which can automatically test the integration of a subset of a collection of software components in a running software application without a manually created integration test.

It is a second object of the invention to provide a method, which can be used to automatically test the integration of a subset of a collection of software components in a running software application without a manually created integration test.

In a first aspect of the invention, a system for automatic testing of a software application while said software application is running comprises at least one processor configured to obtain a test script specification associated with a software component of said software application, said test script specification comprising a plurality of test steps, a test step of said plurality of test steps comprising a reference to a process definition stored in a catalog, said process definition not being implemented by said software component, determine whether a further software component of said software application implements said process definition, create a new version of said test script specification in which said reference to said process definition has been replaced with a reference to an implementation of said process definition in said further software component if said further software component has been determined to implement said process definition, create a test script based on said new version of said test script specification, and control said software application by simulating user input according to said test steps specified in said test script.

Said software application is typically a web application. Said system may comprise a single device or multiple devices. Said software application may be running on another system while said software application is being controlled by simulating said user input. Alternatively, said software application may be running on said system while said software application is being controlled by simulating said user input. For example, said software application may run in a different virtual machine than software which controls said software application. Said software application may be running in a testing environment. Optionally, the catalog may comprise multiple sub catalogs, e.g. one maintained by a composer of the software application and one maintained by the developer of the platform.

When a composer is able to compose a software application by selecting a subset of components from a collection of software components, some of the selected components are normally interrelated. Rather than let the composer of the software application manually create an integration test, it is more beneficial to let the developer of the software component create one or more test script specifications that also test the interrelation of the software component with other components. These one or more test script specifications may be included in the software component.

When a composer is able to select, from a collection of components, the components that the composer wants to include in the software application, the composer often needs to assume the role of software developer and write glue code to let components interact. The glue code refers to the specific components selected by the composer. With certain (e.g. prescriptive low code) platforms, it is not necessary to write glue code. For example, a software component may refer to data types and process definitions stored in a catalog and the development platform can then search for one or more further software components which produces those data types and implements those process definitions, e.g. in the subset of components selected by the composer of the software application.

This way, robust and versatile software applications may be composed even by people without a software specialization, e.g. business analysts, consultants or key users. In the context of this patent application, any person composing a software application from pre-existing software components is referred to as a software composer. With such a (development) platform, when the selected components are integrated, a reference to a data type, specified in a component that consumes data of this data type, may be complemented with a reference to the (related) further component that produces data of this data type.

Moreover, with such a (development) platform, it becomes possible to let the developer of a software component create a test script specification that comprises a reference to a process definition which is not implemented in the software component itself. The developer of the software component selects this process definition from the catalog but does not know yet whether the composer of the software application will select any further software component that implements this process definition and if so, which one. If the composer of the software application selects a further software component that implements this process definition, the reference to the process definition is replaced, in a new version of the test script specification, during the integration/weaving step, with a reference to an implementation of this process definition in this further software component.

The test script created based on this new version of the test script specification can be used to automatically test the integration of the subset of the collection of software components in the running software application without a specific, manually created integration test. The automatic testing can be performed in an efficient manner, i.e. without wasting processor resources on checks as to whether a certain software component is present before executing a test step related to that component. A user interaction simulation software library may be used to perform the testing. For example, tools like Selenium WebDriver may be used to simulate the user input according to the test steps specified in the test script.

Thus, with this system, an automatic way of testing individually composed software applications is achieved, which is especially suitable for software applications which have been composed using Novulo's software components, plug-ins and datasets. This automatic way of testing may be used to save resources, avoid individually construed test scripts, and eliminate risks of errors and regression issues.

Said at least one processor may be configured to check whether said software application responds to said simulated user input in an unexpected manner and report results of said check. Said results may include a screen shot of the software application's user interface when an error occurred. Alternatively or additionally, said results may include a date/time stamp indicating when an error occurred, to allow the cause of the error to be determined more easily.

Said implementation of said process definition in said further software component may be associated with a field or button, said field or button may be associated with an identifier, and said at least one processor may be configured to find, based on said identifier, an element in a webpage produced by said running software application and control said software application by simulating user input to said field or button based on information associated with said element. By associating fields and/or buttons with (at least locally unique) identifiers and including these identifiers in the web pages produced by the software application, it becomes easy to locate the fields and/or buttons, even when the user interface changes. As a result, the system does not need to know the positions of the fields and/or buttons in the user interface.

Said implementation of said process definition in said further software component may be associated with a user interface plugin and with a test function in said user interface plugin and said at least one processor may be configured to find said test function in a webpage produced by said running software application and call said test function on said webpage. User interface plugins may be used for advanced user interface functionality. It is beneficial not to require a developer of a user interface plugin to define fields and/or buttons in the user interface. To still be able to test the functionality of the user interface plugins, the user interface plugins may implement test functions, which may be then called in the test steps.

Said at least one processor may be configured to search for said further software component in an application specification associated with said software application, said application specification specifying that said software application comprises one or more software components, said one or more software components including said software component. For example, a composer of a software application may first select the components that the composer wants to include in the software application, which will be specified in the application specification, and if the software component has been selected, a check may take place, e.g. as part of the integration process, whether the composer has selected a further software component which implements the process definition.

Said at least one processor may be configured to omit said test step from said new version of said test script specification if no further software component of said software application implements said process definition. For example, said at least one processor may be configured to omit said test step from said new version of said test script specification if no further software component of said software application implements said process definition and said test step has been marked as optional. Said at least one processor may be configured to generate an error message if no further software component of said software application implements said process definition and said test step has been marked as mandatory.

A further test step of said plurality of test steps may comprise a further reference to a process implemented in said software component and both said test script specification and said new version of said test script specification may comprise said further reference. Alternatively, the further test may comprise a further reference to a further process definition stored in said catalog, said further process definition being implemented by said software component, and said at least one processor may be configured to replace said further reference to said further process definition in said test script specification with a reference to an implementation of said further process definition in said software component in said new version of said test script specification. Thus, process definitions may be stored in the catalog for all processes, even for the processes used only by the software component itself.

Said process definition may be implemented by a process which has been associated with a button provided by said further software component or by a process which specifies that data needs to be entered into a field provided by said further software component, for example. The latter process may be implemented solely for the purpose of testing or also as part of a use-case process description included in the further software component. This use-case process definition may be used to create help information for users of the software application. Alternatively, said process definition may be implemented by a further test script specification included in said further software component, for example.

Said at least one processor may be configured to control said software application by simulating user input according to said test steps specified in said test script based on application-specific test data associated with said software application. For example, in one software application, the revenue in the sales ledger may be <NUM> and in another software application, the revenue in the sales ledger may be <NUM>. One of the test steps in the test script may need this data to check whether the sale of a product entered in a previous test step has been booked correctly. Said application-specific test data may comprise a plurality of values. Each of these values may be associated with one of the test steps.

In a second aspect of the invention, a computer-implemented method of automatic testing of a software application while said software application is running comprises obtaining a test script specification associated with a software component of said software application, said test script specification comprising a plurality of test steps, a test step of said plurality of test steps comprising a reference to a process definition stored in a catalog, said process definition not being implemented by said software component, determining whether a further software component of said software application implements said process definition, creating a new version of said test script specification in which said reference to said process definition has been replaced with a reference to an implementation of said process definition in said further software component if said further software component has been determined to implement said process definition, creating a test script based on said new version of said test script specification, and controlling said software application by simulating user input according to said test steps specified in said test. The method may be performed by software running on a programmable device. This software may be provided as a computer program product.

Said method may comprise checking whether said software application responds to said simulated user input in an unexpected manner and reporting results of said check. Said results may include a screen shot of the software application's user interface when an error occurred. Alternatively or additionally, said results may include a date/time stamp indicating when an error occurred, to allow the cause of the error to be determined more easily.

Said implementation of said process definition in said further software component may be associated with a field or button, said field or button may be associated with an identifier, and said method may comprise finding, based on said identifier, an element in a webpage produced by said running software application and controlling said software application by simulating user input to said field or button based on information associated with said element.

Said implementation of said process definition in said further software component may be associated with a user interface plugin and with a test function in said user interface plugin and said method may comprise finding said test function in a webpage produced by said running software application and calling said test function on said webpage.

Said method may comprise searching for said further software component in an application specification associated with said software application, said application specification specifying that said software application comprises one or more software components, said one or more software components including said software component.

Said method may comprise omitting said test step from said new version of said test script specification if no further software component of said software application implements said process definition. For example, said method may comprise omitting said test step from said new version of said test script specification if no further software component of said software application implements said process definition and said test step has been marked as optional. Said method may comprise generating an error message if no further software component of said software application implements said process definition and said test step has been marked as mandatory.

A further test step of said plurality of test steps may comprise a further reference to a process implemented in said software component and both said test script specification and said new version of said test script specification may comprise said further reference. Alternatively, the further test may comprise a further reference to a further process definition stored in said catalog, said further process definition being implemented by said software component, and said method may comprise replacing said further reference to said further process definition in said test script specification with a reference to an implementation of said further process definition in said software component in said new version of said test script specification.

Said process definition may be implemented by a process which has been associated with a button provided by said further software component or by a process which specifies that data needs to be entered into a field provided by said further software component, for example.

Said method may comprise controlling said software application by simulating user input according to said test steps specified in said test script based on application-specific test data associated with said software application.

A non-transitory computer-readable storage medium stores at least a first software code portion, the first software code portion, when executed or processed by a computer, being configured to perform executable operations for automatic testing of a software application while said software application is running.

The executable operations comprise obtaining a test script specification associated with a software component of said software application, said test script specification comprising a plurality of test steps, a test step of said plurality of test steps comprising a reference to a process definition stored in a catalog, said process definition not being implemented by said software component, determining whether a further software component of said software application implements said process definition, creating a new version of said test script specification in which said reference to said process definition has been replaced with a reference to an implementation of said process definition in said further software component if said further software component has been determined to implement said process definition, creating a test script based on said new version of said test script specification, and controlling said software application by simulating user input according to said test steps specified in said test.

Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a "circuit", "module" or "system.

<FIG> illustrates the operation of an example platform for composing a software application from software components. With this example platform, a composer is able to compose a software application by selecting a subset of components from a collection of software components. With this example platform, it is not necessary to write glue code. Instead, a software component refers to data types and process definitions stored in a catalog and the development platform can then search for one or more further software components which produces those data types and implements those process definitions, e.g. in the subset of components selected by the composer of the software application.

Data types may be classified using an ontology. In computer science and information science, an ontology encompasses a representation, formal naming and definition of the categories, properties and relations between the concepts, data and entities that substantiate one, many, or all domains of discourse. More simply, an ontology is a way of showing the properties of a subject area and how they are related, by defining a set of concepts and categories that represent the subject.

The ontology may thus be used as a classification scheme. By classifying a real-world concept based on its properties and relations to other concepts, concepts that may initially appear to be rather different may be mapped onto the same data type. For example, both an insurance and a computer server may be described using the data type 'system' based on shared properties such as 'can be sold', 'has an owner', 'may be mutated after sale'. This way, the size of the catalogue is kept manageable, and reusability of data types is maximized. This, in turn, reduces resources needed for storage, maintenance and updating of data types. A data type typically corresponds to a concept, e.g. a 'person' or a 'currency' which describes the real world.

With this example platform, when the selected components are integrated, a reference to a data type, specified in a component that consumes data of this data type, may be complemented with a reference to the further component that produces data of this data type. This way, robust and versatile software applications may be composed even by people without a software specialization, e.g. business analysts, consultants or key users.

In the example of <FIG>, four software components <NUM>-<NUM> have been developed by software developers and have been included in a component library. In practice, thousands of software components may be included in a component library. Furthermore, different versions of a software component may be included in the component library. In this description, the different versions of a software component are collectively referred to as a component family. In the example of <FIG>, component <NUM> is version <NUM> of a component family A, component <NUM> is version <NUM> of a component family B, component <NUM> is version <NUM> of a component family C, and component <NUM> is version <NUM> of a component family D.

Each component produces one or more data types and/or consumes one or more data types. This is specified by the component developer. The component developer selects from a catalog which data types are produced by his software component and which data types are consumed by his software component. If the component developer wants his software component to produce a new data type, i.e., a data type not yet stored in the catalog, he typically needs to ask the owner of the catalog to add this data type to the catalog. Existing entries in a catalog are typically not modified, but the catalog can be extended with new entries.

In the example of <FIG>, the catalog <NUM> comprises three data types DT1, DT2, and DT3. Each of the data types is associated with an identifier (ID1, ID2, and ID3 respectively in the example of <FIG>). In the example of <FIG>, component <NUM> produces data of data type DT1 and component <NUM> consumes data of data type DT1. For the sake of simplicity, <FIG> does not show which data types are consumed or produced by components <NUM> and <NUM>. The identifier of DT1, i.e. ID1, is stored in the software components <NUM> and <NUM>.

Each component implements one or more processes and typically multiple processes. For each process that should be callable by another component, a process definition is stored in the catalog. In the example of <FIG>, no process definitions are stored in the catalog for processes that do not need to be callable by another component, but it would be possible to store process definitions for these processes as well. In the example of <FIG>, the component <NUM> implements processes P1 and P2 that should be callable by other components and component <NUM> implements processes_P1 and _P2 that do not need to be callable by other components. Component <NUM> calls processes_P1, _P2, P1, and P2.

The component developer selects from a catalog which process definitions are called by his software component and which process definitions are implemented by his software component. If the component developer wants his software component to implement a new process type, he typically needs to ask the owner of the catalog to add this process definition to the catalog. In the example of <FIG>, the developer of component <NUM> does not need to select process definitions for processes _P1 and _P2, as they do not need to be called by other components and are therefore not stored in catalog <NUM>.

For the sake of simplicity, <FIG> does not show which processes are implemented and called by components <NUM> and <NUM> or which processes are called by component <NUM>. In the example of <FIG>, the catalog <NUM> comprises three process definitions: a process definition <NUM> for P1 and process definitions for P2 and P3. Each of the process definitions is associated with an identifier (ID11, ID12, and ID13 respectively in the example of <FIG>).

A composer can select the software components that he wishes to include in his software application from the component library. This results in an application specification. A typical application specification comprises tens to hundreds of software components. In the example of <FIG>, the composer first selects a component <NUM> that consumes data of data type DT1 and then selects a component <NUM> that produces data of this data type DT1. This is reflected in the application specification <NUM>. In general, there may be multiple components in the component library that produce data of the same data type. If a data type is consumed by a component in the application specification, there must be exactly one component that produces that data type in the application specification. There is no limit on the number of components in the application specification that consumes a given data type.

After the composer has selected the desired components, he can instruct the system to weave the components into an integrated software application. In the example of <FIG>, the system creates a software application <NUM> and further creates a configuration <NUM>. In the example of <FIG>, the components <NUM> and <NUM> are weaved together in order to create the software application <NUM>. As part of this process, the reference to the data type DT1 in component <NUM> is replaced by a reference to the implementation of that data type in component <NUM>. Next, code may be generated based on the results of this weaving process. Alternatively, a runnable model may be generated based on the results of this weaving process. This runnable model is run by a model interpreter without generating intermediate code.

Thus, the created software application comprises code or a runnable model. In the created software application, the individual components may no longer be distinguishable for the end-user. Even in this case, it may still be possible to for the composer to distinguish the individual components. For example, the server running the software application may offer a user interface which identifies each software component uniquely.

In the example of <FIG>, the created software application is transferred to a system <NUM> where the software application is deployed. System <NUM> executes the code or interprets the model of the software application. In other examples, the software application may be deployed on the same system where the software application was composed. The software application is typically a web application. The software application may run on a server and may be used using a client device connected to the server.

With, for example, the platform of <FIG>, it becomes possible to automatically generate a test script from the selected/integrated components and to use this test script to automatically test the integration of the subset of the collection of software components in the running software application without a specific, manually created integration test.

<FIG> shows a first embodiment of the system for automatic testing of a software application while the software application is running. In this first embodiment, the system is a computer <NUM>. In the example of <FIG>, the software application is running on a server <NUM>. The computer <NUM> and the server <NUM> are connected via the Internet <NUM>. A composer uses the computer <NUM> and a monitor <NUM> connected to the computer <NUM> to create the software application.

The computer <NUM> comprises a receiver <NUM>, a transmitter <NUM>, a processor <NUM>, and storage means <NUM>. The processor <NUM> is configured to obtain a test script specification associated with a software component of the software application. The test script specification may be included in the software component. The test script specification comprises a plurality of test steps. A test step of the plurality of test steps comprises a reference to a process definition stored in a catalog. The process definition is not implemented by the software component.

The processor <NUM> is further configured to determine whether a further software component of the software application implements the process definition, create a new version of the test script specification in which the reference to the process definition has been replaced with a reference to an implementation of the process definition in the further software component if the further software component has been determined to implement the process definition.

The processor <NUM> is further configured to create a test script based on the new version of the test script specification and control the software application running on server <NUM> by simulating user input according to the test steps specified in the test script.

With platforms such as the one shown in <FIG>, it becomes possible to let the developer of a software component create a test script specification that comprises a reference to a process definition which is not implemented in the software component itself. The developer of the software component selects this process definition from the catalog but does not know yet whether the composer of the software application will select any further software component that implements this process definition and if so, which one.

If the composer of the software application selects a further software component that implements this process definition, the reference to the process definition is replaced, in a new version of the test script specification, during the integration/weaving step, with a reference to an implementation of this process definition in this further software component.

With this system, an automatic way of testing individually composed software applications is achieved, which is especially suitable for software applications which have been composed using Novulo's software components, plug-ins and datasets. This automatic way of testing may be used to save resources, avoid individually construed test scripts, and eliminate risks of errors and regression issues.

A user interaction simulation software library may be used to perform the testing. For example, tools like Selenium WebDriver may be used to simulate the user input according to the test steps specified in the test script.

In addition to the new version of the test script specification, one or more further versions of the test script specification may be generated as part of the process of creating the test script from the test script specification. In addition to being based on the new version of the test script specification, the test script may be based on the configuration <NUM> of <FIG>.

The test script specification is typically a modelled process that comprises other processes and actions which represent the user interaction and checks that should be performed consecutively. The new version of the test script specification may be a representation in code of that process, which may comprise two lists per process: the processes/actions that it comprises and the transitions between these actions. Based on this new version, the user may be asked to provide application-specific test data and the test script may then be created. The test script may comprise generated XML files with the test steps that need to be performed and the corresponding test data.

In the embodiment of <FIG>, the software components are integrated by computer <NUM> and computer <NUM> is also used to control the software application running on server <NUM> by simulating user input according to the test steps specified in the test script.

In the embodiment of <FIG>, the system <NUM> comprises a computer <NUM> and a server <NUM>. The computer <NUM> comprises a receiver <NUM>, a transmitter <NUM>, a processor <NUM>, and storage means <NUM>. The server <NUM> comprises a receiver <NUM>, a transmitter <NUM>, a processor <NUM>, and storage means <NUM>.

The processor <NUM> of the server <NUM> is configured to obtain the test script specification associated with the software component, determine whether a further software component of the software application implements the process definition, create the new version of the test script specification in which the reference to the process definition has been replaced with a reference to an implementation of the process definition in the further software component if the further software component has been determined to implement the process definition, and create the test script based on the new version of the test script specification.

The processor <NUM> of the computer <NUM> is configured to obtain the test script and control the software application running on server <NUM> by simulating user input according to the test steps specified in the test script. The computer <NUM> may obtain the test script from the server <NUM>, for example, after the software application has been transferred to the server <NUM>, and typically after the user has entered application-specific test data. Alternatively, the computer <NUM> may obtain the test script from the server <NUM>, for example.

In an alternative embodiment, the processor <NUM> of the server <NUM> is configured to control the software application running on server <NUM> by simulating user input according to the test steps specified in the test script and the computer <NUM> and the monitor <NUM> are only used as interface to the server <NUM>.

In the embodiment of the computer <NUM> shown in <FIG>, the computer <NUM> shown in <FIG>, and the server <NUM> shown in <FIG>, the device comprises one processor <NUM>, <NUM>, or <NUM>. In an alternative embodiment, the device comprises multiple processors. The processor <NUM>, <NUM>, or <NUM> may be a general-purpose processor, e.g. from Intel or AMD, or an application-specific processor. The processor <NUM>, <NUM>, or <NUM> may run a Windows or Unix-based operating system for example. The storage means <NUM>, <NUM>, or <NUM> may comprise one or more memory units. The storage means <NUM>, <NUM>, or <NUM> may comprise one or more hard disks and/or solid-state memory, for example. The storage means <NUM>, <NUM>, or <NUM> may be used to store an operating system, applications and application data, for example.

The receiver <NUM>, <NUM>, or <NUM> and the transmitter <NUM>, <NUM>, or <NUM> may use one or more wired and/or wireless communication technologies such as Ethernet and/or Wi-Fi (IEEE <NUM>) to communicate with other devices via the Internet <NUM>. In an alternative embodiment, multiple receivers and/or multiple transmitters are used instead of a single receiver and a single transmitter. In the embodiments shown in <FIG>, a separate receiver and a separate transmitter are used. In an alternative embodiment, the receiver and the transmitter are combined into a transceiver. The device may comprise other components typical for a computer/server such as a power connector. The invention may be implemented using a computer program running on one or more processors.

Preferably, the software application does not run on the system that controls the software application by simulating user input according to the test steps specified in the test script, but this is not excluded. In the latter case, the software application preferably runs on a different virtual machine.

A first embodiment of the method of computer-implemented method of automatic testing of a software application while the software application is running is shown in <FIG>. The method may be performed by the computer <NUM> of <FIG> or the computers <NUM> and <NUM> of <FIG>, for example.

A step <NUM> comprises obtaining a test script specification associated with a software component of the software application. The test script specification comprises a plurality of test steps and a test step of this plurality of test steps comprises a reference to a process definition stored in a catalog. This process definition is not implemented by the software component. The test script specification may be included in the software component itself.

A step <NUM> comprises determining whether a further software component of the software application implements the process definition. A step <NUM> comprises creating a new version of the test script specification in which the reference to the process definition has been replaced with a reference to an implementation of the process definition in the further software component if the further software component has been determined (in step <NUM>) to implement the process definition.

A step <NUM> comprises creating a test script based on the new version of the test script specification, which was created in step <NUM>. A step <NUM> comprises controlling the software application by simulating user input according to the test steps specified in the test script, which was created in step <NUM>.

A second embodiment of the method of computer-implemented method of automatic testing of a software application while the software application is running is shown in <FIG>. This second embodiment is an extension of the first embodiment of <FIG>. In the embodiment of <FIG>, step <NUM> of <FIG> is implemented by a step <NUM>.

Step <NUM> comprises searching for the further software component in an application specification associated with the software application. The application specification specifies that the software application comprises one or more software components. The one or more software components include the software component. An example of such an application specification is application specification <NUM> of <FIG>.

The process definition referred to in the test step may be, for example, a use-case process definition or a test process definition. A use-case process definition is referred to in a use case. User instructions for using the software application may be automatically generated from such use cases and thus from the processes implemented in such use cases. A test process definition is used for a process that is defined for the purpose of testing. The catalog may further comprise regular process definitions. A regular process definition is used for a process that can be performed during normal execution of the software application.

An implementation of a process definition may refer to another process definition and/or to an internal process (for which no process definition is stored in the catalog). For example, for use-case and test process definitions, there may be level <NUM> processes that correspond to scenarios, level <NUM> processes that correspond to scenario steps, and level <NUM> processes that correspond to simple actions.

<FIG> shows an example of an implementation <NUM> of a level <NUM> use-case process in component <NUM> of <FIG>. The process implementation _P1 comprises three subprocesses <NUM>, <NUM>, and <NUM>. An implementation of a user-case process may include a choice between multiple subprocesses. In the example of <FIG>, either process <NUM> or process <NUM> is chosen in dependence on the outcome of choice <NUM>.

Subprocess <NUM> is an internal process _P2 implemented in component <NUM>. Subprocesses <NUM> and <NUM> are processes P1 and P2 defined in the catalog and not implemented in component <NUM> but by component <NUM> of <FIG>. The first block <NUM> is a process information block PI that comprises meta information. This implementation <NUM> can beneficially be used to automatically generate user instructions for using the software application, i.e. for using the functionality relating to the component <NUM>.

<FIG> shows an example of an implementation <NUM> of a test process in component <NUM> of <FIG>. The process implementation _T1 comprises two of the three subprocesses <NUM>, <NUM>, and <NUM> of implementation <NUM> of the user-case process: processes <NUM> and <NUM>. A main difference between a test process and a use-case process is that the test process comprises a sequence of processes, i.e. a sequence of test steps, and no choices. For that reason, implementation <NUM> only comprises one of processes <NUM> and <NUM>. Another test process definition may have an implementation that comprises step <NUM> instead of step <NUM>. The test process starts with the same process information block <NUM> as the regular process. After block <NUM>, processes <NUM> and <NUM> are specified in sequence.

In the example of <FIG>, the subprocess <NUM> may be associated with a button provided by the software component <NUM> or may specify, e.g. in an information process block of the subprocess, that data needs to be entered into a field provided by the software component <NUM>. The subprocess <NUM> may be implemented by a test script specification/process included in the further software component <NUM>.

Since test step <NUM> refers to a process implemented in the software component <NUM> itself, both the test script specification obtained in step <NUM> and the new version of the test script specification created in step <NUM> can comprise the same reference to process <NUM> (_P2).

A third embodiment of the method of computer-implemented method of automatic testing of a software application while the software application is running is shown in <FIG>. This second embodiment is an extension of the first embodiment of <FIG> only shows what happens if it is determined in step <NUM> that a further software component of the software application implements the process definition, i.e. that step <NUM> is performed in this case.

<FIG> shows what could happen if it is determined in step <NUM> that no further software component of the software application, e.g. no further component included in the application specification, implements the process definition. In this case, a step <NUM> is performed in the embodiment of <FIG>. Step <NUM> comprises creating a new version of the test script specification in which the test step is omitted. After step <NUM> or step <NUM> has been performed, steps <NUM> and <NUM> are performed as described in relation to <FIG>.

A fourth embodiment of the method of computer-implemented method of automatic testing of a software application while the software application is running is shown in <FIG>. This fourth embodiment is a variation on the third embodiment of <FIG>. If it is determined in step <NUM> that no further software component of the software application implements the process definition, a step <NUM> is performed after step <NUM>.

Step <NUM> comprises determining whether the test step has been marked as mandatory or optional. If the test step has been marked as optional, step <NUM> of <FIG> is performed. If the test step has been marked as mandatory, a step <NUM> is performed. Step <NUM> comprises generating an error message. After getting an error message, the composer of the software application could include a further software component which implements the process definition in the software application and cause the method to restart at step <NUM>.

The implementation of the process definition in the further software component may be associated with a field or button. This association may be specified in a process information block, for example. Similar associations may be specified in a process information block of the software component.

<FIG> shows an example of meta information specified in a process implementation which does not use a user interface plugin. In the example of <FIG>, the process information of a process in the software component specifies that the (use-case or test) process is executed on a page <NUM> titled "Inventory Counts" comprises three UI elements <NUM>, <NUM>, and <NUM>. These UI elements <NUM>, <NUM>, and <NUM> are fields titled "Owner", "Description", and "Warehouse", respectively. Similarly, the process may refer to a process definition which is implemented by the further software component and the process information block of this implementation may comprise a field titled "Journal entry scheme".

Each of the UI elements is associated with an identifier. For example, UI element <NUM> may be associated with the identifier <NUM> and the field titled "Journal entry scheme" may be associated with the identifier <NUM>. As previously described, components may also be associated with identifiers. For example, the software component may be associated with the identifier "556b2105-24e2-<NUM>-84bf-f87d9c5982ab" and the further software component may be associated with the identifier "497f09c1-1faf-459e-bba7-db3ea8c877c4". If a process has been associated with a button (not shown in <FIG>), this button may also be associated with an identifier. An example of an XML test script for the process is listed below:
<?xml version="<NUM>"?>
<testscenario description="Inventory count" startpagekey="556b2105-24e2-<NUM>-84bf-
f87d9c5982ab:<NUM>" record="businessprocessscenarios:load(<NUM>)">
<scenariosteps>
<scenario step sequence="<NUM>" record="businessprocessscenariosteps:load(<NUM>)">
<scenariostepactions>
<scenariostepaction sequence="<NUM>" type="ClickButton"
structureid="556b2105-24e2-<NUM>-84bf-f87d9c5982ab: <NUM>" />
<scenariostepaction sequence="<NUM>" type="SearchLinkInput"
structureid="556b2105-24e2-<NUM>-84bf-f87d9c5982ab:<NUM>">
<value>(//scenariostepactiondata[@structurekey="556b2105-24e2-<NUM>-84bf-
f87d9c5982ab:<NUM>"])[<NUM>]/value</value>
<searchlinkgridinfo>
<findcolumnvalue>(//scenariostepactiondata[@structurekey="556b2105-
24e2-<NUM>-84bf-f87d9c5982ab:<NUM>"])[<NUM>]/searcheolumnvalue</findcolumnvalue>
<findcolumnfield>this$contact_name$$</findcolumnfield>
</searchlinkgridinfo>
</scenariostepaction>
<scenariostepaction sequence="<NUM>" type="TypeInTextField"
structureid="556b2105-24e2-<NUM>-84bf f87d9c5982ab:<NUM>">
<value>(//scenariostepactiondata[@structurekey="556b2105-24e2-<NUM>-84bf-
f87d9c5982ab:<NUM>"])[<NUM>]/value</value>
</scenariostepaction>
<scenariostepaction sequence="<NUM>" type="SearchLinkInput"
structureid="556b2105-24e2-<NUM>-84bf f87d9c5982ab:<NUM>" usedefaultvalue="true"
/>
<scenariostepaction sequence="<NUM>" type="SearchLinkInput"
structureid="497f09c1-1faf-459e-bba7-db3ea8c877c4:<NUM>" usedefaultvalue="true" />
<scenariostepaction sequence="<NUM>" type="ClickButton"
structureid="556b2105-24e2-<NUM>-84bf-f87d9c5982ab: <NUM>" />
</scenariostepactions>
</scenariostep>.

The above steps describe which UI elements needs to be clicked or filled-in consecutively and comprise references to data fields in the datafile (//scenariostepactiondata) from where the data to be filled in can be obtained. An example of such a data file is listed below:
<?xml version="<NUM>"?>
<testscenariodata description="Inventory count">
<metadata>. </metadata>
<scenariostepdatas>
<scenariostepdata>
<scenariostepactiondatas>
<scenariostepactiondata structurekey="556b2105-24e2-<NUM>-84bf-f87d9c5982ab:<NUM>">
<value>Name of company</value>
</scenariostepactiondata>
<scenariostepactiondata structurekey="556b2105-24e2-<NUM>-84bf-f87d9c5982ab:<NUM>">
<value>Automatic Test demo Inventory Count</value>
</scenariostepactiondata>
</scenariostepactiondatas>
</scenariostepdata>.

Step <NUM> of <FIG> and/or <NUM> may comprise finding, based on the identifier, an element in a webpage produced by the running software application and controlling the software application by simulating user input to the field or button based on information associated with the element. A testrunner may process the 'scenariostepaction' elements from the script one by one. For example, for the step with sequence number <NUM> of type 'TypeInTextField', the testrunner may search for the element with id '556b2105-24e2-<NUM>-84bf-f87d9c5982ab: <NUM>'. When a software application is started in test-mode, these IDs are included in the html source code to make them easy to find.

Instead of, or in addition to, specifying own fields and/or buttons, a process may a user interface plug-in to interact with the user. In this case, a test function may be specified in a process information block of subprocess <NUM> or subprocess <NUM> of <FIG>, for example. <FIG> shows an example of meta information specified in a process implementation which uses a user interface plugin.

In the example of <FIG>, the process information specifies that a page <NUM> titled "Relations" comprises a UI element <NUM> which specifies a function titled show_message, which is associated with a button, message content <NUM> for that function, and a test function <NUM> titled "clickShowMessageButton". Step <NUM> of <FIG> and/or <NUM> may comprise finding this test function <NUM>, i.e. the name "clickShowMessageButton", in a webpage produced by the running software application and call the test function <NUM> on the webpage. The test function <NUM> may be a JavaScript function, for example.

Examples of user interface plugins are a calendar and a WYSIWYG text editor. A plug-in generally comprises one or more DLLs and an XML file that describes how the platform should deal with the plug-in. The XML file may also describe how a component developer should configure the plug-in. For example, in a calendar plug-in, a developer may need to configure the data type of appointments and configure which fields of the appointments contain start time and end time. An example of such a step in the scenariofile is listed below:
<scenariostepaction sequence=" <NUM>" type="TestFunction"
structureid="<NUM>-9abc-def0-<NUM>-56789abcdef0: <NUM>">
<testfunction>clickShowMessageButton</testfunction>
<testparameter>add</testparameter>
</scenariostepaction.

In this case, the structureid is optional, as not every plug-in is represented by a UI element that appears on the page. A plug-in normally ensures that the test functions they offer are registered in JavaScript, e.g. as window. <pluginname>. <functionname>. In this case, the testrunner may call:
var result =
window. tests['novulomessageextensions']['clickShowMessageButton']('add');.

Calling a test function typically results in an object with at least an attribute "success" and if the value of this attribute is false, an attribute "error". In this case, {"success": true} or {"success": false, "error": "Button 'add' not found. "} may be returned. Depending on this result, the test may proceed, or a failure of the test may be reported.

A fifth embodiment of the method of computer-implemented method of automatic testing of a software application while the software application is running is shown in <FIG>. This fifth embodiment is an extension of the first embodiment of <FIG>. In the embodiment of <FIG>, steps <NUM>, <NUM>, <NUM>, and <NUM> are performed after step <NUM>. Step <NUM> of <FIG> is implemented by step <NUM>.

Step <NUM> comprises obtaining application-specific test data associated with the software application. The application-specific test data is typically provided by the composer of the software application. Step <NUM> comprises controlling the software application by simulating user input according to the test steps specified in the test script, which was created in step <NUM>, based on application-specific test data obtained in step <NUM>.

Step <NUM> is performed after each test step performed in step <NUM>. Step <NUM> comprises determining whether the software application responds to the simulated user input in an unexpected manner. If so, this response is stored in memory, at least temporarily. Steps <NUM> and <NUM> are repeated until all test steps have been performed.

After all test steps have been performed, step <NUM> is performed. Step <NUM> comprises reporting the results of the check in step <NUM>. Step <NUM> may comprise obtaining stored response information or even a stored report. The results may comprise a screen shot of the software application's user interface when an error occurred and/or a date/time stamp indicating when an error occurred.

<FIG> shows a test script <NUM> being created and used in the platform of <FIG>. The test script <NUM> is created when the components <NUM> and <NUM> specified in the application specification <NUM> are weaved together in order to create the software application <NUM> or at a later time. This test script <NUM> is used by system <NUM> of <FIG> to control the software application <NUM> which runs on system <NUM> and uses configuration <NUM>. The system <NUM> also uses application-specific test data <NUM>, as described in relation to step <NUM> of <FIG>.

For example, after the software application <NUM> has been transferred to the system <NUM>, a user may be able to use the software application <NUM> itself to enter application-specific test data <NUM>. The user may then use an application running on system <NUM> to generate the test script <NUM> and transfer this test script <NUM> to system <NUM>. If the test script <NUM> does not include the application-specific test data <NUM>, the application-specific test data <NUM> may be transferred to system <NUM> separately.

The embodiments of <FIG>, and <FIG> differ from each other in multiple aspects, i.e. multiple steps have been added or replaced. In variations on these embodiments, only a subset of these steps is added or replaced and/or one or more steps is omitted. For example, the embodiment of <FIG> may be combined with the embodiment of <FIG> and/or with the embodiment of <FIG>.

<FIG> depicts a block diagram illustrating an exemplary data processing system that may perform the method as described with reference to <FIG>, and <FIG>.

Examples of input devices may include, but are not limited to, a keyboard, a pointing device such as a mouse, or the like.

In various embodiments, the application <NUM> may be stored in the local memory <NUM>, he one or more bulk storage devices <NUM>, or separate from the local memory and the bulk storage devices.

Claim 1:
A system (<NUM>,<NUM>) for automatic testing of a software application (<NUM>) while said software application (<NUM>) is running, said system (<NUM>,<NUM>) comprising at least one processor (<NUM>,<NUM>,<NUM>) configured to:
- obtain a test script specification (<NUM>) associated with a software component (<NUM>) of said software application (<NUM>), said test script specification comprising a plurality of test steps (<NUM>-<NUM>), a test step (<NUM>) of said plurality of test steps (<NUM>-<NUM>) comprising a reference to a process definition (<NUM>) stored in a catalog (<NUM>), said process definition (<NUM>) not being implemented by said software component (<NUM>),
- determine whether a further software component of said software application (<NUM>) implements said process definition (<NUM>),
- create a new version of said test script specification (<NUM>) in which said reference to said process definition (<NUM>) has been replaced with a reference to an implementation of said process definition (<NUM>) in said further software component (<NUM>) if said further software component (<NUM>) has been determined to implement said process definition (<NUM>),
- create a test script (<NUM>) based on said new version of said test script specification, and
- control said software application (<NUM>) by simulating user input according to said test steps specified in said test script (<NUM>).