Patent Description:
In recent years, BDD has emerged as an agile software development approach for the specification and execution of automated acceptance tests of software programs. BDD was introduced by Dan North in <NUM> to simplify Test-Driven Development (TDD), see for example <NPL>. TDD is a software development methodology which essentially states that for each unit of software, a software developer must define specific test sets for the unit first, then implement the unit and finally verify that the implementation of the unit makes the tests succeed. BDD combines Test-Driven Development (TDD), Object-Oriented Analysis (OOA), Object-Oriented Design (OOD) and Domain-Driven Design (DDD) to provide a unified language and approach for handling such a software development process from requirements analysis to implementation.

BDD is largely facilitated through the use of a simple domain-specific language (DSL) using natural language constructs (e.g., English-like sentences) that can express the behavior and the expected outcomes of the software. This 'ubiquitous language' can be understood and jointly used by quality managers, domain experts, software developers and customers. BDD employs a semi-formal format for behavioral specification of the software, which is borrowed from user story specifications from the field of object-oriented analysis and design.

To this end, each software unit is decomposed into so-called scenarios, each scenario testing one individual aspect of the software. Each scenario is in turn divided into test steps, which describe a desired outcome of the respective aspect of the software starting from given initial conditions and running through predefined events. Each scenario with its test steps is formulated as a natural language script, which can later be translated into executable test scripts in an automated way. The executable test scripts can then be executed as automated tests for testing the software for its correct implementation. The software requirements within the test scripts are usually written in "given-when-then" sentences based on the ubiquitous language of domain-driven design. This is intended to facilitate the transition between the language used to define the domain-driven requirements and the programming language used to implement them.

One test automation framework widely used for automated acceptance tests written in BDD style is called Cucumber, which comprises a plain language parser called Gherkin. The desired behavior of the software is formulated within Gherkin in a declarative way:.

Such descriptive languages are semi-formal with the capital words (GIVEN, WHEN, THEN) serving as pre-designated keywords. Due to the simple grammar and the natural language keywords, the BDD requirements can be understood and manually executed by technical testers. Cucumber runs through these keywords and processes them step by step, thereby mapping every non-capital phrase following these keywords to a parameterized function call. Traditionally, Ruby scripts were used for this purpose within Cucumber, which replace the test steps by automated program calls and thus make the BDD description automatically executable. However, Cucumber now supports a variety of different programming languages through various implementations, including Java and C#.

BDD is easy to understand and straightforward to implemented. However, in large and complex use cases, the approach with its textual and manually created scenarios may lack the manageability to handle a large number of scenarios and complex tests sets while ensuring completeness and consistency. For the development of complex systems, approaches like Model Based Testing (MBT) or Keyword Based Testing (KBT) are often seen as more appropriate. In particular, MBT approaches allow reviewing and verifying the completeness and consistency of even complex test scenarios using a visual representation of the scenarios, e.g. using diagrams in Unified Modelling Language (UML). However, MBT has to be individually embedded into the existing development and test process for each software component.

In document <CIT> a test script to test a web application is generated from a test case and web objects extracted from a web application. A web application testing tool may be invoked to test a functionality of the web application by executing the test script.

Document <CIT> describes a method for improving automated software testing, which includes identifying software elements of a software test specification executed by a hardware device of an IT system. Existing software objects associated with a software module for testing are mapped to the identified software elements and with physical operational values of the software module. The identified software elements of the software test specification and associated software parameters are verified and software values of the identified software elements are extracted.

Natural language processing and context analysis capabilities are used to determine if an object name referred in a test step refers to an existing object.

Document <CIT> describes a method for automatic generation of a test script. The method includes acquiring a plurality of test steps from a database, the plurality of test steps being associated with a test case and including one or more words in natural language. The method also includes identifying, using natural language processing on the plurality of test steps, one or more actions to be performed in a testing process. The method includes generating, based on the identified one or more actions, the test script to perform the plurality of test steps. The method further includes identifying, by performing the natural language processing on the plurality of test steps, an expected test result associated with each test step. The method additionally includes generating, a validation script based on the expected test result associated with each of the plurality of test steps.

Document <CIT> describes a touchless testing platform employed to, for example, create automated testing scripts, sequence test cases, and implement determine defect solutions. In one aspect, a method includes the actions of receiving a log file that includes log records generated from a code base; processing the log file through a pattern mining algorithm to determine a usage pattern; generating a graphical representation based on an analysis of the usage pattern; processing the graphical representation through a machine learning algorithm to select a set of test cases from a plurality of test cases for the code base and to assign a priority value to each of the selected test cases; sequencing the set of test cases based on the priority values; and transmitting the sequenced set of test cases to a test execution engine.

Document <NPL>, proposes an assisted flow for BDD where the user enters into a dialog with the computer which suggests code pieces extracted from the sentences. For this purpose, natural language processing techniques are exploited. This allows for a semi-automatic transformation from acceptance tests to source code stubs and thus provides a first step towards an automatization of BDD.

Against this background, it is an object of the present invention to find solutions with improved convenience and automatization for the verification of complex software packages.

This object is achieved by a method having the features of claim <NUM>, a data processing system having the features of claim <NUM> a computer program product having the features of claim <NUM> and a data storage medium having the features of claim <NUM>.

According to an aspect of the invention, a computer-implemented method for automated verification of a software program in a behavior-driven development environment comprises receiving, with a data processing system, test scenarios, each test scenario defining an expected behavior of the software program in consecutive test steps, which are formulated in a domain-specific language using natural language phrases and which describe a desired outcome of the software program for predefined events based on given initial conditions; importing test step definitions from the behavior-driven development environment; determining for each test step of the test scenarios if the test step matches with one of the test step definitions on basis of the natural language phrases of the test step; assigning all matched test steps to the corresponding test step definitions; applying natural language processing, NLP, on the natural language phrases of any test steps remaining unmatched, wherein the NLP provides a confidence level for each unmatched test step to correspond to one of the test step definitions; assigning any unmatched test step to the corresponding test step definition when the confidence level surpasses a first predefined matching probability; and at least one of: generating graphical test models for the test scenarios on basis of the assigned test step definitions; and generating executable test scripts for the test scenarios on basis of the assigned test step definitions. The method further comprises updating, when the confidence level is above the first predefined matching probability, the respective test step definition on basis of the natural word phrases of the respective test step; and adding, when the confidence level is below a second predefined matching probability, a test step definition to the behavior-driven development environment corresponding to the respective test step.

According to another aspect of the invention, a data processing system comprises a processor configured to perform a method according to the invention.

According to yet another aspect of the invention, a computer program comprises executable program instructions configured to, when executed, perform the method according to the invention.

According to yet another aspect of the invention, a non-transient computer-readable data storage medium comprises executable program instructions configured to, when executed, perform the method according to the invention.

The non-transient computer-readable data storage medium may comprise, or consist of, any type of computer memory, in particular semiconductor memory such as a solid-state memory. The data storage medium may also comprise, or consist of, a CD, a DVD, a Blu-Ray-Disc, an USB memory stick, a memory card (e.g. an SD card) or the like.

One idea of the present invention is to provide the means to utilize the benefits of BDD with its easy to use and natural language based scenarios while maintaining the required manageability for large, complex development projects. To this end, the proposed solution automatically assigns the test steps of each scenario with already existing test step definitions of a test automation framework (from the integrated BDD development environment). If a literal and/or unambiguous matching is not possible, e.g. because the respective scenario was written in a different style and/or uses different wording, then the NLP algorithm is used to find a best match of the respective test step among the existing test step definitions. If the probability of this best match is high enough to provide a correct/likely fit between test step and test step definition, e.g. if it has a matching probability of at least <NUM>% or <NUM>% or more, then the test step is assigned to the respective test step definition.

The present approach allows an efficient mapping of BDD step phrases to a test automation framework of the integrated BDD development environment and supports the structured development of the necessary framework code. Furthermore, it facilitates the automated generation and synchronization of a graphical test model from BDD scenarios so that the advantages both from BDD and MBT methods may be utilized even for large and complex development projects. The graphical test models may be used to visualize, review and modify test cases so that the consistency and completeness of the BDD scenarios can be ensured. For example, missing scenarios may be identified based on a test model review. As another example, similar scenarios may be combined into a single scenario. Furthermore, the ability to use MBT techniques adds an additional abstraction level and supports the change management. The executable test scripts may be automatically generated directly based on the assigned test step definitions and/or after a verification of the scenario(s) on basis of the generated test model.

Advantageous embodiments and improvements of the present invention are found in the subordinate claims.

According to the invention, the method further comprises updating, when the confidence level is above the first predefined matching probability, the respective test step definition on basis of the natural word phrases of the respective test step. Hence, the existing phrase pattern definitions from the BDD test automation framework may be adapted to include alternative and/or modified test step definitions corresponding to the matched test steps. The first predefined matching probability may be set to a high confidence value of <NUM>% or more so that there is a high probability for a match between the test step and the test step definition.

According to the invention, the method further comprises adding, when the confidence level is below a second predefined matching probability, a test step definition to the behavior-driven development environment corresponding to the respective test step. Hence, in case that the confidence level is lower than this reference probability, which may be for example <NUM>% or similar, it is decided that the test step does not match to any existing test step definition. Instead, the test step is used to define a new test step definition, which is then added to the BDD test automation framework and may be used further.

According to an embodiment, a user verification may be requested if the confidence level is below the first predefined matching probability but above a second predefined matching probability. For example, the first predefined matching probability may be set to <NUM>% or <NUM>% and the second predefined matching probability may be set to <NUM>%. If the confidence level is above the first predefined matching probability, then the test step is considered to match the respective test step definition, which may then be updated based on the formulation of the test step. If the confidence level is below the second predefined matching probability, then the test step does not match any of the existing definitions and hence may be used to define a new one. However, in the intermediate range between <NUM>% and <NUM>% (or <NUM>%), the situation may be unclear, i.e. the test step may or may not match one of the existing definitions. In that case a user input may be required to settle the further procedure, that is if a new definition is introduced, if an existing definition is updated or if the scenario is discarded, etc..

According to an embodiment, the method may further comprise feeding the user verification to a machine learning algorithm of the NLP. For example, commonality criteria of the NLP may be adjusted based on the verification results, e.g. reduce the relevance of certain phrases, identify invalid commonalities and/or commonalities not yet identified. Then, the commonality detection accuracy may be compared in future executions to check whether the optimized criteria did improve the accuracy of the NLP. In the long run, this may reduce the effort for manual validation and improve the accuracy of the commonality detection over time. After the required training and optimization of the NLP engine, the algorithm of the invention may detect and match phrases in a fully unattended and automated way.

According to an embodiment, generating the graphical test models may comprise combining similar test scenarios on basis of test steps assigned to the same test step definition.

According to an embodiment, generating the graphical test models may comprise identifying test data within the test scenarios based on the natural language phrases.

According to an embodiment, the graphical test models may comprise unified modeling language diagrams, i.e. UML diagrams.

According to an embodiment, the method may further comprise comparing the graphical test models with the test scenarios to determine if the graphical test models are in compliance with the expected behavior of the software program. Hence, based on the generated graphical test model, missing and/or incorrect scenarios may be identified.

The invention will be explained in greater detail with reference to exemplary embodiments depicted in the drawings as appended.

The accompanying drawings are included to provide a further understanding of the present invention and are incorporated in and constitute a part of this specification. The drawings illustrate the embodiments of the present invention and together with the description serve to explain the principles of the invention. Other embodiments of the present invention and many of the intended advantages of the present invention will be readily appreciated as they become better understood by reference to the following detailed description. In the figures, like reference numerals denote like or functionally like components, unless indicated otherwise.

<FIG> shows a data processing system <NUM> with a processor <NUM> performing a method M according to an embodiment of the invention. Certain aspects of the method M are exemplified in <FIG> and <FIG>.

Besides the processor <NUM>, the data processing system <NUM> may include the usual components like an accessible memory, a storage unit, an input unit, an output unit and so on (not shown). The processing unit <NUM>, as used herein, means any type of computer or computational circuit, such as, but not limited to, a microprocessor unit, a microcontroller, a graphics processing unit, a digital signal processing unit, or any other type of processing circuit.

The method M provides automated verification of a software program in a behavior-driven development environment, e.g. an integrated development environment, which may comprise a BDD test automation framework like Cucumber, SpecFlow, Behave or similar having a library with BDD scripts and phrase pattern/test step definitions.

The method M comprises under M0 receiving, with the data processing system <NUM>, test scenarios <NUM>, e.g. by importing them from the BDD development environment. Each test scenario <NUM> defines an expected behavior of the software program in consecutive test steps <NUM>. The test scenario <NUM> and thus the test steps <NUM> are formulated in a domain-specific language using natural language phrases and describe a desired outcome ("THEN") of the software program for predefined events ("WHEN") based on given initial conditions ("GIVEN"). The test scenarios <NUM> thus represent a specification and/or requirements of the software program in a chronological sequence.

As a simple example, a registering/login software may comprise the following (schematic) scenarios, wherein the keywords GIVEN, WHEN, THEN each define a respective test step <NUM>:.

The method M further comprises under M1 importing test step definitions <NUM> from the behavior-driven development environment. For the example above, such existing test step definitions <NUM> may look like this (formulated in an arbitrary programming language, e.g. Jave or C#):.

Next, the method M comprises under M2 determining for each test step <NUM> of the test scenarios <NUM> if the test step <NUM> matches with one of the test step definitions <NUM> on basis of the natural language phrases of the test step <NUM>. The method M further comprises under M3 assigning all matched test steps <NUM> to the corresponding test step definitions <NUM>.

In the example from above, matched steps may comprise:.

However, a literal one-to-one matching may not be possible for all test steps <NUM>. For example, "WHEN pressing login" is different form "WHEN press login" due to the different usage of the word "press". However, both test steps <NUM> are similar and thus natural language processing (NLP) may be used to recognize these similarities. In a similar vein, "GIVEN a not registered user has entered some credentials" is similar to:
@Given("^a user has entered [* credentials]$") and "WHEN click on Register" is similar to:
@When("^Click on Registration button$").

To identify these similarities, the method M further comprises under M4 applying NLP on the natural language phrases of any test steps <NUM> remaining unmatched. The NLP provides a confidence level for each unmatched test step <NUM> to correspond to one of the test step definitions <NUM>. The method M further comprises under M5 assigning any unmatched test step <NUM> to the corresponding test step definition <NUM> when the confidence level surpasses a first predefined matching probability.

<FIG> shows an example, where the NLP is called under M4 as soon as one of the test steps <NUM> cannot be matched under step M2, M3. The NLP provides a confidence level, which is then compared with two predefined matching probabilities, a high probability of <NUM>% and a low probability of <NUM>%. If the confidence level is above <NUM>%, then the corresponding test step <NUM> is assigned to the respective test step definition <NUM> under M5. The test step definitions <NUM> may be updated by including the assigned test step <NUM> in repository of test step definitions <NUM> within the BDD framework (cf. reference sign T1 in <FIG>). In case that the confidence level is below <NUM>%, a new test step definition <NUM> is added to the BDD test automation framework that corresponds to the not yet existing test step <NUM> (cf. reference sign T2 in <FIG>).

If the confidence level is between <NUM>% and <NUM>%, a user verification is requested (cf. middle quadratic box in <FIG>), which is then used under T3 as input for a machine learning algorithm of the NLP to improve the accuracy of the NLP in future runs. For example, the results of the assignments may be analyzed with respect to commonalities not identified, e.g. "Select" should be equal to "Click", or invalid commonalities, e.g. "Start screen" should be different from "Registration screen". Furthermore, the relevance of certain words or phrases may be reduced, e.g. in case of irrelevant words like "some". By optimizing the NLP engine, manual interventions for the NLP application may be reduced and/or completely avoided in the consecutive runs of the method M.

In the example from above, the assignment of test steps <NUM> and test step definitions <NUM> may look like this after running the NLP:.

Updated test step definitions <NUM> may comprise:.

Next the method M comprises under M6 generating graphical test models <NUM> for the test scenarios <NUM> on basis of the assigned test step definitions <NUM>. The graphical test models <NUM> may be represented, for example, by Unified Modelling Language diagrams. Here, similar scenarios <NUM> may be combined on basis of the assignment of test steps <NUM> to test step definitions <NUM>. An example is shown in <FIG>, where two graphical test models <NUM> are generated from the above example, namely the case where a user has entered credentials and the case where a user has not entered credentials. The credentials may be invalid or valid or the user may not be registered at all. These different test data (user credentials: invalid, not registered, valid) may be identified by the NLP based on the natural language phrases and may be used to combine similar test scenarios <NUM>, as it is shown in <FIG>.

As indicated in <FIG>, the method M may comprise under T4 comparing the graphical test models <NUM> with the test scenarios <NUM> to determine if the graphical test models <NUM> are in compliance with the expected behavior of the software program. For example, based on these graphical test models <NUM>, missing scenarios <NUM> may be identified, e.g. a user with expired credentials or a registration attempt of an already registered user. Moreover, optimized test scenarios <NUM> may be generated, e.g.:.

The method M further comprises under M7 generating executable test scripts <NUM> for the test scenarios <NUM> on basis of the assigned test step definitions <NUM>. To this end, existing BDD tools and frameworks may be utilized, e.g. Cucumber or similar.

As a result, the method M described above provides the means to utilize the benefits of BDD (with easy to use, natural language based scenarios) while maintaining the required manageability for large, complex development projects. The generation and synchronization of a test model from BDD scenarios allows to utilize the advantages both from BDD and MBT methods, especially for large complex development projects. The model based review and generation of test cases ensures the consistency and completeness of the BDD scenarios. The ability to use MBT techniques to add an additional abstraction level bridges the gap between RE focused usage of BDD and a BDD based test automation approach. The automated step matching using machine learning allows an efficient mapping of BDD step phrases to a test automation framework and supports the structured development of the necessary framework code.

In the foregoing detailed description, various features are grouped together in one or more examples or examples with the purpose of streamlining the disclosure. It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other examples will be apparent to one skilled in the art upon reviewing the above specification.

Claim 1:
A computer-implemented method (M) for automated verification of a software program in a behavior-driven development environment, the method (M) comprising:
receiving (M0), with a data processing system (<NUM>), test scenarios (<NUM>), each test scenario (<NUM>) defining an expected behavior of the software program in consecutive test steps (<NUM>), which are formulated in a domain-specific language using natural language phrases and which describe a desired outcome of the software program for predefined events based on given initial conditions;
importing (M1) test step definitions (<NUM>) from the behavior-driven development environment;
determining (M2) for each test step (<NUM>) of the test scenarios (<NUM>) if the test step (<NUM>) matches with one of the test step definitions (<NUM>) on basis of the natural language phrases of the test step (<NUM>);
assigning (M3) all matched test steps (<NUM>) to the corresponding test step definitions (<NUM>);
applying (M4) natural language processing, NLP, on the natural language phrases of any test steps (<NUM>) remaining unmatched, wherein the NLP provides a confidence level for each unmatched test step (<NUM>) to correspond to one of the test step definitions (<NUM>);
assigning (M5) any unmatched test step (<NUM>) to the corresponding test step definition (<NUM>) when the confidence level surpasses a first predefined matching probability; and at least one of:
generating (M6) graphical test models (<NUM>) for the test scenarios (<NUM>) on basis of the assigned test step definitions (<NUM>); and
generating (M7) executable test scripts (<NUM>) for the test scenarios (<NUM>) on basis of the assigned test step definitions (<NUM>); further comprising:
updating (T1), when the confidence level is above the first predefined matching probability, the respective test step definition (<NUM>) on basis of the natural word phrases of the respective test step (<NUM>); and
adding (T2), when the confidence level is below a second predefined matching probability, a test step definition (<NUM>) to the behavior-driven development environment corresponding to the respective test step (<NUM>).