Method and system for automating the process of testing of software applications

The invention relates to a method and a system for testing interactive applications on at least one TV device according to at least one test scenario. The test scenario is run and monitored by a test driver system separate from the at least one TV device, wherein the test scenario comprises a series of user interaction steps and at least one target state of the application and/or the at least one TV device to be achieved after a specified user interaction step, the method comprising the steps of: a. integrating a test definition helper library into the interactive application, wherein the test definition helper library is configured to obtain information on the state of the interactive application and/or the at least one TV device; b. providing the modified interactive application to the at least one TV device; c. running the at least one test scenario, wherein the test driver system non-transparently performs the series of user interaction steps on the at least one TV device; and d. comparing the information on the state of the interactive application and/or the at least one TV device obtained by the test definition helper library with the at least one target state after the specified user interaction step.

BACKGROUND

The present invention relates to a method and a system for testing interactive applications on at least one TV device according to at least one test scenario.

Connected TV devices are rapidly gaining popularity these days and a multitude of interactive applications for TV devices already exist. Before releasing to production every interactive application must be properly tested to ensure that all execution paths function properly, the edge cases are covered and the application is fully compatible with all TV devices on the target platform. Testing interactive applications manually has a number of obvious drawbacks, among other error proneness due to human mistakes, long time-to-market, increased costs for QA team and testing devices, etc. Known automatic testing approaches, such as described in U.S. Pat. No. 8,185,877 B1, are not capable to be used for the testing of interactive applications on TV devices. The testing of TV devices therefore requires highly skilled technicians, workflow and development cycle changes, and pose a number of conditions and restrictions on the interactive application development style.

SUMMARY AND DEFINITIONS

In view of this, a method and a system are provided for testing interactive applications on TV devices which provides provide a simple and user friendly way of defining and performing test scenarios.

The test scenario is run and monitored by a test driver system separate from the at least one TV device. The test scenario comprises a series of user interaction steps and at least one target state of the application and/or the at least one TV device to be achieved after a specified user interaction step. The method comprises the further steps of:a. integrating a test definition helper library into the interactive application, wherein the test definition helper library is configured to obtain information on the state of the interactive application and/or the at least one TV device;b. providing the modified interactive application to the at least one TV device;c. running the at least one test scenario, wherein the test driver system non-transparently performs the series of user interaction steps on the at least one TV device; andd. comparing the information on the state of the interactive application and/or the at least one TV device obtained by the test definition helper library with the at least one target state after the specified user interaction step.

First of all, a few terms used within the present invention are explained. Hereinafter the term “TV device” is used to refer to a television or to a device being used in connection with a television in order to display image and/or video content on the television. Examples of such a TV device include but are not limited to: Television sets, Set Top Boxes, or other devices having a capability to broadcast or transmit image and/or video content to a TV, such as PC sticks, gaming consoles, broadcasting devices, multimedia devices, streaming devices, phones, tablets and other mobile devices or other devices which possess a screen beaming capability.

Most of the present TV device platforms support interactive applications which may be built using either common web technologies or SDKs provided by device manufacturers or custom software. The term “interactive application” denotes any kind of application which can be run on a TV device. Interactive applications within the meaning of the present invention particularly comprise applications installable from the device manufacturer's portal and applications that are complementing a TV channel. Applications from the first group are in the following referred to as SmartTV applications. Applications from the second group use different technology standards in different countries, wherein the prevailing technology standard is HbbTV (Hybrid broadcast/broadband TV). Applications from the second group are referred to as broadcast applications in the following. It will be appreciated that the present invention supports testing of both broadcast applications and SmartTV applications.

Within the present invention, the term “target state” denotes a state of the interactive application and/or the TV device which is expected after a corresponding specified user interaction step in case the interactive application and/or the TV device functions properly.

The test driver system (TDS) runs the test scenario in a way, which is non-transparent to the application and/or the TV device being tested. The term non-transparent within this context means that the application and/or the TV device on which the user interaction step or series of user interaction steps is performed is not able to distinguish between a user interaction step coming from the TDS and a regular (i.e. manual) user input signal coming from an end-user.

The TDHL is integrated into the interactive application. Within the present invention the term “modified interactive application” is used in order to denote an interactive application comprising the test definition helper library.

It will be appreciated that the integration of the TDHL into an application is a very simple process that usually requires no more than 1-4 lines of code. For example, in case of a TV application built using web technologies the TDHL together with a unique application id (UAID) may be integrated by adding a script tag to every HTML page of the application. In case of applications built with other technologies then web, this generally means to include a library and execute a predefined initialization call for passing over the UAID. It will be appreciated that TDHL is a very carefully crafted software that usually has none or very insignificant impact on the overall functioning of the application. Since TDHL poses generally no conditions on the application, developers do not have to make any adjustments to the application code—the development process may simply ignore the presence of TDHL. It will be appreciated, that in some embodiments the TDS may be implemented to inject the TDHL with UAID into an application automatically without a manual intervention from the developer.

Just before the developer signs the application over for testing he injects TDHL into it. TDHL may create a persistent data connection between the device on which the app is running and the TDS. However, usually a persistent data connection is not required at all times. In some embodiments TDHL may be configured to collect test related results continuously and deliver them all at once to TDS after data connection becomes available. It preferably acts as a man-in-the-middle between the users input and the application. During the runtime TDHL may maintain a persistent or non-persistent connection to TDS to provide information about the application runtime internals in real time. TDS can instruct it to perform inspection of various UI elements of the application as well as its non-visual properties in order to perform a test.

One of the main ideas is to integrate the test definition helper library into the interactive application, wherein the test definition helper library is configured to obtain information on the state of the interactive application and/or the at least one TV device. Thus, the present invention disclosure suggests to modify the interactive application to be tested. The functionality to obtain information on the state of the interactive application and/or the at least one TV device is directly provided to the interactive application. It is not necessary to monitor the state of the interactive application from “outside” of the interactive application, for instance by means of capturing and analyzing the display output of the application directed to a TV device. Rather, the relevant information is directly obtained by the TDHL being integrated into the interactive application and may be forwarded to the test driver system for further analysis. This considerably simplifies the testing process, as TDS may readily be configured to supervise test execution on each device, collect results from each device and compile a comprehensive report.

A further advantage of the present invention relates to the TDS being configured to non-transparently perform the series of user interaction steps on the at least one TV device. Due to the non-transparency the TV device cannot distinguish between user interaction steps generated by the TDS and a regular user input signal generated by the end user. This is crucial in order to provide a realistic test scenario which does not differ from a corresponding “real operating scenario” of the TV device.

An effort has been made to make automated testing of TV devices more approachable and understandable for individuals without requiring any special technical knowledge. With respect to the testing process itself it will be appreciated, that the person performing the test does not need to know the internals of the application intimately. The test operator is only expected to know the applications specification, i.e. expected behavior and response to a particular user action. The test operator may build a number of test cases with the help of an optionally provided graphical user interface (GUI) on any TV device currently at his disposal. After the test case is thus prepared it can be scheduled for unattended execution on a number of TV devices that the application is targeting.

In a preferred embodiment the data obtained by the test definition helper library is transmitted to the test driver system, wherein the comparison in step d. is preferably performed by the test driver system. Thus, the information on the application and TV device state can be effectively analyzed and further processed.

Preferably, the information on the state of the interactive application and/or the at least one TV device obtained by the test definition helper library comprises at least one of the following:a TV device identification value,information on runtime objects of the interactive application,information on view objects of the interactive application,information on data traffic caused by the interactive application,console output of the interactive application,operating system runtime changes and/orshared memory status.

Obtaining one or more of the above noted data allows to generate an accurate analysis on the state of the interactive application and/or the TV device.

It is furthermore preferred that the test driver system comprises a translation module to translate the user interaction steps of the test scenario into user input signals adapted to each of the at least one TV device.

The interactive TV application may be a SmartTV application or a broadcast application, wherein the test driver system is preferably configured to automatically install or provide the modified interactive application on the at least one TV device. The idea to use a test driver system being capable to automatically provide the modified interactive application to the at least one TV device is particularly beneficial, as it considerably simplifies the test preparation process. In particular, it is not required that each TV device is individually attended to in order to install the modified application. Thus time and costs are saved.

In case the interactive application is a broadcast application, it is preferred that the test driver system provides the at least one TV device with a television signal, preferably with a DVB-signal, wherein the television signal is preferably adapted to provide the modified interactive application to the at least one TV device. Herein the term DVB-signal denotes signals being configured to deliver digital television, such as, in particular, DVB-T, DVB-T2, DVB-S, DVB-S2, DVB-C, DVB-C2, etc. Preferably the DVB-signal comprises an application information table (AIT). With this idea, a particularly beneficial method is provided for automatically providing the modified interactive application to the at least one TV device. Within this embodiment it is not necessary to perform any modifications or adaptions on the at least one TV device. It is simply sufficient to set the at least one TV device into its usual operation mode in which it receives the television signal provided by the test driver system, wherein the modified interactive application is provided by the television signal.

In a preferred embodiment the test driver system comprises an infrared emitter, a wireless radio wave transmitter and/or a network connection module for non-transparently performing the series of user interaction steps on the at least one TV device. Due to this measure typical user interaction steps can be carried out by the test driver system. The wireless radio wave transmitter may, for instance, use the Bluetooth® standard. If, for example, the interactive application running on the TV device is controlled by HTTP Api calls, the network connection module may be adapted to send such HTTP Api calls in order to non-transparently perform the series of user interaction steps. If, as a further example, the TV device is a PC-stick navigable by a standard keyboard, the network connection module may be configured to send emulated keyboard signals to the PC-stick in order to non-transparently perform the series of user interaction steps.

The test driver system may be configured to intercept network traffic of the at least one TV device to provide predefined versions of network resources requested by the interactive TV application.

Subject of the present disclosure is furthermore a method for defining a test scenario for use in a testing method as described above, wherein the method uses a reference TV device and comprises the following steps:a. integrating a test definition helper library into the interactive application, wherein the test definition helper library is configured to obtain information on the state of the interactive application and/or the at least one TV device;b. providing the modified interactive application to a reference TV device;c. providing a manual user input signal to the reference TV device, wherein the test definition helper library and/or the test driver system is configured to monitor the manual user input signal;d. monitoring the information on the state of the interactive application and/or the reference TV device subsequent to the manual user input signal provided in step c.;e. defining a specified user interaction step and a corresponding target state of a test scenario, wherein the specified user interaction step is based on the monitored manual user input signal and wherein the target state is based on the monitored state of the interactive application and/or the reference TV device obtained in step d.

The reference TV device preferably is a TV device on which the interactive application is already tested, wherein the interactive application functions properly on the reference TV device. The reference TV device can, thus, be used to define the target state of the test scenario and the corresponding user interaction steps. The inventive method allows the definition of the test scenario in an efficient and straightforward way, as simply a manual user input signal can be directed at the reference TV device, wherein the manual user input signal and the target state of the interactive application and/or the TV device is automatically monitored and used to define the test scenario.

Subject of the present disclosure is furthermore a test system being adapted to perform the testing method as described above, wherein the system comprises a test driver system and at least one TV device. The at least one TV device comprises the interactive application, wherein the test definition helper library is integrated into the interactive application. The test definition helper library is configured to obtain information on the state of the interactive application and/or the at least one TV device. Furthermore, the TDS is adapted to run the at least one test scenario by non-transparently performing the series of user interaction steps on the at least one TV device. The system is furthermore adapted to compare the information on the state of the interactive application and/or the at least one TV device obtained by the test definition helper library with the at least one target state after the specified user interaction step.

Preferably, the test driver system is adapted to carry out the comparison.

In a preferable embodiment, the test driver system comprises a translation module configured to translate the user interaction steps of the test scenario into user input signals adapted to each of the at least one TV device.

It is furthermore preferred that the test driver system comprises an executive unit configured to emit a user input signal adapted to each of the at least one TV device, wherein the executive unit preferably comprises an infrared emitter, a wireless radio wave transmitter and/or a network connection module. Preferably, an executive unit is provided for each of the at least one TV devices.

In another embodiment, the test driver system furthermore comprises a learning unit configured to receive a manual user input signal adapted to the at least one TV device, wherein the test driver system comprises a conversion module configured to convert the manual user input signal into a user interaction step of the test scenario.

The test driver system may comprise a broadcasting unit configured to automatically install or provide the modified interactive application on the at least one TV device, wherein the interactive application preferably is a broadcast application, wherein the broadcasting unit preferably provides the at least one TV device with a television signal, preferably a DVB-signal, wherein the television signal is preferably adapted to provide the modified interactive application to the at least one TV device.

The system may be further developed by means of the features which have been described in relation with the methods.

It will be appreciated that it is generally possible to apply the idea of the present disclosure to any software application independently of its implementation technology or target platform. However, the present independent claims are explicitly restricted to the testing of interactive applications running on TV devices.

DETAILED DESCRIPTION

An exemplary embodiment of the inventive system is illustrated inFIG. 1. The main components of the system are a test driver system (TDS)14and a test definition helper library (TDHL)15which is integrated into an interactive application running on TV devices18.

The TDS14comprises a central server24, a control unit21, an executive unit20, a translation unit19, a broadcasting unit17, a learning unit22, and a conversion module23. A graphical user interface (GUI)13is running on the central server24and may be accessed via the internet. The local arrangement of the elements of TDS14shown inFIG. 1is an exemplary arrangement only and does not imply any restrictions on the actual location of these elements.

The central server24is a network server that hosts all of the TDS14software. The central server24may be distributed on different physical computer machines that may be located in different geographical locations. The central server24is connected to the control unit21which controls the TV devices18. Control unit21and TV devices18are located in the same IP network. As illustrated inFIG. 2, the central server24may be connected to different groups27of TV devices18, wherein each group is controlled by a separate control device (not shown), and wherein each group27and associated control device is part of a separate IP network. Several test operators26may connect with the central server24by means of user accounts and simultaneously perform test operations on the groups27of TV devices18.

Referring back toFIG. 1, the control unit21controls all TV devices18and reports to the central server24. It creates and maintains a persistent connection to the central server24. After pairing with user account, it can listen to commands from central server24and forward them to control unit21or other control units being located on different IP networks, or collect local metrics. When used in tandem with routing unit (explained below), it can provide http proxy capable of translating domains and injecting testing scripts. The control unit21may be a microcomputer such as Raspberry Pi®. Nevertheless any computing device with a CPU28architecture based on, for example, ARM®, x86 or AMD64® and running Linux®, FreeBSD® or Windows® can be deployed as control unit. Those skilled in the art will appreciate, that any modern home PC is suitable for becoming a control unit21.

The broadcasting unit17is used to broadcast a DVB-signal including an application information table (AIT), which is received by the TV devices18. The TV devices18are enabled to run broadcast applications and are tuned to the DVB-signal broadcasted by the broadcasting unit17. The DVB-signal comprises an interactive application into which the TDHL is integrated. The interactive application comprising the TDHL automatically starts on the TV devices18due to an autostart record in the AIT of the DVB-signal. Those skilled in the art will appreciate that there is at least one other way to achieve this by using a static AIT with a specially crafted autostart application that connects to TDS14and then itself starts the target application on behalf of TDS14. The broadcasting unit may be a cable/terrestrial modulator, such as a DekTec DTU-215° or any other device with capabilities as described above.

The TDHL15is a software component being able to inspect the runtime of the interactive application and/or the view objects of the application, listen, collect and/or transmit to TDS the network traffic made by the application, listen and/or collect and/or transmit to TDS the console output of the application, perform screenshots or video capture of the renderings on the devices screen and/or transmit them to TDS14. In some embodiments the TDHL15may utilize available platform tools to perform inspections and measurements outside the application being tested, for example inspect the shared memory or the system internals. TDHL15is written, compiled and/or transpiled to the language supported by the platform of the interactive application being tested. For example, the TDHL15for a broadcast application is written, compiled and/or transpiled to JavaScript. Another feature of the TDHL15is, that it poses no restriction on neither the TV device18nor the interactive application running on that device.

GUI13is a web based graphical user interface application but may in other embodiments be any other type of application capable of running on a personal computer or similar device. GUI13serves for defining user interaction steps as well as complete test scenarios. GUI13communicates with the control unit21of TDS14via central server24. GUI13usually does not communicate with other components of the system directly. GUI13may offer means for performing advanced data analysis based on historical test executions as well as a statistical overview. The GUI13is a graphical user interface application and can be used for creating and managing test cases, viewing test results and reports, viewing test history, managing devices currently in the system or adding new device, sharing devices with other users, managing app configuration, automatic test creation based on previous tests and machine learning algorithms.

It allows for creating tests in the natural test language as described below. In some embodiments GUI13may also allow adding tests of different type then the above, for example unit tests or integration tests. GUI13interacts with TDS14directly. In some embodiments GUI13may help the test operator to create a test condition automatically. Test operator opens GUI13and opens the application to be tested on the TV devices18. Test operator presses a “Select view element” button in GUI13which triggers the element selection mode. In element selection mode the GUI13makes it possible for the test operator to select a particular view element and automatically retrieve its current properties. Provided that GUI13is launched on a personal computer or similar device, in some embodiments pressing the “Select view element” could cause pairing of the movements of the mouse attached to PC, on which the GUI13is running, with the movements of a visual pointer on the TV device18. Clicking the mouse button can retrieve the properties of the element currently being pointed to with said pointer. In some embodiments the test operator may be given an option to specify a selector string that is used to identify the view element within the application's view tree. Such string may be a CSS or XPath selector string if the application being tested is HTML based. Alternatively it may be of any format recognizable by either the TDHL15or the underlying platform.

GUI13may also provide the user with a repository of the view element either preselected automatically by TDHL15or saved manually by the test operator. The thus selected view element may be then highlighted in the application as to provide a better orientation for the test operator. Those skilled in the art will appreciate that there are a number of other ways of selecting a particular view element.

GUI13may furthermore provide means of historical and comparative analysis of the test(s). In particular, it may offer tools to either fully or partially automatically determine at which point the application started failing and what was the cause for that failure. In some embodiments GUI13may generate textual and graphical reports and allow for a deeper drilldown into the test results and application internals.

GUI13may also serve as a tool for application debugging and/or development. For example GUI13may provide tools of launching an application while watching internet traffic caused by that application as well as its console output. GUI13may report errors or potential errors in the application fully automatically by merely letting the TDHL15perform the inspection routine of the application.

In some embodiments GUI13may also provide means for running the test iteratively (step by step) or executing it all at once on one or more TV devices. GUI13may also provide means of scheduling the test execution on one or more TV devices either remote or local.

In some embodiments GUI13may also provide means of synchronizing the test progress with the application state on the device. Those skilled in the art will appreciate that the test (any UI test) is in the essence a sequence of the user input steps that modify the application state. The GUI13may be able to warn the test operator whether or not the state of the application on the TV device18corresponds to the test progress inside the GUI13.

After the definition of a user interaction step by means of GUI13, the translation module19of TDS14translates the user interaction step into user input signals adapted to each of the at least one TV device. The user input signals are forwarded to the executive units20each of which comprise infrared emitters. The infrared emitters are able to emit infrared waves of length 940 nm which can be received by TV devices18. The user input signals are sent to TV devices18by means of the infrared emitters. The state of the interactive application and/or of the TV devices18is changed by the user input signals. In other embodiments executive units20may additionally or alternatively comprise a Bluetooth® transmitter and/or a network connection module configured to emit the user input signals to the TV devices18.

The executive units20further comprise an interface capable of connecting to control unit21via an interface such as USB, Ethernet, Wi-Fi, Bluetooth®.

It will be appreciated by the person skilled in the art, that the executive unit20may require additional modules or may be just a software. For example, if the TV device is controlled by HTTP Api calls, the executive unit20may be a piece of software running on the control unit21. Another example: if a device is navigable by a standard keyboard, the control unit21may include a keyboard emulation controller. Another example: if the remote control of the TV device is based on the Bluetooth® standard, the present invention may use an executive unit20equipped with a programmable Bluetooth® interface.

Test Language

Each test consists of an arbitrary amount of steps. Each step can be expressed in a synthetic yet human readable language. The test aims to model the actual behavior of an actual tester during a QA process. Supposing that MyApp is an app that responds to the navigational cursor buttons. Here is an example of a navigational test with 7 steps:1. Open MyApp or fail with a timeout after 5000 ms.2. Press button “DOWN” once.3. Press button “RIGHT” 3 times.4. If “video teaser” does not exist then fail test5. If “video teaser” background color is not equal to rgba(0, 0, 0, 1); then fail test6. Press button “OK” once.7. Wait until element “video player” appears or fail with a timeout after 5000 ms.

As is obvious from the test definition, this exemplary simple test aims to open the app, execute4button presses, verify the presence of the element “video teaser” and consequently check if it has correct background color. Then OK button will be pressed and the video player will be expected to appear.

It will be appreciated that this test definition is applicable to any interactive application on any TV device platform. The only assumption that this test makes is that the application responds to navigational buttons. Some embodiments of the inventive system may formulate navigational clauses applicable to cover navigational specifics of a particular platform.

The GUI13makes it generally possible to define tests exactly like the one example above. Operator can define each step by selecting options that modify the step from a very general to a very narrow one. For example to create a step that holds a test condition:1. Operator presses a button “Add condition step”.2. A following step is inserted into the test case: If [condition] then [continue test].3. Operator clicks on the [condition] and gets to pick the type of the condition out of the suitable options. The exact selection of options varies based on the platform and the type of application being tested. It will be appreciated however that the available conditions are thus that allow describing any present state of the application. For example for a web based application the list would be not limited to the following: [view element, videoplayer, page location, variable value]. For the sake of the example suppose the operator selects the value of [videoplayer] for [condition].4. The step is transformed like so: if [videoplayer] [has] [property] then [continue test].5. Operator can expand the [property] part in the same way as he expanded the [condition] part in the step 3. The [has] part can also be changed if necessary. Suppose that operator selects property “status” for [property]6. The step is transformed like so: if [videoplayer] [has] [status] [equal to] [stopped] then [continue test]. The GUI tries to autocomplete as much as possible in order to reduce the effort of the operator, so it would fill out the “playing” as the value of the current state of the background color. With step 6 the condition is completely expanded and the operator can proceed to the next step in the test case.

The invention gives operator the ability to define, for instance, that the step should be considered successful, if the condition result is true. This is controlled by the [continue test] part. Operator is able to change this to one of continue test, fail test, stop test, issue a warning.

Test Definition

Every test case in the exemplary system may be based on a single rule. Assuming that the application's content data remain unchanged, arbitrary fixed sequence of user interaction with the application must cause the application to enter an identical state on every target device after execution. I.e. if the user does the same thing to the application it should give him back the same result.

The process of test definition assumes that the test operator has a “reference TV device” at his disposal. For instance, one of the TV devices18may be a reference TV device. Alternatively, the reference device may be connected to the system in addition to TV devices18. On the reference TV device, the application to be tested functions without error. Having at least one such device is generally necessary since without attachment to the application state a test step is of no value.

It will be appreciated that the particular test step is able to be synchronized with the corresponding state of the application and/or the TV device18. It will be appreciated that the GUI13makes it possible for the test operator to fast forward or rewind to any test step and to view the application state on the TV device18. This can be a very important feature since the test operator is editing the test case based on the captured state of the application which is the only reliable source of information for the test step.

The exemplary system furthermore allows for recording manual user input signals directed to the application and/or the reference TV device18as well as the state of the application and/or the TV device18after each interaction. In other words, a test case can be recorded by simply using the application in the same way as the end user would. To start recording the test operator opens the application containing the TDHL15on the reference TV device, logs into his account at GUI13and presses “Start recording test” button in GUI13. This activates the watch function in TDHL15which intercepts every manual user input signal before it is passed over to the application and inspects the application state after this interaction. TDHL15sends its findings to TDS14which in turn forwards it to GUI13. As a result of this a new step of the test case appears instantaneously inside of the GUI13.

In some cases the TDHL15may not be capable of intercepting manual user input signals. The exemplary TDS14therefore furthermore comprises a learning unit22configured to receive manual user input signals. Such a manual user input signal may be, for instance, an optical signal (such as an infrared signal sent by a conventional remote control), an HTTP Api call, or a standard keyboard input signal. The received signals are forwarded to a conversion module23which converts the signals into a corresponding user interaction step. In addition, subsequent to each manual user input, the state of the application is monitored by the TDHL15and forwarded to the control unit21. By combining the user interaction step with the monitored state of the application, a test scenario can be created.

The learning unit22is an electronic device which may comprise the following further elements:an interface capable of connecting to control unit21via either of USB, Ethernet, Wi-Fi, Bluetooth® or any other interface,an optical sensor, which can be complemented by either a sensor measuring carrier frequency, or an array of sensors each filtering different carrier spectrum,a microcontroller for processing the data captured by the optical sensor.

As mentioned above, a particular TV device18may feature a method of control alternative to the traditional remote control. For example, the TV device18may be able to respond to API commands sent via HTTP. The learning device22may therefore be capable of receiving those alternative control methods as well. Every control unit21can utilize one or more learning units22in a particular group of devices.

Additionally, the TDHL15of the exemplary embodiment logs other events that emerge during the test recording session: data traffic, operating system runtime changes, memory status, etc. To achieve maximum precision TDHL15filters out events that might not have been caused by user interactions with the app or events that happen too often (hundreds of time per second). This events may be logged as a separate step in the recorded test case.

Test operator can delete or modify parts of the recording to further strengthen or broaden the test. Test operator can also define a reference element manually by a direct selection.

It will be appreciated that the exemplary embodiment employs predictive modelling algorithms and pattern recognition to offer automatic test creation based on static input parameters. For example, tagging a particular element of the application will cause the TDHL15to find its “siblings” that is elements with similar properties, repeat the action originally defined for the tagged element adapted to the context of the sibling and undertake an intelligent guess on whether or not the result of the operation is correct. Another example: If a certain sequence of the user input has caused the application to fail once, the TDHL15will be able to detect situations when it is likely that the application would fail again were it subjected to the same or similar user input sequence. In such cases TDHL15calculates the statistical variation determining the most likely failure scenario. Consequently and without supervision TDHL15generates a comprehensive test execution plan to cover the likely-to-fail user input scenarios.

It will be appreciated that the exemplary embodiment is able to control execution of tests different from the described above. For instance the system is able to execute and collect results of unit tests, integration tests or other types of tests commonly created by the software developers/QA personnel.

Test Execution

While the testing principle is the same for every platform, the actual test execution differs for every platform and the complexity of the system governing the process varies greatly. The TDS14component of the exemplary embodiment encapsulates the platform differences providing a unified application programming interface (API) for controlling an arbitrary amount of TV devices18of a particular platform in parallel. The following methods can be carried out by the exemplary TDS15: install application, launch application, send user input (key event, mouse event, touch event, other HID event), destroy application, recover from application failure, report application state, restart/reinitialize device, report component state, report runtime state, run a shell or platform specific command.

In the topic of test execution the TDS generally distinguishes between the reference TV device (used for test definition) and a test device (used for unattended execution of a ready test case). The function of a reference TV device is described above. It will be appreciated that the API methods work with arbitrary amount of test TV devices18at the same time and that the devices can join and leave the device pool at any point of time without influencing the operations of TDS14. It will be appreciated that the present disclosure allows for TV devices18to freely participate in the test device pool. A cluster of test devices, e.g. on the same (IP) network is also called a “device zoo”.

The device zoo comprising TV devices18is controlled by control unit21which is a dedicated computer machine with preinstalled software that is part of TDS14. In this case, the control unit21is also called zoo master. The zoo master is on the same network as other zoo members. A Raspberry Pi® may, for example, be utilized as a zoo master. Depending on the platform, additional hardware and software may be required to manage applications on the members of the zoo. Furthermore the system may be able to utilize devices belonging to the end users for executing tests when not in use by the end user.

Further Details of the Exemplary Embodiments

Hereinafter the term “emulator” shall refer to a computer software that emulates the behavior of a certain TV device. It shall be known to the persons skilled in the art, that it is a common practice of a hardware manufacturer to release a virtual machine containing software identical to that installed on the TV device. The “emulator” shall refer to such a virtual machine.

Hereinafter the term “platform” refers to the software preinstalled on the device at the point of leaving the manufacturing premises as well as modified versions of such software. This includes for example the device firmware, the operating system or a special set of applications installed on the device. A single physical device may have one or multiple platforms installed and supported.

TV devices18of different types, situated in different geographic locations and owned by different users or entities are supported.

A further embodiment is depicted inFIG. 3. In the embodiment ofFIG. 3the TDS14comprises a routing unit30and a network middleware unit31. The routing unit30is a specially configured network routing device. The TV devices18are connected to the routing unit by Ethernet connections (solid lines) or by a wireless network connection32. The routing unit30, in turn, is connected to control unit21and further to central server24. The network traffic between the TV devices18and the control unit21and/or central server24flows through the routing unit30. Routing unit30may inspect the network traffic and on behalf of the control unit21and/or central server collect ip addresses, mac addresses, spoof DNS records, rewrite destination ips or provide VPN services.

The routing unit30comprises an open API, either running DD-WRT® or Mikrotik®. The routing unit30is configured to automatically provide the TDHL15(not shown) to the interactive application running on the TV devices18, thus making the manual developer effort unnecessary.

Furthermore, the TDS14of the embodiment ofFIG. 4comprises a network middleware unit31. The network middleware unit31is used as a complement to the routing unit30, however, in other embodiments the network middleware unit31may also be used as an alternative to routing unit30. The network middleware unit31comprises a special software which allows for at least partial traffic interception and inspection. The network middleware unit provides predefined versions of network resources requested by the interactive TV application. The implementation method is as follows:

Inside the GUI13(not shown) a user will specify the URL of the application to be tested, additionally he will indicate, that the network middleware unit should be deployed when testing this application. In such a case, the central server24will negotiate with the network middleware unit31a map of network addresses. For the sake of example, consider an app that runs on the domain example.com and uses resources from a.example.com and b.example.com. Furthermore, suppose that domains *.invention.tld are set to resolve to the IP address of the same network middleware unit31.

Thus upon request from the central server24the network middleware unit31can create the following map for these example domains:

The central server24will govern itself by the above table. When starting the test of example.com it will instruct the TV device to open the corresponding domain from the mapping above, in this case x.invention.tld. Since the central server24resolves to the network middleware unit31, the request will arrive there. The network middleware unit31will recognize the x.invention.tld as the one generated by itself and will perform a network request to example.com—the corresponding domain in the map in order to fetch the data from that resource. After retrieving the data it will replace all occurrences of example.com with x.invention.tld inside the data. It will furthermore perform replacement according to the table above.

It will be appreciated by a person skilled in the art, that this approach will result in a resource of example.com and all related and linked to resources to be transmitted to the TV device with an alias of x.invention.tld (or according to the mapping described in the table).

FIG. 4shows several elements of a further embodiment of the present invention. An executive unit20is provided for each of TV devices18. The executive units20are connected to the control unit21. Furthermore a learning unit22is connected with the control unit21.

In contrast to the embodiment ofFIG. 1, the embodiment ofFIG. 3comprises several image capture units25, each of which is dedicated to a TV device18. The image capture units25are camera devices installed in way that allows filming the screen of the TV devices18. The image capture units25transmit the captured image data back to the control unit21, which further sends the captured image data to central server24(not shown). The control unit21or central server24assigns the captured images to the results of the test currently in progress. Generally, no image recognition is performed nor is any data mined from the image data by the present invention. The images purely serve for the purposes of illustrating a test cycle. In other embodiments, however, the image data may be used as an additional source of information on the state of the interactive application and/or the at least one TV device. This means that in addition to the information obtained by the TDHL of the present invention, further information may be obtained by image recognition or other kind of data mining performed on the image data.

Some embodiments may also support capturing the image from devices without a screen, such as Set-Top-Boxes.

Some or all elements of the exemplary embodiment can be combined to a single electronic device, for cheaper or easier installation.

For example control unit21and executive unit20can be combined, for instance by combining an infrared emitter with a Raspberry Pi® used as a control unit21. Learning unit22and executive unit20can be combined. In this case they share the same microcontroller and communication protocols, and the combined device can have sensors as well as emitters. The routing unit30can be integrated into control unit21, if the control unit21has more than one network interface. Image capture unit25can also be integrated into control unit21, for instance, if the control unit21is a notebook with a webcam pointed to the testing device.