Replaying operations on widgets in a graphical user interface

A method for replaying operations on a widget in a graphical user interface (GUI) includes acquiring, from an operation record, operation information related to the widget and a first unique identification (UID) of the widget. A widget having a second UID is determined, wherein the second UID is generated based on a call stack for creating the widget in the GUI. The second UID is the same as the first UID. It may be determined whether the widget on the GUI is operable. An operation is executed on the widget according to the operation information related to the widget.

BACKGROUND

Embodiments of the present invention relate to graphical user interfaces, and more specifically, to methods, systems and computer program products for replaying the operations on widgets in a graphical user interface.

Robotic Process Automation (RPA) is the technology that allows a user to configure computer software, or a software “robot,” to emulate and integrate actions of a user interacting with a digital system, such as via a graphical user interface (GUI), to perform a process. An RPA robot can use the GUI to capture data, manipulate applications, and otherwise perform the process just like the user does. The RPA robot can interpret, trigger responses, and communicate with other systems in order to perform a vast variety of repetitive tasks.

BRIEF SUMMARY

Embodiments of the present invention are directed to a method, a system and computer program product. According to one embodiment, operation information related to a widget and a first unique identification (UID) of the widget are required from an operation record. A widget having a second UID is determined. The second UID is generated based on a call stack for creating the widget in the GUI during a replaying process. The second UID is the same as the first UID. It is determined whether the widget on the GUI is operable. An operation is executed on the widget according to the operation information related to the widget, which may be in response to the widget being operable.

DETAILED DESCRIPTION

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

An RPA system can replay operations on a plurality of widgets (control elements, e.g., button, check box, label, scroll bar, drop-down list, icon, link, and the like) in a graphical user interface (GUI) during a replaying process. So it is necessary to identify each widget in the GUI. In a traditional way, a widget in the GUI can be identified based on one or more attribute values of the widget such as name, class etc. Some widgets don't have the attribute value(s), so it is difficult to meet unique and consistent requirements of identifying a plurality of widgets in the GUI. Moreover, developers of the RPA system normally have no permission to change the source code of the GUI. In another way, a widget in the GUI can be identified based on the location of the widget in the GUI. However, the widget can't be accurately located if resolution of a screen of the GUI has been changed during the replaying process.

Therefore, it's desirable to provide a solution to precisely identify each widget of the plurality of widgets in a GUI when replaying the operations on a plurality of widgets. According to embodiments of the present disclosure, a new solution is proposed to identify a widget in the GUI by a unique identification (UID) of the widget generated based on a call stack for creating the widget in both a recording process and the replaying process. For the widget, since a UID generation strategy adopted in the replaying process is the same as that in the recording process, so the UID of the widget generated in the replaying process is the same as that generated in the recording process. With the UID of the widget, the widget can be accurately identified during the replaying process.

With reference now toFIG. 4, a RPA system400according to an embodiment of the present disclosure is depicted. The RPA system400may include a recording engine402and a replaying engine403. The recording engine402and the replaying engine403can be integrated with a graphical user interface (GUI)401in a browser. The recording engine402and the replaying engine403can also be implemented as a plugin in a browser. The GUI401is a user interface that allows users to interact with electronic devices through graphical icons and visual widgets, instead of text-based user interfaces, typed command labels or text navigation. The recording engine402can record operations on a plurality of widgets in the GUI401and store operation information related to the plurality of widgets as operation records in the storage404.

After acquiring the operation records from the storage404, the replaying engine403can enable a browser to replay the operations on a plurality of widgets in the GUI401. Although the recording engine402and the replaying engine403are illustrated in the same block400inFIG. 4, those skilled in the art may understand that the recording engine402and the replaying engine403can be implemented in a same server or in different servers in a distributed system. The recording process and the replaying process will be described respectively by referring to the RPA system400inFIG. 4.

The recording process:

FIG. 5describes an example method500for recording an operation on a widget in a graphical user interface (GUI) according to an embodiment of the present disclosure. The method500can be implemented in the recording engine402. The method500comprises S502-S512. For similarity, the method500describes how to record an operation on a widget.

At S502, in response to a creation of a widget in the GUI, a call stack for creating the widget is acquired. The GUI can include a plurality of widgets. Each widget of the plurality of widgets is a control element in the GUI. Each widget can facilitate a specific type of user-computer interaction, and appear as a visible part of an application that the GUI belongs to. Some widgets such as labels, buttons, dropdown menus, scroll bars and check boxes and the like can support an interaction with the user. Other widgets such as windows, panels, and tabs and the like can act as containers that group the widgets added to them. As known, a call stack is a stack data structure in which information about the active subroutines of a computer program is stored. The call stack is used to track a return address at which each active subroutine returns control back when finishing execution. Some GUIs can provide APIs (application programming interface) to monitor the creation of the widget. The APIs can be invoked to monitor the creation of the widget and to acquire the call stack for creating the widget. For example, a software application can use methods to create a widget. These methods can be hooked by a hooked code when the methods are invoked to create the widget. The hooked code can acquire the call stack for creating the widget. It shall be appreciated by those skilled in the art, a variety of hooking methods may be implemented to acquire the call stack for creating the widget.

At S504, a unique identification (UID) of the widget can be generated based on the acquired call stack. The UID of the widget is a unique identifier which can identify a widget. A generation rule for the UID of the widget can be predefined based on information related to the call stack for creating the widget by those who record the operation on the widget.

As the call stack is organized as a stack, a calling routine, which is a routine to call the above active subroutines to create the widget, can push the return address of the above active subroutines into the stack, and a called active subroutine, when it finishes, pulls or pops the return address off the call stack and transfers control to that address. The call stack is composed of stack frames. Each stack frame corresponds to a call to a subroutine which has not yet terminated with a return address. The stack frame at the top of the stack is for a routine being executed currently. An active subroutine is one that has been called but has not yet completed execution after which the control should be handed back to the return address. If, for example, a routine R1calls a subroutine R2multiple times from different places, R2must know respective return addresses when completing respective executions. The stack frame usually includes at least the following items (in push order):

arguments (parameter values) passed to the called subroutine (if any);

the return address back to the calling routine (e.g. in the R2stack frame, an address back to the routine R1); and

space for the local variables of the calling routine (if any).

According to one embodiment of the present disclosure, the UID of the widget can be generated based on the arguments passed to the called subroutine, or the return address back to the calling routine, or both the arguments passed and the return address. A rule for generating a UID for a widget can be based on information related to the call stack for creating the widget. A method name, arguments, return address, or any combination of method name, arguments, return address may be used to generate a UID, provided that the UID generation technique generates a unique identifier for each widget.

At S506, events of the widget are subscribed. According to an embodiment, the recording engine402can create a recording object to subscribe the events of the widget. The creation of the recording object is not dependent on S502and S504. According to an embodiment, the creation of the recording object can be in parallel with S502and S504. For example, the recording object can be created in response to the creation of the widget in the GUI. According to other embodiments, the recording object can be created before or after S502and S504. Those skilled in the art shall appreciate that a variety of methods may be implemented for creating the recording object to subscribe the events of the widget. According to one embodiment, the recording object can receive a notification when an event of the widget, for example button click event, occurs. In another embodiments, the recording engine402can subscribe the events of the widget in other ways, such as the recording engine402subscribes the events of the widget directly.

At S508, it is determined whether the widget is operable. At S510, in response to the widget being operable, operation information related to the widget are acquired. After the GUI is loaded and the widget can be operable, a user can make operations on the widget, for example, click a button or input parameters etc. According to an embodiment, since the recording object can subscribe the events of the widget, all operations made on the widget can be received by the recording object. The operation information related to the widget can be acquired from the recording object.

At S512, the operation information related to the widget and the UID of the widget are recorded as an operation record. A plurality of operation records may be recorded in sequence of operation. Once the recording engine402acquires operation information related to the widget, the recording engine402can record the operation information related to the widget and the UID of the widget as the operation record in sequence of operation in the storage404. The operation information related to the widget may comprise an operation type on the widget, the sequence of the operation, parameters related to the operation (if any) and the like, wherein the sequence of operation can be determined according to the time when the operation on the widget occurs. For similarity, the method500describes how to record an operation on a widget. The method500can applied to record operations on at least one widget of the plurality of widgets. For at least one widget of a plurality of widgets in the GUI, an operation record corresponding to the at least one widget of the plurality of widgets can be generated. The generated at least one operation record can be saved in the storage404in a form of a file or a table in a database.

FIG. 6shows a simple example GUI according to one embodiment of present disclosure. There are three widgets in the GUI. The three widgets include a user name input box602, a password input box604and a sign in button606. Once the three widgets are created, call stacks for the three widgets can be acquired from the hooked code. For example, a call stack for the user name input box in Table 1 indicates an invoking relationship among the methods and also indicates return addresses of the methods. The invoking relationship among the methods indicates that a method “loadLoginPage” invokes a method “layoutWidgets”, the method “layoutWidgets” invokes a method “createIDBox”, and the method “createIDBox” invokes a method “_createWidget”.

The UID of the widget can be generated based on the acquired call stack. A UID generation method can be implemented with an algorithm as long as the algorithm can make sure the generated UID of the widget is unique. According to one embodiment, the UIDs of each widget of three widgets can be generated based on a hash value of the method names and all the return addresses in the corresponding call stack. For example, Table 2 shows the three generated UlDs of the three widgets: “3Z9F45bqwrQ” for the user name input box602, “aR56OPweWe” for the password input box604, and “1cwUf6dveU” for the sign in button606.

The UID of the widget can be generated based on the acquired call stack. A UID generation method may be implemented with any algorithm as long as the algorithm can make sure the generated UID of the widget is unique. According to one embodiment, the UIDs of each widget of three widgets can be generated based on a hash value of the method names and the return addresses in the corresponding call stack. For example, Table 2 shows the three generated UIDs of the three widgets: “3Z9F45bqwrQ” for the user name input box602, “aR56OPweWe” for the password input box604, and “1cwUf6dveU” for the sign in button606.

According to an embodiment, the recording engine402can create three recording objects to subscribe to the events of the three widgets respectively. As shown in Table 3, the record object 1 is created to record the events of the user name input box602with a UID “3Z9F45bqwrQ”, the record object 2 is created to record the events of the password input box604with a UID “aR56OPweWe”, and the record object 3 is created to record the events of the sign in button606with a UID “1cwUf6dveU”. In this embodiment, the creation of the three operation records is not intended to limit the embodiments disclosed. Those in the art may understand that there are many solutions to record the operations on the widgets, for example, the recording engine402may record the operations on the widgets by itself directly.

After the GUI shown inFIG. 6is loaded in a browser with the RPA system400and the three widgets are operable, the user can make operations on the widgets. All those operations can be recorded automatically in the three record objects. For example, at first, the user inputs “administrator” in the user name input box602. Then the user inputs “password” in the password input box604. At last, the user click the sign in button606. All operations made on the three widgets can be received by the respective recording objects. For example, the record object 1 can receive the “administrator” input operation. The record object 2 can receive the “password” input operation. The record object 3 can receive the click operation. The three record objects can acquire the operation information related to the corresponding widget. For example, the operation information related to the user name input box602includes an operation type “input” and an operation parameter “administrator”. The operation information related to the password input box604includes the operation type “input” and an operation parameter “password”. The operation information related to the sign in button606includes an operation type “click”. Then the recording engine402can record operation information related to each widget and the UID of each widget as an operation record in sequence of operation. Table 4 shows an example table, in which the three operation records 1, 2, 3 are recorded in sequence of operation. The operation record 1 corresponding to the user name input box602includes an operation sequence “1”, the UID of the user name input box602“3Z9F45bqwrQ”, the operation type “input” and the operation parameter “administrator”. The operation record 2 corresponding to the password input box604includes an operation sequence “2”, the UID of the password input box604“aR56OPweWe”, the operation type “input” and the operation parameter “password”. The operation record 3 corresponding to the sign in button606includes an operation sequence “3”, the UID of the sign in button606“1cwUf6dveU” and the operation type “click”. The operation records in Table 4 is for purposes of illustration but are not intended to be exhaustive or limited to the embodiments disclosed. Those in the art should understand that the operation records can also be saved in a form of a file such as XML file or other data structures.

The replaying process:

FIG. 7describes an example method700for replaying operations on a plurality of widgets in a GUI according to an embodiment of the present invention. The method700can be implemented in the replaying engine403. The method700comprises S702-S710. For similarity, the method700describes how to replay an operation on a widget.

At S702, operation information related to a widget and a UID of a widget can be acquired from an operation record generated during the recording process. The UID for the widget may be referred to as the first UID of the first widget. As noted above, operation records may be saved during the recording process in the storage404in form of a file or a table. In a replaying process, e.g., the method700, the operation record can be acquired at S702from a file or table, such as Table 4 saved in the storage404.

At S704, for at least one widget of a plurality of widgets in a GUI, a UID corresponding to at least one widget of the plurality of widgets is generated based on a call stack for creating the at least one widget of the plurality of widgets. For example, assume the GUI shown inFIG. 6has been created or rendered, UIDs for each of the widgets602,604, and606may be generated based on the call stacks for creating the respective widgets602,604, and606.

At S706, a widget is determined or selected from the plurality of widgets in the created GUI based on the first UID of the widget. The UID of the determined or selected widget may be referred to as a second UID. The second UID is the same as the first UID. A technical problem of locating a particular widget among a plurality of widgets is solved by the widget having a unique identifier, and the second UID being the same as the first UID.

At S708, it may, in some embodiments, be determined whether the widget is operable on the GUI. Sometimes a widget that is visible will be disabled or otherwise not functional. In these cases, even though the widget has been rendered, it is not operable, i.e., the widget is not capable of executing an operation. In one embodiment, an input may be applied to a widget and a result of applying the input monitored to determine whether the widget is operable.

At S710, an operation, e.g., a replay operation, on the widget can be executed according to the operation information. S710may be in response to the widget being operable, although this condition is not required. It should be noted that the generation of the second UID at S704is not dependent on S702. According to an embodiment, the generation of the second UID at S704can be parallel with S702. According to an embodiment, the second UID at S704can be generated before or after S702.

According to an embodiment, the generating the second UID for each widget of the plurality of widgets in the GUI comprises: the call stack for creating each widget of the plurality of widgets in the GUI is acquired in response to each widget of the plurality of widgets being created in the GUI during the replaying process, and the second UID for each widget of the plurality of widgets in the GUI is generated based on the acquired call stack during the replaying process.

According to an embodiment, the first UID of the widget is generated based on the call stack for creating the widget in the GUI during a recording process in which operations on a plurality of widgets in the GUI are recorded. The recording process has been described in detail here before. The generation of the first UID comprises the following actions. First, the call stack for creating the widget in the GUI is acquired in response to the widget being created in the GUI during the recording process. Second, the first UID is generated based on the acquired call stack during the recording process. Those skilled in the art may understand that the call stack for creating the widget in the GUI during the recording process is the same as the call stack for creating the widget in the GUI during the replay process due to replaying operations on the same widget in the GUI of the same software application. As the same UID generation method is adopted during the recording process and the replay process, the first UID of the widget is the same as the second UID of the widget.

According to an embodiment, the operation record is created during the recording process. The creation of the operation record further comprises the following actions. At first, events of the widget are subscribed. Then operation information related to the widget is acquired in response to the widget being operable. At last, the operation information related to the widget and the first UID of the widget are recorded as the operation record.

According to an embodiment, it may be determined whether the widget is operable in the GUI, which may comprise: determining whether the widget is visible in the GUI, then it is determining whether the widget is operable in the GUI in response to the widget being visible. According to an embodiment, the executing an operation on the widget according to the operation information related to the widget further comprises the following actions. First, a replaying object to refer to the widget is created. Then, the replaying object is instructed to execute the operation on the widget according to the operation information related to the widget.

FIG. 8describes an example flow chart of process800for replaying operations on a plurality of widgets in a GUI according to an embodiment of the present invention.

At S802, an operation record in sequence of operation is acquired. For example, operations on the three widgets in the GUI shown inFIG. 6recorded during the recording process are to be replayed in a browser with the RPA system400. At S802, one of the operation records 1, 2 and 3 can be acquired from Table 4 according to the replay sequence of operations.

At S804, operation information related to a widget and a first UID of the widget can be acquired from the acquired operation record. For example, the operation record 1 can be acquired from Table 4, the operation record 1 containing: operation type “input,” operation parameter “administrator,” and the UID of the Widget “3Z9F45bqwrQ.”

At S806, it is determined whether a widget having the first UID exists, e.g., a widget having the first UID was created during the replay process. After the GUI is loaded in a browser with the RPA system400, for at least one widget of a plurality of widgets in the GUI, a UID corresponding to the at least one widget of the plurality of widgets is generated based on a call stack for creating the at least one widget of the plurality of widgets. According to an embodiment, a mapping relationship between a created widget and its corresponding UID can be recorded in, for example, a data structure or a file document, e.g., Table 5 described below. It may be checked whether the first UID can be found in the data structure or the file document recording the mapping relationship between the plurality of widgets in the GUI and the respective UIDs. If the check result is YES, it means the widget having the first UID can be retrieved from the plurality of widgets in the GUI, then the process800may go to S808. If the check result is NO, it means a widget having the first UID does not exist, then the process800go to the end.

Herein, the UID corresponding to the at least one widget of the plurality of widgets is generated based on a call stack for creating the at least one widget of the plurality of widgets. The generation of the UID corresponding to the at least one widget of the plurality of widgets may comprise the following actions. First, in response to a creation of the at least one widget of the plurality of widgets in the GUI, a call stack for creating the at least one widget of the plurality of widgets during the replaying process can be acquired. Second, the UID corresponding to the at least one widget of the plurality of widgets can be generated based on the acquired call stack. For at least one widget of the plurality of widgets in the GUI, the UID generation method adopted in the replaying process is the same as that in the recording process, e.g., S504. The details of the UID generation method for a widget may refer to S504, which is not repeated herein. So for the same widget, the corresponding UID generated during the replaying process should be the same as the first UID generated during the recording process.

At S808, it may be determined, in some embodiments, whether the widget is operable. According to an embodiment, the replaying engine403can create a replaying object to refer to the widget. The replaying object can check the status of the widget. If the check result indicates that the widget is operable, then the process800may go to S810. In fact, it is possible that the widget may be inoperable when it is checked for the first time. So, in some embodiments, the replaying object may check multiple times until the widget is operable. If the widget is still inoperable after being checked at least once or, in some embodiments, a predetermined number of times, e.g., 5 or 10 times, then the process800is ended.

At S810, if the widget is operable, the operation, e.g., a replay operation, on the widget can be executed according to the operation information related to the widget. According an embodiment, the replaying object can call method(s) or function(s) related to the operation of the widget. After the replaying object completes the operation on the widget, the replaying engine403can get a response from the replaying object. Then the replaying engine403may check whether there is a next operation record in the storage404. If the check result is YES, then the process800may go back to S802and repeat the whole process800. If the check result is NO, it means all operation records have been processed, then the process800may be ended.

According to one embodiment of present disclosure, after the GUI is loaded in a browser with the RPA system400, the replaying engine403can replay operations on the GUI shown inFIG. 6. There are three widgets shown in the GUI. The three widgets include the user name input box602, the password input box604and the sign in button606. After the three widgets were created, the call stacks for the three widgets can be acquired from the hooked code. The UIDs of the three widgets can be generated based on the respective call stacks. Since the UI generation method adopted in the replaying process is the same as that in the recording process, each of the UIDs generated in the replaying process is the same as that generated in the recording process. For example, Table 5 shows the generated UIDs of the three widgets602,604and606in the replaying process. It can be seen that the generated UIDs of the three widgets602,604and606in the replaying process in Table 5 are the same as those generated in the recording process in Table 2.

The replaying engine403can read the three operation records in Table 4 in sequence of operation. The operation information related to each widget and the UID of each widget can be acquired from each operation record. For example, at first, the operation record 1 can be read from Table 4. The operation type “input” and the operation parameter “administrator” as well as the UID of the Widget “3Z9F45bqwrQ” can be acquired from the operation record 1. Then the replaying engine403may check if any widget having the UID “3Z9F45bqwrQ” can be retrieved in Table 5 using the UID “3Z9F45bqwrQ” acquired from the operation record 1. In this case, the user name input box602can be retrieved to be the widget having the UID “3Z9F45bqwrQ”. Then, the replaying engine403may create replaying object 1 to refer to the user name input box602, as shown in Table 6. The replaying object 1 can check the status of the user name input box602. If the check result indicates that the user name input box602can be operable, then the replaying object 1 can input “administrator” into the user name input box602. After completion of the input operation, the replaying object 1 may return a response to the replaying engine403.

Then, the replaying engine403may check whether there is a next operation record to be processed in Table 4. In the following, the operation record 2 can be read from Table 4 in sequence of operation. The replaying process for the operation record 2 is similar to that for the operation record 1 described above. The operation type “input” and the operation parameter “password” as well as the UID of the Widget “aR56OPweWe” can be acquired from the operation record 2. Next, the password input box604can be retrieved to be the widget having the UID “aR56OPweWe”. Then the replaying engine403may create replaying object 2 to refer to the password input box604, as shown in Table 6. The replaying object 2 can check the status of the password input box604. If the check result indicates that the password input box604can be operable, then the replaying object 1 can input “password” into the password input box604. After completion of the input operation, the replaying object 2 may return a response to the replaying engine403. At last, the operation record 3 can be read from Table 4. The operation type “click” and the UID of the Widget “1cwUf6dveU” can be acquired from the operation record 3. Next, the sign in button606can be retrieved to be the widget having the UID “1cwUf6dveU”. Then, the replaying engine403may create replaying object 3 to refer to the sign in button606, as shown in Table 6. The replaying object 3 can check the status of the sign in button606. If the check result indicates that the sign in button606can be operable, then the replaying object 3 can click the sign in button606. After completion of the click operation, the replaying engine403can get a response from the replaying object 3. The whole replaying process is ended after all operation records have been processed. In above embodiment, the creation of the three replaying objects for replaying the operations on the three widgets is only illustrated, but are not intended to limit the embodiments disclosed. Those in the art may understand that there are many solutions to execute the operations on the widgets. For example, the replaying engine403may read each operation record and execute each corresponding operation by itself.