Automated application interaction using a virtual operator

A computer-implemented method for automating interaction with a computer system includes linking a control computer system to an input interface and to an output interface of a client computer system, which is operative for producing user interface images on a display device. The control computer system executes distinct software modules that include a virtual operator for simulating actions of a human operator. Execution of the software modules causes the control computer system to capture an image from the output interface, and to recognize information in the image. In response to the information, the virtual operator controls an input device to automatically execute predetermined operations on the client computer system via the input interface.

FIELD OF THE INVENTION

The present invention relates generally to software solutions and specifically to providing automatic interaction among business software applications using a virtual operator.

BACKGROUND OF THE INVENTION

Modern organizations typically employ interactive business software solutions, including multiple operative products that may support diverse business units. Such products typically include software applications to provide support for business requirements, e.g., financial operations, customer relationship management, human resources, professional services, purchasing, and distribution.

The software applications may run on computerized systems, which can include legacy as well as modern systems. Often, at least a portion of the software and hardware of the computerized systems are tailored to support customized customer-specific functionality. Customizations may be applied at the time a new application is obtained, or as business needs change. Techniques utilized to enable a given system to be customized range from source code customization by professional services experts to interactive tool based approaches that allow end customers to make some changes themselves. At the end of the day, regardless of whether the software applications are “out-of-the-box”, customized, legacy or modern, they all need to be interactive to allow the data in each system to be accessible, shareable, and consistent.

BRIEF SUMMARY

An embodiment of the present invention provides a computer-implemented method for automating interaction with a computer system, which is carried out by linking a control computer system to an input interface of a client computer system and to an output interface of the client computer system, which is operative for producing user interface images on a display device. Distinct software modules are executed with the control computer system, including a virtual operator for simulating actions of a human operator. Execution of the software modules causes the control computer system to capture an image from the output interface and to recognize information in the image. In response to the information, the virtual operator controls an input device to automatically execute predetermined operations on the client computer system via the input interface.

Other embodiments of the invention provide computer software product and apparatus for carrying out the above-described method. Still other embodiments of the invention provide techniques for configuring a computer software product for carrying out the above-described method cooperatively with computer apparatus.

DETAILED DESCRIPTION OF EMBODIMENTS

Overview

Embodiments of the present invention that are described hereinbelow provide improved methods, products and systems for automating interaction with a computer system. When integrating multiple modern software systems with legacy software systems, for example, it is a primary goal to maximize the efficiency of the integration by minimizing the effort necessary for data to flow among the systems.

In the past, approaches to the issue have included manual data updates and queries to the legacy software systems, wherein human operators have prepared data in the required format, and have manually keyed the prepared data into the legacy software systems. This approach has proven to be costly in terms of resources, suffers from poor throughput, and is typically inaccurate. Another historical approach has been to replace the legacy software systems with new computerized systems designed to accurately integrate the data flow while providing the necessary functionality. This approach has proven to be quite expensive, wherein the replacement systems take significant resources to develop, often fail to meet the objectives of the design, and are typically provided later than scheduled.

Yet another approach from the past has been to develop automated processes to convert data off-line, e.g., the processes are scheduled to operate during non-production hours, changing data in a modern format to a format acceptable to the legacy software systems. One drawback of this historical approach includes a lack of direct connectivity among the modern and legacy software systems. For example, verification that a client address stored in a legacy software system is accurate can take an unacceptably long time. This approach has also proven to be extremely expensive from a strategic point of view, as the conversion processes are typically “hard-wired”, or precisely tailored to match the data structures of the legacy software systems. Constant updates to the conversion processes are required as the modern software systems evolve.

Embodiments of the present invention provide a virtual operator that emulates actions performed by a human operator using legacy software. The virtual operator interacts with the legacy software via an automated software system interfacing between modern and legacy software systems, or simply among any number of non-integrated software systems. In the context of the present patent application and claims, the term “virtual operator” refers to any software application or process that is capable of automatically performing actions on computer systems by interaction with a user interface, thus simulating identical actions performed by a human operator. In the context of the present patent application and claims, the term “legacy software system” refers to any software application that executes on a computer system and that continues to be used in spite of available newer technology, typically since the software application still meets the requirements of at least some of the users of the software application.

In embodiments of the invention, a control computer system is linked to input and output interfaces of a client computer system that executes legacy computer software. The client computer system produces user interface images on a display device, typically in response to data entered via the input interface, e.g., using a keyboard or mouse. A virtual operator running on the control computer system simulates actions of a human operator, as described hereinbelow. The virtual operator captures one of the images from the output interface, and recognizes information in the image. Typically, the virtual operator uses an image analyzer to compare the image with a collection of image templates. In one example, the image analyzer uses optical character recognition (OCR) to analyze the image. In another example, complex processes such as creating hierarchical relationships among areas of the screen may be used in order to apply disambiguation via semantic entity identification.

In some embodiments, the virtual operator verifies that the recognized information matches a known pattern. An image verifier is typically used to corroborate the match by extracting some type of marker from the image, such as text indicative of a screen function. For example, an image template may be used to make comparisons using the screen title text to identify the image. In another example, the image verifier may confirm that the currently displayed image on the display device of the client computer system is an address change screen.

The virtual operator controls the input device, automatically executing predetermined operations on the client computer system. The input device is typically used to input entry data comprising simulated keystrokes, mouse clicks, and other operator actions according to an input script, as described hereinbelow. In the present example, entry data is input to the input interface of the client computer system via the input device to change a customer address. Alternatively, predetermined operations may be executed to simulate other human operator actions, to query legacy software systems, or to insert and update data, as required. Embodiments of the present invention provide a single reliable, high-speed interface among multiple software systems, such as between a modern software system and a legacy software system.

System Description

Reference is now made toFIG. 1, which is a block diagram that schematically illustrates a system10for automating interaction with a computer system, in accordance with a disclosed embodiment of the invention. System10typically comprises a control computer system12and at least one client computer system14. Control computer system12may interface with multiple client computer systems, which are not shown for the sake of simplicity. Although portions of system10shown inFIG. 1and other drawing figures herein are shown as comprising a number of separate functional blocks, these blocks are not necessarily separate physical entities, but rather may represent, for example, different computing tasks or data objects stored in a memory that is accessible to a processor of control computer system12. Embodiments of the present invention enable control computer system12to automatically interact with the multiple client computer systems irrespective of their hardware and software configurations, as described hereinbelow. Control computer system12typically comprises a general-purpose control computer16, which is programmed in software to carry out the functions that are described herein. Control computer system12also comprises an input device18and a storage unit20, as described hereinbelow.

Control computer16comprises a processor22, a capture interface24, and a memory26. Capture interface24may comprise a Bluetooth® adapter, an Infrared Data Association (IrDA) device, a cable connection, or any communication interface for capturing image data that allows control computer16to receive output from a screen, e.g., in the form of an image bitmap. A virtual operator28, an image analyzer module30, and an image verifier module32are held in memory26. Virtual operator28, image analyzer module30, and image verifier module32may be downloaded to control computer16in electronic form, over a network, for example, or they may alternatively be provided on tangible media, such as optical, magnetic or electronic memory media. Further alternatively, at least some of the functions of control computer16may be carried out by dedicated electronic logic circuits. Although the embodiment relates to one particular system for automating interaction with a computer system, the principles of automated interaction that are implemented in system10may similarly be applied, mutatis mutandis, in other types of automated computer interaction systems using other techniques for automatically inputting entry data to client computer systems.

Input device18may comprise any suitable device or device simulator that allows control computer16to produce a sequence of device signals such as keyboard keystrokes or mouse clicks for input to a communications interface, which is typically connected to another computer system. One example of a suitable input simulator is the Microsoft Windows™ SendInput function for the Windows user interface, available at the Microsoft Developer Network website (msdn.microsoft.com). The SendInput function provides a tool to support simulation of keystrokes, mouse motions, and button clicks. Storage unit20may be any storage device that is capable of storing images captured by capture interface24, image templates, and any other data stored for retrieval by virtual operator28.

Client computer system14typically comprises a general-purpose client computer34, which is programmed in software to carry out the functions that are described herein. Client computer system14also comprises a screen36, as described herein. Client computer34comprises an input interface38, which may comprise any suitable communications interface that receives input signals representative of actions performed using input devices such as keyboards and computer mice. Input device18may provide simulated input, and as described hereinbelow, software applications may be used to enable remote control of client computer system14over a network. Thus embodiments of the present invention provide interoperability between client computer system14and input device18without necessitating any modifications to input interface38. Screen36comprises a display device and an output interface40, which may comprise a Bluetooth® adapter, an Infrared Data Association (IrDA) device, a cable connection, or any communication interface for outputting image data that allows client computer34to export visual display data, e.g., in the form of a compressed image.

System10may include devices42for producing a software product on a computer-readable medium, for example a CD or DVD writer. Devices employing many other technologies are also suitable for use as devices42, for example, writers adapted to tapes, hard disc drives, RAM, ROM, flash memory devices, floppy drives, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and magneto-optical discs. In all of these technologies patterns or transitions in the physical state of the medium are formed therein. These patterns and sequences of transitions encode instructions or data, e.g., binary numbers that are acceptable to the processor. The nature of the state transitions varies with the particular medium, and includes changes in magnetic domains in magnetic media, formation of pits or bumps on an optical disk, etc. Once the transitions are established, the medium may be transferred to another computing device44, which is capable of reading the medium, recognizing the transitions, and decoding them to identify executable instructions or data.

In an alternative embodiment of the present invention, control computer system12and client computer system14may run on a single computer system106as separate processes. Communication between the systems may comprise events passed between the separate processes via any suitable application programming interface known in the art. The present embodiment provides for simplified image capture and input signal generation due to the processes running on single computer system106, while eliminating the need to physically link multiple computer systems.

Reference is now made toFIG. 2, which is a schematic, pictorial illustration of a user interface screen, which displays user interface images, in accordance with a disclosed embodiment of the invention. Client computer34(FIG. 1) presents a user interface image46having application data fields48on screen36. For the purposes of illustration, image46is described using an example of an action performed on a software application to determine policy holder data in association with an insurance policy maintained in a legacy software system. Image46comprises a screen function label50, a policy number field52, a policy holder name field54and data fields48.

Client computer system14(FIG. 1) is adapted to execute at least one software application. The software application may be responsive to input operations, e.g., by signals sent to input interface38, as described in more detail hereinbelow. In the present example, the software application provides storage and maintenance functionality for policy holder data for an insurance firm. Typically, the software application generates images such as image46that are then displayed on screen36in response to the actions. For example, a logon session can involve performing multiple actions in the predetermined order required to enter login and password information.

The software application may require additional input operations that cause client computer34(FIG. 1) to display a different image on screen36. Virtual operator28typically captures the image from screen36or from output interface40, using any image capture method known in the art. Virtual operator28uses image analyzer module30to recognize information in the image. A pseudocode listing of a sample image analysis that may be used in the context of the present example, whereby the software application is running on a mainframe computer, is presented hereinbelow in Listing1. Function find_nearest in Listing1tries to locate each application data field based upon relative coordinates of the data field, for example, policy number field52.

Image verifier module32(FIG. 1) is used to confirm that the recognized information matches a pattern that is known to virtual operator28, as described hereinabove. A pseudocode listing of a sample screen recognition routine that may be used in this context is presented hereinbelow in Listing2. In the present example, the image contains a marker describing the function of the image, in the text of screen function label50. A pseudocode listing of a sample screen verification routine that may be used in this context is presented hereinbelow in Listing3. In the present example, screen function label50is verified to match expected screen function label values.

Typically, the software application provides some images that allow information entry, whereas other images are displayed that only allow information to be viewed. Some of the images provided by the software application may indicate to virtual operator28(FIG. 1) that information entry has been completed. The input operations required for executing actions for each software application are known in advance, in the form of the above-noted input script, which is typically stored in storage unit20.

Automatic Application Interaction

Reference is now made toFIG. 3, which is a flow chart that schematically illustrates a method for automating interaction with a computer system, in accordance with a disclosed embodiment of the invention. Shown by way of example, the name of a policy holder at an insurance firm may need to be changed due to an update in the marital status of the policy holder. Virtual operator28would therefore need to simulate several actions of a human operator in order for to automatically interact with a legacy software application running on client computer system14to successfully negotiate the requirement.

Control computer system12(FIG. 1) is linked to client computer system14in a computer system linking step56. Typically, input device18is linked to input interface38, and output interface40is linked to capture interface24.

Control computer system12(FIG. 1) executes virtual operator28to simulate actions of a human operator, thus automatically interacting with the software application running on client computer system14, in a virtual operator executing step58. Virtual operator28(FIG. 1) causes processor22to perform the steps listed hereinbelow.

Capture interface24(FIG. 1) captures an image, e.g., image46displayed on screen36(FIG. 2) in an image capturing step60. Additionally or alternatively, image46may be captured by capture interface24via output interface40. In the current example, output interface40is a universal serial bus (USB), and image46is captured from the software application.

As described hereinabove, image analyzer module30may be invoked by virtual operator28(FIG. 1) to analyze image46. Image analyzer module30recognizes information in image46in an information recognition step62, as described hereinabove in reference to the pseudocode listing presented in Listing1. A simple text layout is easily recognized by image analyzer module30, while comparing image46to image templates stored in storage unit20. The example provided by image46(FIG. 2), taken from a sample mainframe computer application, may be analyzed with ease by image analyzer module30. However, more complex analyses may be required if geometric primitives, e.g., rectangles or lines, are utilized by the software application running on client computer system14, as described hereinabove.

Virtual operator28(FIG. 1) typically uses image verifier module32to verify that the recognized information matches a known pattern in a recognized information pattern matching verification step64. In the current example, image verifier module32verifies that the marker describing the function of image46matches the image template used by image analyzer module30in information recognition step62. Screen function label50(FIG. 2), the marker in the present example, has the value “INQ100”, and is verified against the image template by image verifier module32, thus confirming the identification of image46as matching the known pattern of the address change screen of the software application delineated by the image template.

Virtual operator28responds to the pattern verification by image verifier module32, by controlling input device18to automatically execute predetermined operations on client computer system14in a device controlling and operation executing step66. These operations may be specified in an input script. A library of such input scripts may be stored in storage unit20. A pseudocode listing of routines to generate data input that may be used in this context is presented hereinbelow in Listing4. Virtual operator28causes input device18to automatically execute predetermined operations, e.g., inputting entry data to client computer system14, thereby simulating actions of the human operator. In an embodiment of the present invention, virtual operator28uses RealVNC© server and client software applications, available from RealVNC Ltd, Betjeman House, 104 Hills Road, Cambridge. Cambridgeshire, CB2 1LQ. UK, to control client computer system14over a network. The RealVNC applications utilize remote framebuffer (also known as Remote Frame Buffer or RFB) functionality, a protocol which provides remote access to graphical user interfaces at the framebuffer level, and is available from the RealVNC web site (realvnc.com). Thus, the RealVNC applications provide the ability to send the output of input device18to input interface38and to place device events into the input event queue of the operating system of client computer system14. The device events are then processed on client computer system14as though a physical input device had been used to generate the events.

Each input script provides entry data for performing actions that have been precisely tailored to execute operations for a specific software application on a particular computer system via input device18. In the present example, the entry data comprises keystrokes and mouse clicks input by input device18to execute the operations delineated by an input script. The input script provides the entry data that is necessary when using the software application running on client computer system14in order to automatically select policy holder name field54and to automatically revise the policy holder name of the example.

The method then terminates at a final step68.

In tests performed utilizing an embodiment ofFIG. 1of the above-described automated method, improved results were typically observed. The tests were performed using a single general-purpose computer running the Microsoft Windows operating system, wherein both control computer system12and client computer system14were implemented as processes running on the computer. In an exemplary test, a human operator interacted with multiple screens provided by a software application used for performance of a beneficiary change process in an insurance policy management system. Some application screens needed to be reviewed to verify information visually, and other application screens were used to apply changes utilizing the verified information. The human operator typically required approximately two minutes to complete a specific delineated set of actions. Application of the automated method resulted in identical results in an average of ten seconds, for a typical improvement in efficiency of approximately an order of magnitude.

Alternative Embodiments

Reference is now made toFIG. 4, which is a block diagram of an alternative system70for automating interaction with a computer system, in accordance with alternate embodiments of the invention. Apart from the differences described below, the operation of alternative system70is generally similar to that of system10(FIG. 1). As mentioned hereinabove, virtual operator28may interface with multiple client computer systems to automatically interact with software applications executing on the multiple client computer systems. In some embodiments, control computer system12utilizes additional software modules, e.g., duplicate virtual operator, image analyzer, and image verifier modules. In alternate embodiments, a single instance of each distinct software module is required to interface with the multiple client computer systems and multiple software applications running thereon.

In some embodiments, a client computer system72(FIG. 1) captures images from a scanner74linked to a client computer76, and the images are typically stored in a database78, as described hereinbelow. In an alternative embodiment, images are provided via any conventional image provider80. In yet another alternative embodiment, captured images are received via a computer network82after having been sent as image data over the computer network from a remote computer system.

In an alternative embodiment, a human operator may be used to verify simulated actions of virtual operator28(FIG. 1) prior to the entry data being input automatically to the client computer system. For example, when erroneous actions could have harmful effects, or significant risk is incurred by applying updates to the software system, the human operator may perform a review of the entry data on a control screen84and may provide a formal approval, before the entry data is entered to the software application, as described hereinbelow.

For purposes of illustration, in an embodiment of the present invention, a software application running on client computer system72(FIG. 4) is used to manage back-office services for an insurance provider. The software application provides services comprising receiving payments from customers, associating the payments with insurance policies having outstanding monetary balances, and selectively applying the payments to the balances.

Reference is now made toFIG. 5, which is a schematic, pictorial illustration of scanned and reference images using the embodiment ofFIG. 4. In the present embodiment, client computer76(FIG. 4) scans a check received from a client, and captures a check image86using scanner74, storing check image86in database78. Client computer76processes check image86and identifies an account number image88, a bank account owner signature image90, and a payment amount image92. The software application processes the images, deriving a bank account owner name94, (“Jane Doe”), an account number96, (“1234567890”), and a payment amount98, (“$122.50”), in the present example.

The software application searches database78(FIG. 4) and locates an insurance policy holder, (“Jane Doe”), that matches bank account owner name94, and provides client computer system72with an insurance policy holder listing image, which is substantially similar to image46(FIG. 2) for display by client computer76on screen36. The policy holder listing image typically displays the insurance policy number, the name of the customer, or insurance policy holder, and any outstanding monetary balance due. The software application displays check image86and the images identified thereupon, in addition to the processed check data, e.g., bank account owner name94, account number96, and payment amount98.

In the present example, virtual operator28(FIG. 4) may capture the insurance policy holder listing image from screen36. Image analyzer module30may be used by virtual operator28to recognize check image86and the images displayed thereupon, account number image88, bank account owner signature image90, and payment amount image92. Virtual operator28uses image verifier module32to verify that the images match the processed check data, typically stored in database78in the present example. Image verifier module32matches the images to the processed check data and in response to the matching, virtual operator28controls input device18to automatically input entry data to client computer system72to confirm processing of the check.

In an alternative embodiment of the present invention, after virtual operator28(FIG. 4) has simulated the actions of a human operator by confirming that the check should be applied to the insurance policy holder's balance, an additional step may be taken for the purpose of safety. The software application provides an additional verification image100to the human operator on control screen84. Additional verification image100comprises the images derived from check image86, the processed check data, an acceptance button icon102, and a rejection button icon104. The human operator can compare bank account owner signature image90to bank account owner name94, for example, and if satisfied, accept the check for payment of the insurance policy by using acceptance button icon102. Alternatively, rejection button icon104may be used by the human operator to reject the check for payment of the balance.

In the previous description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent to one skilled in the art, however, that the present invention may be practiced without these specific details. In other instances, well-known circuits, control logic, and the details of computer program instructions for conventional algorithms and processes have not been shown in detail in order not to obscure the present invention unnecessarily.