Methods and systems for automating computer application tasks using application guides, markups and computer vision

In one aspect, a computerized method useful for automating computer application tasks using application guides, markups and computer vision includes the step of automating a human-implemented computer task. This includes determining a human workflow via a set of images. This also includes automatically deriving the workflow based on the set of images and a set of human user inputs. The method includes the step of, based on the automated human-implemented computer task, automatically implementing a decisions on behalf of a human user. The method includes the step of, implementing the automated human-implemented computer task using an in-application guide or a computer-vision step to determine a task the human user has performed and then performing the same task as the human user.

BACKGROUND

There is a need to enable any human computer task to be automated simply by observing human workflow via a set of images and deriving the workflow based on images and human inputs. Accordingly, improvements to ability to fully or partially automate human computer tasks and take decisions on behalf of the human are desired.

SUMMARY OF THE INVENTION

In one aspect, a computerized method useful for automating computer application tasks using application guides, markups and computer vision includes the step of automating a human-implemented computer task. This includes determining a human workflow via a set of images. This also includes automatically deriving the workflow based on the set of images and a set of human user inputs. The method includes the step of, based on the automated human-implemented computer task, automatically implementing a decisions on behalf of a human user. The method includes the step of, implementing the automated human-implemented computer task using an in-application guide or a computer-vision step to determine a task the human user has performed and then performing the same task as the human user.

DESCRIPTION

Disclosed are a system, method, and article of automating computer application tasks using application guides, markups and computer vision. The following description is presented to enable a person of ordinary skill in the art to make and use the various embodiments. Descriptions of specific devices, techniques, and applications are provided only as examples. Various modifications to the examples described herein can be readily apparent to those of ordinary skill in the art, and the general principles defined herein may be applied to other examples and applications without departing from the spirit and scope of the various embodiments.

Definitions

Example definitions for some embodiments are now provided.

Application programming interface (API) can specify how software components of various systems interact with each other.

Cloud computing can involve deploying groups of remote servers and/or software networks that allow centralized data storage and online access to computer services or resources. These groups of remote serves and/or software networks can be a collection of remote computing services.

Computer vision is an interdisciplinary field that deals with how computers can be made to gain high-level understanding from digital images or videos. Computer vision tasks can include methods for acquiring, processing, analyzing and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information (e.g., in the forms of decisions, etc.).

Document Object Model (DOM) is a cross-platform and language-independent application programming interface that treats an HTML, XHTML, or XML document as a tree structure where in each node is an object representing a part of the document. The DOM model represents a document with a logical tree. Each branch of the tree ends in a node, and each node contains objects. DOM methods allow programmatic access to the tree; with them you can change the document's structure, style or content. Nodes can have event handlers attached to them. Once an event is triggered, the event handlers get executed.

Hypertext Markup Language (HTML) is the standard markup language for creating web pages and web applications.

Machine learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that can teach themselves to grow and change when exposed to new data. Example machine learning techniques that can be used herein include, inter alia: decision tree learning, association rule learning, artificial neural networks, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, and/or sparse dictionary learning.

Exemplary Methods and Screenshots

Computer software tasks can include simple tasks (e.g. downloading W-2 for income tax reporting, etc.) and/or complex tasks (e.g. reporting expenses at the end of a business trip and getting approval from a manager, etc.). The tasks can be daily tasks (e.g. reporting working time at the end of a work day, etc.) and/or once a year tasks (e.g. setting up medical preferences at the medical election time, etc.). The tasks can start and end in one application and/or start in-app application and continue on to another application. Tasks can also start in an application installed on a desktop computer and end in a hand-held computer.

FIG.1illustrates an example process100for automating computer application tasks using application guides, markups and computer vision, according to some embodiments. In step102, a human computer task can be automated by observing a human workflow via a set of images and deriving the workflow based on images and human inputs. In step104, process100can fully or partially automate human computer tasks and take decisions on behalf of the human. Step104enables a computer to perform same task as a human user and take decision based on human provided intelligence at task creation time. In step106, process100implements the automation relies upon an in-app guide and/or computer vision to understand what human has been doing and doing same for the human.

By leveraging process100, a computer can implement the following example processes.download a W-2 report when instructed to do so by human (e.g. by observing how one human has done, same task can be done for another human user);complete an expense report task simply by reading an image consisting of a restaurant bill (by observing how one human does expense reporting same tasks can be done for another human);perform a business task like book a hotel or book a flight ticket based on given instructions and decision criteria; etc.

FIG.2provides another example process200for automating computer application tasks using application guides, markups and computer vision, according to some embodiments. In step202, an application software is installed on a web browser via a web-browser extensions. The application software has ability to record a software workflow when instructed to do by human user based on start and end actions. Mouse click results in capture of underlying screen along with necessary background information (e.g. Document Object Model and/or associated HTML elements in case of web browser)

In step204, an application software is installed on a computer machine via a desktop application. The application software has the ability to record a software workflow when instructed to do by human user based on start and end actions and/or record an underlying API response from underlying operating system (e.g. current windows, current application in focus, current screen in focus, etc.).

In step206, an application software is installed on a mobile machine via a mobile application. The application software has the ability to record a software workflow when instructed to do by a human user based on start and end actions and/or record every underlying API response from underlying operating system (e.g. current windows, current application in focus, current screen in focus, etc.).

FIGS.3A-F illustrate an example process300for expense reporting inside a web browser via screenshots302-312, according to some embodiments. An application installed on computer can be instructed to record every screen where a specific mouse action like click or enter is detected. More specifically, an application installed on a user's computer can be instructed to record every screen where a specific mouse action like click or enter operation is detected. By capturing and analyzing a set of screenshots302-306, a workflow can be created which tell computer-application automator as to how to report an expense given a set of input signals (e.g. date, amount and vendor).

The automator can review at each image and identify how to return back to same screen using a set of repeated input elements as documents in screen shots308. To start an expense reporting transaction automator uses a URL (e.g. identified in pink above via http://expenses), a label (identified in yellow above ‘Expense reporting Portal’) and click on a button ‘Create’ (identified in red rectangle). The signal to teach the computer system can be given explicitly by human (via in-app guides) and/or derived by reviewing images from a specified number of such transactions where triggering event has happened (e.g. click of a mouse). Computer vision can further be used to understand text behind buttons and gather further meta data about application (e.g. using browser HTML elements, API responses from WIn32 applications, etc.). The automator can further implement the tasks on behalf of the human by reading input signals from an input device (e.g. human can speak “Report my expenses for $100 for McDonalds on Aug. 24, 2018”) and the automator can understand and can execute human tasks via a set of screens.

Screen shot312shows the user to communicate that transaction has been completed. A report can be sent to the human user as well. By reviewing at URL and label on the screen, the automator can determine whether it has reached its goal and/or if further human help is needed.

FIG.4illustrates an example process400for computer vision for task automation, according to some embodiments. Process400can leverage the relevant DOM and images. In step402, process400can analyze images and DOM. In step404, process400can build image map and click maps. In step406, process400can read the red border (e.g. seeFIGS.5A-C) and determine location of new and enable users to click on new with just vision.

FIGS.5A-C illustrates an example set of screen shots502-506for an implementation of process400, according to some embodiments. The processes used herein use both DOM (HTML element like classid, divid, etc.) and images. The processes provided herein leverage both DOM and computer vision to make progress.

Example Systems

FIG.6depicts an exemplary computing system600that can be configured to perform any one of the processes provided herein. In this context, computing system600may include, for example, a processor, memory, storage, and I/O devices (e.g., monitor, keyboard, disk drive, Internet connection, etc.). However, computing system600may include circuitry or other specialized hardware for carrying out some or all aspects of the processes. In some operational settings, computing system600may be configured as a system that includes one or more units, each of which is configured to carry out some aspects of the processes either in software, hardware, or some combination thereof.

FIG.6depicts computing system600with a number of components that may be used to perform any of the processes described herein. The main system602includes a motherboard604having an I/O section606, one or more central processing units (CPU)608, and a memory section610, which may have a flash memory card612related to it. The I/O section606can be connected to a display614, a keyboard and/or other user input (not shown), a disk storage unit616, and a media drive unit618. The media drive unit618can read/write a computer-readable medium620, which can contain programs622and/or data. Computing system600can include a web browser. Moreover, it is noted that computing system600can be configured to include additional systems in order to fulfill various functionalities. Computing system600can communicate with other computing devices based on various computer communication protocols such a Wi-Fi, Bluetooth® (and/or other standards for exchanging data over short distances includes those using short-wavelength radio transmissions), USB, Ethernet, cellular, an ultrasonic local area communication protocol, etc.

CONCLUSION