Automation identification diagnostic tool

An automation identification diagnostic tool (AIDT) is disclosed that records and analyzes actions within a program that are taken by an agent to accomplish certain tasks. By recording the action steps to accomplish a task in a recording session data file, the AIDT is then able to analyze the recording session data file to identify steps for conversion to automated tasks.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to Indian Provisional Patent Application No. 201641030353, filed Sep. 6, 2016, and Indian Non-Provisional Patent Application No. 201641030353, filed Apr. 5, 2017, the entirety of which are hereby incorporated by reference herein.

BACKGROUND

Due to the lack of available technology to improve the efficiency of certain manual tasks or to automate these tasks, many enterprises still rely on manual processes fulfilled by agents to operate.

Accordingly, it is desirable to create advancements in computer and networking technology for automating manual processes and identifying areas for improved efficiencies.

DETAILED DESCRIPTION

The methods, devices, systems, and other features discussed below may be embodied in a number of different forms. Not all of the depicted components may be required, however, and some implementations may include additional, different, or fewer components from those expressly described in this disclosure. Variations in the arrangement and type of the components may be made without departing from the spirit or scope of the claims as set forth herein. Further, variations in the processes described, including the addition, deletion, or rearranging and order of logical operations, may be made without departing from the spirit or scope of the claims as set forth herein.

An automated identification diagnostic tool (AIDT) is disclosed for recording manual processes implemented on a user's desktop computer, analyzing the recorded manual processes to identify manual processes that may potentially be automated, and generating a report that identifies manual processes that may be automated, identifies manual processes that may not be automated, and identifies manual processes that may be automated. The AIDT may be implemented as a web-based application configured for remote users to access the AIDT running on a server or other cloud location. The AIDT may also be implemented as a desktop application that is installed locally on a user's computer and is capable of running offline within a local computing environment.

The AIDT is comprised of at least three components: 1) A desktop recorder, 2) an AIDT analyzer, and 3) a workflow diagram generator. According to some embodiments, the workflow diagram generator may create a workflow diagram, as described herein. In addition or alternatively, according to some embodiments the workflow diagram generator may automatically create automation code (e.g. Solution Design Document (SDD), Process Design Instruction (PDI), Object Design Instruction (ODI) and business process (BP) code) in addition to, or instead of, the workflow diagram. The automation code may be generated to be run by a particular Robot Process Automation (RPA) tool to automate a particular process, where the RPA tool has been selected by the AIDT based on an analysis by the AIDT of past performance of the selected RPA tool in automating the particular process.

The Desktop Recorder is activated by an agent for recording manual processes implemented by the agent on their computer that is desired to be analyzed for automation. The Desktop Recorder records the agent's keyboard actions (e.g., mouse clicks and x & y coordinates; keyboard presses; desktop screen object detection (e.g., identify buttons and text fields selected by user)) as well as identify the application currently being accessed and receiving the agent's keyboard actions. The Desktop Recorder may further measure a length of time elapsed for the process, measure a length of time elapsed for each step in the process, count a number of steps in the process, and provide a graphical user interface for controlling the recording stop, start, and pause functions. The Desktop Recorder may generate a data file that describes the information recorded by the Desktop Recorder, where the data file may be forwarded to the AIDT Analyzer for automation analysis. The data file may include, for example, a Solution Design Document (SDD), a Process Design Instruction (PDI), an Object Design Instruction (ODI), or business process (BP) code.

Each detected keyboard activity may further cause the Desktop Recorder to capture a screenshot of the graphical user interface display at the time the keyboard activity was detected. The keyboard activity may be, for example, related to the activation of one or more keys on a keyboard (e.g., a function key, or enter key), or may further refer to a mouse click (e.g., left or right mouse click). Each screenshot captured by the Desktop Recorder may include identification information relating the screenshot to the corresponding keyboard activity that caused the screenshot to be captured and/or the corresponding application receiving the keyboard activity. According to some embodiments, a cognitive bot may be utilized within the Desktop Recorder to capture a portion of the desktop screen surrounding an area where the mouse click occurred, where the portion of the desktop screen that is captured by the cognitive bot is less than the entire desktop screen image. For a task that requires many steps to implement, this cognitive bot running within the Desktop Recorder may result in an overall memory storage savings as the enhanced screenshot being captured will be significantly smaller than the full desktop images that may be captured without the cognitive bot. The desktop area surrounding the mouse click captured by the cognitive bot may be a predetermined area set by the cognitive bot universally for all mouse clicks. In addition or alternatively, the cognitive bot may set the captured desktop area surrounding the mouse click dependent on the current application running during the mouse click (e.g., different predetermined area assigned for specific applications), or the object being clicked (e.g., different predetermined area assigned to different detected objects being clicked). For example, the cognitive bot may set the captured desktop area surrounding the mouse click to ensure that an object being clicked by the mouse click is captured in full by the captured desktop area. The Desktop Recorder may also capture a screenshot based on the detection of a specific predetermined keyboard activity (e.g., mouse click and function keys selected in combination)

The data file and screenshots may be automatically gathered by the Desktop Recorder as the Desktop Recorder monitors the information related to the detected keyboard activity. The data file and screenshots may be automatically combined into a discovery document that describes the keyboard activity of the user. The Desktop Recorder may further generate the following documents and include them into the discovery document: a process design document, a solution design document, an object design instruction, and/or a process design instruction may further be automatically created and included as part of the discovery document.

The manual process being recorded may correspond to a specific task available while using a specific application. For example, the process may relate to an invoice payment task using the Excel application. Multiple different instances of the invoice payment task being implemented, as accomplished by the same or different agent, may be recorded by the Desktop Recorder for analysis by the AIDT Analyzer.

The AIDT Analyzer receives the discovery document and runs an analysis on the discovery document to determine which of the recorded processes are capable of being automated. For example, the recorded processes in the discovery document may be compared against a list of predetermined processes that are known to be automatable. The list of predetermined processes may include, for example, the list described by table1200inFIG. 12.

When a recorded process in the discovery document matches a process in the list of predetermined processes that are known to be automatable, the AIDT Analyzer may tag the recorded process as being automatable. When a recorded process does not match up to a process in the list of predetermined processes that are known to be automatable, the AIDT Analyzer may tag the recorded process as not being automatable. There may also be instances where a recorded process is determined by the AIDT Analyzer as being potentially automatable. Such recorded processes may be tagged as being possibly automatable. As additional instances of the recorded processes (that includes repeated instances of the process tagged as being possibly automatable) are analyzed by the AIDT Analyzer, additional information for making a final determination as to whether the process tagged as being possibly automatable is automatable or not.

After running the analysis on the discovery document, the AIDT Analyzer may display a graphical user interface (GUI) describing the results of the analysis. An exemplary version of the resulting GUI is shown by first exemplary GUI800inFIG. 8. Another exemplary version of the resulting GUI is shown by second exemplary GUI900inFIG. 9.

An output of the AIDT Analyzer may include a text file that describes each of the automatable processes. An output of the AIDT Analyzer may also include an encrypted data file that includes the information displayed in, for example, GUI800or GUI900, as well as the screenshots captured by the Desktop Recorder. The output from the AIDT Analyzer may be referred to as a workflow diagram, where the workflow diagram describes each step taken by the user as reflected by the keyboard activity detected by the Desktop Recorder. An exemplary workflow diagram1100is illustrated inFIG. 11. The workflow diagram may be generated as, for example, a MS Visio document. Alternatively, the workflow diagram may be embedded into parts of a process design document and/or solution design document.

The output(s) of the AIDT Analyzer are then transmitted to a Robot Process Automation (RPA) tool to implement the automation. The RPA tool is responsible for receiving the workflow diagram, identify the automatable tasks, and generate the software code for implementing the automatable tasks. After the software code for implementing the automatable tasks is generated, the RPA tool may execute the software code to implement the automatable tasks. Going forward, the user may utilize the generated software code to run future iterations of the automatable tasks.

According to some embodiments, the AIDT Analyzer may generate the workflow diagram and recommend one or more RPA tools for implementing the automation described in the workflow diagram. Each RPA tool available to the AIDT Analyzer may be monitored and graded for each instance of automation the RPA tool implements historically. By monitoring and grading the RPA tool's performance, the AIDT Analyzer may create a record that tracks the different RPA tool's performance in automating specific types of tasks, or task types, so that eventually workflow diagrams may be assigned to the RPA tool having a history of high performance in automating the specific tasks described in the workflow diagram. Therefore the assignment of workflow diagram to known RPA tools may be made according to a set of rules developed by the AIDT Analyzer to assign workflow diagrams to the RPA tool having a history of high performance in automating the specific tasks described in the workflow diagram.

According to some embodiments, the AIDT Analyzer may recommend one or more RPA tools for implementing automation for processes detected by the AIDT Analyzer without the specific input of the workflow diagram. Each RPA tool available to the AIDT Analyzer may be monitored and graded for each instance of automation the RPA tool implements historically. By monitoring and grading the RPA tool's performance, the AIDT Analyzer may create a record that tracks the different RPA tool's performance in automating specific types of tasks, or task types, so that eventually workflow diagrams may be assigned to the RPA tool having a history of high performance in automating the specific tasks detected by the AIDT Analyzer. The assignment of workflow diagram to known RPA tools may accordingly be made according to a set of rules developed by the AIDT Analyzer to assign workflow diagrams to the RPA tool having a history of high performance in automating the specific tasks detected by the AIDT Analyzer.

FIG. 1illustrates exemplary system architecture for an application platform system100that includes component devices for implementing the described features of the AIDT150. Application platform system100includes an application server140configured to include the hardware, software, and/or middleware for operating the AIDT150. Application server140is shown to include a processor141, a memory144, a communication interface142, and a database interface143.

Application platform system100further includes a database library120configured to store application data such as the automation rules described by table1200inFIG. 12. The database library120is configured to provide data to the AIDT150either directly, or through a network130, via the database interface143. The data stored on the database library120may include a list of enterprise process steps recorded by a desktop recorder, such as the desktop recorder described inFIG. 3. In addition, the database library120may store enterprise process descriptions that describe a known list of actions that are included for known enterprise processes. For example, the database library120may identify a first enterprise process (e.g., human resources process for updating employee paid time off calculation) and describe each known step included to accomplish the first enterprise process. The enterprise process descriptions stored in the database library120may later be referenced to recognize an enterprise process in progress, and/or recognize a recorded enterprise process for automation.

The application platform system100communicates with any number and type of communication devices110, where communication device110may include well known computing systems, environments, and/or configurations that may be suitable for implementing features of the AIDT150such as, but are not limited to, smart phones, tablet computers, personal computers (PCs), server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, network PCs, server computers, minicomputers, mainframe computers, embedded systems, distributed computing environments that include any of the above systems or devices, and the like.FIG. 1shows that the communication device110includes a processor111, a memory114configured to store the instructions for operating a web application browser, an input/output device113, and a communication interface112. A user operating the communication device110may run the web application browser to access the AIDT150running on the application server140. In addition or alternatively, the AIDT150and corresponding data for operating the AIDT150may be installed on a user's computing device (e.g., communication device110) to run locally when a network connection to network130is not available or when enhanced security measures are required for isolating the user's computing device from external influences.

AIDT150may be a representation of software, hardware, circuitry, and/or middleware configured to implement features of the AIDT150. For example, the AIDT150may be a web-based application operating, for example, according to a .NET framework within the application platform system100.

More specifically, the AIDT150may include a desktop recorder circuitry151, an AIDT analysis report program (i.e., AIDT analyzer) circuitry152, and a workflow diagram generator circuitry153. Each of the desktop recorder circuitry151, the AIDT analysis report program circuitry152, and the workflow diagram generator circuitry153may be a representation of software, hardware, circuitry, and/or middleware configured to implement respective features of the AIDT150.

Desktop recorder circuitry151may be configured to include desktop recording controls enabling a user (e.g., authorized user) to configure and control certain features of AIDT150. Desktop recorder circuitry151may further be configured to include security controls for authenticating users to access, use, and/or configure features of the AIDT150. Desktop recorder circuitry151may further be configured to include log-in controls that control a log-in process enabling the communication device110to log-in and access the AIDT150running on application server140.

To access the AIDT150, a user may control the communication device110to request access to the application server140hosting the AIDT150. The access request may be transmitted to the application server140through the network130, via the communication interface112. The access request signal may be received by the application server140via the communication interface142, and further received by the desktop recorder circuitry151. Desktop recorder circuitry151may then control a log-in protocol by receiving and authenticating log-in credentials included in the access request signal (e.g., user name, password, authentication of the communication device110, etc.). According to some embodiments, the desktop recorder circuitry151may also control a security protocol by determining the user's access level based on the log-in credentials included in the access request signal. According to some embodiments, the desktop recorder circuitry151may grant authenticated user's identified as having certain predetermined access levels the capability to configure features and/or attributes of the AIDT150. After successfully passing the log-in protocol and/or security protocol, a user may be allowed to operate the AIDT150from the user's communication device110.

The desktop recorder circuitry151may be embodied by a desktop recorder GUI300as illustrated inFIG. 3. The desktop recorder GUI300may be displayed on the screen of the communication device as the user is manually interacting with a running program (e.g., MS Excel, MS Word). The desktop recorder GUI300includes a settings option301, a record button302for commencing recording of the user's action on the communication device110, a step counter303that displays a number of user action steps recorded by the desktop recorder circuitry151, a pause recording button304for pausing the current recording session, a time elapsed indicator305for displaying a length of time elapsed for the current recording session, a stop button306for stopping recording of the current recording session, a transaction display307for identifying the current recording session, and a close button308for closing out and ending the AIDT150running on the communication device110.

By activating the settings option301, the desktop recorder GUI300initiates display of a settings menu309. The settings menu309includes an “open screenshot reference” option, a “generate workflow” option, a “global productivity hub (GPH) analysis page” option, a “turn ON/OFF screen capture” option, an “image quality” option, and an “about” option for displaying information on the AIDT150.

Referring back toFIG. 1, the AIDT analysis report program circuitry152may be responsible for executing an analysis of recording session data obtained by the desktop recorder circuitry151. The results of the analysis may be presented by a first AIDT analysis report GUI800illustrated inFIG. 8, or a second AIDT analysis report GUI900illustrated inFIG. 9. The AIDT analysis report program circuitry152may be caused to generate the first AIDT analysis report GUI800or second AIDT analysis report GUI900in response to a selection of the “open screenshot reference” option in the settings menu309.

The first AIDT analysis report GUI800includes an option to select a date of the recording session and a transaction name of the recording session that will be analyzed. The first AIDT analysis report GUI800also includes a step analysis section801that displays a total number of steps in the recording session, a total time of the recording session, and a data graph that represents steps from the recording session that has been determined to be automatable, steps that are determined to be non-automatable, and steps that are potentially automatable and require further analysis. The first AIDT analysis report GUI800also includes a timeline section803that generates a visual color-coded timeline of the recording session that locates steps from the recording session that has been determined to be automatable, steps that are determined to be non-automatable, and steps that are potentially automatable and require further analysis. The first AIDT analysis report GUI800also includes a transaction details section804that lists and describes each recorded step from the recording session. The first AIDT analysis report GUI800also includes a pdf generator option805, where selection of the pdf generator option805will cause the AIDT analysis report program circuitry152to generate a pdf document including the information presented in the first AIDT analysis report GUI800. The first AIDT analysis report GUI800also includes an application summary section802.

The second AIDT analysis report GUI900is an exemplary GUI according to different embodiments of the AIDT150. The information presented by the second AIDT analysis report GUI900is similar to the information presented by the first AIDT analysis report GUI900. However, in the second AIDT analysis report GUI900the recording session analysis information is split between a summary tab and a detailed tab. The recording session analysis information presented by selection of the summary tab in the second AIDT analysis report GUI900is analogous to the transaction details described by the transaction details section804in the first AIDT analysis report GUI800. For example, detailed transaction GUI1000illustrates the second AIDT analysis report GUI900when the detailed tab is selected.

The AIDT analysis report program circuitry152may further analyze the recording session to identify processes that may be optimized by reducing a number of redundant steps. For example, the AIDT analysis report program circuitry152may compare multiple executions of a process and identify repetitive steps, branching, and decision points, to determine steps within the process that may be optimized by eliminating redundant steps without detriment to the overall goal of the process.

Referring back toFIG. 1, the workflow diagram generator153is responsible for generating a workflow diagram that describes each step within the recording session. The workflow diagram may be generated in a text document, visual document, or other viewable electronic document format. An exemplary workflow diagram1100is illustrated inFIG. 11. The workflow diagram generator153may generate the workflow diagram1100in response to a selection of the “generate workflow” option in the settings menu309. In addition to the sequential workflow provided by the workflow diagram1100, the workflow diagram generator153may generate other workflow diagrams that include decision trees and/or loops to identify repetitive tasks that are captured during the recording session.

To generate the workflow diagram, the workflow diagram generator153further compares the steps described in the received recording session against previously received recording sessions and generated workflow diagrams. The historical data may relate to recording sessions that captured tasks included in the currently received recording session. The historical data may also relate, or not relate, to the same user recorded in the currently received recording session. The historical data may further be relied upon, in addition to the steps in the currently received recording session, to calculate an average handling time (AHT) for a task (e.g., inputting a set of invoices into an invoice chart in Excel, or calculating employee's overtime work amount from timesheets) described by the currently received recording session.

In addition or alternatively, the workflow diagram generator153generates automation code without specifically formatting the automation code into the workflow diagram. The automation code similarly is generated based on the historical data included in the recording session data from previous recording sessions. After identifying a specific process for automation, the workflow diagram generator153may first determine a recommended RPA tool for automating the specific process. Then, the workflow diagram generator153may generate the automation code in a data format known to be executable by the recommended RPA tool. After the automation code is generated by the workflow diagram generator153, the AIDT150transmits the automation code to the recommended RPA tool for automation of the process described in the automation code.

Each of communication device110, database library120, and application server140may include one or more components of computer system200illustrated inFIG. 2.FIG. 2illustrates exemplary computer architecture for computer system200. Computer system200includes a network interface220that allows communication with other computers via a network226, where network226may be represented by network130inFIG. 1. Network226may be any suitable network and may support any appropriate protocol suitable for communication to computer system200. In an embodiment, network226may support wireless communications. In another embodiment, network226may support hard-wired communications, such as a telephone line or cable. In another embodiment, network226may support the Ethernet IEEE (Institute of Electrical and Electronics Engineers) 802.3x specification. In another embodiment, network226may be the Internet and may support IP (Internet Protocol). In another embodiment, network226may be a LAN or a WAN. In another embodiment, network226may be a hotspot service provider network. In another embodiment, network226may be an intranet. In another embodiment, network226may be a GPRS (General Packet Radio Service) network. In another embodiment, network226may be any appropriate cellular data network or cell-based radio network technology. In another embodiment, network226may be an IEEE 802.11 wireless network. In still another embodiment, network226may be any suitable network or combination of networks. Although one network226is shown inFIG. 2, network226may be representative of any number of networks (of the same or different types) that may be utilized.

The computer system200may also include a processor202, a main memory204, a static memory206, an output device210(e.g., a display or speaker), an input device212, and a storage device216, communicating via a bus208.

Processor202represents a central processing unit of any type of architecture, such as a CISC (Complex Instruction Set Computing), RISC (Reduced Instruction Set Computing), VLIW (Very Long Instruction Word), or a hybrid architecture, although any appropriate processor may be used. Processor202executes instructions224stored on one or more of the main memory204, static memory206, or storage device215. Processor202may also include portions of the computer system200that control the operation of the entire computer system200. Processor202may also represent a controller that organizes data and program storage in memory and transfers data and other information between the various parts of the computer system200.

Processor202is configured to receive input data and/or user commands through input device212. Input device212may be a keyboard, mouse or other pointing device, trackball, scroll, button, touchpad, touch screen, keypad, microphone, speech recognition device, video recognition device, accelerometer, gyroscope, global positioning system (GPS) transceiver, or any other appropriate mechanism for the user to input data to computer system200and control operation of computer system200and/or operation of the AIDT150. Input device212as illustrated inFIG. 2may be representative of any number and type of input devices.

Processor202may also communicate with other computer systems via network226to receive instructions224, where processor202may control the storage of such instructions224into any one or more of the main memory204(e.g., random access memory (RAM)), static memory206(e.g., read only memory (ROM)), or the storage device216. Processor202may then read and execute instructions224from any one or more of the main memory204, static memory206, or storage device216. The instructions224may also be stored onto any one or more of the main memory204, static memory206, or storage device216through other sources. The instructions224may correspond to, for example, instructions that make up the AIDT150.

Although computer system200is represented inFIG. 2as a single processor202and a single bus208, the disclosed embodiments applies equally to computer systems that may have multiple processors and to computer systems that may have multiple busses with some or all performing different functions in different ways.

Storage device216represents one or more mechanisms for storing data. For example, storage device216may include a computer readable medium222such as read-only memory (ROM), RAM, non-volatile storage media, optical storage media, flash memory devices, and/or other machine-readable media. In other embodiments, any appropriate type of storage device may be used. Although only one storage device216is shown, multiple storage devices and multiple types of storage devices may be present. Further, although computer system200is drawn to contain the storage device216, it may be distributed across other computer systems that are in communication with computer system200, such as a server in communication with computer system200. For example, when computer system200is representative of communication device110, storage device216may be distributed across to application server140when communication device110is in communication with application server140during operation of the AIDT150.

Storage device216may include a controller (not shown) and a computer readable medium222having instructions224capable of being executed by processor202to carry out functions of the AIDT150. In another embodiment, some or all of the functions are carried out via hardware in lieu of a processor-based system. In one embodiment, the controller included in storage device216is a web application browser, but in other embodiments the controller may be a database system, a file system, an electronic mail system, a media manager, an image manager, or may include any other functions capable of accessing data items. Storage device216may also contain additional software and data (not shown), for implementing described features.

Output device210is configured to present information to the user. For example, output device210may be a display such as a liquid crystal display (LCD), a gas or plasma-based flat-panel display, or a traditional cathode-ray tube (CRT) display or other well-known type of display in the art of computer hardware. Accordingly in some embodiments, output device210displays a user interface. In other embodiments, output device210may be a speaker configured to output audible information to the user. In still other embodiments, any combination of output devices may be represented by the output device210.

Network interface220provides the computer system200with connectivity to the network226through any compatible communications protocol. Network interface220sends and/or receives data from the network226via a wireless or wired transceiver214. Transceiver214may be a cellular frequency, radio frequency (RF), infrared (IR) or any of a number of known wireless or wired transmission systems capable of communicating with network226or other computer device having some or all of the features of computer system200. Bus208may represent one or more busses, e.g., USB, PCI, ISA (Industry Standard Architecture), X-Bus, EISA (Extended Industry Standard Architecture), or any other appropriate bus and/or bridge (also called a bus controller). Network interface220as illustrated inFIG. 2may be representative of a single network interface card configured to communicate with one or more different data sources. For example, according to some embodiments the communications interface142(i.e., a communications network interface) and database interface143(i.e., a database network interface) may be separate network interface card hardware components dedicated to communicating with different data sources. In other embodiments, communications interface142and database interface143may be representative of a single network interface card hardware component configured to communicate with different data sources.

Computer system200may be implemented using any suitable hardware and/or software, such as a personal computer or other electronic computing device. In addition, computer system200may also be a portable computer, laptop, tablet or notebook computer, PDA, pocket computer, appliance, telephone, server computer device, or mainframe computer.

FIG. 13illustrates a flow diagram1300of logic describing an exemplary operation of the AIDT150after opening and running the AIDT150on communication device110. The desktop recorder GUI300may be displayed on the desktop screen of the communication device110.

An agent selects their account or client name from a list of available names, or input a new account or client name to associate the current recording session (1301). By receiving the agent identification information, the AIDT150identifies the agent for subsequent processing.

The agent may further input a recording session transaction name for the current recording session (1302). By receiving the recording session transaction name, the AIDT150may assign subsequently recorded information under the recording session transaction name.

The agent controls settings of the AIDT150by selecting the settings option301on the desktop recorder GUI300(1303). For example, by selecting the settings option301and further selecting the “recordings options,” the agent may customize recording options for the recording session. Recording options GUI1400illustrated inFIG. 14displays exemplary recording options that may be available including a screen capture on or off option, image quality selection option, and a mainframe function keys activation option. Recording options GUI1410displays the available customization under the mainframe function keys activation option in more detail.

The agent commences the recording session to record the agent's action with a program running on the communication device110, and also to capture desktop screenshots when this option is activated (1304). The recording session may be commenced by selecting the record button302on the desktop recorder GUI300. The screenshots captured during the recording session may be encrypted and stored at a data storage location (e.g., database library120). A cognitive bot running as part of the AIDT150may control the capture of screenshots to include only a surrounding area around the mouse click, instead of capturing the entire desktop, as described herein.

After a passage of time, the agent may select either the pause recording button304or the stop button306on the desktop recorder GUI300to cease recoding of the recording session (1305). Detecting selection of either the pause recording button304or the stop button306causes the AIDT150to cease recording the recording session.

The agent generates and views an AIDT analysis report on the recording session by, for example, selecting the global productivity hub (GPH) analysis page option or the equivalent from the settings menu309on the desktop recorder GUI300(1306). The AIDT analysis report for the recording session may be generated to embody the first AIDT analysis report GUI800or second AIDT analysis report GUI900described herein.

The agent may open screenshot references that have been captured and saved by selecting the open screenshot reference option or the equivalent from the settings menu309on the desktop recorder GUI300(1307). In response to selecting the open screenshot reference option, a screenshot selection GUI400may be displayed that includes an option to browse for the screenshot references from their storage location, and an option to decrypt the selected screenshot references and identify a storage location for the decrypted screenshot reference.FIG. 5illustrates a first exemplary screenshot500of the recording session, which includes the desktop recorder GUI300. The first exemplary screenshot500also includes an information section501that describes a step (e.g., step1) associated with the first exemplary screenshot500, a time and date the first exemplary screenshot500was captured, a description of the agent's action that caused the screenshot to be captured (e.g., left [mouse] click), and an x and y coordinate on the display screen that identifies the location of the action (e.g., left [mouse] click) that caused the screenshot to be captured.FIG. 6illustrates a second exemplary screenshot600of the recording session. The second exemplary screenshot600also includes an information section601that describes a step (e.g., step2) associated with the second exemplary screenshot600, a time and date the second exemplary screenshot600was captured, a description of the agent's action that caused the second exemplary screenshot600to be captured (e.g., left [mouse] click), the name of the file and program in which the agent's action was received (e.g., Stock price monitoring sheet—Excel), and an x and y coordinate on the display screen that identifies the location of the action (e.g., left [mouse] click) that caused the second exemplary screenshot600to be captured.FIG. 7illustrates a third exemplary screenshot700of the recording session. The third exemplary screenshot700also includes an information section701that describes a step (e.g., step3) associated with the third exemplary screenshot700, a time and date the third exemplary screenshot700was captured, a description of the agent's action that caused the third exemplary screenshot700to be captured (e.g., left [mouse] click), and an x and y coordinate on the display screen that identifies the location of the action (e.g., left [mouse] click) that caused the third exemplary screenshot700to be captured.

Referring back to the flow diagram1300, a workflow diagram may be generated by selection of the “generate workflow” option in the settings menu (1308).FIG. 11illustrates the exemplary workflow diagram1100that may be generated by the workflow diagram generator153of the AIDT150. The workflow diagram generator153may further create one or more of a process design document (PDD), a process design instruction (PDI), an object design instruction (ODI), a solution design instruction (SDI), or a solution design document (SDD). The workflow diagram may be a collection of visual data and corresponding descriptions that are embedded into the PDD and SDD.

The workflow diagram may then be transmitted to an RPA tool, where the RPA tool receives the workflow diagram, extracts information describing a process detected from the user's interactions with the application from the recording session, and generates an automated process for a process described in the workflow diagram (1309). The AIDT150may further suggest a particular RPA tool for automating the process described in the workflow diagram based on historical data describing the performance of available RPA tools in automating the process described in the workflow diagram. The AIDT150may select the available RPA tool with the best past performance in automating the process described in the workflow diagram.

FIG. 15illustrates a flow diagram1500of logic describing another exemplary operation of the AIDT150after opening and running the AIDT150on communication device110. According to the flow diagram1500, the AIDT150does not generate a workflow diagram, and an RPA tool may generate an automation process without receiving the workflow diagram as an input. The desktop recorder GUI300may be displayed on a desktop screen of the communication device110.

An agent may select their account or client name from a list of available names, or input a new account or client name to associate the current recording session (1501). By receiving the agent identification information, the AIDT150identifies the agent for subsequent processing.

The agent may further input a recording session transaction name for the current recording session (1502). By receiving the recording session transaction name, the AIDT150may assign subsequently recorded information under the recording session transaction name.

The agent controls settings of the AIDT150by selecting the settings option301on the desktop recorder GUI300(1503). For example, by selecting the settings option301and further selecting the “recordings options,” the agent may customize recording options for the recording session. Recording options GUI1400illustrated inFIG. 14displays exemplary recording options that may be available including a screen capture on or off option, image quality selection option, and a mainframe function keys activation option. Recording options GUI1410displays the available customization under the mainframe function keys activation option in more detail.

The agent commences the recording session to record the agent's action with a program running on the communication device110, and also to capture desktop screenshots when this option is activated (1504). The recording session may be commenced by selecting the record button302on the desktop recorder GUI300. The screenshots captured during the recording session may be encrypted and stored at a data storage location (e.g., database library120). A cognitive bot running as part of the AIDT150may control the capture of screenshots to include only a surrounding area around the mouse click, instead of capturing the entire desktop, as described herein.

After a passage of time, the agent may select either the pause recording button304or the stop button306on the desktop recorder GUI300to cease recoding of the recording session (1505). Detecting selection of either the pause recording button304or the stop button306causes the AIDT150to cease recording the recording session.

The agent generates and views an AIDT analysis report on the recording session by, for example, selecting the global productivity hub (GPH) analysis page option or the equivalent from the settings menu309on the desktop recorder GUI300(1506). The AIDT analysis report for the recording session may be generated to embody the first AIDT analysis report GUI800or second AIDT analysis report GUI900described herein.

The agent opens screenshot references that have been captured and saved by selecting the open screenshot reference option or the equivalent from the settings menu309on the desktop recorder GUI300(1307). In response to selecting the open screenshot reference option, a screenshot selection GUI400may be displayed that includes an option to browse for the screenshot references from their storage location, and an option to decrypt the selected screenshot references and identify a storage location for the decrypted screenshot reference.FIG. 5illustrates a first exemplary screenshot500of the recording session. The first exemplary screenshot500also includes an information section501that describes a step (e.g., step1) associated with the first exemplary screenshot500, a time and date the first exemplary screenshot500was captured, a description of the agent's action that caused the screenshot to be captured (e.g., left [mouse] click), and an x and y coordinate on the display screen that identifies the location of the action (e.g., left [mouse] click) that caused the screenshot to be captured.FIG. 6illustrates a second exemplary screenshot600of the recording session. The second exemplary screenshot600also includes an information section601that describes a step (e.g., step2) associated with the second exemplary screenshot600, a time and date the second exemplary screenshot600was captured, a description of the agent's action that caused the second exemplary screenshot600to be captured (e.g., left [mouse] click), the name of the file and program in which the agent's action was received (e.g., Stock price monitoring sheet—Excel), and an x and y coordinate on the display screen that identifies the location of the action (e.g., left [mouse] click) that caused the second exemplary screenshot600to be captured.FIG. 7illustrates a third exemplary screenshot700of the recording session. The third exemplary screenshot700also includes an information section701that describes a step (e.g., step3) associated with the third exemplary screenshot700, a time and date the third exemplary screenshot700was captured, a description of the agent's action that caused the third exemplary screenshot700to be captured (e.g., left [mouse] click), and an x and y coordinate on the display screen that identifies the location of the action (e.g., left [mouse] click) that caused the third exemplary screenshot700to be captured.

Referring back to the flow diagram1500, the AIDT150selects an appropriate RPA tool for automating a process detected from the recording session data, including the recording session and the information created to generate the AIDT analysis report (1508). The AIDT150may select a particular RPA tool from a plurality of known RPA tools for automating a specific process described in the recording session data based on the RPA tool's historical performance in automating the specific process. This way the AIDT150selects the available RPA tool with a proven history of high performance in automating the specific process.

After selecting the appropriate RPA tool, the AIDT150generates automation code for execution by the selected RPA tool (1509). The automation code includes instructions for automating the specific process from the record session data. The automation tool may be in an RPA code format that is readable and executable by the selected RPA tool. Following the generation of the automation code, the AIDT150may transmit the automation code to the selected RPA tool for execution and automation of the specific process.

With the added capability provided by the AIDT150of monitoring a user's desktop actions and recognizing automatable processes from the monitored desktop actions, additional enterprise features related to optimizing digital desktop procedures is possible. One such feature is the installation of a desktop publishing tool (DTP tool) onto a computer desktop, similar to the desktop recorder GUI300shown inFIG. 3. The DTP tool is displayed as a visual tool on the computer desktop to monitor the user's actions while the user is going through steps of a known enterprise process. As the DTP tool detects the steps of the known enterprise process, a visual indicator notifies the user of each step included in the known enterprise process, and further provides an alert to the user when a step is skipped by the user. According to some embodiments, the DTP tool may communicate with the AIDT150to automate the skipped step when the enterprise process has previously been automated by the AIDT150.

The DTP tool solves the problem of potential human error by effectively monitoring the user's desktop actions and alerting the user of any steps that are missed.

The DTP Tool

FIG. 16shows an exemplary system architecture for an application platform system600that includes component devices from the application platform system100that is further updated to include the DTP tool601working within the AIDT150. In particular, the memory144has been updated to store the DTP tool601. The DTP tool may be a representation of software, hardware, circuitry, and/or middleware configured to implement features of the DTP tool601. For example, the DTP tool601may be a web-based application operating, for example, according to a .NET framework within the application platform system600. The DTP tool601works together with the AIDT150to accomplish the monitoring, alerting, and automating features described herein. In particular, the DTP tool601is authorized to access recording session data and enterprise process descriptions stored on the database library120by the AIDT150.

FIG. 17shows a flow diagram1700of logic describing an exemplary process for detecting missing steps of an enterprise process implemented by the DTP tool601. The DTP tool601may be graphically represented by the DTP widget1910shown in the exemplary computer desktop1900inFIG. 19. The DTP widget1910is displayed on the computer desktop1900along with, for example, the desktop recorder GUI300. Reference to the computer desktop1900is made during the description of the flow diagram1700.

A user about to commence an enterprise process inputs a process identifier into, for example, either a search input1911or a process name input1912included in the DTP widget1910(1701). The process identifier is received by the DTP tool601through the process identifier input to the search input1911or process name input1912. The process identifier may be a name (e.g., paid time off update, payroll update, or other enterprise process described by a predetermined name), a numeric identification, or other identifier for identifying a predetermined enterprise process. When the user does not know the exact process identifier, search terms may be input to the search input1911. When the user knows the exact process identifier, the process identifier may be input directly into the process name input1912.

The DTP tool601receives the process identifier and/or search terms from the process name input1912and/or search input1911, respectively, and accesses the database library120to select a corresponding enterprise process description (1702). The enterprise process description selected from the database library120will include a description of the enterprise process and a list of steps known to be required for successfully accomplishing the enterprise process. After the enterprise process description is selected by the DTP tool601, the list of steps included in the enterprise process description may be pre-populated into the step tracker1914included in the DTP widget1910.

As the user then begins interacting with an application to commence the enterprise process, the DTP tool601monitors the user's desktop actions through the DTP widget1910(1703). The DTP tool601may control the desktop recorder GUI300to record the desktop actions.

The DTP tool601compares each monitored desktop action to the list of steps known to be included in the enterprise process description (1704). Each monitored desktop action that is determined by the DTP tool601to correspond to a step included in the enterprise process description based on this comparison is highlighted or otherwise displayed in the step tracker1914.

The DTP tool601considers whether a step included in the enterprise process description is skipped (1705). The DTP tool601may determine a step has been skipped when consecutive steps from the enterprise process description are not detected by the DTP tool601based on the monitoring of the desktop actions.

When a missing step is detected by the DTP tool601, an alert is presented to the user through the DTP widget1910(1706). The alert may be a visual alert highlighting the missing step in the step tracker1914, or other visual alert presented through the DTP widget1910. The step track1914shown inFIG. 19highlights step3as being skipped by displaying double stars “**” next to the process step description.

When a missing step is not detected, the completion status of the enterprise process is updated by displaying a percent completed information in a status display1913of the DTP widget1910(1707). The percent completed information may be calculated by the DTP tool601as a number of enterprise process steps detected from the user's desktop actions, divided by the total number of corresponding steps included in the enterprise process description for completion of the enterprise process.

When additional steps of the enterprise process remain, the DTP tool601reverts back to monitoring the desktop actions (1708). When no further steps remain, the DTP tool601presents a completion report through the DTP widget1910(1709). The completion report may identify the missed steps (i.e., errors) by the user detected by the DTP tool601during the enterprise process.

FIG. 18shows a flow diagram1800of logic describing an exemplary process for detecting missing steps of an enterprise process implemented by the DTP tool601, that includes automation features provided by the AIDT150. The DTP tool601may be graphically represented by the DTP widget1910shown in the exemplary computer desktop1900inFIG. 19. The DTP widget1910is displayed on the computer desktop1900along with, for example, the desktop recorder GUI300. Reference to the computer desktop1900is made during the description of the flow diagram1700.

A user about to commence an enterprise process inputs a process identifier into, for example, either a search input1911or a process name input1912included in the DTP widget1910(1801). The process identifier is received by the DTP tool601through the process identifier input to the search input1911or process name input1912. The process identifier may be a name (e.g., paid time off update, payroll update, or other enterprise process described by a predetermined name), a numeric identification, or other identifier for identifying a predetermined enterprise process. When the user does not know the exact process identifier, search terms may be input to the search input1911. When the user knows the exact process identifier, the process identifier may be input directly into the process name input1912.

The DTP tool601receives the process identifier and/or search terms from the process name input1912and/or search input1911, respectively, and accesses the database library120to select a corresponding enterprise process description (1802). The enterprise process description selected from the database library120will include a description of the enterprise process and a list of steps known to be required for successfully accomplishing the enterprise process. After the enterprise process description is selected by the DTP tool601, the list of steps included in the enterprise process description may be pre-populated into the step tracker1914included in the DTP widget1910.

As the user then begins interacting with an application to commence the enterprise process, the DTP tool601monitors the user's desktop actions through the DTP widget1910(1803). The DTP tool601may control the desktop recorder GUI300to record the desktop actions.

The DTP tool601compares each monitored desktop action to the list of steps known to be included in the enterprise process description (1804). Each monitored desktop action that is determined by the DTP tool601to correspond to a step included in the enterprise process description based on this comparison is highlighted or otherwise displayed in the step tracker1914.

The DTP tool601considers whether a step included in the enterprise process description is skipped (1805). The DTP tool601may determine a step has been skipped when consecutive steps from the enterprise process description are not detected by the DTP tool601based on the monitoring of the desktop actions.

When a missing step is detected by the DTP tool601, an alert is presented to the user through the DTP widget1910(1806). In addition or alternatively, the DTP tool601transmits an automation request to automatically implement the missing step to the AIDT150. When the AIDT150has previously automated the enterprise process currently being monitored by the DTP tool601, the AIDT150has access to the automation code for automating the steps of the enterprise process, including the missing step. Thus when the AIDT150receives the automation request and the missing step requested in the automation request has previously been automated by the AIDT150, the AIDT150may automatically implement the missing step. This saves the user from having to go back and implement the missing step manually.

When a missing step is not detected, the completion status of the enterprise process is updated by displaying a percent completed information in a status display1913of the DTP widget1910(1807). The percent completed information may be calculated by the DTP tool as a number of enterprise process steps detected from the user's desktop actions, divided by the total number of corresponding steps included in the enterprise process description for completion of the enterprise process.

When additional steps of the enterprise process remain, the DTP tool reverts back to monitoring the desktop actions (1808). When no further steps remain, the DTP tool presents a completion report through the DTP widget1910(1809). The completion report may identify the missed steps (i.e., errors) by the user detected by the DTP tool during the enterprise process.

The implementations may be distributed. For instance, the circuitry may include multiple distinct system components, such as multiple processors and memories, and may span multiple distributed processing systems. Parameters, databases, and other data structures may be separately stored and managed, may be incorporated into a single memory or database, may be logically and physically organized in many different ways, and may be implemented in many different ways. Example implementations include linked lists, program variables, hash tables, arrays, records (e.g., database records), objects, and implicit storage mechanisms. Instructions may form parts (e.g., subroutines or other code sections) of a single program, may form multiple separate programs, may be distributed across multiple memories and processors, and may be implemented in many different ways. Example implementations include stand-alone programs, and as part of a library, such as a shared library like a Dynamic Link Library (DLL). The library, for example, may contain shared data and one or more shared programs that include instructions that perform any of the processing described above or illustrated in the drawings, when executed by the circuitry.