Patent ID: 12217030

The features and advantages of aspects will become more apparent from the detailed description set forth below when taken in conjunction with the drawings, in which like reference characters identify corresponding elements throughout. In the drawings, like reference numbers generally indicate identical, functionally similar, and/or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference number.

DETAILED DESCRIPTION

I. Introduction

The following detailed description discloses numerous aspects. The scope of the present patent application is not limited to the disclosed aspects, but also encompasses combinations of the disclosed aspects, as well as various modifications to the disclosed aspects.

References in the specification to “one aspect,” “an aspect,” “an example aspect,” “one embodiment,” “an embodiment,” “an example embodiment,” and/or the like, indicate that the aspect/embodiment described may include a particular feature, structure, or characteristic, but every aspect/embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same aspect/embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an aspect/embodiment, it is submitted that it is within the knowledge of one skilled in the relevant art(s) to affect such feature, structure, or characteristic in connection with other aspects/embodiments whether or not explicitly described.

In the discussion, unless otherwise stated, adjectives such as “substantially,” “approximately,” and “about” modifying a condition or relationship characteristic of a feature or features of an aspect of the disclosure, are understood to mean that the condition or characteristic is defined to be within tolerances that are acceptable for operation of the aspect for an application for which it is intended.

If the performance of an operation is described herein as being “based on” one or more factors, it is to be understood that the performance of the operation may be based solely on such factor(s) or may be based on such factor(s) along with one or more additional factors. Thus, as used herein, the term “based on” should be understood to be equivalent to the term “based at least on.”

Furthermore, it should be understood that spatial descriptions (e.g., “above,” “below,” “up,” “left,” “right,” “down,” “top,” “bottom,” “vertical,” “horizontal,” etc.) used herein are for purposes of illustration only, and that practical implementations of the structures and drawings described herein can be spatially arranged in any orientation or manner. Additionally, the drawings may not be provided to scale, and orientations or organization of elements of the drawings may vary in aspects.

Numerous exemplary aspects are described as follows. It is noted that any section/subsection headings provided herein are not intended to be limiting. Aspects are described throughout this document, and any type of aspect may be included under any section/subsection. Furthermore, aspects disclosed in any section/subsection may be combined with any other aspects described in the same section/subsection and/or a different section/subsection in any manner.

Section II below describes example aspects for generating automations via natural language processing. Section III below describes example computing devices that may be used to implement features of the aspects described herein. Section IV below describes additional examples and advantages, and finally, Section V provides some concluding remarks.

II. Example Aspects for Generating Automations Via Natural Language Processing

Methods for generating automations via natural language processing are performed by computing systems as described herein for various aspects. For example, a system receives natural language input from a user interface, and generates a workflow based at least on the natural language input, where the workflow includes one or more steps to build an automation (where the automation may be considered a workflow, or “flow” herein for distinction, during its own operation). The system provides at least one step of the one or more steps to the user interface, and populates a first field and a second field in the user interface that each correspond to the at least one step, where the first field is populated with a parameter based at least on the natural language input, and the second field is populated based at least on the parameter, thereby improving the user interface and its ease of use for the user. The system then builds the automation and/or deploys the automation for use.

Existing solutions require a significant learning curve for new users to get to a point where they can build automation flows. Accordingly, such users may inadvertently introduce programming errors, improper parameters, bugs, etc., into the automations and/or application, which can affect the computing device or system on which such automations and/or applications execute. For example, such issues may cause the device or system to incur an unnecessary expenditure of compute resource(s) (e.g., processing cycles, memory, storage, power, etc.), may require additional expenditure for time spent programming, or even cause a device or system to crash. The techniques described herein prevent such disastrous scenarios, as the generation of automations via natural language processing by systems herein automatically learns and/or generates the automations on behalf of the user from natural language inputs, and thus avoids errors, delays, and other inefficiencies with which the learning curve for new users is rife.

Aspects herein provide a low-code automation platform that enables users who do not have development skills to build automation processes. These automations may be referred to as flows and are based on a workflow of steps from which the automations are generated/created. That is, a low-code automation building experience where users can describe in plain language what they want to automate and have a flow automation created is provided herein. Users are enabled to speak or type in natural language what they want to automate, and aspects described herein generate for them the automation flow that performs it. In some aspects, by way of example and not limitation, a GPT-3 model that has been fined-tuned is configured to take the natural language input description provided by the user, and output an automation flow.

Described aspects are applicable to any type of server/host implementation, including but not limited to cloud-based platforms. Aspects may be incorporated into, or be a part of, software applications/services for generating automations, flows, bots, etc., such as Microsoft® Power Automate™ from Microsoft Corporation of Redmond, WA. Additionally, aspects are described herein by way of example and for purposes of illustration, however, other implementations and architectures are also adaptable to, and contemplated for, the described aspects and are contemplated herein.

These and other aspects will be described in further detail below, in association with the Figures, and in Sections/Subsections that follow.

Systems, devices, and apparatuses herein may be configured in various ways for generating automations via natural language processing. For instance,FIG.1will now be described.FIG.1is a block diagram of a workflow development system100(also “system100”), according to an example aspect. As shown inFIG.1, system100includes a computing device102, storage104, a first network-based application124A, a second network-based application124B, and a server134. Server134includes a workflow generator106and an automation library118that stores steps for automations, flows, bots, etc. (e.g., in a storage). Workflow generator106includes an UI generator110and a workflow logic generator112. Computing device102includes a display screen108and a browser136. Storage104stores a local application122. System100is described as follows.

Computing device102may be any type of stationary or mobile computing device, including a mobile computer or mobile computing, a wearable computing device, or a stationary computing device such as a desktop computer or PC (personal computer). Server134may include one or more server devices and/or other computing devices.

Local application122in storage104is an example of an application accessible by computing device102without communicating over a network. Local application122may be configured to perform data processing and/or data hosting operations when executed by a processor of computing device102, and may provide data132to flows/automations created by workflow generator106during runtime of those flows/automations. Local application122may be any type of local application/service, such as but not limited to, a database application (e.g., QuickBooks®, a Microsoft® Excel® spreadsheet), a messaging application (e.g., Microsoft® Outlook®), a productivity application (e.g., Microsoft® Word®, Microsoft® PowerPoint®, etc.), or another type of application/service. AlthoughFIG.1shows a single local application, any number of local applications may be present at computing device102, including numbers in the tens, hundreds, or even greater numbers of local applications.

First and second network-based applications124A and124B are examples of network-based applications, also referred to as “cloud” applications or services. Network-based applications124A and124B are accessible by computing device102over network126, may be configured to perform data processing and/or data hosting operations, and may provide data130A and130B, respectively, to flows/automations created by workflow generator106during runtime thereof. Network-based applications124A and124B may each be any type of web accessible applications/services, such as database applications, collaborative document management and storage system applications, social networking applications, messaging applications, financial services applications, news applications, search applications, web-accessible productivity applications, cloud storage and/file hosting applications, etc. Examples of such applications/services include, without limitation, a web-accessible SQL (structured query language) database, Salesforce.com™, Facebook®, Twitter®, Instagram®, Yammer®, LinkedIn®, Yahoo!® Finance, The New York Times® (at www.nytimes.com), Google search, Microsoft® Bing, Google Docs™, Microsoft® Office365and/or services/applications thereof, Dropbox™, Microsoft SharePoint™, etc. AlthoughFIG.1shows two network-based applications/services, any number of network-based applications/services may be accessible over network126, including numbers in the tens, hundreds, thousands, or even greater numbers of network-based applications/services. For simplicity and brevity, “applications” hereinafter also includes “services,” as would be understood by persons of skill in the relevant art(s) having the benefit of this disclosure.

Note that data128, data130A, data130B, and data132may each include any type of data, including messages, notifications, calculated data, retrieved data, files, attachments, and/or any other type of information requested or usable by a flow or automation.

Computing device102and server134may each include at least one network interface that enables communications with each other and with network-based applications124A and124B over network126. Examples of such a network interface, wired or wireless, include an IEEE 802.11 wireless LAN (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth™ interface, a near field communication (NFC) interface, etc. Further examples of network interfaces are described elsewhere herein. Examples of network126include a local area network (LAN), a wide area network (WAN), a personal area network (PAN), and/or a combination of communication networks, such as the Internet.

Workflow generator106is configured to be operated/interacted with to create automations/flows by generating workflows thereof comprising steps as described herein. For instance, a developer may access workflow generator106by interacting with an application at computing device102capable of accessing a network-based application, such as browser136. The developer may use browser136to traverse a network address (e.g., a uniform resource locator) to workflow generator106, which invokes a workflow generator GUI116(e.g., as a web page) in a browser window114. The developer is enabled to interact via natural language with workflow generator GUI116to generate an automation workflow from which an automation, flow, bot, etc., is created. An example of workflow generator106includes, but is not limited to, Microsoft® Power Automate™. In some aspects, workflow generator106is configured to perform one or more of receive natural language input from a user interface, generate an automation workflow based at least on the natural language input, the automation workflow including one or more steps to build an automation, provide at least one step of the one or more steps to the user interface, populate a first field and a second field in the user interface that each correspond to the at least one step, the first field being populated with a parameter based at least on the natural language input, and the second field being populated based at least on the parameter, and build the automation and/or deploy the automation.

As shown inFIG.1, workflow generator106includes UI generator110and workflow logic generator112. UI generator110is configured to transmit workflow GUI information140(A) (e.g., one or more web pages, image content, etc.) to browser136to be displayed as workflow generator GUI116in display screen108in browser window114. Workflow generator GUI116may be interacted with by the developer via natural language inputs (voice, text, etc.) to proactively/automatically generate automation workflows, comprising steps, into a flow, automation, etc., as illustrated in the FIGS. described below. It should also be noted that a GUI herein may also be described as a UI, generally, and that both graphical and non-graphical UIs are contemplated for aspects herein as would be understood by persons of skill in the relevant art(s) having the benefit of this disclosure.

For example, a user maybe presented with one or more automation workflow steps, in workflow generator GUI116, associated with a local or network-based application. In one example scenario, consider that a user enters natural language input of “I want to store email attachments to OneDrive” in workflow generator GUI116. One result of this input is that workflow generator GUI116displays the one or more steps to generation the automation for storing the user's email attachments to OneDrive® (see exemplary description of FIGS. below).

Browser136is configured to provide natural language inputs140(B) from workflow generator GUI116to UI generator110at server134, which workflow generator106utilizes to generate an automation(s). As part of this process, a user may be provided automation steps120for the determined automation that are stored in automation library118via workflow generator GUI116. Such steps may include parameters138(A) and/or configuration values that are automatically determined and populated by workflow generator106as well as parameters and/or configuration values that the user may provide via workflow generator GUI116.

Browser136stores the automation workflow steps and parameters/parameter values, corresponding configuration information, and/or workflow step sequence information, etc., as constructed workflow information138(B). Constructed workflow information138(B) is transmitted to workflow logic generator112at server134. Workflow logic generator112generates an automation based at least on the assembled workflow steps and workflow information138(B). The flow represented by the automation may subsequently be invoked or deployed at runtime by an end user, or may be automatically invoked based on a condition of the flow/automation being met subsequent to deployment (e.g., a new email with an attachment is received).

During runtime of the automation/flow, the automation may invoke operation of one or more local or network-based applications associated with the automation/flow steps120of the automation. Each flow step120may receive input data from or transmit output data to the one or more local or network-based applications. Such input or output data may include, for example, input data132received from or sent to local application122, data130A received from or sent to network-based application124, data130B received from or sent to network-based application124B, etc.

Workflow generator106may operate and/or be configured in various ways, to enable development of a flow or an automation herein. For instance, in aspects, workflow generator106may be configured as in system200ofFIG.2and/or operate in accordance with flowchart300ofFIG.3.

Referring now toFIG.2, a block diagram of a system200is shown for generating automations via natural language processing, according to an example aspect. System200is configured to be an aspect of system100ofFIG.1, e.g., server134and workflow generator106. Like- or similarly-named components of system100illustrated in system200may be aspects thereof. System200is described as follows.

System200includes a computing system202, which is an aspect of server134ofFIG.1, in aspects, and which is any type of server or computing system, as mentioned elsewhere herein, or as otherwise known, including one or more cloud-based servers that support multi-tenancies. As shown inFIG.2, computing system202includes one or more processors (“processor”)204, one or more of a memory and/or other physical storage device (“memory”)206, as well as one or more network interfaces (“network interface”)226. Computing system202includes one or more automation libraries (“automation library”)224which may be an aspect of automation library118ofFIG.1, and a workflow generator208that is an aspect of workflow generator106ofFIG.1. Workflow generator208is configured to perform aspects of generating automations via natural language processing, as described herein, including but without limitation, those described above for workflow generator106ofFIG.1, and/or the like. In aspects, while not shown for brevity and illustrative clarity, workflow generator208comprises a portion of a server or web application/service. Computing system202also includes one or more automations (“automation(s)”)220that are generated by workflow generator208based on natural language inputs, and one or more natural language (NL) model(s)222that are used in the generation of one or more automations220, as described herein. In aspects, automation(s)220, NL model(s)222, and/or automation library224may be stored in memory206.

System200also includes additional components (not shown for brevity and illustrative clarity) including, but not limited to, components and subcomponents of other devices and/or systems herein, as well as those described below with respect toFIGS.19-20, such as an operating system, according to aspects.

Processor204and memory206may respectively be any type of processor circuit(s) and memory that is described herein, and/or as would be understood by a person of skill in the relevant art(s) having the benefit of this disclosure. Processor204and memory206may each respectively comprise one or more processors or memories, different types of processors or memories, remote processors or memories, and/or distributed processors or memories. Processor204may be multi-core processors configured to execute more than one processing thread concurrently, and cores of processor204may be grouped with portions of memory206as processing resources allocated for hosting services and/or for generating automations via natural language processing. Processor204may comprise circuitry that is configured to execute computer program instructions such as, but not limited to, aspects of workflow generator208, which may be implemented as computer program instructions, as described herein.

Memory206may include volatile storage portions such as a random access memory (RAM) and/or persistent storage portions such as hard drives, non-volatile RAM, and/or the like, to store or be configured to store computer program instructions/code as described herein, as well as to store other information and data described in this disclosure including, without limitation, workflow manager208, including one or more of the components thereof as described herein, as well as automation(s)220, NL model(s)222, and/or automation library(ies)224, etc.

Network interface226may be any type or number of wired and/or wireless network adapter, modem, etc., configured to enable system200, including computing system202, to communicate with other devices and/or systems over a network, including cloud networks, such as communications between computing system202and other devices, systems, hosts, of system100inFIG.1over a network such as network126of which network interface226may connect.

In some aspects, UI generator210is an aspect of UI generator106ofFIG.1and is configured to perform similar or like operations/functions. Similarly, in some aspects, workflow logic generator212is an aspect of workflow logic generator112ofFIG.1and is configured to perform similar or like operations/functions.

NL model trainer218is configured to train NL model(s)222, which may be performed in conjunction with an intermediate coder214that is configured to provide a coding language, e.g., an intermediate language, with logical expressions that are absent from the standard workflow definitional language utilized by a standard coder216. In some aspects, steps for automation stored in automation library224are, in whole or in part, in the standard workflow definitional language utilized by a standard coder216. Intermediate coder214and standard coder216are each configured to convert between these languages for training of NL model(s)222(e.g., from the standard workflow definitional language to the intermediate language) and for operation/use thereof (e.g., from the intermediate language to the standard workflow definitional language). In other words, the NL models of NL model(s)222enable the generation of workflows for automations, as described herein. These workflows are characterized, in aspects, as “programs” that describe a set of steps, actions, operations, application programming interfaces (APIs), and/or the like, to call to automate a user's workflow. Further details are described below.

Workflow logic generator212is configured to utilize NL model(s)222to generate flows, automations, bots, and/or the like, based at least on NL inputs provided by a user via a UI/GUI (e.g., as described above forFIG.1and in the FIGS. described below), as exemplarily described below forFIG.3.

As noted above forFIGS.1and2, aspects herein provide for generating automations via natural language processing. System100ofFIG.1and/or system200ofFIG.2may be configured to perform such functions and operations. It is further contemplated that the systems and components described above are configurable to be combined in any way. Further details regarding the operations and configurations of system200, and its various sub-components, are provided in the aspects herein, including those shown in the instant FIGS., discussed in their associated descriptions.

Referring also no toFIG.3, in which a flowchart300is shown. Flowchart300depicts a process for generating automations via natural language processing, according to an example aspect. Further structural and operational examples will be apparent to persons skilled in the relevant art(s) based at least on the following descriptions. Flowchart300is described as follows with respect to system100ofFIG.1and/or system200ofFIG.2, as described above.

Flowchart300begins at step302. In step302, natural language input is received from a user interface. For example, workflow generator208and/or UI generator210of system200receive NL inputs from workflow generator GUI116via browser136, as similarly described above and described in further detail below.

In step304, an automation is designed based on the natural language input. In step304, an automation workflow may be generated based at least on the natural language input to design the automation. The automation workflow includes one or more steps to design and build an automation that includes one or more operations. Steps to design and build automations are stored in automation library224, and are determined by workflow logic generator212based at least on the NL inputs (e.g., by receiving the natural language input at the machine learning NL model, which may be utilized by workflow logic generator212, to generate an output according to the standard workflow definitional language). Accordingly, the steps to design and/or generate the automation for an appropriate workflow are determined.

In step306, a user interface is populated with parameters for operations of the automation, the parameters derived from the natural language input. In aspects herein, a step(s) of the workflow is provided to the user interface. For instance, and as described herein, a step(s) of the workflow is/are displayed to a user via the GUIs described herein, e.g., inFIG.5described below. As noted herein, workflow generator208is configured to provide the steps to a user device, e.g., a browser or the like, for display to the user. Steps may be displayed together in a single screen, or separately in multiple screens. The steps correspond to the automation operations desired by the user, and represent the automation workflow generally. For example, and as described, some input fields for a given step are automatically populated (also “pre-populated” herein) with a parameter value(s) based on the NL input (e.g., in the example scenario “I want to store email attachments to OneDrive,” the first step may have auto-populated fields for an email application “inbox”), while some other fields for the same or a different step are populated based on value “inbox,” as well as here on the NL input “attachments” (e.g., “attachments” may be auto-populated, along with “attachment name” and “attachment content” parameters). In some aspects, a first field and a second field are populated in the user interface that each correspond to a step or to steps of the workflow and to operations of the automation, where the first field is populated with a first parameter value based at least on the natural language input, and the second field is populated with a second parameter value based at least on the parameter value. These parameters thus correspond to operations of the automation that is designed, as the operations are represented in the steps of the workflow shown in the GUI.

In step308, building of the automation based on the parameters is controlled in response to an approval received via the user interface. For instance, when the workflow automation is saved, e.g., by a user via the GUI, a corresponding automation is built from design and parameters entered via the workflow population and/or the user input population. That is, the designed automations are built, e.g., as a final step, based at least on automatically populated parameters and/or final user inputs, e.g., as described below, and when saved are stored, e.g., by memory206, as automation(s)220, and are associated with the user via the user's accounts/identity. Ones of automation(s)220are stored locally to a user in embodiments, and also may be stored regionally or globally in the cloud and/or an on-premise server for deployment.

In step310, the automation is deployed subsequent to the automation being built. In aspects, saving of the automation after building in step308may cause the system to deploy the automation for the user, or in some aspects, the user may manually deploy the automation via interaction with the GUI.

The example aspects provided above will be described in additional exemplary detail below with reference toFIGS.4-9.

FIG.4shows a diagram of a graphical user interface (GUI)400for designing and generating automations via natural language processing, in an example aspect. GUI400may be a further aspect of those described above. GUI400is generated by UI generator210of system200inFIG.2, in some aspects. GUI400includes, inter alia, a workflow automation begin control402and a natural language input field404. In some aspects, UI generator210generates and provides GUI400based on a user selection of a NL workflow automation element of a UI presented to the user by UI generator210or by another webpage.

Workflow automation begin control402may be any type of UI control, e.g., a button, a checkbox, a link, etc., which may be activated by a user's selection thereof. Selection of workflow automation begin control402causes a step-by-step walkthrough for generating a suggested or recommended workflow automation, in one aspect, or causes a generic a step-by-step walkthrough for generating a workflow automation to be customized by a user. In some aspects, a separate workflow automation begin control402-1is included in GUI400to generate a workflow automation based on the user's natural language input, as described herein.

Natural language input field404is configured to receive natural language inputs from a user, e.g., via keyboard, voice, stylus-related actions, and/or the like, which specify, in natural language, a type of automation the user desires to generate. As shown in GUI400, by way of illustrative example, a user input406of “I want to store email attachments to OneDrive” is entered in natural language input field404.

Natural language input field404is pre-populated with an example suggestion for an automated workflow in some aspects, for instance, “store responses to forms in a spreadsheet,” as illustrated. In some aspects, user activity with one or more services and/or applications associated with the user's account or credentials may be utilized to recommend the example suggestion in natural language input field404, while in other aspects, popular workflow automations used by a number of other users may be suggested in natural language input field404.

Inputs entered and/or activations made, and/or indicia thereof, for workflow automation begin control402and/or natural language input field404are provided from GUI400to workflow generator208and/or UI generator210of system200inFIG.2, as noted in step302of flowchart300inFIG.3, for automation design/generation via a workflow according to the inputs and/or activations, as described for step302and step304.

FIG.5shows a diagram of a user interface (UI)500for designing and generating automations via natural language processing, in an example aspect. GUI500may be a further aspect of those described above. GUI500is generated by UI generator210of system200inFIG.2, in some aspects. GUI500includes, inter alia, a display of the natural language from user input406, as well as steps (also “actions” or “operations”) of the workflow automation and input fields therefor, as described below. Such steps are stored in automation library224, in some aspects as noted herein.

For example, GUI500includes a workflow step display element502having an input field504associated therewith. In aspects herein, step display element502corresponds to user input406(“I want to store email attachments to OneDrive”), specifically to the “email” portion, and input field504is auto-/pre-populated with an identifier that is determined based at least on user input406, and represents an operation in the designed automation. As illustratively shown by way of example, in furtherance of the scenario described above forFIG.4and GUI400, user input406indicates that the user desires to utilize a workflow to create an automation that automatically saves attachments received by email to a user's storage account, here OneDrive® from Microsoft Corp. of Redmond, WA. Accordingly, step display element502is labeled as “On new email” and input field504is pre-populated with “Inbox,” which may correspond to Outlook® from Microsoft Corp. of Redmond, WA, as the user may be authenticated currently to their Outlook® account. Additionally, user input406may specify “Outlook” as the email service/application in some scenarios. That is, as described above for step306in flowchart300ofFIG.3, a user interface is populated with parameters for operations of the automation, where the parameters are derived from the natural language input. In some aspects, a first field and a second field are populated in the user interface that each correspond to a workflow step(s) representing an operation of the designed automation, the first field being populated with a first parameter value based at least on the natural language input, and the second field being populated with a second parameter value based at least on the first parameter value.

As noted, in some aspects, this automatic population or pre-population is determined by applications and/or services associated with the user and their accounts, for which the user may be logged in via user credentials, or which may be identified by user credentials entered as part of the workflow automation generation process described herein. In a variation of the example above, user input406may be “store email attachments separately to my account,” and appropriate applications and services, e.g., Outlook® and OneDrive® are determined from the user's profile and/or accounts.

In either case of providing, selecting, and/or determining applications and/or services for workflows in automation designs and/or generations, GUI500may include controls such as links, buttons, etc., to allow the user to manually customize the applications and/or services (e.g., the “folder” icon of input field504, an “advanced options” control, as shown, etc.).

Also illustrated for GUI500is a step display element506, which is a second workflow step(s) representing an operation of the designed automation, and which also corresponds to user input406(“I want to store email attachments to OneDrive”), specifically to the “attachments” and “OneDrive” portions thereof. A step display element includes one or more sub-steps and/or one or more input fields, in some aspects. Step display element506includes a sub-step display element506-1for file creation, as relates to the example for user input406and its “attachments” and “OneDrive” portions. This also corresponds to the operation in the designed automation to perform this function. Step display element506includes a number of input fields, for example, as shown: an input field508for selecting an output from step display element502, an input field510for a file name option, and an input field512for a file content option. According to aspects, GUI500has input field508, input field510, and input field512each being automatically- or pre-populated, based on user input406and step display element502, with “Attachments,” Attachments Name(s),” and “Attachments Content,” respectively, as similarly noted above with respect to step308of flowchart300.

Also included in this example for GUI500is an input field514for a “folder path” that is not pre-populated in the instant example, although it is contemplated herein that a default folder path for the user, either specified by the user or determined by workflow generator208(inFIG.2) from the user's account, services, applications, etc. An input field herein that is not pre-populated may allow a user to enter input therein, may include an activator, icon, link, button, etc., exemplarily shown inFIG.5as a selectable folder icon, that enables a user to navigate to folders, files, etc., and/or may include a dropdown list element with options from which the user can make a selection. Additionally, in some aspects, a user is prompted by a prompt516for providing input, selections, etc., for input fields, which is displayed in GUI500.

Further user changes and customizations are also contemplated herein, as illustrated by the following non-limiting examples. For instance, step display elements may include a selectable control in their title bars or elsewhere, such as an ellipsis (‘ . . . ’), which enables users to alter the associated steps. Similarly, a pre-populated input field includes a selectable control, e.g., an ‘x’, that enables the user to remove the pre-populated selection determined from a prior step, in some aspects, and to select or provide their own input as described above for input field514. Aspects also provide for users to add additional steps beyond what is provided for user input406by selecting an add step element518.

FIG.6shows a diagram of a graphical user interface (GUI)600for generating automations via natural language processing, in an example aspect. GUI600may be a further aspect of those described above. GUI600is generated by UI generator210of system200inFIG.2, in some aspects. GUI600includes, inter alia, a display of the natural language from user input406and input field514, as well as the steps (or “actions” or “operations”) of the workflow automation and input fields noted above for GUI400and GUI500, which are not each illustrated for clarity and brevity of description.

In the illustrated aspect, GUI600is shown in furtherance of a user providing input for input field514, as described above. For example, GUI600includes an input602which designates the folder path for input field514as an “Email attachments” folder. In some aspects, responsive to the user entering/providing input602for input field514, a prompt604for the user is displayed in GUI600, which highlights attention to workflow automation options606. Workflow automation options606includes, in some aspects and by way of non-limiting example, selectable options for the user to add comments to the workflow automation, to save the workflow automation, and to test and/or check the workflow automation.

When the workflow automation is saved, a corresponding automation is built, saved, and/or deployed as described above in step308and step310of flowchart300inFIG.3. In embodiments, saving causes the automation to be built and/or deployed, in some aspects, and may include running the automation.

FIGS.7,8A, and8Bwill now be described for additional aspects of those described above.

FIG.7shows a diagram of a graphical user interface (GUI)700for generating automations via natural language processing, in an example aspect. GUI700may be a further aspect of those described above. GUI700is generated by UI generator210of system200inFIG.2, in some aspects. GUI700includes, inter alia, a display of the natural language from user input406and input field514, as well as an overview702of the steps of the workflow from the automation exemplarily noted above.

In the illustrated aspect, GUI700is shown in furtherance of a user completing entry of parameters, customizations, etc., and/or saving via workflow, as described above, or a designed automation. For example, subsequent to a user entering the folder path for input602of input field514in GUI600ofFIG.6, described above, and/or subsequent to the user saving the automation, overview702of the steps of the workflow from the automation is illustrated in GUI700for the user. Additionally, as illustrated by way of example, an account overview704is illustrated, in some aspects, which allows the user to verify the account for the applications and/or services in the automation as designed and represented in the workflow steps. Additionally, the user is enabled to configure permissions, accounts, etc., via controls of account overview704.

The user is thus enabled to select a continue control element706to proceed with the automation as summarized in GUI700for building and/or deployment.

FIG.8Ashows a diagram of a graphical user interface (GUI)800A for generating automations via natural language processing, in an example aspect. GUI800A is generated by UI generator210of system200inFIG.2, in some aspects. In the illustrated aspect, GUI800A is shown in furtherance of a user providing user input406in GUI400ofFIG.4.

GUI800A illustrates example aspects in which a user may not be authenticated to a service or application which is to be included in the automation and workflow generation thereof, and/or aspects in which a user is enabled to select/confirm the service or application. For example, GUI800A shows an input field802in which a user is prompted to enter an email account for the example user input406described above. Continuing with the example aspect above for user input406, the user may enter an email address for their Outlook® account into input field802.

FIG.8Bshows a diagram of a graphical user interface (GUI)800B for generating automations via natural language processing, in an example aspect. GUI800B is generated by UI generator210of system200inFIG.2, in some aspects, and GUI800B is in furtherance of the example aspect of GUI800A inFIG.8Adescribed above.

In the illustrated aspect, subsequent to the user entering an email address for their email account into input field802, a sign in element804is presented in GUI800B that enables the user to enter their account credentials, e.g., sign in, for the email account provided for input field802. When sign in element804is activated and valid credentials are entered for the user, e.g., via a sign in UI, the workflow for the automation proceeds to details and parameters for email attachments, as described herein.

InFIG.9, a flow diagram900for generating automations via natural language processing is shown, according to an example aspect. Flow diagram900includes a training flow902and an inference flow910, each for NL models as described herein.

As previously noted, the NL models of NL model(s)222inFIG.2enable the generation of workflows for designed automations, and these workflows are characterized, in aspects, as “programs” that describe a set of steps, actions, operations, application programming interfaces (APIs), and/or the like, to call to automate a user's workflow. The original workflow definition format may be, or be based on a definitional language such as, JSON in a tree-like format that is unlike any common programming languages and on which a NL model has been trained. In order to improve the performance of NL model inferences when generating workflows to design automations and to enable the workflow to determine auto-/pre-population parameters for operations of the automation, an intermediate language is provided that includes JSON format equivalents, conceptually, while also providing logical expressions that are absent from formats such as JSON. To enable such an implementation for training and performing inferences, combinations of the intermediate language and a user's NL prompt are provided as additional operations.

For example, training flow902includes a step904in which a user's NL prompt input is combined with the definitional workflow JSON. In step906, rather than training the NL model from step904, an intermediate language representation of the definitional workflow JSON is generated and included with the user's NL prompt input to enable logical operations in the generation of workflows to design automations based on the logical expressions in the intermediate language. For instance, as shown in the examples above, when a user's NL input includes an email program/service, e.g., Outlook®, and includes performing an action on items in the user's Inbox, the logical expressions of the intermediate language enable a determination of parameters, e.g., “Inbox,” such as: IF “Outlook”, THEN SELECT PARAMETER “Inbox” for received email. Similarly, in subsequent steps (e.g., step display element506inFIG.5), the logical expressions of the intermediate language enable the further determination of parameters, e.g., pre-populating email attachments based on the selection of “Inbox” and the NL input of the user. Accordingly, the intermediate language representation and the NL prompt are utilized to generate, via training, an NL Model in step908.

Likewise, inference flow910utilizes the intermediate language representation, in a reversed order, for generation of workflow to design automations. For instance, a user's NL input prompt is received in step912, and is processed based on the NL model in step914. Rather than determining the workflow directly from the output of the NL model in step914, however, an intermediate representation at step916is generated from the NL model output prior to converting this representation into the workflow JSON, with the user's NL input, in step918for workflow generation that will design an automation, as described herein.

III. Example Mobile and Computing Device Aspects

Aspects described herein may be implemented in hardware, or hardware combined with software and/or firmware. For example, aspects described herein may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer readable storage medium. Alternatively, aspects described herein may be implemented as hardware logic/electrical circuitry.

As noted herein, the aspects described, including but not limited to, system100inFIG.1, system200inFIG.2, and the GUIs ofFIGS.4-8B, along with any components and/or subcomponents thereof, as well any data structures, and operations and portions of flowcharts/flow diagrams described herein and/or further examples described herein, may be implemented in hardware, or hardware with any combination of software and/or firmware, including being implemented as computer program code configured to be executed in one or more processors and stored in a computer readable storage medium, or being implemented as hardware logic/electrical circuitry, such as being implemented together in a system-on-chip (SoC), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a trusted platform module (TPM), and/or the like. A SoC may include an integrated circuit chip that includes one or more of a processor (e.g., a microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and/or further circuits and/or embedded firmware to perform its functions.

Aspects described herein may be implemented in one or more computing devices similar to a mobile system and/or a computing device in stationary or mobile computer aspects, including one or more features of mobile systems and/or computing devices described herein, as well as alternative features. The descriptions of computing devices provided herein are provided for purposes of illustration, and are not intended to be limiting. Aspects may be implemented in further types of computer systems, as would be known to persons skilled in the relevant art(s).

FIG.10shows a block diagram of an exemplary mobile device1000including a variety of optional hardware and software components, shown generally as components1002. Any number and combination of the features/elements of components1002may be included in a mobile device aspect, as well as additional and/or alternative features/elements, as would be known to persons skilled in the relevant art(s). It is noted that any of components1002can communicate with any other of components1002, although not all connections are shown, for ease of illustration. Mobile device1000can be any of a variety of mobile devices described or mentioned elsewhere herein or otherwise known (e.g., cell phone, smartphone, handheld computer, Personal Digital Assistant (PDA), etc.) and can allow wireless two-way communications with one or more mobile devices over one or more communications networks1004, such as a cellular or satellite network, or with a local area or wide area network.

The illustrated mobile device1000can include a controller or processor referred to as processor circuit1010for performing such tasks as signal coding, image processing, data processing, input/output processing, power control, and/or other functions. Processor circuit1010is an electrical and/or optical circuit implemented in one or more physical hardware electrical circuit device elements and/or integrated circuit devices (semiconductor material chips or dies) as a central processing unit (CPU), a microcontroller, a microprocessor, and/or other physical hardware processor circuit. Processor circuit1010may execute program code stored in a computer readable medium, such as program code of one or more applications1014, operating system1012, any program code stored in memory1020, etc. Operating system1012can control the allocation and usage of the components1002and support for one or more application programs1014(a.k.a. applications, “apps”, etc.). Application programs1014can include common mobile computing applications (e.g., email applications, calendars, contact managers, web browsers, messaging applications) and any other computing applications (e.g., word processing applications, mapping applications, media player applications).

As illustrated, mobile device1000can include memory1020. Memory1020can include non-removable memory1022and/or removable memory1024. The non-removable memory1022can include RAM, ROM, flash memory, a hard disk, or other well-known memory storage technologies. The removable memory1024can include flash memory or a Subscriber Identity Module (SIM) card, which is well known in GSM communication systems, or other well-known memory storage technologies, such as “smart cards.” The memory1020can be used for storing data and/or code for running the operating system1012and the applications1014. Example data can include web pages, text, images, sound files, video data, or other data sets to be sent to and/or received from one or more network servers or other devices via one or more wired or wireless networks. Memory1020can be used to store a subscriber identifier, such as an International Mobile Subscriber Identity (IMSI), and an equipment identifier, such as an International Mobile Equipment Identifier (IMEI). Such identifiers can be transmitted to a network server to identify users and equipment.

A number of programs may be stored in memory1020. These programs include operating system1012, one or more application programs1014, and other program modules and program data. Examples of such application programs or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the systems described above, including the workflow development and execution systems described in reference toFIGS.1-18B.

Mobile device1000can support one or more input devices1030, such as a touch screen1032, microphone1034, camera1036, physical keyboard1038and/or trackball1040and one or more output devices1050, such as a speaker1052and a display1054.

Other possible output devices (not shown) can include piezoelectric or other haptic output devices. Some devices can serve more than one input/output function. For example, touch screen1032and display1054can be combined in a single input/output device. The input devices1030can include a Natural User Interface (NUI).

Wireless modem(s)1060can be coupled to antenna(s) (not shown) and can support two-way communications between processor circuit1010and external devices, as is well understood in the art. The modem(s)1060are shown generically and can include a cellular modem1066for communicating with the mobile communication network1004and/or other radio-based modems (e.g., Bluetooth1064and/or Wi-Fi1062). Cellular modem1066may be configured to enable phone calls (and optionally transmit data) according to any suitable communication standard or technology, such as GSM, 3G, 4G, 5G, etc. At least one of the wireless modem(s)1060is typically configured for communication with one or more cellular networks, such as a GSM network for data and voice communications within a single cellular network, between cellular networks, or between the mobile device and a public switched telephone network (PSTN).

Mobile device1000can further include at least one input/output port1080, a power supply1082, a satellite navigation system receiver1084, such as a Global Positioning System (GPS) receiver, an accelerometer1086, and/or a physical connector1090, which can be a USB port, IEEE 1394 (FireWire) port, and/or RS-232 port. The illustrated components1002are not required or all-inclusive, as any components can be not present and other components can be additionally present as would be recognized by one skilled in the art.

FIG.11depicts an exemplary implementation of a computing device1100in which aspects may be implemented. For example, aspects described herein may be implemented in one or more computing devices or systems similar to computing device1100, or multiple instances of computing device1100, in stationary or mobile computer aspects, including one or more features of computing device1100and/or alternative features. The description of computing device1100provided herein is provided for purposes of illustration, and is not intended to be limiting. Aspects may be implemented in further types of computer systems, servers, and/or clusters, etc., as would be known to persons skilled in the relevant art(s).

As shown inFIG.11, computing device1100includes one or more processors, referred to as processor circuit1102, a system memory1104, and a bus1106that couples various system components including system memory1104to processor circuit1102. Processor circuit1102is an electrical and/or optical circuit implemented in one or more physical hardware electrical circuit device elements and/or integrated circuit devices (semiconductor material chips or dies) as a central processing unit (CPU), a microcontroller, a microprocessor, and/or other physical hardware processor circuit. Processor circuit1102may execute program code stored in a computer readable medium, such as program code of operating system1130, application programs1132, other programs1134, etc. Bus1106represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. System memory1104includes read only memory (ROM)1108and random access memory (RAM)1110. A basic input/output system1112(BIOS) is stored in ROM1108.

Computing device1100also has one or more of the following drives: a hard disk drive1114for reading from and writing to a hard disk, a magnetic disk drive1116for reading from or writing to a removable magnetic disk1118, and an optical disk drive1120for reading from or writing to a removable optical disk1122such as a CD ROM, DVD ROM, or other optical media. Hard disk drive1114, magnetic disk drive1116, and optical disk drive1120are connected to bus1106by a hard disk drive interface1124, a magnetic disk drive interface1126, and an optical drive interface1128, respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer-readable instructions, data structures, program modules and other data for the computer. Although a hard disk, a removable magnetic disk and a removable optical disk are described, other types of hardware-based computer-readable storage media can be used to store data, such as flash memory cards, digital video disks, RAMs, ROMs, and other hardware storage media.

A number of program modules may be stored on the hard disk, magnetic disk, optical disk, ROM, or RAM. These programs include operating system1130, one or more application programs1132, other programs1134, and program data1136. Application programs1132or other programs1134may include, for example, computer program logic (e.g., computer program code or instructions) for implementing aspects described herein, such as but not limited to, system100inFIG.1, system200inFIG.2, and the GUIs ofFIGS.4-8B, along with any components and/or subcomponents thereof, as well any data structures, and operations of the flowcharts/flow diagrams described herein, including portions thereof, and/or further examples described herein.

A user may enter commands and information into the computing device1100through input devices such as keyboard1138and pointing device1140. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, a touch screen and/or touch pad, a voice recognition system to receive voice input, a gesture recognition system to receive gesture input, or the like. These and other input devices are often connected to processor circuit1102through a serial port interface1142that is coupled to bus1106, but may be connected by other interfaces, such as a parallel port, game port, or a universal serial bus (USB).

A display screen1144is also connected to bus1106via an interface, such as a video adapter1146. Display screen1144may be external to, or incorporated in computing device1100. Display screen1144may display information, as well as being a user interface for receiving user commands and/or other information (e.g., by touch, finger gestures, virtual keyboard, etc.). In addition to display screen1144, computing device1100may include other peripheral output devices (not shown) such as speakers and printers.

Computing device1100is connected to a network1148(e.g., the Internet) through an adaptor or network interface1150, a modem1152, or other means for establishing communications over the network. Modem1152, which may be internal or external, may be connected to bus1106via serial port interface1142, as shown inFIG.11, or may be connected to bus1106using another interface type, including a parallel interface.

A TPM may be connected to bus1106, and may be an aspect of any TPM, as would be understood by one of skill in the relevant art(s) having the benefit of this disclosure. For example, the TPM may be configured to perform one or more functions or operations of TPMs for various aspects herein.

As used herein, the terms “computer program medium,” “computer-readable medium,” “computer-readable storage medium,” and “computer-readable storage device,” etc., are used to refer to physical hardware media. Examples of such physical hardware media include the hard disk associated with hard disk drive1114, removable magnetic disk1118, removable optical disk1122, other physical hardware media such as RAMs, ROMs, flash memory cards, digital video disks, zip disks, MEMs, nanotechnology-based storage devices, and further types of physical/tangible hardware storage media (including memory1120ofFIG.11). Such computer-readable media and/or storage media are distinguished from and non-overlapping with communication media and propagating signals (do not include communication media and propagating signals). Communication media embodies computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wireless media such as acoustic, RF, infrared and other wireless media, as well as wired media. Aspects are also directed to such communication media that are separate and non-overlapping with aspects directed to computer-readable storage media.

As noted above, computer programs and modules (including application programs1132and other programs1134) may be stored on the hard disk, magnetic disk, optical disk, ROM, RAM, or other hardware storage medium. Such computer programs may also be received via network interface1150, serial port interface1142, or any other interface type. Such computer programs, when executed or loaded by an application, enable computing device1100to implement features of aspects discussed herein. Accordingly, such computer programs represent controllers of the computing device1100.

Aspects are also directed to computer program products comprising computer code or instructions stored on any computer-readable medium or computer-readable storage medium. Such computer program products include hard disk drives, optical disk drives, memory device packages, portable memory sticks, memory cards, and other types of physical storage hardware.

IV. Additional Examples and Advantages

As described, systems and devices embodying the techniques herein may be configured and enabled in various ways to perform their respective functions for generating automations via natural language processing. In aspects, one or more of the steps or operations of any flowchart and/or flow diagram described herein may not be performed. Moreover, steps or operations in addition to or in lieu of those in any flowchart and/or flow diagram described herein may be performed. Further, in examples, one or more operations of any flowchart and/or flow diagram described herein may be performed out of order, in an alternate sequence, or partially (or completely) concurrently with each other or with other operations.

As described herein, systems, devices, components, etc., of the aspects that are configured to perform functions and/or operations are also contemplated as performing such functions and/or operations.

Any additional examples and aspects described in this Section may be applicable to examples disclosed in any other Section or subsection of this disclosure.

Aspects provided in this description provide for systems, devices, and methods for generating automations via natural language processing. For instance, such a system is described herein. The system includes a program memory that stores program code, and a processing system, based on the program code, to perform one or more operations. The one or more operations include to: receive natural language input from a user interface; design an automation based on the natural language input; populate a user interface with parameters for operations of the automation, the parameters derived from the natural language input; and control, in response to an approval received via the user interface, building of the automation based on the parameters.

In an aspect of the system, the processing system is to: subsequent to the user interface being populated with the parameters: provide a prompt corresponding to a blank field of the user interface for user input thereto, and receive the user input via the user interface; and the building of the automation is also based on the user input.

In an aspect of the system, the automation is designed also based on a machine learning model that is trained using natural language prompts and a second workflow definitional language that is derived from a first workflow definitional language, the second workflow definitional language including logical expressions that are absent from the first workflow definitional language; and the processing system is to: receive the natural language input at the machine learning model to generate an output according to second workflow definitional language, and convert the output into the first workflow definitional language to design the automation.

In an aspect of the system, the second automation workflow definitional language includes logical expressions that are absent from the second automation workflow definitional language; and the second automation workflow definitional language enables step iterations that are absent from the first automation workflow definitional language.

In an aspect of the system, the processing system is to: populate a first field in the user interface with a first parameter derived from the natural language input; and populate a second field in the user interface with a second parameter that is also derived from the first parameter.

In an aspect of the system, the user interface accepts a modification by a user to alter the first parameter or the second parameter, or the second parameter is also derived at least from account information associated with the user.

In an aspect of the system, the operations are determined based on a library associated with an application or a service associated with the application, or the operations are for one of: a combination of web-based applications; a combination of web-based services; and a combination of web based applications and web based services.

In an aspect of the system, the processing system is to: deploy the automation subsequent to the automation being built.

A method performed by a computing system is also described herein. The method includes receiving natural language input from a user interface; designing an automation based on the natural language input; populating a user interface with parameters for operations of the automation, the parameters derived from the natural language input; and controlling, in response to an approval received via the user interface, building of the automation based on the parameters.

In an aspect, the method includes, subsequent to the user interface being populated with the parameters: providing a prompt corresponding to a blank field of the user interface for user input thereto, and receiving the user input via the user interface; and the building of the automation is also based on the user input.

In an aspect of the method, at least one of the one or more operations based at least on a machine learning model that is trained utilizing at least natural language prompts and a second automation workflow definitional language that derived from a first automation workflow definitional language.

In an aspect of the method, the automation is designed also based on a machine learning model that is trained using natural language prompts and a second workflow definitional language that is derived from a first workflow definitional language, the second workflow definitional language including logical expressions that are absent from the first workflow definitional language; and the method includes: receiving the natural language input at the machine learning model to generate an output according to second workflow definitional language, and converting the output into the first workflow definitional language to design the automation.

In an aspect of the method, the second automation workflow definitional language includes logical expressions that are absent from the second automation workflow definitional language; and the second automation workflow definitional language enables step iterations that are absent from the first automation workflow definitional language.

In an aspect, the method includes populating a first field in the user interface with a first parameter derived from the natural language input, and populating a second field in the user interface with a second parameter that is also derived from the first parameter.

In an aspect of the method, the user interface accepts a modification by a user to alter the first parameter or the second parameter; or the second parameter is also derived at least from account information associated with the user.

In an aspect of the method, the operations are determined based on a library associated with an application or a service associated with the application; or the operations are for one of: a combination of web-based applications; a combination of web-based services; and a combination of web based applications and web based services

A computer-readable storage medium having program instructions recorded thereon that, when executed by a processing system, perform a method, is also described. The method includes receiving natural language input from a user interface; designing an automation based on the natural language input; populating a user interface with parameters for operations of the automation, the parameters derived from the natural language input, including: populating a first field in the user interface with a first parameter derived from the natural language input, and populating a second field in the user interface with a second parameter that is also derived from the first parameter; and controlling, in response to an approval received via the user interface, building of the automation based on the parameters.

In an aspect of the computer-readable storage medium, the method further includes, subsequent to the user interface being populated with the parameters: providing a prompt corresponding to a blank field of the user interface for user input thereto, and receiving the user input via the user interface; and the building of the automation is also based on the user input.

In an aspect of the computer-readable storage medium, at least one of the one or more operations based at least on a machine learning model that is trained utilizing at least natural language prompts and a second automation workflow definitional language that derived from a first automation workflow definitional language.

In an aspect of the computer-readable storage medium, the automation is designed also based on a machine learning model that is trained using natural language prompts and a second workflow definitional language that is derived from a first workflow definitional language, the second workflow definitional language including logical expressions that are absent from the first workflow definitional language; and the method includes: receiving the natural language input at the machine learning model to generate an output according to second workflow definitional language, and converting the output into the first workflow definitional language to design the automation.

In an aspect of the computer-readable storage medium, the second automation workflow definitional language includes logical expressions that are absent from the second automation workflow definitional language, and the second automation workflow definitional language enables step iterations that are absent from the first automation workflow definitional language.

In an aspect of the computer-readable storage medium, the user interface accepts a modification by a user to alter the first parameter or the second parameter; and the building of the automation based on the parameters includes building the automation utilizing the modification.

V. Conclusion

While various aspects of the disclosed subject matter have been described above, it should be understood that they have been presented by way of example only, and not limitation. It will be understood by those skilled in the relevant art(s) that various changes in form and details may be made therein without departing from the spirit and scope of the aspects as defined in the appended claims. Accordingly, the breadth and scope of the disclosed subject matter should not be limited by any of the above-described exemplary aspects, but should be defined only in accordance with the following claims and their equivalents.