Systems and methods for automated skill creation and selection

Described are systems and methods for extending the capabilities of client communication devices that are communicatively coupled to the Internet and capable of executing supported applications. An automated skills-creation process locates forms on the World Wide Web that requires user interaction, develops skills in support of those interactions, and loads the skills into a skills database. The skills thus developed can simplify subsequent user interactions with the related web resources, such as to facilitate interactions that are partially or wholly based on speech.

BACKGROUND

A “skill” is a device capability. Amazon's ALEXA, a so-called intelligent personal assistant capable of voice interaction, provides examples in which a user can enable and disable skills, using the ALEXA application (app) or a web browser, as one would install and remove apps on a mobile device. Skills can be called up using manual user-interface (UI) devices, such as a keyboard or mouse, or can be called up using voice commands. People and institutions are rapidly developing skills for accomplishing myriad tasks. There nevertheless remains a demand for skill development.

DETAILED DESCRIPTION

FIG. 1diagrammatically depicts information flow100in support of systems and methods for extending the capabilities of client communication devices that are communicatively coupled to the Internet and capable of executing supported applications. The information flow is automated to locate information resources on the World Wide Web (“web resources”) that require user interaction, develop skills in support of those interactions, and load the skills into a skills database105. The skills thus developed and stored can simplify subsequent user interactions with the related web resources, such as to facilitate interactions that are partially or wholly based on speech. For example, an automated skills-creation system might comb the web to identify web forms associated with banking websites and develop the requisite skills for interacting with those resources to e.g. open accounts, check balances, and transfer money. Rather than conventionally interacting with the corresponding bank by filling in various form fields using a mouse and keyboard, as intended by the designer of the web interface, a related “skill” in database105might walk the user through the account process as would a human teller, receiving the requisite form data via verbal communication converted to ASCII. Skills development of this kind can expand to support interaction with a nearly unlimited number of service providers. The resultant ease of use would be a boon for mobile-device users.

Information flow100relies on components that can be supported by different economic entities (e.g., one or more cloud-based service providers interconnected via the Internet). A content-processing engine110engages a crawler service115to automatically crawl the World Wide Web in a methodical, automated manner to discover web forms in need of skills. Engine110can select specific types of resources, such as to emphasize popular skills types and reduce risks associated with malicious websites. Where engine110has access to browsing histories, engine110can target skill creation by popularity rather than or in addition to via crawling. The latter is advantageous, however, in that skills can be created in advance of user access.

Crawler service115employs an HTML processor120running a headless browser (a web browser without a graphical user interface) to access web resources via their uniform resource locators (URLs), issuing a request122to download the contents of a website125(e.g. zoho.com). Processor120responsively downloads the contents of zoho.com, e.g. two pages page1.html and page2.html, and relays the full website contents to crawler service115, which in turn relays the contents125to content-processing engine110.

Engine110parses the HTML data from the selected web resource, such as a web form or forms on pages page1.html and page2.html, and attempts to exercise form links and fields. A web form allows users to enter data to send to a server for processing. Web forms can resemble paper or database forms because web users fill out the forms using e.g. check boxes, radio buttons, or text fields.

In this example, engine110runs an automated skill-builder that parses a sign-up form on page page1.html for an online account for data inputs and builds a skill that gathers the requisite data. The form may simply require a name and a password or may require more information that can be gathered from various resources. For example, a skill to open an account with a particular service provider might gather user data from the user, the user's device, and an accessible database. The skill might also ask for or suggest a password, arrange for the transfer of money, etc. Skills may also be independent of service providers. A hotel-reservations skill called up by a user seeking a place to stay may be capable of interacting with multiple hotel websites, selecting among them based on location, cost, user preferences, etc. Once developed, a user could execute the skill by interrogating an entity with access to content processing engine110. Engine110could add, remove, or update skills responsive to information obtained by periodically crawling the Web for web forms. Engine110represents each skill it creates or builds as a set of instructions and data required to execute the skill and stores this skill information130in skills database105.

FIG. 2depicts a networked communication system200that allows a user205access to a skills server210via a remote device215and a wide-area network220, such as the Internet. Skills server210has or has access to skills database105and includes content-processing engine110as detailed above in connection withFIG. 1. Skills server210additionally supports or includes a skill-detection engine225and a skill management-and-execution service230. Device215, a mobile phone in this example, can be other types of client devices that support a voice user interface and have access to networked resources.

Skills database105stores skills associated with many web resources, including a skill to access an online bank account with a movie website240. User205can browse website240manually using device215; however, such browsing requires manual device access that can be inconvenient. User205can instead initiate a call to skill server210using a voice command or chat interface to a skill application (app)235running on device215. Calls to skill app235are initiated using a leading term “Zia” in this example. User205might ask “Zia, open an account on the movie website” or “Zia, play a movie from the movie website.” Skill-detection engine225responsively checks skills database105for the requested skill. Should the skill exist, skill management and execution server230downloads the skill from database105, gathers the requisite information, and performs the skill on behalf of user205. The requisite information may be acquired from a user database250, device235, user205, the resource of interest, or a combination of these and other information sources. For example, if the user's request is to watch a movie the skill might create or log into the website of the requested resource, elicit a genre or title, process payment, and direct the movie to a browser on device235. If the use's request is to attend a movie, execution server might additionally present user205with show times and locations.

Where device215support apps with overlapping capabilities, skills can aid in app selection. For example, user205issues a voice command to the Zia app235to “get directions to Pleasanton Calif.” on a device that has or has access to multiple apps capable of answering the user's request. Skill server210aids in selecting which app or apps have the requested skill, if any, and to select between suitable apps if there is more than one. Skill server210can query e.g. database250for information relating supported applications to capabilities of computing device215to match at least one of the applications supported by device215to the expressed device capability parsed from the command.

App selection based on capability can be accomplished on or aided by information stored by or from device215. Selecting which apps to execute may be based on a user prompt, or may be based on some desired metric, such as a record of which app works better, receives higher reviews, was selected last time by the user, or which provider pays a small fee. Ambiguities between skill selections can also be resolved with reference to some characteristic of the user, such as location or various personal data stored in an accessible database. Skill server210can relate skills to teams, employees, categories, etc. External skills can be developed and customized for an organization. For example, an Uber skill might call a driver but also submit the requisite expense to the company. Such a skill could be internal or shared between companies.

FIG. 3diagrammatically depicts information flow300in support of invoking skills in accordance with one embodiment. As detailed previously, content processing engine110and other elements crawls the Internet for forms, automatically develops corresponding skills, and populates skills database105with skill information (305). A user employs device215to invoke a skill by name, typically via a voice command (310). Execution service230queries database105for stored details required for skill execution (315). In this example, those skill details320include inputs from device215that are required for skill execution, details such as the name and location of the user expressing interest in movie. Execution service230retrieves these details by requesting inputs325to be collected from the user or user's device215. Service230can also collect skill information from third-party databases, such as show times and theater locations. Execution service230then awaits inputs325from device215, and any other information required for skill execution, and returns a response330to device215. In the movie example, response330might include an electronic movie ticket and receipt.

Execution service230can additionally update and maintain skills as needed for improved performance and ease of use. Service230may, for example, monitor the outcome of an executed skill and update the capability accordingly. The example ofFIG. 3includes an error message335from device215. A movie theater targeted by the invoked skill may have modified their web interface to require additional user input from the time of skill creation, for example, in which case app235would report a failure to complete the skill. The user would suffer the minor inconvenience of being prompted for the missing input. Report335would allow execution service230to update the skill, possibly working with engine110, to seek the requisite additional user input and pass updated skill information340to skills database105. Subsequent users of the skill would thus be presented with an updated, corrected version. Skills may likewise be updated by content processing engine110periodically exercising skills via a test browser. Stale skills can be updated automatically or with human support.

FIG. 4is a flowchart400illustrating how content processing engine110builds skills with which to populate and update database105in accordance with one embodiment. This example assumes a single URL represented as a website405with form fields indicative of the possibility of building an applicable skill. Engine110assigns the skill a form identifier (ID)410and parses the page or pages to identify the requisite inputs and input types415(e.g., form fields, activatable icons, captchas).

Many forms and fields are sufficiently descriptive that users can fill in the requisite data (e.g., the “name” field is labeled “name”). Engine110executes an automated skill-builder process that parses such forms for the requisite information to develop a speech-based skill. The automated skill builder might be sufficiently generic that it could be applied to web forms generally. Alternatively, a relatively small set of skill builders tailored for different languages, business types, local norms, etc., could be used. In any case, one or an army of web crawlers (also known as web spiders or web robots) could automatically crawl the World Wide Web in a methodical, automated manner to discover web forms in need of skills, apply suitable available skills, and prompt human intervention where no suitable skill can be created.

Engine110can use actual or fictitious data to gather what input is available (e.g., the timing or location of a service) and derive a skill label (420). Engine110then exercises the form by generating questions and collecting simulated user responses (425). Simulated user responses can be taken from fictitious users during skill development. This step may seek human intervention, as where a captcha requires recognition. Engine110considers and addresses success/failure response messages (430) until the skill under development succeeds. Content-processing engine110then stores the newly created or updated skill with corresponding skill identifier in skills database105.

The information flow detailed above in connection withFIGS. 1-4is automated to locate web resources that require user interaction, develop skills in support of those interactions, and store the skills. The skills thus developed can dramatically simplify subsequent user interactions with the related web resources, such as to facilitate interactions that are partially or wholly based on speech.

Skills can be grouped by entity, skill category, or any combination of these and other logical connections. Skills can further be nested within other skills. A user might execute a skill to open a bank account that prompts the user to select a bank, and in so doing call up a second skill nested within the first. In the alternative, the entire nested structure could be considered a “bank” skill with individual-bank or branch sub-skills. A user could thus call up a bank skill (“Zia, help me open a bank account”) that solicits location and other preference information from the user, selects a bank-specific skill accordingly, and steps the user through an account set-up process stylized and tested for the desired bank. A third-party service, such as available from Google, can perform the speech recognition required to convert voice commands to digital representations of the contents of those commands. Skill server210can note stumbling blocks and corrections to improve a skill based on user experience.

In some embodiments skill server210builds skills for a website visitor “on the fly,” with the user's experience guiding skill creation. For example, the skill builder could note when skill creation stumbles or fails, noting steps the user takes to overcome problems. Failure could also be reported to a human to fix the skill. Multiple skills can also be provided for the same task to accommodate user differences. Skills for a given task can also evolve with changes in e.g. the content of an associated website or other variations in users or context.

The Zoho intelligent assistant (Zia) could maintain a large database of skills and device capabilities. The user might ask “Give me directions home,” likely supported locally via a GPS app, or “Open me an account with Bank of America (BofA),” likely not supported locally because the user would not yet have a BofA app. The Zia app could check whether the BofA skill is available on the device, and if not check the Zia server to see whether such a skill is available at all, and if so to assist the user in opening the BofA account. The skill might require getting the requisite device support (e.g., downloading an app), if any, etc. The skill may exist in the cloud because (1) another Zia user developed the skill, (2) Zia developed the skill in connection with another user successfully opening a BofA account, (3) the skills crawler created the skill, or (4) a combination of these. The skills builder might also build the skill on the fly for the first requesting user and store a successful version of the skill for subsequent requests. This process is analogous to progressively better searches that flow from repeated queries and answers except that here the “answers” are activities rather than results.

FIG. 5(prior art) depicts a general-purpose computing system500that can serve as a client or a server depending on the program modules and components included. One or more computers of the type depicted in computing system500can be configured to perform operations described with respect toFIGS. 1 through 4. Those skilled in the art will appreciate that the invention may be practiced using other system configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, and the like.

Computing system500includes a conventional computer520, including a processing unit521, a system memory522, and a system bus523that couples various system components including the system memory to the processing unit521. The system bus523may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The system memory includes read only memory (ROM)524and random-access memory (RAM)525. A basic input/output system526(BIOS), containing the basic routines that help to transfer information between elements within the computer520, such as during start-up, is stored in ROM524. The computer520further includes a hard disk drive527for reading from and writing to a hard disk, not shown, a solid-state drive528(e.g. NAND flash memory), and an optical disk drive530for reading from or writing to an optical disk531(e.g., a CD or DVD). The hard disk drive527and optical disk drive530are connected to the system bus523by a hard disk drive interface532and an optical drive interface534, respectively. The drives and their associated computer-readable media provide nonvolatile storage of computer readable instructions, data structures, program modules and other data for computer520. Other types of computer-readable media can be used.

A number of program modules may be stored on the hard disk, solid state disk528, optical disk531, ROM524or RAM525, including an operating system535, one or more application programs536, other program modules537, and program data538. A user may enter commands and information into the computer520through input devices such as a keyboard540, microphone541, and pointing device542. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit521through a serial port interface546that is coupled to the system bus, but may be connected by other interfaces, such as a parallel port, game port or a universal serial bus (USB). A monitor547or other type of display device is also connected to the system bus523via an interface, such as a video adapter548. In addition to the monitor, computers can include or be connected to other peripheral devices (not shown), such as speakers and printers.

The computer520may operate in a networked environment using logical connections to one or more remote computers, such as a remote computer549. The remote computer549may be another computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer520, although only a memory storage device550has been illustrated inFIG. 5. The logical connections depicted inFIG. 5include a network connection551, which can support a local area network (LAN) and/or a wide area network (WAN). Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

Computer520includes a network interface553to communicate with remote computer549via network connection551. In a networked environment, program modules depicted relative to the computer520, or portions thereof, may be stored in the remote memory storage device. It will be appreciated that the network connections shown are exemplary and other means of establishing a communication link between the computers may be used.

Variations of these embodiments, including embodiments in which features are used separately or in any combination, will be obvious to those of ordinary skill in the art. Therefore, the spirit and scope of the appended claims should not be limited to the foregoing description. In U.S. applications, only those claims specifically reciting “means for” or “step for” should be construed in the manner required under 35 U.S.C. section 112(f).