Patent Description:
The invention is defined according to the features of the independent claims.

A computerized personal assistant may be configured to use a variety of different skills to serve natural language queries submitted by a user (e.g., via a natural language user interface (NLUI) configured to receive audio and/or textual natural language queries).

As used herein, "skill" includes any operation or set of related operations which may be performed by a computerized personal assistant to assist a user. Non-limiting examples of assistance that may be provided by a skill include <NUM>) gathering and outputting information and/or multimedia content (e.g., automatically searching a multimedia streaming service for streaming content); <NUM>) recording information for later retrieval (e.g., memos, notes, to-do lists, commitments); <NUM>) communicating with other users (e.g., by sending an email or other message); <NUM>) controlling electrical and/or mechanical appliances (e.g., automatically controlling a home thermostat); and/or <NUM>) interacting with a cooperating computer service or device (e.g., preparing content to be displayed at a user device such as a mobile phone).

In some examples, using a skill may include communicatively coupling the computerized personal assistant to a cooperating computer service in order to cause the cooperating computer service to perform at least part of the operation to assist the user. For example, using a skill may include communicatively coupling to a cooperating computer service having an associated application programming interface (API), and automatically invoking the API to assist the user. More generally, a skill may be configured to invoke one or more APIs in any suitable manner to assist the user. Non-limiting examples of APIs include <NUM>) APIs for purchasing goods and/or services (e.g., API for scheduling a ride, API for ordering food for delivery); <NUM>) APIs for subscription multimedia content (e.g., API for logging into a streaming video service and retrieving licensed video content); <NUM>) APIs for gathering information (e.g., API for retrieving directions from a map service); <NUM>) APIs for performing calculations and/or transforming data; <NUM>) APIs for storing and/or retrieving user data (e.g., API for collaborative document editing, API for sharing photographs); and/or <NUM>) APIs for interacting with specific hardware devices and/or cooperating computer services (e.g., APIs for using sensor data collected at a user device such as a mobile phone). Skills configured to invoke a plurality of APIs may provide assistance for any suitable task(s) using the APIs.

A computerized personal assistant may be configured to use a wide variety of different skills. However, a user of the computerized personal assistant may not know what skills are available and/or how to use available skills. As such, when the user wants to perform a task, although they may wish to receive assistance from the computerized personal assistant, they may not know if the computerized personal assistant is capable of providing assistance. Even if the user suspects that the computerized personal assistant is configured to help, the user may not know an appropriate query that would cause the computerized personal assistant to provide the desired assistance. As used herein, "query" includes any utterance (e.g., questions, verbal commands, non-verbal-commands, and/or spoken or written words, phrases, and/or sentences) which may be processed by a computerized personal assistant in order to determine how to assist the user (e.g., what skill to utilize to assist the user). Utterances include audible speech as well as any other actions by a user signifying lexical content, e.g., typing text and/or symbols, inking handwritten text and/or symbols, and/or gestures. Similarly, "conversation" is used herein to refer to a spoken and/or textual exchange between a user and a computerized personal assistant.

Even when a user has previously engaged in conversation with a computerized personal assistant and thereby become familiar with some of the skills of the computerized personal assistant, the user may be unfamiliar with other skills of the computerized personal assistant. In some examples, the computerized personal assistant may be updated to provide new skills not previously seen by users of the computerized personal assistant. In some examples, the computerized personal assistant may provide skills that pertain to a domain-specific task that a user is unlikely to engage in frequently, or skills that pertain to a task the user has never previously attempted to complete with the assistance of the computerized personal assistant. As a result, a user of the computerized personal assistant may not become familiar with a skill of the computerized personal assistant, and may be unable to pose appropriate queries to complete a task using the skill.

Accordingly, <FIG> shows a computing environment <NUM> including a computerized personal assistant <NUM> that is configured to facilitate the discovery of skills by a user <NUM> through conversation with the user <NUM>. Computerized personal assistant <NUM> is communicatively coupled, via a computer network <NUM>, to a personalization database <NUM>, a skills database <NUM>, a classifier machine <NUM>, a validation machine <NUM>, and optionally further communicatively coupled to a cooperating computer service <NUM> and/or to a user device <NUM>. Computer network <NUM> may be any suitable computer network, e.g., the Internet. Cooperating computer service <NUM> may be any suitable computer service, e.g., a provider of an API. User device <NUM> may be any suitable computing device of user <NUM>, e.g., a personal computer or a mobile phone.

Facilitating discovery of skills by a user <NUM> through conversation with the user <NUM> may improve an efficiency and/or performance of computerized personal assistant <NUM> aiding user <NUM> in the completion of various tasks. For example, when relevant skills are discovered through conversation, user <NUM> may efficiently learn about how to cause computerized personal assistant <NUM> to provide assistance with a task by simply asking a question about the task, without needing to divert attention from the task. Furthermore, facilitating discovery of skills through conversation may allow user <NUM> to learn about a skill that is particularly well suited to a task. For example, computerized personal assistant <NUM> may help user <NUM> to discover a skill that will achieve particularly good performance when completing a task, in terms of computational efficiency (e.g., processor load and/or network bandwidth usage), user satisfaction, elapsed time to complete the task, monetary costs to complete the task, and/or any other suitable considerations of performance for completion of tasks by computerized personal assistant <NUM>.

Furthermore, computerized personal assistant <NUM> may ensure, through conversation with user <NUM>, that all relevant information for completing a task is collected prior to engaging a skill. Accordingly, computerized personal assistant <NUM> may defer engaging skills until sufficient information is collected to engage a pertinent skill that is likely to succeed in assisting user <NUM> with completing the task. Similarly, by facilitating discovery of pertinent skills and collecting relevant information before engaging skills, computerized personal assistant <NUM> may avoid engaging skills that are unlikely to succeed in assisting user <NUM> with completing the task. Accordingly, computerized personal assistant <NUM> may increase a computational efficiency and/or reduce a cost of completing tasks (e.g., by reducing processor load and/or network bandwidth usage).

Computerized personal assistant <NUM> includes a NLUI <NUM> configured to receive queries. Optionally, in some examples, NLUI <NUM> may include a microphone <NUM> configured to capture speech audio. Accordingly, the queries received by NLUI <NUM> may include the speech audio captured by the microphone. In some examples, NLUI <NUM> is configured to receive user speech audio and to represent the user speech audio as text.

Alternately or additionally, NLUI <NUM> may include a keyboard or any other text input device configured to receive a query in the form of text. In some examples, NLUI <NUM> may include an inking input device and the user input received by NLUI <NUM> may include user handwriting and/or user gestures captured by the inking input device. As used herein, "inking input device" includes any device or combination of devices which may allow a user to provide inking input. For example, an inking input device may include any device which allows the user to indicate a series of two- or three-dimensional positions relative to a display or any other surface, e.g., <NUM>) a capacitive touch screen controlled by a user's finger; <NUM>) a capacitive touch screen controlled by a stylus; <NUM>) a "hover" device including a stylus and a touch screen configured to detect a position of the stylus when the stylus is in proximity to the touch screen; <NUM>) a mouse; or <NUM>) a video game controller. In some examples, an inking input device may alternately or additionally include a camera configured to detect user gestures. For example, a camera may be configured to detect gestures based on three-dimensional movements of the user's hand. Alternately or additionally, a camera (e.g., a depth camera) may be configured to detect movements of the user's hand as two-dimensional positions relative to a surface or plane, e.g., relative to a plane defined by a front side of a viewing frustum of the camera.

Optionally, in some examples, NLUI <NUM> may include a speaker <NUM> configured to output responses to queries in the form of speech audio. Alternately or additionally, NLUI <NUM> may include a display configured to display responses to queries in the form of text. Alternately or additionally, NLUI <NUM> may be configured to send responses, via network <NUM>, to a user device <NUM> (e.g., a mobile phone) in order to output speech audio via a speaker of user device <NUM> and/or to display response text at a display of user device <NUM>. Alternately or additionally, NLUI <NUM> may be configured to send any suitable notification to user <NUM> (e.g., via any suitable messaging protocol, via visible lights, via displayed symbols, text, and/or images, and/or via audible sounds and/or synthesized speech).

Computerized personal assistant <NUM> includes a skill discovery machine <NUM>. Skill discovery machine <NUM> is configured to facilitate discovery of skills to assist user <NUM>. Skill discovery machine <NUM> may facilitate discovery of a skill based on a query received at NLUI <NUM> of computerized personal assistant <NUM> and/or based on a current context of user <NUM>.

Skill discovery machine <NUM> is configured to assess a match confidence for a candidate skill, the match confidence indicating a quality of match between a current context and a reference context previously associated with the candidate skill. In some examples, the reference context may include a seed query defined by an exemplary natural language sentence. In some examples, a reference context comprises a historical activity context including state data of a computer service (e.g., cooperating computer service <NUM>), wherein the state data is associated with a recognized activity that was engaged in in by a historical user of the computer service, and wherein the operation to assist the user pertains to the recognized activity.

In some examples, skill discovery machine <NUM> is configured to communicatively couple to skills database <NUM> and/or classifier machine <NUM> to select a skill from skills database <NUM> based on the current context of user <NUM>, in order to facilitate the discovery of the candidate skill. Selecting a candidate skill from skills database <NUM> is based on assessing a match confidence for the candidate skill, the match confidence indicating a quality of match between the current context and a reference context previously associated with the candidate skill.

As used herein, "context" includes any information relating to user <NUM> which is available to computerized personal assistant <NUM> and which may pertain to a current activity of user <NUM>. Non-limiting examples of such context information include: <NUM>) a history of interaction between user <NUM> and computerized personal assistant <NUM> (e.g., past queries) and/or interaction between user <NUM> and cooperating computer service <NUM>; <NUM>) a current state of cooperating computer service <NUM>; <NUM>) a current state of user device <NUM> (e.g., based on sensor data captured by a sensor of user device <NUM> or based on an application program executing on user device <NUM>); <NUM>) stored commitments, appointments, communications, or other logistical information input by the user <NUM> and/or received from other users at cooperating computer service <NUM> and/or user device <NUM>; <NUM>) stored preferences of user <NUM>, <NUM>) queries received at NLUI <NUM>, and/or <NUM>) current audio or video observations of the user. In some examples, a context of a user is determined based on recognizing natural language features of a query received at NLUI <NUM> in order to identify one or more of an intent expressed by the query and/or a constraint defined by the query. In some examples, a context of a user is determined based on recognizing an intent of the user based on a current and/or past activity of the user, e.g., based on state data of an application program executing on user device <NUM> or state data of cooperating computer service <NUM>.

As used herein, "intent" includes any outcome intended by the user, e.g., a goal the user would like to be fulfilled by a response to a query, or a task the user is performing using a cooperating computer service. "Constraint" includes any detail specified by a context, e.g., an entity mentioned in the query, an event mentioned in the query, an entity indicated by state data of cooperating computer service <NUM>, or any other suitable details (e.g., time/date/location details). Computerized personal assistant <NUM>, using NLUI <NUM>, may recognize an intent of a context, e.g., an intent of a query or an intended task a user may be engaged in based on a state of a cooperating computer service <NUM>. For example, if user <NUM> asks computerized personal assistant <NUM> to "play disco music," computerized personal assistant <NUM> may recognize an intent to play music, and a constraint that the genre of the music should be disco. In some examples, computerized personal assistant <NUM> may recognize one or more predefined constraints that should be associated with a received query based on an intent of the query. For example, when an intent of a query is to play music, computerized personal assistant <NUM> may recognize that one or more of the "genre," "song," and/or "artist" constraints should be specified in order to play music.

Personalization database <NUM> may store one or more current contexts of user <NUM>. For example, responsive to interaction with user <NUM>, computerized personal assistant <NUM>, cooperating computer service <NUM>, and/or user device <NUM> may communicatively couple to personalization database <NUM> in order to store relevant data in personalization database <NUM>. In some examples, personalization database <NUM> may be a component of computerized personal assistant <NUM>. For example, a storage device of computerized personal assistant <NUM> may store personalization data and make such personalization data available to provide functionality of personalization database <NUM>. In some examples, computerized personal assistant <NUM> may communicatively couple to personalization database <NUM> in order to access personalization data stored in personalization database <NUM>. Accordingly, computerized personal assistant <NUM> may retrieve a current context of user <NUM>, in order to facilitate discovery of a skill based on such context. Alternately or additionally, computerized personal assistant <NUM>, cooperating computer service <NUM>, and/or user device <NUM> may maintain one or more current contexts of user <NUM>, independently of personalization database <NUM>. Accordingly, computerized personal assistant <NUM> may facilitate discovery of a skill based on any suitable current context (e.g., any combination of queries received at natural language user interface <NUM> and/or current contexts maintained by computerized personal assistant <NUM>, personalization database <NUM>, cooperating computer service <NUM>, and/or user device <NUM>).

Computerized personal assistant <NUM> further includes a skill execution machine <NUM>. Skill execution machine <NUM> is configured to determine a current query (e.g., by receiving, from the NLUI <NUM>, a current query captured by microphone <NUM>). Skill execution machine <NUM> is further configured, responsive to receiving the current query, to operate skill discovery machine <NUM> to assess a match confidence for a candidate skill. In some examples, skill execution machine <NUM> is configured to operate skill discovery machine <NUM> with regard to each candidate skill of a plurality of candidate skills, e.g., so as to select a candidate skill having a relatively higher match confidence among the plurality of candidate skills. Accordingly, through the operation of skill discovery machine <NUM>, skill execution machine <NUM> may select a candidate skill for executing an assistive action and/or complementary help action of the candidate skill. Skill execution machine <NUM> is further configured to execute instructions defining an assistive action associated with the candidate skill, to perform an operation to assist the user, responsive to the match confidence exceeding a predefined match confidence threshold. Skill execution machine <NUM> is further configured to execute instructions defining a complementary help action associated with the candidate skill, to output a predefined response to educate the user about the assistive action, responsive to the match confidence not exceeding the predefined match confidence threshold.

Skills database <NUM> is configured to include a plurality of skills for computerized personal assistant <NUM>. Skills database <NUM> may include available skills and/or unavailable skills. Computerized personal assistant <NUM> is configured to perform operations to assist user <NUM> according to any of the available skills, while computerized personal assistant <NUM> may need further configuration in order to utilize unavailable skills. For example, skills database <NUM> may include an unavailable skill that may only be utilized if computerized personal assistant <NUM> is configured to provide a user credential associated with a user account of a cooperating computer service (e.g., cooperating computer service <NUM>). In an example, an unavailable skill is a skill for playing streaming media content from a streaming media service, and the unavailable skill may only be utilized if computerized personal assistant <NUM> is configured to log in to the streaming media service using a user account of user <NUM>.

Skills database <NUM> is configured to define, for a candidate skill of the plurality of skills, an assistive action, a reference context previously associated with the assistive action, and a complementary help action previously associated with the assistive action. An assistive action for a skill is defined by instructions executable by computerized personal assistant <NUM> to perform an operation to assist a user. For example, an assistive action may include instructions to gather sensor data from computerized personal assistant <NUM> and/or user device <NUM>, and to invoke one or more APIs based on the gathered sensor data, in order to perform an operation to assist the user using the one or more APIs. A reference context associated with an assistive action may be any suitable context as described above, e.g., a query or a state of a cooperating computer service <NUM>. A complementary help action associated with an assistive action is defined by instructions executable by computerized personal assistant <NUM> to output a predefined response to educate the user about the assistive action. For example, a complementary help action may be defined by instructions to output a predefined response describing how to use the assistive action. In some examples, the predefined response may be output via speaker <NUM> of computerized personal assistant <NUM>.

Skills database <NUM> is configured to allow adding new skills by receiving data defining, for a new skill: instructions executable by computerized personal assistant <NUM> to perform an assistive action of the new skill, a reference context associated with the assistive action, and instructions executable by computerized personal assistant <NUM> to perform a complementary help action associated with the assistive action. The instructions to perform the complementary help action may be configured to output a predefined response to educate user <NUM> about the assistive action. Accordingly, in some examples, defining the complementary help action may be based on supplying the predefined response, and skills database <NUM> is configured to automatically define instructions to perform the complementary help action based on the supplied predefined response.

Assessing the match confidence based on a current context of user <NUM> may include operating classifier machine <NUM> to match the current context to a reference context associated with the candidate skill. In some examples, skill discovery machine <NUM> is configured to access skills database <NUM> in order to select an initial plurality of available skills and, for each skill in the plurality of available skills, to operate classifier machine <NUM> to select a matching skill. In some examples, functionality of skills database <NUM> and classifier machine <NUM> may be provided by a single computer service and/or computer device, e.g., skills database <NUM> and classifier machine <NUM> may be implemented on a common server. In some examples, skills database <NUM> and/or classification machine <NUM> may be a component of computerized personal assistant <NUM>. In some examples, skills database <NUM> and/or classifier machine <NUM> are configured to automatically cooperate to automatically select a skill based on a current context. For example, skills database <NUM> may be configured to receive a current context and to automatically operate classifier machine <NUM> to select a matching skill and return the matching skill to skill discovery machine <NUM>.

Classifier machine <NUM> includes a machine learning classifier previously trained to assess, for a candidate skill of the plurality of available skills in skills database <NUM>, a match confidence for the candidate skill indicating a quality of match between the current context of user <NUM> and a reference context previously associated with the candidate skill.

By matching a current context to a reference context of the candidate skill, classifier machine <NUM> may thereby assess a relevance of the candidate skill to the current context, e.g., when the current context and the reference context are similar, the candidate skill may be applicable to both contexts. For example, when the current context is based on a current query received at NLUI <NUM>, the reference context may be a similar reference query (e.g., a substantially identical query or a semantically equivalent rephrasing of the current query). Accordingly, the current query and the reference query may express a similar intent and/or define similar constraints, and the candidate skill is likely applicable to the current query as well as the reference query.

Optionally, in some examples, classifier machine <NUM> is configured to pre-process natural language data of a current context (e.g., natural language data derived from a query, and/or natural language data associated with a state of a cooperating computer service <NUM>) at language preprocessing <NUM>. Language preprocessing <NUM> may include any suitable preprocessing, e.g., tokenization, dependency and/or constituency parsing, correcting spelling, stemming words and replacing words with synonyms, and/or any other suitable techniques to transform natural language data of the current context into a standardized form (e.g., a parse tree including stemmed, correctly spelled words).

Optionally, in some examples, classifier machine <NUM> is configured to process a current context (e.g., after pre-processing at language preprocessing <NUM>) via one or both of a classifier pipeline <NUM> and/or a lexical pipeline <NUM>. In some examples, classifier machine <NUM> is configured to process the current context at each of classifier pipeline <NUM> and lexical pipeline <NUM>, independently and substantially in parallel.

Classifier pipeline <NUM> includes one or more machine learning (ML) and/or artificial intelligence (AI) models previously trained to classify a current context and a reference context in order to assess a match confidence indicating a quality of match between the current context and the reference context. Classifier pipeline <NUM> may include any suitable ML and/or AI models.

In some examples, classifier pipeline <NUM> includes a statistical classifier (e.g., support vector machine, logistic regression, or any other suitable statistical classifier). In some examples, classifier pipeline <NUM> includes a deep neural network model, e.g., deep semantic similarity model (DSSM). More generally, classifier pipeline <NUM> may include any ML model that may be trained based on training data (e.g., supervised training using labelled examples). Classifier pipeline <NUM> is configured to recognize a semantic match between the current context and the reference context, even when natural language features of the current context and the reference context differ lexically. In some examples, classifier pipeline <NUM> may include a plurality of different AI and/or ML models configured as an ensemble model. For example, the ensemble model may be based on any suitable ensemble method, such as bagging and/or boosting (e.g., AdaBoost).

Lexical pipeline <NUM> is configured to apply one or more rules for assessing a match confidence indicating a quality of match between the current context and the reference context based on lexical content of the current context and the reference context. Lexical pipeline <NUM> is configured to detect lexical features associated with the current context and/or reference context (e.g., lexical features of a query or lexical features of a natural language feature associated with a context). In some examples, lexical pipeline <NUM> is configured to detect keywords (e.g., according to a keyword whitelist) and/or lexical patterns (e.g., ordered sequences of words or patterns according to a formal grammar such as a regular expression grammar or a context-free grammar). In some examples, lexical pipeline <NUM> is configured to detect absence of keywords (e.g., keyword blacklist) and/or lexical patterns.

In some examples, lexical pipeline <NUM> is configured to assess a match confidence between the current context and the reference context based on similar lexical features being present in both contexts. In some examples, lexical pipeline <NUM> is configured to classify a current context based on lexical features. For example, when a current context is a query including the words "help," or similar lexical content related to assistance, lexical pipeline <NUM> is configured to classify the query as a help query. As used herein, "help query" includes any query to assist with discovering skills and/or configuring computerized personal assistant <NUM>. In some examples, computerized personal assistant <NUM> is configured to perform a complementary help action associated with a skill responsive to recognizing a help query that may be related to the skill (e.g., a query having a high match confidence assessed by classifier pipeline <NUM> while also being recognized as a help query by lexical pipeline <NUM>).

Classifier pipeline <NUM> and lexical pipeline <NUM> may cooperate to robustly assess matches between current contexts and reference contexts. Classifier pipeline <NUM> and lexical pipeline <NUM> may each assess a match confidence for a current context with regard to a reference context. In some examples, a match confidence assessed by classifier machine <NUM> may be based on match confidences output by classifier pipeline <NUM> and lexical pipeline <NUM> (e.g., a weighted sum of match confidences).

In some examples, classifier machine <NUM> may assess a match between the current context and each of a plurality of different reference contexts having varying match confidences indicating varying quality of match with the current context. Accordingly, classifier machine <NUM> is configured to assess, for each available skill of the plurality of available skills included in skills database <NUM>, a candidate match confidence for the available skill. Classifier machine <NUM> may be further configured to rank the different reference contexts based on candidate match confidences, and to select a candidate skill having a highest candidate match confidence, at ranking <NUM>. In this manner, classifier machine <NUM> may facilitate the discovery of a skill having a reference context that most closely matches a current context, even when multiple skills may match to varying extents. In some examples, classifier machine <NUM> may rank the different reference contexts based on a single match confidence value (e.g., based on a weighted sum of candidate match confidences assessed by classifier pipeline <NUM> and/or lexical pipeline <NUM>). In some examples, classifier machine <NUM> may rank the different reference contexts based on a plurality of different values generated during the matching process. For example, the reference contexts may be ranked based on separate match confidences assessed by classifier pipeline <NUM> and/or lexical pipeline <NUM>, based on a value assessed by one or more AI and/or ML models of classifier pipeline <NUM> (e.g., based on a value assessed by a hidden unit of a deep neural network model such as DSSM), and/or based on an individual rule applied by lexical classifier <NUM> (e.g., based on a match between the reference context and the current context with regard to a particular keyword).

In some examples, assessing a match confidence for a current context and a reference context is based on matching an intent of the current context to an intent of the reference context. In some examples, assessing a match confidence for a current context and a reference context is based on matching one or more constraints associated with the current context to one or more constraints associated with the reference context. For example, a current context and a reference context may have a relatively higher match confidence assessed when the two contexts each define the same constraint, and/or when the two contexts have one or more similar constraints.

Based on assessing a match confidence at classifier machine <NUM>, a skill matching the current context may be selected in any suitable manner. For example, the skill may be selected based on having an associated reference context that has a relatively highest match confidence with the current context. In some examples, the skill may be selected at classifier machine <NUM>. In some examples, the skill may be selected at skills database <NUM> and/or computerized personal assistant <NUM> based on the match confidence assessed by classifier machine <NUM> (e.g., classifier machine <NUM> may send, to skills database <NUM> and/or computerized personal assistant <NUM>, one or more match confidence values associated with skills in skills database <NUM> for selection).

In some examples, computerized personal assistant <NUM> is configured to operate skill execution machine <NUM> in order to execute the instructions defining an assistive action of the candidate skill responsive to selection of the skill. In some examples, skill execution machine <NUM> is configured to only execute instructions defining an assistive action of the candidate skill when the match confidence for the skill exceeds a predefined match threshold confidence, e.g., so as to only perform operations that were highly likely to be intended by user <NUM>. In some examples, skill execution machine <NUM> is configured to defer execution of instructions defining an assistive action of the candidate skill when the match confidence for the skill does not exceed the predefined match confidence threshold, e.g., so as to gather more information and/or ask user <NUM> a disambiguating question before executing the instructions. In some examples, skill execution machine <NUM> is configured to execute instructions defining a complementary help action associated with the skill instead of or in addition to executing the instructions defining the assistive action of the skill (e.g., when the match confidence does not exceed the predefined match confidence threshold).

Instructions may be executed at any of the computer devices in computing environment <NUM> of <FIG> (e.g., computerized personal assistant <NUM>, network <NUM>, personalization database <NUM>, skills database <NUM>, classifier machine <NUM>, validation machine <NUM>, cooperating computer service <NUM>, and/or user device <NUM>). In some examples, executing the instructions may include skills database <NUM> sending a command to computerized personal assistant <NUM> to execute further instructions configured to perform the assistive action and/or the complementary help action associated with a candidate skill. In some examples, executing the instructions may include computerized personal assistant <NUM> sending a command to cooperating computer service <NUM> to execute further instructions. More generally, an assistive action and/or a complementary help action may be defined by instructions that may be partially executed by any suitable combination of computer devices, thereby causing computing environment <NUM> and/or other cooperating computers to collectively provide the assistive action and/or the complementary help action.

Although exemplary computing environment <NUM> is described with regard to computerized personal assistant <NUM>, the methods described in the present disclosure (e.g., method <NUM>) may be enacted on any suitable computing device, e.g., a personal computing device or a cloud service instantiated on one or more network-accessible computing devices. As used herein, computerized personal assistant <NUM> generally refers to a local and/or remote set of computing devices configured to facilitate discovery of skills from skills database <NUM> and/or to answer queries by using a skill. In some examples, the computerized personal assistant <NUM> includes both <NUM>) a local computing device that includes input devices (e.g., keyboard, microphone, camera) and output devices (e.g., display, speaker) for interacting with a user, and <NUM>) a cooperating remote computer device (e.g., cloud computing device) configured to facilitate discovery of skills from skills database <NUM>, and/or answer queries using a skill. In some examples, one or more components of exemplary computing environment <NUM> may be included in a common enclosure, e.g., as components of computerized personal assistant <NUM>. For example, computerized personal assistant <NUM> may be implemented as an all-in-one computing device include components providing functionality of personalization database <NUM>, skills database <NUM>, classifier machine <NUM>, and validation machine <NUM>. In some examples, computerized personal assistant <NUM> may further provide functionality of one or more cooperating computer service, e.g., cooperating computer service <NUM>. In some examples, computerized personal assistant <NUM> may be implemented as an application program configured for execution on user device <NUM>.

Computerized personal assistant <NUM> may encounter many different current contexts of user <NUM>. For example, computerized personal assistant <NUM> may receive many different queries from user <NUM>. Moreover, queries received from user <NUM> may vary in lexical form and may express many different intents of user <NUM> and/or constraints related to the different intents. As such, there may be many different ways to issue a query to computerized personal assistant <NUM>. Furthermore, in some cases, a query can be served by immediately using a skill to assist user <NUM>, whereas in other cases more information may be required before computerized personal assistant <NUM> can use a skill to assist user <NUM> with regard to the query. Similarly, there may be many different activity contexts of user <NUM>. In some cases, using a skill may be appropriate given a particular activity context, but in other cases a skill may be only partially relevant to the activity context or more information may be required before computerized personal assistant can use the skill to assist user <NUM> with regard to the activity context.

Accordingly, <FIG> depicts a method <NUM> for facilitating discovery of a skill of a computerized personal assistant, e.g., computerized personal assistant <NUM>. In some examples, computerized personal assistant <NUM> may perform method <NUM> in order to expose a relevant skill based on a current context of user <NUM> thereby helping user <NUM> to learn what skills are available for computerized personal assistant <NUM> and to learn how to use the available skills. For example, computerized personal assistant <NUM> may perform method <NUM> in order to expose a relevant skill during conversation with user <NUM>, based on a current context of user <NUM> (e.g., a current context defined by a query posed by user <NUM> during the conversation).

At <NUM>, method <NUM> includes communicatively coupling computerized personal assistant <NUM> to a computer database including a plurality of available skills for computerized personal assistant <NUM>, e.g., skills database <NUM>.

At <NUM>, method <NUM> includes recognizing a current context of a user, e.g., user <NUM>. Optionally, in some examples, at <NUM>, the current context of user <NUM> includes a user query received at a NLUI, e.g., at NLUI <NUM>. For example, as shown in <FIG> at <NUM>, user <NUM> issues a query asking if computerized personal assistant <NUM> can "play `<NUM> music.

At <NUM>, method <NUM> includes operating a previously-trained machine learning classifier to assess a match confidence, the match confidence indicating a quality of match between the current context and a reference context of the candidate skill. Accordingly, in order to facilitate discovery of a candidate skill pertaining to the user query at <NUM>, computerized personal assistant <NUM> and/or skill discovery machine <NUM> may operate previously-trained machine learning classifier <NUM> to assess a match confidence for the candidate skill, and/or to assess match confidences for a plurality of candidate skills in order to select the candidate skill from among the plurality of candidate skills.

In some examples, assessing the match confidence for the candidate skill is based on assessing a match confidence for the current context with regard to a reference context of the candidate skill. In some examples, the reference context may include a seed query defined by an exemplary natural language sentence. Accordingly, the assistive action and the associated operation to assist the user may pertain to the seed query. For example, the candidate skill may be a skill having an assistive action for playing music using a streaming music service. Accordingly, a seed query associated with the candidate skill may be the query, "Play music by `The Beatles'," e.g., a query expressing an intent to play music and a constraint that the music artist should be 'The Beatles. ' Accordingly, the operation to assist the user may include playing music by `The Beatles' using the streaming music service.

At <NUM>, method <NUM> includes determining whether the match confidence exceeds a predefined threshold. The predefined threshold may be defined in any suitable manner. For example, the predefined threshold may be defined based on assessing an accuracy, precision, and/or recall of the previously trained machine learning classifier with regard to labelled training data (e.g., supervised training and/or cross-validation using the labelled training data).

At <NUM> and as depicted in <FIG>, the match confidence does not exceed the predefined threshold. For example, a match confidence assessed for the current query "Can you play `<NUM> music?" with regard to a seed query "Play music by `The Beatles'" may not exceed the predefined threshold due to a difference in form/phrasing of the questions. Alternately or additionally, the match confidence may not exceed the predefined threshold due to constraints of the queries, e.g., the seed query has a constraint pertaining to a music artist, whereas the current query has a constraint about an era of music, which may not be useable for specifying music to stream using the streaming music service. Accordingly, at <NUM>, method <NUM> includes executing instructions defining a complementary help action of the candidate skill, responsive to the match confidence not exceeding the predefined match confidence threshold. As shown in <FIG>, computerized personal assistant <NUM> outputs a predefined response at <NUM> to educate user <NUM>.

Optionally, at <NUM> and as shown in <FIG>, outputting the predefined response to educate the user about the assistive action may include outputting a natural language description of the assistive action, e.g., computerized personal assistant <NUM> can play music if given a genre, artist, or song name.

Optionally, at <NUM> and as shown in <FIG>, outputting the predefined response may include outputting an exemplary query for which the machine learning classifier is configured to output a match confidence value greater than the predefined match confidence threshold, so that the computerized personal assistant <NUM> is configured to perform the operation associated with the candidate skill to assist the user, responsive to receiving the exemplary query. For example, computerized personal assistant <NUM> is configured to respond to the query "Play rock music," by automatically playing rock music. In some examples, the exemplary query may be similar to a seed query or may be based on a seed query, e.g., "Play rock music" has a similar lexical form to the seed query "Play music by 'the Beatles'," and includes a genre constraint suitable for streaming music using the streaming music service.

After computerized personal assistant <NUM> outputs the predefined response to educate user <NUM> about how to use the skill for playing music, user <NUM> may continue the conversation by issuing another query. For example, at <NUM>, user <NUM> issues another query saying that computerized personal assistant <NUM> should "play disco music. " This new query is lexically similar to the exemplary query and includes a genre constraint similar to the exemplary query. Accordingly, computerized personal assistant <NUM> is configured to recognize the new context of the user based on the new query (e.g., according to method <NUM> at <NUM>), to operate the previously-trained machine learning classifier to assess a match confidence for the new context and a reference context of a candidate skill (e.g., according to method <NUM> at <NUM>), and to determine that the match confidence exceeds the predefined match confidence threshold (e.g., according to method <NUM> at <NUM>).

At <NUM> method <NUM> includes determining whether the candidate skill is available. As shown in <FIG>, the candidate skill for playing music is available, and accordingly, method <NUM> further includes, at <NUM>, executing the instructions defining the assistive action of the candidate skill, responsive to the match confidence exceeding the predefined match confidence threshold. Accordingly, computerized personal assistant <NUM> acknowledges that music will begin playing at <NUM>, thereby fulfilling the intent of the user's query.

<FIG> shows a further example of computerized personal assistant <NUM> helping user <NUM> according to method <NUM>. As shown in <FIG>, computerized personal assistant <NUM> recognizes a current context of user <NUM> (e.g., according to method <NUM> at <NUM>) based on a query at <NUM> in which the user asks the device to play rock music using a specific music service, "MusicService2. " Accordingly, computerized personal assistant <NUM> is configured to operate the previously-trained machine learning classifier to assess a match confidence for a candidate skill, e.g., a skill for playing music using MusicService2.

In some examples, user <NUM> might ask to play a particular music track that is only available on a particular music service (e.g., songs from an artist and/or album that is distributed on the particular music service and not on other music services available to computerized personal assistant <NUM>). Accordingly, computerized personal assistant <NUM> may infer a constraint implied by user <NUM>'s query, indicating that a music service making the music track available should be used for playing the music track. By inferring the music service to use for streaming music, computerized intelligent assistant <NUM> may alleviate a burden on user <NUM> to know and/or specify a service for playing the streaming content. Computerized personal assistant <NUM> may similarly infer and select an appropriate provider for any other kind of service (e.g., streaming movies and/or television services, food services, ride hailing services, etc.) based on constraint(s) implied by user <NUM>'s query. For example, computerized personal assistant <NUM> may infer and select a food service based on availability of a menu item user <NUM> is interested in ordering and/or based on a geographic coverage area of available food services. Similarly, computerized personal assistant <NUM> may infer and select a streaming movie service based on availability of a movie choice of user <NUM> within catalogs of movies offered by available streaming movie services.

Based on the query at <NUM>, the previously-trained machine learning classifier may assess a relatively high match confidence for the candidate skill so that the match confidence exceeds the predefined match confidence threshold at <NUM>. Accordingly, at <NUM>, method <NUM> includes determining whether the candidate skill is available. However, the candidate skill may be unavailable because user <NUM> has not previously configured a user account for MusicService2.

At <NUM>, method <NUM> includes, responsive to the match confidence for the unavailable skill exceeding the predefined match confidence threshold, outputting a predefined response describing the unavailable skill and educating the user about configuring computerized personal assistant <NUM> to cause the unavailable skill to become available. Accordingly, as shown in <FIG>, computerized personal assistant <NUM> is configured to output a predefined response previously associated with the skill (e.g., when the skill was added to skills database <NUM>). For example, at <NUM>, computerized personal assistant <NUM> outputs a predefined response describing how music may be played using MusicService2, indicating that first a user account for MusicService2 is required, and describing how to set up the user account. In the example, computerized personal assistant <NUM> is further configured to offer to display, on the user's phone (e.g., user device <NUM>) an 'Accounts' settings page for configuring the account for MusicService2. Accordingly, at <NUM>, user <NUM> accepts this offer and at <NUM>, computerized personal assistant <NUM> acknowledges that user <NUM> can continue the set-up process on their phone. In this manner, although the candidate skill for playing music with MusicService2 may have been initially unavailable, computerized personal assistant <NUM> may facilitate discovery and usage of the candidate skill by guiding the user <NUM> through a configuration process to make the candidate skill available.

<FIG> shows a further example of a conversation between computerized personal assistant <NUM> and user <NUM> in which computerized personal assistant <NUM> facilitates the discovery of a skill according to method <NUM>. As shown in <FIG>, at <NUM>, user <NUM> asks computerized personal assistant <NUM> to order food for delivery. Computerized personal assistant <NUM> may recognize a relatively high match confidence for each of two different skills for ordering food, using either "FoodService1" or "FoodService2. " However, the match confidence for each of the two different skills may be below the predefined threshold (e.g., even when the user's intent to order food is recognized, the user's query may not have a sufficiently high match with either of the two skills, because the user's query does not sufficiently define constraints such as which particular skill/food service to use).

Computerized personal assistant <NUM> is configured to perform a complementary help action, wherein the complementary help action defines a disambiguating question. Accordingly, computerized personal assistant <NUM> may output, via NLUI <NUM>, the disambiguating question, by asking whether to use FoodService1 or FoodService2 at <NUM>. Computerized personal assistant <NUM> is further configured to receive, via NLUI <NUM>, an answer to the disambiguating question, e.g., user <NUM>'s answer at <NUM>. Computerized personal assistant <NUM> is configured to interpret the answer to the disambiguating question based on conversational history with user <NUM>, e.g., by interpreting user <NUM>'s reply "the first one" to indicate the first option provided at <NUM>, namely FoodService1. Accordingly, computerized personal assistant <NUM> may recognize a new context based on the conversation history with user <NUM>, wherein the new context indicates that user <NUM> intends to order food using FoodService1 in particular.

Even after user <NUM> says to use FoodService1 at <NUM>, a new match confidence for the current context with regard to a reference context of a candidate skill may not exceed the predefined match confidence threshold. For example, even when the user's intent to order food using FoodService1 is recognized, the user's query may not have a sufficiently high match with a skill for ordering food using FoodService1, because the user's query does not sufficiently define constraints such as what restaurant to order from and/or what menu item to order. Accordingly, as shown at <NUM>, <NUM>, <NUM>, and <NUM>, computerized personal assistant <NUM> may pose further disambiguating questions to user <NUM> in order to recognize a new context, e.g., to recognize that user <NUM> intends to order food using FoodService1 and specifically intends to order a "veggie burrito" from "The Burrito Place. " Accordingly, computerized personal assistant <NUM> is configured to output a response at <NUM> acknowledging that an order was placed.

More generally, computerized personal assistant <NUM> is configured to output a series of predefined responses to pose a series of disambiguating questions whenever a match confidence for a current context is insufficiently high to exceed the predefined match confidence threshold with regard to one or more reference contexts associated with available and/or unavailable skills. For example, computerized personal assistant <NUM> may pose disambiguating questions to determine a particular skill (e.g., when an intent of a user is recognized but more than one skill may be applicable for serving the intent, as shown in <FIG>), and/or to determine constraints associated with the intent. In some examples, user <NUM> may ask computerized personal assistant <NUM> for additional options, e.g., if user <NUM> does not want to use FoodService1 or FoodService2, user <NUM> may ask if there are any other skills available for ordering food. Accordingly, computerized personal assistant <NUM> may provide further options when such options are available, e.g., by indicating that FoodService3 is another option. When no such further options are available, computerized personal assistant <NUM> is configured to provide a generic "fallback" response, e.g., to indicate that no further options are available for ordering food.

In some examples, computerized personal assistant <NUM> may automatically disambiguate a user intent based on a current context, in order to automatically determine what skill to use and/or to automatically determine constraints associated with the user intent.

In some examples, the current context of user <NUM> includes a user preference (e.g., a user preference stored in personalization database <NUM>), and assessing the match confidence for a candidate skill includes assessing a higher match confidence for the candidate skill based on the assistive action of the candidate skill satisfying the user preference. For example, computerized personal assistant <NUM> may have previously recognized a preference of user <NUM> to user FoodService1 instead of FoodService2 when possible, e.g., based on a frequency of user <NUM> using FoodService1 or based on user <NUM> explicitly stating a preference for using FoodService1 in the future. Accordingly, computerized personal assistant may recognize that user <NUM> likely wishes to use FoodService1 and may assess a higher match confidence for a skill for using FoodService1 as compared to a skill for using FoodService2.

In some examples, the current context of the user <NUM> includes, for each available skill of the plurality of available skills in skills database <NUM>, a utilization frequency indicating a degree of past utilization of the available skill by user <NUM>. Computerized personal assistant <NUM> is configured to facilitate discovery of underutilized skills having a relatively lower utilization frequency. For example, user <NUM> may be unaware of such underutilized skills. Accordingly, computerized personal assistant <NUM> is configured to assess a higher match confidence for a candidate skill based on the candidate skill having a relatively lower utilization frequency, as compared to other skills.

Returning to <FIG>, optionally, in some examples, at <NUM>, the current context of user <NUM> is a current activity context including state data of a computer service (e.g., cooperating computer service <NUM>). A candidate skill may have an associated reference context comprising a historical activity context including state data of a computer service (e.g., cooperating computer service <NUM>), wherein the state data is associated with the recognized activity being engaged in by a historical user of the computer service, and wherein the operation to assist the user pertains to the recognized activity. A high quality match between the current activity context and the historical activity context may indicate that a current activity of user <NUM> is the recognized activity. Accordingly, as the operation to assist the user pertains to the recognized activity, computerized personal assistant <NUM> may utilize the skill to assist the user with the recognized activity.

In some examples, computerized personal assistant <NUM> may automatically and/or proactively execute instructions defining a complementary help action of a candidate skill, responsive to assessing a match confidence for the candidate skill. For example, if user <NUM> is at an airport, user device <NUM> may detect that user <NUM> is at the airport (e.g., based on a global positioning system (GPS) sensor included in the user device <NUM>), and accordingly, a current context of user <NUM> recognized by computerized personal assistant <NUM> may include the fact that user <NUM> is at the airport. Skills database <NUM> may include a skill for providing travel assistance (e.g., by displaying itineraries, maps, and/or tickets as user <NUM> enters the airport and throughout travel). However, user <NUM> may not have discovered the skill for providing travel assistance. Accordingly, based on the current context and/or based on underutilization of the skill for providing travel assistance, computerized personal assistant <NUM> may proactively execute instructions to perform the complementary help action for the travel skill. For example, the complementary help action for the travel skill may include outputting a predefined response, at NLUI <NUM>, educating the user <NUM> about the travel skill. More generally, based on recognizing that the travel skill may be applicable to the user <NUM>'s current context, computerized personal assistant <NUM> may provide any suitable notifications to user <NUM>, e.g., by displaying information about the travel skill at a display of user device <NUM>.

In some examples, a skill may be associated with more than one predefined response, e.g., a predefined response for proactively performing a complementary help action and a predefined response for performing a complementary help action responsive to a query. In an example, a predefined response for proactively educating a user about a travel skill may include a brief description of the travel skill for educating the user about the skill (e.g., in case the user did not yet know about the skill), while a predefined response for educating the user about the travel skill responsive to a query may include a more in-depth description of how to use the travel skill including an exemplary query (e.g., in case the user may already know about the skill but may benefit from additional in-depth information about the skill). More generally, a skill may be associated with any suitable number of predefined responses, e.g., pertaining to different contexts, queries, intents, and/or constraints relating to the skill.

Returning to <FIG>, facilitating the discovery of skills may be continually improved by training classifier machine <NUM>. For example, classifier machine <NUM> may be trained using supervised training with regard to labelled data, e.g., based on labelled pairs where a labelled pair includes an exemplary current context and an exemplary reference context, labelled to indicate a quality of match between the contexts.

Validation machine <NUM> is configured to validate the operation of classifier machine <NUM> with regard to historical contexts (e.g., historical queries and historical activity contexts). Validation machine <NUM> is configured to receive a training example context, which may be any suitable context, e.g., a historical context previously recognized by computerized personal assistant <NUM>. For an available skill of the plurality of available skills included in skills database <NUM>, validation machine <NUM> is configured to operate classifier machine <NUM> to assess a training confidence indicating a quality of match between the training example context and a training reference context previously associated with the available skill. In some cases, the training confidence may be close to a predefined match confidence threshold, while not exceeding the predefined match confidence threshold. Accordingly, the training confidence may indicate that the training example context and the training reference context are a "near match," wherein the available skill is likely to be at least partially relevant to the training example context.

In some cases of "near match" between a training example context and a reference context, classifier machine <NUM> would potentially assess a lower and/or higher match confidence for the training example context and the training reference context if more training data were provided to classifier machine <NUM>. Accordingly, validation machine <NUM> is configured to bring the "near match" to the attention of a trainer (e.g., a user of computerized personal assistant <NUM>) in order to receive, from the trainer, additional training data and/or feedback.

Validation machine <NUM> is configured to receive, from the trainer, a second training reference context different than the first training reference context, and to update the skills database <NUM> to associate the second training reference context with the assistive action of the available skill. In some examples, the second training reference context may be semantically and/or lexically similar to the training example context. Accordingly, based on the association between the available skill and the second training reference context, classifier machine <NUM> may assess a higher match confidence for the training example context with regard to the available skill.

Alternately or additionally, validation machine <NUM> is configured to receive, from the trainer, an indication that the training example context does in fact match the training reference context, and to train classifier <NUM> so that after training, classifier <NUM> is configured to assess an updated match confidence that does exceed the predefined match confidence threshold. Data received from the trainer may confirm whether an available skill is actually a good match for a training example context. Accordingly, by updating classifier machine <NUM> and/or skills database <NUM> based on such data, validation machine <NUM> may cause classifier machine <NUM> to more appropriately assess match confidence for example contexts that may be a "near match" for available skills, thereby facilitating the discovery of such skills when the example contexts occur.

Classifier machine <NUM> may determine the quality of match in any suitable fashion, e.g., using any suitable combination of state-of-the-art and/or future natural language processing (NLP), ML and/or AI techniques, as will be described further below. For example, classifier pipeline <NUM> may be implemented using any suitable combination of such techniques.

More generally, components of computing environment <NUM>, including personalization database <NUM>, skills database <NUM>, validation engine <NUM>, computerized personal assistant <NUM> in general (and NLUI <NUM>, skill discovery machine <NUM>, and skill execution machine <NUM> in particular) may be implemented using any suitable combination of state-of-the-art and/or future NLP, ML, and/or AI techniques. For example, components of computing environment <NUM> may process natural language inputs, make decisions and/or assess confidence values for decisions, and more generally perform any methods and processes disclosed herein using NLP, ML, and/or AI techniques.

Non-limiting examples of techniques that may be incorporated in an implementation of computing environment <NUM> include support vector machines, multi-layer neural networks, convolutional neural networks (e.g., including temporal convolutional neural networks for processing natural language sentences), word embedding models (e.g., GloVe or Word2Vec), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom Filters, Neural Turing Machine and/or Neural Random Access Memory), unsupervised spatial and/or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and/or k-means clustering) and/or graphical models (e.g., Markov models, conditional random fields, and/or AI knowledge bases).

In some examples, methods and processes performed by components of computing environment <NUM> (e.g., method <NUM>) may be implemented using one or more differentiable functions, wherein a gradient of the differentiable functions may be calculated and/or estimated with regard to inputs and/or outputs of the differentiable functions (e.g., with regard to training data, and/or with regard to an objective function). Methods and processes of computing environment <NUM> may be at least partially determined by a set of trainable parameters of computing environment <NUM> (e.g., including trainable parameters for each component of computing environment <NUM>). Accordingly, trainable parameters of computing environment <NUM> may be adjusted through any suitable training procedure, in order to continually improve functioning of the methods and processes described herein.

Non-limiting examples of training procedures include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on classes derived from unsupervised clustering methods), reinforcement learning (e.g., deep Q learning based on feedback, such as feedback from user <NUM> regarding quality of assistance provided by computing assistant <NUM>) and/or generative adversarial neural network training methods. In some examples, a plurality of components of computing environment <NUM> may be trained simultaneously with regard to an objective function measuring performance of collective functioning of the plurality of components (e.g., with regard to feedback from user <NUM> or with regard to labelled training data), in order to improve such collective functioning. In some examples, one or more components of computing environment <NUM> may be trained independently of other components (e.g., offline training with regard to historical data).

In particular, such methods and processes may be implemented as a computer-application program or service, an API, a library, and/or other computer-program product.

For example, computing system <NUM> may be a computerized personal assistant <NUM>. Alternately or additionally, computing system <NUM> may be a personalization database <NUM>, a skills database <NUM>, a classifier machine <NUM>, a validation machine <NUM>, a cooperating computer service <NUM>, and/or a user device <NUM>.

In an example, a computerized personal assistant comprises: a natural language user interface (NLUI) including a microphone and a speaker; a skill discovery machine configured to communicatively couple to a computer database including a plurality of available skills for the computerized personal assistant, wherein the computer database defines, for a candidate skill: an assistive action defined by instructions executable by the computerized personal assistant to perform an operation to assist a user; a reference query previously associated with the assistive action; and a complementary help action previously associated with the assistive action, wherein the complementary help action is defined by instructions executable by the computerized personal assistant to output a predefined response to educate the user about the assistive action. In this example or any other example, the computerized personal assistant further comprises: a skill execution machine configured to: receive, from the NLUI, a current query captured by the microphone; responsive to receiving the current query, operate the skill discovery machine to assess a match confidence for the candidate skill, the match confidence indicating a quality of match between the current query and the reference query; execute the instructions defining the assistive action responsive to the match confidence exceeding a predefined match confidence threshold; and execute the instructions defining the complementary help action responsive to the match confidence not exceeding the predefined match confidence threshold, and output the predefined response via the speaker.

In an example, a method of facilitating the discovery of a skill of a computerized personal assistant comprises: communicatively coupling the computerized personal assistant to a computer database including a plurality of available skills for the computerized personal assistant, wherein the computer database defines, for a candidate skill: an assistive action defined by instructions executable by the computerized personal assistant to perform an operation to assist a user; a reference context previously associated with the assistive action; and a complementary help action previously associated with the assistive action, wherein the complementary help action is defined by instructions executable by the computerized personal assistant to output a predefined response to educate the user about the assistive action. In this example or any other example, the method further comprises: recognizing a current context of the user; operating a previously-trained machine learning classifier to assess a match confidence for the candidate skill, the match confidence indicating a quality of match between the current context and the reference context; executing the instructions defining the assistive action responsive to the match confidence exceeding a predefined match confidence threshold; and executing the instructions defining the complementary help action responsive to the match confidence not exceeding the predefined match confidence threshold. In this example or any other example, the reference context includes a seed query defined by an exemplary natural language sentence; the operation to assist the user pertains to the seed query; and the current context of the user includes a user query received via a natural language user interface. In this example or any other example, the reference context includes a historical activity context including state data of a computer service, the state data associated with a recognized activity of a historical user of the computer service; the operation to assist the user pertains to the recognized activity; and the current context of the user includes a current activity context including state data of the computer service, the state data indicating the user is using the computer service to engage in the recognized activity. In this example or any other example, executing the instructions defining the complementary help action of the candidate skill is performed proactively, responsive to assessing a match confidence for the candidate skill. In this example or any other example, the current context of the user includes a user preference; and assessing the match confidence includes assessing a higher match confidence for the candidate skill based on the assistive action satisfying the user preference. In this example or any other example, the current context of the user includes, for each available skill of the plurality of available skills, a utilization frequency indicating a degree of past utilization of the available skill by the user; and assessing the match confidence includes assessing a higher match confidence for the candidate skill based on the candidate skill having a relatively lower utilization frequency. In this example or any other example, the method further comprises: operating the previously-trained machine learning classifier to assess, for each available skill of the plurality of available skills, a candidate match confidence for the available skill; and ranking the plurality of available skills by candidate match confidence. In this example or any other example, the candidate skill is an available skill having a highest candidate match confidence. In this example or any other example, the computer database further includes an unavailable skill, and the method further comprises: operating the previously-trained machine learning classifier to assess an unavailable skill match confidence indicating a quality of match between the current context and a reference context previously associated with the unavailable skill; and responsive to the unavailable skill match confidence exceeding the predefined match confidence threshold, outputting a response describing the unavailable skill and educating the user about configuring the computerized personal assistant to cause the unavailable skill to become available. In this example or any other example, outputting the predefined response to educate the user about the assistive action includes outputting, via a NLUI, a natural language description of the assistive action. In this example or any other example, the predefined response to educate the user about the assistive action includes an exemplary query for which the previously-trained machine learning classifier is configured to output a match confidence value greater than the predefined match confidence threshold, so that the computerized personal assistant is configured to perform the operation to assist the user responsive to receiving the exemplary query. In this example or any other example, the complementary help action further defines a disambiguating question, and the method further comprises: outputting, via a natural language user interface (NLUI), the disambiguating question; receiving, via the NLUI, an answer to the disambiguating question; and responsive to receiving the answer, operating the computerized personal assistant to execute the instructions defining the assistive action of the candidate skill. In this example or any other example, the method further comprises: receiving a training example context; for an available skill of the plurality of available skills: operating the previously-trained machine learning classifier to assess a training confidence indicating a quality of match between the training example context and a first training reference context previously associated with the available skill, wherein the training confidence does not exceed the predefined match confidence threshold; receiving, from a trainer, a second training reference context; and updating the computer database to associate the second training reference context with the assistive action of the available skill. In this example or any other example, the method further comprises: receiving a training example context; for an available skill of the plurality of available skills: operating the previously-trained machine learning classifier to assess a training confidence indicating a quality of match between the training example context and a training reference context previously associated with the available skill, wherein the training confidence does not exceed the predefined match confidence threshold; receiving, from a trainer, an indication that the training example context matches the training reference context; and training the previously-trained machine learning classifier, so that a further-trained machine learning classifier is configured to assess an updated confidence, wherein the updated confidence exceeds the predefined match confidence threshold.

In an example, a computer system comprises: a logic machine; and a storage machine holding instructions executable by the logic machine to: communicatively couple to a computer database including a plurality of available skills for the computer system, wherein the computer database defines, for a candidate skill: an assistive action defined by instructions executable by the computerized personal assistant to perform an operation to assist a user; a reference context previously associated with the assistive action; and a complementary help action previously associated with the assistive action, wherein the complementary help action is defined by instructions executable by the computerized personal assistant to output a predefined response to educate the user about the assistive action. In this example or any other example, the instructions are further executable to: recognize a current context of the user; operate a previously-trained machine learning classifier to assess a match confidence for the candidate skill, the match confidence indicating a quality of match between the current context and the reference context; perform the assistive action responsive to the match confidence exceeding a predefined match confidence threshold; and perform the complementary help action responsive to the match confidence not exceeding the predefined match confidence threshold. In this example or any other example, the reference context includes a seed query defined by an exemplary natural language sentence; the operation to assist the user pertains to the seed query; and the current context of the user includes a user query received via a natural language user interface. In this example or any other example, the reference context includes a historical activity context including state data of a computer service, the state data associated with a recognized activity of a historical user of the computer service; the operation to assist the user pertains to the recognized activity; and the current context of the user includes a current activity context including state data of the computer service, the state data indicating the user is using the computer service to engage in the recognized activity. In this example or any other example, the complementary help action of the candidate skill is performed proactively, responsive to assessing a match confidence for the candidate skill. In this example or any other example, the current context of the user includes a user preference; and assessing the match confidence includes assessing a higher match confidence for the candidate skill based on the assistive action satisfying the user preference. In this example or any other example, the current context of the user includes, for each available skill of the plurality of available skills, a utilization frequency indicating a degree of past utilization of the available skill by the user; and assessing the match confidence includes assessing a higher match confidence for the candidate skill based on the candidate skill having a relatively lower utilization frequency.

Claim 1:
A computerized personal assistant (<NUM>), comprising:
a natural language user interface, NLUI (<NUM>), including a microphone (<NUM>) and a speaker (<NUM>);
a skill discovery machine (<NUM>) configured to communicatively couple to a computer database including a plurality of available skills for the computerized personal assistant (<NUM>), wherein the computer database defines, for a candidate skill:
an assistive action defined by instructions executable by the computerized personal assistant (<NUM>) to perform an operation to assist a user (<NUM>), wherein the assistive action comprises: gathering and outputting information and/or multimedia content; recording information for later retrieval; communicating with at least one other user; controlling electrical and/or mechanical appliances; or interacting with a cooperating computer service or device;
a reference context previously associated with the assistive action; and
a complementary help action previously associated with the assistive action, wherein the complementary help action is defined by instructions executable by the computerized personal assistant (<NUM>) to output a predefined response to educate the user (<NUM>) about the assistive action; and
wherein the computerized personal assistant (<NUM>) is configured to:
recognize a current context of the user (<NUM>);
operate a previously-trained machine learning classifier to assess a match confidence for the candidate skill, the match confidence indicating a quality of match between the current context and the reference context, including assessing a higher match confidence for the candidate skill based on the candidate skill having a relatively lower utilization frequency by the user (<NUM>), compared to other available skills of the plurality of available skills;
execute the instructions defining the assistive action responsive to the match confidence exceeding a predefined match confidence threshold; and
execute the instructions defining the complementary help action responsive to the match confidence not exceeding the predefined match confidence threshold.