Patent Description:
An automated assistant and/or other application(s) can provide a user with recommended content including video, audio, images, web pages, news articles, videos, recipes, blog posts, electronic books, memes, social media posts, and/or additional types of content. In some cases, content can be recommended in response to a specific request for the content by the user. Additionally or alternatively, content can be proactively rendered to the user without a specific request for content by the user.

Document <CIT> discloses a method and an apparatus for recommending information to users within a social network. The method builds a recommendation list with at least one two-tuple, where each two-tuple comprises a target user name and an information item and ranks the recommendation list by using two-tuples in the recommendation list as a basic unit. By selecting a two-tuple in the recommendation list, the user can recommend a corresponding information item to a user represented by a target user name. An apparatus is also provided by using a builder for building for a user a recommendation list comprising at least one two-tuple and a sorter for ranking the recommendation list by using two-tuples in the recommendation list as a basic unit, such that, by selecting a two-tuple in the recommendation list.

In <CIT> systems and methods are described for providing information to a recommending user reflecting an expected, or predicted, desirability of a recommendation of a media item for a potential recommendation recipient or a group of potential recommendation recipients. In one embodiment, the recommending user selects a media item to potentially recommend to other users. For each potential recommendation recipient or group of potential recommendation recipients, an expected desirability value reflecting an expected desirability of the media item is determined. The expected desirability values are then presented to the recommending user. Based on the expected desirability values, the recommending user then selects one or more of the potential recommendation recipients or groups of potential recommendation recipients to which to send a recommendation for the media item.

<CIT> discloses a system and method for providing content recommendation to users on a site. The system includes a processor and a memory storing instructions that, when executed, cause the system to: receive data describing a page on a site visited by a first user; query one or more sources to retrieve one or more candidate items for the first user; determine one or more source types related to the one or more candidate items; determine one or more scores for the one or more candidate items based on an interest profile and the one or more source types; rank the one or more candidate items; determine that an invitation trigger event causing a presentation of an invitation state occurs; determine that a recommendation trigger event causing a presentation of a recommendation state occurs during the invitation state; and provide a recommendation state to the first user.

<CIT> is related to a recommender system, a computer-implemented recommending method, a corresponding computer readable medium and a corresponding computer program. A recommender system is configured to send an electronic suggestion signal to a respective user of a user database in dependence of a first like-degree of a specific item, which has been determined based on items that the respective user has already recommended to his contacts. The electronic suggestion signal suggests the respective user to recommend the specific item to one or more of his contacts. In that way, personal based recommendations are stimulated.

Implementations described herein are directed towards generating candidate endorsements for items of content using an ensemble of nominators; selecting an endorsement from the candidate endorsements based on a determined score for the candidate endorsements; and rendering the items of content with the selected endorsements. An endorsement is a communication (e.g., natural language communication(s) and/or additional types of communications) indicating why a user should interact with an item of content. For example, a recommendation for a smart doorbell can have a corresponding endorsement of "users of your smart thermostat frequently use this device". An endorsement can include a single phrase, a dialog with multiple turns in the conversation, and/or additional communication(s) with the user. As used herein, a "dialog session" may include a logically self-contained exchange of one or more messages between a user and a computing system such as an automated assistant client (and in some cases, other human participants). A computing system may differentiate between multiple dialog sessions with a user based on various signals, such as passage of time, change of user context (e.g., location, before/during/after a scheduled meeting, etc.) between sessions, detection of one or more intervening interactions between the user and a client device other than the dialog (e.g., the user switches applications for a while, the user walks away from then later returns to a client device), locking/sleeping of the client device between sessions, change of client devices used to interface with one or more instances of the automated assistant, etc. User interaction with content can include, selecting, selecting and viewing for at least a threshold duration, and/or other interaction(s) with the content.

In a variety of implementations, an ensemble of nominators can be used in determining a group of candidate endorsements for a recommended item of content. As used herein, a nominator can describe a variety of types of logical structures implemented in software including a machine learning model used in processing an item of content to determine a candidate endorsement. Nominators can include a variety of types of nominators, where each type is used in generating a distinct type of endorsement. In many implementations, nominator types can include recommender nominators, summarization nominators, fact nominators, baseline nominators, and/or additional types of nominators.

In general, endorsements may provide some additional information about the item of content with which they are associated. For example, recommender nominators are used in generating an endorsement that provides an explanation of why a recommendation engine selected the item of content. In many implementations, recommender nominators can make use of integrated path gradients, concept activation vectors, rationalization tags, and/or additional techniques that provide an explanation for a given item of content. For example, a recommender nominator can generate an endorsement of "this automated assistant client device is recommended because you own a smart thermostat".

Summarization nominators can be used in generating an endorsement by mining documents (e.g., reviews of the content, forum posts on the content, descriptions of the content, and/or additional documents relating to the content). For example, a summarization nominator can generate an endorsement of "Many reviews said this was a great restaurant for a date".

Fact nominators are used in generating an endorsement based on facts related to the item of content. In a variety of implementations, the facts are unknown to the user. For example, a fact nominator can generate an endorsement of "This movie has a lot of fun car chases".

Baseline nominators are used in generating an endorsement based on heuristic(s) which, for example, can be used to perform computation(s) on meta-data corresponding with the recommended item. For example, endorsements generated using a baseline nominator can include: "<NUM>,<NUM> people liked this"; "Ends in <NUM> days"; "Hypothetical Band will be attending"; "You've visited this place before"; "You have rated this place <NUM> stars".

In a variety of implementations, an ensemble of nominators can include a variety of types of nominators (e.g., the ensemble includes a recommender nominator, a facts nominator, and a baseline nominator). Furthermore, the ensemble of nominators can contain multiple of the same type of nominator (e.g., a first facts nominator, a second facts nominator, and a third facts nominator). Additional or alternative combinations of nominators can be included in an ensemble of nominators in accordance with a variety of implementations.

In some implementations, each nominator is used in generating a single endorsement for each item of content. In other implementations, each nominator is used in generating multiple endorsements for each item of content (e.g., a nominator generates two endorsements, three endorsements, and/or additional numbers of endorsements).

An endorsement can be selected for presentation to a user based on scores determined for each of the candidate endorsements. The score can be indicative of the likelihood the user will interact with an item of content based on a specific candidate endorsement. A scoring system can be utilized that generates a uniform type of score across the various types of nominators. Different types of nominators can utilize a variety of data sources in generating a candidate endorsement. For instance, a summarization monitor can utilize content reviews, a baseline nominator can utilize content item metadata, etc..

In many implementations, a score for a candidate endorsement can be determined utilizing a variety of sub-scores, such as a nominator sub-score, a strategy sub-score, an entity personalization sub-score, and/or additional sub score(s). A nominator sub-score can indicate how relevant the nominator views that endorsement to be, without using any user context (and can often be determined using the nominator itself). This score can be calibrated, for example, by using offline processing that monitors the distribution of scores of each nominator across a sample of items. In many implementations, a nominator sub-score is dependent on the endorsement being scored as well as the item being recommended to the user.

Additionally or alternatively, a strategy sub-score can predict how effective a particular endorsement strategy (i.e., a particular nominator) will be for a specific user. For example, one or more attributes of the user can be included in a user profile, such as an attribute indicating a user is cost-conscious. For cost-conscious users it may be advantageous to boost a price-based endorsement strategy. In other words, the strategy sub-score can be dependent on the nominator generating the endorsement as well as a user profile of the user to which content will be recommended.

Furthermore, an entity personalization sub-score can determine the extent to which the candidate endorsement is relevant to a particular user. In many implementations, the entity personalization sub-score can be dependent on the candidate endorsement being scored as well as the user profile of the user to which the content is being recommended. The entity personalization sub-score can capture how well topical interests of the user (e.g., as indicated by the user profile) are captured by the information contained in the endorsement.

Once a score is determined for each candidate endorsement, an endorsement can be selected to be rendered and/or spoken to the user with the recommended item of content. In many implementations, the candidate endorsement with the highest score is selected. Additionally or alternatively, in implementations which include rendering one or more endorsements with each item of content, each candidate endorsement with a score above a threshold value can be selected for rendering to the user with the item of content.

In many implementations, it can be determined whether a user interacts with a specific item of rendered content. Based on this determination, one or more nominators can be updated. For example, the nominator used in generating a selected endorsement can be updated when a user interacts with the item of content and/or when the user does not interact with the item of content. Additional or alternative nominators can be updated based on the user interaction with the content. For example, all other nominators (excluding the nominator used in generating a selected endorsement) can be updated with a user interacts with the item of content.

Recommended content can include an initial order for rendering the items of content for the user. In many implementations, the score of the selected endorsements corresponding with the items of content can be used to change the order of the items of content prior to rendering the items of content.

Accordingly, various implementations set forth techniques for enabling navigation to user desired content - and to do so in a manner that enables navigation to the content to be more efficient through the use of endorsements generated by an ensemble of nominators. For example, candidate endorsements can be identified based on an ensemble of nominators personalized for a user. These nominators can be personalized based on whether a user interacted with an item of content based on the endorsement. By providing a user with an endorsement of content highly relevant to the user, the endorsements can be used to more efficiently identify content desired by the user.

System resources (e.g., processor cycles, memory, battery power, and/or additional resources of a computing system) can be conserved by providing recommended items of content with selected endorsements. For example, a user can more quickly make a determination whether he or she will interact with an item of content, and in some cases can make a determination whether to interact with an item of content without having to select the item of content. In other words, endorsements can reduce the likelihood a user will select an item of content and quickly make a determination they do not wish to interact with the entire item of content, thus causing the user to select an additional item of content. Reduced interactions with multiple items of content can shorten the duration of the overall user-system interaction, conserving various resources through enabling the interaction to cease earlier. Additionally or alternatively, the time saved from the shortened duration of the overall user-system interaction can lead to higher user satisfaction with the product.

Other implementations may include a non-transitory computer readable storage medium storing instructions executable by one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s)), and/or tensor processing unit(s) (TPU(s)) to perform a method such as one or more of the methods described above and/or elsewhere herein. Yet other implementations may include a system of one or more computers and/or one or more robots that include one or more processors operable to execute stored instructions to perform a method such as one or more of the methods described above and/or elsewhere herein.

<FIG> illustrates an example environment in which implementations disclosed herein may be implemented. The example environment of <FIG> includes client device <NUM> which may be associated with user interface input/output device(s) <NUM>, content <NUM>, and/or nominators <NUM>. Client device <NUM> can include recommendation engine <NUM> for generating recommended items of content to be presented to a user of the client device, and can include endorsement engine <NUM> for determining an endorsement to present with each recommended item of content. In many implementations, endorsement engine <NUM> can include a variety of additional and/or alternative engines including nominator engine <NUM>, endorsement selection engine <NUM>, scorer engine <NUM>, reranker engine <NUM>, and/or additional engine(s) (not depicted). The user interface input/output device(s) <NUM> may include, for example, a physical keyboard, a touch screen (e.g., implementing a virtual keyboard or other textual input mechanism), a microphone, a camera, a display screen, speaker(s), and/or additional device(s) to capture user input and/or render output for a user. The user interface input/output devices <NUM> may be incorporated with one or more client devices <NUM> of a user. For example, a mobile phone of the user may include the user interface input/output device(s) <NUM>; a standalone personal assistant hardware device may include the user interface input/output device(s) <NUM>; or a first computing device may include user interface input device(s) <NUM> and a separate computing device may include user interface output device(s) <NUM>; etc..

In some implementations, client device <NUM> may include an automated assistant (not depicted), and all or aspects of the automated assistant may be implemented on computing device(s) that are separate and remote from the computing device that contains user interface input/output device(s) <NUM> (e.g., all or aspects may be implemented "in the cloud"). In some of those implementations, those aspects of the automated assistant may communicate with client device <NUM> via one or more networks (not depicted) such as a local area network (LAN) and/or a wide area network (WAN) (e.g., the internet).

Some non-limiting examples of client device <NUM> that may include the user interface input/output device(s) <NUM> include one or more of: a desktop computing device, a laptop computing device, a standalone hardware device at least in part dedicated to an automated assistant, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the user (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative client devices may be provided.

Client device <NUM> may include one or more memories for storage of data and software applications, one or more processors for accessing data and executing applications, and other components that facilitate communication over a network. The operations performed by client device <NUM> may be distributed across multiple computing devices. For example, computing programs running on one or more computers in one or more locations can be coupled to each other through a network.

As illustrated in <FIG>, client device <NUM> can determine recommended items of content and endorsements for those items of content. Output device(s) of user interface input/output device (s) <NUM> can be utilized in rendering the recommended items of content and the corresponding endorsements for the user of the client device. As used herein, rendering content includes rendering an entire item of content, rendering a hyperlink to an item of content, rendering a thumbnail image identifying the item of content, rendering a summary of an item of content and/or additional representation(s) of an item of content. Recommended content can include a variety of digital content including: video, audio, image, web pages, news articles, videos, recipes, blog posts, electronic books, memes, social media posts, restaurant recommendations, and/or additional types of content a user can interact with via client device <NUM>.

Recommendation engine <NUM> can generate one or more recommended items of content for a user, such as recommend one or more items of content <NUM>. For example, recommendation engine can generate content <NUM>, content <NUM>, and content <NUM> as recommended content for the user. In many implementations, the recommended items of content have an inherent order determined by recommendation engine <NUM>. In some implementations, recommendation engine <NUM> can generate recommended items of content in response to a request from a user (e.g., a natural language spoken request from a user for content recommendations). In additional and/or alternative implementations, recommendation engine <NUM> can preemptively generate recommended items of content for a user. In a variety of implementations, recommendation engine <NUM> can be remote from client device <NUM>, such as recommendation engine <NUM> determining recommended items of content via cloud computing and transmitting the recommended items of content to the client device <NUM> via the Internet. Additionally or alternatively, recommendation engine <NUM> can be integrated as part of endorsement engine <NUM>. In other words, endorsement engine <NUM> can generate recommended items of content as well as determine endorsements to present with the recommended items of content.

Although endorsement engine <NUM> is shown as integral with client device <NUM>, this is not meant to be limiting. In various implementations, one or more components of endorsement engine <NUM> may be implemented in whole or in part on remote computing system(s) forming a "cloud" infrastructure. Endorsement engine <NUM> can include nominator engine <NUM>, endorsement selection engine <NUM>, scorer engine <NUM>, reranker engine <NUM>, and/or additional engines (not depicted). Nominator engine <NUM>, can generate candidate endorsements for recommended items of content using an ensemble of nominators <NUM>. In some implementations, nominator engine <NUM> can select the ensemble of nominators from nominators <NUM> to be utilized in generating candidate endorsements for recommended items of content. Different ensemble(s) of nominators can be associated with different types of content. For example, the ensemble of: nominator A, nominator B, and nominator C can be utilized in generating endorsements for recipe recommendations. In contrast, the ensemble of: nominator D, nominator E, nominator F, and nominator G can be utilized in generating endorsements for recommended news articles. Although this example illustrates nonoverlapping ensembles of nominators, nominator(s) included in a first ensemble can overlap with nominator(s) included in a second ensemble (e.g., the ensemble of nominators for a first type of content includes nominators A, B, C, and D, and the ensemble of nominators for a second type of content includes nominators B, C, D, E and F). Additional and/or alternative combinations of nominators can be included in ensembles of nominators in accordance with implementations described herein.

Nominator engine <NUM> can utilize an ensemble of nominators to generate candidate endorsements for recommended items of content. Each nominator in the ensemble can be utilized to generate candidate endorsement(s) for each recommended item of content. In many implementations, each nominator in the ensemble can be utilized to generate a single candidate endorsement. Additionally or alternatively, in some implementations, each nominator in the ensemble can be utilized to generate two, three, four, and/or other number(s) of candidate endorsements. Furthermore, in many implementations, each nominator in the ensemble can generate a distinct candidate recommendation. In some other implementations, two or more nominators can generate the same or substantially similar candidate endorsements. For example, nominator A can generate the candidate endorsement of "reviews say this a romantic restaurant for a date" and nominator B can generate a substantially similar endorsement of "reviewers say this is a romantic restaurant for a date".

Additionally or alternatively, nominator engine <NUM> can be used to update one or more nominators in response to whether the user interacted with the recommended item of content. For example, one or more parameters (e.g., one or more network weights) of one or more nominators can be updated through backpropagation. In many implementations, nominators can be personalized for a user by associating the nominators with a known profile for the user. In other words, the nominators of the user profile can be updated based on when the user associated with the profile interacts with (or does not interact with) a recommended item of content. A known user profile can be associated with the user based on one or more determined characteristics such as voice signatures (e.g., text independent voice identification, text dependent voice identification, and/or other voice identification), facial recognition, image feeds, motion characteristics, and/or other data.

Endorsement selection engine <NUM> can select an endorsement to present with each recommended item of content. The endorsement to present with an item of content can be selected based on the scores of each candidate endorsement determined using scorer engine <NUM> (as described below). In many implementations, endorsement selection engine <NUM> can select the candidate endorsement with the highest score to present with the recommended item of content. Additionally or alternatively, endorsement selection engine <NUM> can select more than one candidate endorsement to present with an item of content (e.g., item of content A is rendered with endorsement <NUM> and endorsement <NUM>). In some such implementations, endorsement selection engine <NUM> can select the candidate endorsements with the highest scores, select all candidate endorsements with a score satisfying a criterion (such as exceeding a threshold value), and/or select candidate endorsements based on other criteria.

Scorer engine <NUM> can determine a score for each candidate endorsement in accordance with a variety of implementations. A score can be in indication of the likelihood a user will interact with an item of content based on the endorsement. In many implementations, a score can be determined utilizing a variety of sub-scores including a nominator sub-score, a strategy sub-score, an entity personalization sub-score, and/or additional sub-score(s). Utilizing a variety of types of score information can provide a uniform scoring across nominators that determine candidate recommendations by accessing a variety of different data sources. A nominator sub-score can depend on the candidate endorsement being scored as well as the item of content recommended to the user. In other words, a nominator sub-score can indicate how relevant the nominator which generated the candidate endorsement views the candidate endorsement. In many implementations, a nominator sub-score can be determined using the nominator itself as part of the process of generating the candidate endorsement. Additionally or alternatively, scorer engine <NUM> can generate the nominator sub-score.

Scorer engine <NUM> can determine the strategy sub-score corresponding to a candidate endorsement. In many implementations, the strategy sub-score can depend on the nominator generating the candidate endorsement as well as the user to which the content is recommended. In other words, the strategy sub-score provides an indication of how effective a particular endorsement strategy (i.e., a particular nominator) will be in generating an endorsement for a particular user.

Furthermore, scorer engine <NUM> can determine an entity personalization sub-score corresponding to a candidate endorsement. In many implementations, the entity personalization sub-score can be dependent on the candidate endorsement being scored as well as a user profile associated with the user for which the item of content is recommended. In other words, the entity personalization sub-score can provide an indication of how relevant the candidate endorsement is to a particular user.

As described above, recommended items of content can include an order of how to render the items of content for the user from the recommendation engine. For example, the recommendation can determine item of content A to render first, item of content B to render second, and item of content C to render third. Reranker engine <NUM> can utilize the scores of the endorsements selected to provide with the items of content to change the order the items of content are rendered for the user. In other words, reranker engine can be utilized to change the order of items of content are displayed to the user by rendering items of content the user is more likely to interact with (based on the endorsement score) first.

Endorsement engine <NUM> can additionally or alternatively incorporate signals for other engines of the client device when scoring and/or raking candidate endorsements including: a sentiment analyzer, a syntactic checker, a consistency checker, and/or additional signals to increase the accuracy of selected endorsements and/or increase a user's ability to understand endorsements.

<FIG> illustrates an example of utilizing an ensemble of nominators to generate item endorsements in accordance with various implementations described herein. Recommendation engine <NUM> (which may correspond to and/or share characteristics with recommendation engine <NUM> in <FIG>) selects one or more items of content <NUM> to recommend to a user. For example, recommendation engine <NUM> can select item <NUM>, item <NUM><NUM>, and item N <NUM>. Additional and/or alternative items of content can be selected for recommendation to the user. In a variety of implementations, recommendation engine <NUM> selects items of content <NUM>, <NUM>, <NUM> in response to a request for content by the user. For example, a user can provide a spoken request of "show me devices that will work with my smart thermostat", and recommendation engine <NUM> can determine item <NUM> - "smart smoke detector", item <NUM> - "smart camera A", and item N - "smart camera B" as items of content responsive to the user's request. Additionally or alternatively, recommendation engine <NUM> can preemptively determine items of content to recommend to the user. For example, the system can determine a user is near the client device, and can preemptively determine recommended items of content <NUM>, <NUM>, <NUM>.

Recommended items of content <NUM>, <NUM>, <NUM> can be processed using an ensemble of nominators such as nominator <NUM><NUM>, nominator <NUM><NUM>, and nominator M <NUM>. In many implementations, each nominator <NUM>, <NUM>, <NUM> can determine a candidate endorsement corresponding to each recommended item of content. Item of content <NUM> and associated candidate endorsements <NUM> can include endorsement <NUM> (determined using nominator <NUM>), endorsement <NUM> (determined using nominator <NUM><NUM>), and endorsement M (determined using nominator M <NUM>). Similarly, item of content <NUM> and associated candidate endorsements <NUM> can include endorsement <NUM> (determined using nominator <NUM><NUM>), endorsement <NUM> (determined using nominator <NUM><NUM>), and endorsement M (determined using nominator M <NUM>). Additionally or alternatively, item of content N and associated candidate endorsements <NUM> can include endorsement <NUM> (determined using nominator <NUM><NUM>), endorsement <NUM> (determined using nominator <NUM><NUM>), and endorsement M (determined using nominator M <NUM>).

Scorer engine <NUM> (which may correspond to or share characteristics with scorer engine <NUM> in <FIG>) can determine a score for each candidate endorsement, as well as select a candidate endorsement to present with the item of content. For example, for item of content <NUM> and corresponding candidate endorsements <NUM>, scorer engine <NUM> can determine a score for candidate endorsement <NUM>, candidate endorsement <NUM>, and candidate endorsement M. Similarly, scorer engine <NUM> can determine a score for each candidate endorsement of item <NUM> and corresponding candidate endorsements <NUM>, as well as determine a score for each candidate endorsement of item N and corresponding candidate endorsements <NUM>. In many implementations, a score can be based on a variety of sub-scores including a nominator sub-score, a strategy sub-score, and an entity personalization sub-score. Determining scores of candidate endorsements is described herein with respect to process <NUM> of <FIG>.

In many implementations, recommended items of content and the selected endorsements <NUM> can include, for example, item <NUM>, endorsement i; item <NUM>, endorsement j; and item N, endorsement k. Recommended items of content and selected endorsements <NUM> can be rendered for the user via the client device.

<FIG> is a flowchart illustrating an example process <NUM> of rendering items of content and corresponding endorsements, where the endorsements are selected from a plurality of candidate endorsements determined using an ensemble of nominators according to implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of client device <NUM> of <FIG>. Moreover, while operations of process <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and/or added.

At block <NUM>, the system receives items of content recommended for a user of a client device. Recommended content items can be determined, for instance, using a recommendation engine such as recommendation engine <NUM> in <FIG> or recommendation engine <NUM> in <FIG>. In many implementations, the recommendation engine is integrated locally with the system (e.g., the recommendation engine is local to a client device). Additionally or alternatively, the recommendation engine can be remote from the system (e.g., the recommendation engine is in the cloud and provides recommended content to a client device via the internet). For example, the system can receive recipes recommended for the user such as: "banana pancakes recipe", "French toast recipe" and "breakfast casserole recipe".

At block <NUM>, the system determines, for each recommended item of content, candidate endorsements using an ensemble of nominators. Candidate endorsements for a recommended item of content can be determined using an ensemble of nominators, where each nominator is used in determining a corresponding candidate endorsement. Process <NUM> of <FIG> as described herein illustrates a process of determining candidate endorsements using an ensemble of nominators. For example, candidate endorsements of "this recipe was selected because you enjoy breakfast recipes", "this recipe is low in calories", "this recipe was invented by Hypothetical Famous Chef", and "<NUM>,<NUM> people liked this recipe" can be generated for the recommended item of content "banana pancakes recipe".

At block <NUM>, the system selects an endorsement to present with each item of content using a score determined for each candidate endorsement. In many implementations, the score for each candidate endorsement can be determined based on a nominator sub-score, a strategy sub-score, an entity personalization sub-score, and/or additional factors. Determining scores for candidate endorsements is described herein with respect to process <NUM> of <FIG>. Furthermore, an endorsement can be selected from the candidate endorsements based on the score to present with the item of content to the user. For example, the candidate endorsement with the highest score can be selected to present with the item of content. For example, candidate endorsements for the recommended item of content "banana pancakes recipe" can be scored as: "this recipe was selected because you enjoy breakfast recipes" - score <NUM>; "this recipe is low in calories" - score <NUM> "this recipe was invented by Hypothetical Famous Chef" - score <NUM>; and "<NUM>,<NUM> people liked this recipe" - score <NUM>. In many implementations, the system can select the candidate endorsement with the highest score to present to the user with the item of content (e.g., "this recipe is low in calories" is selected because it has the highest score (<NUM>) of scores <NUM>, <NUM>, <NUM>, and <NUM>). Additionally or alternatively, the system can select the candidate endorsement with the lowest score to present with the item of content (e.g., "this recipe was selected because you enjoy breakfast recipes" is selected because it has the lowest score (<NUM>) out of scores <NUM>, <NUM>, <NUM>, and <NUM>).

In a variety of implementations, the system can select candidate endorsements with a score over a threshold value. For example, the system can select "this recipe is low in calories" and "this recipe was invented by Hypothetical Famous Chef" as having a score over a threshold value <NUM>. As another example, the system can select "this recipe is low in calories", "this recipe was invented by Hypothetical Famous Chef", and "<NUM>,<NUM> users liked this recipe" can be selected as endorsements as having a score over a threshold value of <NUM>.

In many implementations, endorsements to present with items of content can be selected to increase diversity of the selected for endorsements. For example, the system can ensure a diversity of endorsement to prevent all endorsements from being similar (e.g., to prevent every endorsement presented with the recommended recipes from being "this recipe is low in calories"). In many implementations, determinantal point processes can be utilized for diversity of endorsements.

At block <NUM>, the system renders the recommended items of content and the corresponding selected endorsements for the user via the client device. For example, the system can render the recommended items of content and the selected endorsements via a screen, a speaker, and/or additional user interface output device(s).

<FIG> is a flowchart illustrating an example process <NUM> of generating candidate endorsements according to implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of client device <NUM> of <FIG>. Moreover, while operations of process <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and/or added.

At block <NUM>, the system selects an unprocessed item of content. For example, an item of content without corresponding candidate endorsements can be selected.

At block <NUM>, the system processes the item of content using a nominator to generate a candidate endorsement. For example, a nominator in the ensemble of nominators that has not yet been used in generating a candidate endorsement can be selected.

At block <NUM>, the system determines whether any nominators in the ensemble of nominators is capable of generating further candidate endorsements for the item of content. If any nominators remain, the system proceeds back to block <NUM> and processes the item of content using an unused nominator to generate an additional candidate endorsement before proceeding to block <NUM> and determining if any nominators in the ensemble have not been utilized. If all nominators in the ensemble of nominators have been utilized in generating candidate endorsements, the system proceeds to block <NUM>.

At block <NUM>, the system determines whether any recommended items of content have not been processed using the ensemble of nominators to generate candidate endorsements. If candidate endorsements have not been determined for recommend item(s) of content, the system proceeds back to block <NUM> and an unprocessed item of content is selected before proceeding to blocks <NUM>, <NUM>, and <NUM>. If candidate endorsements have been determined for all recommended items of content, the system ends.

<FIG> is a flowchart illustrating an example process <NUM> of determining scores for candidate endorsements and selecting an endorsement to present with items of content according to implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as one or more components of client device <NUM> of <FIG>. Moreover, while operations of process <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and/or added.

At block <NUM>, the system selects an unprocessed recommended item of content. For example, the system can determine an item of content without a selected endorsement (i.e., the item of content has corresponding candidate endorsements, and an endorsement to present with the item of content has not been selected from the candidate endorsements).

At block <NUM>, the system selects an unscored candidate endorsement corresponding with the item of content selected at block <NUM>. As an illustrative example in accordance with many implementations disclosed herein, a user A can be recommended item of content B, where candidate endorsement C is determined for recommended item of content B using nominator D.

At block <NUM>, the system determines a nominator sub-score based on the candidate endorsement and the selected item of content. For example, the nominator subs-core can be determined using the candidate endorsement C and the recommended item of content B.

At block <NUM>, the system determines a strategy sub-score based on the nominator utilized in generating the candidate endorsement and a profile of the user. For example, the strategy sub-score can be determined using nominator D and one or more attributes of a user profile associated with user A.

At block <NUM>, the system determines an entity personalization sub-score based on the candidate endorsement (i.e., the candidate endorsement selected at block <NUM>) and the profile of the user. For example, the entity personalization sub-score can be determined using candidate endorsement C and one or more attributes of the user profile associated with user A.

At block <NUM>, the system determines a score for the candidate endorsement using the nominator sub-score determined at block <NUM>, the strategy sub-score determined at block <NUM>, the entity personalization sub-score determined at block <NUM>, and/or additional score components (e.g., additional sub-score(s), additional user information, etc.).

At block <NUM>, the system determines whether any candidate endorsements corresponding with the item of content selected at block <NUM> have not been scored. If candidate endorsement(s) are unscored, the system proceeds back to block <NUM> to select an unscored candidate endorsement, before proceeding to blocks <NUM>, <NUM>, <NUM>, and <NUM>. If all candidate endorsements corresponding with the selected item of content have been scored, the system proceeds to block <NUM>.

At block <NUM>, the system selects an endorsement to present with the item of content using the scores of the candidate endorsements corresponding with that item of content. For example, the system can select the candidate endorsement with the highest score.

At block <NUM>, the system determines whether any recommended items of content with corresponding candidate endorsements have not been processed. If so, the system proceeds back to block <NUM> and selects an unprocessed item of content with corresponding candidate endorsements before proceeding to blocks <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. If the system has selected an endorsement to present with every recommended item of content, the system proceeds to block <NUM>.

At block <NUM>, the system renders the items of content and the selected endorsements for the user. Optionally, the system can change the order one or more items of content are rendered for the user based on the endorsement scores of the selected endorsements. For example, recommended items of content can include item A, item B, and item C, where item A should be rendered first, item B should be rendered second, and item C should be rendered third. Item A can have a selected endorsement with a score of <NUM>, item B can have a selected endorsement with a score of <NUM>, and item C can have a selected endorsement with a score of <NUM>. In many implementations, the items of content can be rendered in a new order, where item B is rendered first, item C is rendered second, and item A is rendered third, based on the scores of the selected endorsements.

Turning to <FIG>, an example environment in which implementations disclosed herein can be implemented. <FIG> includes a client computing device <NUM>, which execute an instance of an automated assistant client <NUM>. One or more cloud-based automated assistant components <NUM> can be implemented on one or more computing systems (collectively referred to as a "cloud" computing system) that are communicatively coupled to client device <NUM> via one or more local and/or wide area networks (e.g., the Internet) indicated generally at <NUM>.

An instance of an automated assistant client <NUM>, by way of its interactions with one or more cloud-based automated assistant components <NUM>, may form what appears to be, from the user's perspective, a logical instance of an automated assistant <NUM> with which the user may engage in a human-to-computer dialog. It thus should be understood that in some implementations, a user that engages with an automated assistant client <NUM> executing on client device <NUM> may, in effect, engage with his or her own logical instance of an automated assistant <NUM>. For the sakes of brevity and simplicity, the term "automated assistant" as used herein as "serving" a particular user will often refer to the combination of an automated assistant client <NUM> executing on a client device <NUM> operated by the user and one or more cloud-based automated assistant components <NUM> (which may be shared amongst multiple automated assistant clients of multiple client computing devices). It should also be understood that in some implementations, automated assistant <NUM> may respond to a request from any user regardless of whether the user is actually "served" by that particular instance of automated assistant <NUM>.

The client computing device <NUM> may be, for example: a desktop computing device, a laptop computing device, a tablet computing device, a mobile smartphone computing device, a standalone interactive speaker, a smart appliance, and/or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative client computing devices may be provided. Additionally or alternatively, operations of client computing device <NUM> may be distributed between multiple computing devices. For example, one or more operations of client computing device <NUM> may be distributed between a mobile smartphone and a vehicle computing device. Furthermore, operations of client computing device <NUM> may be repeated between multiple computing devices (which in some cases may be communicatively coupled). As a further example, a mobile smartphone as well as a vehicle interface device may each implement operations of automated assistant <NUM>, such as a mobile smartphone and a vehicle interface device both including an invocation engine (described below). In various implementations, the client computing device <NUM> may optionally operate one or more other applications that are in additional to automated assistant client <NUM>, such as a message exchange client (e.g., SMS, MMS, online chat), a browser, and so forth. In some of those various implementations, one or more of the other applications can optionally interface (e.g. via an application programming interface) with the automated assistant <NUM>, or include their own instance of an automated assistant application (that may also interface with the cloud-based automated assistant component(s) <NUM>).

Automated assistant <NUM> engages in human-to-computer dialog sessions with a user via user interface input and output devices of the client device (not pictured). To preserve user privacy and/or to conserve resources, in many situations a user must often explicitly invoke the automated assistant <NUM> before the automated assistant will fully process a spoken utterance. The explicit invocation of the automated assistant <NUM> can occur in response to certain user interface input received at the client device <NUM>. For example, user interface inputs that can invoke the automated assistant <NUM> via the client device <NUM> can optionally include actuations of a hardware and/or virtual button of the client device <NUM>. Moreover, the automated assistant client can include one or more local engines <NUM>, such as an invocation engine that is operable to detect the presence of one or more spoken invocation phrases. The invocation engine can invoke the automated assistant <NUM> in response to detection of one or more of the spoken invocation phrases. For example, the invocation engine can invoke the automated assistant <NUM> in response to detecting a spoken invocation phrase such as "Hey Assistant", "OK Assistant", and/or "Assistant". The invocation engine can continuously process (e.g., if not in an "inactive" mode) a stream of audio data frames that are based on output from one or more microphones of the client device <NUM>, to monitor for an occurrence of a spoken invocation phrase. While monitoring for the occurrence of the spoken invocation phrase, the invocation engine discards (e.g., after temporary storage in a buffer) any audio data frames that do not include the spoken invocation phrase. However, when the invocation engine detects an occurrence of a spoken invocation phrase in processed audio data frames, the invocation engine can invoke the automated assistant <NUM>. As used herein, "invoking" the automated assistant <NUM> can include causing one or more previously inactive functions of the automated assistant <NUM> to be activated. For example, invoking the automated assistant <NUM> can include causing one or more local engines <NUM> and/or cloud-based automated assistant components <NUM> to further process audio data frames based on which the invocation phrase was detected, and/or one or more following audio data frames (whereas prior to invoking no further processing of audio data frames was occurring).

The one or more local engine(s) <NUM> of automated assistant <NUM> are optional, and can include, for example, the invocation engine described above, a local speech-to-text ("STT") engine (that converts captured audio to text), a local text-to-speech ("TTS") engine (that converts text to speech), a local natural language processor (that determines semantic meaning of audio and/or text converted from audio), and/or other local components. Because the client device <NUM> is relatively constrained in terms of computing resources (e.g., processor cycles, memory, battery, etc.), the local engines <NUM> may have limited functionality relative to any counterparts that are included in cloud-based automated assistant components <NUM>.

Automated assistant client <NUM> can additionally include a content agent engine (not pictured) which can be utilized by automated assistant client <NUM> in accordance with a variety of implementations including: generating a content agent, determining content related to a user request using a content agent, determining content using a content agent without receiving a user request, etc..

Cloud-based automated assistant components <NUM> leverage the virtually limitless resources of the cloud to perform more robust and/or more accurate processing of audio data, and/or other user interface input, relative to any counterparts of the local engine(s) <NUM>. Again, in various implementations, the client device <NUM> can provide audio data and/or other data to the cloud-based automated assistant components <NUM> in response to the invocation engine detecting a spoken invocation phrase, or detecting some other explicit invocation of the automated assistant <NUM>.

The illustrated cloud-based automated assistant components <NUM> include a cloud-based TTS module <NUM>, a cloud-based STT module <NUM>, and a natural language processor <NUM>. In some implementations, one or more of the engines and/or modules of automated assistant <NUM> may be omitted, combined, and/or implemented in a component that is separate from automated assistant <NUM>. Further, in some implementations automated assistant <NUM> can include additional and/or alternative engines and/or modules.

Cloud-based STT module <NUM> can convert audio data into text, which may then be provided to natural language processor <NUM>. In various implementations, the cloud-based STT module <NUM> can covert audio data into text based at least in part on indications of speaker labels and assignments that are provided by an assignment engine (not illustrated).

Cloud-based TTS module <NUM> can convert textual data (e.g., natural language responses formulated by automated assistant <NUM>) into computer-generated speech output. In some implementations, TTS module <NUM> may provide the computer-generated speech output to client device <NUM> to be output directly, e.g., using one or more speakers. In other implementations, textual data (e.g., natural language responses) generated by automated assistant <NUM> may be provided to one of the local engine(s) <NUM>, which may then convert the textual data into computer-generated speech that is output locally.

Natural language processor <NUM> of automated assistant <NUM> processes free form natural language input and generates, based on the natural language input, annotated output for use by one or more other components of the automated assistant <NUM>. For example, the natural language processor <NUM> can process natural language free-form input that is textual input that is a conversion, by STT module <NUM>, of audio data provided by a user via client device <NUM>. The generated annotated output may include one or more annotations of the natural language input and optionally one or more (e.g., all) of the terms of the natural language input. In some implementations, the natural language processor <NUM> is configured to identify and annotate various types of grammatical information in natural language input. For example, the natural language processor <NUM> may include a part of speech tagger (not depicted) configured to annotate terms with their grammatical roles. Also, for example, in some implementations the natural language processor <NUM> may additionally and/or alternatively include a dependency parser (not depicted) configured to determine syntactic relationships between terms in natural language input.

In some implementations, the natural language processor <NUM> may additionally and/or alternatively include an entity tagger (not depicted) configured to annotate entity references in one or more segments such as references to people (including, for instance, literary characters, celebrities, public figures, etc.), organizations, locations (real and imaginary), and so forth. The entity tagger of the natural language processor <NUM> may annotate references to an entity at a high level of granularity (e.g., to enable identification of all references to an entity class such as people) and/or a lower level of granularity (e.g., to enable identification of all references to a particular entity such as a particular person). The entity tagger may rely on content of the natural language input to resolve a particular entity and/or may optionally communicate with a knowledge graph or other entity database to resolve a particular entity.

In some implementations, the natural language processor <NUM> may additionally and/or alternatively include a coreference resolver (not depicted) configured to group, or "cluster," references to the same entity based on one or more contextual cues. For example, the coreference resolver may be utilized to resolve the term "there" to "Hypothetical Café" in the natural language input "I liked Hypothetical Café last time we ate there.

In some implementations, one or more components of the natural language processor <NUM> may rely on annotations from one or more other components of the natural language processor <NUM>. For example, in some implementations the named entity tagger may rely on annotations from the coreference resolver and/or dependency parser in annotating all mentions to a particular entity. Also, for example, in some implementations the coreference resolver may rely on annotations from the dependency parser in clustering references to the same entity. In some implementations, in processing a particular natural language input, one or more components of the natural language processor <NUM> may use related prior input and/or other related data outside of the particular natural language input to determine one or more annotations.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of one or more of the processes of <FIG>, <FIG>, and/or <FIG>, as well as to implement various components depicted in <FIG>.

In addition, some implementations include one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s), and/or tensor processing unit(s) (TPU(s)) of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the methods described herein. Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the methods described herein.

Claim 1:
A method (<NUM>) implemented by one or more processors, the method comprising:
receiving (<NUM>) a plurality of items of content to be recommended to a user of a client device;
for each item of content to be recommended for the user:
determining (<NUM>) a plurality of candidate endorsements corresponding to the item of content by processing the item of content using a plurality of nominators including a variety of types of nominators, each nominator generating one of the candidate endorsements, wherein each type of nominator generates candidate endorsements based on a distinct corpus of information related to the item of content, and where each candidate endorsement provides a natural language description of information relating to the item of content;
determining a score for each candidate endorsement by processing the plurality of candidate endorsements corresponding to the item of content using a ranking agent, where the score is indicative of a likelihood the user will interact with the item of content based on the candidate endorsement, and where the ranking agent determines the score uniformly across the plurality of nominators;
selecting (<NUM>) an endorsement from the plurality of candidate endorsements to present with the item of content based on the score of each candidate endorsement corresponding to the item of content; and
rendering (<NUM>) the plurality of items of content and the selected endorsements corresponding to the plurality of items for the user via one or more user interface output devices of the client device.