Voice based interaction based on context-based directives

A voice assistant service (VAS) may receive an audio request from a user via a first device via a voice assistant application executed by the first device. The VAS may determine that the audio request references a context queue that stores log information about user-interaction with a second device. The VAS may analyze the log information from the context queue to determine a context of the audio request as a supplemented request. The VAS may determine a response based on the supplemented text request. The response may be an audio response and/or an action, such as a computing action. The VAS may provide an audible response to the voice assistance application for output to the user.

BACKGROUND

Many different ways have been introduced to allow users to interact with computing devices, such as through use of mechanical devices (e.g., keyboards, mice, etc.), touch screens, motion, gesture, and even using natural language input such as speech. Furthermore, many of these computing devices are further connected to remote computing resources, such as cloud-based resources, that extend functionality afforded by the local computing devices.

As computing devices in homes and offices continue to evolve, users expect a more seamless and timely experience when interacting with cloud-based resources through the local computing devices. Additionally, users expect a more robust set of services when interacting with cloud-based resources through the local computing devices. In particular, users expect accurate, intuitive, and relatively quick interactions with computing devices in order to instruct the computing devices to perform desired functions.

In some environments, use of a voice assistant service may augment computing operations performed using conventional devices or applications, which may rely on traditional modes of input/output.

DETAILED DESCRIPTION

This disclosure is directed to augmenting information provided by content provider using a first device or first application using a voice assistant service via a second device or second application.

The voice assistant system may include any system and/or device that receives audio commands from a user, processes the audio, possibly using speech to text algorithms and/or natural language processing (NLP) algorithms, to determine text, returns a reply based on the text, converts the reply to an audio output using text to speech algorithms, and causes a speaker to output the audio output. Examples of voice assistant systems include Alexa® provided by Amazon.com® of Seattle, Wash., Siri® provided by Apple Corp.® of Cupertino, Calif., and Cortana® provided by Microsoft Corp.® of Redmond, Wash. The voice assistant system may include a user device that typically includes at least a network interface, a microphone, and a speaker. The user device may be a smart phone, a dedicated device, and/or other devices controlled by users and located proximate to the users. The voice assistant system may include a service engine, which may be stored in a remote location (e.g., via remote computing devices such as in a cloud computing configuration, etc.), stored in a local device (e.g., a smartphone, a dedicated voice assistant device, etc.) and/or a combination of both.

In some embodiments, the voice assistant system may enable a user to order items, select fulfillment details for the items, pay for the items, and/or perform other tasks to enable the user to obtain information or the items using voice activated commands and without reliance on a graphical user interface. In various embodiments, the voice assistant system may leverage prior user interaction data, user profile information, and/or other user information during interaction with a user to supplement voice inputs received from a user.

In an example, use-case, a user may view content on a browser application, such as an item the user may desire to purchase. The user may make an audible request to a voice assistant application, which may be running on the same device as the browser application or on a second device. The audible request may include a reference to the content provided by the browser application, such as “buy this” or “show me more,” among other possible requests that are associated with the content and rely on the knowledge of the content to provide context for the audible request. A voice assistance service may associate use of the voice assistant application with use of the browser application, such as by way of a shared user profile associated with the applications. In some embodiments, the voice assistant service may access log information provided to a context queue, which may store some, but possibly not all, information pertaining to a request or action, and metadata associated with content provided by the browser application and/or other applications associated with the voice assistant service via the user profile or by association with other data. The voice assistant service may analyze the content provided by the browser application, possibly stored as a log or as other information, to determine a context of the audible request, and thus to determine a supplemented audible request. The voice assistance service may then generate a response to the supplemented audible request, which may be provided back to a user by the voice assistance application or by other outputs, such as by refreshing information provided by the browser application. In some embodiments, the voice assistance service may initiate a transaction to acquire an item on behalf of a user, where the item is viewed by the user via the browser application, but the transaction is preformed via the voice assistant application that is a different application than the browser application.

In some embodiments, the voice assistant application may access, after receiving user consent, information from another application operating on the same device, such as a browser application or other third party application. For example, the voice assistant application may receive an audible request. The voice assistant application may then access information associated with an application that has is associated with user activity within a threshold amount of time, or otherwise referred to as “active” or “in focus” by a user. For example, the threshold period of time may be a number of seconds or minutes, such as five seconds, one minute, or other amounts of time. The voice assistant application may access data from the application, such as a uniform resource locator (URL), an item identifier, an image or screenshot, or other data that indicates content provided by the application. The voice assistant application may transmit this information form the application to a voice assistance service to enable augmenting the audible request, such as to provide context to the audible request. For example, if the voice assistant application receives an audible request of “buy this”, the voice assistant service may determine the item represented by the term “this” and may create a response accordingly.

The voice assistant service may user information from different applications and different devices to augment a response to an audible request. For example, the voice assistant service may cause the voice assistant application to output an audio reply that complements information presented by a different application. The voice assistant service may answer a question, initiate a transaction, store data for later retrieval by a user via a user device, cause the application to refresh and provide different information to the user, and/or perform or provide other augmented services or information.

FIG. 1is a schematic diagram of an illustrative computing environment100that includes a voice assistant service102that augments information provided by a content provider104. The environment may include a user device106operated by a user108. The user device106may exchange information from the content provider104. As an example, the user device104may execute a browser application that enables the user108to interact with content provided by the content provider104. For example, the content provider104may host an electronic marketplace that enables the user108to consume products and/or services, referred to collectively herein as “items”.

The environment100may also include a voice assistant device110that operates a voice assistant application. The voice assistant device110may be the same devices as the user device106or may be a different device (as shown inFIG. 1). The voice assistant device110may exchange information with the voice assistant service102to provide audible dialog between the user108and the voice assistant service102. The dialog may be used to acquire information (what is “x”?, what is the weather for today?, Tell me the news headlines, etc.), initiate computing actions (e.g., buy this, initiate a transaction, save this for later, play this song, etc.), and/or perform other actions.

In accordance with one or more embodiments, the voice assistant device110may be one of any electronic devices that are at least partially controlled using speech commands issued by the user108. In some embodiments, the voice assistant device110may be a dedicated speech recognition device that includes few or no other input controls (e.g., few or no buttons, motion devices, imaging devices, etc.). Instead, the voice assistant device110may receive input from users by receiving spoken commands, which are converted to signals by the voice assistant device110and/or by a cloud service, and then processed, such as by an exchange of data with voice assistant service102. The voice assistant service102may be any service that provides data to the voice assistant device110in response, directly or indirectly, from the voice assistant device110. The voice assistant service102need not be configured for speech recognition since speech recognition may be performed prior to sending a request to one of the services in some instances. The voice assistant service102may provide may different types of information, entertainment, or data, such as by proving music, directions, documents, performing tasks (adding things to lists, setting reminders, starting a timer, adding a calendar event, etc.), and so forth. As discussed herein, the voice assistant service102may be configured to engage in a dialog to receive an order of one or more items from the user, including facilitating selection and confirmation of items, and cause those items to be fulfilled and delivered to the use, or for other tasks or fulfillment of audible requests. The voice assistant service102and the voice assistant device110may be in communication via one or more networks114, which include wired and/or wireless networks.

The environment100enables a user to share information exchanged with the content provider104with the voice assistant service102to enable the voice assistant service102to augment the information provided by the content provider104. In some embodiments, a context queue112may enable sharing some, but not all, information between the content provider104and the voice assistant service102, as explained further below by way of an example. The devices discussed above may communicate by way of one or more networks114, which may include wireless and/or wired networks.

In this example, the user108may use the user device106to access a user interface (UI)116to view information about an item118. The user108may desire to use the voice assistant device110to provide augmented information (including actions) related to the item. Here, the user108may issue a voice request to the voice assistant device110of “buy this”. Currently, such a request would result in an error or a request for clarifying information from the voice assistance system102, such as a response of “I'm sorry, I don't understand . . . ”. However, contextual information may be provided to the voice assistance service102, via the context queue112.

To provide context information to the voice assistant service102, the content provider104and the voice assistant service102may share associated information about the user108. The shared information may include a same user profile. For example, the user108may be logged in using a user profile to interact with the content provider104. The user108may be logged in to the same user profile, or possibly an associated user profile, to interact with the voice assistant service102via the voice assistant device110. This linking of information may create a conduit, via the context queue112, for the content provider104to share contextual information, after receiving user permission, with the voice assistant service102. User permissions may be created in advance, and notifications of such sharing of information may be provided to the user, such as by providing messages to the user (e.g., “voice assistance enabled” on a UI, etc.).

In response to providing information to the user device106to populate the UI116about information pertaining to the item118, the content provider104may provide log information120to the context queue. The log information may include some, but not all, of the information provided to the user device106. In some embodiments, the log information may provide basic or specific information, such as an idem identifier, a URL, or other information that may be used to determine context by the voice assistance service102. The log information120may be associated with metadata, such as a timestamp, usage metrics, user interaction data, and/or other types of data.

Meanwhile, the voice assistant service102may receive a message122of the audible request of “buy this”, which was received via a microphone of the voice assistance device110. The voice assistance service102may analyze the message122and determine that context information may be available to deduce a meaning of the audible request. The voice assistance service102may request information from the context queue112to determine a context. For example, the voice assistant service102may submit a request to the context queue112. The request may include parameters, such as the user profile associated with the message122, among other possible parameters. In return, the context queue112may provide one or more logs to the voice assistance service102, including the log information120.

The voice assistant service102may analyze the log information120to determine a context of the message122, such as to initiate a transaction for the item118to be acquired on behalf of the user108. For example, the voice assistant device102may initiate a transaction with the content provider104, with another service provider, or using its own service, to acquire the item. In various embodiments, the voice assistant service102may perform other operations once the context of the message122is determined. For example, the voice assistant service102may provide an audible response to the user108via a voice assistant device110, may cause the content provider104to refresh content provided to the user device106, may store data on behalf of the user108possibly in association with a user account, and/or may perform other operations in response to the message.

Although the example above discusses use of a context queue that operates as an intermediary between the content provider104and the voice assistant service102, the user device106(or software operating thereon) may provide a similar functionality as discussed below. Thus, the voice assistance service102may receive contextual information from a different application (or “app”) running on the user device106.

FIG. 2is a block diagram of an illustrative computing architecture200of a voice assistant service, such as the voice assistant service102. The computing architecture200may be implemented in a distributed or non-distributed computing environment. The computing architecture200may include one or more processors202and one or more computer-readable media204that stores various modules, applications, programs, or other data. The computer-readable media204may include instructions that, when executed by the one or more processors202, cause the processors to perform the operations described herein.

In some embodiments, the computer-readable media204may store a voice assistant engine206. The voice assistant engine206may further include a language module208, a context module210, and a fulfillment module212, which are described in turn. The voice assistant service may access or include data stores including user data214, catalog data216, and the context queue112. The voice assistant engine206and data stores may be stored together or in a distributed arrangement.

The language module208may process language inputs received from the user, such as an audible request. For example, the language module208may speech-to-text (STT) and/or text-to-speech (TTS) algorithms to convert between text and speech. The language module208may deploy automated speech recognition (ASR) algorithms and/or natural language processing (NLP) algorithms to facilitate processing information received from a user. In some embodiments, the language module208may determine means for pronouns and/or anaphoras by analyzing use of language rules and information collected from a user from a prior user input and/or from user history via the user data214. The user data may enable linking the user with contextual information, as discussed below via the context module210.

The context module210may provide additional information for processing an audible request, such as by providing context to enable the voice assistant engine204to understand the request, and thus be equipped to respond to the request. The context module210may interface with the context queue212to obtain contextual information (e.g., log information and/or metadata), which may provide context to an audible request. In some embodiments, the context module210may provide some parameters to the context queue212, such as a user profile, to assist in locating or accessing contextual information which may be securely stored by the context queue212. In various embodiments, the context module210may analyze the contextual information to determine context. For example, the context module210may utilize an image analysis module (e.g., an optical character recognition (OCR) algorithm or other image analysis algorithms) to extract and analyze information from the context queue112. In some embodiments, the context module210may identify specific information, such as an item identifier (e.g., a number, code, or other identifier of a specific item), which may be used to provide context for the audible request.

The fulfillment module212may process a supplemented request, which may be the audible request that is supplemented by contextual information from the context queue112, thereby augmenting information provided by the content provider. For example, the fulfillment module212may determine information about an item associated with the audible request (via the contextual information), such as via the catalog data216. The fulfillment module212may generate a response to deliver to the voice assistance device (e.g., the voice assistant device110or the user device106, etc.). In some embodiments, the fulfillment module212may provide information to a content provider, such as the content provider104, which may be used to refresh information provided to the user's device (e.g., refresh information associated with a URL accessed by the user device106, etc.). In various embodiments, the fulfillment module212may store information for later retrieval by the user, such as a list of items to acquire and/or other types of information. The fulfillment module212may provide other types of computing actions on behalf of the user, such as perform searches in data repositories, modify settings, and/or perform other actions requested by the user and supplemented by the contextual information from the context queue112. In some embodiments, a response may be processed by the language module208prior to transmission to the voice assistance device, such as to perform TTS or NLU via associated algorithms.

FIG. 3is a schematic diagram of an illustrative device300that includes at least a first application that can share information with a second application in communication with a voice assistant service. The device300may include an operating system layer302and any number of applications304. Some applications306(1)-(N) may be authorized for association with a voice assistant application308. For example, these applications may share context information with the voice assistant application308. The context information may be shared by the device, which may act as a context queue as described above. Thus, the device may track information and user interaction with authorized applications, and provide this information from time to time to the voice assistant application308, such as in response to an audible request provided to the voice assistant application308. In some embodiments, the voice assistant application308may receive application data from a particular application306(N) that received user manipulation within a threshold period of time. This particular application306(N) (referred to at times as a second application) may be “in focus” by a user, and manipulated just prior to or during receipt of the audible request by the voice assistant application308. In some embodiments, the voice assistant application308may be a module, plug-in, extension, or other software add-on to the particular application306(N). However, the voice assistant application308and the particular application306(N) may be separate applications. In various embodiments, the voice assistant application308may be part of a core software or the operating system layer302.

The voice assistant application308may provide context information gained from the particular application306(N) to the voice assistant service102. The voice assistant service102may then process the request, such as “buy this” using the context information that may accompany the audible request or be associated with the audible request.

FIG. 4is a flow diagram of an illustrative process400to augment information from a content provider using a voice assistant service. The process400is described with reference to the environment100and the computing architecture200and may be performed by the voice assistant service102. Of course, the process400may be performed in other similar and/or different environments.

At402, the voice assistant service102may receive an audio request with reference to other source of information, such as information viewed by the user via a different application or via a different device. For example, the audio request may include language that refers to another sources of data, such as “this” or “that”, or the request may not provide information specific enough to complete a request, such as “buy this toaster.” The other source of information may be provided by the content provider104, which may be associated with the voice assistant service102via a common user profile. The audio request may be an audio file of speech from a user. The voice assistant service102may process the audio request, via a language module208, such as to determine text of the request and/or other information about the request, and possibly the association with the content provider.

At404, the voice assistant service102may access, after receiving appropriate user authorization(s), the context queue112populated by content providers to determine a context of the audio request. For example, the voice assistant service102may send parameters and/or other information to the context queue112to request and receive log information. The log information may include an image, textual data, or other information and/or metadata to enable the voice assistant service102to determine a context of the audio request. As an example, the log information may include an item identifier that may provide a subject of the audio request, such as an audio request of “buy this” or “recommend other items similar to that”.

At406, the voice assistant service102may analyze source(s), such as the log information, to determine context of audio request. The voice assistant service102may use information in the audio request to determine the context. For example, the request may specify “buy this toaster”, but the request may not specify which model. The model information may be found in the context queue in associated log information. In some embodiments, the voice assistant service102, via the context module210, may use algorithms to analyze the log information from the context queue112to determine the context information. For example, the voice assistant service102may use a OCR algorithm to inspect text captured in an image or from a URL associated with the log data. The voice assistant service102may inspect multiple logs from different user devices to determine which one maps to the audio request, possibly using usage history, and which log includes relevant contextual information. For example, when the log include information from multiple devices, one device may be identified as a likely matching device when it's browsing history represented in the log information is similar to questions received via the voice assistant service102(e.g., both include topics related to baby items, for example).

At408, the voice assistant service102may determine whether to request more information to determine a context of an audio request. For example, when the voice assistant service102includes enough information to understand and respond to an audio request that is supplemented by context information derived from the context queue, then the voice assistant service102may not request additional information from the user. When the voice assistant service102does not include enough information to understand and respond to the audio request that is supplemented by context information derived from the context queue (following the “yes” route from the decision operation408), then the process400may advance to an operation410.

At410, the voice assistant service102may request additional information from the user by sending an audio request to the user via the voice assistant device or voice assistant application. The request may be a choice, such as “did you mean “buy a toaster or diapers”. In some instances, the response may be a request for more specific information, such as “Please provide more information so I can fulfill your request”.

When the voice assistant service102does include enough information to understand and respond to the audio request that is supplemented by context information derived from the context queue (following the “no” route from the decision operation408), then the process may advance to an operation412. At412, the voice assistant service102may implement supplemented audio request. For example, the voice assistant service102may provide an audio response to the user via the voice assistant device or voice assistant application.

FIG. 5is a flow diagram of an illustrative process500to determine context of a voice request based on content provided by a different application running on a device. The process500is described with reference to the environment100and the computing architecture200and may be performed by the voice assistant service102. Of course, the process500may be performed in other similar and/or different environments.

At502, the voice assistant service102may receive an audio request from a voice assistant application on a device with reference to another application executed by the same device. For example, the audio request may include language that refers to data from a different application as “this” or “that”, or the request may not provide information specific enough to complete a request, such as “tell me more about this.” The audio request may be an audio file of speech from a user. The voice assistant service102may process the audio request, via a language module208, such as to determine text of the request and/or other information about the request.

At504, the voice assistant service102may determine a recently active application executed by the device, which also executes the voice assistant application. The active application may be a second application that received user manipulation within a threshold period of time relative to issuance of the audio request at the operation502.

At506, the voice assistant service102may receive and analyze information from the application (e.g., the second application) to determine context of audio request. The voice assistant service102may use information in the audio request to determine the context. For example, the request may specify “when was this song recorded”, but the request may not specify which exact song. The song information may be found by inspecting information accessed from the second application. In some embodiments, the voice assistant service102, via the context module210, may use algorithms to analyze the information from the second application to determine the context information. For example, the voice assistant service102may use a OCR algorithm to inspect text captured in an image or from a URL associated with the second application.

At508, the voice assistant service102may determine whether to request more information to determine a context of an audio request. For example, when the voice assistant service102includes enough information to understand and respond to an audio request that is supplemented by information from the second application, then the voice assistant service102may not request additional information from the user. When the voice assistant service102does not include enough information to understand and respond to the audio request that is supplemented b by information from the second application (following the “yes” route from the decision operation508), then the process500may advance to an operation510.

At510, the voice assistant service102may request additional information from the user by sending an audio request to the user via the voice assistant application. The request may be a choice, such as “did you mean “song ABC or song XYZ”. In some instances, the response may be a request for more specific information, such as “Please provide more information so I can fulfill your request”.

When the voice assistant service102does include enough information to understand and respond to the audio request that is supplemented by information from the second application (following the “no” route from the decision operation508), then the process may advance to an operation512. At512, the voice assistant service102may implement supplemented audio request. For example, the voice assistant service102may provide an audio response to the user via the voice assistant application.

FIG. 6is a flow diagram of an illustrative process600to determine outputs of a voice assistant service to augment content provided by a different application or device. The process600is described with reference to the environment100and the computing architecture200and may be performed by the voice assistant service102. Of course, the process600may be performed in other similar and/or different environments.

At602, the voice assistant service102may analyze information to determine context of an audio request. The operation602may be similar to the operation406and/or the operation506discussed above. For example, the voice assistant service102may inspect information provided by another application running on a same device or may inspect information in the context queue, which is populated by content providers that also interact with a same user via a user device.

At604, the voice assistant service102may determine an output action to fulfil the supplemented request. For example, the voice assistant may determine an output format or action based on various options available to the particular configuration. Some options may be unavailable for some types of request.

At606, the voice assistant service102may determine whether to perform an audio authentication. The audio authentication may authenticate a user and/or access to perform an action, such as execute a command, access information, or for other reasons. For example, some legacy devices may not support the latest security authentication mechanisms enforced by a server. Due to this, user may have to perform an additional authentication step for certain actions or purchases. This two-factor authentication usually involves sending an email and entering the One Time Password and/or re-logging in to complete an action or transaction. However, audio authentication may be used by the voice assistant service as a solution. As an example, the voice assistant service102may analyze the audio request as authentication information to allow a privileged computing action. The analysis may include analyzing an account associated with the audio request, a time of the request, a location associated with the request, a voice signature of the request, and/or other attributes of the audio request.

When audio authentication is to be performed (following the “yes” route from the decision operation608), then the process600may continue at the operation608. At608, audio authentication may be performed.

At608, in some embodiments, the voice assistant service may receive an audio request and authenticate based on receipt of this audio request (e.g., based on the logged in state of the voice assistant application or voice assistant device). Here, the audio request may be linked or associated with a user account (e.g., via backend processing using shared resources), which is also associated with the content viewed on the second application. A confirmation page or reply (possibly via audio) may confirm completion of this action.

At608, in various embodiments, the voice assistant service102may receive a code generated by the second application or may provide a code to be input to the second application for authentication between the applications. Here, the authentication may be performed using input via the first application (e.g., the voice assistant application, etc.) and/or the second, different application. For example, the code may be a sequence of numbers/letters that can be spoken by the user for receipt by the voice assistant service or manually input into the second application. Ultimately, back end processing may confirm the code is correct, which may be performed by sharing information across the content provider and the voice assistant service, for example.

Following operation608, or when no audio authentication is to be performed (following the “no” route from the decision operation608), then the process600may continue at the operation610.

At610, the voice assistant service102may determine an output action. The output may be determined and implemented by the fulfillment module212.

Following a first route “A” from the decision operation610, the voice assistant service102may advance to an operation612. At612, the voice assistant service102may cause output of an audio response that answers a question posed in the audio request, and supplemented by the context information. For example, the audio request may be a question such as “what year was this song recorded” and the operation612may include “the song ‘Tangled up in Blue’ was recorded in 1974 by Bob Dylan”.

Following a first route “B” from the decision operation610, the voice assistant service102may advance to an operation614. At614, the voice assistant service102may initiate a transaction for an item on behalf of the user. For example, the audio request may include “buy this toaster.” By supplementing the request with context information, the voice assistant service102may determine a specific brand and model of the toaster, and may initiate a transaction for the item via the operation614. In some embodiments, the voice assistant service102may initiate the transaction via a content provider, and possibly a content provider that identified the context to the voice assistant service102.

Following a first route “C” from the decision operation610, the voice assistant service102may advance to an operation616. At616, the voice assistant service102may store data or other information on behalf of the user for later retrieval by the user (e.g., in response to a later request from the user, etc.). For example, the voice assistant service102may receive an audio request of “save this book to my list” or “add this to my shopping cart.” At616, the voice assistant service102may add data to a list, such as by storing data in association with the user profile after supplementing the request with the contextual information.

Following a first route “D” from the decision operation610, the voice assistant service102may advance to an operation618. At618, the voice assistant service102may cause the content provider to refresh data provided to a user device to fulfill, at least in part, the audio request. For example, the voice assistant service102may receive a request of “show me more books like this” while the user is viewing an electronic bookstore via a second application or device. The voice assistant service102may determine the book referred to as “this” via the contextual information. At618, the voice assistant service102may transmit a request to the content provider to cause a refresh of information served to a device associated with the user who made the audio request, to cause output of more books, such as by graphically outputting recommended books similar to the book referenced via the contextual information.

Following a first route “E” from the decision operation610, the voice assistant service102may advance to an operation620. At620, the voice assistant service102may provide other outputs as a reply to the audio request. As an example, the voice assistant service102may provide at620a request for more information, an error, a playback of media associated with the request, or other information that satisfies or is expected to satisfy the audio request supplemented with the contextual information.

FIG. 7is a flow diagram of an illustrative process700to analyze a request received by voice assistant service and to refresh graphical content provided to a device based on the request. The process700is described with reference to the environment100and the computing architecture200and may be performed by the voice assistant service102. Of course, the process700may be performed in other similar and/or different environments.

At702, the voice assistant service102may receive an audio request with reference to a related source (e.g., information on an application or other device used by a same user having a same user profile, etc.). The operation702may be similar to the operation402and/or the operation502discussed above.

At704, the voice assistant service102may access information to determine a context of the audio request. The voice assistant service102may access a context queue (e.g., similar to the operation404). The voice assistant service102information from another application (e.g., similar to the operation504).

At706, the voice assistant service102may determine the context of the audio request to determine a supplemented request. The voice assistant service102may access a context queue (e.g., similar to the operation404). The voice assistant service102may determine the context from analyzing log information from the context queue (e.g., similar to the operation406). The voice assistant service102may determine the context from analyzing information from a second application running on a same device that provided the audio request (e.g., similar to the operation506).

At708, the voice assistant service102may determine a response to the supplemented request. The response may be to provide graphical information to the user via the second application or via the different device.

At710, the voice assistant service102may cause a change in content provided by an application (e.g., the second application and/or another device). The voice assistant service102may transmit a request to a content provider associated with the context information from the context queue or associated with the second application. The voice assistant service102may request the content provider provide certain information to the user via a refresh operation (e.g., an AJAX call or other techniques to refresh content provided to an application such as a browser or dedicated application or “app”.). As an example, the audio request may be “show me more books like this one”. The voice assistant service102may transmit a request at710to the content provider to request delivery of recommended books associated with the supplemented request (e.g., associated with a specific book), and provided to the user via a refresh of content or other change of content via the content provider.

FIG. 8is a flow diagram of an illustrative process800exchange information between various devices using a context queue to enable providing context to voice-based requests for a voice assistant service. The operations in the process800are shown under devices that may perform the operations. However, the operations may be performed by other devices in some instances. For example, the user device106may operate the voice assistant application, and may be a same device as the voice assistant device110in some instances.

The process800may begin at an operation802where a content provider104sends data to a user device106, which is received an operation804and displayed or output via the user device at an operation806.

At808, the content provider104may send log data to the context queue112. The log data may be an image, a URL, metadata, an item identifier, and/or other data. At an operation810, the context queue112may store the log information, possibly with metadata such as a timestamp, an associated user profile, and/or other information.

Meanwhile, at an operation812, the voice assistant device110may detect an audio request via a microphone. At an operation814, the voice assistant service102may receive the request from the voice assistant device110.

At an operation816, the voice assistant service102may request log information from the context queue112, which may get the request at an operation818and may fulfill the request at an operation820. For example, the voice assistant service102may pass parameters to the context queue112, such as a user profile and a time period, which may be used to query for associated log information that provides contextual information for the audio request received at the operation814. At an operation822, the voice assistant service102may receive the data (e.g., the log information) from the context queue112.

At an operation824, the voice assistant service102may analyze the data (e.g., the log information) to determine a context of the audio request received at the operation814. At an operation826, the voice assistant service102may create a response to the audio request, which is supplemented by the context from the operation824.

The voice assistant service102may provide the response, via audio, to the voice assistant device110at an operation828. At the operation828, the voice assistant device may cause output of the audio response.

In some embodiments, the response created at the operation826may be output via the user device106as discussed in the process700described with reference toFIG. 7. At826, the voice assistant service102may send a request to the content provider to refresh content provided to the user device106. The content provider104may refresh the content or otherwise cause output of certain content informed based at least in part by the voice assistant service102via the operation826. At an operation832, the user device110may output the refreshed content for display to the user.

FIG. 9shows an illustrative user interface (UI)900that indicates compatibility with a voice assistant service. The UI900may include content provided by the content provider104shown inFIG. 1. The UI900may show an item902which may be associated with an item identifier904and a source906, such as a URL. The UI900may provide other information908about the item. The UI900may provide other items910, such as related items, advertisements, recommendations, and/or other items.

The UI900may provide an indication912that indicates that a voice assistant service is enabled for this content. For example, when the content provider stores information about the UI900in the context queue, the content provider may cause display of the indication912. The indication912may serve to inform users of an ability to make audio request related to this content as described above.

The context queue may include different types of log information to represent the information included in the UI900. In some embodiments, the log information may include the source906, such as a URL or network address. The log information may include the item identifier904. The log information may include an image of the UI900or an image of a portion of the UI900. The log information may include text included in the UI900, such as HTML code used to create the UI900. The log information may include combinations of the aforementioned information and/or other information. Analysis of the log information may provide contextual information to the voice assistant service to enable fulfillment of an audio request as disclosed above. For example, an OCR algorithm may be used to extract information from an image stored as the log information.

CONCLUSION