Patent Publication Number: US-11664032-B2

Title: Multi-agent input coordination

Description:
CROSS-REFERENCE TO RELATED APPLICATION(S) 
     The present application claims priority to and is a continuation of U.S. patent application Ser. No. 16/385,106, filed Apr. 16, 2019. All sections of the aforementioned application(s) and/or patent(s) are incorporated herein by reference in their entirety. 
    
    
     TECHNICAL FIELD 
     This disclosure generally relates to virtual assistants that can listen for commands or sounds and perform various control functions within an environment such as a building. More specifically, this disclosure relates to mechanisms that automatically coordinate responses and activities of multiple virtual assistant software agents being used together in or on the same premises. 
     BACKGROUND 
     A virtual assistant can be accessed through a software agent in a smart device. Examples of virtual assistants include Google Assistant™, Apple&#39;s Siri™, Amazon&#39;s Alexa™, and Microsoft&#39;s Cortana™. Deployment of multiple agents for virtual assistants, each with listening capability, is becoming increasingly common in homes and business. In some instances, these agents simultaneously record and attempt to respond user requests. 
     SUMMARY 
     In one example, a system includes a wireless communication interface and a processor communicatively coupled to the wireless communication interface, wherein the processor is configured to perform operations. The operations include identifying at least one audio request received through two or more agents in a network and determining at which of the agents to actively process a selected audio request of the at least one audio request using at least one of location context, person interaction context, or secondary trait analysis. The audio request(s) include simultaneous audio requests received through at least two of the agents, at least two differing audio requests received from different requesters, or both. 
     In an additional example, a method includes identifying, by a processor, at least one audio request received through at least two agents in a network and determining, by the processor, at which of the agents to actively process a selected audio request of the at least one audio request using at least one of location context or secondary trait analysis. The audio request includes simultaneous audio requests received through at least two agents, at least two differing audio requests received from different requesters, or both. 
     In a further example, a non-transitory computer-readable medium includes instructions that are executable by a computing device for causing the computing device to perform operations for multi-agent input coordination. The operations include identifying at least one audio request received through two or more agents in a network and determining at which of the agents to actively process a selected audio request of the at least one audio request using at least one of location context or secondary trait analysis. The audio request includes simultaneous audio requests received through at least two agents, at least two differing audio requests received from different requesters, or both. 
    
    
     
       BRIEF DESCRIPTION OF THE FIGURES 
       These and other features, aspects, and advantages of the present disclosure are better understood when the following Detailed Description is read with reference to the accompanying drawings. 
         FIG.  1    is an architectural-style illustration of a multi-agent input coordination environment according to some aspects of the present disclosure. 
         FIG.  2    is a block diagram depicting a smart device that provides multi-agent input coordination according to some aspects of the present disclosure. 
         FIG.  3    is a block diagram depicting a system for a multi-agent input coordination environment according to some aspects of the present disclosure. 
         FIG.  4    is a flowchart illustrating a process for providing multi-agent input coordination according to some aspects of the present disclosure. 
     
    
    
     DETAILED DESCRIPTION 
     Certain aspects of this disclosure relate to acoustic collaboration of multiple listening agents deployed in smart devices on a premises. This acoustic collaboration can reduce or avoid incidences of inaccurate recognition of appropriate actions and user frustration with command execution that can result when agents simultaneously record and attempt to respond to user requests. Certain aspects of this disclosure relate to improving the accuracy of identifying requests and specifying where that request should be actively processed, improving quality of detection and providing better understanding of user commands and user intent throughout the premises. 
     In one example, a processor carries out operations including identifying at least one audio request received through at least two agents in a network and determining at which of the agents to actively process a selected audio request using at least one of location context or secondary trait analysis. The audio request can include a simultaneous audio request received through at least two agents, at least two differing audio requests received from different requesters, or both. 
     In some aspects, determinations are made using secondary trait analysis including, as examples, footstep recognition, non-language-sound cadence, habit pattern analysis, or tonal context. In some aspects, determinations are made using location context, including, as examples, localization, movement, or spatial usage restrictions. 
     In some aspects, the processor activates an attention token at the agent at which the selected audio request is to be honored and displays an indication of the attention token. In some aspects, the processor automatically sorts ambient sounds into sound categories, and uses the sound categories to provide location context for determining where to honor the selected audio request. In aspects, actions to be taken by an agent are determined at least in part by a state machine to take into account previous audio requests and actions. 
     Detailed descriptions of certain examples are discussed below. These illustrative examples are given to introduce the reader to the general subject matter discussed here and are not intended to limit the scope of the disclosed concepts. The following sections describe various additional aspects and examples with reference to the drawings in which like numerals indicate like elements, and directional descriptions are used to describe the illustrative examples but, like the illustrative examples, should not be used to limit the present disclosure. 
       FIG.  1    is an architectural-style illustration of a multi-agent input coordination environment (MICE) according to some aspects of the present disclosure. Premises  100  includes three listening agents. One listening agent is deployed in smart device  102 , another listening agent is deployed in smart device  104 , and a third listening agent is deployed in smart device  106 . Requester  108  is in room  110  with smart device  102 . Requester  112  is in room  114  with smart device  106 . Smart device  104  is located in room  116 . Nobody is present in room  114  with smart device  106 . The agents in premises  100  form a local network mesh, for example, using a local wireless local area network (WLAN). 
     Still referring to  FIG.  1   , the multiple listening agents shown in premises  100  are joined in the space. Each agent can probe for others within a network or spatial location. The network can be organized as a layer running over the WLAN and can be organized as a centralized network or a mesh network. The specific examples presented assume a mesh network paradigm is used. Optionally, other audio sources that are not incorporated into smart devices can join the network as listening agents. A microphone in such a device can be dispatched to low coverage or high interference areas such as where a user cannot otherwise be heard due to high noise or distance to existing listening devices. 
       FIG.  2    is a block diagram depicting the smart device  102 . For purposes of this example, it can be assumed that smart device  102  is a smart speaker. Smart device  102  includes an amplification block  202  connected to antenna  210 . Amplification block  202  includes a power amplifier for Wi-Fi transmission, as well as preamplifiers or amplifiers for boosting received signals. Smart device  102  also includes a processor  204 , which is implemented as part of system-on-a-chip (SOC)  206 . A dual-band wireless LAN (WLAN) interface block  208  is communicatively coupled to SOC  206 . The dual-band WLAN interface supports the 2.4 GHz and the 5.8 GHz bands. Also included in smart device  102  is flash storage  209  and random-access memory (RAM)  211 . RAM  211  can include various devices and possibly memory dedicated to specific purposes such storing encryption keys, MAC addresses, and the like for access by processor  204  when the smart device is in operation. Input/output (I/O) block  212  drives status LEDs (not shown), receives input from a microphone (now shown) and provides output to a speaker (not shown). Within the dual band WLAN interface  208 , transmitted and received information can be converted to and from radio frequencies (RF), and filtering using baseband or intermediate frequency circuitry can be applied. The SOC  206  is specifically designed to implement smart speaker functions, and also performs basic signal processing, e.g., synchronization, coding and decoding. 
     Still referring to  FIG.  2   , the functions of the SOC  206  and the other aforementioned blocks can be directed and controlled by the processor  204 , which can be a general-purpose microprocessor, digital signal processor (DSPs), application specific integrated circuit (ASIC). Supporting control logic can include various types of signal conditioning circuitry, including analog-to-digital converters, digital-to-analog converters, input/output buffers, etc. The flash storage  209  shown in  FIG.  2    includes at least one array of non-volatile memory cells. RAM  211  includes at least one array of dynamic random-access memory (DRAM) cells. The content of the flash memory may be pre-programmed and write protected thereafter, whereas the content of other portions of the RAM may be selectively modified and/or erased. The flash memory therefore, is non-transitory computer-readable medium that is used to store operating system software or firmware, including computer program code instructions  250 , which are executable by processor  204  to serve as the local MICE agent and carry out the multi-agent input coordination as described herein. Flash storage  209  can also be used to store credentials and encryption keys for longer periods. SOC  206  also contains on-board memory  260  that can serve as a non-transitory medium to store computer program code, credentials, MAC addresses, encryption keys, etc. It cannot be overemphasized that smart device  102  is but one example of a smart device. 
       FIG.  3    is a block diagram depicting an example of a system  300  for multi-agent input coordination according to some aspects of the present disclosure. Local agent  250  in smart device  102  on premises  100  includes spatial localization module  302 , audio cleanup module  304 , state machine  306 , agent response coordinator  308 , and attention token flag  310 . Agent  250  is connected to stored sound category data  320  and external network  324 . External network  324  in this example includes a high-level awareness agent  326 , external services  328 , and a MICE cloud agent  332 . Sound category data  320  can be stored in the same non-transitory memory device as agent  250 , on a local server, or in external network  324 . Other agents in  FIG.  1    can include local agents having the same or a similar structure as that described here with respect to local agent  250 . 
     Still referring to  FIG.  3   , spatial localization module  302  provides localization through advanced use of sound understanding, optionally without explicit optical input as from a camera, or external labels as might be provided by beacons, global positioning system (GPS data) or manual tags. A localized position within a space can be determined via audio cues provided from various agents through agent coordination or using other spatial cues such as Bluetooth beacons or the WLAN signal. 
     Audio cleanup includes improving signal quality from acoustic collaboration of multiple listeners. A MICE agent can in some aspects identify simultaneous audio requests and specify where that request should be honored for high-quality understanding. The MICE agent can use passive observation to record a feed and cancel out noise on a secondary feed. Alternatively, strong signals in some areas may cause the agent to always reject commands (e.g. no agent should listen to sounds that originate in a bedroom). 
     Continuing with  FIG.  3   , audio cleanup module  304  includes localized data  340  and context data  342 . Localized data  340  describes where the particular agent is working on the premises and specifies audio enhancement routines. For example, a tile bathroom or kitchen may produce echoes that obscure audio requests. Context data describes audio enhancement based on removal of conflicting sounds as determined from other agents using the local network mesh and causes the removal of conflicting background sounds from buffered audio. Examples of such conflicting background sounds include users in other rooms, ambient music, and location-wide noise. Agent  250  can, over time, automatically sort ambient sounds into sound categories stored as sound category data  320 , and audio cleanup module  304  can use the sound categories to provide audio cleanup. 
     State machine  306  of agent  250  in  FIG.  3    includes localized data  346  and authorized data  348 . In addition to setting a current state based on previous audio requests and previous actions taken, a determination between multiple possible states can at least in part take into account localized data  346  that describes or depends on where an agent is located on the premises. Sound category data  320  can also inform the state machine&#39;s determination of a current state. Authorized data  348  describes who is authorized to provide audio requests, where certain audio requests are authorized, or both. 
     Continuing with  FIG.  3   , agent response coordinator  308  determines at which of the agents to actively process a selected audio request. In some aspects, active processing of a request means to honor the request by programmatically taking action or seeking to take action, as distinct from processing that may be carried out by one or more agents to localize the request. Agent response coordinator  308  may cause the smart device to actively process the request locally, or it may determine that another agent should honor the request and communicate that decision to the appropriate agent over the local network mesh. Attention token flag  310  is alternatively set to a value that represents either an activated or an inactivated attention token for the smart device at which agent  250  is operating. An agent can metaphorically grab an attention token to cue a user as to which agent is expecting to act on a request by displaying an indication such as a flashing LED. The user can then move nearer to another smart device if a correction is needed. 
     External network  324  of  FIG.  3    includes high-level awareness  326 . High-level awareness  326  is a module that optionally runs in the cloud to store and provide configuration information to smart device agents on the premises. It can provide some of the same functions as a MICE agent remotely. In this example, the functions are connected with such things as master user authorization and approximate location in order to eliminate the need to have local agents manage these functions. External services  328  are all the services typically provided to smart devices that are related to location awareness, including in some examples, GPS services. MICE cloud agent  332  duplicates the functions of local agents such as local agent  250 . MICE cloud agent  332  can provide these functions remotely to older smart devices without a fully capable local agent, or can provide them remotely when a user does not wish to enable the local agent or the local agent cannot function due to technical issues. The functions of a MICE agent as described herein can be provided locally, by the network cloud, or by a combination of both. 
       FIG.  4    is a flowchart illustrating a process  400  for providing multi-agent input coordination according to some aspects of the present disclosure. At block  402 , processor  204  automatically sorts ambient sounds into sound categories over time and the resulting sound category data is stored. At block  404 , processor  204  identifies audio requests received through listening agents in the network. At block  406 , processor  204  performs secondary trait analysis including footstep recognition, non-language sound cadence, habit pattern analysis, and tonal context to indicate urgency, mood, tone, etc. Non-language sound cadence can include the cadence of groans, shouts, breaths, or sighs. Habit patterns include phrases and sounds uttered frequently such as saying the word okay at the beginning of sentences. The local agent can also use this information to frame context query responses, such as a quiet query mode to have a calming effect or a happy mode when appropriate. 
     At block  412  of  FIG.  4   , processor  204  determines contexts such as location context and person interaction context using localization, movement, and spatial use restrictions. Determining location context or person interaction context can include identification of individuals within the relevant space. Determining location context can also include triangulation and network features such as GPS, echolocation, visual identification and voiceprint recognition. Location context can also include detecting user groups that may be participating in a multi-player social experience. With such an arrangement, processor  204  executing agent  250  can localize generic requests to the appropriate agents. Localizing a request can include determining, for example, where an audio request such as, turn on light, applies. Such an audio request might apply near an entry door instead of at a desk or in the kitchen. 
     Still referring to  FIG.  4   , state machine  306  run by processor  204  is set to its current state at block  414  and the current state is stored in memory, for example, in RAM  211 . At block  416 , processor  204  executes the agent response coordinator  308  to determine at which agent (or agents) to actively process a selected audio request. At block  418 , the attention token is activated and an attention indication is displayed on the smart device corresponding to the selected agent. At block  420 , processor  204  determines if the request is valid based on spatial uses for the agent, location context, available connectivity, and other factors such as system-wide restrictions or prohibitions. If the request is valid, at block  422 , processor  204  takes action based on the selected audio request and stored current state. If the request is not valid, processor  204  issues a denial response at block  424 . 
     In certain aspects, the system can enforce spatial usage restrictions per agent and optionally restrict functions to specific smart device positioning. For example, the system can prevent a user from telling a dishwasher to start unless the user is near the dishwasher. States or actions within a single space can expire or be restricted. For example, banking requests can only be permitted at a desk, and the permission to issue valid banking requests can be set to expire. States or actions can be limited by the presence of certain individuals. The presence or lack thereof of a specified individual or specified individuals in a location can be referred to as person interaction context. Actions can be taken or restricted (which can be considered an action) based on person interaction context either alone or in combination with other factors. For example, a child can be prevented from turning on a television unless a parent is nearby. 
     In certain aspects, the system can allow privacy in different areas by removing conflicts between devices. Optionally, the users present near each agent can influence the personal assistant&#39;s behavior at that agent. For example, if children are near the agent, the personal assistant can adopt a friendlier or slower speaking voice than when the space is occupied solely by adults. In some aspects the need for passwords or access tokens can be eliminated by the system using user classification and ranking to allow secure access to electronic files or to other computer resources. 
     In certain aspects, agents can cooperator to accomplish system-wide updates of changing characteristics of users (visual, audio, etc.) to naturally progress the identification of a user to account for aging, growth, etc. Each spatialized agent can adapt different behaviors as appropriate. Smart devices with appropriate smart assistant agents can be carried on the person or embedded in clothing, and joined to the local network mesh as users walk down corridors. 
     Unless specifically stated otherwise, throughout this specification terms such as “processing,” “computing,” “determining,” “identifying,” or the like refer to actions or processes of a computing device, such as one or more computers or a similar electronic computing device or devices that manipulate or transform data represented as physical electronic or magnetic quantities within memories, registers, or other information storage devices, transmission devices, or display devices of the computing platform. 
     The system or systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device can include any suitable arrangement of components that provides a result conditioned on one or more inputs. Suitable computing devices include multipurpose microprocessor-based computing systems accessing stored software that programs or configures the computing system from a general-purpose computing apparatus to a specialized computing apparatus implementing one or more aspects of the present subject matter. Any suitable programming, scripting, or other type of language or combinations of languages may be used to implement the teachings contained herein in software to be used in programming or configuring a computing device. 
     Aspects of the methods disclosed herein may be performed in the operation of such computing devices. The order of the blocks presented in the examples above can be varied—for example, blocks can be re-ordered, combined, or broken into sub-blocks. Certain blocks or processes can be performed in parallel. 
     The foregoing description of the examples, including illustrated examples, of the subject matter has been presented only for the purpose of illustration and description and is not intended to be exhaustive or to limit the subject matter to the precise forms disclosed. Numerous modifications, adaptations, and uses thereof will be apparent to those skilled in the art without departing from the scope of this subject matter. The illustrative examples described above are given to introduce the reader to the general subject matter discussed here and are not intended to limit the scope of the disclosed concepts.