ROUTING OF USER COMMANDS ACROSS DISPARATE ECOSYSTEMS

A system for routing commands issued by a passenger of a vehicle to a Smart Home and/or an Internet of Things (IoT) ecosystem via a connection manager. Issued commands are obtained from utterances using speech recognition and analyzed using natural language understanding and natural language processing. Using the output of the natural understanding analysis, the connection manager determines where to send the command by identifying a target Smart Home and/or IoT ecosystem.

TECHNICAL FIELD

Disclosed herein are systems and methods for routing of user commands across disparate Smart Home/IoT ecosystems.

BACKGROUND

Consumers today are increasingly more connected to their environment whether it be their home, work, or vehicle. For example, smart home devices and systems have become ubiquitous within homes.

Multiple types of devices can be included within a smart home system. For example, a typical system can include smart speakers, smart thermostats, smart doorbells, and smart cameras. In such a system, each device can interact with the other devices and be controlled by a user from a single point of control. Connectivity amongst devices and single-point control can typically be accomplished only when each device within a system is manufactured by a single manufacturer or otherwise specifically configured to integrate. The integrated smart devices together with the smart home system can be called a smart home ecosystem or an internet-of-things (IoT) ecosystem.

Characteristics of an IoT ecosystem are interoperability between devices configured to receive and transmit data over a similar protocol or using a similar application program interface. IoT ecosystems typically have a shared hub comprising at least a management application and data repository for the data obtained from the devices. Additionally, these ecosystems typically require the devices to execute on a particular operating system such as the Android® or iOS® operating system.

IoT ecosystems are designed to restrict the types of devices permitted within the ecosystem. For example, the Google Home ecosystem integrates with Google's Nest products. End users can only achieve interoperability between devices manufactured by the same company, or one designed specifically to function with specific other manufacturers devices. Restricting interoperability in such a way reduces an end user's ability to select a disparate group of devices, instead end users must only buy devices manufactured by the same manufacturer or devices that all use the same communication protocol and/or control application or operating system.

It would therefore be advantageous to provide an integration platform within a vehicle, such as a car, enabling end users to interact with their existing smart home ecosystems including ecosystem configuration using a single control application.

SUMMARY

Described herein are systems and methods for using a connection manager to direct voice or multimodal commands received by an automotive assistant via cloud-based, artificial intelligence for vehicles. The automotive assistant or the cloud-based AI can determine where to route the commands based on an analysis of an utterance and/or other input. The analysis can include speech recognition and natural language understanding. Natural language understanding or natural language processing can be applied to recognized speech to determine a destination ecosystem.

Once a target ecosystem is identified, the automotive assistant or the cloud-based AI can identify an IoT ecosystem over which the command should be transmitted and then transmit the command. Transmission can include modifying the command into a format accepted by the target ecosystem, and/or using natural language understanding or natural language processing to modify the content of the command.

The cloud-based AI, or a module executing within the cloud-based AI, can receive feedback from the target ecosystem regarding the routed command, and use the received feedback to modify one or more natural language understanding or natural language processing models. Modifying the models can have the effect of modifying a list of target ecosystems associated with the originally received commands.

Described herein is a system for routing commands where the system includes a recognition module that receives one or more utterances from a head unit of a vehicle. The one or more utterances can include at least one command. Using this one command, the recognition module can identify a target ecosystem and a connection manager can transmit the at least one command to the target ecosystem.

The recognition module can include an automatic speech recognition module that transcribes the one or more utterances into text. The recognition module comprises a natural language understanding (NLU) module that interprets a meaning of the one or more utterances and identifies the target ecosystem based on the interpreted meaning. The NLU module can identify the target ecosystem using one or more NLU models accessed by the NLU module.

The connection manager receives feedback from the target ecosystem about the at least one command, and the NLU module updates the one or more NLU models based on the feedback.

DETAILED DESCRIPTION

Disclosed herein is a system for selecting a target ecosystem from a plurality of ecosystems based on a spoken utterance, and transmitting that utterance to the selected ecosystem. In a vehicle, a user may wish to control ecosystems such as personal assistant devices, home automation systems, etc., that are external to the vehicle. In some examples, the ecosystems may include Google® Home and SimpliSafe® ecosystems, Alexa® Home System, etc. Based on the utterance and contextual data associated therewith, the system may select an ecosystem, transmit the utterance or associated command to the ecosystem, and return a confirmation to the user interface within the vehicle to indicate to the user that the command has been carried out.

Illustrated inFIG. 1is an example embodiment of a system10for providing a vehicle assistant15that uses multiple modes of input to provide services to passengers within a vehicle. The system10may include a head unit (“HU”)30arranged within the vehicle. The system10may also include a phone or mobile device35communicatively linked to the HU30. The HU30and/or the mobile device35can be in communication with a cloud-based application20that provides functionality to the vehicle assistant15. Within the cloud-based application20is a natural language understanding (“NLU”) module25, an automatic speech recognition (“ASR”) module60, and authentication module40, a connection module65and an arbitration engine70. The connection module65can include a connection manager50, an authentication cache45and a cases cache55. Communication over the “cloud” may involve data transfer via wide area and/or local area networks, such as the Internet, Global Positioning System (GPS), cellular networks, Wi-Fi, Bluetooth, etc. Further, such communication may provide for communication between the vehicle an external or remote servers and/or databases, as well as other external applications, systems, vehicles, etc. This communication network may provide navigation, music or other audio, program content, marketing content, internet access, speech recognition, cognitive computing, artificial intelligence, to the vehicle. While the cloud-based application20is described as being cloud-based, other forms of storage and communication may also be contemplated.

The HU30and/or the mobile device35can reside within a vehicle, where a vehicle can be any machine able to transport a person or thing from a first geographical place to a second different geographical place that is separated from the first geographical place by a distance. Vehicles can include, but not be limited to: an automobile or car; a motorbike; a motorized scooter; a two wheeled or three wheeled vehicle; a bus; a truck; an elevator car; a helicopter; a plane; a boat; or any other machine used as a mode of transport. Further, the vehicle104may be autonomous, partially autonomous, self-driving, driverless, or driver-assisted vehicles. The vehicle104may be an electric vehicle (EV), such as a battery electric vehicle (BEV), plug-in hybrid electric vehicle (PHEV), hybrid electric vehicle (HEVs), etc.

Head units30can be the control panel or set of controls within the vehicle that are used to control operation of the vehicle. The HU30typically includes one or more processors or micro-processors capable of executing computer readable instructions. A vehicle HU30can be used to execute one or more applications such as navigation applications, music applications, communication applications, or assistant applications. In some instances, the HU30can integrate with one or more mobile devices35in the vehicle. In other instances, a phone or mobile device35can operate or provide the background applications for the HU30such that the HU30is a dummy terminal on which the mobile device35application is projected. In other instances, the HU30can access the data plan or wireless connectivity provided by a mobile device35to execute one or more wireless-based applications.

A HU30can communicate with a vehicle assistant15that can be provided in part by a cloud-based application20. The cloud-based application20can provide one or more services to a vehicle either via the vehicle assistant15or directly to the HU30. In some instances, the cloud-based application20can execute entirely in a remote location, while in other instances aspects of the cloud-based application20can be either cached in the HU30or executed locally on the HU30and/or mobile device35. In still other instances, multiple aspects of the cloud-based application20can be embedded in the HU30and executed thereon.

The cloud-based application20can provide natural language understanding (“NLU”) or automatic speech recognition (“ASR”) services. An ASR module60can provide the speech recognition system and language models needed to recognize utterances and transcribe them to text. The ASR module60can execute entirely within the context of the cloud-based application20, or aspects of the ASR module60can be distributed between the cloud-based application20and embedded applications executing on the HU30. The NLU module25provides the natural language understanding applications and natural language understanding models needed to understand the intent and meaning associated with recognized utterances. The NLU module25can include models specific to the vehicle assistant15, or specific to one or more IoT ecosystems. Example ecosystems82are illustrated inFIG. 2but in general may be non-vehicle systems configured to carry out commands external and remote from the vehicle, such as systems within a user's home, etc.

In some embodiments, the cloud-based application20can be referred to as a cloud- based artificial intelligence20, or cloud-based AI. The cloud-based AI20can include artificial intelligence and machine learning to modify ASR and NLU modules60,25based on feedback from target ecosystems.

An authentication module40can be included within the cloud-based application20and can be used to authenticate a user or speaker to any of the cloud-based application20, the vehicle assistant15, or a connected IoT ecosystem. The authentication module40can perform authentication using any of the following criteria: the VIN (vehicle identification number) of the vehicle; a voice biometric analysis of an utterance; previously provided login credentials; one or more credentials provided to the HU30and/or the cloud-based application20by the mobile device

35; or any other form of identification. Authentication credentials can be cached within the cloud-based application20, or in the case of the IoT ecosystems, within the connection module65.

The connection module65can be used to provide access to the vehicle assistant15and one or more IoT ecosystems. Within the connection module65is a connection manager50that manages which IoT ecosystem to connect to. The connection manager50can access databases within the connection module65, including a cache of authentication cache45and a cache of cases55. Cases55may be predetermined workflows that dictate the execution of applications according to a specified timeline and set of contexts. The connection manager50can access cases55within the cache45to determine which IoT ecosystem to connect with and where to send information.

A vehicle assistant15can be the interface that end users (i.e., passengers and/or drivers) interact with to access Smart Home and/or IoT ecosystems, send commands to the cloud-based application20or Smart Home or IoT ecosystems, or create automation case routines. The vehicle assistant15can be referred to as an automotive assistant, an assistant, or the CERENCE Assistant. In some instances, the vehicle assistant15can include the CERENCE Drive 2.0 framework which can include one or more applications that provide ASR and NLU services to a vehicle. The vehicle assistant15may be an interface configured to integrate different products and applications such as text to speech applications, etc. The vehicle assistant15can include a synthetic speech interface and/or a graphical user interface that is displayed within the vehicle.

Communication between the vehicle, cloud-based application20, and target ecosystem may be carried out via cloud communication and may involve data transfer via wide area and/or local area networks, such as the Internet, Global Positioning System (GPS), cellular networks, Wi-Fi, Bluetooth, etc. This communication network may provide for communication between the vehicle and external or remote servers and/or databases, as well as other external applications, systems, vehicles, etc. This communication network may provide navigation, music or other audio, program content, marketing content, internet access, speech recognition, cognitive computing, artificial intelligence, to the vehicle.

The “modules” discussed herein may include one or more computer hardware processors coupled to one or more computer storage devices for performing steps of one or more

methods as described herein and may enable the system10to communicate and exchange information and data with systems and subsystems. The modules, vehicle, cloud AI, HU30, mobile device35, and vehicle assistant15, among other components may include one or more processors configured to perform certain instructions, commands and other routines as described herein. Internal vehicle networks may also be included, such as a vehicle controller area network (CAN), an Ethernet network, and a media oriented system transfer (MOST), etc. The internal vehicle networks may allow the processor to communicate with other vehicle systems, such as a vehicle modem, a GPS module and/or Global System for Mobile Communication (GSM) module configured to provide current vehicle location and heading information, and various vehicle electronic control units (ECUs) configured to corporate with the processor.

The vehicle may include various sensors and input devices as part of the system10. For example, the vehicle may include at least one microphone. The microphone132may be configured to receive audio signals from within the vehicle cabin, such as acoustic utterances including spoken words, phrases, or commands from vehicle occupants. The microphone may include an audio input configured to provide audio signal processing features, including amplification, conversions, data processing, etc.

Other sensors, such as GPS, occupant detection, vehicle safety system, cameras, timers, clocks, etc., may also be included in the vehicle. These systems may provide contextual data associated with the utterances to aid in selecting the appropriate ecosystem.

Referring toFIG. 2, the system10includes the vehicle assistant15, cloud-based application20, and ecosystem application programming interfaces (APIs)80. The ecosystem APIs80may correspond to various ecosystems82. As explained above, the ecosystems82may include various smart systems outsides of the vehicle systems such as personal assistant devices, home automation systems, etc. In some example, the ecosystems may include Google® Home and SimpliSafe® ecosystems, Alexa® Home System, etc. The cloud-based application20can include a configuration management service (not shown) that permits end users to on-board IoT ecosystems. These IoT ecosystems can be a home automation ecosystem or any other ecosystem that permits wireless-enabled devices to interoperate, communicate with each other, and be controlled by a single application or control point. The system10can provide end-users (i.e., car

manufacturer OEMs, and/or car owners/users) with the ability to access multiple IoT ecosystems. For example, an end user can choose to access the Google® Home and SimpliSafe® ecosystems. In the future, if the end user wants to access the Alexa® Home System, the end user can use the configuration management service to on-board the Alexa® Home System ecosystem. This may be done, for instance, by establishing a connection with the specific device and the cloud-based application20so that the cloud-based application20may communicate with the ecosystem. This may be done in a set up mode via a user interface on the mobile device35, or via an interface within the vehicle.

In some instances, the configuration management service can include a storage repository for storing configuration profiles, an application program interface (API) for providing an end-user with the ability to on-board new ecosystems, and various backend modules for configuring access to an ecosystem. On-boarding and establishing a connection with an ecosystem requires access to that ecosystem's API or suite of APIs80. The cloud-based application20includes an API access module that provides an interface between the cloud-based application20and the ecosystem API(s)80.

The vehicle assistant15can be a front end to the cloud-based application20such that the vehicle assistant15receives utterances and serves them to the cloud-based application20for processing. The vehicle assistant15can also manage authentication.

For example, the vehicle assistant15can receive an utterance and send the utterance to the cloud-based application20for processing. Within the cloud-based application20, the ASR module60uses ASR applications and language models to translate the utterance to text.

The NLU module25then uses the translated text of the utterance and various other types of information to determine the intent of the utterance. Other types of information or utterance data may be contextual data indicative of non-audio contextual circumstances of the vehicle or driver. The contextual data may include, but not be limited to: the time of day; the day of the week; the month; the weather; the temperature; where the vehicle is geographically located; how far away the vehicle is located from a significant geographic location such as the home of the driver; whether there are additional occupants in the vehicle; the vehicle's identification number;

the biometric identity of the person who spoke the utterance; the location of the person who spoke the utterance within the vehicle; the speed at which the vehicle is traveling; the direction of the driver's gaze; whether the driver has an elevated heart rate or other significant biofeedback; the amount of noise in the cabin of the vehicle; or any other relevant contextual information. Using this information, the NLU module25can determine whether the utterance included a command and to which ecosystem the command is directed.

For example, a driver of an automobile can say “turn on the lights”. The vehicle assistant15can send this utterance to the cloud-based application20where the utterance is translated by the ASR module60to be “turn on the lights”. The NLU module25can then use the fact that it is five o′clock at night, half a mile from the driver's home to know that the command should be sent to the driver's Alexa® Home System. The cloud-based application20can then send the command to the driver's Alexa® Home System and receive a confirmation from the driver's Alexa® Home System that the command was received and executed. Based on this received confirmation, the cloud-based application20can update the NLU/NLP models of the NLU module25to increase the certainty around the determination that when the driver of the car is a half mile from their house at five o'clock at night and utters the phrase “turn on the lights”, the utterance means that the cloud-based application20should instruct the driver's Alexa® Home System to turn on the lights.

In another example, a driver of an automobile can say “lock the doors”. The vehicle assistant15can send this utterance to the cloud-based application20where the utterance is translated by the ASR module60to be “lock the doors”. The NLU module25can then use the fact that the driver is more than ten miles from home to determine that the command should be sent to the driver's SimpliSafe® system. The cloud-based application20can then send the command to the driver's SimpliSafe® system and receive a confirmation from the driver's SimpliSafe® system that the command was received and not executed. Based on this received confirmation, the cloud-based application20can update the NLU/NLP models of the NLU module25so that when a “lock the doors” command is received, the command is not sent to the driver's SimpliSafe® system.

As explained above, the vehicle assistant15can be the interface that end users (i.e., passengers and/or drivers) interact with to access IoT ecosystems82, send commands to the cloud-based application20or IoT ecosystems82, or create cases. The vehicle assistant15can be referred to as an automotive assistant, an assistant, or the CERENCE Assistant. In some instances, the vehicle assistant15can include the CERENCE Drive 2.0 framework18which can include one or more applications that provide ASR and NLU services to a vehicle. The vehicle assistant15may be an interface configured to integrate different products and applications such as text to speech applications, etc. The vehicle assistant15can include a synthetic speech interface and/or a graphical user interface22that is displayed within the vehicle.

The user interface22may be configured to display information relating to the target ecosystem. For example, once the ecosystem is selected, the user interface22may display an image or icon associated with that ecosystem82. The user interface22may also display a confirmation that the target ecosystem82received the command and also when the ecosystem82has carried out the command. The user interface22may also display a lack of response to the command by the ecosystem82as well.

FIG. 3illustrates an example process300for the routing system ofFIGS. 1 and 2. The process300may begin at step302where the cloud-based application20or ecosystem82asks for permission to access the vehicle assistant15. This may be facilitated via the mobile device35. This request may include a user specific secret or key in order to identify the user and/or the application. The vehicle assistant15may return an authentication token at step304via a direct really simple syndication (DRSS)24. This may allow one cloud system to communicate with another (e.g., vehicle system15and cloud-based application20.) At step306, the authentication is stored in a database, such as the authentication database45. At step308, the authentication confirmation is returned to the mobile device35.

Once authenticated, at block310, the vehicle assistant15may receive an utterance at the head unit30. In the example shown inFIG. 3, the utterance may be “turn on the lights.” The command may be received by speakers within the vehicle or at the mobile device35. The command may be received by the vehicle assistant15, including a speech recognition framework such as the CERENCE Drive 2.0 framework.

At step312, the vehicle assistant15may forward the utterance to the ASR module60within the cloud-based application20. At step314, the ASR module60may process the utterance to convert the text to speech and the may NLU module25to determine the intent of the utterance. Concurrently or near concurrently, at step316, an embedded speech recognizer, such as CERENCE VoCon ASR engine, may process speech recognition at the head unit30. At step318, the ASR module60may forward the ASR and NLU results to an arbitration engine70. The arbitration engine may evaluate the utterance. This may include evaluating the cloud domain and/or the topic probability. This may include an intermediate result of the utterance. The intermediate result may include a best guess at the intent of the utterance. For example, the intermediate result may indicate that the utterance is a command that is system-directed and intended to direct an action to a particular device. In this example, it is to turn on the lights.

At step322, the arbitration engine70returns the intermediate result to the ASR module60. Meanwhile, at step324, the vehicle assistant15awaits the intermediate result until the ASR module60returns the intermediate result at step326. At step328, the vehicle assistant15evaluates the intermediate result.

Concurrently or near concurrently to sending the intermediate result to the ASR module60, the arbitration engine70also forwards the ASR and NLU results to the connection manager50or the authentication module40at step330. The connection manager50may be a cloud interface configured to allow for the cloud dialog. At step332, the connection manager50may forward the ASR results to the DRSS24service. The DRSS24service may be part of the connection module65or the connection manager50. The DRSS24service may also be a separate communication engine.

The DRSS24service may then evaluate which assistant or IoT ecosystem82to use, receive the necessary authentication tokens, and convert the ASR back to an audio signal at steps334,336,338, respectively. At step340, the DRSS24service may send the token and the audio command to the vehicle assistant15. The vehicle assistant15may then return an audio response to the DRSS24service at step342, and the DRSS24service may in turn transmit the audio response to the ASR module60at step344. The ASR module60may return an ASR result to the DRSS24service at step346. The DRSS24service may then return a text to speech (TTS)

response to the connection manager50at step348, which may in turn transmit the TTS response to the ASR module60at step350. At step352, the ASR module60may transit its response back to the vehicle assistant15.

Computing devices described herein generally include computer-executable instructions where the instructions may be executable by one or more computing devices such as those listed above. Computer-executable instructions, such as those of the virtual network interface application202or virtual network mobile application208, may be compiled or interpreted from computer programs created using a variety of programming languages and/or technologies, including, without limitation, and either alone or in combination, JavaTM, C, C++, C#, Visual Basic, JavaScript, Python, TypeScript, HTML/CSS, Swift, Kotlin, Perl, PL/SQL, Prolog, LISP, Corelet, etc. In general, a processor (e.g., a microprocessor) receives instructions, e.g., from a memory, a computer-readable medium, etc., and executes these instructions, thereby performing one or more processes, including one or more of the processes described herein. Such instructions and other data may be stored and transmitted using a variety of computer-readable media.

Accordingly, it is to be understood that the above description is intended to be illustrative and not restrictive. Many embodiments and applications other than the examples provided would be apparent upon reading the above description. The scope should be determined, not with reference to the above description, but should instead be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled. It is anticipated and intended that future developments will occur in the technologies discussed herein,

and that the disclosed systems and methods will be incorporated into such future embodiments. In sum, it should be understood that the application is capable of modification and variation.