Patent Description:
<CIT> discloses a voice-based system comprising a local speech interface device and a remote control service. A user may interact with the system using speech to obtain services and perform functions. The system may allow a user to install applications to provide enhanced or customized functionality. Such applications may be installed on either the speech interface device or the control service. The control service receives user speech and determines user intent based on the speech. If an application installed on the control service can respond to the intent, that application is called. Otherwise, the intent is provided to the speech interface device which responds by invoking one of its applications to respond to the intent.

LIND RET AL: "THE NETWORK VEHICLE - A GLIMPSE INTO THE FUTURE OF MOBILE MULTI-MEDIA" discloses a vision for the future convergence of computers, the communications infrastructure, and the automobile. It features functions such as: satellite video, Internet access, virtual navigation, remote vehicle diagnostics and control, games, mobile office, automotive web site, and customized real-time stock quotes and sports scores. These features are enabled by an integrated planar antenna that is capable of multiple satellite reception, a client-server network architecture, and human-vehicle-interfaces such as color reconfigurable head-up and head-down displays, steering wheel controls, voice recognition, text-to-speech, and large touch screen active matrix liquid crystal displays (LCDs).

According to an aspect of the disclosure, a system to selectively invoke applications for execution can include a data processing system of a vehicle. A natural language processor component executed by the data processing system can receive, via an interface, an input audio signal. The natural language processor component can parse the input audio signal to identify a request and a command corresponding to the request. A configuration tracker executed by the data processing system can identify a first functionality of a first digital assistant application hosted on the data processing system in the vehicle and a second functionality of a second digital assistant application accessible via a client device communicatively coupled to the data processing system. An application selector executed by the data processing system can determine that one of the first functionality or the second functionality supports the command corresponding to the request. The application selector can select one of the first digital assistant application hosted on the data processing system or the second digital assistant application accessible via the client device based on the determination that one of the first functionality or the second functionality supports the command. The application selector can invoke one of the first digital assistant application or the second digital assistant application, selected based on the determination that one of the first functionality or the second functionality supports the command.

According to an aspect of the disclosure, a method to validate vehicular functions can include a natural language processor component executed by a data processing system of a vehicle receiving, via an interface, an input audio signal. The method can include the natural language processor component parsing the input audio signal to identify a request and a command corresponding to the request. The method can include a configuration tracker executed by the data processing system identifying a first functionality of a first digital assistant application hosted on the data processing system in the vehicle and a second functionality of a second digital assistant application accessible via a client device communicatively coupled to the data processing system. The method can include an application selector executed by the data processing system, determining that one of the first functionality or the second functionality supports the command corresponding to the request. The method can include the application selector selecting one of the first digital assistant application hosted on the data processing system or the second digital assistant application accessible via the client device based on the determination that one of the first functionality or the second functionality supports the command. The method can include the application selector invoking one of the first digital assistant application or the second digital assistant application, selected based on the determination that one of the first functionality or the second functionality supports the command.

The foregoing information and the following detailed description include illustrative examples of various aspects and implementations and provide an overview or framework for understanding the nature and character of the claimed aspects and implementations.

Following below are more detailed descriptions of various concepts related to, and implementations of, methods, apparatuses, and systems to selectively invoke applications for execution in vehicular environments. The vehicular environment can include the vehicle and its surroundings, including a radius around the vehicle (e.g. <NUM> feet) as well as devices that are connected to the vehicle via wired or wireless electronic communication systems. The various concepts introduced above and discussed in greater detail below may be implemented in any of numerous ways.

The present disclosure is generally directed to a data processing system to selectively invoke applications for execution in vehicular environments. The systems and methods described herein include a data processing system in a vehicle that can receive an audio input query, which can also be referred to herein as an audio input signal. From the audio input query, the data processing system identifies a request. The request can correspond to a command to perform one of various functions, such as a vehicular function (e.g., to open a window or to activate a windshield wiper) or a non-vehicular function (e.g., to make a phone call or to access a calendar). Digital assistant applications can carry, obtain, respond to, or process the command extracted from the audio input query. The digital assistant application can be a program or script executed on the data processing system, the vehicle, and the client device (e.g., a smartphone, tablet, laptop, desktop, etc.) interfacing with the data processing system via the cloud. The digital assistant application can receive audio input queries, process the requests associated with such queries using natural language processing algorithms, and then present an audio response in a conversational manner.

There are multiple digital assistant applications available in the vehicular environment, each with various functionalities to carry out the requested command. Certain commands are performed by a primary digital assistant application hosted on the data processing system in the vehicle, while other commands are performed by a secondary digital assistant application accessible via a client device (e.g., smartphone or tablet). The client device and the data processing system have established a communication link with each other Even with commands that may be carried out either on the data processing system or through the client device, some commands can be executed faster or more efficiently at one digital assistant application versus the other digital assistant application because such commands may be performed via network access. With vehicular environments, however, there may be limited connectivity to the network. The data processing system on the vehicle itself may not be connected to the network. In addition, the client device may move in and out of range of wireless networks. As such, the functionalities of either the primary or secondary digital assistant application may vary.

To carry out the identified command, the data processing system can identify functionalities of the primary digital assistant application hosted in the vehicle and functionalities of the secondary digital assistant application accessible via the client device. With the functionalities of both applications identified, the data processing system can determine which digital assistant application supports the command as requested in the audio input query. Based on the determination, the data processing system can select which of the primary or secondary digital assistant application to use in carrying out the requested command. Once selected, the data processing system can invoke the corresponding digital assistant application. When the primary digital assistant application is selected, the data processing system can execute the requested command via the primary digital assistant application. Conversely, when the secondary digital assistant application is selected, the data processing system can send a command to the client device via the communication link to carry out the requested command using the secondary digital assistant application.

The present solution can improve efficiency and effectiveness of auditory data packet processing by selecting the appropriate digital assistant application to perform the command of the request contained in the auditory data packet. As the functionalities may vary across the multiple digital assistant applications, the present solution also can reduce network bandwidth and computing resource consumption by selecting the digital assistant application that can perform the command more efficiently than the other applications. From a human-computer interaction (HCI) perspective, the seamless transition in selectively invoking one of multiple digital assistant applications may improve the utility of all the digital assistant applications available in the vehicular environment.

<FIG> illustrates an example system <NUM> to selectively invoke applications for execution in vehicular environments. The system <NUM> can include at least one data processing system <NUM>, one or more client devices <NUM>, and one or more vehicles <NUM>. The one or more client devices <NUM> can be communicatively coupled to the one or more vehicles <NUM>, and vice-versa. The at least one data processing system <NUM>, one or more client devices <NUM>, and one or more vehicles <NUM> can be communicatively coupled to one another via the network <NUM>.

The data processing system <NUM> can include an interface <NUM>. The data processing system <NUM> can include a natural language processor (NLP) component <NUM> to parse audio-based inputs. The data processing system <NUM> can include an interface management component <NUM> to detect and manage the interfaces of other devices in the system <NUM>. The data processing system <NUM> can include an audio signal generator component <NUM> to generate audio-based signals. The data processing system <NUM> can include a direct action application programming interface (API) <NUM>. The data processing system <NUM> can include a response selector component <NUM> to select responses to audio-based input signals. The NLP component <NUM>, the interface management component <NUM>, the audio signal generator component <NUM>, the data repository <NUM>, the direct action API <NUM>, and the response selector component <NUM> may be part of a digital assistant application <NUM>. The data processing system <NUM> can include a data repository <NUM>. The data repository <NUM> can store parameters <NUM>, policies <NUM>, response data <NUM>, and templates <NUM>. The data processing system <NUM> can also include a configuration tracker component <NUM>, an application selector component <NUM>, and a performance evaluator component <NUM>, among others.

The functionalities of the data processing system <NUM>, such as the digital assistant application <NUM>, can be included or otherwise be accessible from the one or more client devices <NUM> and the one or more vehicles <NUM>. The functionalities of the data processing system <NUM> may correspond to the functionalities or interface with the digital assistant application <NUM> executing on the client devices <NUM> or the vehicles <NUM>. The client devices <NUM> can each include and execute a separate instance of the one or more components of the digital assistant application <NUM>. The client devices <NUM> can otherwise have access to the functionalities of the components of the digital assistant application <NUM> on a remote data processing system <NUM> via the network <NUM>. For example, the client device <NUM> can include the functionalities of the NLP component <NUM> and access the remainder of the components of the digital assistant application <NUM> via the network <NUM> to the data processing system <NUM>. The vehicle <NUM> can include and execute a separate instance of the components of the data processing system <NUM>, such as the digital assistant application <NUM>, the configuration tracker component <NUM>, the application selector component <NUM>, and the performance evaluator component <NUM>. For example, the digital assistant application <NUM> may be pre-installed on a head unit of a vehicle <NUM>. The digital assistant applications <NUM> accessible or executing on the client devices <NUM> and the vehicles <NUM> may each have different functionalities.

The client devices <NUM> and the vehicles <NUM> can each include at least one logic device such as a computing device having a processor to communicate with each other with the data processing system <NUM> via the network <NUM>. The client devices <NUM> and the vehicles <NUM> can include an instance of any of the components described in relation to the data processing system <NUM>. The client devices <NUM> and the vehicles <NUM> can include an instance of the digital assistant application <NUM>. The client devices <NUM> can include a desktop computer, laptop, tablet computer, personal digital assistant, smartphone, mobile device, portable computer, thin client computer, virtual server, speaker-based digital assistant, or other computing device. The vehicle <NUM> can be a car, truck, motorcycle, airplane, blimp, boat, submarine, or other transportation device. The vehicle <NUM> can include one or more processors that can execute an instance of the digital assistant application <NUM> or any component thereof. The processors can be a component of the head unit disposed in the vehicle <NUM>, such as a head unit in an automobile.

The components of the system <NUM> can communicate over a network <NUM>. In some implementations, one or more of the client devices <NUM> can be located within an instance of the vehicle <NUM>. For example, the client devices <NUM> can be the mobile phone of a driver driving the vehicle <NUM>. In some implementations, one or more of the client devices <NUM> can be remote to the vehicles <NUM>. For example, after the driver parks and exits the vehicle <NUM> for work, the driver's mobile phone may be remote to the vehicle <NUM>.

The network <NUM> can include, for example, a point-to-point network, a broadcast network, a wide area network, a local area network, a telecommunications network, a data communication network, a computer network, an ATM (Asynchronous Transfer Mode) network, a SONET (Synchronous Optical Network) network, a SDH (Synchronous Digital Hierarchy) network, an NFC (Near-Field Communication) network, a local area network (LAN), a wireless network or a wireline network, and combinations thereof. The network <NUM> can include a wireless link, such as an infrared channel or satellite band. The topology of the network <NUM> may include a bus, star, or ring network topology. The network <NUM> can include mobile telephone networks using any protocol or protocols used to communicate among mobile devices, including advanced mobile phone protocol (AMPS), time division multiple access (TDMA), code-division multiple access (CDMA), global system for mobile communication (GSM), general packet radio services (GPRS), or universal mobile telecommunications system (UMTS). Different types of data may be transmitted via different protocols, or the same types of data may be transmitted via different protocols.

The network <NUM> can include a short-range communication link (e.g., ranging up to <NUM> meters) established between the client devices <NUM> and the vehicles <NUM>, using Bluetooth, Bluetooth Low Energy, Dedicated Short-Range Communications (DSRC), or Near-Field Communications (NFC) protocols. Using such protocols, the data processing system <NUM> can establish a communication link with one or more client devices <NUM> via the interface <NUM>. The data processing system <NUM> can establish a communication link with one or more vehicles <NUM> via the interface <NUM>. The short-range communication link may be established between the client devices <NUM> and the vehicles <NUM> via pairing protocol.

The client devices <NUM> can include sensors <NUM>, speakers <NUM>, peripheral devices <NUM>, interfaces <NUM>, and transducers <NUM>. The client devices <NUM> can execute an instance of the digital assistant application <NUM>. The vehicles <NUM> can include sensors <NUM>, speakers <NUM>, peripheral devices <NUM>, interfaces <NUM>, and transducers <NUM>. The vehicles <NUM> can execute an instance of the NLP component <NUM>. The peripheral devices <NUM> for the client device <NUM> can include user input/output devices, such as a keyboard, a monitor, and headphones, among others. The peripheral devices <NUM> for the vehicle <NUM> can include any other devices, such as a garage door opener, a window opener, a passenger door opener, a trunk opener, a multimedia device player, and a temperature control, among others.

The client devices <NUM> and the vehicle <NUM> can include at least one sensor <NUM>, at least one transducer <NUM>, at least one audio driver, and at least one speaker <NUM>. The sensor <NUM> includes a microphone. The sensor <NUM> can also include at least one of a GPS sensor, proximity sensor, ambient light sensor, temperature sensor, motion sensor, accelerometer, or gyroscope. The sensor can include an occupancy or weight sensor. The transducer <NUM> can convert the audio input into an electronic signal. The audio driver can include a script or program executed by one or more processors of the client devices <NUM> or the vehicle <NUM> to control the speaker <NUM>. The speaker <NUM> can render audio signals by converting electrical signals into audible waves.

The client devices <NUM> and the vehicle <NUM> can be associated with an end user that enters voice queries as input audio signals into the client devices <NUM> or the vehicle <NUM> (via the sensor <NUM>) and receives audio output in the form of a computer generated voice that can be provided from the data processing system <NUM>. In response to the input audio signals, the client devices <NUM> and the vehicle <NUM> can also receive action data structures to perform predetermined functions or actions. The interface <NUM> can receive or provide data messages to the direct action API <NUM> of the data processing system <NUM> and enable communication between the components of the system <NUM>. The client devices <NUM> and the vehicle <NUM> can also include a user interface that enables a user to interact with the components of the system <NUM>.

The data processing system <NUM> can include at least one server having at least one processor. For example, the data processing system <NUM> can include a plurality of servers located in at least one data center or server farm. The data processing system <NUM> can determine from an audio input signal a request and a trigger keyword associated with the request. Based on the request and trigger keyword, the data processing system <NUM> can generate or select response data. The response data can be audio-based or text-based. For example, the response data can include one or more audio files that, when rendered, provide an audio output or acoustic wave. The data within the response data can also be referred to as content items. The response data can include other content (e.g., text, video, or image content) in addition to audio content.

The data processing system <NUM> can include multiple, logically grouped servers and facilitate distributed computing techniques. The logical group of servers may be referred to as a data center, server farm, or a machine farm. The servers can be geographically dispersed. A data center or machine farm may be administered as a single entity, or the machine farm can include a plurality of machine farms. The servers within each machine farm can be heterogeneous-one or more of the servers or machines can operate according to one or more type of operating system platform. The data processing system <NUM> can include servers in a data center that are stored in one or more high-density rack systems, along with associated storage systems, located for example in an enterprise data center. In this way, the data processing system <NUM> with consolidated servers can improve system manageability, data security, the physical security of the system, and system performance by locating servers and high performance storage systems on localized high performance networks. Centralization of all or some of the data processing system <NUM> components, including servers and storage systems, and coupling them with advanced system management tools allows more efficient use of server resources, which saves power and processing requirements and reduces bandwidth usage. Each of the components of the data processing system <NUM> can include at least one processing unit, server, virtual server, circuit, engine, agent, appliance, or other logic device such as programmable logic arrays configured to communicate with the data repository <NUM> and with other computing devices.

The data processing system <NUM> can include the data repository <NUM>. The data repository <NUM> can include one or more local or distributed databases and can include a database management system. The data repository <NUM> can include computer data storage or memory and can store one or more parameters <NUM>, one or more policies <NUM>, response data <NUM>, and templates <NUM>, among other data. The parameters <NUM>, policies <NUM>, and templates <NUM> can include information such as rules about a voice based session between the client devices <NUM>, the data processing system <NUM>, and the vehicle <NUM>. The parameters <NUM>, policies <NUM>, and templates <NUM> can also include information for another digital assistant application <NUM> received via the interface <NUM> from another source (e.g., the data processing system <NUM>, the client devices <NUM>, and the vehicles <NUM>). For example, the parameters <NUM>, policies <NUM>, and templates <NUM> stored in the data repository <NUM> of a digital assistant application <NUM> hosted on a vehicle <NUM> can include the parameters <NUM>, policies <NUM>, and templates <NUM> from the data repository <NUM> of a digital assistant application <NUM> accessible via the client device <NUM>, and vice-versa. In this manner, the parameters <NUM>, policies <NUM>, and templates <NUM> of different digital assistant applications <NUM> may be shared and used by one another. The response data <NUM> can include content items for audio output or associated metadata, as well as input audio messages that can be part of one or more communication sessions with the client devices <NUM>.

An application, script, program, or other components that are associated with the data processing system <NUM> can be installed at the client devices <NUM> or the vehicle <NUM>. The application can enable the client devices <NUM> or vehicle <NUM> to communicate input audio signals (and other data) to the interface <NUM> of the data processing system <NUM>. The application can enable the client devices <NUM> and the vehicle <NUM> to drive components of the client devices <NUM> and the vehicle <NUM> to render the output audio signals.

The NLP component <NUM> can receive input audio signals. The data processing system <NUM> can receive the input audio signal from the client devices <NUM> or the vehicle <NUM> (e.g., via the transducers <NUM>) included in a data packet. A first device can execute the NLP component <NUM>, and the NLP component <NUM> can receive the input audio signal from a second device. For example, a client device <NUM> can receive the input audio signal from and the data processing system <NUM> can execute the NLP component <NUM>.

The NLP component <NUM> can convert input audio signals into recognized text by comparing the input audio signal against a stored, representative set of audio waveforms and choosing the closest matches. The representative waveforms can be generated across a large set of input audio signals. Once the input audio signal is converted into recognized text, the NLP component <NUM> can match the text to words that are associated, for example, via a learning phase, with actions or output audio signals.

From the input audio signal, the NLP component <NUM> identifies at least one request or at least one command corresponding to the request. The request can indicate intent or subject matter of the input audio signal. The command indicates a type of action likely to be taken. For example, the NLP component <NUM> can parse the input audio signal to identify at least one request to open the vehicle's windows or skip to a next audio file in a music playlist. The command can include at least one word, phrase, root or partial word, or derivative indicating an action to be taken. The request can also include a trigger keyword. The trigger keyword may be associated with a particular digital assistant application accessible via the client device <NUM> or running on the vehicle <NUM>. For example, the trigger keywords "go" may be associated with the digital assistant application accessible via the client device <NUM>, and "ok" may be associated with the digital assistant application running on the vehicle <NUM>. Which digital assistant application is selected to execute the indicated command is detailed herein.

The NLP component <NUM> determines whether the at least one command corresponding to the request of the input audio signal relates to a vehicle functionality of the vehicle <NUM> or to a non-vehicle functionality of the client device <NUM>. Upon extraction of the intent, subject matter, and the command from the input audio signal, the NLP component <NUM> compares the word, phrases, root or partial words, or derivatives to a semantic knowledge graph. The semantic knowledge graph may specify a set of words, phrases, root or partial words, or derivatives relating to a vehicle functionality and non-vehicle functionality. For example, "windshield" may be marked as related to a vehicle functionality and "phonebook" may be marked as related to a non-vehicle functionality. The semantic knowledge graph may also specify a relationship between each word, phrase, root or partial word, or derivative. The semantic knowledge graph may be maintained at the data repository <NUM>. Based on the comparison with the semantic knowledge graph, the NLP component <NUM> can determine whether the word, phrase, root or partial word, or derivative of the command relates to a vehicle functionality or a non-vehicle functionality.

The response selector component <NUM> can obtain information from the data repository <NUM> where it can be stored as part of the response data <NUM>. The response selector component <NUM> can query the data repository <NUM> to select or otherwise identify response phrases or content items, e.g., from the response data <NUM>.

The audio signal generator component <NUM> can generate or otherwise obtain an output signal that includes the content item. The data processing system <NUM> can execute the audio signal generator component <NUM> to generate or create an output signal corresponding to the content item or request. For example, once a request is fulfilled, the audio signal generator component <NUM> can generate an audio output signal that includes the phrase "The action was completed.

The interface <NUM> can be a data interface or a network interface that enables the components of the system <NUM> to communicate with one another. The interface <NUM> of the data processing system <NUM> can provide or transmit one or more data packets that include the action data structure, audio signals, or other data via the network <NUM> to the client devices <NUM> or vehicle <NUM>. For example, the data processing system <NUM> can provide the output signal from the data repository <NUM> or from the audio signal generator component <NUM> to the client devices <NUM>. The data processing system <NUM> can also instruct, via data packet transmissions, the client devices <NUM> or the vehicle <NUM> to perform the functions indicated in the action data structure. The output signal can be obtained, generated, transformed to, or transmitted as one or more data packets (or other communications protocol) from the data processing system <NUM> (or other computing device) to the client devices <NUM> or the vehicle <NUM>.

The direct action API <NUM> of the data processing system <NUM> can generate, based on, for example, the request, action data structures. The action data structure can include data or instructions for the execution of a specified action to satisfy the request. In some implementations, the action data structure can be a JSON formatted data structure or an XML formatted data structure. The direct action API <NUM> may be prevented from carrying out the action specified in the request until invocation by the application selector component <NUM>. The details of the functionalities of the application selector component <NUM> in relation to the direct action API <NUM> are explained below.

Depending on the action specified in the request, the direct action API <NUM> can execute code or a dialog script that identifies the parameters required to fulfill the request. The action data structures can be generated responsive to the request. The action data structure can be included in messages that are transmitted to or received by the client devices <NUM> or the vehicle <NUM>. Based on the request parsed by the NLP component <NUM>, the direct action API <NUM> can determine to which of the client devices <NUM> or the vehicles <NUM> the message should be sent for processing. Whether the client device <NUM> or the vehicle <NUM> is selected to process the message is detailed below with respect to the application selector component <NUM>. The direct action API <NUM> can package the request into an action data structure for transmission to the vehicle <NUM>. The direct action API <NUM> can access a vehicle ID from the response data <NUM> to determine which vehicle is associated with the user that generated the request. Once received, the vehicle <NUM> can process the action data structure and can perform the indicated action. The direct action API <NUM> can also package the request into an action data structure for execution by the client device <NUM>. Once received, the client device <NUM> can process the action data structure using the digital assistant application <NUM> or one or more applications running on the client device <NUM>.

The action data structure can include information for completing the request. For example, the action data structure can be an XML or JSON formatted data structure that includes attributes used in completing or otherwise fulfilling the request. The attributes can include a location of the vehicle <NUM>, a location of the client devices <NUM>, an authorization level of a user associated with a client devices <NUM>, a vehicle identifier, an interface identifier, a vehicular state, or a request state. In some implementations, the request state includes one or more attributes that should be satisfied before the action is fulfilled. For example, with the request "Ok, change the song," the request state may have the attribute {requestor: [authorized, passenger]}, indicating that the request should be an explicitly authorized user or a passenger in the vehicle.

The direct action API <NUM> can retrieve a template <NUM> from the repository <NUM> to determine which fields or attributes to include in the action data structure. The direct action API <NUM> can determine necessary parameters and can package the information into an action data structure. The direct action API <NUM> can retrieve content from the repository <NUM> to obtain information for the attributes of the data structure.

The direct action API <NUM> can populate the fields with data from the input audio signal. The direct action API <NUM> can also populate the fields with data from the client devices <NUM> or the vehicle <NUM>, or from another source. The direct action API <NUM> can prompt a user for additional information when populating the fields. The templates <NUM> can be standardized for different types of actions, such as playing media files through the vehicle's head unit, responding to messages, and performing functions within the vehicle <NUM>. The action data structure can initially be generated by a direct action API <NUM> executed by a remote data processing system <NUM>. The remote data processing system <NUM> can transmit the action data structure to the data processing system <NUM> of the vehicle <NUM>, which can add fields and attributes to the action data structure.

The direct action API <NUM> can obtain response data <NUM> (or parameters <NUM> or policies <NUM>) from the data repository <NUM>, as well as data received with end user consent from the client devices <NUM> to determine location, time, user accounts, and logistical or other information in order to reserve a car from the car share service. The response data <NUM> (or parameters <NUM> or policies <NUM>) can be included in the action data structure. When the content included in the action data structure includes end user data that is used for authentication, the data can be passed through a hashing function before being stored in the data repository <NUM>.

The data processing system <NUM> can include, interface, or otherwise communicate with the configuration tracker component <NUM>. The configuration tracker component <NUM> can be hosted on and executed from the vehicle <NUM>. The configuration tracker component <NUM> can identify a functionality of each digital assistant application. The configuration tracker component <NUM> can identify each digital assistant application <NUM> running on or otherwise accessible via a client device <NUM>. The configuration tracker component <NUM> can identify each digital assistant application <NUM> running on the vehicle <NUM>. The client device <NUM> may be communicatively coupled (e.g., via network <NUM> or via a short-range communication link) to the data processing system <NUM> or the vehicle <NUM>. With the communication link established, each digital assistant application <NUM> accessible from the vehicle <NUM>, such as the digital assistant application <NUM> running on the vehicle <NUM> and the digital assistant application <NUM> accessible via the client device <NUM>, may be prevented from executing the action specified by the command until invoked by the application selector component <NUM>. In addition, the transducer <NUM> at the client device <NUM> may also be disabled until the digital assistant application <NUM> is invoked by the application selector component <NUM>. Details of the functionality of the application selector component <NUM> are explained below.

The functionalities of the digital assistant application <NUM> hosted on the vehicle <NUM> (sometimes referred to as a "primary digital assistant application") may differ from the functionalities of the digital assistant application <NUM> accessible via the client device <NUM> (sometimes referred to as a "secondary digital assistant application"). For example, the digital assistant application <NUM> installed on the vehicle <NUM> may activate various function on the vehicle <NUM>, such as opening a door, turning on a windshield, or changing temperature controls. On the other hand, the digital assistant application <NUM> accessible through the client device <NUM> may carry out other functions, such as making a telephone call or accessing a calendar. For functionalities of the digital assistant application <NUM> accessible via the client device <NUM> that do not differ from the functionalities of the digital assistant application <NUM> hosted on the vehicle <NUM>, one of the digital assistant applications <NUM> may be able to execute the functionality faster or more efficiently.

The configuration tracker component <NUM> can identify the functionalities of each digital assistant application <NUM> accessible via the client device <NUM>. There may be multiple digital assistant applications <NUM> accessible via multiple client devices <NUM>. The configuration tracker component <NUM> can identify an application profile of the digital assistant application <NUM>. The application profile may include metadata for the digital assistant application <NUM> and may indicate a name and a type for the application. The configuration tracker component <NUM> can identify a version number of the digital assistant application <NUM>. The configuration tracker component <NUM> can identify a connectivity of the client device <NUM> to the vehicle <NUM> or to an external network (e.g., the network <NUM>) for accessing the remote data processing system <NUM>. The configuration tracker component <NUM> can identify one or more routines or processes (e.g., application programming interface (API) calls) supported or used by the digital assistant application <NUM>. The configuration tracker component <NUM> can identify one or more peripheral devices <NUM> interfacing with the client device <NUM>. The one or more peripheral devices <NUM> may include input/output devices, such as a monitor, keyboard, and card-reader, among others, connected to the client device <NUM>. The configuration tracker component <NUM> can identify consumption of computing resources at the client device <NUM> (e.g., CPU time, remaining battery life, or memory use, for example) through which the digital assistant application <NUM> is accessed.

The configuration tracker component <NUM> can identify the functionalities of the digital assistant application <NUM> hosted on the vehicle <NUM>. The configuration tracker component <NUM> can identify an application profile of the digital assistant application <NUM>. The application profile may include metadata for the digital assistant application <NUM> and may indicate a name and a type for the application. The configuration tracker component <NUM> can identify a version number of the digital assistant application <NUM>. The version number of the digital assistant application <NUM> hosted on the vehicle <NUM> may differ from the version number of the digital assistant application <NUM> accessible via the client device <NUM>. The configuration tracker component <NUM> can identify a connectivity of the vehicle <NUM> to the client device <NUM> or to an external network (e.g., the network <NUM>) for accessing the remote data processing system <NUM>. The configuration tracker component <NUM> can identify one or more routines or processes (e.g., application programming interface (API) calls) supported or used by the digital assistant application <NUM>. The configuration tracker component <NUM> can identify one or more peripheral devices <NUM> interfacing with the vehicle <NUM>. The one or more peripheral devices <NUM> may include various devices in the vehicle <NUM>, such as a temperature control, passenger door opener, trunk opener, garage door opener, a steering wheel, and multimedia device, among others accessible by the digital assistant application <NUM> running from the vehicle <NUM>. The configuration tracker component <NUM> can identify consumption of computing resources (e.g., CPU time, remaining battery life, memory use, etc.) at the vehicle <NUM> hosting the digital assistant application <NUM>.

The data processing system <NUM> can include, interface, or otherwise communicate with the application selector component <NUM>. The application selector component <NUM> can be hosted on and executed from the vehicle <NUM>. The application selector component <NUM> can determine whether the functionality of the digital assistant application <NUM> accessible via the client device <NUM> or the functionality of the digital assistant application <NUM> hosted on the vehicle <NUM> can support the command corresponding to the request. The application selector component <NUM> can identify an action specified by the command as processed by the NLP component <NUM>. The application selector component <NUM> can parse the action data structure generated by the direct action API <NUM> to identify the specified action.

To determine that the specified action is (or is not) supported by the digital assistant application <NUM>, the application selector component <NUM> can compare the operational capabilities or functionality of the digital assistant application <NUM> accessible via the client device <NUM> to the action specified by the command. The application selector component <NUM> can also compare the operational capabilities or functionality of the digital assistant application <NUM> hosted on the vehicle <NUM>. The application selector component <NUM> can determine that the functionality of the digital assistant application <NUM> can support the action specified by the command based on combination of factors. The application selector component <NUM> can determine that the action corresponding to one or more routines or processes of the digital assistant application <NUM> corresponding to the action specified by the command. Responsive to the determination, the application selector component <NUM> can determine that the functionality of the digital assistant application <NUM> supports the action. The application selector component <NUM> can determine that the action specified by the command is performed via an external network. Responsive to the determination, the application selector component <NUM> can determine that the functionality of the digital assistant application <NUM> supports the action when there is access to the external network (e.g., network <NUM>). The application selector component <NUM> can determine the action specified by the command is performed via a particular peripheral device <NUM>. Responsive to the determination, the application selector component <NUM> can identify whether the particular peripheral device <NUM> is interfacing with the client device <NUM>, and the particular peripheral devices <NUM> is interfacing with the vehicle <NUM>. The application selector component <NUM> can determine the action specified by the command is performed by a particular type of application (e.g., the digital assistant application <NUM> at the vehicle <NUM> or the digital assistant application <NUM> on the client device <NUM>). Responsive to the determination, the application selector component <NUM> can determine whether the application profile of the digital assistant application <NUM> is a type of application that can carry out the specification action.

Based on the determination of the functionality of the respective digital applications <NUM>, the application selector component <NUM> can select one of the digital assistant application <NUM> accessible via the client device <NUM> or the functionality of the digital assistant application <NUM> hosted on the vehicle <NUM>. For example, the application selector component <NUM> can set the digital assistant application <NUM> hosted together on the vehicle <NUM> as the default digital assistant application and set the digital assistant application <NUM> accessible via the client device <NUM> as the secondary digital assistant application. In the absence of the secondary digital assistant application, the application selector component <NUM> can select the primary digital assistant application to carry out the command. The application selector component <NUM> can determine the functionality of the digital assistant application <NUM> accessible via the client device <NUM> can support the action specified by the command. Responsive to the determination, the application selector component <NUM> can select the digital assistant application <NUM> accessible via the client device <NUM>. The application selector component <NUM> can determine that there are multiple digital assistant applications <NUM> accessible via the one or more client devices <NUM>. Responsive to the determination that there are multiple digital assistant applications <NUM>, the application selector component <NUM> can select the one digital assistant application <NUM> accessible via the client device <NUM> whose functionality can support the action specified by the command. Conversely, the application selector component <NUM> can determine the functionality of the digital assistant application <NUM> hosted on the vehicle <NUM> but not that of the digital assistant application <NUM> accessible via the client device <NUM> can support the action specified by the command. Responsive to the determination, the application selector component <NUM> can select the digital assistant application <NUM> hosted on the vehicle <NUM>.

When both functionalities of the digital assistant applications <NUM> accessible via the client device <NUM> and hosted on the vehicle <NUM> are determined support the action specified by the command, the digital assistant application <NUM> can select based on which digital assistant application <NUM> can more quickly carry out the action. The application selector component <NUM> can compare the version number of the digital assistant application <NUM> hosted on the vehicle <NUM> with the version number of the digital assistant application <NUM> accessible via the client device <NUM>. The application selector component <NUM> can select the digital assistant application <NUM> with the higher version number, as the corresponding the digital assistant application <NUM> may be more up-to-date. The application selector component <NUM> can compare the computing resources consumption at the vehicle <NUM> with the computing resources consumption at the client device <NUM>. The application selector component <NUM> can select the client device <NUM> or the vehicle <NUM> based on which has the lower computing resource consumption, as the action specified by the command is likely to be executed faster when computing resource consumption is lower.

The application selector component <NUM> can also select the digital assistant application <NUM> accessible via the client device <NUM> or the functionality of the digital assistant application <NUM> hosted on the vehicle <NUM> based on other factors. These other factors can override the selection based on the determination on which digital assistant application <NUM> can support the action specified by the command. The application selector component <NUM> can also select the digital assistant application <NUM> accessible via the client device <NUM> or the digital assistant application <NUM> hosted on the vehicle <NUM> based on an input via a vehicle input interface. The vehicle input interface may be one of the peripheral devices <NUM> connected to the vehicle <NUM>, such as a button on a steering wheel or a touchscreen on a center stack in the vehicle <NUM>. For example, the selection may be based on a length of a button press on the button on the steering wheel of the vehicle <NUM>. The application selector component <NUM> can determine that the length of the button press is greater than or equal to a predetermined threshold (e.g., <NUM> seconds). In response to the determination, the application selector component <NUM> can select the digital assistant application <NUM> hosted on the vehicle <NUM>. The application selector component <NUM> can determine that the length of the button press is shorter than the predetermined threshold. In response to this determination, the application selector component <NUM> can select the digital assistant application <NUM> accessible via the client device <NUM>. A position of the interaction on the touchscreen may also be used to select the digital assistant application <NUM> accessible via the client device <NUM> or the digital assistant application <NUM> hosted on the vehicle <NUM>.

The application selector component <NUM> can also select the digital assistant application <NUM> accessible via the client device <NUM> or the functionality of the digital assistant application <NUM> hosted on the vehicle <NUM> based on the trigger keyword recognized in the audio input signal. The application selector component <NUM> can compare the trigger keyword to a set of keywords associated with corresponding digital assistant applications <NUM>. Based on the comparison of the trigger word detected from the audio input signal with the set of associated keywords, the application selector component <NUM> can select the corresponding digital assistant application <NUM>. For example, there may be two client devices <NUM> in the vehicle <NUM> with respective digital assistant applications <NUM> associated with different trigger keywords. The trigger word "okay" may be associated with the digital assistant application <NUM> on a first client device <NUM> and the trigger word "hey" may be associated with the digital assistant application <NUM> on a second client device <NUM>. Using the set of keywords, when the audio input signal including "hey" is received, the application selector component <NUM> can select the digital assistant application <NUM> on the second client device <NUM>.

The application selector component <NUM> can also select the digital assistant application <NUM> accessible via the client device <NUM> or the functionality of the digital assistant application <NUM> hosted on the vehicle <NUM> based on whether the command is related to a vehicular function. As explained above, the NLP component <NUM> can determine whether the command is related to a vehicular or non-vehicular function using a semantic knowledge graph. Using the determination, the application selector component <NUM> can select the corresponding digital assistant application <NUM>. The application selector component <NUM> can determine that the command is related to a vehicular function. In response to the determination, the application selector component <NUM> can select the digital assistant application <NUM> hosted on the vehicle <NUM>. The application selector component <NUM> can determine that the command is determined to relate to a non-vehicular function. In response to the determination, the application selector component <NUM> can select the digital assistant application <NUM> accessible via the client device <NUM>. The application selector component <NUM> can determine that there are multiple digital assistant applications <NUM> accessible via the one or more client devices <NUM>. In response to the determination, the application selector component <NUM> can select the one digital assistant application <NUM> accessible via the client device <NUM> whose functionality can support the action specified by the command identified to be related to a non-vehicular function.

The application selector component <NUM> can determine that there are multiple digital assistant applications <NUM> accessible whose functionalities can support the action specified by the command. In response to the determination, the application selector component <NUM> can select one of the digital assistant applications <NUM> based on a quality score. These multiple digital assistant applications <NUM> can include the digital assistant applications <NUM> accessible via the one or more client devices <NUM> and the digital assistant application <NUM> hosted on the vehicle <NUM>. The data processing system <NUM> can include, interface, or otherwise communicate with the performance evaluator component <NUM>. The performance evaluator component <NUM> can determine the quality score for each digital assistant application <NUM> accessible via one or more client devices <NUM> based on any number of factors. The performance evaluator component <NUM> can determine the quality score based on a feedback indicator. The feedback indicator may be a numerical value representing a user's response to a prompt inquiring about quality of interaction with the digital assistant application <NUM>. To obtain the feedback indicator, the digital assistant application <NUM> accessible via the client device <NUM> can display the prompt inquiring about quality of interaction on the client device <NUM> displaying the digital assistant application <NUM>. The digital assistant application <NUM> hosted on the vehicle <NUM> or the performance evaluator component <NUM> can also display the prompt inquiring about quality of interaction on the vehicle <NUM> displaying the digital assistant application <NUM>. The prompt may include one or more user interface elements (e.g., buttons, radio buttons, checkboxes, etc.), each indicating the quality of interaction. Once an interaction event is detected on one of the user interface elements on the prompt, the performance evaluator component <NUM> can identify the feedback indicator corresponding to the quality of interaction. The performance evaluator component <NUM> can determine the quality score using the feedback indicator.

The performance evaluator component <NUM> can also determine the quality score for each digital assistant application <NUM> based on a rating score from an application distribution platform. The application distribution platform can run on an external server accessible via the network <NUM>. The external server can host applications, such as the digital assistant applications <NUM>, for download by the client device <NUM> or the vehicle <NUM>. The external server for the application distribution platform may store the rating scores for all the digital assistant applications <NUM>. From the application distribution platform running on the external server, the performance evaluator component <NUM> can identify the rating score for each digital assistant application <NUM> identified as accessible via the client device <NUM> or hosted on the vehicle <NUM>. The performance evaluator component <NUM> can send a request for the rating score to the external server, when there is connectivity with the network <NUM>. Using the rating score for each digital assistant application <NUM>, the performance evaluator component <NUM> can determine the quality score for the corresponding digital assistant application <NUM>.

The performance evaluator component <NUM> can also run an automated test evaluation to determine the quality score for the corresponding digital assistant application <NUM>. The automated test evaluation may be executed using an application or the performance evaluator component <NUM> interfacing with the digital assistant application <NUM>. The automated test evaluation may include a set of audio input signal stimuli and a corresponding expected action data structure for the audio input signal stimuli. The automated test evaluation may be run simultaneous to the execution of the digital assistant application <NUM>. While executing, the automated test evaluation can feed each audio input signal stimulus into the digital assistant application <NUM>. The automated test evaluation can identify an action data structure generated by the digital assistant application <NUM>. The automated test evaluation can compare the action data structure generated by the digital assistant application <NUM> with the expected action data structure. From the comparison, the automated test evaluation can identify a number of matches and a number of mismatches between the two action data structures. Based on the number of matches and the number of mismatches, the performance evaluator component <NUM> can determine the quality score for the digital assistant application <NUM>.

Once the quality score for each digital assistant application <NUM> is determined, the application selector component <NUM> can select the digital assistant application <NUM> with the highest quality score having the functionality to support the action specified by the command. The quality score may be based on any combination of the feedback indicator, the rating score, and the automated test evaluation. The application selector component <NUM> can identify a subset of digital assistant applications <NUM> whose functionality can support the action specified by the command. From the subset of digital assistant applications <NUM>, the application selector component <NUM> can identify the digital assistant application <NUM> with the highest quality score as determined by the performance evaluator component <NUM>.

The application selector component <NUM> can invoke the digital assistant application <NUM> selected based on the determination that the functionality of the digital assistant application <NUM> can support the action specified by the command. To invoke the selected digital assistant application <NUM>, the application selector component <NUM> can identify the action data structure corresponding to the command. The application selector component <NUM> can call the direct action API <NUM> of the selected digital assistant application <NUM> to execute and process the action data structure corresponding to the command in the audio input signal. The application selector component <NUM> can determine that the digital assistant application <NUM> selected is accessible via the client device <NUM>. Responsive to the determination, the application selector component <NUM> can assign control of the input interfaces (e.g., the sensor <NUM>, the interface <NUM>, the transducer <NUM>, and other peripheral devices <NUM>) of the vehicle <NUM> to the client device <NUM>. For example, the command may be to play an audio file on the client device <NUM>, and the digital assistant application <NUM> selected is accessible via the client device <NUM>. To carry out the command, the application selector component <NUM> can assign control of the speakers <NUM> from the vehicle <NUM> to the digital assistant application <NUM> accessible via the client device <NUM> (sometimes referred to in such instances as a "brought-in digital assistant application"), such that the audio is played from the speakers <NUM> of the vehicle <NUM>.

In invoking the selected digital assistant application <NUM>, the application selector component <NUM> can use application data from a non-selected digital assistant application <NUM>. The application selector component <NUM> can access the data repository <NUM> of the non-selected digital assistant application <NUM> to retrieve the parameters <NUM>, the policies <NUM>, the response data <NUM>, the templates <NUM>, and other information (e.g., phone numbers, calendar events, etc.). For example, when the action specified by the command is to call a particular phone number, the application selector component <NUM> can retrieve the phone number from the non-selected digital assistant application <NUM> (or another application) running on a first client device <NUM>. The application selector component <NUM> can store the retrieved application data in the local data repository <NUM>. Prior to invocation, the application selector component <NUM> can modify the action data structure to include the application data retrieved from the non-selected digital assistant application <NUM>. The application selector component <NUM> can then call the direct action API <NUM> of the selected digital assistant application <NUM> using the direct action structure modified by the application data retrieved from the non-selected digital assistant application <NUM>. Continuing with the previous example, the application selector component <NUM> can call the selected digital assistant application <NUM> accessible via a second client device <NUM> to make the phone call using the phone number retrieved from the first client device <NUM>.

<FIG> illustrates a top view of the vehicle <NUM> and illustrates the interior cabin of the vehicle <NUM>. In the vehicular environment <NUM>, the interior cabin of the vehicle <NUM> can include four seats, a plurality of speakers <NUM>, and two client devices <NUM>(<NUM>) and <NUM>(<NUM>). The two client devices <NUM>(<NUM>) and <NUM>(<NUM>) may have been carried in by one or more passengers of the vehicle <NUM>. The vehicle <NUM> can also include a head unit <NUM>. The head unit <NUM> can execute one or more of the components described in relation to the data processing system <NUM>, such as a digital assistant application <NUM>(<NUM>), the configuration tracker component <NUM>, the application selector component <NUM>, and the performance valuator component <NUM>. The client device <NUM>(<NUM>) can execute or make accessible one or more of the components described in relation to the data processing system <NUM>, such a digital assistant application <NUM>(<NUM>). The client device <NUM>(<NUM>) also can execute or make accessible one or more of the components described in relation to the data processing system <NUM>, such a digital assistant application <NUM>(<NUM>). The client devices <NUM>(<NUM>) and <NUM>(<NUM>) may have established communication links (e.g., via pairing) with the head unit <NUM>.

Referring to <FIG> among others, a transducer <NUM> within the vehicle <NUM> may detect an audio input signal uttered by a passenger within the vehicle <NUM>. Upon detection of the audio input signal, the NLP component <NUM> operating from the head unit <NUM> of the vehicle <NUM> can parse the audio input signal to identify a command. Using the command parsed from the audio input signal, the direct action API <NUM> running in the head unit <NUM> of the vehicle <NUM> can generate an action data structure. The action data structure can include data or instructions for the execution of a specified action to satisfy the command.

The configuration tracker component <NUM> operating from the head unit <NUM> of the vehicle <NUM> can identify each digital assistant application <NUM>(<NUM>)-(<NUM>) running on or in communication with the head unit <NUM>. The digital assistant applications <NUM>(<NUM>)-(<NUM>) may have differing functionalities. For example, the digital assistant application <NUM>(<NUM>) running in the head unit <NUM> may support vehicle-related commands (e.g., "pop the trunk") while the digital assistant application <NUM>(<NUM>) of the client device <NUM>(<NUM>) and the digital assistant application <NUM>(<NUM>) of the client device <NUM>(<NUM>) may support non-vehicle related commands (e.g., "play a song"). The configuration tracker component <NUM> can identify one or more functionalities of each of the identified digital assistant applications <NUM>(<NUM>)-(<NUM>), such as the application profile, version number, connectivity to the network <NUM> or the vehicle <NUM>, peripheral devices <NUM>, consumption of computing resources, and supported routines, among others.

Having identified the functionalities of the digital assistant applications <NUM>(<NUM>)-(<NUM>), the application selector component <NUM> can determine which digital assistant applications <NUM>(<NUM>)-(<NUM>) can support the action specified by the command. The application selector component <NUM> can select the digital assistant application <NUM>(<NUM>)-(<NUM>) based on which can support the action specified by the command. When two or more of the digital assistant application <NUM>(<NUM>)-(<NUM>) can support the action specified by the command, the application selector component <NUM> can select one of the digital assistant applications <NUM>(<NUM>)-(<NUM>) using other factors, such as the version number and consumption of computing resources, among others. Once determined, the application selector component <NUM> can invoke the selected digital assistant application <NUM>(<NUM>)-(<NUM>) by calling the respective direct action API <NUM> to execute in accordance with the generated action data structure. When application data from one digital assistant application <NUM>(<NUM>) is to be used by the selected digital assistant application <NUM>(<NUM>), the application data may be transferred from the selected digital assistant application <NUM>(<NUM>) to the non-selected digital assistant application <NUM>(<NUM>). The application selector component <NUM> can modify the data structure using the transferred data structure. The direct action API <NUM> may be called to carry out the action specified by the command.

<FIG> illustrates an example method <NUM> to selectively invoke applications for execution. The method <NUM> may be implemented or executed by the system <NUM> described above in conjunction with <FIG> and <FIG> or system <NUM> detailed below in conjunction with <FIG>. The method <NUM> can include receiving an audio signal (BLOCK <NUM>). The method <NUM> can include parsing the audio signal to identify a command (BLOCK <NUM>). The method <NUM> can include generating an action data structure (BLOCK <NUM>). The method <NUM> can include identifying a functionality of a digital assistant application (BLOCK <NUM>). The method <NUM> can include determining that the functionality supports the command (BLOCK <NUM>). The method <NUM> can include selecting the digital assistant application (BLOCK <NUM>). The method <NUM> can include invoking the selected digital assistant application (BLOCK <NUM>).

The method <NUM> can include receiving an audio signal (BLOCK <NUM>). A data processing system can receive the input audio signal. For example, an NLP component, executed by the data processing system, can receive the input audio signal. The data processing system (and the NLP component) can be a component of or otherwise executed by a client device, a vehicle, or be a standalone device. The microphone at the client device or the vehicle can detect the input audio signal and then the respective client device or vehicle can transmit the input audio signal to the data processing system. For example, an application executed on the client device can detect a user speaking "Ok, open the sunroof. " The detected utterance can be encoded into an input audio signal and transmitted to the NLP component of the data processing system or vehicle.

The method <NUM> can include parsing the audio signal to identify a command (BLOCK <NUM>). The NLP component can parse the input audio signal to identify a request in the input audio signal. The NLP component can identify a vehicle associated with the request. The NLP component can identify a fulfillment interface associated with the request and the vehicle. The fulfillment interface can be the interface of one of the client device or the vehicle that will execute the action data structure to fulfill the request of the input audio signal.

In the above example, the request can be to open the sunroof. In this example, the fulfillment interface can be the interface of the vehicle that includes the sunroof. In some implementations, the vehicle can be explicitly stated in the input audio signal. For example, a user may assign nicknames to his vehicle (e.g., the user may name his red Toyota Camry the "red car"). When explicitly stated, the input audio signal could be "OK, open the sunroof of my red car. " In some implementations, the data processing system can determine which vehicles associated with the user's account can fulfill the action. For example, the user's red car may include a sun roof and the user's blue car may not. After receiving the input audio signal "Ok, open the sunroof. " The data processing system may automatically select the user's red car. In some implementations, the data processing system may ask the user for confirmation of the vehicle.

The method <NUM> can include generating an action data structure (BLOCK <NUM>). The direct action API can generate the data structure that can be transmitted and processed by the client device or the vehicle to fulfil the request of the input audio signal. For example, continuing the above example, the direct action API can generate a first action data structure for opening the sunroof of the user's car. The direction action API can generate the action data structure using a template retrieved from the data processing system's data repository. The action data structure can include fields used to fulfill the request. For example, for the request to open the sunroof, the action data structure can include a field (or attribute) for the vehicle ID to which the action data structure should be transmitted when the action data structure is approved.

The method <NUM> can include identifying a functionality of a digital assistant application (BLOCK <NUM>). For example, a configuration tracker component can identify each digital assistant application in the vehicle. The digital assistant applications can be hosted on the vehicle itself or may be accessible via the client device. All of the digital assistant applications in the vehicle may be communicatively linked via short-range communications. For each digital assistant application identified, the configuration tracker component can identify the functionality of the digital assistant application, such as the application profile, version number, connectivity to the network <NUM> or the vehicle <NUM>, peripheral devices <NUM>, consumption of computing resources, and supported routines, among others.

The method <NUM> can include determining that the functionality supports the command (BLOCK <NUM>). For instance, an application selector component can determine whether the functionality identified for the corresponding digital assistant application supports the action specified by the command. Some commands may be related to a vehicle function (e.g., "open the sunroof"), while other commands may be related to a non-vehicle function (e.g., "call John"). The determination whether the command relates to a vehicle function or a non-vehicle function may be performed by the NLP component. The application selector component can compare the functionality of each digital assistant application with the action specified by the command.

The method <NUM> can include selecting the digital assistant application (BLOCK <NUM>). For example, the application selector component can select the digital assistant application determined to be able to support the action specified by the command. When the functionality of the digital assistant application hosted on the vehicle can support the action specified by the command, the application selector component can select the digital assistant application hosted on the vehicle, in lieu of the application accessible via the client device. When the functionality of the digital assistant application accessible via the client device can support the action specified by the command, the application selector component can select the digital assistant application hosted on the vehicle, in lieu of the application hosted on the vehicle. When both digital assistant applications can support the action specified by the command, the application selector component can use other factors (e.g., latency or computing resources consumed) to select one of the multiple digital assistant applications.

The method <NUM> can include invoking the selected digital assistant application (BLOCK <NUM>). For instance, the application selector component can invoke the digital assistant application selected based on the determination that the application can support the action supported by the command. The application selector component can call the direct action API of the selected digital assistant application using the action data structure for the command. The action data structure may be modified by the application selector component using application data from the non-selected digital assistant application.

The computer system or computing device <NUM> can include or be used to implement the system <NUM> or its components such as the data processing system <NUM>. The computing system <NUM> includes a bus <NUM> or other communication component for communicating information and a processor <NUM> or processing circuit coupled to the bus <NUM> for processing information. The computing system <NUM> can also include one or more processors <NUM> or processing circuits coupled to the bus for processing information. The computing system <NUM> also includes main memory <NUM>, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus <NUM> for storing information and instructions to be executed by the processor <NUM>. The main memory <NUM> can be or include the data repository <NUM>. The main memory <NUM> can also be used for storing position information, temporary variables, or other intermediate information during execution of instructions by the processor <NUM>. The computing system <NUM> may further include a read-only memory (ROM) <NUM> or other static storage device coupled to the bus <NUM> for storing static information and instructions for the processor <NUM>. A storage device <NUM>, such as a solid state device, magnetic disk or optical disk, can be coupled to the bus <NUM> to persistently store information and instructions. The storage device <NUM> can include or be part of the data repository <NUM>.

The computing system <NUM> may be coupled via the bus <NUM> to a display <NUM>, such as a liquid crystal display or active matrix display, for displaying information to a user. An input device <NUM>, such as a keyboard including alphanumeric and other keys, may be coupled to the bus <NUM> for communicating information and command selections to the processor <NUM>. The input device <NUM> can include a touch screen display <NUM>. The input device <NUM> can also include a cursor control, such as a mouse, a trackball, or cursor direction keys, for communicating direction information and command selections to the processor <NUM> and for controlling cursor movement on the display <NUM>. The display <NUM> can be part of the data processing system <NUM>, the client computing devices <NUM>, or other component of <FIG>, for example.

Although an example computing system has been described in <FIG>, the subject matter including the operations described in this specification can be implemented in other types of digital electronic circuitry or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

For situations in which the systems discussed herein collect personal information about users, or may make use of personal information, the users may be provided with an opportunity to control whether programs or features that may collect personal information (e.g., information about a user's social network, social actions, or activities; a user's preferences; or a user's location), or to control whether or how to receive content from a content server or other data processing system that may be more relevant to the user. In addition, certain data may be anonymized in one or more ways before it is stored or used, so that personally identifiable information is removed when generating parameters. For example, a user's identity may be anonymized so that no personally identifiable information can be determined for the user, or a user's geographic location may be generalized where location information is obtained (such as to a city, postal code, or state level), so that a particular location of a user cannot be determined. Thus, the user may have control over how information is collected about him or her and used by the content server.

The subject matter and the operations described in this specification can be implemented in digital electronic circuitry or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, e.g., one or more circuits of computer program instructions, encoded on one or more computer storage media for execution by, or to control the operation of, data processing apparatuses. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. A computer storage medium can be, or be included in, a computer-readable storage device, a computer-readable storage substrate, a random or serial-access memory array or device, or a combination of one or more of them. While a computer storage medium is not a propagated signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal. The computer storage medium can also be, or be included in, one or more separate components or media (e.g., multiple CDs, disks, or other storage devices).

The terms "data processing system," "computing device," "component," or "data processing apparatus" encompass various apparatuses, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or multiple ones, or combinations of the foregoing. The apparatus can include special-purpose logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). The components of system <NUM> can include or share one or more data processing apparatuses, systems, computing devices, or processors.

A computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program can correspond to a file in a file system. A computer program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code).

The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs (e.g., components of the data processing system <NUM>) to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatuses can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field-programmable gate array) or an ASIC (application-specific integrated circuit). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD-ROM and DVD-ROM disks.

The subject matter described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a web browser through which a user can interact with an implementation of the subject matter described in this specification, or a combination of one or more such back end, middleware, or front end components.

The computing system such as system <NUM> or system <NUM> can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network (e.g., the network <NUM>). In some implementations, a server transmits data (e.g., data packets representing a content item) to a client device (e.g., for purposes of displaying data to and receiving user input from a user interacting with the client device). Data generated at the client device (e.g., a result of the user interaction) can be received from the client device at the server (e.g., received by the data processing system <NUM> from the client devices <NUM> or the vehicle <NUM>).

The separation of various system components does not require separation in all implementations, and the described program components can be included in a single hardware or software product. For example, the NLP component <NUM> and the direct action API <NUM> can be a single component, app, or program, or a logic device having one or more processing circuits, or part of one or more servers of the data processing system <NUM>.

Having now described some illustrative implementations, it is apparent that the foregoing is illustrative and not limiting, having been presented by way of example. In particular, although many of the examples presented herein involve specific combinations of method acts or system elements, those acts and those elements may be combined in other ways to accomplish the same objectives. Acts, elements, and features discussed in connection with one implementation are not intended to be excluded from a similar role in other implementations.

The use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized in that," and variations thereof herein, is meant to encompass the items listed thereafter, equivalents thereof, and additional items, as well as alternate implementations consisting of the items listed thereafter exclusively.

Any references to implementations, elements, or acts of the systems and methods herein referred to in the singular may also embrace implementations including a plurality of these elements, and any references in plural to any implementation, element, or act herein may also embrace implementations including only a single element. References to any act or element being based on any information, act, or element may include implementations where the act or element is based at least in part on any information, act, or element.

Any implementation disclosed herein may be combined with any other implementation or embodiment, and references to "an implementation," "some implementations," "one implementation," or the like are not necessarily mutually exclusive and are intended to indicate that a particular feature, structure, or characteristic described in connection with the implementation may be included in at least one implementation or embodiment. Such terms as used herein are not necessarily all referring to the same implementation. Any implementation may be combined with any other implementation, inclusively or exclusively, in any manner consistent with the aspects and implementations disclosed herein.

References to "or" may be construed as inclusive so that any terms described using "or" may indicate any of a single, more than one, and all of the described terms. A reference to "at least one of 'A' and 'B'" can include only 'A', only 'B', as well as both 'A' and 'B'. Such references used in conjunction with "comprising" or other open terminology can include additional items.

Where technical features in the drawings, detailed description, or any claim are followed by reference signs, the reference signs have been included to increase the intelligibility of the drawings, detailed description, and claims.

Claim 1:
A method implemented by one or more processors of a vehicle computing device of a vehicle (<NUM>), the method comprising:
receiving, via one or more microphones of the vehicle computing device, an input audio signal that captures a request provided by a user of the vehicle;
identifying, based on processing the input audio signal, an action to be performed responsive to the request provided by the user;
determining, based on comparing the request or data derived from the request with a semantic knowledge graph, a type of the action to be performed responsive to the request provided by the user;
identifying at least a first digital assistant application (<NUM>(<NUM>)) and a second digital assistant application (<NUM>(<NUM>)) that is accessible by the vehicle computing device, wherein the first digital assistant application is integral to the vehicle computing device and the second digital assistant application is integral to a client device (<NUM>(<NUM>)) of the user of the vehicle, and wherein the second digital assistant application is accessible by the vehicle computing device via a communication link between the client device and the vehicle computing device;
in response to determining that the type of the action to be performed responsive to the request provided by the user corresponds to a first type of action, wherein the first type of action includes controlling one or more functionalities of the vehicle:
selecting, from among at least the first digital assistant application and the second digital assistant application, the first digital assistant application to perform the action; and
causing the first digital assistant application to perform the action to satisfy the request; and
in response to determining that the type of the action to be performed responsive to the request provided by the user corresponds to a second type of action:
selecting, from among at least the first digital assistant application and the second digital assistant application, the second digital assistant application to perform the action; and
causing the second digital assistant application to perform the action to satisfy the request.