Patent Description:
Vehicles may include an in-vehicle computing system, such as a head unit for an infotainment system, which may provide multimedia and control functions. For example, an in-vehicle computing system may provide navigational, multimedia playback, telephonic, social media interactive, mobile device charging, and/or other functions and receive user input to control elements of the vehicle and in-vehicle computing system. In order to provide such functionality, in-vehicle computing systems include complex and costly processing and memory resources that enable the execution of an operating system and various applications. Further, given such extended functionality, the control of such in-vehicle computing system may be overly complex for a user whose attention should be focused on safely operating the vehicle.

<CIT> describes a vehicle driving assist interoperating with a plurality of mobile terminals and a vehicle equipped with the same, the vehicle driving assist comprising: a display unit; a microphone configured to receive voice uttered by a user in a vehicle; a wireless communication unit, disposed inside the vehicle, configured to wirelessly communicate with a plurality of mobile terminals; a position sensing unit for sensing the positions of the plurality of mobile terminals within the vehicle; and a controller for transmitting voice data to any one of the plurality of mobile terminals through the wireless communication unit so that the voice data corresponding to the voice inputted from the microphone can be processed, wherein the plurality of mobile terminals process the voice data through different voice recognition algorithms. <CIT>, <CIT>, <CIT>, and <CIT> relate to background information.

The dependent claims recite selected optional features.

In order to improve the control of an in-vehicle computing system, the in-vehicle computing system may accept voice commands as user input for controlling elements of the vehicle and the in-vehicle computing system. However, as the basic functionality of an in-vehicle computing system may already utilize the majority of processing capability and memory resources of the in-vehicle computing system, the in-vehicle computing system may not be able to support a complex voice control system. To overcome the limitations of an in-vehicle computing system, the processing capability and memory resources of a mobile device positioned with the vehicle may be leveraged to provide sophisticated voice control of vehicle components and the in-vehicle computing system.

Embodiments are disclosed for in-vehicle systems and methods of controlling vehicle systems with a mobile device in communication with the in-vehicle system. An example system for controlling a vehicle includes a microphone, at least one vehicle component, and an interface system communicatively coupled to the microphone and the at least one vehicle component, the interface system configured to detect, via the microphone, an utterance of a vocal command by an occupant of the vehicle, transmit the utterance to at least one of a virtual personal assistant (VPA) of the vehicle and a VPA of a mobile device positioned within the vehicle, receive control signals from at least one of the VPA of the vehicle and the VPA of the mobile device, the control signals generated according to the vocal command, and transmit the control signals to the at least one vehicle component for controlling the at least one vehicle component.

An example method of controlling vehicle systems of a vehicle with a mobile device includes receiving, at an in-vehicle interface system via a microphone, an utterance by an occupant of the vehicle, transmitting the utterance from the in-vehicle interface system to the mobile device, wherein the mobile device processes the utterance to determine a command, receiving, at the in-vehicle interface system, a control signal from the mobile device, the control signal generated by the mobile device according to the command, and transmitting the control signal from the in-vehicle interface system to at least one vehicle system of the vehicle for controlling the at least one vehicle system.

An example system for controlling a vehicle system of a vehicle includes a mobile device positioned with the vehicle, and an interface system integrated into the vehicle and communicatively coupled to the vehicle system and the mobile device, the interface system configured to receive, via a microphone, an utterance of a command by an occupant of the vehicle, transmit the utterance from the interface system to the mobile device, receive, at the interface system, a control signal from the mobile device, and transmit the control signal from the interface system to the vehicle system for controlling the vehicle system, wherein the mobile device is configured to receive the utterance from the interface system, convert the utterance to a natural language text, identify the command in the natural language text, generate the control signal according to the command, and transmit the control signal to the interface system.

The disclosure may be better understood from reading the following description of non-limiting embodiments, with reference to the attached drawings, wherein below:.

A virtual personal assistant (VPA) may include an automatic speech recognition system that detects a vocal command in a vocal utterance and performs an action responsive to the vocal command. For example, a VPA may detect a vocal command (e.g., "Turn up the volume," "Roll down the windows," "Navigate to the nearest gas station," "Play classical music," and so on), and then perform the command, if possible. While some functions, such as adjusting audio settings or adjusting the windows, may easily be carried out by an in-vehicle computing system, other functions, such as navigating to a particular location or playing a specific song by a specific musical artist on the stereo, may not be easily carried out by the in-vehicle computing system. As mentioned above, a mobile device may include available processing and/or storage resources that equal or exceed those resources typically used to provide the functionality associated with an in-vehicle computing system. In order to provide reliable voice control of vehicle systems, the disclosure provides systems and methods for selectively routing vocal commands to a VPA installed on a mobile device or to a VPA of an in-vehicle computing system.

<FIG> shows an example partial view of an interior of a cabin <NUM> of a vehicle <NUM>, in which a driver and/or one or more passengers may be seated. Vehicle <NUM> of <FIG> may be a motor vehicle including drive wheels (not shown) and an internal combustion engine <NUM>. Internal combustion engine <NUM> may include one or more combustion chambers which may receive intake air via an intake passage and exhaust combustion gases via an exhaust passage. Vehicle <NUM> may be a road automobile, among other types of vehicles. In some examples, vehicle <NUM> may include a hybrid propulsion system including an energy conversion device operable to absorb energy from vehicle motion and/or the engine and convert the absorbed energy to an energy form suitable for storage by an energy storage device. Vehicle <NUM> may include a fully electric vehicle, incorporating fuel cells, solar energy capturing elements, and/or other energy storage systems for powering the vehicle.

As shown, an instrument panel <NUM> may include various displays and controls accessible to a driver of vehicle <NUM>, such as a touch screen <NUM> of an in-vehicle interface system (e.g., an infotainment system), an audio system control panel, and an instrument cluster <NUM>. While the example system shown in <FIG> includes audio system controls that may be performed via a user interface of an in-vehicle interface system such as touch screen <NUM> without a separate audio system control panel, in other embodiments, the vehicle <NUM> may include an audio system control panel, which may include controls for a conventional vehicle audio system such as a radio, compact disc player, MP3 player, and so on. The audio system controls may include features for controlling one or more aspects of audio output via speakers <NUM> of a vehicle speaker system. For example, the audio system controls may control a volume of audio input, a distribution of sound among the individual speakers of the vehicle speaker system, an equalization of audio signals, and/or any other aspect of the audio output.

Instrument cluster <NUM> may include various gauges such as a fuel gauge, tachometer, speedometer, and odometer, as well as indicators and warning lights. A steering wheel <NUM> may project from the instrument panel <NUM> below instrument cluster <NUM>. Optionally, steering wheel <NUM> may include controls <NUM> which may be used in conjunction with touch screen <NUM> to navigate features of an in-vehicle interface system and to control the in-vehicle interface system. As an illustrative example, controls <NUM> may include a button <NUM> which activates or initiates voice control of the in-vehicle interface system. For example, responsive to the button <NUM> being pressed and/or while the button <NUM> is being pressed, a microphone <NUM> may record one or more utterances by a driver or other passenger of the vehicle. The one or more utterances recorded via the microphone <NUM> may then be used to navigate or control one or more features of the in-vehicle interface system if the one or more utterances comprises a voice command.

In addition to the components depicted in <FIG>, it will be appreciated that instrument panel <NUM> may include additional components such as door and window controls, a cigarette lighter which may also be used as a low-voltage power outlet, a glove compartment, and/or any other suitable elements. In one or more embodiments, control of in-vehicle climate via climate control system vents <NUM> may be performed using touch screen <NUM> and thus no separate climate control interface may be included in instrument panel <NUM>. In alternative embodiments, however, a separate climate control interface may be provided.

The cabin <NUM> may include one or more sensors for monitoring the vehicle, the user, and/or the environment. For example, the cabin <NUM> may include one or more seat-mounted pressure sensors <NUM> configured to measure the pressure applied to the seat to determine the presence of a user. The cabin <NUM> may include one or more door sensors <NUM> configured to monitor door activity, such as the opening and/or closing of the door, the locking of the door, the operation of a window of the door, and/or any other suitable door activity event. A humidity sensor <NUM> may be included to measure the humidity content of the cabin <NUM>. A microphone <NUM> may be included to receive user input in the form of voice commands, to enable a user to conduct telephone calls, and/or to measure ambient noise in the cabin <NUM>. It is to be understood that the placement of the sensors illustrated in <FIG> is exemplary, and one or more additional or alternative sensors may be positioned in an engine compartment, on an external surface of the vehicle, and/or in other suitable locations for providing information regarding the operation of the vehicle, ambient conditions of the vehicle, a user of the vehicle, and so on.

The cabin <NUM> may also include one or more user objects, such as mobile device <NUM>, that are stored in the vehicle before, during and/or after traveling. The mobile device may include a smart phone, a tablet, a laptop computer, a portable media player, and/or any suitable mobile computing device. The mobile device <NUM> may be connected to the in-vehicle interface system via communication link <NUM>. The communication link <NUM> may be wired (e.g., via Universal Serial Bus [USB], Mobile High-Definition Link [MHL], High-Definition Multimedia Interface [HDMI], and so on) or wireless (e.g., via BLUETOOTH, WI-FI, Near-Field Communication [NFC], and so on) and configured to provide two-way communication between the mobile device <NUM> and the in-vehicle interface system. For example, the communication link <NUM> may provide sensor and/or control signals from in-vehicle systems and the touch screen <NUM> to the mobile device <NUM> and may provide control and/or display signals from the mobile device <NUM> to in-vehicle systems and the touch screen <NUM>. The communication link <NUM> may also provide power to the mobile device <NUM> from an in-vehicle power source in order to charge an internal battery of the mobile device <NUM>.

While the mobile device <NUM> is illustrated as being spatially separated from the in-vehicle interface system and connected via a substantially external communication link (e.g., a cable or radiofrequency signal), it is to be understood that a mobile device storage structure <NUM> or other storage structure may be formed in the instrument panel <NUM> or another location in the vehicle to hold the mobile device <NUM> in a particular location. The mobile device storage structure <NUM> may include an integrated connector <NUM> to which the mobile device <NUM> may be attached or "docked" for providing a substantially internal communication link between the mobile device <NUM> and the interface system.

<FIG> shows a block diagram of an in-vehicle interface system <NUM> and related elements with which the in-vehicle interface system <NUM> is in communication. The in-vehicle interface system <NUM> may be located and/or integrated within a vehicle <NUM>, such as vehicle <NUM> of <FIG>. The in-vehicle interface system <NUM> may communicate with one or more elements within the vehicle, including but not limited to vehicle systems connected via an in-vehicle interconnect, such as Controller-Area-Network (CAN) bus <NUM>. It is to be understood that any suitable number and/or combination of interconnects may be used to permit communication between the in-vehicle interface system <NUM>, or simply the interface system <NUM>, and various in-vehicle components, including but not limited to CAN buses, Media Oriented Systems Transport (MOST) buses, Ethernet-based interconnects, and so on. Interconnects may communicate directly with in-vehicle components and/or may communicate with such components via intervening processors. In some embodiments, one or more in-vehicle components may communicate directly with the interface system <NUM> without or in addition to communicating with the interface system <NUM> via the CAN bus <NUM>.

The interface system <NUM> may pass information from the CAN bus to an external mobile device <NUM> in order to leverage the memory <NUM> and processor <NUM> as well as other computing resources of the mobile device <NUM> for controlling various in-vehicle elements of the vehicle <NUM>. The interface system <NUM> may be connected to the mobile device <NUM> via a wired connection in some embodiments. In additional or alternative embodiments, the interface system <NUM> may include a wireless gateway and/or a protocol <NUM>, enabling wireless connectivity between the interface system <NUM> and the mobile device <NUM>. The mobile device <NUM> similarly includes a protocol <NUM> for enabling communication between the interface system <NUM> and the mobile device <NUM>. The protocols <NUM> and <NUM> may comprise a standard protocol such as Smart Device Link (SDL), for example, or a proprietary protocol.

The display <NUM> may display images to provide visual feedback related to navigation, media playback, telephone, vehicle system control (e.g., cabin temperature control, cruise control settings, odometer output/settings, vehicle diagnostics control, vehicle operating mode control, and so on), and/or other vehicle-related functions. In some embodiments, the display <NUM> may include a touch screen <NUM>, such as touch screen <NUM> of <FIG>, enabling user input for controlling the vehicle-related functions to be received via the display <NUM>.

The mobile device <NUM> may receive signals from the interface system <NUM> and/or the display <NUM>, process the signals based on non-transitory instructions stored on the memory <NUM> and executed by the processor <NUM>, and output display and/or control signals to the display <NUM> and/or the interface system <NUM>. In some embodiments, the mobile device <NUM> may communicate with a remote service <NUM> via a network <NUM>. In such embodiments, the mobile device <NUM> may leverage the computing resources of the remote service <NUM> and/or additional information stored at the remote service <NUM> to perform and/or assist with the processing of the signals from the interface system <NUM>. For example, the mobile device <NUM> may utilize the remote service <NUM> to assist with generating a response to a particular user input. Although one remote service is illustrated in <FIG>, it is to be understood that the mobile device <NUM> may communicate with one or more remote services including remote service <NUM> during operation.

As illustrated, the mobile device <NUM> is separate from and not integrated within the vehicle <NUM>, although the mobile device <NUM> may be located within the vehicle cabin during travel and while connected to the interface system <NUM> (e.g., via a communication link, such as communication link <NUM> of <FIG>). In contrast, each of the other systems and elements illustrated within the vehicle <NUM> in <FIG> may be integrated within the vehicle. The mobile device <NUM> may include computing resources (e.g., memory <NUM> and processor <NUM>) that are separate from computing resources integrated within the vehicle <NUM> (e.g., the computing resources utilized by the components illustrated within the vehicle <NUM> of <FIG>). While the mobile device <NUM> may operate in accordance with an operating system and run one or more applications, in some embodiments, the interface system <NUM> and/or display <NUM> may not include an operating system and/or computing resources for operating in accordance with an operating system or running applications. For example, in such embodiments, the display <NUM> and the interface system <NUM> may not provide any user interface when disconnected from the computing resources (e.g., memory <NUM> and processor <NUM>) of the mobile device <NUM>. In some embodiments, the display <NUM> and the interface system <NUM> may not include a memory device and/or a processor. In other embodiments, the display <NUM> and the interface system <NUM> may include a memory device and/or a processor capable of providing basic functionality related to the elements (e.g., displaying images in accordance with display instructions and/or transmitting/receiving signals). In such embodiments, the memory device(s) and/or processor(s) of the display <NUM> and/or the interface system <NUM> may not include or be capable of running an operating system and/or providing a user interface when operating independently from the mobile device <NUM>.

Display signals may control the output of display <NUM>, while control signals may control one or more other vehicle systems <NUM> in communication with the interface system <NUM>. For example, the vehicle systems <NUM> may include controllable elements related to the engine and/or auxiliary elements, such as windshield wipers, windows, doors/door locks, headlights, air conditioning systems, and so on. The control signals may also control audio output at one or more speakers <NUM> of the vehicle <NUM>. For example, the control signals may adjust audio output characteristics such as volume, equalization, audio image (e.g., the configuration of the audio signals to produce audio output that appears to a user to originate from one or more defined locations), audio distribution among a plurality of speakers <NUM>, and so on.

User input may be accepted via one or more in-vehicle controls <NUM> transmitted to the interface system <NUM> and/or the mobile device <NUM> as additional or alternative input sources to the optional touch screen of display <NUM>, for example. The in-vehicle controls <NUM> may include any suitable user-actuatable element within and/or on the vehicle, including but not limited to any number or combination of buttons, sliders, switches, knobs, levers, joysticks, key pads, pedals, and so on. For example, in-vehicle controls <NUM> may include control elements within a cabin of the vehicle <NUM>, such as steering wheel controls (e.g., steering wheel-mounted audio system controls, cruise controls, windshield wiper controls, headlight controls, turn signal controls, and so on), instrument panel controls, accelerator/brake/clutch pedals, a gear shift, door/window controls positioned in a driver or passenger door, seat controls, cabin light controls, audio system controls, cabin temperature controls, and so on. Control elements positioned on an outside of a vehicle (e.g., controls for a security system) may also be connected to the interface system <NUM> via the CAN bus <NUM>. The control elements of the in-vehicle controls <NUM> may be physically and permanently positioned on and/or in the vehicle for receiving user input, even when the mobile device <NUM> is removed from the vehicle.

As a specific example, the in-vehicle controls <NUM> may include a microphone <NUM> integrated within the cabin of the vehicle <NUM> which records or senses user input such as voice control commands that are routed to the interface system <NUM> and/or the mobile device <NUM>. As another example, the steering wheel controls and/or the instrument panel controls of the in-vehicle controls <NUM> may include the button <NUM> depicted in <FIG> for initiating voice control, wherein pressing the button <NUM> indicates that utterances sensed by the microphone <NUM> comprise voice commands. In this way, utterances not intended for voice control of vehicle elements may not be inadvertently processed by the interface system <NUM> and/or the mobile device <NUM> for controlling one or more elements of the vehicle <NUM>. Additionally or alternatively, the user may indicate that an utterance is intended for voice control by uttering a wake word (e.g., a particular voice data string, which may be a word, phrase, tone, or other predefined sound) for voice control. For example, the interface system <NUM> and/or the mobile device <NUM> may monitor audio recorded by the microphone <NUM> and specifically process the audio to detect the wake word without otherwise processing or handling the audio, and in response to detecting the wake word being uttered by an occupant of the vehicle <NUM>, the interface system <NUM> and/or the mobile device <NUM> may activate respective VPAs <NUM> and <NUM> for processing an utterance sensed by the microphone <NUM> and following the wake word or trigger word, as described further herein.

The CAN bus <NUM> may communicate with one or more vehicle sensors <NUM> in order to provide an indication of operating states and/or conditions of the vehicle to the interface system <NUM>. The interface system <NUM> may pass the sensor signals from the vehicle sensors <NUM> to the mobile device <NUM> in order to provide contextual information relating to the operation and/or environment of the vehicle and/or user to the mobile device <NUM>. The vehicle sensors <NUM> may include any suitable sensor, such as the in-cabin sensors described herein above with regard to <FIG>, vehicle operating state sensors (e.g., sensors monitoring fuel storage, engine temperature, oil storage, engine speed, vehicle speed, transmission/gear settings, tire pressure, traction, diagnostics information, and so on), and environmental sensors (e.g., a rear camera and/or other sensors monitoring ambient conditions, such as temperature, humidity, pressure, road surface/conditions, and so on). Upon receiving the user input and/or the sensor signals at the mobile device <NUM>, the mobile device <NUM> may determine an adjustment to the display <NUM> and/or control instructions for the one or more vehicle systems <NUM>.

Information regarding power states and/or power control signals may be exchanged between a power module <NUM> and the mobile device <NUM> via the CAN bus <NUM> and the interface system <NUM>. For example, the mobile device <NUM> may receive information from the power module <NUM> and update the display <NUM> to provide feedback to the user regarding the power state of the one or more vehicle components. The mobile device <NUM> may additionally or alternatively provide control signals to the power module <NUM> based on user input, information received from the vehicle sensors <NUM>, and/or information received from the power module <NUM>.

An antenna <NUM> may provide AM/FM radio signals to the interface system <NUM> for processing at the mobile device <NUM> and/or the interface system <NUM>. For example, the mobile device <NUM> may control the speakers <NUM> to control audio broadcast by AM/FM radio stations and control the display <NUM> to display information related to the AM/FM radio stations and/or content being broadcast (e.g., information broadcast alongside the audio via a radio data system).

In order to communicate with the varied vehicle systems and mobile device <NUM> as described above, the interface system <NUM> may include one or more modules in some embodiments for resolving different communication protocols and other issues that may arise while passing signals between disparate systems. For example, the interface system <NUM> may include a video switching module (not shown) to synchronize and/or format a video feed from a rear camera for display on the display <NUM>. As another example, the interface system <NUM> may include an encoder (not shown) and/or a decoder (not shown) for converting signals received from one system and/or device to a format that is compatible with or otherwise able to be interpreted by a target system and/or device. It is to be understood that one or more of the above-described modules are optional, and in some embodiments, the interface system <NUM> may pass received signals to another system and/or device without providing any adjustment to the signals.

The interface system <NUM> includes a device platform <NUM> comprising an infotainment platform which provides primitives to communicate with components of the vehicle <NUM>. For example, the device platform <NUM> may generate the user interface for display via the display <NUM> and/or communicate with one or more components of the vehicle <NUM>, including but not limited to the speakers <NUM> via the audio amplifier <NUM>, the vehicle sensors <NUM>, the power module <NUM>, the vehicle systems <NUM>, the in-vehicle controls <NUM>, and/or the antenna <NUM>, for providing commands to one or more of the vehicle components and/or receiving signals from one or more of the vehicle components.

The interface system <NUM> includes a virtual personal assistant (VPA) <NUM> for supporting voice dialogs between a user of the vehicle <NUM> and the interface system <NUM>. The VPA <NUM> comprises a speech processing module that processes voice commands, such as voice commands received from the microphone <NUM>. The VPA <NUM> thus performs automatic speech recognition on a received vocal utterance received via the microphone <NUM> to convert the vocal utterance to natural language text, and processes the natural language text to identify a vocal command from the vocal utterance. The VPA <NUM> may further provide an audio response, for example, by using text-to-speech, which may be output via the speakers <NUM>.

To implement the command of the voice command identified by the VPA <NUM>, the interface system <NUM> further includes a VPA services module <NUM> that provides access to device functions. In particular, the VPA services module <NUM> provides a catalog of device functions and an interface for executing device functions which map commands to appropriate components of the vehicle <NUM>. For example, a voice utterance received by the microphone <NUM> may be processed by the VPA <NUM> to identify a command "change equalizer to jazz" in the voice utterance. The VPA services module <NUM> links the command from the voice utterance identified by the VPA <NUM> to a corresponding device function which may be executed, for example, by the device platform <NUM>. For example, the VPA services module <NUM> may route the "change equalizer to jazz" command identified by the VPA <NUM> to the device platform <NUM> which in turn controls the audio equalization settings of the audio amplifier <NUM> and/or the speakers <NUM> to a combination of equalizations settings predetermined and optimized for jazz music. Alternatively, rather than providing the command to the device platform <NUM>, in some examples the VPA services module <NUM> may directly communicate with and control one or more components of the vehicle <NUM> according to the command.

The interface system <NUM> further includes a VPA switch <NUM> for arbitrating between the VPA <NUM> of the vehicle <NUM> and the VPA <NUM> of the mobile device <NUM>. The VPA switch <NUM> routes an utterance sensed by the microphone <NUM>, for example, to the VPA <NUM> of the interface system <NUM>, to the VPA <NUM> of the mobile device <NUM>, or to both the VPA <NUM> and the VPA <NUM>. Various methods for routing an utterance to one or more of the VPA <NUM> and the VPA <NUM> are described further herein with regard to <FIG>.

As mentioned above, the mobile device <NUM> includes a VPA <NUM> for supporting voice dialogs between a user of the vehicle <NUM> and the interface system <NUM>. Similar to the VPA <NUM> of the interface system <NUM>, the VPA <NUM> of the mobile device <NUM> comprises a speech processing module that processes voice commands, such as voice commands received from the microphone <NUM> and/or from a microphone (not shown) of the mobile device <NUM>. The VPA <NUM> thus performs automatic speech recognition on a received vocal utterance received via the microphone <NUM> to convert the vocal utterance to natural language text, and processes the natural language text to identify a vocal command from the vocal utterance. The VPA <NUM> may further provide an audio response, for example, by using text-to-speech, which may be output via the speakers <NUM>.

The mobile device <NUM> further includes a VPA services module <NUM> providing a catalog of device functions and an interface for executing device functions which map commands to appropriate components of the vehicle <NUM>. The VPA services module <NUM> may further map commands to a mobile device platform <NUM>.

The VPA <NUM> of the mobile device <NUM> and the VPA <NUM> of the vehicle <NUM> may be identical, in some examples. In other examples, the VPA <NUM> may include extended functionality with respect to the VPA <NUM>. As a specific example, the VPA <NUM> may be equipped with a limited vocabulary specific to control of various vehicle components, and therefore the VPA <NUM> may be limited to enabling voice control of vehicle components. In contrast, the VPA <NUM> may be equipped with a larger vocabulary which is not limited to control of various vehicle components. Further, the VPA <NUM> may access an even larger vocabulary by communicating with a remote service <NUM> via the network <NUM>. As an example, the VPA <NUM> may support adjusting control of the audio amplifier <NUM> and the speakers <NUM>, for example by adjusting volume or equalization, and further the VPA <NUM> may support controlling the device platform <NUM> to change radio transmissions received via the antenna <NUM>. However, if the user vocally commands the VPA <NUM> to play a particular song by a particular artist, the vocabulary of the VPA <NUM> may be too limited to understand the command. Furthermore, the functionality of the device platform <NUM> may not support streaming music that is not stored locally. In contrast, the VPA <NUM> may be able to process the vocal command to play the particular song by the particular artist, and furthermore may be able to stream the particular song via the mobile device platform <NUM> and/or the remote service <NUM>.

<FIG> is a flow chart of an example method <NUM> for routing an utterance by a user to a virtual personal assistant (VPA) of a vehicle or a mobile device. The method <NUM> may be performed by a system within a vehicle, such as an in-vehicle interface system. Further, one or more actions of the method <NUM> may be performed by an in-vehicle interface system, while other actions of the method <NUM> may be performed by a mobile device positioned within the vehicle and communicatively coupled to the in-vehicle interface system.

Method <NUM> begins at <NUM>. At <NUM>, method <NUM> includes receiving an utterance via a microphone. The utterance comprises one or more words spoken by a person in the vehicle, which may comprise a vocal command to control one or more components of the vehicle. A VPA switch of the in-vehicle interface system, such as VPA switch <NUM> of the interface system <NUM>, may receive the utterance via the microphone, which may comprise a microphone of the vehicle such as microphone <NUM> or the microphone <NUM>, or alternatively may comprise a microphone of the mobile device. Further, method <NUM> may receive the utterance responsive to detecting a wake word uttered by the user or occupant of the vehicle. Method <NUM> may receive the utterance spoken after the utterance of the wake word, and may complete reception of the utterance in response to an additional trigger word or in response to a threshold duration elapsing after receiving the utterance without sensing additional utterance(s) by the user via the microphone. As another example, method <NUM> may receive the utterance responsive to detecting a voice control button being pressed, such as the button <NUM>, being pressed. Further, the VPA switch <NUM> may receive the utterance via the microphone. As described further herein, the VPA switch <NUM> determines in method <NUM> whether to route the utterance to a vehicle VPA or to a mobile device VPA.

After receiving the utterance, method <NUM> continues to <NUM>. At <NUM>, a VPA switch such as the VPA switch <NUM> determines if a vehicle VPA, such as the VPA <NUM> of the interface system <NUM>, exists or is present. For example, a vehicle VPA such as the VPA <NUM> may comprise an optional component of the vehicle, such as the vehicle <NUM>, and so in some instances the vehicle may not be equipped with a vehicle VPA in order to reduce cost of the vehicle. Nevertheless, the interface system <NUM> may include the VPA switch <NUM> for routing utterances in case a vehicle VPA is eventually installed in the vehicle. Method <NUM> therefore determines whether a vehicle VPA is installed within the vehicle and connected to the VPA switch <NUM>, for example.

If a vehicle VPA is present ("YES"), method <NUM> continues to <NUM>. At <NUM>, the VPA switch routes the utterance to the vehicle VPA to generate a response comprising one or more control signals according to the utterance. For example, method <NUM> may process the utterance with the VPA <NUM> to perform automatic speech recognition. That is, the VPA <NUM> converts the utterance to natural language text. The VPA <NUM> may further process the natural language text to identify a command in the utterance. As an example, the VPA <NUM> may compare the natural language text derived from the utterance to commands stored in the catalog of commands in the VPA services module <NUM>. The VPA <NUM> may thus determine that at least a portion of the natural language text corresponds to a command. In some examples, the VPA <NUM> may determine that the natural language text does not include a command responsive to even a portion of the natural language text not matching to any command in the catalog of commands stored in the VPA services module <NUM>. In any case, the VPA <NUM> generates a response to the utterance comprising a control signal to be transmitted to one or more components by the VPA services module <NUM>.

Continuing at <NUM>, method <NUM> includes controlling one or more elements of the vehicle according to the response generated at <NUM>. In an instance wherein the utterance does not include a command or at least a valid command for controlling a component of the vehicle, the response may indicate that the utterance did not include a valid command, and so method <NUM> may control one or more elements of the vehicle, such as the display device <NUM> and/or the speakers <NUM>, to inform the user that the utterance did not include a valid command. For example, method <NUM> may control the display device <NUM> to display a message indicating that the utterance did not include a valid command, and/or method <NUM> may control the speakers <NUM> to playback a tone or a voice message indicating that the utterance did not include a valid command. Method <NUM> then returns. Thus, in some examples, method <NUM> may use the vehicle VPA to control one or more components of a vehicle responsive to and according to an utterance spoken by a user of the vehicle.

However, referring again to <NUM>, if a vehicle VPA is not present ("NO"), method <NUM> continues to <NUM>. At <NUM>, method <NUM> includes determining whether a mobile device, such as the mobile device <NUM> or <NUM>, is connected and active. For example, a mobile device may be determined to be connected upon establishing a wired or wireless connection to an interface system, such as interface system <NUM> of <FIG>, and/or upon completing a data transaction between the mobile device and the interface system (e.g., receiving data from the mobile device, performing a handshake operation, and so on). The mobile device may be determined to be active while the mobile device is running a vehicle interface application and/or otherwise capable of receiving signals from the vehicle, processing the received signals, and transmitting signals to control one or more vehicle systems.

Responsive to determining that the mobile device is not connected and/or not active ("NO") at <NUM>, method <NUM> proceeds to <NUM>. At <NUM>, method <NUM> includes outputting an alert to connect and/or activate the mobile device. For example, method <NUM> may display an image on a display device integrated in the vehicle (e.g., display <NUM> of <FIG>). Additionally or alternatively, method <NUM> may play a pre-recorded audio message or audio tone via one or more speakers integrated in the vehicle (e.g., speakers <NUM> of <FIG>). Method <NUM> then returns. Thus, method <NUM> may not control one or more components of the vehicle according to the utterance if a vehicle VPA is not present and a mobile device is not connected and active.

Referring again to <NUM>, if method <NUM> determines that a mobile device is connected and active ("YES"), method <NUM> continues to <NUM>. At <NUM>, method <NUM> includes transmitting the utterance to the mobile device. For example, for the interface system <NUM>, the VPA switch <NUM> routes the utterance, via the protocol <NUM>, to the mobile device <NUM>. The mobile device <NUM> then processes the utterance with a VPA installed on the mobile device (e.g., VPA <NUM> of <FIG>) and generates a response for the utterance. At <NUM>, method <NUM> includes receiving the response generated by the VPA of the mobile device from the mobile device. For example, the VPA services module <NUM> of the mobile device <NUM> may transmit the response, via the protocol <NUM> of the mobile device <NUM>, to the interface system <NUM> of the vehicle <NUM>. The interface system <NUM> receives the response, for example via the protocol <NUM>, and at <NUM>, method <NUM> includes controlling one or more elements or components of the vehicle according to the response. Method <NUM> then returns.

Thus, in some examples, the VPA switch <NUM> may route an utterance to the vehicle VPA <NUM> or to the mobile device VPA <NUM> depending on whether the vehicle VPA <NUM> is installed in the vehicle <NUM>.

However, wherein both the vehicle VPA <NUM> and the mobile device VPA <NUM> are present in the vehicle <NUM>, the VPA switch <NUM> routes an utterance to both VPAs <NUM> and <NUM>. As an example, <FIG> is a flow chart of an example method <NUM> for parallel processing of an utterance according to an embodiment. In particular, method <NUM> relates to simultaneous or parallel processing of an utterance with a vehicle VPA of an in-vehicle interface system <NUM> and a mobile device VPA of a mobile device <NUM>, and controlling one or more vehicle components <NUM> according to the output of the vehicle VPA and/or the mobile device VPA. The vehicle VPA of the in-vehicle interface system <NUM> may correspond to the VPA <NUM> of the interface system <NUM>, and the mobile device VPA of the mobile device <NUM> may correspond to the VPA <NUM> of the mobile device <NUM> described hereinabove. The one or more vehicle components <NUM> may correspond to one or more components of the vehicle <NUM> described hereinabove, including but not limited to a display <NUM>, vehicle systems <NUM>, speakers <NUM> and/or an audio amplifier <NUM>, in-vehicle controls <NUM>, vehicle sensors <NUM>, a power module <NUM>, an antenna <NUM>, a device platform <NUM>, and so on.

Method <NUM> begins at <NUM>. At <NUM>, the interface system <NUM> receives an utterance, sensed for example by a microphone in the vehicle such as microphone <NUM>, at a VPA switch such as VPA switch <NUM>. At <NUM>, the VPA switch transmits the utterance from the interface system <NUM> to the mobile device <NUM>, for example via protocols <NUM> and <NUM>. At <NUM>, the mobile device <NUM> receives the utterance from the interface system <NUM>. At <NUM>, the mobile device <NUM> processes the utterance with the mobile device VPA to generate a mobile device VPA response. At <NUM>, the mobile device <NUM> transmits the mobile device VPA response to the interface system <NUM>. The interface system <NUM> receives the mobile device VPA response at <NUM>.

Meanwhile, as the mobile device <NUM> processes the utterance as described above, method <NUM> proceeds from <NUM> to <NUM>. At <NUM>, the interface system <NUM> processes the utterance with a vehicle VPA to generate a vehicle VPA response. At <NUM>, the interface system <NUM> evaluates the vehicle VPA response and the mobile device VPA response to determine one or more control signals for one or more target vehicle components specified by the responses. As an illustrative example, the VPA switch may evaluate the vehicle VPA response and the mobile device VPA response according to which response is received first. As another example, the VPA switch may evaluate the vehicle response and the mobile device VPA response to determine whether the responses are different. If the responses are the same, the VPA switch uses either response as a control signal. However, if the responses are different, the VPA switch may determine that one of the responses is better than the other, and may select the best response for use as the control signal. For example, the utterance may comprise a vocal command to navigate to a particular location with a location name. The vehicle VPA response may comprise a control signal to initiate a navigation system in the device platform, for example, as well as a control signal for the display device and/or the speakers to indicate that the VPA could not understand the location name. Meanwhile, the mobile device VPA response may comprise a control signal to initiate the navigation system for navigating to the particular location. The VPA switch may select the mobile device VPA response as the control signal as the mobile device VPA response is a complete response whereas the vehicle VPA response is an incomplete response. As yet another example, the VPA switch may use the response received first as a control signal, and may evaluate the response received second to determine whether the second response is superior to the first response. For example, in the above example wherein the mobile device VPA accurately resolves the location name while the vehicle VPA does not resolve the location name, the VPA switch may receive the vehicle VPA response first and thus the interface system <NUM> may initiate the navigation system with the indication that the location name was not resolved, and then upon receiving the mobile device VPA response, the interface system <NUM> may override the first response with the second response so that the navigation system navigates to the particular location.

Thus, continuing at <NUM>, the interface system <NUM> transmits the one or more control signals to the one or more target vehicle components. At <NUM>, the one or more target vehicle components of the vehicle components <NUM> perform one or more actions according to the control signal. Method <NUM> then returns.

Thus, the utterance is processed by both the vehicle VPA and the mobile device VPA, and one or more vehicle components are subsequently controlled according to the response from the vehicle VPA, the response from the mobile device VPA, or both responses.

In other examples, the VPA switch may selectively route the utterance to the vehicle VPA or the mobile device VPA according to the intention of the user. As an illustrative and non-limiting example, <FIG> is a flow chart of a method <NUM> for selectively distributing an utterance between a vehicle VPA and a mobile device VPA. In particular, method <NUM> relates to determining an intent of an utterance, routing the utterance to either a vehicle VPA of an in-vehicle interface system <NUM> or a mobile device VPA of a mobile device <NUM> according to the intent of the utterance, and controlling one or more vehicle components <NUM> according to the output of the vehicle VPA and/or the mobile device VPA. The vehicle VPA of the in-vehicle interface system <NUM> may correspond to the VPA <NUM> of the interface system <NUM>, and the mobile device VPA of the mobile device <NUM> may correspond to the VPA <NUM> of the mobile device <NUM> described hereinabove. The one or more vehicle components <NUM> may correspond to one or more components of the vehicle <NUM> described hereinabove, including but not limited to a display <NUM>, vehicle systems <NUM>, speakers <NUM> and/or an audio amplifier <NUM>, in-vehicle controls <NUM>, vehicle sensors <NUM>, a power module <NUM>, an antenna <NUM>, a device platform <NUM>, and so on.

Method <NUM> begins at <NUM>. At <NUM>, the interface system <NUM> receives the utterance. At <NUM>, the interface system <NUM> predicts the user intent based on at least a portion of the utterance. For example, the VPA switch of the interface system <NUM> may be configured to convert an initial portion of the utterance (e.g., one or more first words of the utterance) to natural language text and determine whether the natural language text of the initial portion includes a command word. Such a command word in the initial portion of the utterance may indicate the intent of the user. For example, if the initial portion of the utterance includes the word "navigate" or "directions," the VPA switch may determine that the user intent is to obtain directions or assistance navigating to a given location. As another example, if the initial portion of the utterance includes the word "increase" or "decrease," the VPA switch may determine that the user intent is to adjust the settings of a vehicle system, such as the audio system, one or more in-vehicle controls, or one or more vehicle systems (e.g., adjusting cruise control speed, adjusting climate control settings, adjusting audio volume, and so on). Thus, the VPA switch may include a table of command words linked to different intended functions. At <NUM>, the interface system <NUM> determines whether the intent is supported by the vehicle VPA. For example, the table of command words linked to the different intended functions may further specify whether the intended functions can be performed by the vehicle VPA. As an example, functions relating to controlling vehicle components such as the vehicle systems <NUM>, the in-vehicle controls <NUM>, the audio amplifier <NUM>, and the speakers <NUM> may typically be performed by the vehicle VPA. However, other functions such as playing a song by a particular musical artist or navigating to a particular location may require a larger vocabulary for generating the response or may require interaction with a remote service, such as remote service <NUM>. The vocabulary of the vehicle VPA and the functionality of the device platform <NUM> may be limited with respect to such functions, and so in some examples such intended functions may not be supported by the vehicle VPA.

Thus, if the intent is supported by the vehicle VPA ("YES"), method <NUM> continues to <NUM>. At <NUM>, the interface system <NUM> processes the utterance with the vehicle VPA to generate a vehicle VPA response. At <NUM>, one or more vehicle components <NUM> perform an action according to the vehicle VPA response. Method <NUM> then returns.

However, referring again to <NUM>, if the intent is not supported by the vehicle VPA ("NO"), method <NUM> continues to <NUM>. At <NUM>, the interface system <NUM> transmits the utterance to the mobile device <NUM>. At <NUM>, the mobile device <NUM> receives the utterance. At <NUM>, the mobile device processes the utterance with the mobile device VPA to generate a mobile device VPA response. At <NUM>, the mobile device <NUM> transmits the mobile device VPA response to the interface system <NUM>.

At <NUM>, the interface system <NUM> receives the mobile device VPA response. At <NUM>, one or more vehicle components <NUM> perform one or more actions according to the mobile device VPA response. Method <NUM> then returns.

Thus, a method for voice control of a vehicle component includes selectively routing a vocal utterance to an in-vehicle VPA system or to a mobile device VPA system according to an intention of the user. According to one embodiment, a system for controlling a vehicle is provided. The system includes a microphone, at least one vehicle component, and an interface system communicatively coupled to the microphone and the at least one vehicle component. The interface system is configured to: detect, via the microphone, an utterance of a vocal command by an occupant of the vehicle; transmit the utterance to at least one of a virtual personal assistant (VPA) of the vehicle and a VPA of a mobile device positioned within the vehicle; receive control signals from at least one of the VPA of the vehicle and the VPA of the mobile device, the control signals generated according to the vocal command; and transmit the control signals to the at least one vehicle component for controlling the at least one vehicle component.

The description of embodiments has been presented for purposes of illustration and description. For example, unless otherwise noted, one or more of the described methods may be performed by a suitable device and/or combination of devices, such as the vehicle systems and mobile devices described above with respect to <FIG> and <FIG>. The methods may be performed by executing stored instructions with one or more logic devices (e.g., processors) in combination with one or more hardware elements, such as storage devices, memory, hardware network interfaces/antennas, switches, actuators, clock circuits, and so on. The described methods and associated actions may also be performed in various orders in addition to the order described in this application, in parallel, and/or simultaneously. The described systems are exemplary in nature, and may include additional elements and/or omit elements.

Claim 1:
A system for controlling a vehicle (<NUM>), comprising:
a microphone (<NUM>);
at least one vehicle component; and
an interface system (<NUM>) communicatively coupled to the microphone and the at least one vehicle component, the interface system configured to:
detect, via the microphone, an utterance of a vocal command by an occupant of the vehicle;
determine whether a virtual personal assistant, VPA, (<NUM>) of the vehicle is present within the vehicle;
responsive to determining that the VPA of the vehicle is present within the vehicle, transmit the utterance to both the VPA (<NUM>) of the vehicle (<NUM>) and a VPA (<NUM>) of a mobile device (<NUM>) positioned within the vehicle, receive a first control signal from the VPA of the vehicle, receive a second control signal from the VPA of the mobile device, and determine control signals for the at least one vehicle component by evaluating the first control signal and the second control signal;
responsive to determining that the VPA of the vehicle is not present within the vehicle, transmit the utterance to the VPA (<NUM>) of the mobile device (<NUM>), and receive control signals from the VPA of the mobile device; and
transmit the control signals to the at least one vehicle component for controlling the at least one vehicle component.