Context-aware signal conditioning for vehicle exterior voice assistant

A vehicle includes a plurality of microphones to obtain speech from a person outside the vehicle as an input signal and a sensor system to determine a location and orientation of the person relative to the vehicle. The vehicle also includes a controller to determine characteristics of the input signal and to determine whether to perform speech enhancement on the input signal based on one or more of the characteristics and the location and orientation of the person.

INTRODUCTION

The subject disclosure relates to context-aware signal conditioning for a vehicle exterior voice assistant.

Vehicles (e.g., automobiles, trucks, construction equipment, farm equipment, automated factory equipment, shuttle buses) increasingly facilitate interaction between an operator and the vehicle, including audible interaction. An exterior voice assistant refers to a system with microphones and speakers on an exterior of a vehicle. The exterior voice assistant facilitates verbal communication between the vehicle and a person outside the vehicle. The exterior voice assistant may enable reactive assistance, proactive assistance, and social interaction. Reactive assistance refers to the exterior voice assistant responding to a query such as “open liftgate.” Proactive assistance refers to the exterior voice assistant providing anticipatory alerts or recommendations such as “you left a pet in the vehicle.” Social interaction refers to a welcome or farewell greeting, for example. Background noise and reverberation may degrade the quality of the voice signal. The position, orientation, and speaking volume of the person may also affect the voice signal. Accordingly, it is desirable to provide context-aware signal conditioning for the vehicle exterior voice assistant.

SUMMARY

In one exemplary embodiment, a vehicle includes a plurality of microphones to obtain speech from a person outside the vehicle as an input signal and a sensor system to determine a location and orientation of the person relative to the vehicle. The vehicle also includes a controller to determine characteristics of the input signal and to determine whether to perform speech enhancement on the input signal based on one or more of the characteristics and the location and orientation of the person.

In addition to one or more of the features described herein, the sensor system includes an array of ultrawideband (UWB) or Bluetooth Low Energy (BLE) detectors.

In addition to one or more of the features described herein, each of the array of UWB or BLE detectors communicates with a device in possession of the person to determine the location of the person relative to the vehicle and a level of the input signal at each of the plurality of microphones at different locations of the vehicle is used to determine the orientation of the person relative to the vehicle.

In addition to one or more of the features described herein, the controller determines whether to instruct the person to take an action to improve a quality of the input signal.

In addition to one or more of the features described herein, the controller instructs the person to move closer to one of the plurality of microphones or to change the orientation to face the one of the plurality of microphones.

In addition to one or more of the features described herein, the controller determines whether to instruct the person to speak more loudly based on a volume detected at one of the plurality of microphones to which the person is closest.

In addition to one or more of the features described herein, the controller determines whether to perform the speech enhancement on the input signal based on fuzzy logic, on Bayesian probability, on Dempster-Shafer evidential decision-making, or on statistical machine learning.

In addition to one or more of the features described herein, the controller performs the speech enhancement by performing denoising.

In addition to one or more of the features described herein, the controller performs the speech enhancement by performing de-reverberation.

In addition to one or more of the features described herein, the controller performs the speech enhancement by performing a combination of denoising, de-reverberation, and source separation.

In another exemplary embodiment, a method in a vehicle includes arranging a plurality of microphones to obtain speech from a person outside the vehicle as an input signal and arranging a sensor system to determine a location and orientation of the person relative to the vehicle. The method also includes configuring a controller to determine characteristics of the input signal and to determine whether to perform speech enhancement on the input signal based on one or more of the characteristics and the location and orientation of the person.

In addition to one or more of the features described herein, the arranging the sensor system includes arranging an array of ultrawideband (UWB) or Bluetooth Low Energy (BLE) detectors.

In addition to one or more of the features described herein, the method also includes configuring each of the array of UWB or BLE detectors to communicate with a device in possession of the person to determine the location of the person relative to the vehicle and determining the orientation of the person relative to the vehicle based on a level of the input signal at each of the plurality of microphones at different locations of the vehicle.

In addition to one or more of the features described herein, the configuring the controller includes configuring the controller to determine whether to instruct the person to take an action to improve a quality of the input signal.

In addition to one or more of the features described herein, the configuring the controller includes configuring the controller to instruct the person to move closer to one of the plurality of microphones or to change the orientation to face the one of the plurality of microphones.

In addition to one or more of the features described herein, the configuring the controller includes configuring the controller to determine whether to instruct the person to speak more loudly based on a volume detected at one of the plurality of microphones to which the person is closest.

In addition to one or more of the features described herein, the configuring the controller includes configuring the controller to determine whether to perform the speech enhancement on the input signal based on fuzzy logic, on Bayesian probability, on Dempster-Shafer evidential decision-making, or on statistical machine learning.

In addition to one or more of the features described herein, the configuring the controller includes configuring the controller to perform the speech enhancement by performing denoising.

In addition to one or more of the features described herein, the configuring the controller includes configuring the controller to perform the speech enhancement by performing de-reverberation.

In addition to one or more of the features described herein, the configuring the controller includes configuring the controller to perform the speech enhancement by performing a combination of denoising, de-reverberation, and source separation.

DETAILED DESCRIPTION

As previously noted, an exterior voice assistant facilitates verbal interaction between a person outside a vehicle and the vehicle. The accuracy with which a person is assisted may rely in large part on how accurately the person is understood. Embodiments of the systems and methods detailed herein relate to context-aware signal conditioning for a vehicle exterior voice assistant. Speech enhancement may be performed on the input signal from the person. Under some situations, speech enhancement techniques may degrade the quality of the input signal rather than improve it. In those cases, signal improvement may still be attained through guidance to the person speaking to adjust distance, orientation, or volume. As detailed, the current scenario (i.e., context) is identified in order to determine the proper approach to improving the input signal quality.

In accordance with an exemplary embodiment,FIG.1shows a vehicle100that implements context-aware signal conditioning for a vehicle exterior voice assistant125. The exemplary vehicle100shown inFIG.1is an automobile101. The vehicle100is shown with four external microphones110(i.e., microphones arranged to obtain audio from outside the vehicle100) located at each side of the vehicle100. In alternate embodiments, the numbers and locations of the microphones110are not limited by the exemplary illustration. According to exemplary embodiments, the microphones110may be digital array microphones, for example, the vehicle100is also shown with a speaker115. In alternate embodiments, any number of speakers115may be arranged around the vehicle100and may, for example, be co-located with the microphones110. The vehicle100is also shown with a controller120. The controller120may use information from one or more external sensors130(e.g., radar system, lidar system, camera) and/or one or more vehicle sensors135(e.g., inertial measurement unit (IMU), steering angle detector, accelerometer) to control an aspect of operation of the vehicle100. For example, semi-autonomous operation (e.g., collision avoidance, adaptive cruise control) may be controlled by the controller120.

The controller120may be part of the vehicle exterior voice assistant125, along with the microphones110and speaker115. The controller120may implement aspects of signal improvement for the vehicle exterior voice assistant125, as detailed inFIG.2. The controller may include processing circuitry that may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (shared, dedicated, or group) and memory that executes one or more software or firmware programs, a combinational logic circuit, and/or other suitable components that provide the described functionality.

Some of the external sensors130may also be considered part of the vehicle exterior voice assistant125. For example, the external sensors130may include an ultrawideband (UWB) detector305(FIG.3) or a Bluetooth low energy (BLE) detector. These or other known external sensors130can detect a distance to a person using the exterior voice assistant125based on a device140(e.g., keyfob, smartphone) carried by the person. The device140may also identify the person as a speaker authorized to communicate with the exterior voice assistant125. That is, an input signal145may be obtained and processed, as detailed with reference toFIG.2, only when the input signal145is a voice associated with the person carrying the device140. Other authorization mechanisms (e.g., voice biometric-based authorization by the exterior voice assistant) may be used alternatively and are not a part of the signal improvement procedure detailed herein. The processes discussed with reference toFIG.2assume that an authorized speaker provides the input signal145.

When multiple external sensors130(e.g., an array of UWB or BLE detectors305arranged around the vehicle100) are used, the location of the person (i.e., device140) relative to the vehicle100may be determined in addition to the distance. Otherwise, the array of microphones110may be used to determine location and orientation of the person with the device140. The determination of distance, location, and orientation may be implemented according to a number of known techniques and is further discussed with reference toFIG.3.

FIG.2is a process flow of a method200of performing context-aware signal conditioning for a vehicle exterior voice assistant125according to one or more embodiments. The processes of the method200may be performed following the detection of a device140and/or other processes to identify an authorized speaker. As indicated, the method200is performed iteratively while a voice input signal145is provided to the vehicle exterior voice assistant125by an authorized speaker. At block210, obtaining a voice input signal145may use one or more of the microphones110external to the vehicle100as shown inFIG.1, for example. As previously noted, obtaining the input signal145may include identifying the voice as being that of an authorized user (e.g., person with the proper device140such as a keyfob or smartphone) prior to the processes at block210.

At block220, the processes include obtaining the location and orientation of the source of the input signal145(i.e., the authorized speaker). The location may be determined prior to the processes at block210as part of the process of identifying the authorized speaker. That is, for example, the device140may be detected to trigger the authorization process. The location of the device140and, thus, the authorized speaker may be determined as part of this detection. Determining location and orientation of the person providing the input signal145(i.e., the authorized speaker) may involve an array of UWB and/or BLE detectors305, as further discussed with reference toFIG.3, or may involve the microphones110, a combination of the microphones110and UWB and/or BLE detectors305, or another known approach.

For example, an array of UWB and/or BLE detectors305may be among the external sensors130of the vehicle100. A time of flight for a signal from each UWB or BLE detector305to the device140held by the authorized speaker and back may be used to determine the distance between each UWB or BLE detector305of the array and the speaker. The relative distances to each of the UWB or BLE detectors305, in view of the location of each UWB or BLE detector305of the vehicle100, may then be used (e.g., by the controller120) to determine a location of the authorized speaker relative to the vehicle100using a geometric approach. The relative volume level measured at each of the microphones110may be used to determine orientation. The volume may be determined according to EQ. 4, as discussed herein. Alternately, cameras may be among the external sensors130and image processing may be performed to determine an orientation of the authorized speaker relative to the vehicle100.

At block230, characterizing the input signal145refers to obtaining several signal quality measures or, additionally or alternately, to obtaining a standard estimate of speech quality using a standardized methodology (e.g., P.563). As indicated, the location and orientation of the authorized speaker (obtained at block220) may be used to characterize the input signal at block230. For example, characterizing may include obtaining signal-to-noise ratio (SNR). According to an exemplary approach, noise may be estimated based on estimating and updating the background noise spectrum during pauses in the speech of the authorized speaker when the input signal145is not being provided. SNR may be calculated as the ratio of the power of the input signal145to the power of the background noise. Characterizing, at block230, may also include obtaining source-to-artifacts ratio (SAR) as follows:

Obtaining SAR according to EQ. 1 is a beamforming process in which Stargetis the allowed deformation of the input signal, einterfis the allowed deformation of sources other than the authorized speaker, enoiseis the allowed deformation of perturbating noise, and eartifmay correspond to artifacts of the beamforming algorithm such as musical noise, for example, or to deformations induced by the beamforming algorithm that are not allowed. Another exemplary characteristic includes source-to-distortion ratio (SDR) given by:

SDR=10⁢log10⁢starget2einterf+enoise+eartif2[EQ.2]
The SDR reflects the difference between the input signal from the authorized speaker and the estimated signal in the mean-square sense. Yet another exemplary characteristic is source-to-interference ratio (SIR) given by:

SIR=10⁢log10⁢starget2einterf2[EQ.3]
The SIR determines the ratio of energies of the input signal from the authorized speaker and the interference in the separated signal. Volume of the authorized speaker may be determined based on sound pressure level (SPL) given by:

Lp=20⁢log10(pp0)[EQ.4]
In EQ. 4, p is the root mean square sound pressure, p0is the reference sound pressure (e.g., reference sound pressure in air 20 micro Pascals), and SPL is in decibels (dB).

At block240, determining whether the authorized speaker can take action to improve the input signal and/or determining whether speech enhancement will help may both be done, in turn, and in either order. The order in which the determination is made may be based on a condition rather than always being the same. For example, if the signal strength of the input signal145is below a threshold value, then according to an exemplary embodiment, a determination of whether the authorized speaker can take action to improve the input signal145may be made first. Determining whether the authorized speaker can take action to improve the input signal145refers to determining whether to instruct the authorized speaker to take that action. This determination is detailed with reference toFIG.3.

According to an exemplary embodiment, a determination of whether speech enhancement will help may be made after it is determined that the authorized speaker cannot improve the input signal145. This determination uses the characterization (at block230) and may be based on implementing fuzzy logic, a probabilistic approach such as Bayesian probability, a Dempster-Shafer evidential decision-making approach, a statistical machine learning approach, or any other decision-making algorithm. The exemplary case of using fuzzy logic is further detailed with reference toFIG.4. A result of this determination is either that speech enhancement should not be undertaken (i.e., it will not help) or that speech enhancement should be undertaken (i.e., it will help). In the later case, the type of speech enhancement, among known speech enhancement techniques, that should be undertaken (at block260) may also be determined.

Providing guidance to the authorized speaker, at block250, is based on determining (at block240) that the authorized speaker can take action to improve the input signal145. This is further discussed with reference toFIG.3. Performing speech enhancement, at block260, is based on determining (at block240) that speech enhancement should be performed. As previously noted, the determination that speech enhancement should be performed may include a determination of what the speech enhancement should include. For example, speech enhancement may include known techniques such as denoising, de-reverberating, or jointly performing denoising, de-reverberation, and source separation.

Denoising refers to the process of reducing or removing the noise from the acoustic signals. Known denoising algorithms with different levels of complexity and efficiency include ideal channel selection or ideal binary mask, spectral subtractive, subspace, noise estimation and statistical-based methods. De-reverberation refers to addressing reverberation, which is multipath propagation of an acoustic signal from its source to a microphone110. If the authorized speaker is too close to a microphone110, the resulting reverberation is minimal and traditional de-reverberation techniques address noise. If the authorized speaker is too far from the microphone110, the result may be severe distortions including high levels of noise and reverberation. De-reverberation may be implemented using known acoustic echo cancellation (AEC) or known de-reverberation suppression techniques. Source separation refers to recovering an original speech signal from a convolutive mixture of speech signals. The known blind source separation (BSS) technique estimates an original signal through observed signals. Blind signal separation (i.e., blind beamforming), like BSS, exploits statistical characteristics (from block230).

FIG.3illustrates a determination, at block240(FIG.2), of whether an authorized speaker can take action to improve the input signal according to an exemplary embodiment. Continuing reference is made toFIGS.1and2. Two microphones110a,110b(generally referred to as110) are shown in locations of the vehicle100corresponding with locations of two UWB or BLE detectors305a,305b(generally referred to as305). A region310ais shown to correspond with the microphone110aand a region310bis shown to correspond with the microphone110b. Generally, each microphone110may have a corresponding region310. A speaking zone320is indicated for the authorized speaker who is shown carrying the device140. The speaking zone320may be a range of angles of orientation and a distance that originate at the orientation determined for the speaker (at block220).

Ideally, the speaking zone320may be fully within a region310associated with one of the microphones110. This would account for location and orientation. Then, determining whether volume should be increased by the authorized speaker is a straight-forward determination based on the SPL measured at the microphone110whose corresponding region310the speaker is within. Based on the location and orientation determined for the authorized speaker (at block220), the controller120may determine whether the speaking zone320of the authorized speaker is within the region310of any microphone110. If the speaking zone320is not within a region310of any microphone110, then the determination, at block240, may be that the authorized speaker can take action to improve the input signal145.

Then, at block250, audio guidance may be provided by an audio speaker115on or within the vehicle100. The specific guidance may be based on proximity of the authorized speaker to a microphone110and of the speaking zone320with a particular region310. That is, if the authorized speaker is at a distance greater than a threshold distance from any microphone110, the guidance may be for the authorized speaker to move closer to one particular microphone110. If the distance is within the threshold distance but the speaking zone320is not within a region310of the closest microphone110to the authorized speaker, then the guidance may be for the authorized speaker to change their orientation toward the closest microphone110In the exemplary case shown inFIG.3, the speaker may be asked to move closer to the driver-side mirror and to turn to face the mirror where the microphone110ais located so that the speaking zone320is within the region310acorresponding with the microphone110a.

FIG.4is a process flow of a determination, at block240of the method200ofFIG.2, of whether speech enhancement will improve the input signal145according to an exemplary embodiment. The exemplary approach shown inFIG.4is fuzzy logic. As previously noted, other exemplary embodiments for performing the determination at block240may use a Bayesian probability, a Dempster-Shafer evidential decision-making approach, a statistical machine learning approach, or any other decision-making algorithm. At block410, generating linguistic variables from the characteristics generated at block230may also be referred to as fuzzification. In fuzzy logic, fuzzification refers to converting a crisp quantity into a fuzzy quantity. That is, characteristics of the input signal (obtained at block230) such as SNR, SAR, and SDL are converted from values estimated according to formulas, such as EQS. 1-4, into fuzzy quantities (e.g., low, medium, high).

The mapping of the estimates received from block230to fuzzy quantities may be based on established ranges or rules. For example, each linguistic variable derived from each characteristic may be expressed as a quintuple (x, T(x), G, S). The variable name is x and T(x) is the set of linguistic values that the variable with the name x may have. U is the universe of discourse, and G is a syntactic rule that generates the terms in T(x). S is a semantic rule that associates each linguistic value with its meaning (i.e., a fuzzy set defined on U). The set of linguistic values associated with the variable estimated SNR may be {very low, low, medium, high, very high}, for example.

At block420, implementing an inference mechanism relies on a rules database430. The rules database430may include a rule for every combination of every linguistic value of every variable. For example, there may be only two characterizing variables from block230, SNR and noise source identification (NSI). SNR may be associated with five linguistic values {very low, low, medium, high, very high} corresponding to levels of SNR values, and NSI may also be associated with five linguistic values {high static, static, normal, non-static, very non-static} corresponding with types of NSI. In this exemplary case, the rules database430would include twenty-five rules associated with the twenty-five combinations of linguistic values for SNR and NSI. For example, one rule among the twenty-five may correspond to an SNR of “very low” and an NSI of “normal” while another rule corresponds to an SNR of “high” and an NSI of “static.”

The rule applied at block420may be based on the combination of linguistic values corresponding with the characteristics received from block230, as well as from block220(e.g., distance, orientation). The rule results in the output of a decision from block240. As discussed with reference toFIG.2, the decision may be to provide guidance to the speaker (at block250), to perform speech enhancement (at block260) and, according to alternate embodiments, to specify which speech enhancements to perform (e.g., denoising, de-reverberation).