Patent Description:
Recent advances in computing have raised the possibility of realizing many long sought-after voice-control applications. For example, improvements in statistical models, including practical frameworks for effective neural network architectures, have greatly increased the accuracy and reliability of previous speech processing systems. This has been coupled with a rise in wide area computer networks, which offer a range of modular services that can be simply accessed using application programming interfaces. Voice is quickly becoming a viable option for providing a user interface.

While voice control devices have become popular within the home, providing speech processing within vehicles presents additional challenges. For example, vehicles often have limited processing resources for auxiliary functions (such as voice interfaces), suffer from pronounced noise (e.g., high levels of road and/or engine noise) and present constraints in terms of the acoustic environment. Any user interface is furthermore constrained by the safety implications of controlling a vehicle. These factors have made within vehicle voice control difficult to achieve in practice.

Also, despite advances in speech processing, even users of advanced computing devices often report that current systems lack human-level responsiveness and intelligence. Translating pressure fluctuations in the air into parsed commands is incredibly difficult. Speech processing typically involves a complex processing pipeline, where errors at any stage can derail a successful machine interpretation. Many of these challenges are not immediately apparent to human beings, who are able to process speech using cortical and sub-cortical structures without conscious thought. Engineers working in the field, however, quickly become aware of the gap between human ability and state of the art speech processing.

<CIT> (Patent Document <NUM>) describes a combined lip reading and voice recognition multimodal interface system. The system can issue a navigation operation instruction only by voice and lip movements, thus allowing a driver to look ahead during a navigation operation and reducing vehicle accidents related to navigation operations during driving. The combined lip reading and voice recognition multimodal interface system described in <CIT> comprises an audio voice input unit; a voice recognition unit; a voice recognition instruction and estimated probability output unit; a lip video image input unit; a lip reading unit; a lip reading recognition instruction output unit; and a voice recognition and lip reading recognition result combining unit that outputs the voice recognition instruction. While <CIT> provides one solution for in vehicle control, the proposed system is complex and the many interoperating components present increased opportunity for error and parsing failure.

It is desired to provide speech processing systems and methods that more accurately transcribe and parse human utterances. It is further desired to provide speech processing methods that may be practically implemented with real world devices, such as embedded computing systems for vehicles. Implementing practical speech processing solutions is difficult as vehicles present many challenges for system integration and connectivity.

[Patent Document <NUM>] United Patent Specification No. <CIT>.

<CIT> describes systems and methods for performing focus detection, referential ambiguity resolution and mood classification in accordance with multi-modal input data, in varying operating conditions, in order to provide an effective conversational computing environment for one or more users.

The object of the invention is to overcome shortcomings of the prior art.

This object is achieved by the invention defined in the appended independent claims.

Preferred embodiments of the invention are set out in the appended dependent claims.

Certain examples described herein provide methods and systems for processing speech. Certain examples use both audio data and image data to process speech. Certain examples are adapted to address challenges of processing utterances that are captured within a vehicle. Certain examples obtain a speaker feature vector based on image data that features at least a facial area of a person, e.g., a person within the vehicle. Speech processing is then performed using vision-derived information that is dependent on a speaker of an utterance. This may improve accuracy and robustness.

The present invention provides an apparatus for a vehicle-mounted apparatus, as set out in independent claim <NUM>.

The present invention further provides a computer-implemented method of processing an utterance, as set out in independent claim <NUM>.

The present invention also provides a program, as set out in independent claim <NUM>.

The following describes various examples of the present technology that illustrate various interesting aspects. Generally, examples can use the described aspects in any combination.

Certain examples described herein use visual information to improve speech processing. This visual information may be obtained from within a vehicle. In examples, the visual information features a person within the vehicle, e.g., a driver or a passenger. Certain examples use the visual information to generate a speaker feature vector for use by an adapted speech processing module. The speech processing module may be configured to use the speaker feature vector to improve the processing of associated audio data, e.g., audio data derived from an audio capture device within the vehicle. The examples may improve the responsiveness and accuracy of in-vehicle speech interfaces. Certain examples may be used by computing devices to improve speech transcription. As such, described examples may be seen to extend speech processing systems with multi-modal capabilities that improve the accuracy and reliability of audio processing.

Certain examples described herein provide different approaches to generate a speaker feature vector. Certain approaches are complementary and may be used together to synergistically improve speech processing. In one example, image data obtained from within a vehicle, such as from a driver and/or passenger camera, is processed to identify a person and to determine a feature vector that numerically represents certain characteristics of the person. These characteristics may comprise audio characteristics, e.g., a numerical representation of expected variance within audio data for an acoustic model. In another example, image data obtained from within a vehicle, such as from a driver and/or passenger camera, is processed to determine a feature vector that numerically represents certain visual characteristics of the person, e.g., characteristics associated with an utterance by the person. In one case, the visual characteristics may be associated with a mouth area of the person, e.g., represent lip position and/or movement. In both examples, a speaker feature vector may have a similar format, and so be easily integrated into an input pipeline of an acoustic model that is used to generate phoneme data. Certain examples may provide improvements that overcome certain challenges of in-vehicle automatic speech recognition, such as a confined interior of a vehicle, a likelihood that multiple people may be speaking within this confined interior and high levels of engine and environmental noise.

<FIG> shows an example context for a speech processing apparatus. In <FIG>, the context is a motor vehicle. <FIG> is a schematic illustration of an interior <NUM> of a motor vehicle. The interior <NUM> is shown for a front driver side of the motor vehicle. A person <NUM> is shown within the interior <NUM>. In <FIG>, the person is a driver of the motor vehicle. The driver faces forward in the vehicle and observes a road through windshield <NUM>. The person controls the vehicle using a steering wheel <NUM> and observes vehicle status indications via a dashboard or instrument panel <NUM>. In <FIG>, an image capture device <NUM> is located within the interior <NUM> of the motor vehicle near the bottom of the dashboard <NUM>. The image capture device <NUM> has a field of view <NUM> that captures a facial area <NUM> of the person <NUM>. In this example, the image capture device <NUM> is positioned to capture an image through an aperture of the steering wheel <NUM>. <FIG> also shows an audio capture device <NUM> that is located within the interior <NUM> of the motor vehicle. The audio capture device <NUM> is arranged to capture sounds that are made by the person <NUM>. For example, the audio capture device <NUM> may be arranged to capture speech from the person <NUM>, i.e. sounds that are emitted from the facial area <NUM> of the person. The audio capture device <NUM> is shown mounted to the windshield <NUM>, e.g., it may be mounted near to or on a rear-view mirror, or be mounted on a door frame to a side of the person <NUM>. <FIG> also shows a speech processing apparatus <NUM>. The speech processing apparatus <NUM> may be mounted on the motor vehicle. The speech processing apparatus <NUM> may comprise, or form part of, a control system for the motor vehicle. In the example of <FIG>, the image capture device <NUM> and the audio capture device <NUM> are communicatively coupled to the speech processing apparatus <NUM>, e.g., via one or more wired and/or wireless interfaces. The image capture device <NUM> can be located outside the motor vehicle to capture an image within the motor vehicle through window glasses of the motor vehicle.

The context and configuration of <FIG> is provided as an example to aid understanding of the following description. It should be noted that the examples need not be limited to a motor vehicle but may be similarly implemented in other forms of vehicle including, but not limited to: nautical vehicles such as boats and ships; aerial vehicles such as helicopters, planes and gliders; railed vehicles such as trains and trams; spacecraft, construction vehicles and heavy equipment. Motor vehicles may include cars, trucks, sports utility vehicles, motorbikes, buses, and motorized carts, amongst others. Use of the term "vehicle" herein also includes certain heavy equipment that may be motorized while remaining static, such as cranes, lifting devices and boring devices. Vehicles may be manually controlled and/or have autonomous functions. Although the example of <FIG> features a steering wheel <NUM> and dashboard <NUM>, other control arrangements may be provided (e.g., an autonomous vehicle may not have a steering wheel <NUM> as depicted). Although a driver seat context is shown in <FIG>, a similar configuration may be provided for one or more passenger seats (e.g. both front and rear). <FIG> is provided for illustration only and omits certain features that may also be found within a motor vehicle for clarity. In certain cases, the approaches described herein may be used outside of a vehicle context, e.g., may be implemented by a computing device such as a desktop or laptop computer, a smartphone, or an embedded device.

<FIG> is a schematic illustration of the speech processing apparatus <NUM> shown in <FIG>. In <FIG>, the speech processing apparatus <NUM> comprises a speech processing module <NUM>, an image interface <NUM> and an audio interface <NUM>. The image interface <NUM> is configured to receive image data <NUM>. The image data may comprise image data captured by the image capture device <NUM> in <FIG>. The audio interface <NUM> is configured to receive audio data <NUM>. The audio data <NUM> may comprise audio data captured by the audio capture device <NUM> in <FIG>. The speech processing module <NUM> is communicatively coupled to both the image interface <NUM> and the audio interface <NUM>. The speech processing module <NUM> is configured to process the image data <NUM> and the audio data <NUM> to generate a set of linguistic features <NUM> that are useable to parse an utterance of the person <NUM>. The linguistic features may comprise phonemes, word portions (e.g., stems or proto-words), and words (including text features such as pauses that are mapped to punctuation), as well as probabilities and other values that relate to these linguistic units. In one case, the linguistic features may be used to generate a text output that represents the utterance. In this case, the text output may be used as-is or may be mapped to a predefined set of commands and/or command data. In another case, the linguistic features may be directly mapped to the predefined set of commands and/or command data (e.g. without an explicit text output).

A person (such as person <NUM>) may use the configuration of <FIG> to issue voice commands while operating the motor vehicle. For example, the person <NUM> may speak within the interior, e.g., generate an utterance, in order to control the motor vehicle or obtain information. An utterance in this context is associated with a vocal sound produced by the person that represents linguistic information such as speech. For example, an utterance may comprise speech that emanates from a larynx of the person <NUM>. The utterance may comprise a voice command, e.g., a spoken request from a user. The voice command may comprise, for example: a request to perform an action (e.g., "Play music", "Turn on air conditioning", "Activate cruise control"); further information relating to a request (e.g., "Album XY", "<NUM> degrees Fahrenheit", "<NUM> mph for <NUM> minutes"); speech to be transcribed (e.g., "Add to my to do list. " or "Send the following message to user A. "); and/or a request for information (e.g., "What is the traffic like on C?", "What is the weather like today?", or "Where is the nearest gas station?").

The audio data <NUM> may take a variety of forms depending on the implementation. In general, the audio data <NUM> may be derived from time series measurements from one or more audio capture devices (e.g., one or more microphones), such as audio capture device <NUM> in <FIG>. In certain cases, the audio data <NUM> may be captured from one audio capture device; in other cases, the audio data <NUM> may be captured from multiple audio capture devices, e.g., there may be multiple microphones at different positions within the interior <NUM>. In the latter case, the audio data may comprise one or more channels of temporally correlated audio data from each audio capture device. Audio data at the point of capture may comprise, for example, one or more channels of Pulse Code Modulation (PCM) data at a predefined sampling rate (e.g., <NUM>), where each sample is represented by a predefined number of bits (e.g., <NUM>, <NUM> or <NUM> bits per sample - where each sample comprises an integer or float value).

In certain cases, the audio data <NUM> may be processed after capture but before receipt at the audio interface <NUM> (e.g., preprocessed with respect to speech processing). Processing may comprise one or more of filtering in one or more of the time and frequency domains, applying noise reduction, and/or normalization. In one case, audio data may be converted into measurements over time in the frequency domain, e.g., by performing the Fast Fourier Transform to create one or more frames of spectrogram data. In certain cases, filter banks may be applied to determine values for one or more frequency domain features, such as Mel filter banks or Mel-Frequency Cepstral Coefficients. In these cases, the audio data <NUM> may comprise an output of one or more filter banks. In other cases, audio data <NUM> may comprise time domain samples and preprocessing may be performed within the speech processing module <NUM>. Different combinations of approach are possible. Audio data as received at the audio interface <NUM> may thus comprise any measurement made along an audio processing pipeline.

In a similar manner to the audio data, the image data described herein may take a variety of forms depending on the implementation. In one case, the image capture device <NUM> may comprise a video capture device, wherein the image data comprises one or more frames of video data. In another case, the image capture device <NUM> may comprise a static image capture device, wherein the image data comprises one or more frames of static images. Hence, the image data may be derived from both video and static sources. Reference to image data herein may relate to image data derived, for example, from a two-dimensional array having a height and a width (e.g., equivalent to rows and columns of the array). In one case, the image data may have multiple color channels, e.g., comprise three color channels for each of the colors Red Green Blue (RGB), where each color channel has an associated two-dimensional array of color values (e.g., at <NUM>, <NUM> or <NUM> bits per array element). Color channels may also be referred to as different image "planes". In certain cases, only a single channel may be used, e.g., representing a "gray" or lightness channel. Different color spaces may be used depending on the application, e.g., an image capture device may natively generate frames of YUV image data featuring a lightness channel Y (e.g., luminance) and two opponent color channels U and V (e.g., two chrominance components roughly aligned with blue-green and red-green). As with the audio data <NUM>, the image data <NUM> may be processed following capture, e.g., one or more image filtering operations may be applied and/or the image data <NUM> may be resized and/or cropped.

With reference to the example of <FIG>, one or more of the image interface <NUM> and the audio interface <NUM> may be local to hardware within the motor vehicle. For example, each of the image interface <NUM> and the audio interface <NUM> may comprise a wired coupling of respective image and audio capture devices to at least one processor configured to implement the speech processing module <NUM>. In one case, the image and audio interfaces <NUM>, <NUM> may comprise a serial interface over which image and audio data <NUM>, <NUM> may be received. In a distributed vehicle control system, the image and audio capture devices <NUM>, <NUM> may be communicatively coupled to a central systems bus, wherein image and audio data <NUM>, <NUM> may be stored in one or more storage devices (e.g., Random Access Memory or solid-state storage). In this latter case, the image and audio interfaces <NUM>, <NUM> may comprise a communicative coupling of the at least one processor configured to implement the speech processing module to the one or more storage devices, e.g., the at least one processor may be configured to read data from a given memory location to access each of the image and audio data <NUM>, <NUM>. In certain cases, the image and audio interfaces <NUM>, <NUM> may comprise wireless interfaces, wherein the speech processing module <NUM> may be remote from the motor vehicle. Different approaches and combinations are possible.

Although <FIG> shows an example where the person <NUM> is a driver of a motor vehicle, in other applications, one or more image and audio capture devices may be arranged to capture image data featuring a person that is not controlling the motor vehicle such as a passenger. For example, a motor vehicle may have a plurality of image capture devices that are arranged to capture image data relating to people present in one or more passenger seats of the vehicle (e.g., at different locations within the vehicle such as front and back). Audio capture devices may also be likewise arranged to capture utterances from different people, e.g., a microphone may be located in each door or door frame of the vehicle. In one case, a plurality of audio capture devices may be provided within the vehicle and audio data may be captured from one or more of these for the supply of data to the audio interface <NUM>. In one case, preprocessing of audio data may comprise selecting audio data from a channel that is deemed to be closest to a person making an utterance and/or combining audio data from multiple channels within the motor vehicle. As described later, certain examples described herein facilitate speech processing in a vehicle with multiple passengers.

<FIG> shows an example speech processing apparatus <NUM>. For example, the speech processing apparatus <NUM> may be used to implement the speech processing apparatus <NUM> shown in <FIG>. The speech processing apparatus <NUM> may form part of an in-vehicle automatic speech recognition system. In other cases, the speech processing apparatus <NUM> may be adapted for use outside of a vehicle, such as in the home or in an office.

The speech processing apparatus <NUM> comprises a speaker preprocessing module <NUM> and a speech processing module <NUM>. The speech processing module <NUM> may be similar to the speech processing module <NUM> of <FIG>. In this example, the image interface <NUM> and the audio interface <NUM> have been omitted for clarity; however, these may respectively form part of the image input of the speaker preprocessing module <NUM> and the audio input for the speech processing module <NUM>. The speaker preprocessing module <NUM> is configured to receive image data <NUM> and to output a speaker feature vector <NUM>. The speech processing module <NUM> is configured to receive audio data <NUM> and the speaker feature vector <NUM> and to use these to generate linguistic features <NUM>.

The speech processing module <NUM> is implemented by a processor. The processor may be a processor of a local embedded computing system within a vehicle and/or a processor of a remote server computing device (a so-called "cloud" processing device). In one case, the processor may comprise part of dedicated speech processing hardware, e.g., one or more Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) and so-called "system on chip" (SoC) components. In another case, the processor may be configured to process computer program code, e.g., firmware or the like, stored within an accessible storage device and loaded into memory for execution by the processor. The speech processing module <NUM> is configured to parse an utterance of a person, e.g., person <NUM>, based on the audio data <NUM> and the image data <NUM>. In the present case, the image data <NUM> is preprocessed by the speaker preprocessing module <NUM> to generate the speaker feature vector <NUM>. Similar to the speech processing module <NUM>, the speaker preprocessing module <NUM> may be any combination of hardware and software. In one case, the speaker preprocessing module <NUM> and the speech processing module <NUM> may be implemented on a common embedded circuit board for a vehicle.

In one case, the speech processing module <NUM> comprises an acoustic model configured to process the audio data <NUM> and to predict phoneme data for use in parsing the utterance. In this case, the linguistic features <NUM> may comprise phoneme data. The phoneme data may relate to one or more phoneme symbols, e.g., from a predefined alphabet or dictionary. In one case, the phoneme data may comprise a predicted sequence of phonemes; in another case, the phoneme data may comprise probabilities for one or more of a set of phoneme components, e.g., phoneme symbols and/or sub-symbols from the predefined alphabet or dictionary, and a set of state transitions (e.g., for a Hidden Markov Model). The acoustic model may be configured to receive audio data in the form of an audio feature vector. The audio feature vector may comprise numeric values representing one or more of Mel Frequency Cepstral Coefficients (MFCCs) and Filter Bank outputs. In certain cases, the audio feature vector may relate to a current window within time (often referred to as a "frame") and comprise differences relating to changes in features between the current window and one or more other windows in time (e.g., previous windows). The current window may have a width within a w millisecond range, e.g. in one case w may be around <NUM> milliseconds. Other features may comprise signal energy metrics and an output of logarithmic scaling, amongst others. The audio data <NUM>, following preprocessing, may comprise a frame (e.g. a vector) of a plurality of elements (e.g. from <NUM> to over <NUM> elements), each element comprising a numeric representation associated with a particular audio feature. In certain examples, there may be around <NUM>-<NUM> Mel filter bank features, a similar sized set of intra features, a similar sized set of delta features (e.g., representing a first-order derivative), and a similar sized set of double delta features (e.g., representing a second-order derivative).

The speaker preprocessing module <NUM> may be configured to obtain the speaker feature vector <NUM> in a number of different ways. In one case, the speaker preprocessing module <NUM> may obtain at least a portion of the speaker feature vector <NUM> from a memory, e.g., via a look-up operation. In one case, a portion of the speaker feature vector <NUM> comprising an i and/or x vector as set out below may be retrieved from memory. In this case, the image data <NUM> may be used to determine a particular speaker feature vector <NUM> to retrieve from the memory. For example, the image data <NUM> may be classified by the speaker preprocessing module <NUM> to select one particular user from a set of registered users. The speaker feature vector <NUM> in this case may comprise a numeric representation of features that are correlated with the selected particular user. In another case, the speaker preprocessing module <NUM> may compute the speaker feature vector <NUM>. For example, the speaker preprocessing module <NUM> may compute a compressed or dense numeric representation of salient information within the image data <NUM>. This may comprise a vector having a number of elements that is smaller in size than the image data <NUM>. The speaker preprocessing module <NUM> in this case may implement an information bottleneck to compute the speaker feature vector <NUM>. In one case, the computation is determined based on a set of parameters, such as a set of weights, biases and/or probability coefficients. Values for these parameters may be determined via a training phase that uses a set of training data. In one case, the speaker feature vector <NUM> may be buffered or stored as a static value following a set of computations. In this case, the speaker feature vector <NUM> may be retrieved from a memory on a subsequent utterance based on the image data <NUM>. Further examples explaining how a speaker feature vector may be computed are set out below. In a case, where the speaker feature vector <NUM> comprises a component that relates to lip movement, this component may be provided on a real-time or near real-time basis and may not be retrieved from data storage.

In one case, a speaker feature vector <NUM> may comprise a fixed length one-dimensional array (e.g., a vector) of numeric values, e.g., one value for each element of the array. In other cases, the speaker feature vector <NUM> may comprise a multi-dimensional array, e.g. with two or more dimensions representing multiple one-dimensional arrays. The numeric values may comprise integer values (e.g., within a range set by a particular bit length - <NUM> bits giving a range of <NUM> to <NUM>) or floating-point values (e.g., defined as <NUM>-bit or <NUM>-bit floating point values). Floating-point values may be used if normalization is applied to the visual feature tensor, e.g., if values are mapped to a range of <NUM> to <NUM> or -<NUM> to <NUM>. The speaker feature vector <NUM>, as an example, may comprise a <NUM>-element array, where each element is an <NUM> or <NUM>-bit value, although the form may vary based on the implementation. In general, the speaker feature vector <NUM> has an information content that is less than a corresponding frame of image data, e.g., using the aforementioned example, a speaker feature vector <NUM> of length <NUM> with <NUM>-bit values is smaller than a <NUM> by <NUM> video frame having <NUM> channels of <NUM>-bit values - <NUM> bits vs <NUM> bits. Information content may be measured in bits or in the form of an entropy measurement.

In one case, the speech processing module <NUM> comprises an acoustic model and the acoustic model comprises a neural network architecture. For example, the acoustic model may comprise one or more of: a Deep Neural Network (DNN) architecture with a plurality of hidden layers; a hybrid model comprising a neural network architecture and one or more of a Gaussian Mixture Model (GMM) and a Hidden Markov Model (HMM); and a Connectionist Temporal Classification (CTC) model, e.g., comprising one or more recurrent neural networks that operates over sequences of inputs and generates sequences of linguistic features as an output. The acoustic model may output predictions at a frame level (e.g., for a phoneme symbol or sub-symbol) and use previous (and in certain cases future) predictions to determine a possible or most likely sequence of phoneme data for the utterance. Approaches such as beam search and the Viterbi algorithm may be used on an output end of the acoustic model to further determine the sequence of phoneme data that is output from the acoustic model. Training of the acoustic model may be performed time step by time step.

In the case that the speech processing module <NUM> comprises an acoustic model and the acoustic model comprises a neural network architecture (e.g., is a "neural" acoustic model), the speaker feature vector <NUM> may be provided as an input to the neural network architecture together with the audio data <NUM>. The speaker feature vector <NUM> and the audio data <NUM> may be combined in a number of ways. In a simple case, the speaker feature vector <NUM> and the audio data <NUM> may be concatenated into a longer combined vector. In another case, different input preprocessing may be performed on each of the speaker feature vector <NUM> and the audio data <NUM>, e.g., one or more attention, feed-forward and/or embedding layers may be applied and then the result of these layers combined. Different sets of layers may be applied to the different inputs. In other cases, the speech processing module <NUM> may comprise another form of statistical model, e.g., a probabilistic acoustic model, wherein the speaker feature vector <NUM> comprises one or more numeric parameters (e.g., probability coefficients) to configure the speech processing module <NUM> for a particular speaker.

The example speech processing apparatus <NUM> provides improvements for speech processing within a vehicle. Within a vehicle there may be high levels of ambient noise, such as road and engine noise. There may also be acoustic distortions caused by the enclosed interior space of the motor vehicle. These factors may make it difficult to process audio data in comparative examples, e.g., the speech processing module <NUM> may fail to generate linguistic features <NUM> and/or generate poorly matching sequences of linguistic features <NUM>. However, the arrangement of <FIG> allows the speech processing module <NUM> to be configured or adapted based on speaker features determined based on the image data <NUM>. This provides additional information to the speech processing module <NUM> such that it may select linguistic features that are consistent with a particular speaker, e.g., by exploiting correlations between appearance and acoustic characteristics. These correlations may be long-term temporal correlations such as general facial appearance and/or short-term temporal correlations such as particular lip and mouth positions. This leads to greater accuracy despite the challenging noise and acoustic context. This may help reduce utterance parsing errors, e.g., improve an end-to-end transcription path, and/or improve the audio interface for performing voice commands. In certain cases, the present example may be able to take advantage of an existing driver-facing camera that is normally configured to monitor the driver to check for drowsiness and/or distraction. In certain cases, there may be a speaker dependent feature vector component that is retrieved based on a recognized speaker and/ a speaker dependent feature vector component that comprises mouth movement features. The latter component may be determined based on a function that is not configured for individual users, e.g. a common function for all users may be applied, but the mouth movement would be associated with a speaker. In certain other cases, the extraction of mouth movement features may be configured based on a particular identified user.

<FIG> shows an example speech processing apparatus <NUM>. The speech processing apparatus <NUM> shows additional components that may be used to implement the speaker preprocessing module <NUM> in <FIG>. Certain components shown in <FIG> are similar to their counterparts shown in <FIG> and have similar reference numerals. The features described above with reference to <FIG> may also apply to the example <NUM> of <FIG>. Like the example speech processing apparatus <NUM> of <FIG>, the example speech processing apparatus <NUM> of <FIG> comprises a speaker preprocessing module <NUM> and a speech processing module <NUM>. The speech processing module <NUM> receives audio data <NUM> and a speaker feature vector <NUM> and computes a set of linguistic features <NUM>. The speech processing module <NUM> may be configured in a similar manner to the examples described above with reference to <FIG>.

In <FIG>, a number of subcomponents of the speaker preprocessing module <NUM> are shown. These comprise a face recognition module <NUM>, a vector generator <NUM> and a data store <NUM>. Although these are shown as subcomponents of the speaker preprocessing module <NUM> in <FIG>, in other examples they may be implemented as separate components. In the example of <FIG>, the speaker preprocessing module <NUM> receives image data <NUM> that features a facial area of a person. The person may comprise a driver or passenger in a vehicle as described above. The face recognition module <NUM> performs facial recognition on the image data to identify the person, e.g., the driver or passenger within the vehicle. The face recognition module <NUM> may comprise any combination of hardware and software to perform the facial recognition. In one case, the face recognition module <NUM> may be implemented using an off-the-shelf hardware component such as a B5T-<NUM> supplied by Omron Electronics Inc. In the present example, the face recognition module <NUM> detects a user based on the image data <NUM> and outputs a user identifier <NUM>. The user identifier <NUM> is passed to the vector generator <NUM>. The vector generator <NUM> uses the user identifier <NUM> to obtain a speaker feature vector <NUM> associated with the identified person. In certain cases, the vector generator <NUM> may retrieve the speaker feature vector <NUM> from the data store <NUM>. The speaker feature vector <NUM> is then passed to the speech processing module <NUM> for use as described with reference to <FIG>.

In the example of <FIG>, the vector generator <NUM> may obtain the speaker feature vector <NUM> in different ways depending on a set of operating parameters. In one case, the operating parameters comprise a parameter that indicates whether a particular number of speaker feature vectors <NUM> have been computed for a particular identified user (e.g., as identified by the user identifier <NUM>). In one case, a threshold is defined that is associated with a number of previously computed speaker feature vectors. If this threshold is <NUM>, then the speaker feature vector <NUM> may be computed for a first utterance and then stored in the data store <NUM>; for subsequent utterances the speaker feature vector <NUM> may be retrieved from the data store <NUM>. If the threshold is greater than <NUM>, such as n, then n speaker feature vectors <NUM> may be generated and then the (n+<NUM>)th speaker feature vector <NUM> may be obtained as a composite function of the previous n speaker feature vectors <NUM> as retrieved from the data store <NUM>. The composite function may comprise an average or an interpolation. In one case, once the (n+<NUM>)th speaker feature vector <NUM> is computed, it is used as a static speaker feature vector for a configurable number of future utterances.

In the example above, the use of the data store <NUM> to save a speaker feature vector <NUM> may reduce run-time computational demands for an in-vehicle system. For example, the data store <NUM> may comprise a local data storage device within the vehicle, and as such a speaker feature vector <NUM> may be retrieved for a particular user from the data store <NUM> rather than being computed by the vector generator <NUM>.

In one case, at least one computation function used by the vector generator <NUM> may involve a cloud processing resource (e.g., a remote server computing device). In this case, in situations of limited connectivity between a vehicle and a cloud processing resource, the speaker feature vector <NUM> may be retrieved as a static vector from local storage rather than relying on any functionality that is provided by the cloud processing resource.

In one case, the speaker preprocessing module <NUM> may be configured to generate a user profile for each newly recognized person within the vehicle. For example, prior to, or on detection of an utterance, e.g., as captured by an audio capture device, the face recognition module <NUM> may attempt to match image data <NUM> against previously observed faces. If no match is found, then the face recognition module <NUM> may generate (or instruct the generation of) a new user identifier <NUM>. In one case, a component of the speaker preprocessing module <NUM>, such as the face recognition module <NUM> or the vector generator <NUM>, may be configured to generate a new user profile if no match is found, where the new user profile may be indexed using the new user identifier. Speaker feature vectors <NUM> may then be associated with the new user profile, and the new user profile may be stored in the data store <NUM> ready to be retrieved when future matches are made by the face recognition module <NUM>. As such an in-vehicle image capture device may be used for facial recognition to select a user-specific speech recognition profile. User profiles may be calibrated through an enrollment process, such as when a driver first uses the car, or may be learnt based on data collected during use.

In one case, the speaker processing module <NUM> may be configured to perform a reset of data store <NUM>. At manufacturing time, device <NUM> may be empty of user profile information. During usage, new user profiles may be created and added to the data store <NUM> as described above. A user may command a reset of stored user identifiers. In some cases, the reset may be performed only during professional service, such as when an automobile is maintained at a service shop or sold through a certified dealer. In some cases, the reset may be performed at any time through a user provided password.

In an example where the vehicle comprises multiple image capture devices and multiple audio capture devices, the speaker preprocessing module <NUM> may provide further functionality to determine an appropriate facial area from one or more captured images. In one case, audio data from a plurality of audio capture devices may be processed to determine a closest audio capture device associated with the utterance. In this case, a closest image capture device associated with the determined closest audio capture device may be selected and image data <NUM> from this device may be sent to the face recognition module <NUM>. In another case, the face recognition module <NUM> may be configured to receive multiple images from multiple image capture devices, where each image comprises an associated flag to indicate whether it is to be used to identify a currently speaking user. In this manner, the speech processing apparatus <NUM> of <FIG> may be used to identify a speaker from a plurality of people within a vehicle and configure the speech processing module <NUM> to the specific characteristics of that speaker. This may also improve speech processing within a vehicle in a case where multiple people are speaking within a constrained interior of the vehicle.

In certain examples described herein, a speaker feature vector, such as speaker feature vector <NUM> or <NUM> may comprise data that is generated based on the audio data, e.g., audio data <NUM> or <NUM> in <FIG> and <FIG>. This is shown by the dashed line in <FIG>. In one case, at least a portion of the speaker feature vector may comprise a vector generated based on factor analysis. In this case, an utterance may be represented as a vector M that is a linear function of one or more factors. The factors may be combined in a linear and/or a non-linear model. One of these factors may comprise a speaker and session independent supervector m. This may be based on a Universal Background Model (UBM). Another one of these factors may comprise a speaker-dependent vector w. This latter factor may also be dependent on a channel or session or a further factor may be provided that is dependent on the channel and/or the session. In one case, the factor analysis is performed using a Gaussian Mixture Model (GMM). In a simple case, a speaker utterance may be represented by a supervector M that is determined as M = m + Tw, where T is a matrix defining at least a speaker subspace. The speaker-dependent vector w may have a plurality of elements with floating point values. The speaker feature vector in this case may be based on the speaker-dependent vector w. One method of computing w, which is sometimes referred to as an "i-vector", is described by<NPL>. In certain examples, at least a portion of the speaker feature vector comprises at least portions of an i-vector. The i-vector may be seen to be a speaker dependent vector that is determined for an utterance from the audio data.

In the example of <FIG>, the vector generator <NUM> may compute an i-vector for one or more utterances. In a case, where there are no speaker feature vectors stored within data store <NUM>, an i-vector may be computed by the vector generator <NUM> based on one or more frames of audio data for an utterance <NUM>. In this example, the vector generator <NUM> may repeat the per-utterance (e.g., per voice query) i-vector computation until a threshold number of computations have been performed for a particular user, e.g., as identified using the user identifier <NUM> determined from the face recognition module <NUM>. In this case, after a particular user has been identified based on the image data <NUM>, the i-vector for the user for each utterance is stored in the data store <NUM>. The i-vector is also used to output the speaker feature vector <NUM>. Once the threshold number of computations have been performed, e.g., <NUM> or so i-vectors have been computed, the vector generator <NUM> may be configured to compute a profile for the particular user using the i-vectors that are stored in the data store <NUM>. The profile may use the user identifier <NUM> as an index and may comprise a static (e.g., non-changing) i-vector that is computed as a composite function of the stored i-vectors. The vector generator <NUM> may be configured to compute the profile on receipt of a (n+<NUM>)th query or as part of a background or periodic function. In one case, a static i-vector may be computed as an average of the stored i-vectors. Once the profile is generated by the vector generator <NUM> and stored in the data store <NUM>, e.g., using the user identifier to associate the profile with the particular user, then it may be retrieved from the data store <NUM> and used for future utterance parsing in place of computation of the i-vector for the user. This can reduce the computation overhead of generating the speaker feature vector and reduce i-vector variance.

In certain examples, the speaker feature vector, such as speaker feature vector <NUM> or <NUM> may be computed using a neural network architecture. For example, in one case, the vector generator <NUM> of the speaker preprocessing module <NUM> of <FIG> may comprise a neural network architecture. In this case, the vector generator <NUM> may compute at least a portion of the speaker feature vector by reducing the dimensionality of the audio data <NUM>. For example, the vector generator <NUM> may comprise one or more Deep Neural Network layers that are configured to receive one or more frames of audio data <NUM> and output a fixed length vector output (e.g., one vector per language). One or more pooling, non-linear functions and SoftMax layers may also be provided. In one case, the speaker feature vector may be generated based on an x-vector as described by <NPL>).

An x-vector may be used in a similar manner to the i-vector described above, and the above approaches apply to a speaker feature vector generated using x-vectors as well as i-vectors. In one case, both i-vectors and x-vectors may be determined, and the speaker feature vector may comprise a supervector comprising elements from both an i-vector and an x-vector. As both i-vectors and x-vectors comprise numeric elements, e.g., typically floating-point values and/or values normalized within a given range, that may be combined by concatenation or a weighted sum. In this case, the data store <NUM> may comprise stored values for one or more of i-vectors and x-vectors, whereby
once a threshold is reached a static value is computed and stored with a particular user identifier for future retrieval. In one case, interpolation may be used to determine a speaker feature vector from one or more i-vectors and x-vectors. In one case, interpolation may be performed by averaging different speaker feature vectors from the same vector source.

In the case where the speech processing module comprises a neural acoustic model, a fixed-length format for the speaker feature vector may be defined. The neural acoustic model may then be trained using the defined speaker feature vector, e.g., as determined by the speaker preprocessing module <NUM> or <NUM> in <FIG> and <FIG>. If the speaker feature vector comprises elements derived from one or more of i-vector and x-vector computations, then the neural acoustic model may "learn" to configure acoustic processing based on speaker specific information that is embodied or embedded within the speaker feature vector. This may increase acoustic processing accuracy, especially within a vehicle such as a motor vehicle. In this case, the image data provides a mechanism to quickly associate a particular user with computed or stored vector elements.

<FIG> shows an example speech processing module <NUM>. The speech processing module <NUM> may be used to implement the speech processing modules <NUM>, <NUM> or <NUM> in <FIG>, <FIG> and <FIG>. In other examples, other speech processing module configurations may be used.

As per the previous examples, the speech processing module <NUM> receives audio data <NUM> and a speaker feature vector <NUM>. The audio data <NUM> and the speaker feature vector <NUM> may be configured as per any of the examples described herein. In the example of <FIG>, the speech processing module <NUM> comprises an acoustic model <NUM>, a language model <NUM> and an utterance parser <NUM>. As described previously, the acoustic model <NUM> generates phoneme data <NUM>. Phoneme data may comprise one or more predicted sequences of phoneme symbols or sub-symbols, or other forms of proto-language units. In certain cases, multiple predicted sequences may be generated together with probability data indicating a likelihood of particular symbols or sub-symbols at each time step.

The phoneme data <NUM> is communicated to the language model <NUM>, e.g., the acoustic model <NUM> is communicatively coupled to the language model <NUM>. The language model <NUM> is configured to receive the phoneme data <NUM> and generate a transcription <NUM>. The transcription <NUM> may comprise text data, e.g., a sequence of characters, word-portions (e.g., stems, endings and the like) or words. The characters, word-portions and words may be selected from a predefined dictionary, e.g., a predefined set of possible outputs at each time step. In certain cases, the phoneme data <NUM> may be processed before passing to the language model <NUM> or may be pre-processed by the language model <NUM>. For example, beam forming may be applied to probability distributions (e.g. for phonemes) that are output from the acoustic model <NUM>.

The language model <NUM> is communicatively coupled to an utterance parser <NUM>. The utterance parser <NUM> receives the transcription <NUM> and uses this to parse the utterance. In certain cases, the utterance parser <NUM> generates utterance data <NUM> as a result of parsing the utterance. The utterance parser <NUM> may be configured to determine a command, and/or command data, associated with the utterance based on the transcription. In one case, the language model <NUM> may generate multiple possible text sequences, e.g., with probability information for units within the text, and the utterance parser <NUM> may be configured to determine a finalized text output, e.g., in the form of ASCII or Unicode character encodings, or a spoken command or command data. If the transcription <NUM> is determined to contain a voice command, the utterance parser <NUM> may be configured to execute, or instruct execution of, the command according to the command data. This may result in response data that is output as utterance data <NUM>. Utterance data <NUM> may comprise a response to be relayed to the person speaking the utterance, e.g., command instructions to provide an output on the dashboard <NUM> and/or via an audio system of the vehicle. In certain cases, the language model <NUM> may comprise a statistical language model and the utterance parser <NUM> may comprise a separate "meta" language model configured to rescore alternate hypotheses as output by the statistical language model. This may be via an ensemble model that uses voting to determine a final output, e.g., a final transcription or command identification.

<FIG> shows with a solid line an example where the acoustic model <NUM> receives the speaker feature vector <NUM> and the audio data <NUM> as an input and uses the input to generate the phoneme data <NUM>. For example, the acoustic model <NUM> may comprise a neural network architecture (including hybrid models with other non-neural components) and the speaker feature vector <NUM> and the audio data <NUM> may be provided as an input to the neural network architecture, wherein the phoneme data <NUM> is generated based on an output of the neural network architecture.

The dashed lines in <FIG> show additional couplings that may be configured in certain implementations. In a first case, the speaker feature vector <NUM> may be accessed by one or more of the language model <NUM> and the utterance parser <NUM>. For example, if the language model <NUM> and the utterance parser <NUM> also comprise respective neural network architectures, these architectures may be configured to receive the speaker feature vector <NUM> as an additional input, e.g., in addition to the phoneme data <NUM> and the transcription <NUM> respectively. If the utterance data <NUM> comprises a command identifier and one or more command parameters, the complete speech processing module <NUM> may be trained in an end-to-end manner given a training set with ground truth outputs and training samples for the audio data <NUM> and the speaker feature vector <NUM>.

In a second implementation, the speech processing module <NUM> of <FIG> may comprise one or more recurrent connections. In one case, the acoustic model may comprise recurrent models, e.g. LSTMs. In other cases, there may be feedback between modules. In <FIG> there is a dashed line indicating a first recurrent coupling between the utterance parser <NUM> and the language model <NUM> and a dashed line indicating a second recurrent coupling between the language model <NUM> and the acoustic model <NUM>. In this second case, a current state of the utterance parser <NUM> may be used to configure a future prediction of the language model <NUM> and a current state of the language model <NUM> may be used to configure a future prediction of the acoustic model <NUM>. The recurrent coupling may be omitted in certain cases to simplify the processing pipeline and allow for easier training. In one case, the recurrent coupling may be used to compute an attention or weighting vector that is applied at a next time step.

<FIG> shows an example speech processing apparatus <NUM> that uses a neural speaker preprocessing module <NUM> and a neural speech processing module <NUM>. In <FIG>, the speaker preprocessing module <NUM>, which may implement modules <NUM> or <NUM> in <FIG> and <FIG>, comprises a neural network architecture <NUM>. In <FIG>, the neural network architecture <NUM> is configured to receive image data <NUM>. In other cases, the neural network architecture <NUM> may also receive audio data, such as audio data <NUM>, e.g., as shown by the dashed pathway in <FIG>. In these other cases, the vector generator <NUM> of <FIG> may comprise the neural network architecture <NUM>.

In <FIG>, the neural network architecture <NUM> comprises at least a convolutional neural architecture. In certain architectures there may be one or more feed-forward neural network layers between a last convolutional neural network layer and an output layer of the neural network architecture <NUM>. The neural network architecture <NUM> may comprise an adapted form of the AlexNet, VGGNet, GoogLeNet, or ResNet architectures. The neural network architecture <NUM> may be replaced in a modular manner as more accurate architectures become available.

The neural network architecture <NUM> outputs at least one speaker feature vector <NUM>, where the speaker feature vector may be derived and/or used as described in any of the other examples. <FIG> shows a case where the image data <NUM> comprises a plurality of frames, e.g., from a video camera, wherein the frames feature a facial area of a person. In this case, a plurality of speaker feature vectors <NUM> may be computed using the neural network architecture, e.g., one for each input frame of image data. In other cases, there may be a many-to-one relationship between frames of input data and a speaker feature vector. It should be noted that using recurrent neural network systems, samples of the input image data <NUM> and the output speaker feature vectors <NUM> need not be temporally synchronized, e.g., a recurrent neural network architecture may act as an encoder (or integrator) over time. In one case, the neural network architecture <NUM> may be configured to generate an x-vector as described above. In one case, an x-vector generator may be configured to receive image data <NUM>, to process this image data using a convolutional neural network architecture and then to combine the output of the convolutional neural network architecture with an audio-based x-vector. In one case, known x-vector configurations may be extended to receive image data as well as audio data and to generate a single speaker feature vector that embodies information from both modal pathways.

In <FIG>, the neural speech processing module <NUM> is a speech processing module such as one of modules <NUM>, <NUM>, <NUM> that comprises a neural network architecture. For example, the neural speech processing module <NUM> may comprise a hybrid DNN-HMM/GMM system and/or a fully neural CTC system. In <FIG>, the neural speech processing module <NUM> receives frames of audio data <NUM> as input. Each frame may correspond to a temporal window, e.g., a window of w ms that is passed over time series data from an audio capture device. The frames of audio data <NUM> may be asynchronous with the frames of image data <NUM>, e.g., it is likely that the frames of audio data <NUM> will have a higher frame rate. Again, holding mechanisms and/or recurrent neural network architectures may be applied within the neural speech processing module <NUM> to provide temporal encoding and/or integration of samples. As in other examples, the neural speech processing module <NUM> is configured to process the frames of audio data <NUM> and the speaker feature vectors <NUM> to generate a set of linguistic features. As discussed herein, reference to a neural network architecture includes one or more neural network layers (in one case, "deep" architectures with one or more hidden layers and a plurality of layers), wherein each layer may be separated from a following layer by non-linearities such as tanh units or REctified Linear Units (RELUs). Other functions may be embodied within the layers including pooling operations.

The neural speech processing module <NUM> may comprise one or more components as shown in <FIG>. For example, the neural speech processing module <NUM> may comprise an acoustic model comprising at least one neural network. In the example of <FIG>, the neural network architectures of the neural speaker preprocessing module <NUM> and the neural speech processing module <NUM> may be jointly trained. In this case, a training set may comprise frames of image data <NUM>, frames of audio data <NUM> and ground truth linguistic features (e.g., ground truth phoneme sequences, text transcriptions or voice command classifications and command parameter values). Both the neural speaker preprocessing module <NUM> and the neural speech processing module <NUM> may be trained in an end-to-end manner using this training set. In this case, errors between predicted and ground truth linguistic features may be back propagated through the neural speech processing module <NUM> and then the neural speaker preprocessing module <NUM>. Parameters for both neural network architectures may then be determined using gradient descent approaches. In this manner, the neural network architecture <NUM> of the neural speaker preprocessing module <NUM> may "learn" parameter values (such as values for weights and/or biases for one or more neural network layers) that generate one or more speaker feature vectors <NUM> that improve at least acoustic processing in an in-vehicle environment, where the neural speaker preprocessing module <NUM> learns to extract features from the facial area of a person that help improve the accuracy of the output linguistic features.

Training of neural network architectures as described herein is typically not performed on in-vehicle devices (although this could be performed if desired). In one case, training may be performed on a computing device with access to substantive processing resources, such as a server computer device with multiple processing units (whether CPUs, GPUs, Field Programmable Gate Arrays - FPGAs - or other dedicated processor architectures) and large memory portions to hold batches of training data. In certain cases, training may be performed using a coupled accelerator device, e.g., a couplable FPGA or GPU-based device. In certain cases, trained parameters may be communicated from a remote server device to an embedded system within the vehicle, e.g. as part of an over-the-air update.

<FIG> shows an example speech processing module <NUM> that uses a speaker feature vector <NUM> to configure an acoustic model. The speech processing module <NUM> may be used to implement, at least in part, one of the speech processing modules described in other examples. In <FIG>, the speech processing module <NUM> comprises a database of acoustic model configurations <NUM>, an acoustic model selector <NUM> and an acoustic model instance <NUM>. The database of acoustic model configurations <NUM> stores a number of parameters to configure an acoustic model. In this example, the acoustic model instance <NUM> may comprise a general acoustic model that is instantiated (e.g., configured or calibrated) using a particular set of parameter values from the database of acoustic model configurations <NUM>. For example, the database of acoustic model configurations <NUM> may store a plurality of acoustic model configurations. Each acoustic model configuration may be associated with a different user, including one or more default acoustic model configurations that are used if a user is not detected or a user is detected but not specifically recognized.

In certain cases, the speaker feature vector <NUM> may be used to represent a particular regional accent instead of (or as well as) a particular user. This may be useful in countries such as India where there may be many different regional accents. In this case, the speaker feature vector <NUM> may be used to dynamically load acoustic models based on an accent recognition that is performed using the speaker feature vector <NUM>. For example, this may be possible in the case that the speaker feature vector <NUM> comprises an x vector as described above. This may be useful in a case with a plurality of accent models (e.g. multiple acoustic model configurations for each accent) that are stored within a memory of the vehicle. This may then allow a plurality of separately trained accent models to be used.

In one case, the speaker feature vector <NUM> may comprise a classification of a person within a vehicle. For example, the speaker feature vector <NUM> may be derived from the user identifier <NUM> output by the face recognition module <NUM> in <FIG>. In another case, the speaker feature vector <NUM> may comprise a classification and/or set of probabilities output by a neural speaker preprocessing module such as module <NUM> in <FIG>. In the latter case, the neural speaker preprocessing module may comprise a softmax layer that outputs "probabilities" for a set of potential users (including a classification for "unrecognized"). In this case, one or more frames of input image data <NUM> may result in a single speaker feature vector <NUM>.

In <FIG>, the acoustic model selector <NUM> receives the speaker feature vector <NUM>, e.g., from a speaker preprocessing module, and selects an acoustic model configuration from the database of acoustic model configurations <NUM>. This may operate in a similar manner to the example of <FIG> described above. If the speaker feature vector <NUM> comprises a set of user classifications then the acoustic model selector <NUM> may select an acoustic model configuration based on these classifications, e.g., by sampling a probability vector and/or selecting a largest probability value as a determined person. Parameter values relating to a selected configuration may be retrieved from the database of acoustic model configurations <NUM> and used to instantiate the acoustic model instance <NUM>. Hence, different acoustic model instances may be used for different identified users within the vehicle.

In <FIG>, the acoustic model instance <NUM>, e.g., as configured by the acoustic model selector <NUM> using a configuration retrieved from the database of acoustic model configurations <NUM>, also receives audio data <NUM>. The acoustic model instance <NUM> is configured to generate phoneme data <NUM> for use in parsing an utterance associated with the audio data <NUM> (e.g., featured within the audio data <NUM>). The phoneme data <NUM> may comprise a sequence of phoneme symbols, e.g., from a predefined alphabet or dictionary. Hence, in the example of <FIG>, the acoustic model selector <NUM> selects an acoustic model configuration from the database <NUM> based on a speaker feature vector, and the acoustic model configuration is used to instantiate an acoustic model instance <NUM> to process the audio data <NUM>.

The acoustic model instance <NUM> may comprise both neural and non-neural architectures. In one case, the acoustic model instance <NUM> may comprise a non-neural model. For example, the acoustic model instance <NUM> may comprise a statistical model. The statistical model may use symbol frequencies and/or probabilities. In one case, the statistical model may comprise a Bayesian model, such as a Bayesian network or classifier. In these cases, the acoustic model configurations may comprise particular sets of symbol frequencies and/or prior probabilities that have been measured in different environments. The acoustic model selector <NUM> thus allows a particular source (e.g., person or user) of an utterance to be determined based on both visual (and in certain cases audio) information, which may provide improvements over using audio data <NUM> on its own to generate phoneme sequence <NUM>.

In another case, the acoustic model instance <NUM> may comprise a neural model. In this case, the acoustic model selector <NUM> and the acoustic model instance <NUM> may comprise neural network architectures. In this case, the database of acoustic model configurations <NUM> may be omitted and the acoustic model selector <NUM> may supply a vector input to the acoustic model instance <NUM> to configure the instance. In this case, training data may be constructed from image data used to generate the speaker feature vector <NUM>, audio data <NUM>, and ground truth sets of phoneme outputs <NUM>. Such a system may be jointly trained.

<FIG> and <FIG> show example image preprocessing operations that may be applied to image data obtained from within a vehicle, such as a motor vehicle. <FIG> shows an example image preprocessing pipeline <NUM> comprising an image preprocessor <NUM>. The image preprocessor <NUM> may comprise any combination of hardware and software to implement functionality as described herein. In one case, the image preprocessor <NUM> may comprise hardware components that form part of image capture circuitry that is coupled to one or more image capture devices; in another case, the image preprocessor <NUM> may be implemented by computer program code (such as firmware) that is executed by a processor of an in-vehicle control system. In one case, the image preprocessor <NUM> may be implemented as part of the speaker preprocessing module described in examples herein; in other cases, the image preprocessor <NUM> may be communicatively coupled to the speaker preprocessing module.

In <FIG>, the image preprocessor <NUM> receives image data <NUM>, such as an image from image capture device <NUM> in <FIG>. The image preprocessor <NUM> processes the image data to extract one or more portions of the image data. <FIG> shows an output <NUM> of the image preprocessor <NUM>. The output <NUM> may comprise one or more image annotations, e.g., metadata associated with one or more pixels of the image data <NUM> and/or features defined using pixel co-ordinates within the image data <NUM>. In the example of <FIG>, the image preprocessor <NUM> performs face detection on the image data <NUM> to determine a first image area <NUM>. The first image area <NUM> may be cropped and extracted as image portion <NUM>. The first image area <NUM> may be defined using a bounding box (e.g., at least top left and bottom right (x, y) pixel co-ordinates for a rectangular area). The face detection may be a pre-cursor step for face recognition, e.g., face detection may determine a face area within the image data <NUM> and face recognition may classify the face area as belonging to a given person (e.g., within a set of people). In the example of <FIG>, the image preprocessor <NUM> also identifies a mouth area within the image data <NUM> to determine a second image area <NUM>. The second image area <NUM> may be cropped and extracted as image portion <NUM>. The second image area <NUM> may also be defined using a bounding box. In one case, the first and second image areas <NUM>, <NUM> may be determined in relation to a set of detected facial features <NUM>. These facial features <NUM> may comprise one or more of eyes, nose and mouth areas. Detection of facial features <NUM> and/or one or more of the first and second areas <NUM>, <NUM> may use neural network approaches, or known face detection algorithms such as the Viola-Jones face detection algorithm as described in "<NPL>. In certain examples, one or more of the first and second image areas <NUM>, <NUM> are used by the speaker preprocessing modules described herein to obtain a speaker feature vector. For example, the first image area <NUM> may provide input image data for the face recognition module <NUM> in <FIG> (i.e. be used to supply image data <NUM>). An example that uses the second image area <NUM> is described with reference to <FIG> below.

<FIG> shows an effect of using an image capture device configured to capture electromagnetic radiation having infra-red wavelengths. In certain cases, image data <NUM> obtained by an image capture device such as image capture device <NUM> in <FIG>, may be impacted by low light situations. In <FIG>, the image data <NUM> contains areas of shadow <NUM> that partially obscure the facial area (e.g., including the first and second image areas <NUM>, <NUM> in <FIG>). In these cases,
an image capture device that is configured to capture electromagnetic radiation having infra-red wavelengths may be used. This may comprise providing adaptations to the image capture device <NUM> in <FIG> (e.g., such as removable filters in hardware and/or software) and/or providing a Near-Infra-Red (NIR) camera. An output from such an image capture device is shown in schematically as image data <NUM>. In image data <NUM>, the facial area <NUM> is reliably captured. In this case, the image data <NUM> provides a representation that is illumination invariant, e.g., that is not affected by changes in illumination, such as those that may occur in night driving. In these cases, the image data <NUM> may be that provided to the image preprocessor <NUM> and/or the speaker preprocessing modules as described herein.

In certain examples, the speaker feature vector described herein may comprise at least a set of elements that represent mouth or lip features of a person. In these cases, the speaker feature vector may be speaker dependent as it changes based on the content of image data featuring the mouth or lip area of a person. In the example of <FIG>, the neural speaker preprocessing module <NUM> may encode lip or mouth features that are used to generate the speaker feature vectors <NUM>. These may be used to improve the performance of the speech processing module <NUM>.

<FIG> shows another example speech processing apparatus <NUM> that uses lip features to form at least part of a speaker feature vector. As with previous examples, the speech processing apparatus <NUM> comprises a speaker preprocessing module <NUM> and a speech processing module <NUM>. The speech processing module <NUM> receives audio data <NUM> (in this case, frames of audio data) and outputs linguistic features <NUM>. The speech processing module <NUM> may be configured as per other examples described herein.

In <FIG>, the speaker preprocessing module <NUM> is configured to receive two different sources of image data. In this example, the speaker preprocessing module <NUM> receives a first set of image data <NUM> that features a facial area of a person. This may comprise the first image area <NUM> as extracted by the image preprocessor <NUM> in <FIG>. The speaker preprocessing module <NUM> also receives a second set of image data <NUM> that features a lip or mouth area of a person. This may comprise the second image area <NUM> as extracted by the image preprocessor <NUM> in <FIG>. The second set of image data <NUM> may be relatively small, e.g., a small cropped portion of a larger image obtained using the image capture device <NUM> of <FIG>. In other examples, the first and second sets of image data <NUM>, <NUM> may not be cropped and may comprise copies of a set of images from an image capture device. Different configurations are possible - cropping the image data may provide improvements in processing speed and training, but neural network architectures may be trained to operate on a wide variety of image sizes.

The speaker preprocessing module <NUM> comprises two components in <FIG>: a feature retrieval component <NUM> and a lip feature extractor <NUM>. The lip feature extractor <NUM> forms part of a lip-reading module. The feature retrieval component <NUM> may be configured in a similar manner to the speaker preprocessing module <NUM> in <FIG>. In this example, the feature retrieval component <NUM> receive the first set of image data <NUM> and outputs a vector portion <NUM> that consists of one or more of an i-vector and an x-vector (e.g., as described above). In one case, the feature retrieval component <NUM> receives a single image per utterance whereas the lip feature extractor <NUM> and the speech processing module <NUM> receive a plurality of frames over the time of the utterance. In one case, if a facial recognition performed by the feature retrieval component <NUM> has a confidence value that is below a threshold, the first set of image data <NUM> may be updated (e.g., by using another/current frame of video) and the facial recognition reapplied until a confidence value meets a threshold (or a predefined number of attempts is exceeded). As described with reference to <FIG>, the vector portion <NUM> may be computed based on the audio data <NUM> for a first number of utterances, and then retrieved as a static value from memory once the first number of utterances is exceeded.

The lip feature extractor <NUM> receives the second set of image data <NUM>. The second set of image data <NUM> may comprise cropped frames of image data that focus on a mouth or lip area. The lip feature extractor <NUM> may receive the second set of image data <NUM> at a frame rate of an image capture device and/or at a subsampled frame rate (e.g., every <NUM> frames). The lip feature extractor <NUM> outputs a set of vector portions <NUM>. These vector portions <NUM> may comprise an output of an encoder that comprises a neural network architecture. The lip feature extractor <NUM> may comprise a convolutional neural network architecture to provide a fixed-length vector output (e.g., <NUM> or <NUM> elements having integer or floating-point values). The lip feature extractor <NUM> may output a vector portion for each input frame of image data <NUM> and/or may encode features over time steps using a recurrent neural network architecture (e.g., using a Long Short Term Memory - LSTM - or Gated Recurrent Unit - GRU) or a "transformer" architecture. In the latter case, an output of the lip feature extractor <NUM> may comprise one or more of a hidden state of a recurrent neural network and an output of the recurrent neural network. One example implementation for the lip feature extractor <NUM> is described by <NPL>.

In <FIG>, the speech processing module <NUM> receives the vector portions <NUM> from the feature retrieval component <NUM> and the vector portions <NUM> from the lip feature extractor <NUM> as inputs. In one case, the speaker preprocessing module <NUM> may combine the vector portions <NUM>, <NUM> into a single speaker feature vector; in another case, the speech processing module <NUM> may receive the vector portions <NUM>, <NUM> separately yet treat the vector portions as different portions of a speaker feature vector. The vector portions <NUM>, <NUM> may be combined into a single speaker feature vector by one or more of the speaker preprocessing module <NUM> and the speech processing module <NUM> using, for example, concatenation or more complex attention-based mechanisms. If the sample rates of one or more of the vector portions <NUM>, the vector portions <NUM> and the frames of audio data <NUM> differ then a common sample rate may be implemented by, for example, a receive-and-hold architecture (where values that more vary more slower are held constant at a given value until a new sample values are received), a recurrent temporal encoding (e.g., using LSTMs or GRUs as above) or an attention-based system where an attention weighting vector changes per time step.

The speech processing module <NUM> may be configured to use the vector portions <NUM>, <NUM> as described in other examples set out herein, e.g., these may be input as a speaker feature vector into a neural acoustic model along with the audio data <NUM>. In an example where the speech processing module <NUM> comprises a neural acoustic model, a training set may be generated based on input video from an image capture device, input audio from an audio capture device and ground-truth linguistic features (e.g., the image preprocessor <NUM> in <FIG> may be used to obtain the first and second sets of image data <NUM>, <NUM> from raw input video).

In certain examples, the vector portions <NUM> may also include an additional set of elements whose values are derived from an encoding of the first set of image data <NUM>, e.g., using a neural network architecture such as <NUM> in <FIG>. These additional elements may represent a "face encoding" while the vector portions <NUM> may represent a "lip encoding". The face encoding may remain static for the utterance whereas the lip encoding may change during, or comprise multiple "frames" for, for the utterance. Although <FIG> shows an example that uses both a lip feature extractor <NUM> and a feature retrieval component <NUM>, in one example the feature retrieval component <NUM> may be omitted. In this latter example, a lip-reading system for in-vehicle use may be used in a manner similar to the speech processing apparatus <NUM> of <FIG>.

<FIG> show an example where the vehicle as described herein is a motor vehicle. <FIG> shows a side view <NUM> of an automobile <NUM>. The automobile <NUM> comprises a control unit <NUM> for controlling components of the automobile <NUM>. The components of the speech processing apparatus <NUM> as shown in <FIG> (as well as the other examples) may be incorporated into this control unit <NUM>. In other cases, the components of the speech processing apparatus <NUM> may be implemented as a separate unit with an option of connectivity with the control unit <NUM>. The automobile <NUM> also comprises at least one image capture device <NUM>. For example, the at least one image capture device <NUM> may comprise the image capture device <NUM> shown in <FIG>. In this example, the at least one image capture device <NUM> may be communicatively coupled to, and controlled by, the control unit <NUM>. In other examples, the at least one image capture device <NUM> is in communication with the control unit <NUM> and remotely controlled. As well as the functions described herein, the at least one image capture device <NUM> may be used for video communications, e.g., voice over Internet Protocol calls with video data, environmental monitoring, driver alertness monitoring etc. <FIG> also shows at least one audio capture device in the form of side-mounted microphones <NUM>. These may implement the audio capture device <NUM> shown in <FIG>.

The image capture devices described herein may comprise one or more still or video cameras that are configured to capture frames of image data on command or at a predefined sampling rate. Image capture devices may provide coverage of both the front and rear of the vehicle interior. In one case, a predefined sampling rate may be less than a frame rate for full resolution video, e.g., a video stream may be captured at <NUM> frames per second, but a sampling rate of the image capture device may capture at this rate, or at a lower rate such as <NUM> frame per second. An image capture device may capture one or more frames of image data having one or more color channels (e.g., RGB or YUV as described above). In certain cases, aspects of an image capture device, such as the frame rate, frame size and resolution, number of color channels and sample format may be configurable. The frames of image data may be downsampled in certain cases, e.g., video capture device that captures video at a "<NUM>" resolution of <NUM> x <NUM> may be downsampled to <NUM> x <NUM> or below. Alternatively, for low-cost embedded devices, a low-resolution image capture device may be used, capturing frames of image data at <NUM> x <NUM> or below. In certain cases, even cheap low-resolution image capture devices may provide enough visual information for speech processing to be improved. As before, an image capture device may also include image pre-processing and/or filtering components (e.g., contrast adjustment, noise removal, color adjustment, cropping, etc.). In certain cases, low latency and/or high frame rate image cameras that meet more strict Automotive Safety Integrity Level (ASIL) levels for the ISO <NUM> automotive safety standard are available. Aside from their safety benefits, they can improve lip reading accuracy by providing higher temporal information. That can be useful to recurrent neural networks for more accurate feature probability estimation.

<FIG> shows an overhead view <NUM> of automobile <NUM>. It comprises front seats <NUM> and rear seat <NUM> for holding passengers in an orientation for front-mounted microphones for speech capture. The automobile <NUM> comprises a driver visual console <NUM> with safety-critical display information. The driver visual console <NUM> may comprise part of the dashboard <NUM> as shown in <FIG>. The automobile <NUM> further comprises a general console <NUM> with navigation, entertainment, and climate control functions. The control unit <NUM> may control the general console <NUM> and may implement a local speech processing module such as <NUM> in <FIG> and a wireless network communication module. The wireless network communication module may transmit one or more of image data, audio data and speaker feature vectors that are generated by the control unit <NUM> to a remote server for processing. The automobile <NUM> further comprises the side-mounted microphones <NUM>, a front overhead multi-microphone speech capture unit <NUM>, and a rear overhead multi-microphone speech capture unit <NUM>. The front and rear speech capture units <NUM>, <NUM> provide additional audio capture devices for capturing speech audio, canceling noise, and identifying the location of speakers. In one case, the front and rear speech capture units <NUM>, <NUM> may also include additional image capture devices to capture image data featuring each of the passengers of the vehicle.

In the example of <FIG>, any one or more of the microphones and speech capture units <NUM>, <NUM> and <NUM> may provide audio data to an audio interface such as <NUM> in <FIG>. The microphone or array of microphones may be configured to capture or record audio samples at a predefined sampling rate. In certain cases, aspects of each audio capture device, such as the sampling rate, bit resolution, number of channels and sample format may be configurable. Captured audio data may be Pulse Code Modulated. Any audio capture device may also include audio pre-processing and/or filtering components (e.g., contrast adjustment, noise removal, etc.). Similarly, any one or more of the image capture devices may provide image data to an image interface such as <NUM> in <FIG> and may also include video pre-processing and/or filtering components (e.g., contrast adjustment, noise removal, etc.).

<FIG> shows an example of an interior of an automobile <NUM> as viewed from the front seats <NUM>. For example, <FIG> may comprise a view towards the windshield <NUM> of <FIG>. <FIG> shows a steering wheel <NUM> (such as steering wheel <NUM> in <FIG>), a side microphone <NUM> (such as one of side microphones <NUM> in <FIG>), a rear-view mirror <NUM> (that may comprise front overhead multi-microphone speech capture unit <NUM>) and a projection device <NUM>. The projection device <NUM> may be used to project images <NUM> onto the windshield, e.g., for use as an additional visual output device (e.g., in addition to the driver visual console <NUM> and the general console <NUM>). In <FIG>, the images <NUM> comprise directions. These may be directions that are projected following a voice command of "Find me directions to the Mall-Mart". Other examples may use a simpler response system.

In certain cases, the functionality of the speech processing modules as described herein may be distributed. For example, certain functions may be computed locally within the automobile <NUM> and certain functions may be computed by a remote ("cloud") server device. In certain cases, functionality may be duplicated on the automobile ("client") side and the remote server device ("server") side. In these cases, if a connection to the remote server device is not available then processing may be performed by a local speech processing module; if a connection to the remote server device is available then one or more of the audio data, image data and speaker feature vector may be transmitted to the remote server device for parsing a captured utterance. A remote server device may have processing resources (e.g., Central Processing Units - CPUs, Graphical Processing Units - GPUs and Random-Access Memory) and so offer improvements on local performance if a connection is available. This may be traded-off against latencies in the processing pipeline (e.g., local processing is more responsive). In one case, a local speech processing module may provide a first output, and this may be complemented and/or enhanced by a result of a remote speech processing module.

In one case, the vehicle, e.g., the automobile <NUM>, may be communicatively coupled to a remote server device over at least one network. The network may comprise one or more local and/or wide area networks that may be implemented using a variety of physical technologies (e.g., wired technologies such as Ethernet and/or wireless technologies such as Wi-Fi - IEEE <NUM> - standards and cellular communications technologies). In certain cases, the network may comprise a mixture of one or more private and public networks such as the Internet. The vehicle and the remote server device may communicate over the network using different technologies and communication pathways.

With reference to the example speech processing apparatus <NUM> of <FIG>, in one case vector generation by the vector generator <NUM> may be performed either locally or remotely but the data store <NUM> is located locally within the automobile <NUM>. In this case, a static speaker feature vector may be computed locally and/or remotely but stored locally within the data store <NUM>. Following this, the speaker feature vector <NUM> may be retrieved from the data store <NUM> within the automobile rather than received from a remote server device. This may improve a speech processing latency.

In a case where a speech processing module is remote from the vehicle, a local speech processing apparatus may comprise a transceiver to transmit data derived from one or more of audio data, image data and the speaker feature vector to the speech processing module and to receive control data from the parsing of the utterance. In one case, the transceiver may comprise a wired or wireless physical interface and one or more communications protocols that provide methods for sending and/or receiving requests in a predefined format. In one case, the transceiver may comprise an application layer interface operating on top of an Internet Protocol Suite. In this case, the application layer interface may be configured to receive communications directed towards a particular Internet Protocol address identifying a remote server device, with routing based on path names or web addresses being performed by one or more proxies and/or communication (e.g., "web") servers.

In certain cases, linguistic features generated by a speech processing module may be mapped to a voice command and a set of data for the voice command (e.g., as described with reference to the utterance parser <NUM> in <FIG>). In one case, the utterance data <NUM> may be used by the control unit <NUM> of automobile <NUM> and used to implement a voice command. In one case, the utterance parser <NUM> may be located within a remote server device and utterance parsing may involve identifying an appropriate service to execute the voice command from the output of the speech processing module. For example, the utterance parser <NUM> may be configured to make an application programming interface (API) request to an identified server, the request comprising a command and any command data identified from the output of the language model. For example, an utterance of "Where is the Mall Mart?" may result in a text output of "where is the mall mart" that may be mapped to a directions service API request for vehicle mapping data with a desired location parameter of "mall mart" and a current location of the vehicle, e.g., as derived from a positioning system such as the Global Positioning System. The response may be retrieved and communicated to the vehicle, where it may be displayed as illustrated in <FIG>.

In one case, a remote utterance parser <NUM> communicates response data to the control unit <NUM> of the automobile <NUM>. This may comprise machine readable data to be communicated to the user, e.g., via a user interface or audio output. The response data may be processed and a response to the user may be output on one or more of the driver visual console <NUM> and the general console <NUM>. Providing a response to a user may comprise the display of text and/or images on a display screen of one or more of the driver visual console <NUM> and the general console <NUM>, or an output of sounds via a text-to-speech module. In certain cases, the response data may comprise audio data that may be processed at the control unit <NUM> and used to generate an audio output, e.g., via one or more speakers. A response may be spoken to a user via speakers mounted within the interior of the automobile <NUM>.

<FIG> shows an example embedded computing system <NUM> that may implement a speech processing apparatus as described herein. A system similar to the embedded computing system <NUM> may be used to implement the control unit <NUM> in <FIG>. The example embedded computing system <NUM> comprises one or more computer processor (CPU) cores <NUM> and zero or more graphics processor (GPU) cores <NUM>. The processors connect through a board-level interconnect <NUM> to random-access memory (RAM) devices <NUM> for program code and data storage. The embedded computing system <NUM> also comprises a network interface <NUM> to allow the processors to communicate with remote systems and specific vehicle control circuitry <NUM>. By executing instructions stored in RAM devices through interface <NUM>, the CPUs <NUM> and/or GPUs <NUM> may perform functionality as described herein. In certain cases, constrained embedded computing devices may have a similar general arrangement of components, but in certain cases may have fewer computing resources and may not have dedicated graphics processors <NUM>.

<FIG> shows an example method <NUM> for processing speech that may be performed to improve in-vehicle speech recognition. The method <NUM> begins at block <NUM> where audio data is received from an audio capture device. The audio capture device may be located within a vehicle. The audio data may feature an utterance from a user. Block <NUM> comprises capturing data from one or more microphones such as devices <NUM>, <NUM> and <NUM> in <FIG>. In one case, block <NUM> may comprise receiving audio data over a local audio interface; in another case, block <NUM> may comprise receiving audio data over a network, e.g., at an audio interface that is remote from the vehicle.

At block <NUM>, image data from an image capture device is received. The image capture device may be located within the vehicle, e.g., may comprise the image capture device <NUM> in <FIG>. In one case, block <NUM> may comprise receiving image data over a local image interface; in another case, block <NUM> may comprise receiving image data over a network, e.g., at an image interface that is remote from the vehicle.

At block <NUM>, a speaker feature vector is obtained based on the image data. This may comprise, for example, implementing any one of the speaker preprocessing modules <NUM>, <NUM>, <NUM> and <NUM>. Block <NUM> may be performed by a local processor of the automobile <NUM> or by a remote server device. At block <NUM>, the utterance is parsed using a speech processing module. For example, this may comprise implementing any one of the speech processing modules <NUM>, <NUM>, <NUM>, <NUM> and <NUM>. Block <NUM> comprises number of subblocks. These include, at subblock <NUM>, providing the speaker feature vector and the audio data as an input to an acoustic model of the speech processing module. This may comprise operations similar to those described with reference to <FIG>. In certain cases, the acoustic model comprises a neural network architecture. At subblock <NUM>, phoneme data is predicted, using at least the neural network architecture, based on the speaker feature vector and the audio data. This may comprise using a neural network architecture that is trained to receive the speaker feature vector as an input in additional to the audio data. As both the speaker feature vector and the audio data comprise numeric representations, these may be processed similarly by the neural network architecture. In certain cases, an existing CTC or hybrid acoustic model may be configured to receive a concatenation of the speaker feature vector and the audio data, and then trained using a training set that additionally comprises image data (e.g., that is used to derive the speaker feature vector).

In certain cases, block <NUM> comprises performing facial recognition on the image data to identify the person within the vehicle. For example, this may be performed as described with reference to face recognition module <NUM> in <FIG>. Following this, user profile data for the person (e.g., in the vehicle) may be obtained based on the facial recognition. For example, user profile data may be retrieved from the data store <NUM> using a user identifier <NUM> as described with reference to <FIG>. The speaker feature vector may then be obtained in accordance with the user profile data. In one case, the speaker feature vector may be retrieved as a static set of element values from the user profile data. In another case, the user profile data may indicate that the speaker feature vector is to be computed, e.g., using one or more of the audio data and the image data received at blocks <NUM> and <NUM>. In certain cases, block <NUM> comprises comparing a number of stored speaker feature vectors associated with user profile data with a predefined threshold. For example, the user profile data may indicate how many previous voice queries have been performed by a user identified using face recognition. Responsive to the number of stored speaker feature vectors being below the predefined threshold, the speaker feature vector may be computed using one or more of the audio data and the image data. Responsive to the number of stored speaker feature vectors being greater than the predefined threshold, a static speaker feature vector may be obtained, e.g., one that is stored within or is accessible via the user profile data. In this case, the static speaker feature vector may be generated using the number of stored speaker feature vectors.

In certain examples, block <NUM> may comprise processing the image data to generate one or more speaker feature vectors based on lip movement within the facial area of the person. For example, a lip-reading module, such as lip feature extractor <NUM> or a suitably configured neural speaker preprocessing module <NUM>, may be used. The output of the lip-reading module may be used to supply one or more speaker feature vectors to a speech processing module, and/or may be combined with other values (such as i or x-vectors) to generate a larger speaker feature vector.

In certain examples, block <NUM> comprises providing the phoneme data to a language model of the speech processing module, predicting a transcript of the utterance using the language model, and determining a control command for the vehicle using the transcript. For example, block <NUM> may comprise operations similar to those described with reference to <FIG>.

<FIG> shows an example processing system <NUM> comprising a non-transitory computer-readable storage medium <NUM> storing instructions <NUM> which, when executed by at least one processor <NUM>, cause the at least one processor to perform a series of operations. The operations of this example use previously described approaches to generate a transcription of an utterance. These operations may be performed within a vehicle, e.g. as previously described, or extend an in-vehicle example to situations that are not vehicle-based, e.g., that may be implemented using desktop, laptop, mobile or server computing devices, amongst others.

Via instruction <NUM>, the processor <NUM> is configured to receive audio data from an audio capture device. This may comprise accessing a local memory containing the audio data and/or receiving a data stream or set of array values over a network. The audio data may have a form as described with reference to other examples herein. Via instruction <NUM>, the processor <NUM> is configured to receive a speaker feature vector. The speaker feature vector is obtained based on image data from an image capture device, the image data featuring a facial area of a user. For example, the speaker feature vector may be obtained using the approaches described with reference to any of <FIG>, <FIG>, <FIG> and <FIG>. The speaker feature vector may be computed locally, e.g., by the processor <NUM>, accessed from a local memory, and/or received over a network interface (amongst others). Via instruction <NUM>, the processor <NUM> is instructed to parse the utterance using a speech processing module. The speech processing module may comprise any of the modules described with reference to any of <FIG>, <FIG>, <FIG>, <FIG> and <FIG>.

<FIG> shows that instruction <NUM> may be broken down into a number of further instructions. Via instruction <NUM>, the processor <NUM> is instructed to provide the speaker feature vector and the audio data as an input to an acoustic model of the speech processing module. This may be achieved in a manner similar to that described with reference to <FIG>. In the present example, the acoustic model comprises a neural network architecture. Via instruction <NUM>, the processor <NUM> is instructed to predict, using at least the neural network architecture, phoneme data based on the speaker feature vector and the audio data. Via instruction <NUM>, the processor <NUM> is instructed to provide the phoneme data to a language model of the speech processing module. This may also be performed in a manner similar to that shown in <FIG>. Via instruction <NUM>, the processor <NUM> is instructed to generate a transcript of the utterance using the language model. For example, the transcript may be generated as an output of the language model. In certain cases, the transcript may be used by a control system to execute a voice command, such as control unit <NUM> in the automobile <NUM>. In other cases, the transcript may comprise an output for a speech-to-text system. In the latter case, the image data may be retrieved from a web-camera or the like that is communicatively coupled to the computing device comprising the processor <NUM>. For a mobile computing device, the image data may be obtained from a forward-facing image capture device.

In certain examples, the speaker feature vector received according to instructions <NUM> comprises one or more of: vector elements that are dependent on the speaker that are generated based on the audio data (e.g., i-vector or x-vector components); vector elements that are dependent on lip movement of the speaker that is generated based on the image data (e.g., as generated by a lip-reading module); and vector elements that are dependent on a face of the speaker that is generated based on the image data. In one case, the processor <NUM> may comprise part of a remote server device and the audio data and the speaker image vector may be received from a motor vehicle, e.g., as part of a distributed processing pipeline.

Certain examples are described that relate to speech processing including automatic speech recognition. Certain examples relate to the processing of certain spoken languages. Various examples operate, similarly, for other languages or combinations of languages. Certain examples improve an accuracy and a robustness of speech processing by incorporating additional information that is derived from an image of a person making an utterance. This additional information may be used to improve linguistic models. Linguistic models may comprise one or more of acoustic models, pronunciation models and language models.

Certain examples described herein may be implemented to address the unique challenges of performing automatic speech recognition within a vehicle, such as an automobile. In certain combined examples, image data from a camera may be used to determine lip-reading features and to recognize a face to enable an i-vector and/or x-vector profile to be built and selected. By implementing approaches as described herein it may be possible to perform automatic speech recognition within the noisy, multichannel environment of a motor vehicle.

Certain examples described herein may increase an efficiency of speech processing by including one or more features derived from image data, e.g. lip positioning or movement, within a speaker feature vector that is provided as an input to an acoustic model that also receives audio data as an input (a singular model), e.g. rather than having an acoustic model that only receives an audio input or separate acoustic models for audio and image data.

Certain methods and sets of operations may be performed by instructions that are stored upon a non-transitory computer readable medium. The non-transitory computer readable medium stores code comprising instructions that, if executed by one or more computers, would cause the computer to perform steps of methods described herein. The non-transitory computer readable medium may comprise one or more of a rotating magnetic disk, a rotating optical disk, a flash random access memory (RAM) chip, and other mechanically moving or solid-state storage media. Any type of computer-readable medium is appropriate for storing code comprising instructions according to various example.

Certain examples described herein may be implemented as so-called system-on-chip (SoC) devices. SoC devices control many embedded in-vehicle systems and may be used to implement the functions described herein. In one case, one or more of the speaker preprocessing module and the speech processing module may be implemented as a SoC device. An SoC device may comprise one or more processors (e.g., CPUs or GPUs), random-access memory (RAM - e.g., off-chip dynamic RAM or DRAM), a network interface for wired or wireless connections such as ethernet, WiFi, <NUM>, <NUM> long-term evolution (LTE), <NUM>, and other wireless interface standard radios. An SoC device may also comprise various I/O interface devices, as needed for different peripheral devices such as touch screen sensors, geolocation receivers, microphones, speakers, Bluetooth peripherals, and USB devices, such as keyboards and mice, among others. By executing instructions stored in RAM devices processors of an SoC device may perform steps of methods as described herein.

Certain examples have been described herein and it will be noted that different combinations of different components from different examples may be possible. Salient features are presented to better explain examples; however, it is clear that certain features may be added, modified and/or omitted without modifying the functional aspects of these examples as described.

Claim 1:
A vehicle-mounted apparatus comprising:
an audio interface configured to receive audio data from an audio capture device of the vehicle, the audio data featuring an utterance of a person within the vehicle;
an image interface configured to receive image data from an image capture device to capture images from the vehicle, wherein the image data includes a facial area of the person within the vehicle;
a speaker preprocessing module configured to receive the image data and obtain, based on the image data, a speaker feature vector, wherein obtaining the speaker feature vector comprises:
performing facial recognition on the image data to identify the person within the vehicle;
obtaining user profile data for the person based on the facial recognition; and
obtaining the speaker feature vector in accordance with the user profile data,
wherein obtaining the speaker feature vector in accordance with the user profile data comprises:
comparing a number of stored speaker feature vectors associated with the user profile data with a predefined threshold;
responsive to the number of stored speaker feature vectors being below the predefined threshold, computing the speaker feature vector using the audio data and the image data; and
responsive to the number of stored speaker feature vectors being greater than the predefined threshold, obtaining a static speaker feature vector associated with the user profile data, the static speaker feature vector being generated using the number of stored speaker feature vectors; and
a speech processing module configured to parse the utterance based on the audio data and the speaker feature vector, wherein the speech processing module comprises an acoustic model configured to process the audio data and to predict phoneme data for use in parsing the utterance, wherein the speech processing module is configured to use the speaker feature vector to configure the acoustic model, the speech processing module comprising:
a database of acoustic model configurations;
an acoustic model selector configured to select an acoustic model configuration from the database based on the speaker feature vector; and
an acoustic model instance configured to process the audio data, the acoustic model instance being instantiated based on the acoustic model configuration selected by the acoustic model selector, the acoustic model instance being configured to generate the phoneme data for use in parsing the utterance.