Patent Description:
End-to-end speech recognition with neural transducers in which only one neural network models everything for speech recognition has various advantages both in training and inference. Purely end-to-end training only from transcripts and audio is possible. Only a beam search over one neural network is necessary for inference, resulting in smaller computational cost, smaller memory footprint, and simpler inference engine.

Training of neural transducer models is simple but has some disadvantages. For example, end-to-end training is prone to overfitting because one utterance can serve as only one sample. While neural transducer models include an encoder network that serves as an acoustic model, a prediction network that serves as a language model, and a joint network that combines acoustic and linguistic embeddings from the encoder and prediction networks, end-to-end training does not take such modularity into consideration. As a result, customization of neural transducers are more difficult than the conventional hybrid model in which acoustic and language models are separately trained and can be independently customized. Improved accuracy and more reactive customization are both critical to provide competitive service from the cloud. Hence, there is a need to (<NUM>) improve speech recognition accuracy by mitigating overfitting and (<NUM>) build a model that is more reactive to customization.

<NPL>, examines ways in which RNN-T can achieve better automatic speech recognition (ASR) accuracy via performing auxiliary tasks and proposes using the same auxiliary task as primary RNN-T ASR task, wherein encoder underfitting is addressed by connecting an auxiliary branch to an intermediate encoder layer and applying the same RNN-T loss function (as used for training the primary RNN-T) to the auxiliary branch.

<NPL>, discloses a multi-speaker RNN-T-based speech recognition model in which a first RNN-T and a second RNN-T share a common mixture encoder.

<NPL>, relates to training and deploying a two-pass bidirectional end-to-end speech recognition model which includes a shared/common encoder, a connectionist temporal classification (CTC) decoder that models the alignment of the frames and tokens, a left-to-right/forward attention/prediction decoder, and a right-to-left/backward attention/prediction decoder.

<NPL>, is concerned with improving a streaming/online recurrent neural network transducer (RNN-T) by training an offline RNN-T and utilizing the latter as a teacher to train a student streaming RNN-T, wherein the offline/teacher RNN-T has a bidirectional encoder - as opposed to the online/student RNN-T, which has a unidirectional encoder.

<CIT> relates to training and deploying a two-pass ASR model, wherein audio data is processed using a shared/common encoder, a first-pass portion of the ASR model processes the shared encoder output using a RNN-T decoder to generate RNN-T output including one or more candidate text representation(s) of the utterance captured in the audio data, and a second-pass portion processes the shared encoder output along with the RNN-T output using a listen, attend and spell (LAS) decoder to refine the RNN-T output to generate the final text representation of the audio data.

It is the goal of the present invention to overcome shortcomings of the prior art. This goal is achieved by the present invention as defined in the appended independent claims. Preferred embodiments of the present invention are set forth in the appended dependent claims.

Features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings.

Preferred embodiments of the present invention will now be described, by way of example only, and with reference to the following drawings:.

Embodiments of the present invention are directed to separating acoustic and linguistic information in neural transducer models for end-to-end speech recognition.

Embodiments of the present invention prepare an additional prediction network that predicts label sequences backward. Embodiments of the present invention conduct multi-task training of the neural transducers with the original forward prediction network and the additional backward prediction network. The same encoder network is used during the multi-task learning. Only the original forward prediction network is used for actual tests (inference stage).

In embodiments of the present invention, the encoder network is jointly trained with both forward and backward prediction networks and thus does not capture the linguistic information as a result. Exemplary benefits of this method at least include: (<NUM>) Overfitting to the training data is mitigated; and (<NUM>) The linguistic information and the acoustic information are more separately captured in the prediction and the encoder networks. Resultingly, customization of prediction and encoder networks is more reactive.

<FIG> is a block diagram showing an exemplary computing device <NUM>, in accordance with an embodiment of the present invention. The computing device <NUM> is configured to separate acoustic and linguistic information in neural transducer models for end-to-end speech recognition. In this way, customization of the prediction and encoder networks (see <FIG>) is more reactive.

The computing device <NUM> may be embodied as any type of computation or computer device capable of performing the functions described herein, including, without limitation, a computer, a server, a rack based server, a blade server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor- based system, and/or a consumer electronic device. Additionally or alternatively, the computing device <NUM> may be embodied as a one or more compute sleds, memory sleds, or other racks, sleds, computing chassis, or other components of a physically disaggregated computing device. As shown in <FIG>, the computing device <NUM> illustratively includes the processor <NUM>, an input/output subsystem <NUM>, a memory <NUM>, a data storage device <NUM>, and a communication subsystem <NUM>, and/or other components and devices commonly found in a server or similar computing device. Of course, the computing device <NUM> may include other or additional components, such as those commonly found in a server computer (e.g., various input/output devices), in other embodiments. Additionally, in some embodiments, one or more of the illustrative components may be incorporated in, or otherwise form a portion of, another component. For example, the memory <NUM>, or portions thereof, may be incorporated in the processor <NUM> in some embodiments.

The processor <NUM> may be embodied as any type of processor capable of performing the functions described herein. The processor <NUM> may be embodied as a single processor, multiple processors, a Central Processing Unit(s) (CPU(s)), a Graphics Processing Unit(s) (GPU(s)), a single or multi-core processor(s), a digital signal processor(s), a microcontroller(s), or other processor(s) or processing/controlling circuit(s).

The memory <NUM> may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the memory <NUM> may store various data and software used during operation of the computing device <NUM>, such as operating systems, applications, programs, libraries, and drivers. The memory <NUM> is communicatively coupled to the processor <NUM> via the I/O subsystem <NUM>, which may be embodied as circuitry and/or components to facilitate input/output operations with the processor <NUM> the memory <NUM>, and other components of the computing device <NUM>. For example, the I/O subsystem <NUM> may be embodied as, or otherwise include, memory controller hubs, input/output control hubs, platform controller hubs, integrated control circuitry, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc. ) and/or other components and subsystems to facilitate the input/output operations. In some embodiments, the I/O subsystem <NUM> may form a portion of a system-on-a-chip (SOC) and be incorporated, along with the processor <NUM>, the memory <NUM>, and other components of the computing device <NUM>, on a single integrated circuit chip.

The data storage device <NUM> may be embodied as any type of device or devices configured for short-term or long-term storage of data such as, for example, memory devices and circuits, memory cards, hard disk drives, solid state drives, or other data storage devices. The data storage device <NUM> can store program code for separating acoustic and linguistic information in neural transducer models for end-to-end speech recognition. The communication subsystem <NUM> of the computing device <NUM> may be embodied as any network interface controller or other communication circuit, device, or collection thereof, capable of enabling communications between the computing device <NUM> and other remote devices over a network. The communication subsystem <NUM> may be configured to use any one or more communication technology (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication.

As shown, the computing device <NUM> may also include one or more peripheral devices <NUM>. The peripheral devices <NUM> may include any number of additional input/output devices, interface devices, and/or other peripheral devices. For example, in some embodiments, the peripheral devices <NUM> may include a display, touch screen, graphics circuitry, keyboard, mouse, speaker system, microphone, network interface, and/or other input/output devices, interface devices, and/or peripheral devices.

Of course, the computing device <NUM> may also include other elements (not shown), as readily contemplated by one of skill in the art, as well as omit certain elements. For example, various other input devices and/or output devices can be included in computing device <NUM>, depending upon the particular implementation of the same, as readily understood by one of ordinary skill in the art. For example, various types of wireless and/or wired input and/or output devices can be used. Moreover, additional processors, controllers, memories, and so forth, in various configurations can also be utilized. Further, in another embodiment, a cloud configuration can be used (e.g., see <FIG>). These and other variations of the processing system <NUM> are readily contemplated by one of ordinary skill in the art given the teachings of the present invention provided herein.

As employed herein, the term "hardware processor subsystem" or "hardware processor" can refer to a processor, memory (including RAM, cache(s), and so forth), software (including memory management software) or combinations thereof that cooperate to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem can include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements can be included in a central processing unit, a graphics processing unit, and/or a separate processor- or computing element-based controller (e.g., logic gates, etc.). The hardware processor subsystem can include one or more on-board memories (e.g., caches, dedicated memory arrays, read only memory, etc.). In some embodiments, the hardware processor subsystem can include one or more memories that can be on or off board or that can be dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, basic input/output system (BIOS), etc.).

In some embodiments, the hardware processor subsystem can include and execute one or more software elements. The one or more software elements can include an operating system and/or one or more applications and/or specific code to achieve a specified result.

In other embodiments, the hardware processor subsystem can include dedicated, specialized circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry can include one or more application-specific integrated circuits (ASICs), FPGAs, and/or PLAs.

These and other variations of a hardware processor subsystem are also contemplated in accordance with embodiments of the present invention.

<FIG> shows an exemplary neural transducer architecture <NUM>, in accordance with an embodiment of the present invention.

The neural transducer architecture <NUM> includes a first Recurrent Neural Network Transducer (RNN-T) <NUM> and a second RNN-T <NUM>.

The first RNN-T <NUM> includes a common encoder <NUM>, a forward prediction network <NUM>, a first joint network (hereinafter "joint network" in short) <NUM>, and a softmax block <NUM>. The forward prediction network <NUM> predicts label sequences in a forward direction. The joint network <NUM> combines outputs of both the common encoder <NUM> and the forward prediction network <NUM>. The forward prediction network <NUM>, the joint network <NUM>, and the softmax block <NUM> can be considered to form a first decoder network <NUM>.

The second RNN-T <NUM> includes the common encoder <NUM>, a backward prediction network <NUM>, a second joint network (hereinafter "joint network" in short) <NUM>, and a softmax block <NUM>. The backward prediction network <NUM> predicts label sequences in a backward direction. The joint network <NUM> combines outputs of both the common encoder <NUM> and the backward prediction network <NUM>. The backward prediction network <NUM>, the joint network <NUM>, and the softmax block <NUM> can be considered to form a second decoder network <NUM>.

The common encoder <NUM> serves as an acoustic model. The forward prediction network <NUM> and the backward prediction network <NUM> serve as language models. The joint network <NUM> combines acoustic and linguistic embeddings from the encoder <NUM> and the forward prediction network <NUM>. The joint network <NUM> combines acoustic and linguistic embeddings from the encoder <NUM> and the backward prediction network <NUM>.

The trained first RNN-T <NUM> is used for inference.

The encoder <NUM> takes Xi. T as input, and outputs <MAT>.

The forward prediction network <NUM> takes Yu-<NUM> as input, and outputs <MAT>.

The joint network <NUM> takes <MAT> and <MAT> as inputs, and outputs Zt,u.

The softmax block <NUM> takes Zt,u as input, and outputs P(y|t,u).

The backward prediction network <NUM> takes Yu+<NUM> as input, and outputs <MAT>.

In <FIG>, the following notations apply:.

<FIG> is a diagram showing an exemplary trellis search <NUM> in the forward direction, in accordance with an embodiment of the present invention.

The trellis search <NUM> has a x direction and a y direction. The x direction corresponds to the input sequence x=(x<NUM>,. The y direction corresponds to the forward output sequence Y=(y<NUM>,. Each node in the trellis <NUM> represents a softmax of Zt,u by the softmax block <NUM>.

<FIG> is a diagram showing an exemplary trellis search <NUM> in the backward direction, in accordance with an embodiment of the present invention.

The trellis search <NUM> has a x direction and a y direction. The x direction corresponds to the input sequence x=(x<NUM>,. The y direction corresponds to the output sequence Y=(y<NUM>,. Each node in the trellis <NUM> represents a softmax of Zt,u by the softmax block <NUM>.

<FIG> is a flow diagram showing an exemplary method <NUM> for training a Recurrent Neural Network Transducer (RNN-T), in accordance with an embodiment of the present invention.

At block <NUM>, train, by inputting a set of audio data, a first RNN-T which includes a common encoder, a forward prediction network, and a first joint network combining outputs of both the common encoder and the forward prediction network. The forward prediction network predicts label sequences forward.

In an embodiment, block <NUM> can include block 510A.

At block 510A, form a first output probability lattice from the set of audio data and an output of the first RNN-T with the set of audio data along an x-axis and the output of the first RNN-T along a y-axis. Compute a RNN-T loss on the first output probability lattice of the first RNN_T in an upper right direction.

At block <NUM>, train, by inputting the set of audio data, a second RNN-T which includes the common encoder, a backward prediction network, and a second joint network combining outputs of both the common encoder and the backward prediction network. The backward prediction network predicts label sequences backward. The first RNN-T and the second RNN-T form the RNN-T. The training steps jointly train the forward prediction network and the backward prediction network while ignoring and without capturing linguistic information. The common encoder network is combined with the forward and backward prediction network. Thus the common encoder must not capture word sequence (linguistic) information. Word sequence information needs to be captured by the forward and the backward prediction networks. Only the acoustic information should be captured by the encoder network. The trained first RNN-T is used for inference.

In an embodiment, block <NUM> can include block 520A.

At block 520A, form a second output probability lattice from the set of audio data and an output of the second RNN-T with the set of audio data along an x-axis and the output of the second RNN-T along a y-axis. Compute a RNN-T loss on the second output probability loss of the second RNN-T in a lower left direction.

<FIG> shows an exemplary operating environment <NUM>, in accordance with an embodiment of the present invention.

The environment <NUM> involves a server side <NUM> and a client side <NUM>.

The server side <NUM> includes a speech-based computer processing system. For illustrative purposes, the speech-based computer processing system is an end-to-end speech recognition system <NUM>. The end-to-end speech recognition system <NUM> has improved speech recognition accuracy in accordance with the present principles. It is to be appreciated that block <NUM> can represent any speech-based computer processing system that involves one or more of the following: speech recognition; speaker identification; speaker verification; speaker diarisation; language identification; keyword spotting; emotion detection; automatic translation; court reporting; hands-free computing; home automation; mobile telephony; and so forth.

The client side <NUM> includes a set of workstations <NUM>.

Users at the workstations <NUM> can engage in and/or otherwise use speech recognition sessions. The speech recognition sessions can relate, but are not limited to, customer service, voice dialing, machine control, data searching, data entry, system/facility/entity access, and so forth.

Communications between the server side <NUM> and the client side <NUM> are made through one or more networks <NUM>. In an embodiment, the server side <NUM> is realized using a cloud configuration.

As shown, cloud computing environment <NUM> includes one or more cloud computing nodes <NUM> with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone 754A, desktop computer 754B, laptop computer 754C, and/or automobile computer system 754N may communicate. It is understood that the types of computing devices 754A-N shown in <FIG> are intended to be illustrative only and that computing nodes <NUM> and cloud computing environment <NUM> can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).

Workloads layer <NUM> provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include: mapping and navigation <NUM>; software development and lifecycle management <NUM>; virtual classroom education delivery <NUM>; data analytics processing <NUM>; transaction processing <NUM>; and separating acoustic and linguistic information in neural network models for end-to-end speech recognition <NUM>.

Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as SMALLTALK, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.

Reference in the specification to "one embodiment" or "an embodiment" of the present invention, as well as other variations thereof, means that a particular feature, structure, characteristic, and so forth described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment", as well any other variations, appearing in various places throughout the specification are not necessarily all referring to the same embodiment.

It is to be appreciated that the use of any of the following "/", "and/or", and "at least one of", for example, in the cases of "A/B", "A and/or B" and "at least one of A and B", is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of both options (A and B). As a further example, in the cases of "A, B, and/or C" and "at least one of A, B, and C", such phrasing is intended to encompass the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This may be extended, as readily apparent by one of ordinary skill in this and related arts, for as many items listed.

Claim 1:
A computer-implemented method for training a Recurrent Neural Network Transducer (RNN-T), the method comprising:
training, by inputting a set of audio data, a first RNN-T which comprises a common encoder, a forward prediction network, and a first joint network combining outputs of both the common encoder and the forward prediction network, wherein the forward prediction network predicts label sequences forward; and
training, by inputting the set of audio data, a second RNN-T which comprises the common encoder, a backward prediction network, and a second joint network combining outputs of both the common encoder and the backward prediction network, wherein the backward prediction network predicts label sequences backward, and wherein the trained first RNN-T is used for inference, and wherein the training steps jointly train the forward prediction network and the backward prediction network.