TRAINING DATA SEQUENCE FOR RNN-T BASED GLOBAL ENGLISH MODEL

A computer-implemented method for preparing training data for a speech recognition model is provided including obtaining a plurality of audio data sets, each audio data set having a different acoustic feature and sorting sentences from the plurality of audio data sets so that similar sentences from different audio data sets are positioned closely, while imposing a weak constraint on audio length, to train the speech recognition model.

BACKGROUND

The present invention relates generally to machine learning, and more specifically, to methods and systems for composing an efficient training data sequence for a recurrent neural network transducer (RNN-T) based global English model.

End-to-end models for automatic speech recognition (ASR) have gained popularity in recent years as a way to fold separate components of a conventional ASR system (e.g., acoustic, pronunciation and language models) into a single neural network. Examples of such models include connectionist temporal classification (CTC) based models, recurrent neural network transducer (RNN-T), and attention-based seq2seq models. Among these models, RNN-T is the most suitable streaming end-to-end recognizer, which has shown competitive performance compared to conventional systems.

Before delving into RNN-T, speech recognition continues to evolve to meet the untethered and nimble demands of a mobile environment. New speech recognition architectures or improvements to existing architectures continue to be developed that seek to increase the quality of ASR systems. To illustrate, speech recognition initially employed multiple models where each model had a dedicated purpose. For instance, an ASR system included an acoustic model (AM), a pronunciation model (PM), and a language model (LM). The acoustic model mapped segments of audio (e.g., frames of audio) to phonemes. The pronunciation model connected these phonemes together to form words while the language model was used to express the likelihood of given phrases (e.g., the probability of a sequence of words). Yet although these individual models worked together, each model was trained independently and often manually designed on different datasets.

The approach of separate models enabled a speech recognition system to be fairly accurate, especially when the training corpus (e.g., body of training data) for a given model caters to the effectiveness of the model, but needing to independently train separate models introduced its own complexities and led to an architecture with integrated models. These integrated models sought to use a single neural network to directly map an audio waveform (e.g., input sequence) to an output sentence (e.g., output sequence). This resulted in a sequence-to-sequence approach, which generated a sequence of words (or graphemes) when given a sequence of audio features. Examples of sequence-to-sequence models include “attention-based” models and “listen-attend-spell” (LAS) models. A LAS model transcribes speech utterances into characters using a listener component, an attender component, and a speller component. Here, the listener is a recurrent neural network (RNN) encoder that receives an audio input (e.g., a time-frequency representation of speech input) and maps the audio input to a higher-level feature representation. The attender attends to the higher-level feature to learn an alignment between input features and predicted subword units (e.g., a grapheme or a word piece). The speller is an attention-based RNN decoder that generates character sequences from the input by producing a probability distribution over a set of hypothesized words. With an integrated structure, all components of a model may be trained jointly as a single end-to-end (E2E) neural network. Here, an E2E model refers to a model whose architecture is constructed entirely of a neural network. A fully neural network functions without external and/or manually designed components (e.g., finite state transducers, a lexicon, or text normalization modules). Additionally, when training E2E models, these models generally do not require bootstrapping from decision trees or time alignments from a separate system.

Although early E2E models proved accurate and a training improvement over individually trained models, these E2E models, such as the LAS model, functioned by reviewing an entire input sequence before generating output text, and thus, did not allow streaming outputs as inputs were received. Without streaming capabilities, an LAS model is unable to perform real-time voice transcription. Due to this deficiency, deploying the LAS model for speech applications that are latency sensitive or require real-time voice transcription may pose issues.

Additionally, speech recognition systems that have acoustic, pronunciation, and language models, or such models composed together, may rely on a decoder that has to search a relatively large search graph associated with these models. With a large search graph, it is not conducive to host this type of speech recognition system entirely on-device. Here, when a speech recognition system is hosted “on-device,” a device that receives the audio input uses its processor(s) to execute the functionality of the speech recognition system. For instance, when a speech recognition system is hosted entirely on-device, the processors of the device do not need to coordinate with any off-device computing resources to perform the functionality of the speech recognition system. A device that performs speech recognition not entirely on-device relies on remote computing (e.g., of a remote computing system or cloud computing) and therefore online connectivity to perform at least some function of the speech recognition system. For example, a speech recognition system performs decoding with a large search graph using a network connection with a server-based model.

Unfortunately, being reliant upon a remote connection makes a speech recognition system vulnerable to latency issues and/or inherent unreliability of communication networks. To improve the usefulness of speech recognition by avoiding these issues, speech recognition systems again evolved into a form of a sequence-to-sequence model known as a recurrent neural network transducer (RNN-T). A RNN-T does not employ an attention mechanism and, unlike other sequence-to-sequence models that generally need to process an entire sequence (e.g., audio waveform) to produce an output (e.g., a sentence), the RNN-T continuously processes input samples and streams output symbols, a feature that is particularly attractive for real-time communication. For instance, speech recognition with an RNN-T may output characters one-by-one as spoken.

Accordingly, a need exists for more efficient processes for training data sequences using RNN-T.

SUMMARY

In accordance with an embodiment, a computer-implemented method for preparing training data for a speech recognition model is provided. The computer-implemented method includes obtaining a plurality of audio data sets, each audio data set having a different acoustic feature and sorting sentences from the plurality of audio data sets so that similar sentences from different audio data sets are positioned closely, while imposing a weak constraint on audio length, to train the speech recognition model.

In accordance with another embodiment, a computer program product for preparing training data for a speech recognition model is provided. The computer program product includes a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to obtain a plurality of audio data sets, each audio data set having a different acoustic feature and sort sentences from the plurality of audio data sets so that similar sentences from different audio data sets are positioned closely, while imposing a weak constraint on audio length, to train the speech recognition model.

In accordance with yet another embodiment, a system for preparing training data for a speech recognition model is provided. The system includes a memory and one or more processors in communication with the memory configured to obtain a plurality of audio data sets, each audio data set having a different acoustic feature and sort sentences from the plurality of audio data sets so that similar sentences from different audio data sets are positioned closely, while imposing a weak constraint on audio length, to train the speech recognition model.

In accordance with another embodiment, a computer-implemented method for preparing training data for a speech recognition model is provided. The computer-implemented method includes obtaining a plurality of audio data sets, each audio data set having a different acoustic feature, sorting sentences from the plurality of audio data sets so that similar sentences from different audio data sets are positioned closely to train the speech recognition model, and grouping the similar sentences from the different audio data sets into mini-batches, wherein each mini-batch of the mini-batches includes sentence pairs between different English dialects.

In accordance with yet another embodiment, a computer program product for preparing training data for a speech recognition model is provided. The computer program product includes a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to obtain a plurality of audio data sets, each audio data set having a different acoustic feature, sort sentences from the plurality of audio data sets so that similar sentences from different audio data sets are positioned closely to train the speech recognition model, and group the similar sentences from the different audio data sets into mini-batches, wherein each mini-batch of the mini-batches includes sentence pairs between different English dialects.

In one preferred aspect, the plurality of audio data sets are sampled from data pools each having the different acoustic feature so that the sampled audio data sets include a plurality of sets of similar sentences.

In another preferred aspect, a score penalty is presented to control a variety of sentences.

In yet another preferred aspect, the similar sentences are similar sentences with different dialects of a target language.

In yet another preferred aspect, the speech recognition model is a global speech recognition model for the target language.

In yet another preferred aspect, the similar sentences from the different audio data sets are grouped into mini-batches.

In yet another preferred aspect, each mini-batch of the mini-batches includes sentence pairs between different English dialects.

In yet another preferred aspect, each mini-batch of the mini-batches includes a similar amount of dialect data.

In yet another preferred aspect, a similarity between different English dialects of the similar sentences from different audio data sets is given by:

where F(a, b) is a distance between sentences a and b based on a word vector of n-word sequences and P(d) is a similarity-score dependent penalty not to compose biased text data.

In yet another preferred aspect, the similarity-score dependent penalty is given by:

DETAILED DESCRIPTION

Embodiments in accordance with the present invention provide methods and devices for composing an efficient training data sequence for a recurrent neural network transducer (RNN-T) based global English model. RNN-T models are usually trained with RNN-T loss, which aims to improve the log-likelihood of training data. However, few research work has investigated sequential training criteria for RNN-T models.

In the present circumstances, RNN-T models that are specific to each language are constructed separately. Even for the English language, several models are created independently to achieve a sufficient performance as a practical service, because of strong dialects (accents) in each English-speaking country. For example, American English (US), Australian English (AU), and British English (UK) models are currently deployed as an individual language. From the viewpoint of usability and maintenance cost, it is however practical to construct and deploy a single unified English model (referred to herein as a global English model (GEM)), which processes multiple English dialects with a single model. One beneficial aspect for GEM construction is to compose an efficient training data including multiple dialects with a good balance in terms of data size. Usually, those data sets are imbalanced.

The exemplary embodiments of the present invention alleviate such issues by introducing a method that advantageously does better training data collection (sorting and sampling) for accurate global English model construction.

FIG.1is a block/flow diagram of an exemplary system for organizing a training data sequence based on a metric that similar sentences with different dialects are positioned closely with a weak constraint of audio length for global English model (GEM) construction, in accordance with an embodiment of the present invention.

In a conventional method5, the data10is provided for random sampling12, data sorting14based on audio length, and model training16.

The data10can be, e.g., Australian English text or voice messages, British English text or voice messages, and American English text or voice messages.

In contrast, the exemplary embodiments introduce a method20where the data10is advantageously provided to a data sampler22for data sampling based on sentence similarity between dialects, then to a data sorter24for data sorting based on sentence similarity between dialects, and then model training26is performed.

The data10can be, e.g., Australian English text or voice messages, British English text or voice messages, and American English text or voice messages.

Thus, method20organizes a training data sequence based on a metric that similar sentences with different dialects are advantageously positioned closely with a weak constraint of audio length for global English model (GEM) construction. An advantage is that each mini-batch includes similar sentence pairs between different English dialects. Each mini-batch advantageously includes a similar-amount of dialect data. The same metric can also be applied to data sampling from a large data pool including realistic field data such as customer data of IBM Watson® speech to text (STT) for better GEM construction. How realistic big data can be efficiently leveraged to be organized by better training data of GEM is an advantage of this invention. A beneficial aspect for the data sampling is to introduce a score penalty to control a variety of sentences. This makes a prediction network training not to be overfitted to biased text caused by a strong word-sequence constraint.

IBM Watson® STT technology enables fast and accurate speech transcription in multiple languages for a variety of use cases, including, but not limited to, customer self-service, agent assistance, and speech analytics. IBM Watson® STT is an application programming interface (API) cloud service that enables a person to convert written text into natural sounding audio in a variety of languages and voices within an existing application on, e.g., Watson® assistant.

FIG.2is a block/flow diagram of an exemplary method for organizing a training data sequence based on a metric that similar sentences with different dialects are positioned closely with a weak constraint of audio length for global English model (GEM) construction, in accordance with an embodiment of the present invention.

At block30, a subset DBSof the base dialect is made by randomly selecting utterances from DB.

At block32, DBSLis made by sorting utterances in DBSwith a metric of audio length.

At block34, an unprocessed shortest utterance SiBSLis selected from DBSL.

At block36, set n=1.

At block38, an utterance with the highest similarity is extracted from other dialect Dnas a better training sample for the GEM construction.

At block40, it is determined whether n=N. If NO, then proceed to block42, where n is set to n+1. If YES, then proceed to block44.

At block44, it is determined whether all utterances in DBSLhave been proceed. If NO, then proceed back to block34. If YES, then the process ends.

DBis the training data set of the base dialect.

The smallest amount dialect training data is used as the base set, but not limited to this.

Dnis the training data sets of other dialects. (n=1 . . . N, where N is the number of other dialects).

The similarity between dialects is advantageously given as:

Where F(a, b) is the distance between sentences a and b based on word vector of n-word sequences and P(d) is a similarity-score dependent penalty not to compose biased text data.

FIG.3is a block/flow diagram of an exemplary method for preparing training data for a speech recognition model, in accordance with an embodiment of the present invention.

At block50, obtain a plurality of audio data sets, each audio data set having a different acoustic feature.

At block52, advantageously sort sentences from the plurality of audio data sets so that similar sentences from different audio data sets are positioned closely, while imposing a weak constraint on audio length, to train the speech recognition model.

Moreover, the plurality of audio data sets are advantageously sampled from data pools each having the different acoustic feature so that the sampled audio data sets include a plurality of sets of similar sentences, while presenting a score penalty to control a variety of sentences. Additionally, the similar sentences are similar sentences with different dialects of a target language and the speech recognition model is a global speech recognition model for the target language.

FIG.4illustrates exemplary data sorting by employing the exemplary method versus the conventional method, in accordance with an embodiment of the present invention.

Block60illustrates the random selection from each dialect.

The first few sentences are in Australian English (au), the next few sentences are in British English (uk), and the last few sentences are in American English (us).

Thus, the sentences are grouped by what type of English they are (e.g., au, uk, us), regardless of the words within the sentences or the length of the sentences or any other characteristics of the sentences.

Block70illustrates sentences sorted by audio length.

The first sentence (THANK YOU) is the shortest and listed at the top, whereas the last sentence (OKAY I JUST WANTED TO ASK YOU TO STAY ON THE LINE FOR A MEMOMET WE ARE HERE UNTIL NINE P M) is the longest and listed at the bottom.

Thus, the sentences are listed by length, regardless of any other factors.

Block80, according to the exemplary embodiments, advantageously sorts the sentences by similarity.

For example, the first group82includes 3 sentences that include the phrase “THANK YOU.” Since, the phrase “THANK YOU” is found in all 3 sentences, regardless of dialect or length, such sentences are grouped together as82(based only on similarity).

The second group84also includes 3 sentences. Each sentence includes the phrase “I WILL RING.” Since, the phrase “I WILL RING” is found in all 3 sentences, regardless of dialect or length, such sentences are grouped together as84(based only on similarity).

The third group86also includes 3 sentences. Each sentence includes the phrase “WANTS TO DO” or “WANTS TO KNOW.” Since, such phrases are similar and found in all 3 sentences, regardless of dialect or length, such sentences are grouped together as86(based only on similarity).

The fourth group88also includes 3 sentences. Each sentence includes the phrase “OKAY,” such as “OKAY WE'RE HERE” or “I AH OKAY USED” or “OKAY I JUST WANTED TO ASK.” Since, such phrases are similar and found in all 3 sentences, regardless of dialect or length, such sentences are grouped together as88(based only on similarity).

Therefore, similar sentences with different dialects are advantageously put close to each other, and thus grouped together (e.g., in mini-batches). In other words, the closeness or similarity of the words or phrases is analyzed and evaluated to determine groupings or mini-batches. Each group82,84,86,88can be referred to as a mini-batch. Mini-batches can include, e.g., 3 sentences. However, mini-batches can include anywhere from 3 to 10 sentences.

FIG.5illustrates a system for sorting sentences from a plurality of audio data sets, in accordance with an embodiment of the present invention.

In one example, a first audio data set90is obtained having an acoustic feature92, a second audio data set100is obtained having an acoustic feature102, and a third audio data set110is obtained having an acoustic feature112. Sentences from the audio datasets90,100,110are advantageously sorted, by a sorter115, for similarity or closeness to efficiently train the speech recognition model120. The similar sentences can be grouped into a plurality of mini-batches, as described above with reference toFIG.5.

Moreover, a weak constraint is imposed on the audio length, thus advantageously making the similarity feature or variable or parameter more dominant in determining the mini-batches.

FIG.6is a block/flow diagram of an exemplary processing system for organizing a training data sequence based on a metric that similar sentences with different dialects are positioned closely with a weak constraint of audio length for global English model (GEM) construction, in accordance with an embodiment of the present invention.

FIG.6depicts a block diagram of components of system200, which includes computing device205. It should be appreciated thatFIG.6provides only an illustration of one implementation and does not imply any limitations with regard to the environments in which different embodiments can be implemented. Many modifications to the depicted environment can be made.

Computing device205includes communications fabric202, which provides communications between computer processor(s)204, memory206, persistent storage208, communications unit210, and input/output (I/O) interface(s)212. Communications fabric202can be implemented with any architecture designed for passing data and/or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within a system. For example, communications fabric202can be implemented with one or more buses.

Memory206, cache memory216, and persistent storage208are computer readable storage media. In this embodiment, memory206includes random access memory (RAM)214. In another embodiment, the memory206can be flash memory. In general, memory206can include any suitable volatile or non-volatile computer readable storage media.

In some embodiments of the present invention, program225is included and operated by AI accelerator chip222as a component of computing device205. In other embodiments, program225is stored in persistent storage208for execution by AI accelerator chip222(to implement a training data sequence for RNN-T) in conjunction with one or more of the respective computer processors204via one or more memories of memory206. In this embodiment, persistent storage208includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk drive, persistent storage208can include a solid state hard drive, a semiconductor storage device, read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, or any other computer readable storage media that is capable of storing program instructions or digital information.

Communications unit210, in these examples, provides for communications with other data processing systems or devices, including resources of distributed data processing environment. In these examples, communications unit210includes one or more network interface cards. Communications unit210can provide communications through the use of either or both physical and wireless communications links. Deep learning program225can be downloaded to persistent storage208through communications unit210.

I/O interface(s)212allows for input and output of data with other devices that can be connected to computing system200. For example, I/O interface212can provide a connection to external devices218such as a keyboard, keypad, a touch screen, and/or some other suitable input device. External devices218can also include portable computer readable storage media such as, for example, thumb drives, portable optical or magnetic disks, and memory cards.

Display220provides a mechanism to display data to a user and can be, for example, a computer monitor.

FIG.7is a block/flow diagram of an exemplary cloud computing environment, in accordance with an embodiment of the present invention.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Referring now toFIG.7, illustrative cloud computing environment450is depicted for enabling use cases of the present invention. As shown, cloud computing environment450includes one or more cloud computing nodes410with which local computing devices used by cloud consumers, such as, for example, personal digital assistant (PDA) or cellular telephone454A, desktop computer454B, laptop computer454C, and/or automobile computer system454N can communicate. Nodes410can communicate with one another. They can be grouped (not shown) physically or virtually, in one or more networks, such as Private, Community, Public, or Hybrid clouds as described hereinabove, or a combination thereof. This allows cloud computing environment450to offer infrastructure, platforms and/or software as services for which a cloud consumer does not need to maintain resources on a local computing device. It is understood that the types of computing devices454A-N shown inFIG.7are intended to be illustrative only and that computing nodes410and cloud computing environment450can communicate with any type of computerized device over any type of network and/or network addressable connection (e.g., using a web browser).

FIG.8is a schematic diagram of exemplary abstraction model layers, in accordance with an embodiment of the present invention. It should be understood in advance that the components, layers, and functions shown inFIG.8are intended to be illustrative only and embodiments of the invention are not limited thereto. As depicted, the following layers and corresponding functions are provided:

Hardware and software layer560includes hardware and software components. Examples of hardware components include: mainframes561; RISC (Reduced Instruction Set Computer) architecture based servers562; servers563; blade servers564; storage devices565; and networks and networking components566. In some embodiments, software components include network application server software567and database software568.

Virtualization layer570provides an abstraction layer from which the following examples of virtual entities can be provided: virtual servers571; virtual storage572; virtual networks573, including virtual private networks; virtual applications and operating systems574; and virtual clients575.

Workloads layer590provides examples of functionality for which the cloud computing environment can be utilized. Examples of workloads and functions which can be provided from this layer include: mapping and navigation541; software development and lifecycle management592; virtual classroom education delivery593; data analytics processing594; transaction processing595; and training data sequence for RNN-T20.

Having described preferred embodiments of methods and systems for composing an efficient training data sequence for a recurrent neural network transducer (RNN-T) based global English model (which are intended to be illustrative and not limiting), it is noted that modifications and variations can be made by persons skilled in the art in light of the above teachings. It is therefore to be understood that changes may be made in the particular embodiments described which are within the scope of the invention as outlined by the appended claims. Having thus described aspects of the invention, with the details and particularity required by the patent laws, what is claimed and desired protected by Letters Patent is set forth in the appended claims.