System and method for automatic speech recognition using selection of speech models based on input characteristics

A method and system method for automatic speech recognition using selection of speech models based on input characteristics is disclosed herein. The method includes obtaining speech data from a speaker utilizing a microphone or an audio upload. The system and method select the best speech recognition model to automatically decode the input speech and continuously update models by updating/creating models in a database based on users speech abilities.

CROSS REFERENCE TO RELATED APPLICATION

Not Applicable

Not Applicable

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention generally relates to a speech recognition system.

Description of the Related Art

Speech recognition systems are built by training the system on input acoustic data and transcript speech corpus. This training process leads to a model which is later used to decode new speech into text. The input data results in the models performance on decoding speech, if a model is trained on adult native speakers, it will have difficulty decoding non native, noisy, children and other forms speech that differs from the training data. There is a need to produce an automatic speech recognition system that can handle various cases of speech including those from non-native speakers and children.

The prior art solution is to use speech utterance as input to adaptation and updating the speech model, language identification and select appropriate language speech recognizer, and speech input detection and call right model.

In one prior art solution, languages (and dialects) are considered independently, and a separate acoustic model is trained for each language from scratch. This solution uses deep learning to train a multilingual speech recognition, where some parts are shared among all languages, and other parts are trained separately.

In another prior art solution, an age group classifier is built using linear kernel Support Vector Machines] trained with the Sequential Minimal Optimisation (SMO) algorithm and this used to select which ASR system to use (children, adults, seniors).

In another prior art solution, the solution adapt/builds acoustic and language models based on users profiles distributed over a network. This isn't a system to select the model for speech recognition but to build the acoustic models based on user profiles.

In another prior art solution, the solution uses a tree-structured model for selecting a speaker-dependent model that best matches to the input speech. The solution helps build a more robust speech system that can select the appropriate model based on input speech.

In another prior art solution, the solution selects a speech model based on a length of speech.

In another prior art solution, controller configured to select an acoustic model from a library of acoustic models based on ambient noise in a cabin of the vehicle and operating parameters of the vehicle.

In another prior art solution, the solution selects a language speech system based on the score from language identification system.

BRIEF SUMMARY OF THE INVENTION

The present invention is an automatic speech recognition system and method that uses many speech models, and selects them based on the input data.

One aspect of the present invention is a method for automatic speech recognition using selection of speech models based on input characteristics. The method includes obtaining speech data from a speaker utilizing a microphone or an audio upload. The method also includes receiving speaker and environmental data. The method also includes extracting a plurality of sound features from the speech data. The method also includes classifying the speech data to generate a classified speech data. The method also includes selecting a speech recognition model for the classified speech data. The method also includes decoding the speech data into output text utilizing an automatic speech recognition system. The method also includes identifying a phoneme and a sound map for the classified speech data. The method also includes adapting at least one speech recognition model based on the text, phoneme and sound map.

Another aspect of the present invention is a system for automatic speech recognition using selection of speech models based on input characteristics. The system comprises a server comprising a processor, a sound feature extractor, a context profiler, a speech classifier, a database comprising speech model data, a plurality of speech models, and a speech recognition system. The speech data from a speaker is obtained utilizing a microphone or an audio upload. The context profiler is configured to process the speaker and environmental data. The sound feature extractor is configured to extract a plurality of sound features from the speech data. The speech classifier is configured to classify the speech data to generate a classified speech data. A server is configured to select a speech recognition model for the classified speech data. The automatic speech recognition system is configured to decode the speech data into output text. The server is configured to identify a phoneme and a sound map for the classified speech data. The server is configured to adapt at least one speech recognition model based on the text, phoneme and sound map.

DETAILED DESCRIPTION OF THE INVENTION

The speech recognition system of the present invention performs well for children, adults, native, non-native, noisy and more. The speech recognition system has a database of various speech models. These speech models are very specific with custom acoustic and language data. The system uses the input sound to determine which speech model is best to use. The system selects that speech model to decode and return the transcription. The system validates the transcription and updates the speech models.

The system selects the best speech recognition model to automatically decode the input speech and continuously updates models by updating/creating models in a database based on users speech abilities.

The process extracts speech features, combining the speech features with environmental context to select a speech model to decode speech, then updates and creates a new model that can increase the model inventory. The new models update, as well as get data from a sound map. The sound maps are inventories of phoneme, syllable, lexicon and corpus of specific models. Once a user has uttered enough data to create a sound model, a model will either be updated or a new model will be created.

The system and method preferably utilizes KID SENSE speech recognition for children, and dual language speech recognition used in KADHO English and MOCHU AI.

One embodiment is a system for automatic speech recognition using selection of speech models based on input characteristics. The system comprises a server comprising a processor, a sound feature extractor, a context profiler, a speech classifier, a database comprising speech model data, a plurality of speech models, and a speech recognition system. The speech data from a speaker is obtained utilizing a microphone or an audio upload. The context profiler is configured to process the speaker and environmental data. The sound feature extractor is configured to extract a plurality of sound features from the speech data. The speech classifier is configured to classify the speech data to generate a classified speech data. A server is configured to select a speech recognition model for the classified speech data. The automatic speech recognition system is configured to decode the speech data into output text. The server is configured to identify a phoneme and a sound map for the classified speech data. The server is configured to adapt at least one speech recognition model based on the text, phoneme and sound map.

The speaker and environmental data preferably comprises at least one of a physical location of the speaker, a native speaker, a language learner, a gender of the speaker, and an age of the speaker.

A sound feature extractor of the system extracts features of the sound that can be used by the classifier to learn how to classifier speech inputs. These features could be any combination of Mel Frequency Cepstral Coefficients, Filterbank Energies, Log Filterbank Energies, Spectral Subband Centroids, Zero Crossing Rate, Energy Entropy of Energy, Spectral Centroid Spectral Spread, Spectral Entropy, Spectral Flux, Spectral Rolloff, Chroma Vector Chroma Deviation and more.

At least one of a processor, memory, and GPU is used to extract the plurality of sound features.

A context profiler collects environmental input, user profile if it registered in an application, device location, device language settings, and accomplishments/progress/levels in the software transferring the sound. The context profiler is used to process the speaker and environmental data.

The Sound/Speech Classifier is preferably one of a I-Vector, Neural Network (Feedforward, CNN, RNN), Machine Learning (k Nearest Neighbor kN, Support Vector Machines, Random forests, Extra trees and Gradient boosting.

The database is used to store speech models and tags.

The Speech Model Data is audio, transcripts, and speech files needed for training and build a speech.

The speech recognition model is preferably selected from a plurality of speech recognition models stored on a digital storage medium.

The automatic speech recognition system is used to decode speech.

The speech models are acoustic and language models that are used by the speech recognition system to decode speech. The speech recognition system is HMM (hidden markov model), GMM (gaussian mixture), GMM-HMM, DNN (deep neural network)-HMM, RNN or any speech recognition model that can decode speech based on training.

The speech model update/creation is based on the user speech profile, and sound map database. Once enough phrases have been uttered to cover the sounds required to cover the spectrum determined from an algorithm like the phoneme transitional and probability matrices, the user utterance is used to either update or create a model based on what is available in the database.

Another embodiment is a method for automatic speech recognition using selection of speech models based on input characteristics. The method includes obtaining speech data from a speaker utilizing a microphone or an audio upload, which is retrieving audio in a digital format. The method also includes receiving speaker and environmental data, which is understanding the person behind the speech, are they in a car, classroom, native speaker, language learner, male/female, child/adult. This data is transferred in digital form through memory. The method also includes extracting a plurality of sound features from the speech data, which is obtaining sound features that can be used by the classifier to classify the speech. The method utilizes Utilize computer processors, memory and/or gpu to extract sound features. The method also includes classifying the speech data to generate a classified speech data utilizing computer processors, memory and/or gpu to extract sound features. The method also includes selecting a speech recognition model for the classified speech data, which in combination with the classifier and user data, uses machine learning and/or deep learning to select the appropriate speech model. The models are saved on a digital storage medium. The database can contain digital fingerprints or tags linking to the stored models. The method also includes decoding the speech data into output text utilizing an automatic speech recognition system, which is decoding the speech into text using a trained model. The method utilizes CPU, GPU and the models data stored on the memory to decode the speech. The method also includes identifying a phoneme and a sound map for the classified speech data, which is using user data and the Phoneme Transitional Probability and Matrices to identify the phoneme and syllable range of the individual. The method also includes adapting at least one speech recognition model based on the text, phoneme and sound map. The data can be retrained and a previous model can be adopted to create a new one to increase the diversity of the model database. Data is sent to the server and the model in storage is updated.

The method further comprises analyzing the output text using confidence scoring.

The input of speech has an output of text.

The input of acoustic data has an output of a speech model.

The input Environmental Data (user info, location, language) has an output of a speech model.

The input of speech to text process has an output of more diverse speech models (acoustic data[phonemes, sounds] and language data[words, sentences]).

The input of Lexicon, phoneme, syllable, and speech corpus has an output of Refined acoustic and language models.

FIG. 1is a block diagram of a method100for automatic speech recognition using selection of speech models based on input characteristics. Speech is sent to a speech recognition system at block101. At block102, features are extracted from the speech. At block103, the speech is classified and sent to the database at block104. At block105, an acoustic model is selected and sent to the speech recognition system at block101. At block106, speech factors are collected and sent to the database at107. At block108, a language model is selected.

FIG. 2is a block diagram of a method200for automatic speech recognition using selection of speech models based on input characteristics. Speech is sent to a speech recognition system at block201. At block202, features are extracted from the speech. At block203, the speech is classified and sent to the database at block205. At block204, speech factors are collected and sent to the database at205. At block206, an acoustic model is selected and sent to the speech recognition system at block201.

FIG. 3is a block diagram of a method300for automatic speech recognition using selection of speech models based on input characteristics. At block351, a microphone audio or digital audio is uploaded to the system. At block352, the features are extracted from the audio. The data is sent to a database353, and the data is processed at block354. At block356, the speech is decoded into text. At block357, the text is displayed. At block358, the speech is analyzed on a computer. At block359, the speech model data is updated. At block360, the computer performs training on acoustic and language models. The models are updated after training and sent to database353.

FIG. 4is a block diagram of a score approach process400for a method for automatic speech recognition using selection of speech models based on input characteristics. At block451, speech is inputted into the system. At block452, a speech recognition software is run on the inputted speech. At block453, text is outputted. At block454, the output text is analyzed using confidence scores on each word and a speech model is selected based on what is best for those words.

FIG. 5is a block diagram of a classification approach process500for a method for automatic speech recognition using selection of speech models based on input characteristics. At block501, speech is inputted into the system. At block502, features are extracted. At block503, environmental context is analyzed. At block504, the speech is classified. At block505, the appropriated speech recognition model is selected. At block506, the text is outputted.

FIG. 6is a block diagram of a parallel classification approach process600for a method for automatic speech recognition using selection of speech models based on input characteristics. At block601, speech is inputted into the system. At block602, features are extracted. At block603, environmental context is analyzed. At block604, the speech is classified. At block605, speech recognition is run on the inputted speech. At block606, the appropriated speech recognition model is selected. At block607, the text is outputted. At block608, the final text is selected.

FIG. 7is a block diagram of adapting and improving models process700for a method for automatic speech recognition using selection of speech models based on input characteristics. At block701, speech is inputted into the system. At block702, algorithms are run to selected the most appropriate speech recognition model. At block703, speech recognition models and classifiers are updated based on the new data. At block704, the new/updated model is added to the database to increase the diversity of the database of speech recognition models.

FIG. 8is a block diagram of adapting and improving models process800for a method for automatic speech recognition using selection of speech models based on input characteristics. At block801, speech outputted from a speech recognition system is validated using other speech recognition systems or human interaction. At block802, this data is sent to a speech model files. At block803, a soundmap is used. At block804, further processing using phone transitional probability and matrices is performed on the data. At block805, the speech model is updated. At block806, the updated speech model is sent to the database of speech models. At block807, the updated speech model is classified.

FIG. 9is a block diagram of a communication flow900of the method for automatic speech recognition using selection of speech models based on input characteristics. At block901, an audio file is decoded. At block902, a speech model is selected. At block903, an acoustic model904and a language model905are used in the speech model to decode the audio. At block907, the text is outputted. At block908, the performance of the output text and audio is verified. At block909, the text and audio is saved.

As shown inFIG. 10, a typical mobile communication device25includes an accelerometer301, a headphone jack302, a microphone jack303, a speaker304, a GPS chipset305, a Bluetooth component306, a Wi-Fi component307, a 3G/4G component308, a Baseband Processor (for radio control)309, an applications (or main) processor310, a JTAG (debugger)311, a SDRAM memory312, a Flash memory313, SIM card314, LCD display315, a camera316, a power management circuit317and a battery or power source318.

The mobile devices utilized with the present invention preferably include mobile phones, smartphones, tablet computers, PDAs and the like. Examples of smartphones and the device vendors include the IPHONE® smartphone from Apple, Inc., the DROID® smartphone from Motorola Mobility Inc., GALAXY S® smartphones from Samsung Electronics Co., Ltd., and many more. Examples of tablet computing devices include the IPAD® tablet from Apple Inc., and the XOOM™ tablet from Motorola Mobility Inc.

A mobile communication service provider (aka phone carrier) of the customer such as VERIZON, AT&T, SPRINT, T-MOBILE, and the like mobile communication service providers, provide the communication network for communication to the mobile communication device of the end user.

FIG. 11shows components of a server40. Components of the server40includes a CPU component401, a graphics component402, PCI/PCI Express403, a memory404, a non-removable storage407, a removable storage408, a network interface409, including one or more connections to a fixed network, and SQL databases45a-45d. Included in the memory404, is an operating system405, a SQL server406or other database engine, and application programs/software410. The server also includes at least one computer program configured to receive data uploads and store the data uploads in the SQL database. Alternatively, the SQL server can be installed in a separate server from the venue server.

Communication protocols utilized with the present invention may preferably include but are not limited to XML, HTTP, TCP/IP, Serial, UDP, FTP, Web Services, WAP, SMTP, SMPP, DTS, Stored Procedures, Import/Export, Global Positioning Triangulation, IM, SMS, MMS, GPRS and Flash. The databases used with the system preferably include but are not limited to MSSQL, Access, MySQL, Progress, Oracle, DB2, Open Source DBs and others. Operating system used with the system preferably include Microsoft 2010, XP, Vista, 200o Server, 2003 Server, 2008 Server, Windows Mobile, Linux, Android, Unix, I series, AS 400 and Apple OS.