Patent Description:
Voice-based interfaces are being used more and more often as a mechanism for supporting interactions between humans and machines. These types of interfaces often include an automatic speech recognition (ASR) system, which is designed to convert speech into text. The text can then be displayed, communicated to other users, further processed or used to perform one or more functions, or used in any other suitable manner. This type of functionality is common in various applications, such as voice-based digital personal assistants on mobile electronic devices or other electronic devices.

Document <NPL> discloses large vocabulary continuous speech recognition, LVCSR, based language identification. <CIT> discloses automatic spoken language identification based on phoneme sequence patterns. Document <NPL>. <NPL> discloses Language Identification using parallel phone recognizers followed by language modelling. <CIT> concerns dialect-specific acoustic language modelling and speech recognition.

This disclosure provides a system and method for multi-spoken language detection.

In a first embodiment, a method includes performing, using at least one processor, feature extraction of input audio data to identify extracted features associated with the input audio data. The method also includes detecting, using the at least one processor, a language associated with each of multiple portions of the input audio data by processing the extracted features using a plurality of language models, where each language model is associated with a different language. In addition, the method includes directing, using the at least one processor, each portion of the input audio data to one of a plurality of automatic speech recognition (ASR) models based on the language associated with the portion of the input audio data.

In a second embodiment, an electronic device includes at least one memory and at least one processor operatively coupled to the at least one memory. The at least one processor is configured to perform feature extraction of input audio data to identify extracted features associated with the input audio data. The at least one processor is also configured to detect a language associated with each of multiple portions of the input audio data by processing the extracted features using a plurality of language models, where each language model is associated with a different language. The at least one processor is further configured to direct each portion of the input audio data to one of a plurality of ASR models based on the language associated with the portion of the input audio data.

In a third embodiment, a non-transitory machine-readable medium contains instructions that when executed cause at least one processor of an electronic device to perform feature extraction of input audio data to identify extracted features associated with the input audio data. The medium also contains instructions that when executed cause the at least one processor to detect a language associated with each of multiple portions of the input audio data by processing the extracted features using a plurality of language models, where each language model is associated with a different language. The medium further contains instructions that when executed cause the at least one processor to direct each portion of the input audio data to one of a plurality of ASR models based on the language associated with the portion of the input audio data.

For a more complete understanding of this disclosure and its advantages, reference is now made to the following description taken in conjunction with the accompanying drawings, in which like reference numerals represent like parts:.

As used here, terms and phrases such as "have," "may have," "include," or "may include" a feature (like a number, function, operation, or component such as a part) indicate the existence of the feature and do not exclude the existence of other features. Also, as used here, the phrases "A or B," "at least one of A and/or B," or "one or more of A and/or B" may include all possible combinations of A and B. For example, "A or B," "at least one of A and B," and "at least one of A or B" may indicate all of (<NUM>) including at least one A, (<NUM>) including at least one B, or (<NUM>) including at least one A and at least one B. Further, as used here, the terms "first" and "second" may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate different user devices from each other, regardless of the order or importance of the devices. A first component may be denoted a second component and vice versa without departing from the scope of this disclosure.

It will be understood that, when an element (such as a first element) is referred to as being (operatively or communicatively) "coupled with/to" or "connected with/to" another element (such as a second element), it can be coupled or connected with/to the other element directly or via a third element. In contrast, it will be understood that, when an element (such as a first element) is referred to as being "directly coupled with/to" or "directly connected with/to" another element (such as a second element), no other element (such as a third element) intervenes between the element and the other element.

As used here, the phrase "configured (or set) to" may be interchangeably used with the phrases "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of" depending on the circumstances. The phrase "configured (or set) to" does not essentially mean "specifically designed in hardware to. " Rather, the phrase "configured to" may mean that a device can perform an operation together with another device or parts. For example, the phrase "processor configured (or set) to perform A, B, and C" may mean a generic-purpose processor (such as a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device or a dedicated processor (such as an embedded processor) for performing the operations.

The terms and phrases as used here are provided merely to describe some embodiments of this disclosure but not to limit the scope of other embodiments of this disclosure. All terms and phrases, including technical and scientific terms and phrases, used here have the same meanings as commonly understood by one of ordinary skill in the art to which the embodiments of this disclosure belong. It will be further understood that terms and phrases, such as those defined in commonly-used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined here. In some cases, the terms and phrases defined here may be interpreted to exclude embodiments of this disclosure.

Examples of an "electronic device" according to embodiments of this disclosure may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (such as smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic accessory, an electronic tattoo, a smart mirror, or a smart watch). Other examples of an electronic device include a smart home appliance. Examples of the smart home appliance may include at least one of a television, a digital video disc (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washer, a drier, an air cleaner, a set-top box, a home automation control panel, a security control panel, a TV box (such as SAMSUNG HOMESYNC, APPLETV, or GOOGLE TV), a smart speaker or speaker with an integrated digital assistant (such as SAMSUNG GALAXY HOME, APPLE HOMEPOD, or AMAZON ECHO), a gaming console (such as an XBOX, PLAYSTATION, or NINTENDO), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame. Still other examples of an electronic device include at least one of various medical devices (such as diverse portable medical measuring devices (like a blood sugar measuring device, a heartbeat measuring device, or a body temperature measuring device), a magnetic resource angiography (MRA) device, a magnetic resource imaging (MRI) device, a computed tomography (CT) device, an imaging device, or an ultrasonic device), a navigation device, a global positioning system (GPS) receiver, an event data recorder (EDR), a flight data recorder (FDR), an automotive infotainment device, a sailing electronic device (such as a sailing navigation device or a gyro compass), avionics, security devices, vehicular head units, industrial or home robots, automatic teller machines (ATMs), point of sales (POS) devices, or Internet of Things (IoT) devices (such as a bulb, various sensors, electric or gas meter, sprinkler, fire alarm, thermostat, street light, toaster, fitness equipment, hot water tank, heater, or boiler). Other examples of an electronic device include at least one part of a piece of furniture or building/structure, an electronic board, an electronic signature receiving device, a projector, or various measurement devices (such as devices for measuring water, electricity, gas, or electromagnetic waves). Note that, according to various embodiments of this disclosure, an electronic device may be one or a combination of the above-listed devices. According to some embodiments of this disclosure, the electronic device may be a flexible electronic device. The electronic device disclosed here is not limited to the above-listed devices and may include new electronic devices depending on the development of technology.

In the following description, electronic devices are described with reference to the accompanying drawings, according to various embodiments of this disclosure. As used here, the term "user" may denote a human or another device (such as an artificial intelligent electronic device) using the electronic device.

Definitions for other certain words and phrases may be provided throughout this patent document.

None of the description in this application should be read as implying that any particular element, step, or function is an essential element that must be included in the claim scope. The scope of patented subject matter is defined only by the claims. Use of any other term, including without limitation "mechanism," "module," "device," "unit," "component," "element," "member," "apparatus," "machine," "system," "processor," or "controller," within a claim is understood by the Applicant to refer to structures known to those skilled in the relevant art. <FIG>, discussed below, and the various embodiments of this disclosure are described with reference to the accompanying drawings. However, it should be appreciated that this disclosure is not limited to these embodiments, and all changes and/or equivalents or replacements thereto also belong to the scope of this disclosure. The same or similar reference numerals may be used to refer to the same or similar elements throughout the specification and the drawings.

As noted above, voice-based interfaces are being used more and more often to support interactions between humans and machines. These types of interfaces often include an automatic speech recognition (ASR) system, which converts speech into text. In ASR systems, determining the language that is being spoken by a user is typically a prerequisite for performing automatic speech recognition operations. However, there are a number of technical challenges when it comes to performing automatic language identification, such as overlapping phonemes among different languages, overlapping characteristics of different speakers in the same language (such as native versus non-native speakers or male versus female speakers), and matching broad phoneme distributions between different languages. In some prior approaches, a user is simply given the option of selecting a specific language to be used, such as during setup of a mobile electronic device or other device. In these approaches, the user is often limited to use of that specific language, and the user typically cannot switch between different languages (at least not very easily and often not without repeating the setup process).

This disclosure provides techniques for multi-spoken language detection that, among other things, enable the automatic selection and use of one or more language-specific ASR models to process an input. As described in more detail below, a device receives input data, such as audio data associated with a user utterance. The device performs feature extraction to identify features that are associated with the input data, such as by identifying features of phonemes from the input data. The extracted features are processed using multiple language models (also called language-specific acoustic models) to identify a language associated with each of multiple portions of the input data. Depending on the input data, all portions of the input data may be associated with a single language, or different portions of the input data may be associated with different languages. Each portion of the input data may be routed to a language-specific ASR model for further processing based on the identified language of that portion, and each ASR model is tuned (trained) for a specific language. In some embodiments, the language models may be derived from the ASR models and may process a smaller number of features than the ASR models, since the identification of a language being used in the input data may involve fewer features than the recognition of actual words or phrases contained in the input data.

In this way, these techniques allow an electronic device or other device to receive and process input audio data more effectively since the language or languages associated with the input audio data can be identified automatically. Also, these techniques allow an electronic device or other device to seamlessly transition between processing different languages, without requiring a user to select different languages or re-execute a device setup process to select a different language. This may be particularly useful for users who speak multiple languages and who might switch between different languages in the same input audio data. In addition, the ability to improve the detection of spoken languages can help ASR-based systems provide better responses to user inputs and can enable systems to serve a wider population of potential users.

As a particular example of this functionality, the accuracy of an ASR system is often highly-dependent on the training of the ASR system and the conditions under which input data is provided to the ASR system for processing. While training data used to train an ASR system often contains pure speech from a specific language, input data provided to the ASR system during real-world use seldom has the same characteristics. For instance, users who speak multiple languages may often interact with a device using different languages. Using the techniques described below, one or more language-specific ASR models can be used for each user input depending on the language or languages of that user input. Thus, in some embodiments, the automated detection of a spoken language or languages associated with a particular input can be performed as part of a pre-processing operation in order to identify the language(s) and use the appropriate ASR model(s) for the identified language(s).

<FIG> illustrates an example network configuration <NUM> in accordance with this disclosure. As shown in <FIG>, according to embodiments of this disclosure, an electronic device <NUM> is included in the network configuration <NUM>. The electronic device <NUM> may include at least one of a bus <NUM>, a processor <NUM>, a memory <NUM>, an input/output (I/O) interface <NUM>, a display <NUM>, a communication interface <NUM>, or an event processing module <NUM>. In some embodiments, the electronic device <NUM> may exclude at least one of the components or may add another component.

The bus <NUM> may include a circuit for connecting the components <NUM>-<NUM> with one another and transferring communications (such as control messages and/or data) between the components. The processor <NUM> may include one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor <NUM> may perform control on at least one of the other components of the electronic device <NUM> and/or perform an operation or data processing relating to communication. As described in more detail below, the processor <NUM> may analyze input audio data, automatically identify one or more languages used in the input audio data, and process the input audio data based on the identified language(s).

The memory <NUM> may include a volatile and/or non-volatile memory. For example, the memory <NUM> may store commands or data related to at least one other component of the electronic device <NUM>. According to embodiments of this disclosure, the memory <NUM> may store software and/or a program <NUM>. The program <NUM> may include, for example, a kernel <NUM>, middleware <NUM>, an application programming interface (API) <NUM>, and/or an application program (or "application") <NUM>. At least a portion of the kernel <NUM>, middleware <NUM>, or API <NUM> may be denoted an operating system (OS).

The kernel <NUM> may control or manage system resources (such as the bus <NUM>, processor <NUM>, or memory <NUM>) used to perform operations or functions implemented in other programs (such as the middleware <NUM>, API <NUM>, or application program <NUM>). The kernel <NUM> may provide an interface that allows the middleware <NUM>, API <NUM>, or application <NUM> to access the individual components of the electronic device <NUM> to control or manage the system resources. The application <NUM> includes one or more applications for processing input data based on automated language detection as discussed below. These functions can be performed by a single application or by multiple applications that each carries out one or more of these functions. The middleware <NUM> may function as a relay to allow the API <NUM> or the application <NUM> to communicate data with the kernel <NUM>, for example. A plurality of applications <NUM> may be provided. The middleware <NUM> may control work requests received from the applications <NUM>, such as by allocating the priority of using the system resources of the electronic device <NUM> (such as the bus <NUM>, processor <NUM>, or memory <NUM>) to at least one of the plurality of applications <NUM>. The API <NUM> is an interface allowing the application <NUM> to control functions provided from the kernel <NUM> or the middleware <NUM>. For example, the API <NUM> may include at least one interface or function (such as a command) for file control, window control, image processing, or text control.

The input/output interface <NUM> may serve as an interface that may, for example, transfer commands or data input from a user or other external devices to other component(s) of the electronic device <NUM>. Further, the input/output interface <NUM> may output commands or data received from other component(s) of the electronic device <NUM> to the user or the other external device.

The display <NUM> may include, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a quantum light emitting diode (QLED) display, a microelectromechanical systems (MEMS) display, or an electronic paper display. The display <NUM> can also be a depth-aware display, such as a multi-focal display. The display <NUM> may display various contents (such as text, images, videos, icons, or symbols) to the user. The display <NUM> may include a touchscreen and may receive, for example, a touch, gesture, proximity, or hovering input using an electronic pen or a body portion of the user.

The communication interface <NUM> may set up communication between the electronic device <NUM> and an external electronic device (such as a first electronic device <NUM>, a second electronic device <NUM>, or a server <NUM>). For example, the communication interface <NUM> may be connected with a network <NUM> or <NUM> through wireless or wired communication to communicate with the external electronic device.

The wireless communication may use at least one of, for example, long term evolution (LTE), long term evolution-advanced (LTE-A), code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a cellular communication protocol. The wired connection may include at least one of, for example, universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard <NUM> (RS-<NUM>), or plain old telephone service (POTS). The network <NUM> or <NUM> may include at least one communication network, such as a computer network (like a local area network (LAN) or wide area network (WAN)), the Internet, or a telephone network.

The first external electronic device <NUM> or the second external electronic device <NUM> may be a wearable device or an electronic device <NUM>-mountable wearable device (such as a head mounted display (HMD)). When the electronic device <NUM> is mounted in an HMD (such as the electronic device <NUM>), the electronic device <NUM> may detect the mounting in the HMD and operate in a virtual reality mode. When the electronic device <NUM> is mounted in the electronic device <NUM> (such as the HMD), the electronic device <NUM> may communicate with the electronic device <NUM> through the communication interface <NUM>. The electronic device <NUM> may be directly connected with the electronic device <NUM> to communicate with the electronic device <NUM> without involving with a separate network.

The first and second external electronic devices <NUM> and <NUM> each may be a device of the same type or a different type from the electronic device <NUM>. According to embodiments of this disclosure, the server <NUM> may include a group of one or more servers. Also, according to embodiments of this disclosure, all or some of the operations executed on the electronic device <NUM> may be executed on another or multiple other electronic devices (such as the electronic devices <NUM> and <NUM> or server <NUM>). Further, according to embodiments of this disclosure, when the electronic device <NUM> should perform some function or service automatically or at a request, the electronic device <NUM>, instead of executing the function or service on its own or additionally, may request another device (such as electronic devices <NUM> and <NUM> or server <NUM>) to perform at least some functions associated therewith. The other electronic device (such as electronic devices <NUM> and <NUM> or server <NUM>) may execute the requested functions or additional functions and transfer a result of the execution to the electronic device <NUM><NUM>. The electronic device <NUM> may provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique may be used, for example.

While <FIG> shows that the electronic device <NUM> includes the communication interface <NUM> to communicate with the external electronic device <NUM> or <NUM> or server <NUM> via the network(s) <NUM> and <NUM>, the electronic device <NUM> may be independently operated without a separate communication function, according to embodiments of this disclosure. Also, note that the electronic device <NUM> or <NUM> or the server <NUM> could be implemented using a bus, a processor, a memory, an I/O interface, a display, a communication interface, and an event processing module (or any suitable subset thereof) in the same or similar manner as shown for the electronic device <NUM>.

The server <NUM> may operate to drive the electronic device <NUM> by performing at least one of the operations (or functions) implemented on the electronic device <NUM>. For example, the server <NUM> may include an event processing server module (not shown) that may support the event processing module <NUM> implemented in the electronic device <NUM>. The event processing server module may include at least one of the components of the event processing module <NUM> and perform (or instead perform) at least one of the operations (or functions) conducted by the event processing module <NUM>. The event processing module <NUM> may process at least part of the information obtained from other elements (such as the processor <NUM>, memory <NUM>, input/output interface <NUM>, or communication interface <NUM>) and may provide the same to the user in various manners.

While the event processing module <NUM> is shown to be a module separate from the processor <NUM> in <FIG>, at least a portion of the event processing module <NUM> may be included or implemented in the processor <NUM> or at least one other module, or the overall function of the event processing module <NUM> may be included or implemented in the processor <NUM> shown or another processor. The event processing module <NUM> may perform operations according to embodiments of this disclosure in interoperation with at least one program <NUM> stored in the memory <NUM>.

Although <FIG> illustrates one example of a network configuration <NUM>, various changes may be made to <FIG>. For example, the network configuration <NUM> could include any number of each component in any suitable arrangement. In general, computing and communication systems come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular configuration. Also, while <FIG> illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

<FIG> illustrates an example architecture <NUM> for multi-spoken language detection in accordance with this disclosure. For ease of explanation, the architecture <NUM> shown in <FIG> may be described as being implemented using the electronic device <NUM> in the network configuration <NUM> shown in <FIG>. However, the architecture <NUM> may be used by any suitable device(s) and in any suitable system(s), such as by the server <NUM> in the network configuration <NUM>.

As shown in <FIG>, the architecture <NUM> generally operates to receive an audio-based input <NUM> and generate a text-based output <NUM> representing the audio-based input <NUM>. The audio-based input <NUM> may represent any suitable audio information containing one or more spoken words. For example, the audio-based input <NUM> may represent a digitized (and possibly pre-processed) version of a user utterance, which in some cases may be captured by a microphone or other component of the electronic device <NUM>. The audio-based input <NUM> may include one word or multiple words spoken by at least one user, and a single language or multiple languages may be used by the user(s) when speaking. The text-based output <NUM> represents the audio-based input <NUM> converted into textual form, which is accomplished here using automatic speech recognition. Thus, the architecture <NUM> can process the audio-based input <NUM> to identify the language(s) used in the audio-based input <NUM> and convert words and phrases contained in the audio-based input <NUM> into text in the text-based output <NUM>.

In the architecture <NUM>, the audio-based input <NUM> is provided to a feature extractor <NUM>, which generally operates to process the audio-based input <NUM> and extract various features from the audio-based input <NUM>. The extracted features may represent any suitable characteristics of the audio-based input <NUM>. In some embodiments, the extracted features include phonemes contained in the audio-based input <NUM>. Phonemes represent units of a phonetic system of a language that are perceived to be distinctive sounds in the language.

Words in many languages can be represented using a common set of phonemes, although the usage of those phonemes varies widely based on the language. Thus, the feature extractor <NUM> can extract phoneme features from the audio-based input <NUM> in order to facilitate both (i) the identification of the language or languages used in the audio-based input <NUM> and (ii) the conversion of the audio-based input <NUM> into the text-based output <NUM> through recognition of words and phrases contained in the audio-based input <NUM>.

The feature extractor <NUM> may use any suitable technique for extracting phoneme features or other features from audio-based input <NUM>. For example, the feature extractor <NUM> may operate based on a wavelet decomposition technique or a convolutional neural network technique. Various approaches have been developed for extracting features from audio input data, and additional approaches are sure to be developed in the future. This disclosure is not limited to any specific type of feature extraction technique or any particular implementation of the feature extractor <NUM>. Also, the feature extractor <NUM> may express the extracted features in any suitable format. In some embodiments, for instance, extracted features may have the form of Mel-frequency cepstral coefficients (MFCCs), although the extracted features may be defined in any other suitable manner.

The extracted features of the audio-based input <NUM> are provided from the feature extractor <NUM> to a language-specific acoustic model scoring function <NUM>, which generally operates to identify the likelihood that various portions of the audio-based input <NUM> are from specific languages. In this example, the scoring function <NUM> includes or is used in conjunction with multiple language models <NUM> (also referred to as language-specific acoustic models). The language models <NUM> are associated with different languages (or possibly different dialects of the same language, which are treated as being different languages here). In some embodiments, each language model <NUM> is trained or otherwise configured to calculate the probability that each portion of the audio-based input <NUM> is from the language on which that language model <NUM> has been trained or is otherwise associated. Thus, for example, the first language model <NUM> (LM <NUM>) may receive a first portion of the audio-based input <NUM> and generate a probability that the first portion of the audio-based input <NUM> is from a first language, the second language model <NUM> (LM <NUM>) may receive the first portion of the audio-based input <NUM> and generate a probability that the first portion of the audio-based input <NUM> is from a second language, and so on. This process can be performed by each language model <NUM> in order to generate probabilities that the first portion of the audio-based input <NUM> are from various languages, and this may occur in parallel so that the first portion of the audio-based input <NUM> is processed by all of the language models <NUM> at the same time or in an overlapping manner. Moreover, this process can be performed for second, third, and other portions of the audio-based input <NUM>. The outputs from the scoring function <NUM> represent a collection of probabilities for all portions of the audio-based input <NUM> across all supported languages.

The language-specific acoustic model scoring function <NUM> can support any suitable technique for generating probabilities or otherwise determining the likelihood that various portions of audio-based input <NUM> are associated with various languages. For example, in some embodiments, each of the language models <NUM> may define multiple states and include interstate and intrastate transitions for moving between the states. Each state can apply a particular function to an input data segment in order to generate a probability of that input data segment being in a particular language. As a particular example, in some cases, the output of each language model <NUM> may be expressed as P(phoneklang-y | xi), where P( <IMG> ) represents the calculated probability that a specific phoneme phonek for a specific language lang-y is present in a particular segment xi of the audio-based input <NUM>. Each language model <NUM> can therefore be used to calculate a probability for each phoneme associated with a specific language, and this can be performed for each portion (segment) of the audio-based input <NUM>.

A feature concatenator <NUM> combines the outputs from the scoring function <NUM> to generate feature vectors. The feature vectors here may be generated in any suitable manner. For example, the feature concatenator <NUM> may accumulate the probabilities that are output from the scoring function <NUM> over time to create a matrix of values, which can then be further processed as described below. Note, however, that the feature concatenator <NUM> may combine the outputs from the scoring function <NUM> to generate feature vectors in any other suitable manner. Also, in some embodiments, the feature concatenator <NUM> may combine outputs from the scoring function <NUM> with the original extracted features generated by the feature extractor <NUM> (which may be referred to as "raw" features).

The feature vectors produced by the feature concatenator <NUM> are provided to a neural classification model <NUM>, which processes the feature vectors to identify the language or languages used in the audio-based input <NUM>. The neural classification model <NUM> may use any suitable machine learning technique to process feature vectors and identify languages based on the feature vectors. For example, the neural classification model <NUM> may include at least one convolutional neural network and associated layers, such as normalization and pooling layers, for processing feature vectors. One example implementation of the neural classification model <NUM> is shown in <FIG>, which is described below. However, it should be noted that the specific implementation of the neural classification model <NUM> shown in <FIG> is for illustration only, and the neural classification model <NUM> may be implemented in any other suitable manner.

The neural classification model <NUM> uses the identified language or languages that are associated with the audio-based input <NUM> to control an ASR engine <NUM>, which generally operates to convert the audio-based input <NUM> (or specific portions thereof) into text. Here, the ASR engine <NUM> includes or is used in conjunction with multiple ASR models <NUM>, which are associated with different languages. Each ASR model <NUM> can be used by the ASR engine <NUM> to convert at least part of the audio-based input <NUM> into text in the specific language associated with that ASR model <NUM>. Thus, if the neural classification model <NUM> determines that a single specific language is used in the audio-based input <NUM>, the neural classification model <NUM> can direct the audio-based input <NUM> to the ASR model <NUM> associated with that specific language. If the neural classification model <NUM> determines that multiple specific languages are used in the audio-based input <NUM>, the neural classification model <NUM> can direct different portions of the audio-based input <NUM> to the different ASR models <NUM> associated with those specific languages. The ASR engine <NUM> can then use that specific ASR model <NUM> or those specific ASR models <NUM> to generate the text-based output <NUM> for the audio-based input <NUM>.

In this way, the architecture <NUM> is able to automatically identify the language or languages being used in the audio-based input <NUM> and to automatically route the audio-based input <NUM> or portions thereof to the appropriate ASR model(s) <NUM>. Among other things, this allows the same user or different users of an electronic device <NUM> to interact with the electronic device <NUM> using different languages. This also allows the same user to switch between different languages in the same audio-based input <NUM>. In other words, the architecture <NUM> is able to seamlessly switch between different languages with little or no user input (other than the user or users switching between different languages when speaking).

Note that the features used by the scoring function <NUM> to support language identification may include at least some of the same features used by the ASR engine <NUM> to support language recognition (meaning the recognition of specific words spoken in a specific language). As a result, the language models <NUM> used by the scoring function <NUM> may be similar in form to the ASR models <NUM> used by the ASR engine <NUM>. However, as noted above, the language models <NUM> used by the scoring function <NUM> may require the use of fewer extracted features to perform language identification as compared to the number of extracted features used by the ASR models <NUM> to perform language recognition. As a particular example, in some cases, over one thousand features may be used by the ASR engine <NUM> to perform language recognition, while a smaller set of features (such as one or several hundred features) may be used by the language models <NUM> to perform language identification.

In some embodiments, to support the use of fewer features in the language models <NUM>, a knowledge distillation function <NUM> is used in or in conjunction with the architecture <NUM>. The knowledge distillation function <NUM> operates to take a larger trained model (such as an ASR model <NUM>) and train a smaller model (such as an associated language model <NUM>) to mimic the behavior of or the results obtained from the larger model. In other words, the knowledge distillation function <NUM> can take an ASR model <NUM> that has been trained for language recognition of a particular language and generate a smaller language model <NUM> trained for language identification of that same language. This allows the language models <NUM> to be generated based on trained ASR models <NUM>. Note, however, that the language models <NUM> may be trained in any other suitable manner. The knowledge distillation function <NUM> may use any suitable technique for generating smaller models based on larger trained models. The ASR models <NUM> can also be trained in any suitable manner. For example, each ASR model <NUM> may be trained by providing known words and phrases in a specific language to the feature extractor <NUM> and training the ASR model <NUM> associated with that specific language to recognize the known words and phrases. Various approaches have been developed for training models to recognize words and phrases in various languages, and additional approaches are sure to be developed in the future. This allows additional languages to be easily added to the architecture <NUM>, such as by training an additional ASR model <NUM> for a new language and then using the knowledge distillation function <NUM> to generate a language model <NUM> for the new language.

The architecture <NUM> shown in <FIG> and described above may be used in any suitable manner. For instance, one example use case of the architecture <NUM> is in an electronic device <NUM> or other device used in a home, office, or other setting where different people speak different languages. The electronic device <NUM> may receive and process utterances from various users in different languages and provide responses based on the identified text associated with the user utterances. As a specific example, an electronic device <NUM> can receive a question (such as "How is the weather?") from a first user and provide a response (such as "The weather is warm and sunny. The same electronic device <NUM> could receive the same question from a second user in a different language (such as Korean) and provide the same response to the second user (although the response may also be in Korean). The same electronic device <NUM> could also receive the same question from a third user in a different language (such as Hindi) and provide the same response to the third user (although the response may also be in Hindi). The users are not required to perform any setup operations or otherwise actively perform operations to change the language recognized by the electronic device <NUM>.

Although <FIG> illustrates one example of an architecture <NUM> for multi-spoken language detection, various changes may be made to <FIG>. For example, the scoring function <NUM> may include or be used in conjunction with any suitable number of language models <NUM>, and the ASR engine <NUM> may include or be used in conjunction with any suitable number of ASR models <NUM>.

<FIG> illustrates a more specific example architecture <NUM> for multi-spoken language detection in accordance with this disclosure. In particular, the architecture <NUM> shown in <FIG> may represent a more detailed implementation of part of the architecture <NUM> shown in <FIG>. For this reason, various reference numbers used in <FIG> are also used in <FIG> to represent common components between the architectures <NUM> and <NUM>. For ease of explanation, the architecture <NUM> shown in <FIG> may be described as being implemented using the electronic device <NUM> in the network configuration <NUM> shown in <FIG>. However, the architecture <NUM> may be used by any suitable device(s) and in any suitable system(s), such as by the server <NUM> in the network configuration <NUM>.

As shown in <FIG>, the audio-based input <NUM> is provided to the feature extractor <NUM>, which identifies extracted features from the audio-based input <NUM>. The extracted features are provided to the language-specific acoustic model scoring function <NUM>, which uses the multiple language models <NUM> to identify probabilities or other likelihoods that different portions of the audio-based input <NUM> are in specific languages. The raw extracted features from the audio-based input <NUM> and the probabilities from the scoring function <NUM> are provided to the feature concatenator <NUM>.

The feature concatenator <NUM> combines the raw extracted features and the probabilities to form feature vectors. In this particular example, the feature concatenator <NUM> is implemented using a temporal accumulator, which accumulates various information generated by the feature extractor <NUM> and the scoring function <NUM> over time. Essentially, the feature concatenator <NUM> stacks the information generated by the feature extractor <NUM> and the scoring function <NUM> to form the feature vectors, which are then further processed by the neural classification model <NUM>.

In the specific example shown in <FIG>, information is accumulated by the feature concatenator <NUM> in rows <NUM> and columns <NUM>. Each row <NUM> is associated with data from a different source. In particular, the first row <NUM> in this example includes raw extracted features <NUM> from the feature extractor <NUM>, the second row <NUM> in this example includes probabilities <NUM> calculated using the first language model <NUM>, and the last row <NUM> in this example includes probabilities <NUM> calculated using the last language model <NUM>. Note that additional rows containing probabilities calculated using additional language models <NUM> may similarly be accumulated by the feature concatenator <NUM>. Each column <NUM> is associated with a different time and thereof a different portion of the audio-based input <NUM>. For instance, the first column <NUM> may contain raw extracted features <NUM> and probabilities <NUM> and <NUM> associated with a first portion of the audio-based input <NUM> (such as a first <NUM> millisecond segment), the next column <NUM> may contain raw extracted features <NUM> and probabilities <NUM> and <NUM> associated with a second portion of the audio-based input <NUM> (such as a second <NUM> millisecond segment), and so on. The number of columns <NUM> that are stacked or accumulated by the feature concatenator <NUM> for further processing can vary based on a number of factors. In some cases, the columns <NUM> may span a specified period of time, such as one second, two seconds, three seconds, or five seconds.

Note that the inclusion of the raw extracted features <NUM> from the feature extractor <NUM> in the accumulated information provided by the feature concatenator <NUM> is optional. In other words, the feature concatenator <NUM> may concatenate the probabilities <NUM>, <NUM> from the language models <NUM> and exclude the extracted features <NUM> in the accumulated information. However, it has been determined that the inclusion of the raw extracted features <NUM> in the feature vectors processed by the neural classification model <NUM> may improve language identification by the neural classification model <NUM> in some instances.

The feature vectors provided by the feature concatenator <NUM> are processed by the neural classification model <NUM>, which includes a temporal convolutional neural network (TCNN) <NUM>. The temporal convolutional neural network <NUM> applies a convolution function to the information provided by the feature concatenator <NUM>. More specifically, the temporal convolutional neural network <NUM> applies a convolution function to the information within a window <NUM>, and the window <NUM> moves or slides so that the temporal convolutional neural network <NUM> can apply the convolution function to different subsets of the information from the feature concatenator <NUM>. In this way, the temporal convolutional neural network <NUM> can apply the convolution function in an overlapping manner to the raw extracted features <NUM> and probabilities <NUM> and <NUM> provided by the feature concatenator <NUM>.

In some cases, the window <NUM> may have a height that matches the combined heights of the raw extracted features <NUM> and all probabilities (including the probabilities <NUM> and <NUM>) provided by the feature concatenator <NUM>, in which case the window <NUM> may slide horizontally but not vertically. Note, however, that other embodiments may be used for the window <NUM>. Also, the window <NUM> may have a width that is significantly smaller than the combined widths of the raw extracted features <NUM> and all probabilities (including the probabilities <NUM> and <NUM>) provided by the feature concatenator <NUM>. In some embodiments, the width of the window <NUM> may be based on the number of features being stacked by the feature concatenator <NUM>. As a particular example, if the information from the feature concatenator <NUM> spans two hundred samples widthwise, the window <NUM> may have a width of twenty samples. Again, however, note that other embodiments may be used for the window <NUM>.

The convoluted outputs from the temporal convolutional neural network <NUM> are provided to a batch normalization function <NUM>, which normalizes the convoluted outputs from the temporal convolutional neural network <NUM>. The batch normalization function <NUM> can use any suitable technique to normalize the convoluted outputs from the temporal convolutional neural network <NUM>. The normalized convoluted outputs from the temporal convolutional neural network <NUM> are then provided to another temporal convolutional neural network <NUM>, which may operate in the same or similar manner as the temporal convolutional neural network <NUM>. The temporal convolutional neural network <NUM> may be followed by another batch normalization function. Note that any suitable number of temporal convolutional neural networks and batch normalization functions may be used here. In some embodiments, for instance, the neural classification model <NUM> may include between three and five temporal convolutional neural networks, and each temporal convolutional neural network may be followed by a batch normalization function.

A pooling layer <NUM> processes the convoluted outputs produced by the final temporal convolutional neural network (possibly after normalization) and selects or combines the convoluted outputs for input to an activation function of the neural classification model <NUM>. In this example, the pooling layer <NUM> combines the convoluted outputs from the final temporal convolutional neural network by averaging groups of convoluted values and outputting the average convoluted values. The various temporal convolutional neural networks, batch normalization functions, and pooling layer are used here to model each language that is supported by the architecture <NUM>, allowing these components to effectively score the likelihood of each language being used in the audio-based input <NUM>.

The activation function of the neural classification model <NUM> is implemented in this example using a softmax function <NUM>. The softmax function <NUM> receives the average convoluted values from the pooling layer <NUM> and operates to calculate final probabilities <NUM>, which identify the probability of each language being associated with at least part of the audio-based input <NUM>. Here, the final probabilities <NUM> may be expressed as P(langi | xt), where P( <IMG> ) represents the calculated probability that a specific language langi is associated with a particular time instant xt of the audio-based input <NUM>. For each portion of the audio-based input <NUM>, the probabilities across all languages can be used to identify the ASR model <NUM> that should be used to process that portion of the audio-based input <NUM>.

Note that, in some embodiments, the softmax function <NUM> can also be configured to consider the amount of silence in the audio-based input <NUM> when identifying the language(s) associated with the audio-based input <NUM>. For example, in many cases (even during active conversations), a large percentage of the audio-based input <NUM> may be associated with silence (or at least no one actually speaking). In some cases, the amount of silence in the audio-based input <NUM> may actually exceed <NUM>% of the entire duration of the audio-based input <NUM>. The softmax function <NUM> may therefore operate to exclude portions of the audio-based input <NUM> that are associated with silence. In some embodiments, the softmax function <NUM> may calculate probabilities for various languages while excluding silent portions (also called empty portions) of the audio-based input <NUM> in the following manner: <MAT>.

Here, X represents a length of the audio-based input <NUM> from a time t of zero to a time t of T, and | represents an identity matrix whose size depends on the number of languages associated with non-silent times above a threshold value (<NUM> in this example, although other thresholds may be used here).

Based on the probabilities output by the softmax function <NUM>, the audio-based input <NUM> can be directed in its entirety to a single ASR model <NUM> when a single language is identified, or different portions of the audio-based input <NUM> can be directed to different ASR models <NUM> when multiple language are identified. As a result, inputs in different languages can be routed automatically to appropriate ASR models <NUM> for processing, and portions of the same input in different languages also can be routed automatically to appropriate ASR models <NUM> for processing.

Although <FIG> illustrates a more specific example of an architecture <NUM> for multi-spoken language detection, various changes may be made to <FIG>. For example, the scoring function <NUM> may include or be used in conjunction with any suitable number of language models <NUM>. Also, the feature concatenator <NUM> may temporally accumulate or otherwise combine any suitable number of inputs over any suitable time period. In addition, the neural classification model <NUM> may be implemented using any other suitable machine learning architecture or other architecture.

<FIG> illustrates an example architecture <NUM> for supporting the identification of transitions between multiple languages in accordance with this disclosure. For ease of explanation, the architecture <NUM> shown in <FIG> may be described as being implemented using the electronic device <NUM> in the network configuration <NUM> shown in <FIG>. However, the architecture <NUM> may be used by any suitable device(s) and in any suitable system(s), such as by the server <NUM> in the network configuration <NUM>.

As described above, it is possible in some cases for an audio-based input <NUM> to be associated with multiple languages, such as when a user switches between languages while speaking. In these cases, different portions of the same audio-based input <NUM> are associated with different languages and can be directed to different ASR models <NUM> for processing. Here, the different portions of the audio-based input <NUM> may be identified as being associated with the different languages using the approaches above. However, for certain portions of the audio-based input <NUM> that involve transitions between different languages, it may be more difficult to identify the language to be used. Thus, in some embodiments, additional processing may be performed to identify when the language being used in the audio-based input <NUM> dynamically changes.

As shown in <FIG>, the architecture <NUM> includes a deep sub-network decomposing (DSND) algorithm <NUM>. The DSND algorithm <NUM> here has access to a large language model <NUM> associated with various languages. In some cases, for example, the language model <NUM> may be associated with many languages (such as a universal model). The DSND algorithm <NUM> supports knowledge distillation in order to take the large language model <NUM> and generate multiple sub-network models, such as sub-network models <NUM> and <NUM>.

Each sub-network model <NUM> and <NUM> can be trained to identify a jump between a different set of languages, and outputs of each sub-network model can be treated as a new feature vector used for language identification. For example, one sub-network model <NUM> may be trained to identify when a speaker transitions from English to Korean, and another sub-network model <NUM> may be trained to identify when a speaker transitions from English to Hindi. Note that these are examples only and that sub-network models may be trained to identify when a speaker transitions between any suitable languages. This may allow a digital personal assistant or other ASR-based system to more accurately detect when a user dynamically switches between languages, which may allow the ASR-based system to more accurately respond to multi-language inputs. The DSND algorithm <NUM> supports any suitable knowledge distillation or other technique for generating sub-network models based on a larger model.

The various sub-network models may be used in any suitable manner in the architectures <NUM>, <NUM> described above or in other architectures designed in accordance with this disclosure. For instance, in the architecture <NUM> of <FIG>, the sub-network models may be used after the neural classification model <NUM> to help improve the language detection functionality provided by the neural classification model <NUM>. In the architecture <NUM> of <FIG>, the sub-network models may be used after the last temporal convolutional neural network (and optionally after the last batch normalization function) to again help improve the language detection functionality provided by the neural classification model <NUM>. However, it should be noted that use of sub-network models is not required in either of the architectures <NUM>, <NUM>.

The functionality provided by the sub-network models may be useful when a user speaks a specific language utterance that is embedded within an utterance in a different language. For instance, a user might provide the following utterance as input to an electronic device <NUM>:
"How are you doing" in Hindi is said as "<IMG>. " What is it in Korean?.

The various sub-network models may be used to identify the transitions between English and Hindi in the above example. This may allow the electronic device <NUM> or other device to more effectively route the English portions of the utterance to an English-based ASR model <NUM> and the Hindi portion of the utterance to a Hindi-based ASR model <NUM>. Of course, the example above is for illustration only, and any other suitable multi-language utterance involving any number of languages may be processed here.

Although <FIG> illustrates one example of an architecture <NUM> for supporting the identification of transitions between multiple languages, various changes may be made to <FIG>. For example, the architecture <NUM> may include any other number of sub-network models. Also, any other suitable technique may be used to identify or handle language transitions in input utterances.

<FIG> illustrates an example method <NUM> for multi-spoken language detection in accordance with this disclosure. For ease of explanation, the method <NUM> shown in <FIG> may be described as being performed by the electronic device <NUM> in the network configuration <NUM> shown in <FIG>. However, the method <NUM> may be performed by any suitable device(s) and in any suitable system(s), such as by the server <NUM> in the network configuration <NUM>.

As shown in <FIG>, input audio data is received at step <NUM>. This may include, for example, the processor <NUM> of the electronic device <NUM> receiving audio-based input <NUM> representing a digitized and possibly pre-processed version of a user utterance captured by a microphone of the electronic device <NUM>. The user utterance may include one or more words that are spoken in one or more languages, but the specific word(s) and specific language(s) are not known at this point. Features of the input audio data are extracted at step <NUM>. This may include, for example, the processor <NUM> of the electronic device <NUM> using the feature extractor <NUM> to extract phoneme or phoneme-based features from the audio-based input <NUM>. Of course, any other or additional features of the audio-based input <NUM> may also or alternatively be extracted here.

Likelihoods of each of multiple portions of the input audio data being in specific languages are determined at step <NUM>. This may include, for example, the processor <NUM> of the electronic device <NUM> using the language models <NUM> to process the extracted features and determine probabilities that each portion of the audio-based input <NUM> were spoken in specific languages. The different portions of the audio-based input <NUM> here may represent different segments, such as segments spanning a given time period (like <NUM> milliseconds). However, the different portions of the audio-based input <NUM> may be identified in any other suitable manner. The determined likelihoods and optionally the extracted features of the input audio data are combined to produce feature vectors at step <NUM>. This may include, for example, the processor <NUM> of the electronic device <NUM> using the feature concatenator <NUM> to temporally accumulate or otherwise combine the identified probabilities <NUM>, <NUM> from the language models <NUM> and optionally the raw extracted features <NUM> from the feature extractor <NUM> over time.

A language associated with each portion of the input audio data is detected using the feature vectors and a neural classification model at step <NUM>. This may include, for example, the processor <NUM> of the electronic device <NUM> using the neural classification model <NUM> to determine final probabilities of each portion of the audio-based input <NUM> being associated with specific languages. As a particular example, this may include the processor <NUM> of the electronic device <NUM> using one or more temporal convolutional neural networks, one or more batch normalization functions, at least one pooling layer, and at least one activation function to process the identified probabilities <NUM>, <NUM> and optionally the raw extracted features <NUM> accumulated over a time period. Each temporal convolutional neural network may use a sliding window <NUM> to process its input data and apply a convolution function to that input data.

Each portion of the input audio data is directed to an ASR model based on the identified language for that portion of the input audio data at step <NUM>. This may include, for example, the processor <NUM> of the electronic device <NUM> causing the audio-based input <NUM> to be provided to a single ASR model <NUM> if a single language is associated with the audio-based input <NUM>. This may also include the processor <NUM> of the electronic device <NUM> causing different portions of the audio-based input <NUM> to be provided to different ASR models <NUM> if those portions of the audio-based input <NUM> are associated with different languages. Each portion of the input audio data is converted into text or is otherwise understood using the associated ASR model at step <NUM>. This may include, for example, the processor <NUM> of the electronic device <NUM> using the ASR engine <NUM> with the appropriate ASR model(s) <NUM> to convert the audio-based input <NUM> into a text-based output <NUM>.

Although <FIG> illustrates one example of a method <NUM> for multi-spoken language detection, various changes may be made to <FIG>. For example, while shown as a series of steps, various steps in <FIG> may overlap, occur in parallel, occur in a different order, or occur any number of times.

It should be noted here that while often described above as not requiring user input to identify a language or languages associated with input audio data, some embodiments of this disclosure may use some form of user input when identifying the language or languages associated with input audio data. For example, most users may typically speak only a handful of languages, such as two or three languages (although larger numbers are also possible for certain users). In some instances, a user may identify the language(s) that might be spoken to an electronic device <NUM> or other device, and that information may be used to control which language models <NUM>, ASR models <NUM>, and possibly sub-network models <NUM> and <NUM> are used to process input audio data. A user may identify the language(s) that might be spoken to the device in any suitable manner and at any suitable time(s), such as during a device setup process or by accessing a settings menu. If multiple users might use an electronic device <NUM> or other device, the languages spoken by a specific user might be stored in a user profile for that user, and the user profile may be used to control which language models <NUM>, ASR models <NUM>, and possibly sub-network models <NUM> and <NUM> are used to process input audio data when that user is present. Note that any other suitable technique may be used to limit the number of languages analyzed at any particular time, or all supported languages may be used at any particular time (or at all times).

Claim 1:
A method comprising:
- performing (<NUM>), using at least one processor, feature extraction of input audio data to identify extracted features associated with the input audio data;
- detecting (<NUM>) , using the at least one processor, a language associated with each of multiple portions of the input audio data by processing the extracted features using a plurality of language models, each language model associated with a different language,
wherein each language model is used to determine (<NUM>) a probability that each portion of the input audio data is from a particular language;
wherein detecting the language associated with each portion of the input audio data further comprises:
concatenating the probabilities determined using the language models; and
processing the concatenated probabilities using a neural classification model;
characterised in that
processing the concatenated probabilities using the neural classification model comprises:
processing the concatenated probabilities using a plurality of temporal convolutional neural networks, wherein outputs from at least one of the temporal convolutional neural networks are normalized;
averaging outputs or normalized outputs from a last of the temporal convolutional neural networks; and
determining probabilities that specific languages are associated with specific portions of the input audio data based on the averaged outputs; and
- directing (<NUM>), using the at least one processor, each portion of the input audio data to one of a plurality of automatic speech recognition, ASR, models based on the language associated with the portion of the input audio data, wherein different portions of the input audio data are provided to different ASR models if the different portions of the input audio data are associated with different languages.