End-to-end integration of dialog history for spoken language understanding

Systems, computer-implemented methods, and computer program products to facilitate end to end integration of dialogue history for spoken language understanding are provided. According to an embodiment, a system can comprise a processor that executes components stored in memory. The computer executable components comprise a conversation component that encodes speech-based content of an utterance and text-based content of the utterance into a uniform representation.

BACKGROUND

The subject disclosure relates to spoken language understanding, and more specifically, to end-to-end integration of dialog history for spoken language understanding.

SUMMARY

According to an embodiment, a system can comprise a processor that executes computer executable components stored in memory. The computer executable components comprise a conversational component that encodes speech-based content of an utterance and text-based content of an utterance into a uniform representation.

According to another embodiment, a computer-implemented method can comprise encoding, by a system, operatively coupled to a processor, speech-based content of an utterance and text-based content of the utterance into a uniform representation.

According to another embodiment, a computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to encode speech-based content of an utterance and text-based content of an utterance into a uniform representation.

DETAILED DESCRIPTION

As referenced herein, an “entity” can comprise a client, a user, a computing device, a software application, an agent, a machine learning (ML) model, an artificial intelligence (AI) model, and/or another entity.

Use of spoken language understanding (SLU) has great potential to address a variety of problems in a number of different domains. For example, spoken language understanding can enable speech to be used as an input in various computer products or allow for an entity to control a device through voice commands. Use of dialogue history can be used to improve spoken language understanding by providing context for a speech utterance.

In order to improve performance, existing spoken language understanding systems utilize dialog history in order to resolve ambiguities, co-references, and co-occurrences of the same word. However, existing spoken language understanding systems use dialog history in text form in conjunction with an automatic speech recognition (ASR) component and a natural language understanding (NLU) component. Existing ASR and NLU models are inherently large, having large memory and storage usage, and a model implementing both causes cascading, which leads to an even larger overall model size and can degrade performance of spoken language understanding.

Given problems described above with existing SLU technologies, the present disclosure can be implemented to produce a solution to these problems in the form of systems, computer-implemented methods, and/or computer program products that can facilitate end-to-end integration of dialog history for spoken language understanding by: encoding speech-based content of an utterance and text-based content of the utterance into a uniform representation.

In some embodiments, the present disclosure can be implemented to produce a solution to the problems described above in the form of systems, computer-implemented methods, and/or computer program products that can further facilitate end-to-end integration of dialog history for spoken language understanding by: encoding the utterance and speech-based context of the utterance and the text of the utterance and text-based context of the utterance. An advantage of such systems, computer-implemented methods, and/or computer program products is that they can be used as a hierarchical model with lower level encoder and an upper level encoder, which facilitates end-to-end integration of dialogue history in spoken form.

FIG.1illustrates a flow diagram100of an example, non-limiting method that can facilitate performance of spoken language understanding in accordance with one or more embodiments described herein.

As shown at110, a system, such as spoken language understanding system201described in greater detail below in reference toFIGS.2and3, can receive as input, speech in an audio form. At120, the system can identify an utterance within the speech such as the utterance “I want to travel from New York to Boston.” At130, the system can identify an intent of the utterance or a category to which the utterance belongs using a classifier. In the present example, the intent or category135of the utterance “I want to travel from New York to Boston” is a travel reservation. At140, the system can identify slots and values in the utterance based on the intent or category. For example, the system can identify slots141and142as the departure point and destination point based on the intent135of a travel reservation. The system can then identify values that fit into slots141and142. For example, the system can identify value143“New York” as the departure point and value144“Boston” as the destination point. During this process, context, such as dialog history comprising previous utterances, can assist the system in making more accurate determinations. For example, dialogue history may provide context in order to determine whether “New York” referrers to New York City or New York state.

FIGS.2and3illustrate block diagrams of example, non-limiting systems200and300respectively, that can enable end-to-end integration of dialog history in spoken language understanding. System200can comprise spoken language understanding system201. Spoken language understanding system201of system200can comprise a memory202, a processor203, a conversation component204, and/or a bus218. Spoken language understanding system201of system300, depicted inFIG.3, can further comprise, an utterance component305, a recognition component306, a training component307, and a neural network component308.

It should be appreciated that the embodiments of the subject disclosure depicted in various figures are for illustration only, and as such, the architecture of such embodiments are not limited to the systems, devices, and/or components depicted therein. For example, in some embodiments, system200, system300and/or spoken language understanding system201can further comprise various computer and/or computing-based elements described herein with reference to operating environment1100andFIG.11. In several embodiments, such computer and/or computer-based elements can be used in connection with implementing one or more of the systems, devices, components, and/or computer-implemented operations shown described in connection withFIG.2,FIG.3, and/or other figures disclosed herein.

Memory202can store one or more computer and/or machine readable, writable, and/or executable components and/or instructions that, when executed by processor203(e.g., a classical processor, a quantum processor, and/or another type of processor), can facilitate performance of operations defined by executable component(s) and/or instruction(s). For example, memory202can store computer and/or machine readable, writable, and/or executable components and/or instructions that, when executed by processor203, can facilitate execution of the various functions described herein relating to spoken language understanding system201, conversation component204, utterance component305, recognition component306, training component307, neural network component308, and/or another component associated with spoken language understanding system201.

Memory202can comprise volatile memory (e.g., random access memory (RAM), static RAM (SRAM) dynamic RAM (DRAM), and/or another type of volatile memory) and/or non-volatile memory (e.g., read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), and/or another type of non-volitive memory) that can employ one or more memory architectures. Further examples of memory202are described below with reference to system memory1116andFIG.11. Such examples of memory202can be employed to implement any embodiments of the subject disclosure.

Processor203can comprise one or more types of processors and/or electronic circuitry (e.g., a classical processor, a quantum processor, and/or another type of processor and/or electronic circuitry) that can implement one or more computer and/or machine readable, writable, and/or executable components and/or instructions that can be stored on memory202. For example, processor203can perform various operations that can be specified by such computer and/or machine readable, writable, and/or executable components and/or instructions including, but not limited to, logic, control, input/output (I/O), arithmetic, and/or the like. In some embodiments, processor203can comprise one or more central processing unit, multi-core processor, microprocessor, dual microprocessors, microcontroller, System on a Chip (SOC), array processor, vector processor, quantum processor, and/or another type of processor. Further examples of processor203are described below with reference to processing unit1114andFIG.11. Such examples of processor203can be employed to implement any embodiments of the subject disclosure.

Spoken language understanding system201, memory202, processor203, conversation component204, utterance component305, recognition component306, training component307, neural network component308, and/or another component of spoken language understanding system201as described herein can be communicatively, electrically, operatively, and/or optically coupled to one another via bus218to perform functions of system200, system300, spoken language understanding system201, and/or any components coupled therewith. Bus218can comprise one or more memory bus, memory controller, peripheral bus, external bus, local bus, quantum bus, and/or another type of bus that can employ various bus architectures. Further examples of bus218are described below with reference to system bus1118andFIG.11. Such example of bus218can be employed to implement any embodiments of the subject disclosure.

Spoken language understanding system201can comprise any type of component, machine, device, facility, apparatus, and/or instrument that comprises a processor and/or can be capable of effective and/or operative communication with a wired and/or wireless network. All such embodiments are envisioned. For example, spoken language understanding system201can comprise a server device, a computing device, a general-purpose computer, a special-purpose computer, a quantum computing device (e.g., a quantum computer), a tablet computing device, a handheld device, a sever class computing machine and/or database, a laptop computer, a notebook computer, a desktop computer, a cell phone, a smart phone, a consumer appliance and/or instrumentation, an industrial and/or commercial device, a digital assistant, a multimedia Internet enabled phone, a multimedia player, and/or another type of device.

Spoken language understanding system201can be coupled (e.g., communicatively, electrically, operatively, optically, and/or coupled via another type of coupling) to one or more external systems, sources, and/or devices (e.g., classical and/or quantum computing devices, communication devices, and/or another type of external system, source, and/or device) using a wire and/or cable. For example, spoken language understanding system201can be coupled (e.g., communicatively, electrically, operatively, optically, and/or coupled via another type of coupling) to one or more external systems, sources, and/or devices (e.g., classical and/or quantum computing devices, communication devices, and/or another type of external system, source, and/or device) using a data cable including, but not limited to, a High-Definition Multimedia Interface (HDMI) cable, a recommended standard (RS)232cable, an Ethernet cable, and/or another data cable.

In some embodiments, spoken language understanding system201can be coupled (e.g., communicatively, electrically, operatively, optically, and/or coupled via another type of coupling) to one or more external systems, sources, and/or devices (e.g., classical and/or quantum computing devices, communication devices, and/or another type of external system, source, and/or device) via a network. For example, such a network can comprise wired and/or wireless networks, including, but not limited to, a cellular network, a wide area network (WAN) (e.g., the Internet) or a local area network (LAN). Spoken language understanding system201can communicate with one or more external systems, sources, and/or devices, for instance, computing devices using virtually any desired wired and/or wireless technology, including but not limited to: wireless fidelity (Wi-Fi), global system for mobile communications (GSM), universal mobile telecommunications system (UMTS), worldwide interoperability for microwave access (WiMAX), enhanced general packet radio service (enhanced GPRS), third generation partnership project (3GPP) long term evolution (LTE), third generation partnership project 2 (3GPP2) ultra mobile broadband (UMB), high speed packet access (HSPA), Zigbee and other 802.XX wireless technologies and/or legacy telecommunication technologies, BLUETOOTH®, Session Initiation Protocol (SIP), ZIGBEE®, RF4CE protocol, WirelessHART protocol, 6LoWPAN (IPv6 over Low power Wireless Area Networks), Z-Wave, an ANT, an ultra-wideband (UWB) standard protocol, and/or other proprietary and non-proprietary communication protocols. Therefore, in some embodiments, spoken language understanding system201can comprise hardware (e.g., a central processing unit (CPU), a transceiver, a decoder, quantum hardware, a quantum processor, and/or other hardware), software (e.g., a set of threads, a set of processes, software in execution, quantum pulse schedule, quantum circuit, quantum gates, and/or other software) or a combination of hardware and software that can facilitate communicating information between spoken language understanding system201and external systems, sources, and/or devices (e.g., computing devices, communication devices, and/or another type of external system, source, and/or device).

Spoken language understanding system201can comprise one or more computer and/or machine readable, writable, and/or executable components and/or instructions that, when executed by processor203(e.g., a classical processor, a quantum processor, and/or another type of processor), can facilitate performance of operations defined by such component(s) and/or instruction(s). Further, in numerous embodiments, any component associated with spoken language understanding system201, as described herein with or without reference to the various figures of the subject disclosure, can comprise one or more computer and/or machine readable, writable, and/or executable components and/or instructions that, when executed by processor203, can facilitate performance of operations defined by such component(s) and/or instruction(s). For example, conversation component204, utterance component305, recognition component306, training component307, neural network component308and/or any other components associated with spoken language understanding system201as disclosed herein (e.g., communicatively, electronically, operatively, and/or optically coupled with and/or employed by spoken language understanding system201), can comprise such computer and/or machine readable, writable, and/or executable component(s) and/or instruction(s). Consequently, according to numerous embodiments, spoken language understanding system201and/or any components associated therewith as disclosed herein, can employ processor203to execute such computer and/or machine readable, writable, and/or executable component(s) and/or instruction(s) to facilitate performance of one or more operations described herein with reference to spoken language understanding system201and/or any such components associated therewith.

Spoken language understanding system201can facilitate (e.g., via processor203) performance of operations executed by and/or associated with conversation component204, utterance component305, recognition component306, training component307, neural network component308and/or another component associated with spoken language understanding system201as disclosed herein. For example, as described in detail below, spoken language understanding system201can facilitate (e.g., via processor203): encoding speech-based content of an utterance and text-based content of the utterance into a uniform representation. In another example, as described in detail below, spoken language understanding system201can further facilitate training a hierarchical conversational model using one or more cross-model loss functions for the recognized spoken language understanding categories, slots and values of the speech-based content and the text-based content.

Conversation component204can encode speech-based content of an utterance and text-based content of the utterance into a uniform representation. As described herein, content of an utterance can comprise the utterance itself and context of the utterance, wherein context is the dialog history made up of previous utterance in a conversation. In an embodiment, conversation component204can receive both the speech-based content of an utterance and the text-based content of the utterance. For example, conversation component204can receive speech-based content denoted as Usand text-based content denoted as Ut. Conversation component204can then use a neural network higher level sequence encoder, g(.; ϕ) to encode Usand Utinto a uniform representation. For example, the speech-based content can be encoded as cs=g(Us; ϕ) and the text-based content can be encoded as ct=g(Ut; ϕ), wherein ϕ is a parameter of the encoder. As the sequence encoder is shared between the speech-based content and the text-based content, the text-based content can be used to transfer knowledge from text to speech during the training process described in detail below. In an embodiment, sequence encoder, g(.; ϕ) can be modeled as a 6-layer 1-head transformer.

Utterance component305can encode an utterance and speech-based context of the utterance and the text of the utterance and text-based context of the utterance. For example, an SLU problem can be treated as a context labeling task, where instead of labeling the current utterance, a model can label the entire context (dialog history+current utterance). As such, utterance component305can receive speech-based context Cs={x1s, x2s, . . . , xNs} where xNsis the current utterance and the rest is dialog history. Similarly, utterance component305can also receive text-based context Ct={x1t, x2t, . . . , xNt}, wherein Ctis a transcript of Cs. In an embodiment, utterance component305can comprise a speech encoder component and a text encoder component.

For example, a speech encoder component of utterance component305can use a neural network with the speech encoder fs(.; θs) parameterized by θs. The speech signal xis, from the ithutterance in the context can be encoded as uis=fs(xis; θs). Therefore, the entirety of the encoded speech context can be denoted as Us={uis}i=1N, which can be passed to conversation component204as input. In an embodiment, speech encoder component can use the transcription network of a pretrained recurrent neural network transducer (RNN-T) based ASR model as the speech encoder, which encodes the speech signal directly to give a vector representation. This can be pre-trained for ASR using a 40-dimensional global mean and variance normalized log-mel filterbank features, extracted every 10 ms. These features can be further augmented with Δ and ΔΔ coefficients, every two consecutive frames are stacked, and every second frame is skipped, resulting in 240-dimensional vectors every 20 ms. Thus, xiscan be a sequence of 240-dimensional log-mel features. The encoder, fs(.; θs) can be a 6-layer bidirectional long short-term memory (LSTM) with 640 cells per layer per direction. The speech encoder component can concatenate vectors from the forward and backward direction of the last layer of the LSTM for the last time step to produce a 1280-dimensional vector. This vector can go through a linier layer which shrinks dimensionality of the vector to 768. Thus, uiscan be a 768-dimensional representation of a speech utterance.

Additionally, a text encoder component of utterance component305can use a neural network with the text encoder ft(.; θt) parameterized by θt. The ithtext utterance, xit, can be a sequence of word tokens encoded using Word Piece embeddings, or any other suitable embeddings. As such, text encoder component can encode xitas uit=ft(xit; θt). Therefore, the entirety of the encoded speech context can be denoted as Ut={uit}i=1N, which can be passed to conversation component204as input. In an embodiment, text encoder component can use a pretrained bidirectional encoder representations from transformer (BERT) model. The 768-dimensional token output from this model can be treated as the text representation uit. In an embodiment, the speech encoder component can be a “student” in a student-teacher joint training framework and the text encoder can be a “teacher” in the student-teacher joint training framework. In a student-teacher joint training framework, a “teacher” component is used to transfer knowledge to a “student” component. As such, it should be appreciated the text encoder component can be used during the training process, described in greater detail below, to transfer semantic knowledge to the speech encoder component and then the text encoder component can be discarded after training and/or during performance of SLU operations, thus making the system fully end-to-end.

Recognition component306can recognize at least one of spoken language understanding categories, slots or values from at least one of the speech-based content of the text-based content. For example, recognition component306can comprise a classification layer of a neural network that receives the encoded speech-based content and the encoded text-based content ctand csfrom conversation component204. Examples, of classification layers include support vector machines, Bayesian networks, decisions trees, neural networks, fuzzy logic models, probabilistic classification models, or any other suitable classifier. The classification layer can then generate a prediction of a spoken language understanding category (e.g., an intent), slots, and/or values. In an embodiment, recognition component306can make a prediction of a category or intent. For example, based on the encoded content, the classification layer can determine probabilities that an utterance belongs to one or more categories. Based on the probabilities, the classification layer can select the option with the highest probability as the category or intent. Based on the predicted category or intent, recognition component306can then predict slots that correspond to the predicted category or intent and values to fill the slots. For example, recognition component306can predict that an utterance has an intent of a travel reservation. Based on that intent, recognition component306can then predict slots, such as a departure point and a destination point, and values to fill those slots such as “New York” as a departure point and “Boston” as a destination point. In an embodiment, recognition component306can identify multiple categories or intents. For example, if recognition component306predicts an intent of a travel reservation, recognition component306can then predict an intent of a type of travel reservation such as a flight reservation or a train reservation.

Training component307can train a hierarchical conversation model using one or more cross-model loss functions. In an embodiment, the one or more cross-model loss functions can be based on the recognized spoken language understanding categories, slots and/or values for the speech-based content and/or the text-based content. For example, training component307can compare the categories, slots and values recognized using text to the categories, slots and values recognized using speech. Training component307can the train utterance component305and conversation component204to minimize the difference. In an embodiment, conversation component204and utterance component305can comprise a hierarchical conversational model with utterance component serving as a lower level and conversation component204serving as a top layer. Training component307can then compute binary cross-entropy loss for the speech modalities, denoted as LBCE(θs, ϕ), and for the text modalities, denoted as LBCE(θt, ϕ). In an embodiment, the hierarchical model can be trained in one of three ways, co-trained with speech and text using LBCE(θs, ϕ)±LBCE(θt, ϕ), referred to as HIER-ST, trained with speech only using LBCE(θs, ϕ), referred to as HEIR-S, or trained with text only using LBCE(θt, ϕ), referred to as HIER-T. By using cross-model loss functions, semantic knowledge can be transferred from a text encoder component of utterance component305to a speech encoder component of utterance component305and to conversation component204. In an embodiment, a Euclidean loss function, LEUC, can be computed as the L2 distance between the text and speech representations. Formally,

When compared to a sequence of text tokens, the corresponding sequence of speech frames can be longer, up to five times longer in some instances. As such, when dialog history is used in speech form, the speech sequence can be comparatively long resulting in increased training time for end-to-end models as well as increased memory requirements for the training process. To address this issue, training component307can drop one or more speech frames during training. By dropping speech frames, training time can be decreased while also improving performance. For example, if the length of sequence xisis longer than a hyperparameter l, training component307can randomly drop out some of the frames in xsiuntil the length of xisequals l. As described in greater detail below in regard toFIG.8A, this can result in decreased training time as well as improved performance and accuracy of the trained model.

Neural network component308can perform spoken language understanding by applying the hierarchical conversational model to a speech conversation. For example, in an embodiment, utterance component305and conversation component204can be utilized in a hierarchical neural network model, wherein utterance component305serves as a lower level encoder and conversation component204serves as an upper level encoder. As discussed above in relation to training component307, text is utilized during training to transfer semantic knowledge from a text encoder component to a speech encoder component within utterance component305. As such, when performing SLU after training, only the speech of a conversation is utilized. In an embodiment, neural network component308can receive a speech conversation and a request to perform an SLU operation on the speech conversation. Neural network component308can then apply the hierarchical model comprising utterance component305and conversation component204to the speech conversation. For example, utterance component305can receive speech-based context Cs={x1s,x2s, . . . , xNs} of the speech conversation from neural network component and encode Cs={x1s,x2s, . . . , xNs} as Us={uis}i=1Nas described above. Conversation component204can then encode Usas cs=g(Us; ϕ). This encoding can then be passed to recognition component306. Based on the type of SLU request received by neural network component308, recognition component306can recognize at least one of SLU categories, slots and/or values, which can then be return as an output. It should be appreciated that spoken language understanding system201thereby enables end-to-end integration of dialogue history in SLU operations as text is not utilized to provide context or act as dialogue history during performance of SLU operations. It should also be appreciated, that in an embodiment, SLU operations can be performed without training. For example, neural network component308can perform spoken language understanding by applying the hierarchical model to a speech conversation prior to or absent of the hierarchical model being trained by training component307.

FIG.4illustrates a flow diagram of an example, non-limiting method400that can facilitate training of utterance component305in accordance with one or more embodiments described herein.

As shown, utterance component305can comprise two parts, speech encoder component410and text encoder component420. As described above in reference toFIG.3, in an embodiment, text encoder component420can be a BERT encoder. At412, speech encoder component410can receive speech signal xis, from the ithutterance in speech-based context Cs={x1s,x2s, . . . , xNs}, wherein xNsis the current utterance and the rest is dialog history. At415, speech encoder component410can encode xisusing a neural network with the encoder fs(.; θs) to output the encoded context as uis. It should be appreciated that in an embodiment, speech encoder component410can repeat the encoding process for all utterances in Cs={x1s,x2s, . . . , xNs} to produce the entirety of the encoded speech context denoted as Us={uis}i=1N.

At422, text encoder component420can receive xit, the ithtext utterance from text-based context Ct={x1t, x2t, . . . , xNt}. In an embodiment, xitcan be a sequence of word tokens generated using a suitable word embedding. At425, text encoder component420can encode xitusing the text encoder ft(.; θt) to produce uit=ft(xit; θt). It should be appreciated that in an embodiment, text encoder component420can repeat the encoding process for all utterances in Ct={x1t, x2t, . . . , xNt} to produce the entirety of the encoded text context denoted as Ut={uit}i=1N.

At440, training component307can train speech encoder component410using one or more cross-model loss functions. For example, as described above with reference toFIG.3, training component307can compute the contrastive loss between uisand uitin order to minimize the difference between the two encodings. By doing so, this training process can pass semantic knowledge from text encoder component420to speech encoder component410in order to enable speech encoder component410to perform accurately at deployment time without text encoder component420, thereby enabling end-to-end integration of dialogue history in speech form. As shown, the training at440comprises contrastive loss, but it is to be appreciated that Euclidean loss, or any other suitable binary cross-entropy loss function can be utilized.

FIG.5illustrates a flow diagram of an example, non-limiting method500that can facilitate training of conversation component204in accordance with one or more embodiments described herein.

At530, conversation component204can receive Us={uis}i=1Nas encoded by speech encoder component410. Similarly at540, conversation component204can receive Ut={uit}i=1Nas encoded by text encoder component420. Conversation component204can then encode Usand Utusing the neural network higher level sequence encoder g(.; ϕ) to produce cs=g(Us; ϕ) and ct=g(Ut; ϕ). csand ctcan then be passed to recognition component306, shown here as a classifier. Recognition component306can then generate predictions for SLU categories, slots, and/or values using csand ct. At550, training component307can train conversation component204using binary cross loss entropy between csand ct. It should be appreciated that in an embodiment, the training processes described inFIGS.4and5can be repeated multiple times. For example, the training process can repeat for a set number of iterations, for a set amount of time or until a threshold accuracy level is met or surpassed.

FIG.6illustrates a flow diagram of an example, non-limiting method600that can facilitate performance of spoken language understanding in accordance with one or more embodiments described herein.

At612speech encoder component410, of utterance component305, can receive speech signal xis, from the ithutterance in speech-based context Cs={x1s,x2s, . . . , xNs} , wherein xNsis the current utterance and the rest is dialog history. At615, speech encoder component410can encode xisusing a neural network with the encoder fs(.; θs) to output the encoded context as uis. It should be appreciated that in an embodiment, speech encoder component410can repeat the encoding process for all utterances in Cs={x1s,x2s, . . . , xNs} to produce the entirety of the encoded speech context denoted as Us={uis}i=1N. It should be appreciated that as speech encoder component410was previously trained in reference toFIG.4, text encoder component420is not utilized during performance of SLU operations.

FIG.7illustrates a flow diagram of an example, non-limiting method700that can facilitate performance of spoken language understanding in accordance with one or more embodiments described herein.

At730, conversation component204can receive Us={uis}i=1Nas encoded by speech encoder component410. Conversation component204can then encode Ususing the neural network higher level sequence encoder g(.; ϕ) to produce cs=g(Us; ϕ). cscan then be passed to recognition component306, shown here as a classifier. Recognition component306can then generate predictions for SLU categories, slots, and/or values using cs. It should be appreciated that as conversation component204was previously trained in reference toFIG.5, conversation component204does not receive or utilize Ut={uit}i=1N, as Ut={uit}i=1Nis utilized during to training pass semantic knowledge and is not utilized during performance of SLU operations. As such, it should also be appreciated that this enables end-to-end integration of dialogue history in SLU operations.

FIG.8Aillustrates a chart representation801of the impact of dropping frames during the training process in accordance with one or more embodiments described herein.

Column810shows the number of maximum frames used in training experiments. For example, a number of frames were dropped until the number of frames used in a training experiment equaled the number of max frames in Column810. Column812shows the F1 score of training experiments with different numbers of max frames. F1 score is a value used to measure the accuracy of a trained model and is determined based on a harmonic mean of the precision and recall of the model. Column814shows the training time of training experiments utilizing the various maximum number of speech frames as measured in minutes per epoch.

As shown, row816represents a training experiment utilizing a maximum of 256 frames. Row818represents a training experiment utilizing all speech frames. As shown, row816shows a decrease in training time (e.g., 8 min/epoch as opposed to 27 min/epoch). Accordingly, as shown dropping speech frames during the training process of spoken language understanding system201can provide a reduction in training time while simultaneously providing an increase in performance and accuracy (e.g., F1 score of 61.7 as opposed to F1 score of 59.9).

FIG.8Billustrates a chart representation802of a comparison of the performance and model size of end-to-end models and an existing cascaded style model with gold transcripts in accordance with one or more embodiments described herein.

As shown, column820represents the model tested, column822represents the F1 score (e.g., performance) of the model, and column824represents the number of parameters used in the model (e.g., the size of the model). Row825represents an existing cascade model which comprises an automatic speech recognition component which converts speech into text tokens and then uses a BERT to encode the text context. Row826represents a hierarchical model as described above in relation toFIGS.2and3(e.g., a model utilizing utterance component305and conversation component204) and utilizing contrastive loss in the training process. Row827represents a hierarchical model as described above in relation toFIGS.2and3, utilizing contrastive loss in training in which the sequence encoder g(.; ϕ) utilized by conversation component204is replaced with a 1-layer bidirectional long short-term memory.

As shown, the hierarchical model in row826offers comparable performance (e.g., F1 score) to the exiting model shown in row825, with a score of 61.7 compared to 62.2. Additionally, the hierarchical model in row826utilizes approximately half as many parameters as the existing model shown in row825, with the hierarchical model in row826utilizing 88 parameters as opposed to 168 in the existing model shown in row825. Additionally, the hierarchical model in row827offers a performance reduction (e.g., F1 score) of 0.9% when compared to the F1 score of the existing model shown in row825, while being 64% smaller (e.g., number of parameters). Accordingly, it should be appreciated that both the hierarchical models shown in rows826and827offer similar performance to the existing cascade model shown in row825, while also having reduced size and therefore making the hierarchical models more easily deployed.

FIG.8Cillustrates a chart representation802of a comparison of the performance and model size of end-to-end models and an existing cascaded style model without gold transcripts in accordance with one or more embodiments described herein.

As shown, column830represents the model tested, column832represents the F1 score (e.g., performance) of the model, and column834represents the number of parameters used in the model (e.g., the size of the model). Row835represents an existing cascade model which comprises an automatic speech recognition component which converts speech into text tokens and then uses a BERT to encode the text context. Row836represents a hierarchical model as described above in relation toFIGS.2and3(e.g., a model utilizing utterance component305and conversation component204) and utilizing contrastive loss in the training process. As shown, the existing model represented in row835is susceptible to automatic speech recognition errors and therefor suffers a reduction in F1 score in comparison to the same model when used with gold transcripts as represented in row825of chart802described above. In contrast, the hierarchical model represented in row836has an F1 score of 10% higher than that of the existing model show in row835, thereby illustrating that the hierarchical model represented in row836is end-to-end and therefore robust to ASR errors.

FIG.9illustrates a flow diagram of an example, non-limiting computer-implemented method900that can facilitate end to end integration of dialogue history for spoken language understanding in accordance with one or more embodiments described herein. Repetitive description of like elements and/or processes employed in respective embodiments is omitted for sake of brevity.

At910, computer-implemented method900can comprise encoding, by a system (e.g., spoken language understanding system201, and/or conversation component204) operatively coupled to a processor (e.g., processor203), speech-based content of the utterance and text-based content of the utterance into a uniform representation. For example, as described above in reference toFIGS.2-5, conversation component204can receive Us={uis}i=1Nand Ut={uit}i=1N. Conversation component204can then encode Usand Utusing the neural network higher level sequence encoder g(.; ϕ) to produce cs=g(Us; ϕ) and ct=g(Ut; ϕ).

FIG.10illustrates a flow diagram of an example, non-limiting computer-implemented method1000that can facilitate end to end integration of dialogue history for spoken language understanding in accordance with one or more embodiments described herein. Repetitive description of like elements and/or processes employed in respective embodiments is omitted for sake of brevity.

At1010, computer-implemented method1000can comprise encoding, by a system (e.g., spoken language understanding system201, utterance component305, speech encoder component410and/or text encoder component420) operatively coupled to a processor (e.g., processor203), an utterance and speech-based context of the utterance and the text of the utterance and text-based context of the utterance.

At1020, computer-implemented method900can comprise encoding, by the system (e.g., spoken language understanding system201and/or conversation component204), speech-based content of the utterance and text-based content of the utterance into a uniform representation.

At1030, computer-implemented method1000can comprise recognizing, by the system (e.g., spoken language understanding system201and/or recognition component306), at least one of spoken language understanding categories, slots or values from at least one of the speech-based content or the text-based content.

At1040, computer-implemented method1000can comprise training, by the system (e.g., spoken language understanding system201and/or training component307), a hierarchical conversational model using one or more cross-model loss functions.

At1050, computer-implemented method1000can comprise performing, by the system (e.g., spoken language understanding system201and/or neural network component308), spoken language understanding by applying the hierarchical conversational model to a speech conversation.

Spoken language understanding system201can provide technological improvements to systems, devices, components, operation steps, and/or processing steps associated with spoken language understanding. For example, by training an end-to-end neural network to perform spoken language understanding using dialogue history in speech-from rather than text-form, spoken language understanding system201can utilize dialogue history as context when performing spoken language understanding without first converting the speech to text form. In another example, by dropping speech frames during training, spoken language understanding system201can decrease overall training time while also providing improved accuracy when performing spoken language understanding.

Spoken language understanding system201can provide technical improvements to a processing unit associated with spoken language understanding system201. For example, by training utterance component305and conversation component204to perform spoken language understanding without the text of an utterance or conversation, and thereby decreasing the inputs used to perform spoken language understanding, spoken language understanding system201can reduce the workload of a processing unit (e.g., processor203) that is employed to execute routines (e.g., instructions and/or processing threads) involved in performance of spoken language understanding. In another example, by dropping frames during the training process, spoken language understanding system201can decrease the number of instructions used during a training process while also improving accuracy of the trained model, thereby reducing the workload of the processing unit (e.g., processor203) that is employed to execute the routines (e.g., instructions and/or processing threads) involved in training the model. In these examples, by reducing the workload of such a processing unit (e.g., processor203), spoken language understanding system201can thereby facilitate improved performance, improved efficiency, and/or reduced computational cost associated with such a processing unit.

Spoken language understanding system201can provide technical improvements to a memory unit associated with spoken language understanding system201. For example, by training utterance component305and conversation component204to perform spoken language understanding without text of an utterance or conversation, spoken language understanding system201can decrease the overall model size by not utilizing a speech to text conversion process, thereby reducing storage usage of spoken language understanding system201.

Spoken language understanding system201can employ hardware and/or software to solve problems that are highly technical in nature, that are not abstract and that cannot be performed as a set of mental acts by a human. In some embodiments, one or more of the processes described herein can be performed by one or more specialized computers (e.g., a specialized processing unit, a specialized classical computer, a specialized quantum computer, and/or another type of specialized computer) to execute defined tasks related to the various technologies identified above.

It is to be appreciated that spoken language understanding system201can utilize various combinations of electrical components, mechanical components, and circuity that cannot be replicated in the mind of a human or performed by a human as the various operations that can be executed by spoken language understanding system201and/or components thereof as described herein are operations that are greater than the capability of a human mind. For instance, the amount of data processed, the speed of processing such data, or the types of data processed by spoken language understanding system201over a certain period of time can be greater, faster or different than the amount, speed or data type that can be processed by a human mind over the same period of time.

According to several embodiments, spoken language understanding system201can also be fully operational towards performing one or more other functions (e.g., fully powered on, fully executed, and/or another function) while also performing the various operations described herein. It should be appreciated that such simultaneous multi-operational execution is beyond the capability of a human mind. It should also be appreciated that spoken language understanding system201can include information that is impossible to obtain manually by an entity, such as a human user. For example, the type, amount and/or variety of information included in spoken language understanding system201, and/or conversation component204, utterance component305, recognition component306, training component307, and/or neural network component308can be more complex than information obtained manually by an entity, such as a human user.

In order to provide a context for the various aspects of the disclosed subject matter,FIG.11as well as the following discussion are intended to provide a general description of a suitable environment in which the various aspects of the disclosed subject matter can be implemented.FIG.11illustrates a block diagram of an example, non-limiting operating environment in which one or more embodiments described herein can be facilitated. Repetitive description of like elements employed in other embodiments described herein is omitted for sake of brevity.

With reference toFIG.11, a suitable operating environment1100for implementing various aspects of this disclosure can also include a computer1112. The computer1112can also include a processing unit1114, a system memory1116, and a system bus1118. The system bus1118couples system components including, but not limited to, the system memory1116to the processing unit1114. The processing unit1114can be any of various available processors. Dual microprocessors and other multiprocessor architectures also can be employed as the processing unit1114. The system bus1118can be any of several types of bus structure(s) including the memory bus or memory controller, a peripheral bus or external bus, and/or a local bus using any variety of available bus architectures including, but not limited to, Industrial Standard Architecture (ISA), Micro-Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Firewire (IEEE 1394), and Small Computer Systems Interface (SCSI).

The system memory1116can also include volatile memory1120and nonvolatile memory1122. The basic input/output system (BIOS), containing the basic routines to transfer information between elements within the computer1112, such as during start-up, is stored in nonvolatile memory1122. Computer1112can also include removable/non-removable, volatile/non-volatile computer storage media.FIG.11illustrates, for example, a disk storage1124. Disk storage1124can also include, but is not limited to, devices like a magnetic disk drive, floppy disk drive, tape drive, Jaz drive, Zip drive, LS-110 drive, flash memory card, or memory stick. The disk storage1124also can include storage media separately or in combination with other storage media. To facilitate connection of the disk storage1124to the system bus1118, a removable or non-removable interface is typically used, such as interface1126.FIG.11also depicts software that acts as an intermediary between users and the basic computer resources described in the suitable operating environment1100. Such software can also include, for example, an operating system1128. Operating system1128, which can be stored on disk storage1124, acts to control and allocate resources of the computer1112.

System applications1130take advantage of the management of resources by operating system1128through program modules1132and program data1134, e.g., stored either in system memory1116or on disk storage1124. It is to be appreciated that this disclosure can be implemented with various operating systems or combinations of operating systems. A user enters commands or information into the computer1112through input device(s)1136. Input devices1136include, but are not limited to, a pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, web camera, and the like. These and other input devices connect to the processing unit1114through the system bus1118via interface port(s)1138. Interface port(s)1138include, for example, a serial port, a parallel port, a game port, and a universal serial bus (USB). Output device(s)1140use some of the same type of ports as input device(s)1136. Thus, for example, a USB port can be used to provide input to computer1112, and to output information from computer1112to an output device1140. Output adapter1142is provided to illustrate that there are some output devices1140like monitors, speakers, and printers, among other output devices1140, which require special adapters. The output adapters1142include, by way of illustration and not limitation, video and sound cards that provide a means of connection between the output device1140and the system bus1118. It should be noted that other devices and/or systems of devices provide both input and output capabilities such as remote computer(s)1144.

Computer1112can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s)1144. The remote computer(s)1144can be a computer, a server, a router, a network PC, a workstation, a microprocessor based appliance, a peer device or other common network node and the like, and typically can also include many or all of the elements described relative to computer1112. For purposes of brevity, only a memory storage device846is illustrated with remote computer(s)1144. Remote computer(s)1144is logically connected to computer1112through a network interface1148and then physically connected via communication connection1150. Network interface1148encompasses wire and/or wireless communication networks such as local-area networks (LAN), wide-area networks (WAN), cellular networks, and/or another wire and/or wireless communication network. LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet, Token Ring and the like. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks like Integrated Services Digital Networks (ISDN) and variations thereon, packet switching networks, and Digital Subscriber Lines (DSL). Communication connection(s)1150refers to the hardware/software employed to connect the network interface1148to the system bus1118. While communication connection1150is shown for illustrative clarity inside computer1112, it can also be external to computer1112. The hardware/software for connection to the network interface1148can also include, for exemplary purposes only, internal and external technologies such as, modems including regular telephone grade modems, cable modems and DSL modems, ISDN adapters, and Ethernet cards.

While the subject matter has been described above in the general context of computer-executable instructions of a computer program product that runs on a computer and/or computers, those skilled in the art will recognize that this disclosure also can or can be implemented in combination with other program modules. Generally, program modules include routines, programs, components, data structures, and/or other program modules that perform particular tasks and/or implement particular abstract data types. Moreover, those skilled in the art will appreciate that the inventive computer-implemented methods can be practiced with other computer system configurations, including single-processor or multiprocessor computer systems, mini-computing devices, mainframe computers, as well as computers, hand-held computing devices (e.g., PDA, phone), microprocessor-based or programmable consumer or industrial electronics, and the like. The illustrated aspects can also be practiced in distributed computing environments in which tasks are performed by remote processing devices that are linked through a communications network. However, some, if not all aspects of this disclosure can be practiced on stand-alone computers. In a distributed computing environment, program modules can be located in both local and remote memory storage devices. For example, in one or more embodiments, computer executable components can be executed from memory that can include or be comprised of one or more distributed memory units. As used herein, the term “memory” and “memory unit” are interchangeable. Further, one or more embodiments described herein can execute code of the computer executable components in a distributed manner, e.g., multiple processors combining or working cooperatively to execute code from one or more distributed memory units. As used herein, the term “memory” can encompass a single memory or memory unit at one location or multiple memories or memory units at one or more locations.