Method and apparatus of adaptive textual prediction of voice data

Typical textual prediction of voice data employs a predefined implementation arrangement of a single or multiple prediction sources. Using a predefined implementation arrangement of the prediction sources may not provide a good prediction performance in a consistent manner with variations in voice data quality. Prediction performance may be improved by employing adaptive textual prediction. According to at least one embodiment determining a configuration of a plurality of prediction sources, used for textual interpretation of the voice data, is determined based at least in part on one or more features associated with the voice data or one or more a-priori interpretations of the voice data. A textual output prediction of the voice data is then generated using the plurality of prediction sources according to the determined configuration. Employing an adaptive configuration of the text prediction sources facilitates providing more accurate text transcripts of the voice data.

BACKGROUND OF THE INVENTION

Speech-to-text transcription is commonly used in many applications. The transcription is usually performed by a human agent. However, the use of human agents to transcribe voice data to text is costly, and sometimes the transcription quality is less than satisfactory. With significant advances in speech recognition and language modeling tools, machine-based solutions for speech-to-text transcription are becoming a reality. Such solutions may be used in combination with a human agent or separately.

SUMMARY OF THE INVENTION

According to at least one embodiment, a computerized method or a corresponding apparatus for performing adaptive textual prediction of voice data comprise: determining a configuration of a plurality of prediction sources, used for textual interpretation of the voice data, based at least in part on one or more features associated with the voice data or one or more a-priori interpretations of the voice data; and generating a textual output prediction of the voice data using the plurality of prediction sources according to the configuration determined.

The method further comprises extracting the one or more features associated with the voice data or the one or more a-priori interpretations of the voice data. The one or more features include a signal-to-noise ratio associated with the voice data, complexity measure of a lattice representing at least one a-priori interpretation of the voice data, or an a-priori interpretation of the voice message generated by a human agent. The multiple prediction sources include a language model module, lattice decoder module, or a human agent. The textual output prediction may be provided to a human agent to facilitate generating a final transcript of the voice data. Alternatively, the textual output prediction may be used as the final transcript of the voice data.

In determining the configuration of a plurality of prediction sources, an order according to which the multiple prediction sources are to be applied is determined, weightings associated with the multiple prediction sources are determined, or a subset of the plurality of prediction sources for use in generating the textual output prediction is determined. A representation of the determined configuration may be sent to another device or stored in a database. A database storing a representation of a previous configuration of the plurality of prediction sources may be updated based on the configuration determined. A representation of the determined configuration includes an indication of an order according to which the multiple prediction sources being applied, indication of weightings associated with the multiple prediction sources, or indication of a subset of the plurality of prediction sources for use in generating the textual output prediction.

DETAILED DESCRIPTION OF THE INVENTION

In transcribing voice data into text, the use of human agents alone may be costly and of poor quality sometimes. Agents transcribing hours-long voice data may be under strict time constraints. The voice data may not always have good audio quality. Such factors may result in unsatisfactory transcription results. To address the issues of cost and quality in speech-to-text transcription, computer-based text prediction tools are employed.

Speech recognition applications known in the art may not provide text transcripts corresponding to input speech signals. Instead, an output of a speech recognition application may be in the form of statistical data illustrating different potential interpretations of a respective input speech signal. In addition, speech recognition applications may process a speech signal on a per-utterance or per-phoneme basis and may not consider linguistic rules or the context of the speech, or conversation, associated with the input speech signal. Therefore, the output of a speech recognition application is usually fed to a text prediction source to generate a text prediction corresponding to the input speech signal. A single source or multiple text prediction sources may be applied to a prior text interpretation, e.g., output of a speech recognition application or a transcript by an agent, of a speech signal. While the use of multiple prediction sources usually results in better performance than using a single prediction source, a single arrangement of how such multiple prediction sources are employed may not provide equally good performance under different conditions. In the following, different embodiments of adaptation of multiple text prediction sources are described.

FIG. 1is a block diagram of a system100for speech-to-text transcription according to at least one example embodiment. In the system100, voice data101, e.g., a speech signal, is fed to a speech recognition module110. The speech recognition module110generates text data102as output. The generated text data102may be in the form of statistical data with probability values assigned to text words therein. In other words, the generated text data102may represent potential multiple interpretations of the voice data. The speech recognition module110may include one or more speech recognition applications. A text transcript103of the voice data may also be generated by a human agent through a first agent device180. The output of the speech recognition module102, the text transcript103generated by the agent, or both may be used by the adaptive text prediction module120to generate an output text prediction109of the respective voice data101. The output of the speech recognition module102, the text transcript103generated by the agent, or both may be viewed as a-priori text interpretation(s) of the corresponding voice data101. The output text prediction109provided by the adaptive text prediction module120may be provided to a second agent, associated with a second agent device190, to facilitate providing a final text transcript of the respective voice data101. Alternatively, the output text prediction109generated by the adaptive text prediction module120may be used as the final text transcript of the respective voice data101.

In generating the output text prediction109, the adaptive text prediction module120is configured to employ multiple text prediction sources or tools to the a-priori text interpretations. According to at least one example embodiment, the multiple prediction sources are employed according to adaptable configuration(s). Specifically, the adaptive text prediction module120includes an adaptation module124configured to determine a configuration of the multiple prediction sources based on features105associated with the voice data101, the text data102generated by the speech recognition module110, the text transcript103, or a combination thereof. The adaptive text prediction module120also includes an execution module128configured to execute the determined configuration of the multiple text prediction sources.

The features105may be provided to the adaptation module124by a feature extraction module130. The feature extraction module extracts the features105from voice data101, text data102generated by the speech recognition module110, text transcript103provided by the first agent, or a combination thereof. Examples of the features105include, for example, signal-to-noise ratio of the voice data101, characteristics of the speech recognition module output102, a measure of the accuracy or quality of text transcript103, or the like.

Based on the received features105, the adaptation module124determines the configuration of the multiple prediction sources. According to one scenario, the adaptation module124may analyze the features105to generate further parameters for use in determining the configuration of the multiple prediction sources. Alternatively, the adaptation module124may map the received features105to a particular configuration based on, for example, a mapping table. According to yet another scenario, the adaptation module124may rank or assign a priority value to each of the multiple text prediction sources, based on the received features105, and then determine a configuration based on the ranking or priority values assigned to each text prediction source. The ranking or priority values may be indicative of which text prediction source is to be employed, the order with which a text prediction source is applied, a weighting to be assigned to the output of a text prediction source, or the like.

According to at least one example embodiment, the adaptive text prediction module120is coupled to a database140. The database140may store configuration parameters104associated with each configuration, implementations of different configurations, pointers or application programming interfaces (APIs) for implementations of the different configurations or the like. The database140may alternatively, or in addition, store APIs or implementations of the multiple text prediction sources. As such, a particular configuration may be implemented on the fly by the adaptive text prediction module120using the stored APIs or implementations of the multiple text prediction sources.

Upon determining a configuration of the multiple text prediction sources to be employed, the adaptation module124may inform or instruct an execution module128about the determined configuration. The execution module128is configured to receive the text data102, the text transcript103, or both. The execution module128then applies the determined configuration of the plurality of text prediction sources to one or more of the received a-priori text interpretations, e.g.,102and103. Instructions from the adaptation module may further include an indication of the a-priori text interpretation(s) to be used. Alternatively, such indication may be inherent in the configuration determined or selected by the adaptation module. The execution module may further be configured to retrieve the pointer(s), API(s), or implementation(s) of the selected configuration or of the respective multiple text predictions from the database140.

By executing the selected configuration of the multiple text prediction sources, the execution module128generates an output text prediction109. The output text prediction109may be used as a final text transcript of the voice data101. Alternatively, the output text prediction109may be presented to a second agent, through an agent device190, to facilitate the generation of the final transcript of the voice data101by the second agent. In other words, the second agent may be provided with the voice data audio and the output text prediction109. According to an example scenario, the output text prediction109may be used, for example, as part of an interactive tool which provides or displays prediction(s) of a next word as the second agent types the text transcript of the voice data. According to another scenario, the output text prediction109may be presented to the second agent as a document to be reviewed and edited by the second agent while listening to the voice data audio.

FIG. 2Ais a block diagram illustrating an example configuration200aof the multiple text prediction sources. The example configuration200aof the multiple text prediction sources describes a sequential arrangement of two text prediction sources, a language model (LM) module210afollowed by a lattice decoder module220a. In other words, the a-priori text interpretation202a, e.g., including text data102, text transcript103, or both, is first fed to the LM module210a, which, in turn, generates a first text prediction207a. According to another scenario, no input is fed to the LM module and the LM module, in such case, provides the most likely prediction207abased only on its internal statistical model. For example, for en-US voicemail messages, the LM prediction with no input is likely to be something like “Hi Mike. It's John. Can you call me back?” Where Mike is the most likely recipient name, and John is the most likely caller name. The first text prediction207ais then fed to the lattice decoder module220a. The lattice decoder module220aalso receives a lattice205afrom the speech recognition module110. The lattice is typically a finite state machine representing multiple potential interpretations of the voice data generated by the speech recognition module110. The lattice decoder module220ais configured to parse the lattice205aand generate a second text prediction208a. The second text prediction208acorresponds to the output text prediction109. The second text prediction208ais generated based on the parsed information in the lattice205and the output of the LM module207a. The LM module210amay generate more than one text prediction207a, each with an assigned respective probability or likelihood measure/score. The more than one text predictions207agenerated by the LM module are then used by the lattice decoder module220ato generate the second text prediction208a. The lattice decoder module220amay employ some weighting of the text prediction(s) provided by the LM module210aand the interpretations embedded in the lattice to generate the second text prediction208a.

In a sequential configuration, such as200a, of the multiple text prediction sources, the order of the different text prediction sources is important. For example, in the configuration200athe LM module220ais applied first, and the lattice decoder module210a, is applied second. In an alternative sequential configuration, the lattice decoder module210ais applied first followed by the LM module220a. The order of the multiple text prediction sources, or the corresponding configuration, is determined by the adaptation module124based on features105such as the complexity of the lattice, the signal-to-noise ratio of the voice data101, the text transcript103provided by the first agent, or a combination thereof. With regard to the complexity of the lattice, the more uncertainty is associated with the output of the speech recognition module110, the more complex the lattice is, and the simpler the lattice is, the more reliable the output of the speech recognition module is. In other words, the complexity of the lattice may be viewed as a measure of the reliability of the lattice.

According to at least one example embodiment, if the lattice is determined, e.g., based on a complexity measure, to be simple, the lattice decoder module220ais applied in the beginning of a sequential configuration. Considering a configuration having a LM module210aand a lattice decoder module220a, for example, the order of the LM module210aand the lattice decoder module220awould be reversed compared to the configuration inFIG. 2Awhen the lattice is found to be relatively simple. According to one scenario, a complexity measure of the lattice is the difference between the likelihood or probability scores associated with the best interpretation, e.g., path with highest probability, and the second best interpretation, e.g., path with second highest probability, in the lattice. The larger the difference between such scores, the simpler the lattice is and vise versa. According to another scenario, an alternative complexity measure of the lattice may be the total number of paths in the lattice205a. A person of ordinary skill in the relative art should appreciate that other complexity measures may be used.

The adaptation module124may also use the signal-to-noise ratio of the voice data to determine the configuration of the text prediction sources, or the order of the text prediction sources within the configuration. A high signal-to-noise ratio of the voice data may lead to reliable performance by the speech recognition module110and thus may be indicative of a reliable lattice205a. As such, in the case of a high signal-to-noise ratio, the lattice decoder module220aprecedes the LM module in the determined configuration. If the signal-to-noise is low, the LM module210aprecedes the lattice decoder module220ain the configuration determined by the adaptation module124. In addition, a high signal-to-noise ratio may also be indicative of reliability of the text transcript103generated by the first agent, if available.

FIG. 2Bis a block diagram illustrating an example configuration200bof the multiple text prediction sources. In the configuration200b, the LM module210band the lattice decoder module220bare implemented in parallel. The a-priori text prediction202band the text transcript from a first agent are optional and the LM module210bmay still provide a first text prediction207bwith no a-priori text prediction202bbeing fed to the LM module210b. The text predictions207band208b, e.g., outputs of the LM module210band the lattice decoder module220b, respectively, are fed to a weighting module240b. The weighting module240bapplies weightings, or assigns priority scores, to the text predictions207band208bin order to generate output text prediction209b. The weighting module240bmay further use the text transcript103generated by the first agent, if it's available, in generating the output text prediction209b.

The weighting module240buses features105provided by the feature extraction module130to determine the weights to be applied to each of the text predictions, e.g.,207band208b, or transcripts, e.g.,103, provided to the weighting module240b. Such features105include, for example, the signal-to-noise ratio of the voice data101, the complexity of the lattice205, a measure of the accuracy or quality of the text transcript103, or the like. The weighting module240bmay further use other criteria in applying weighting, or assigning scores, to each of the text predictions, e.g.,207band208b, or transcripts, e.g.,103, provided to the weighting module240b. For example, each of the text prediction sources, e.g.,210band220b, may generate more than one text prediction, e.g.,207band208b. In such case, the weighting module240bmay assign high scores, or apply large weighting, to a text prediction, or a portion of a text prediction, that is the output of more than one text prediction source. For example, a text prediction, or a sentence therein, that appears in the output of the LM module210band the lattice decoder module220bis assigned a relatively higher score than another text prediction, or a sentence therein, that appears in the output of a single text prediction source among210band220b. The weighting module240bmay process text predictions or portions of a text prediction when applying weightings or assigning priority scores.

In the case of a high signal-to-noise ratio of the voice data101, text prediction(s)208bgenerated by the lattice decoder module220bis/are assigned higher priority scores, or larger weightings, than text prediction(s)207bgenerated by the LM module210b. In the case of a low signal-to-noise ratio of the voice data101, however, text prediction(s)207bgenerated by the LM module210bis/are assigned higher priority scores, or larger weightings, than text prediction(s)208bgenerated by the lattice decoder module220b. Text prediction(s)208bgenerated by the lattice decoder module220bmay also be assigned relatively high score(s), or relatively large weighting(s), if the lattice205bhas low complexity. However, if the lattice is determined to be complex, based on a complexity measure, the text prediction(s)208bgenerated by the lattice decoder module220bis assigned relatively low score(s), or relatively small weighting(s). The transcript103, if provided to the weighting module240b, may also be assigned a score or weighting. The weightings or scores are provided by the adaptation module124to the weighting module240b.

FIGS. 3A and 3Bshow tables illustrating example weighting coefficients assigned to different prediction sources under different signal-to-noise ratios. The table inFIG. 3Acorresponds to weightings or scores used in the case of low signal-to-noise ratio of the voice data101. The first column of the table inFIG. 3Ashows scores or weightings for the outputs of LM module210band the lattice decoder module only as no transcript from the first agent is provided. The text prediction(s)207bfrom the LM module210bis/are assigned a score or weighting of 1, while the text prediction(s)208bfrom the lattice decoder module220bis/are assigned a score of 0.75. Such scores may be used, for example, to rank the different inputs to the weighting module240band provide the highest ranked one as the output text prediction209b. In the case where the LM module210b, the lattice decoder module220b, or both provide multiple text predictions with associated probability or likelihood values, the scores or weightings may be used in combination with such probability or likelihood values to determine the output text prediction209b. For example, the probability or likelihood values may be multiplied with the corresponding weightings or scores to generate new scores to be used in determining the output text prediction209b. Alternatively, a different rule or formula may be used to determine the output text prediction209b. The second column of the table inFIG. 3Ashows the scores 0.7, 0.9 and 1, assigned to the text prediction(s)207b, the text prediction(s)208b, and the transcript103, respectively. The score values represent relative weights associated with different predictions.

The table inFIG. 3Billustrates examples scores or weightings used in the case of high signal-to-noise ratio of the voice data101. The first column corresponds to the case where no text transcript103is provided by the first agent and the second column corresponds to the case where a text103transcript is provided by the first agent. Relatively high scores are assigned, in this case, to outputs from both the LM module210band the lattice decoder module220b.

FIG. 4is a flow chart illustrating a method400of text prediction according to at least one example embodiment. At block410, a configuration of a plurality of prediction sources, used for text prediction, is determined based at least in part on one or more features105related to voice data101and/or one or more a-priori text interpretations. In determining the configuration, the features105may be analyzed, and a decision is then made based on the analysis of the features105. One or more scores, or weightings, may be assigned to outputs of different text prediction sources based on the features105or the analysis thereof. The text prediction sources may be ranked based on the features105. The method may further include extracting the features. At block420, an output text prediction, e.g.,109,208aor209b, of the voice data101is generated using the plurality of prediction sources according to the determined configuration.

It should be understood that the example embodiments described above may be implemented in many different ways. In some instances, the various methods and machines described herein may each be implemented by a physical, virtual or hybrid general purpose or application specific computer having a central processor, memory, disk or other mass storage, communication interface(s), input/output (I/O) device(s), and other peripherals. The general purpose or application specific computer is transformed into the machines that execute the methods described above, for example, by loading software instructions into a data processor, and then causing execution of the instructions to carry out the functions described, herein.

As is known in the art, such a computer may contain a system bus, where a bus is a set of hardware lines used for data transfer among the components of a computer or processing system. The bus or busses are essentially shared conduit(s) that connect different elements of the computer system, e.g., processor, disk storage, memory, input/output ports, network ports, etc., that enables the transfer of information between the elements. One or more central processor units are attached to the system bus and provide for the execution of computer instructions. Also attached to the system bus are typically I/O device interfaces for connecting various input and output devices, e.g., keyboard, mouse, displays, printers, speakers, etc., to the computer. Network interface(s) allow the computer to connect to various other devices attached to a network. Memory provides volatile storage for computer software instructions and data used to implement an embodiment. Disk or other mass storage provides non-volatile storage for computer software instructions and data used to implement, for example, the various procedures described herein.

Embodiments may therefore typically be implemented in hardware, firmware, software, or any combination thereof.

In certain embodiments, the procedures, devices, and processes described herein constitute a computer program product, including a computer readable medium, e.g., a removable storage medium such as one or more DVD-ROM's, CD-ROM's, diskettes, tapes, etc., that provides at least a portion of the software instructions for the system. Such a computer program product can be installed by any suitable software installation procedure, as is well known in the art. In another embodiment, at least a portion of the software instructions may also be downloaded over a cable, communication and/or wireless connection.

Embodiments may also be implemented as instructions stored on a non-transitory machine-readable medium, which may be read and executed by one or more processors. A non-transient machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine, e.g., a computing device. For example, a non-transient machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; and others.

Further, firmware, software, routines, or instructions may be described herein as performing certain actions and/or functions of the data processors. However, it should be appreciated that such descriptions contained herein are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc.

It also should be understood that the flow diagrams, block diagrams, and network diagrams may include more or fewer elements, be arranged differently, or be represented differently. But it further should be understood that certain implementations may dictate the block and network diagrams and the number of block and network diagrams illustrating the execution of the embodiments be implemented in a particular way.

Accordingly, further embodiments may also be implemented in a variety of computer architectures, physical, virtual, cloud computers, and/or some combination thereof, and, thus, the data processors described herein are intended for purposes of illustration only and not as a limitation of the embodiments.