Voice Recognition for Voice to Text Generated Typeahead In Suggested Application

Techniques for providing voice to text generated typeahead suggestions for lists of items from voice data are provided. In one aspect, a method for providing typeahead includes: identifying lists of items captured as text from voice data on a Voice Over Internet Protocol connection using voice recognition and natural language processing; and storing the lists of items in memory thereby enabling the typeahead of the lists of items on one or more electronic devices. Further, a suggested application can be provided, and the lists of items are then autocompleted on the one or more electronic devices in the suggested application.

FIELD OF THE INVENTION

The present invention relates to voice recognition-based technology, and more particularly, to techniques for providing voice to text generated typeahead suggestions for lists of items from voice data in suggested applications using voice recognition.

BACKGROUND OF THE INVENTION

Web-based audio/video conference call applications facilitate collaboration amongst parties irrespective of their physical location. During a collaborative session, participants often exchange useful information which needs to be retained for later use. For instance, this information can include lists of items such as addresses, telephone numbers, etc. Participants can simply jot down the information during the collaborative session, and type it again when needed.

Doing so, however, diverts the participants' attention away from the speaker. There is also the risk that the process of later reproducing this information can introduce errors. Further, when the user goes back to type in the information, it can be a tedious task to constantly go back and forth with one's notes to verify each item on the list. For instance, when subsequently entering a list of items into the form fields of an application, such as an address in a web-based mapping service, one must often go back and forth inquiring whether any item of information was missed in the transcription.

Therefore, techniques that automate the process for retaining lists of items from a collaborative session, and intelligently leveraging this information for future use would be desirable.

SUMMARY OF THE INVENTION

The present invention provides techniques for providing voice to text generated typeahead suggestions for lists of items from voice data in suggested applications using voice recognition. In one aspect of the invention, a method for providing typeahead is provided. The method includes: identifying lists of items captured as text from voice data on a Voice Over Internet Protocol connection using voice recognition and natural language processing; and storing the lists of items in memory thereby enabling the typeahead of the lists of items on one or more electronic devices.

For instance, the lists of items can be retrieved from the memory, and the lists of items autocompleted on the one or more electronic devices whenever items from the lists of items are being inputted into the one or more electronic devices (e.g., whenever multiple consecutive characters are being inputted that are contained within strings located in the memory). Further, a suggested application can be provided, and the lists of items autocompleted on the one or more electronic devices in the suggested application.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

As provided above, the exchange of information during conference calls or other web-based collaborative sessions typically involves individual participants jotting down the information from the speaker. Doing so, however, can be tedious, especially when the information includes lists of items that must be precisely recorded for later use such as addresses, telephone numbers, etc. Furthermore, diverting one's attention away from the speaker to write down a list of items can cause a participant to miss out on other pertinent information. There is also the risk that errors are introduced during transcription of the list. Later, when the participant goes back to type in the information, there is the equally tedious task of verifying each item on the list.

Take for instance a scenario where the speaker on a web-based conference call is reciting an address. Traditionally, the participants to the call would individually record the address on their end, such as writing it down on a notepad or using a note-taking application. Later, when any of the participants wants to use the address (e.g., to type or otherwise input it into a web-based mapping service), they have to go back and forth with their notes to ensure that the correct address information is properly entered into each respective field, i.e., street, city, state, zip code, etc.

Advantageously, the present techniques provide an automated system200and process (see below) for actively capturing such lists of items from the speaker as text data using voice recognition, and instantiating that text data in memory for generating typeahead suggestions. The term “typeahead” as used herein generally refers to a prediction tool that provides suggestions for users typing data into the fields of a computer-based form or document. Typeahead may also be referred to herein as ‘autocomplete’ or ‘autosuggest.’ Thus, the terms ‘typeahead,’ “autocomplete” and ‘autosuggest’ may be used interchangeably herein when referring to a computer-based feature that auto-populates text in an application or other field. For instance, if a user is typing a query into a search engine, a typeahead provision might make suggestions based on the first few characters the user enters into the query field. Doing so saves the user the cumbersome task of manually typing each query in its entirety. As will be described in detail below, in accordance with the present techniques, the typeahead feature in connected devices will be leveraged to facilitate use of the captured lists of items. In this manner the participants will not have to recreate the lists themselves whenever they want to use to information, it will be done for them in an automated manner. For instance, again using the exemplary scenario where the speaker recites an address, the present system200will use voice recognition to actively capture voice data from the speaker and derive the (spoken) address as text. That derived text is then stored in memory to enable typeahead of the address. Thus, when any of the participants/users later needs to input the address information into the fields of, e.g., a web-based mapping service, typeahead suggestions can be provided which the participant/user can accept to automatically populate the fields. Doing so advantageously provides a more efficient way for users to input information. For example, the typeahead feature can present autosuggest text to the user on a display or other user interface of the user's device. In one exemplary embodiment, the present typeahead feature can be initiated whenever the participant/user begins typing items contained in the memory (and which correspond to the saved lists of items). As will be described in detail below, the derived text can be instantiated in the memories of the participants' devices, e.g., the participants' computers, mobile devices such as smartphones, etc. However, storing the derived text locally with the end-user is not a requirement, and embodiments are also contemplated herein where system200includes a central database or other repository for storing the lists of items, which can be implemented by system200to generate typeahead in the devices of one or more of the users.

As will be described in detail below, system200can also employ natural language processing to contextualize the captured data into predefined categories. For instance, system200can recognize the derived text as being addresses, telephone numbers and/or connected lists of items. Further, embodiments are contemplated herein where system200also provides a suggested application and automatically distributes the typeahead text to that suggested application. For instance, when a user wants directions for an address on the list, the present system200can also suggest a particular mapping application and autocomplete the address fields with the derived text. The user then simply has to review and submit the query to obtain the results.

Specifically, referring toFIG.2, an exemplary configuration of system200is provided. According to the exemplary embodiment shown inFIG.2, system200includes at least a voice recognition module202, a natural language processing module204, a memory210, a typeahead module208and a recommender210. Rather than taking a passive role, voice recognition module202enables system200to actively capture voice data during a web-based call such as a Voice Over Internet Protocol (VOIP) connection, and derive text using its voice to text capabilities. The phrase “Voice Over Internet Protocol.” as used herein generally refers to any technology that provides voice communications over Internet Protocol networks such as the internet. State of the art voice recognition (also known as speech recognition) uses artificial intelligence and machine learning to identify, understand and process voice data including human speech. Among the capabilities of most voice recognition technology is the ability to translate speech/voice data to text data, i.e., voice to text. Thus, through voice to text, voice recognition module202captures what a speaker or speakers is/are saying during the web-based call and from that speech/voice data derives text data (also referred to herein as ‘derived text’).

In one exemplary, non-limiting example, voice recognition module202employs a neural network such as Long Short-Term Memory (LSTM). LSTM is a recurrent neural network having feedback connections which enables it to process sequences of data like voice data, making it an ideal tool for voice recognition. LSTM can be embodied in a neural network (such as neural network600ofFIG.6—described below), having an input layer, a hidden layer, and an output layer, with the hidden layer containing memory cells and corresponding gate units.

Of particular interest are the lists of items a speaker may mention during the call, such as addresses, telephone numbers, lists of numbered items, etc. Thus, in accordance with the present techniques, the term ‘list’ as used herein generally refers to any meaningful grouping or sequence of ordered items, all belonging to a common category. Thus, for example, if a telephone number is made up of three numbers/hyphen/three numbers/hyphen/four numbers, then a grouping of items that each follows that convention may be considered as a list of telephone numbers. Similarly, if an address is made up a street/city or town/state/zip code, then a grouping of items that each follows that particular convention may be considered as a list of addresses, and so on.

To specifically enable the identification and capturing of lists of items from the speech data, system200employs natural language processing module204. Natural language processing (or NLP) is a machine-learning technique by which computers interpret and manipulate human language in the form of text and/or voice data. Natural language processing combines computational linguistics (i.e., rule-based modeling of human language) with statistical, machine learning, and deep learning models in order to understand the full meaning of the data including the author's/speaker's intent. To do so, the input data can be separated into fragments enabling the grammatical structure of sentences and the meaning of words to be analyzed and understood in their present context.

It is notable that, while natural language processing module204is depicted inFIG.2as being distinct from voice recognition module202, this is done merely to highlight the individual capabilities of system200. Namely, embodiments are contemplated herein where natural language processing is one of the capabilities leveraged by voice recognition module202to understand and process the input voice data from the speaker(s).

Thus, according to an exemplary embodiment, natural language processing module204processes the derived (voice to text) data from voice recognition module202to identify lists of items in the data anytime words arranged in a specific format are detected, e.g., a group or sequence of ordered items that belong to a common category. For instance, as provided above, a sequence of items having the format three numbers/hyphen/three numbers/hyphen/four numbers may be identified by natural language processing module204as a telephone number. A sequence of items having the format street/city or town/state/zip code may be identified by natural language processing module204as an address. Additional examples include, for example, an ordered sequence of items having the format first, second, third, etc., or step 1, step 2, step 3, etc. which natural language processing module204may identify as an ordered list of connected items.

Embodiments are also contemplated herein where, in addition to identifying lists of items in the derived text from voice recognition module202, natural language processing module204further classifies the lists of items into different predefined groups or categories. For instance, natural language processing can employ text classifiers to automatically analyze the derived text and assign a predefined category (or categories) to the analyzed text based on its content. These text classifiers are trained on past/historical data such as that contained in a knowledge corpus (see below). In accordance with the present techniques, some exemplary predefined categories include, but are not limited to, telephone numbers, addresses, ordered lists, etc. (see the specific examples provided above). Advantageously, classifying the lists of items facilitates the subsequent typeahead feature. Namely, as will be described in detail below, the lists of items in a given category can automatically be distributed to the same suggested application (e.g., all derived lists of items that fall into the predefined category ‘Addresses’ can be distributed to the same web-based mapping service), rather than individually evaluating each list each time it is called up by a user.

To further facilitate the identification and classification process, voice recognition module202and natural language processing module204may also be used to capture the context of the conversation (i.e., between speaker and participant(s)). For instance, phrases such as ‘Write this down’ or ‘Here is the list’ can be leveraged to identify and/or categorize the associated text as a list of items. To use an illustrative, non-limiting example, say for instance that the speaker says “The following is the list of telephone numbers: [Telephone Number 1], [Telephone Number 2] and [Telephone Number 3].” Natural language processing module204can leverage this phrase that precedes the items. i.e., [Telephone Number 1]. [Telephone Number 2] and [Telephone Number 3] to a) identify the items as belonging to a list and b) categorize the items as telephone numbers.

This process of contextualizing and classifying the derived text can leverage a knowledge corpus206(e.g., a text database) and/or a repository208of context indicators. For instance, the knowledge corpus206can provide annotated/labeled historical text data that is used to train the text classifiers. For tasks such as text classification, it is notable that labeling of the text data is needed. When text is encountered that is not in the knowledge corpus206, the repository208of context indicators can be used to annotate the text. The term ‘context indicators’ as used herein generally refers to any information that contextualizes an item relative to its content, such as an annotation or label given to text data. These context indicators can be obtained using any commercially-available annotation tool and/or from an open-source library.

Furthermore, embodiments are also contemplated herein where natural language processing module204detects contextual patterns and/or priority in the derived text. For instance, a conversation about a particular topic might list items out of order, however the speaker explains the priority of the items as the speaker goes through them. For instance, in a conversation about the location of a business, the speaker might mention something like “You can't miss it, it's the biggest building in Springfield” but then goes on to provide the exact address starting with the street number. In that case, natural language processing module204can detect the contextual patterns in the transcribed text, recognize that an address is being given and provide the respective items in the correct order for an address format.

As shown inFIG.2, the list of (text) items identified (and preferably classified/categorized) by natural language processing module204are then stored in a memory210. In one exemplary embodiment, memory210is present in one or more connected devices. For instance, participants to the conversation access the Voice Over Internet Protocol connection via their respective devices, i.e., computers, mobile devices such as smartphones, etc. For those users who opt in, instantiating the captured lists in the memory of these users' devices enables a subsequent typeahead feature, e.g., whenever a user begins to type (or otherwise input) items in the recognized list on their connected device (see below). However, storing the recognized lists in the memory of end-user devices is only one possible implementation contemplated herein. For instance, memory210may instead be a cache associated with system200and can be representative of any of the data stores shown inFIG.1, including, but not limited to, volatile memory112, persistent storage113and/or cache121. Again, for users that opt in to the present typeahead features, system200can then detect whenever any of those users is typing the items and autocomplete the list for that user. System200can also go a step further by providing a suggested application for the list, and then auto-populating the field(s) in that suggested application with the derived text. For instance, if a user begins typing an ordered list of the items given by the speaker, system200can suggest a note taking application and automatically populate its fields with the list on the user's device. This avoids the tedious task of having to re-type the complete list word for word.

Thus, the present typeahead feature can be triggered whenever system200detects that a user is typing items contained in memory210. For instance, according to an exemplary embodiment, system200can detect multiple consecutive characters inputted by the user that are contained within strings (e.g., sequences of characters) located in memory210and autosuggest the stored relevant text, preferably in an application suggested by system200.

Namely, typeahead module212can monitor activity on the connected devices, and recognize when a respective user of one of these devices is typing items that coincide with the text stored in memory210. Typeahead module212will then auto-populate the list on the user's device. For instance, when typeahead module212detects that the user is typing multiple consecutive characters that coincide with list items stored in memory210, typeahead module212triggers the typeahead feature for these items. Optionally, typeahead module212may employ a user interface that displays the typeahead suggested text such as by way of a caption or other feature on the user's screen. By way of this interface, the user can opt in or otherwise reject the suggestion being made by system200to autocomplete the fields.

As highlighted above, typeahead module212can work in concert with a recommender214which, in addition to making typeahead suggestions, may also suggest an application for the user that is appropriate for the items in the autocompleted list. For example, if the user begins typing consecutive characters on the user's smartphone or computer from a list of direction items contained in memory210, recommender214can recommend a publicly-available web-based mapping application, and typeahead module212can auto-populate the fields of that mapping application's search function. As highlighted above, system200can optionally display the typeahead suggestions in a caption or other interface on the user's device, allowing the user to accept, reject or even edit the suggested text. Advantageously, by way of this process, the user merely has to type (or otherwise input) the first few characters of the address to receive a fully completed search result in an appropriate mapping application. As illustrated by the examples provided below, the same technique can be applied to lists of items in a wide variety of applications such as, but not limited to, note taking applications, etc.

FIG.3is a diagram illustrating an exemplary methodology300for providing typeahead suggestions performed, for example, by system200above. In step302, voice recognition is used to capture what a speaker or speakers is/are saying during a web-based call and from that speech/voice data derive text using, for example, the voice to text capabilities of voice recognition module202. According to an exemplary embodiment, the web-based call is carried out over a Voice Over Internet Protocol (VOIP) connection.

As provided above, of particular interest in the present techniques are the lists of items a speaker may mention during the call, such as lists of addresses, telephone numbers, numbered/ordered items, etc. Thus, in step304, lists of items are identified in the derived text using natural language processing. For instance, according to an exemplary embodiment, the natural language processing module204of system200is used to detect words arranged in a specific format such as a group or sequence of ordered items that belong to a common category like a sequence of items having the format three numbers/hyphen/three numbers/hyphen/four numbers which natural language processing module204might identify as a telephone number, or a sequence of items having the format street/city or town/state/zip code natural language processing module204might identify as a list of addresses.

Embodiments are also contemplated herein where, in addition to identifying lists of items, natural language processing module204also captures contextual data from the derived text. The term ‘contextual data’ as used herein generally refers to any elements of the derived text that help capture the context of the conversation. For instance, the speaker might identify items as being part of a list by saying something like ‘write this list of steps down.’ While not part of the list itself, this contextual data helps identify the items that follow as a list of steps.

Optionally, in step306, the lists of items identified in the derived text are further classified into different predefined categories. For instance, according to an exemplary embodiment, natural language processing module204is configured to use text classifiers to analyze the lists of items and automatically assign them to a predefined category (or multiple predefined categories as the case may be) based on the content of the lists. The text classifiers can use machine learning or semantically-relevant elements of the text to classify the lists into one or more of the predefined categories. For instance, by way of example only, the predefined categories might include, but are not limited to, telephone numbers, addresses, ordered lists, etc. Categorizing the lists facilitates typeahead into the appropriate suggested application. Namely, the suggested application can be selected based on what predefined category a given list of items is classified in. For instance, all lists categorized as addresses can be distributed to a common web-based mapping application.

As highlighted above, natural language processing module204can use text classifiers trained using labeled data from the knowledge corpus206. The training data can be supplemented with context indicators from the repository208when elements are encountered that are not already in the knowledge corpus206. These context indicators can be obtained using any commercially-available annotation tool and/or from an open-source library.

In step308, the lists of items from step304(preferably classified/categorized in step306) are then stored in memory210. According to an exemplary embodiment, memory210represents the memory present in the connected devices, i.e., computers, mobile devices such as smartphones, etc., of the participants in the web-based call that opt in to the present typeahead features. In that case, step308involves instantiating the lists of items304in the memory of each of those participating devices. Doing so will enable typeahead to be implemented whenever the respective user begins to type (or otherwise input) items in the recognized list (see below). However, embodiments are also contemplated herein where system200employs its own cache (e.g., volatile memory112, persistent storage113and/or cache121inFIG.1) to store the lists of items. System200can then detect when any of the connected users that opts in to the present typeahead features begins typing the items on their respective devices, and autocomplete the list for that user.

Namely, in step310, the list of items is autocompleted from the memory210(via a typeahead feature) on any of the participating connected devices whenever the respective user begins typing the list. For example, if an address is stored in memory210, and one of the connected users begins typing the first few characters, e.g., ‘123 Map1’ into a web-based mapping application, then typeahead module212will detect that the user is recalling one of the lists of items, retrieve that list from memory210, and autocomplete the remaining fields of the web-based mapping application, e.g., with ‘123 Maple Street, Anytown, NY’. According to an exemplary embodiment, the typeahead feature is activated when typeahead module212detects multiple consecutive characters inputted by the user (e.g., by detecting keystrokes) that are contained within strings located in memory210. As highlighted above, system200can optionally present the user with the typeahead suggested text in a caption or other similar user interface on the user's screen which appears, for example, next to the field which will be auto-populated. The user can then be given the option to either accept or reject the suggestions being made by system200to autocomplete the fields.

Preferably, this typeahead suggestion in step310is auto-populated in an application that system200also suggests to the user on the user's connected device. For example, reference is now made toFIG.4which provides one exemplary way for carrying out step310of methodology300. For instance, according to methodology400ofFIG.4, whenever a user begins typing one of the items stored in memory210, such as when the user types (or otherwise inputs) ‘123 Map1’ in a web browser, in step402system200detects that the user is typing one of the stored lists of items. In step404, system200will also recognize that this list has been classified into a predefined category, in this case “address” (see description of step306, above). In step406, recommender214suggests an appropriate application for the retrieved data. For example, if the list in question belongs to the predefined category of addresses, then the recommender214can suggest a web-based mapping application to the user. According to an exemplary embodiment, the recommender214suggests the same application for each of the lists falling under the same predefined category, e.g., all addresses go to the same web-based mapping application. In step408, the list that the user has begun typing is retrieved from the memory210. In step410, typeahead module212then autocompletes the list in the suggested application. It is notable that, while it is a time-saving benefit to the user, selecting the application for typeahead is not a requirement, and embodiments are contemplated herein where the typeahead feature is implemented in an application chosen by the user.

As described above, the captured lists of items can be instantiated in the memory of each of the connected devices that opts in to the present typeahead features. In that case, each participating device will receive and store in its memory the lists of items that were identified from text captured from speech from a web-based call as per steps302and304of methodology300described above, and which have been optionally classified into one or more predefined categories as per step306of methodology300described above. Whenever the respective user of the device begins typing (or in some other way inputting) items on the list (e.g., the user types multiple consecutive characters that are contained within strings located in the device's memory), the device will autocomplete the fields (preferably in a suggested application) using the list retrieved from memory as per step310of methodology300described above.

Alternatively, in the case where system200employs its own cache to store the lists of items (obtained as per steps302-306of methodology300described above), then system200can monitor activity on the participating devices. Whenever system200detects that a respective one of the users is beginning to type (or in another way input) items on the list (e.g., the user types multiple consecutive characters that are contained within strings located in the device's memory), then system200will autocomplete the fields (preferably in a suggested application) using the list retrieved from memory as per step310of methodology300described above.

The present techniques are further illustrated by way of reference to the following non-limiting use case examples. Use Case—Example #1: User A is talking to User B over a Voice Over Internet Protocol (VOIP) connection. User B is telling User A the address and telephone number of a client. The device of User A instantiates the text in its memory. If the present system detects User A beginning to type the address, the autosuggest offers User A a complete address typeahead and opens the most applicable application to auto-populate the data in.

Use Case—Example #2: User A is listening and taking notes in User B's virtual presentation about learning how to code in a particular computer programming language. User B describes a list of dependencies to setup, learn and then finally run a program. The present system200recognizes the items as a list, since User B is numbering them as he talks through the topics. System200then detects that User A is beginning to type items (the numbered list) in the recognized list and offers typeahead containing the complete list within User A's note taking application. For example, in this scenario, system200might generate the following typeahead sample,

Coding Steps for Starters:

Methodology500ofFIG.5provides another exemplary implementation of the present techniques. Referring to methodology500, in step502the users/participants to a web-based conference call are given the option to opt in to the present voice to text generated typeahead feature. In step504, the conference call software integrates with audio streams and the operating system of the participating users, i.e., those that have opted in. In step506, the present system200captures natural language processing (NLP) of speech to text of the words the audience mentions.

In step508, speech to text is generated and a corpus is created anytime items in a specific format (namely that of a list) are detected. In the same manner as described in detail above, these identified lists of items can further be classified into one or more predefined categories such as addresses, lists, telephone numbers, etc. leveraging a knowledge corpus510and/or repository512of context indicators for the classification.

In step514, the speech to text captured lists of items are compared to the users' operating systems and applications. In step516, keystrokes or data entry fields are detected that overlap with the captured lists. As such, the present typeahead feature is employed where, in step518, the (originally spoken) lists are pasted as text into the fields of the applicable application.

As provided above, the present system can employ LSTM for voice recognition. LSTM can be embodied in a neural network such as neural network600ofFIG.6. As shown inFIG.6, neural network600includes a plurality of interconnected processor elements602,604/606and608that form an input layer, at least one hidden layer, and an output layer, respectively, of the neural network600. The connections in neural networks that carry electronic messages between simulated neurons are provided with numeric weights that correspond to the strength or weakness of a given connection. These numeric weights can be adjusted and tuned based on experience, making neural networks adaptive to inputs and capable of learning. Typically, neural networks are trained on labeled sets of training data. Once trained, the neural network can be used for inference. Inference applies knowledge from a trained neural network model and uses it to infer a result. A fully connected layer (typically the last or last few layers in a neural network) is a layer where all of the inputs from one layer are connected to every activation unit of the next layer. The fully connected layer(s) compile the data extracted by previous layers of the neural network to form the final output.