Patent Publication Number: US-7907705-B1

Title: Speech to text for assisted form completion

Description:
BACKGROUND 
     Commonly in business, companies are required to record customer information received over a phone call. This information must be rapidly and accurately recorded to ensure the best possible customer service and satisfaction. The processes used to acquire this information have evolved over time as technologies, such as computer applications, have improved. 
     Typically, upon receiving a call, small businesses record customer information by having an operator take notes onto a paper notepad or a form. These notes may later be transcribed into a computer system or filed in order to keep a historical record of the transaction. Alternatively, in larger businesses, phone operators enter the customer information directly into a computer system while on the phone with the customer. 
     Speech recognition is a process by which speech is converted to text by a computer program. Speech recognition can be used to capture verbal commands or data to be used as information in computer programs. 
     A common use of speech recognition is in dictation systems, such as Dragon Naturally Speaking® (Dragon Naturally Speaking is a registered trademark of Nuance Communications, Inc., headquartered in Burlington, Mass.). In a dictation system, the user speaks into a microphone connected to a computer, and the words spoken are interpreted either as commands to be run on the computer or as transcription that should be entered into, for example, a word processing program. A dictation system typically has a process by which it learns to better recognize the speech of the user in order to improve the accuracy of the speech recognition. 
     In the case of an Interactive Voice Response (IVR) system, speech recognition is used to both capture commands and data in order to facility customer service in a phone operating system. A customer calling into such a system would be prompted by an IVR system with a series of questions that when answered may be confirmed by the IVR system with questions, such as “Did you say billing?” IVR systems are typically used to obtain preliminary information before forwarding to an actual operator or responding to simple inquiries that do no require operator intervention. 
     SUMMARY 
     In general, in one aspect, the invention relates to a method for capturing information from a live conversation between an operator and a customer, comprising monitoring the live conversation between the operator and the customer, recognizing at least one portion of the live conversation as a text portion after converting the live conversation to text, interpreting a cue in the live conversation, relating the cue to an information field associated with a context for the live conversation, and storing information obtained from the text portion into the information field, wherein the information obtained from the text portion comprises at least one word spoken after the cue. 
     In general, in one aspect, the invention relates to a system for capturing information from a live conversation between an operator and a customer, comprising a speech recognition engine, wherein the speech recognition engine is configured to recognize the live conversation as a text portion after converting the live conversation to text, a context designator configured to designate a context for storing information from the text portion obtained by interpreting a cue in the live conversation, and a document completion engine configured to record information obtained from the text portion into an information field within the context, wherein the information obtained from the text portion comprises at least one word spoken after the cue. 
     In general, in one aspect, the invention relates to a computer readable medium comprising instructions to perform a method for capturing information from a live conversation between an operator and a customer, comprising monitoring the live conversation between the operator and the customer, recognizing at least one portion of the live conversation as a text portion after converting the live conversation to text, interpreting a cue in the live conversation, relating the cue to an information field associated with a context for the live conversation, and storing information obtained from the text portion into the information field, wherein the information obtained from the text portion comprises at least one word spoken after the cue. 
     In general, in one aspect, the invention relates to a computer readable medium organized in a library comprising instructions to provide a method for capturing information from a live conversation between an operator and a customer, comprising monitoring the live conversation between the operator and the customer, recognizing at least one portion of the live conversation as a text portion after converting the live conversation to text, interpreting a cue in the live conversation, relating the cue to an information field associated with a context for the live conversation, and storing information obtained from the text portion into the information field, wherein the information obtained from the text portion comprises at least one word spoken after the cue. 
     In general, in one aspect, the invention relates to a computer system for capturing information from a live conversation between an operator and a customer, comprising a processor, a memory, a storage device, and software instruction stored in the memory for enabling the computer system under control of the processor to monitor the live conversation between the operator and the customer, recognize at least one portion of the live conversation as a text portion after converting the live conversation to text, interpret a cue in the live conversation, relate the cue to an information field associated with a context for the live conversation, and store information obtained from the text portion into the information field, wherein the information obtained from the text portion comprises at least one word spoken after the cue. 
     Other aspects of the invention will be apparent from the following description and the appended claims. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG. 1  shows a system in accordance with one or more embodiments of the invention. 
         FIGS. 2-3  show flow charts for converting a live conversation to text for assisted form filling in accordance with one or more embodiments of the invention. 
         FIG. 4  shows a computer system in accordance with on or more embodiments of the invention. 
     
    
    
     DETAILED DESCRIPTION 
     Specific embodiments of the invention will now be described in detail with reference to the accompanying figures. Like elements in the various figures are denoted by like reference numerals for consistency. 
     In the following detailed description of embodiments of the invention, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to one of ordinary skill in the art that the invention may be practiced without these specific details. In other instances, well-known features have not been described in detail to avoid obscuring the invention. 
     In general, embodiments of the invention relate to providing a method and a system for collecting information. More specifically, embodiments of the invention relate using speech recognition technology to capture information electronically. Further, embodiments of the invention provide a method and a system to recognize a live conversation between two individuals as speech, and use cues to interpret and extract information from one or more portions of the live conversation. 
       FIG. 1  shows a system for assisted form completion in accordance with one or more embodiments of the invention. The system architecture includes a customer ( 102 ) interacting with an operator ( 106 ) via a phone system ( 104 ). In one embodiment of the invention, the operator ( 106 ) and the phone system ( 104 ) are communicating with the Document Assistance System (DAS) ( 108 ). The DAS ( 108 ) further includes a Context Designator ( 110 ), a Document Completion Engine ( 112 ), and a Speech Recognition Engine ( 114 ). Optionally, the Speech Recognition Engine ( 114 ) may interact with a grammar repository ( 116 ). Each of the aforementioned components of  FIG. 1  is described below. 
     In one or more embodiments of the invention, the customer ( 102 ) may be any individual that is capable of participating in a live conversation. For example, the customer ( 102 ) may be an individual representing a household or a family, an individual associated with a corporation or company, etc. In one or more embodiments of the invention, the customer may be a vendor, a banker, a business partner, or any other individual that receives or provides goods/services to/from the operator ( 106 ) or the organization represented by the operator ( 106 ). 
     Further, the operator ( 106 ) may be an individual working for a variety of organizations including: a small business, a large corporation, a service center, etc. In one or more embodiment of the invention, the phone system ( 104 ) may be any system incorporating at least one phone and a computer system, such as, a single business phone, a switchboard that operates several phones and phone lines at a time, etc. In one or more embodiments of the invention, the customer ( 102 ) and the operator ( 106 ) conduct a live, real-time conversation, which is channeled through the phone system ( 104 ). Specifically, the customer ( 102 ) may place a phone call to the operator or the operator&#39;s place of employment for some type of customer support. For example, the customer ( 102 ) may provide/request particular information, schedule an appointment to receive services provided by the operator or the operator&#39;s ( 106 ) employer, make a purchase/sell a produce over the telephone, request for a modification to be made to particular information associated with the customer or the customer&#39;s organization, etc. In one or more embodiments of the invention, the phone system ( 104 ) may be used for any phone interaction that is part of a computer workflow (e.g., vendor, supplier, banker, etc.). 
     Further, the live conversation described above may occur over a variety of communication devices. For example, the conversation could occur between two standard phones over a standard phone line, between a digital phone and a standard phone over a hybrid connection, between two digital phones over a digital connection, between two computer applications with voice chat capabilities, etc. 
     In one or more embodiments of the invention, the phone system ( 104 ) is configured to forward the live conversation between the customer ( 102 ) and the operator ( 106 ) to the Document Assistance System (DAS) ( 108 ). As shown in  FIG. 1 , the DAS ( 108 ) includes a Context Designator ( 110 ), a Document Completion Engine ( 112 ), and a Speech Recognition Engine ( 114 ). In one or more embodiments of the invention, the DAS ( 108 ) is responsible for processing the live conversation and capturing information from the live conversation. In one embodiment of the invention, the DAS ( 108 ) uses speech recognition technology to process and capture information from the live conversation. More specifically, the DAS ( 108 ) uses the Speech Recognition Engine ( 114 ) to recognize the live conversation as text. Further, the DAS ( 108 ) is also configured to place the information captured from the live conversation into a particular context associated with a form or document actively in use by the operator. 
     The Speech Recognition Engine ( 114 ) may use a variety of well-known speech recognition technologies to recognize the live conversation between the customer ( 102 ) and the operator ( 106 ) as text. For example, in one embodiment of the invention, the Speech Recognition Engine ( 114 ) may use a restricted grammar to increase accuracy and efficiency. The restricted grammar may be stored in a grammar repository ( 116 ) operatively connected to the Speech Recognition Engine ( 114 ). Restricted grammar provides the Speech Recognition Engine ( 114 ) with a limited vocabulary that the Speech Recognition Engine ( 114 ) can identify from a live conversation. Further, the grammar may be related to a context or an attribute. For example, suppose an operator is processing customer calls at a plumber&#39;s office. In this case, the context could be a form for documenting plumbing problems, and a restricted grammar that is configured to expect certain plumbing terms such as “pipes”, “toilet”, or “leak” may be used by the Speech Recognition Engine ( 114 ). Alternatively, in one embodiment of the invention, the restricted grammar may be a database of “likely vocabulary” which will weight the probabilities of certain words and phrases as higher than other words and phrases. 
     Alternatively, in one embodiment of the invention, a restricted grammar for recognizing speech associated with common information fields may be used. For example, common fields of information may include a customer name, an address, an occupation, a phone number, etc. In this case, the restricted grammar may expect words or phrases associated with such fields of customer information. In one embodiment of the invention, the Speech Recognition Engine ( 114 ) is a learning engine that improves accuracy and performance using corrections made by the operator ( 106 ) such that the Speech Recognition Engine ( 114 ) may be better able to recognize types of customer information in the future. 
     Continuing with  FIG. 1 , the DAS ( 108 ) includes functionality to process the speech recognized by the Speech Recognition Engine ( 114 ) by designating a context for the conversation through the Context Designator ( 110 ). Further, the Context Designator ( 110 ) is responsible for recognizing cues and relating text from the live conversation with fields in the document/form based on the cues. In one embodiment of the invention, the Context Designator ( 110 ) may be operatively connected to a repository (not shown) configured to store documents/forms that are retrieved to record information from the live conversation. Further, in one or more embodiments of the invention, the system of  FIG. 1  may include a computer application ( 107 ) running on a computer system (not shown) configured to present one or more forms to the operator. Thus, the form/document retrieved and automatically completed using the DAS ( 108 ) is presented on-screen to the operator by the computer application ( 107 ). 
     In one embodiment of the invention, the Context Designator ( 110 ) is aware of the current form/document presented by the computer application ( 107 ). In one embodiment of the invention, the developer of the computer application ( 107 ) has embedded code in the computer application ( 107 ) to activate the Context Designator ( 110 ) for that particular form/document. Further, the Context Designator ( 110 ) is provided with all the “metadata” associated with the current form presented by the computer application ( 107 ). The metadata may be in the form of a list of possible cues, an identification (ID) for a data field within the form, and/or the type of data field. Given this information, the Context Designator ( 110 ) includes functionality to “listen for” cues as it obtains a stream of text from the Speech Recognition Engine ( 114 ). Upon “hearing” a cue, the Context Designator ( 110 ) knows the associated data field ID and type. Thus, the Context Designator ( 110 ) is able to inform the Speech Recognition Engine ( 114 ) to look for address data, a phone number, a name, etc. Said another way, in one embodiment of the invention, the Context Designator ( 110 ) provides cues to the speech recognition engine about the type of information to expect. In this manner, the Context Designator creates a “context” for the Speech Recognition Engine ( 114 ). 
     Those skilled in the art will appreciate that the context designated by the Context Designator ( 110 ) may be any number of methods for storing information (e.g., a computer application that interfaces with a repository designed to store customer information, a spreadsheet, etc.). 
     In one embodiment of the invention, a cue is a particular word or phrase that triggers the recording of information from the live conversation. That is, a cue indicates to the DAS ( 108 ) that a portion of the live conversation following the cue needs to be recorded in a form or document. More specifically, in one or more embodiments of the invention, a cue is a word or phrase that indicates that the customer portion of the live conversation following the cue needs to be recorded to complete one or more information fields in a document. For example, a cue may be the word “name,” the phrase “mailing address,” the phrase “For what type of service are you calling today?,” etc. In one or more embodiments of the invention, the cue may be a pre-defined verbal cue (such as the aforementioned words and/or phrases) or visual. A visual cue may include the operator physically moving a cursor to a particular information field, thereby triggering the recording of information into that particular information field. 
     In one embodiment of the invention, the DAS ( 108 ) may process the live conversation using the Document Completion Engine ( 112 ) and the Speech Recognition Engine ( 114 ). Specifically, the Document Completion Engine ( 112 ) stores information into the context designated by the Context Designator ( 110 ). Said another way, the Document Completion Engine ( 112 ) is responsible for actually filling out the document/form presented by the computer application ( 107 ). The Document Completion Engine ( 112 ) uses information from the Context Designator ( 110 ) (e.g., data field IDs) and the Speech Recognition Engine ( 114 ) (i.e., recognized text) to actually populate the document/form in the computer application ( 107 ). In one or more embodiments of the invention, the operator ( 106 ) may proof and correct information filled out by the Document Completion Engine ( 112 ) or enter certain portions of data directly, without having the Document completion Engine ( 112 ) complete those certain portions. 
     In one embodiment of the invention, the speech recognition engine converts the entire live conversation to text, while the document completion engine records only the relevant customer information into a context (i.e., a document or a form used to store customer information). Those skilled in the art will appreciate that the document completion engine may distinguish between the customer and the operator because the Speech Recognition Engine may recognize the difference between the operator&#39;s voice and the customer&#39;s voice. In this manner, the document completion engine may know when the customer is speaking and when the operator is speaking so that information spoken by the operator is not recorded into the form presented by the computer application ( 107 ). 
     In one embodiment of the invention, a library may exist which can be embedded in an existing or new computer application ( 107 ) described above, and the existing/new application “adds” the automatic speech recognition filling of forms to its own functionality by providing information to the library and invoking the library at the appropriate places in the application workflow. In one embodiment of the invention, the DAS ( 108 ) may be such a library, where the DAS ( 108 ) is configured to provide such a utility to an application, and the application is in control of when and how to invoke the automatic speech recognition functionality to complete one or more forms during a live conversation. 
       FIG. 2  shows a flow chart for assisted form completion in accordance with one or more embodiments of the invention. Initially, a customer initiates a live conversation with an operator (Step  200 ). The customer may then make subsequent selections associated with a service or purchase by speaking with the operator. Either verbal clues or operator actions establish a context for the live conversation (Step  202 ). In one embodiment of the invention, the context is designated by converting the live conversation to text, recognizing portions of the live conversation between the customer and the operator as a text portion and using the textual information to determine what type of context is required for that particular live conversation. In one embodiment of the invention, the designated context may be information about the form (metadata) that can help in recognition of information fields in which to store customer information. Those skilled in the art will appreciate that the entire conversation may be converted to text and either portions of the live conversation may be recognized as text portions or the entire converted live conversation may be recognized as a text portion. Alternatively, only portions of the live conversation may be converted to text. 
     Those skilled in the art will appreciate that in one or more embodiments of the invention, initial portions of the customer&#39;s conversation may take place with an automated phone system (i.e., a machine that walks the customer through certain steps in the process of collecting customer information or identifying the customer&#39;s need until an operator takes over). In this case, portions of the context designation and/or speech recognition may occur while the customer is interfacing with the automated phone system, before the live conversation with an operator begins. 
     Continuing with  FIG. 2 , upon designation of the context, the operator may begin interviewing the customer using cues associated with speech recognition and document completion (Step  204 ). That is, at relevant portions of the live conversation, the operator may recite cues signaling that document completion for purposes of recording information into a particular information field should begin. For example, in one embodiment of the invention, the cues may be particular words within verbal phrases such as “Could you please give me your full name?” or “What is your home address?” where the cue is the word “name” and “address.” Alternatively, in one or more embodiments of the invention, cues may be more elaborate by including entire phrases. In one embodiment of the invention, the cues recited by the operator may correspond to information that is included in the context that includes the information field to be completed with customer information. For example, if the information field on a document or form that is used to record customer information is preceded by the phrase “home address,” then that phrase may constitute the cue used by the operator and recognized by the DAS. 
     Further, in one embodiment of the invention, cues may be associated with voice tone or pitch, where the operator is aware that particular words or phrases are spoken in a particular tone, which initiates the speech recognition process. Those skilled in the art will appreciate that the operator could also cue provides visual cues using an input/output interface device such as a mouse or a keyboard. For example, the operator may use the mouse and/or keyboard to place a cursor in a particular field in a document/form. Placing the cursor in a particular place on the document/form may trigger the speech recognition process so that the portion of the live conversation after movement of the cursor into a particular location is recorded into the document/form. Subsequently, upon using cues to signal which information field is active, the document/form is automatically filled in with information obtained from a portion of the live conversation recognized as text (Step  206 ). 
     Those skilled in the art will appreciate that the portion of the live conversation that is used to record information in to a particular context may be a customer portion of the live conversation or an operator portion of the live conversation. For example, in one embodiment of the invention, after the cue is invoked to trigger the speech recognition process, the customer&#39;s works/phrases may be unclear, in which the operator may clarify the portion of the live conversation after the cue, and the operator&#39;s words/phrases may be recorded into a particular context. 
     Simultaneously or subsequent to the completion of the recording of information into the document/form, the document/form may be reviewed to check whether the speech recognition was performed accurately and to confirm the validity of the information (Step  208 ). If the information recorded from the live conversation is correct, the recorded information is saved (Step  214 ). However, if a recording error is found, the error may be corrected manually before saving the information (Step  212 ). 
     Those skilled in the art will appreciate that corrections of errors in recorded information may occur while the live conversation continues and while other information is being recorded into other portions of the document/form. At this stage, a determination is made as to whether all necessary information (e.g., customer information) has been obtained and recorded (Step  216 ). If more information is necessary, the live conversation continues as described above, and Steps  204 - 216  are repeated. If all necessary customer information has been recorded, then the process ends. 
       FIG. 3  shows a flow chart for assisted form completion in accordance with one or more embodiments of the invention. Particularly,  FIG. 3  shows a flow chart for the process of assisted form completion from the perspective of the DAS. Initially, a context associated with the live conversation is designated (Step  300 ). In one embodiment of the invention, the context may be designated by an application running on a computer system. Subsequently, the live conversation between the operator and the customer is processed by monitoring the live conversation and recognizing the speech as a portion of text (Step  302 ). That is, the live conversation is converted to text, after which portions of the live conversation are recognized as text portions. 
     At this stage, the live conversation recognized as a portion of text may be processed to identify any cues in the live conversation (Step  304 ). Next, each recognized cue is related to an information field in the context (i.e., the form or document used to record customer information) (Step  306 ). That is, for example, the cue “name” would be related to the name field within the context. Next, information from the text portion spoken after the recognized cue is obtained (Step  308 ). More specifically, in one or more embodiments of the invention, the information from the portion of text following each recognized cue may be recorded as information relating to the information field (Step  310 ). That is, the information recorded into an information field may be obtained from the recognized portion of text after identifying cues after which relevant information to be recorded in one or more information fields is likely to be spoken. Those skilled the art will appreciate that if the live conversation continues, then Steps  304 - 312  are repeated until all relevant customer information is obtained and recorded into the document/form or other context being used to capture customer information electronically. 
     In one embodiment of the invention, the end of the live conversation may be recognized using a cue, so that the live conversation can stop being converted to text. For example, the end of the live conversation (or at least the end of the portion of the live conversation from which customer information is extracted) may be recognized when the operator resumes speaking or when the operator says “Thank you” or other cue phrases. 
     Upon recording relevant information, additional processing on the information may be performed to remove any surplus information. In one embodiment of the invention, the Document Completion Engine may have access to a list of common phrases that should be removed before completing (i.e., filling in) the information field. For example, when interpreting customer information associated with a name field, the Document Completion Engine may remove the phrase “my name is” before entering the customer information into the name field. 
     The following is an example of the process described in  FIGS. 2 and 3  above. Consider the scenario in which a customer calls a plumber&#39;s office to make an appointment for plumbing work that needs to be performed on the customer&#39;s home. When the customer calls the plumber&#39;s office, an operator answers the phone and initiates a live, real-time conversation with the customer. As soon as the operator learns the nature of the customer&#39;s call, the operator can designate a context for the conversation, in this case the context being a work request form. The DAS can then display a work request form and begin monitoring (i.e., listening to) the live conversation between the customer and the operator. The operator may then ask the customer “What is your full name?” where the phrase “full name” is a designated cue for completion of the name field in the work request form. The customer may then reply, “My name is Jon Smith,” which would be recognized as text and sent to the Document Completion Engine as relating to the name field. 
     In this case, the text “My name is Jon Smith” may be processed by first consulting a repository for surplus phrases and then removing the phrase “My name is” from the text. The name field may subsequently be completed with the remaining text “Jon Smith.” Alternatively, the operator may remove surplus portions of recorded text upon reviewing the work request form. In one embodiment of the invention, the operator may review the information in the name field and ask the customer how he spells his first name. Notice that this part of the conversation did not have a cue from the operator and is not processed by the DAS. Learning that the customer&#39;s first name has been misspelled and should be “John,” the operator may correct the information in the name field while simultaneously continuing the interview to ask for the customer&#39;s home address. The DAS recognizes the cue “address” and begins monitoring the live conversation for information related to the home address field. This information is processed in a similar manner as described above with respect to the customer&#39;s name. For the address information, a restricted grammar specialized for addresses is used to recognize the customer&#39;s answer with greater accuracy and efficiency. The operator can continue to interview the customer in a similar fashion as described above until all the information fields of the document have been completed and all relevant customer information has been recorded to satisfy the purpose of the customer&#39;s call. 
     The invention may be implemented on virtually any type of computer regardless of the platform being used. For example, as shown in  FIG. 4 , a networked computer system ( 400 ) includes a processor ( 402 ), associated memory ( 404 ), a storage device ( 406 ), and numerous other elements and functionalities typical of today&#39;s computers (not shown). The networked computer ( 400 ) may also include input means, such as a keyboard ( 408 ) and a mouse ( 410 ), and output means, such as a monitor ( 412 ). The networked computer system ( 400 ) is connected to a local area network (LAN) or a wide area network. (e.g., the Internet) (not shown) via a network interface connection (not shown). Those skilled in the art will appreciate that these input and output means may take other forms, now known or later developed. Further, those skilled in the art will appreciate that one or more elements of the aforementioned computer ( 400 ) may be located at a remote location and connected to the other elements over a network. 
     Further, the invention may be implemented on a distributed system having a plurality of nodes, where each portion of the invention (e.g., Speech Recognition Engine, Document Completion Engine, Context Designator, etc.) may be located on a different node within the distributed system. In one embodiment of the invention, the node corresponds to a computer system. Alternatively, the node may correspond to a processor with associated physical memory. 
     While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of this disclosure, will appreciate that other embodiments can be devised which do not depart from the scope of the invention as disclosed herein. Accordingly, the scope of the invention should be limited only by the attached claims.