Techniques for automatically analyzing a transcript and providing interactive feedback pertaining to interactions between a user and other parties

Techniques are provided to allow a user to interact with a computer to automatically analyze a transcript and provide interactive feedback pertaining to interactions between the user and other parties. This may be accomplished by dividing the transcript into text sequences, such as sentences, and matching each text sequence against a set of rules that define patterns that relate text sequences to particular characteristic categories. These matches can be further scored and ranked to allow particular text sequences to be interactively displayed to the user in response to selection of a particular categorization.

BACKGROUND

Individuals often have a need to interact. When not physically co-located, telephones as well as conferencing software running on networked computers may allow them to conduct a conversation or presentation remotely. An individual may record these conversations or presentations using the conferencing software. An individual may also record a conversation or presentation using a computing device (e.g., a smart phone) even when it is not being conducted over a network. If the individual wishes to later have easy access to the contents of the conversation or presentation, the recording may be transcribed, either by a person or by computer transcription software. Users may then search the transcription as needed.

SUMMARY

Although recording and transcribing conversations and presentations allows for easy searching if the user knows what words to search for, it can be difficult for users to evaluate the conversation or presentation for effectiveness and other qualities.

Thus, improved techniques allow a user to interact with a computer to automatically analyze a transcript and provide interactive feedback pertaining to interactions between the user and other parties. This may be accomplished by dividing the transcript into text sequences, such as sentences, and comparing each text sequence against a set of rules that define patterns that relate text sequences to particular characteristic categories. These matches can further be scored and ranked to allow particular text sequences to be interactively displayed to the user in response to selection of a particular categorization.

One embodiment is directed to a method of interactively communicating with a user to generate automated feedback pertaining to the user's interactions with other parties, performed by a computing device. The method includes (1) obtaining a plurality of text sequences provided in a text transcription, the text transcription based on a recording of an interaction, (2) matching each of the plurality of text sequences against a set of predetermined rules arranged across multiple predetermined categories to yield a set of matches, each rule specifying a respective set of text relationships that must be present in a text sequence to yield a match, each category pertaining to a particular characteristic of the interaction for which the rules in that category test to determine whether the sequence indicates the characteristic to which the category pertains, each match identifying (a) a particular text sequence of the plurality of text sequences, (b) a particular category of the multiple predetermined categories, and (c) a particular rule which matched the particular text sequence, (3) identifying confidence scores for the set of matches from a database of confidence scores, each confidence score reflecting a confidence that the particular text sequence identified by its respective match actually possesses the characteristic to which the particular category identified by that match pertains, (4) for each unique sequence/category pairing, selecting a match that identifies that unique sequence/category pairing having a highest identified confidence score and assigning that highest identified confidence score to that unique sequence/category pairing, (5) selecting a plurality of the unique sequence/category pairings for display based on their respective assigned confidence scores, and (6) causing text of the sequences of each of the selected unique sequence/category pairings to be interactively displayed within a graphical user interface (GUI) on a display device, the GUI allowing the user to select a particular category of the multiple categories for display, wherein once the user has selected a selected category for display, causing text of the sequences of each of the selected unique sequence/category pairings to be interactively displayed within the GUI on the display device includes causing sequences indicated to possess the characteristic to which the selected category pertains to be displayed within the GUI on the display device. Other embodiments are directed to corresponding apparatuses, computer program products, and systems for performing similar methods.

DETAILED DESCRIPTION

Embodiments of the invention will now be described. It is understood that such embodiments are provided by way of example to illustrate various features and principles of the invention, and that the invention hereof is broader than the specific example embodiments disclosed.

Improved techniques are provided to allow a user to interact with a computer to automatically analyze a transcript and provide interactive feedback pertaining to interactions between the user and other parties. This may be accomplished by dividing the transcript into text sequences, such as sentences, and matching each text sequence against a set of rules that define patterns that relate text sequences to particular characteristic categories. These matches can be further scored and ranked to allow particular text sequences to be interactively displayed to the user in response to selection of a particular categorization.

FIG. 1shows an example environment30in which embodiments of the improved techniques hereof can be practiced. Here, a human user32interacts with one or more other parties36using a user device34that makes a recording44of the interaction42between the user32and the one or more other parties36. The interaction42may be a bidirectional interaction such as a conference or a unidirectional (or mostly unidirectional) interaction such as a presentation. Interaction42includes an audio recording of speech by the parties32,36. In some embodiments, the interaction42may be conducted across a network38with a person36using a remote device40. In some embodiments, the interaction42is local between the users32,36both within recording range of the user device34(e.g., a smart phone). In some embodiments, several parties32,36may be located adjacent to the user device34while a teleconference is conducted with yet another user36at remote device40.

Transcription server46receives a copy of the recording44over network38and creates a transcription48of the interaction42. In some embodiments, transcription server46may also divide the transcription into a set of non-overlapping text sequences50(e.g., sentences). For example, transcription server46may look for pauses exceeding a threshold length of time (e.g., 0.5 seconds or 1 second) within the recording44, breaking up the text sequences50between those pauses. In another example, text transcription server may grammatically analyze the transcription48using a parser to identify sentences or other grammatical structures (e.g., phrases or clauses) and to set each such identified sentence or other grammatical structure as a separate text sequence50. In other embodiments, transcription server46does not divide the transcription48into text sequences50, that job being instead done by another entity (e.g., computing device52).

The network38may be any type of network or combination of networks, such as a local area network (LAN), a wide area network (WAN), the Internet, and/or some other type of network or combination of networks, for example. The user device34, remote devices40, and transcription server46may connect to the network38using various technologies, such as Ethernet, Token Ring, or Wi-Fi, for example.

Computing device52connects to network38as well. In some embodiments, computing device52may also function as user device32, a remote device40, or transcription server46. User32or another person may communicate with computing device52in order to request analysis of transcript48and interactive feedback pertaining to the interaction42between the user32and one or more of the other parties36. In one embodiment, user32(or other person, hereinafter omitted) communicates with computing device52remotely over network38by interacting with a web page provided by computing device52via a web browser operating on the user device34and displayed on a display device60connected to the user device34. In another embodiment, user32communicates directly with computing device by operating user interface (UI) devices (including display device60as well as user input devices, not shown) connected to UI circuitry58of the computing device52. Display device60may be any kind of device capable of displaying images to user32. Display device60may be, for example, a CRT, LCD, plasma, or LED monitor or embedded display screen. User input devices allow user32to directly interact with the computing device52or the user device34, depending on the embodiment. User input devices may include, for example, a keyboard, keypad, mouse, trackpad, trackball, touch-sensitive screen, etc.

Computing device52may be any kind of computing device, such as, for example, a personal computer, a workstation, a server, an enterprise server, a laptop, a tablet, a smartphone, etc. Typically, computing device52is a server. Computing device52includes processing circuitry56, network interface circuitry54, memory64, and interconnection circuitry (not depicted). Computing device52may also include UI circuitry58to connect to display device60and user input circuitry in embodiments in which user32communicates directly with computing device52.

Processing circuitry56may be any kind of processor or set of processors configured to perform operations, such as, for example, a microprocessor, a multi-core microprocessor, a digital signal processor, a system on a chip, a collection of electronic circuits, a similar kind of controller, or any combination of the above.

Network interface circuitry54may include one or more Ethernet cards, cellular modems, cable modems, DSL modems, Wireless Fidelity (Wi-Fi) wireless networking adapters, and other devices for connecting to a network (e.g., network38), such as a LAN, WAN, cable system, DSL system, cellular data network, etc.

UI circuitry58may connect to one or more UI devices (e.g., user input devices and display device60), which allow a user (e.g., user32) to directly interact with the computing device52. UI circuitry58may include, for example, a graphics adapter for connecting to a display device60and one or more communications buses. These communications buses may connect to, for example, a keyboard, mouse, trackpad, etc.

The memory64may include both volatile memory (e.g., random access memory, RAM), and non-volatile memory, such as one or more read-only memories (ROMs), disk drives, solid-state drives, and the like. At a minimum, memory64includes system memory, typically RAM. The processing circuitry56and the memory64together form control circuitry, which is constructed and arranged to carry out various methods and functions as described herein, e.g., alone or in coordination with similar control circuitry on another data storage system. Also, the memory64includes a variety of software constructs realized in the form of executable instructions. When the executable instructions are run by the processing circuitry56, the processing circuitry56is caused to carry out the operations of the software constructs. Although certain software constructs are specifically shown and described, it should be understood that the memory64typically includes many other software constructs, which are not shown, such as an operating system and various applications, processes, and daemons. Applications configured to run on processing circuitry56when stored in non-transitory form, either in the volatile portion or the non-volatile portion of memory64or both, form a computer program product. The processing circuitry56running one or more of these applications thus forms a specialized circuit constructed and arranged to carry out the various processes described herein.

Memory64stores an interactive analytics application66as well as interactive analytics GUI code70. The interactive analytics GUI code70is configured to cause an interactive analytics GUI62to be displayed on display device60. In some embodiments, interactive analytics GUI code70is sent by a web server application68also running on the computing device52to the user device34over network38so that the interactive analytics GUI code70can be executed by a web browser (not depicted) running on the user device34to render interactive analytics GUI62on display device60. In other embodiments, interactive analytics GUI code70runs locally on the computing device52to cause the interactive analytics GUI62to be rendered on display device60via UI circuitry58.

Memory64also stores a copy of transcription58, including text sequences50(depicted as text sequences50-1,50-2,50-3, . . . ,50-n), a set72of rules74, confidence score database (DB)76, match list82, a category/sequence mapping90, a set of selections92of text sequences50(depicted as selections92(a),92(b), . . . ), a threshold minimum score94, and a threshold maximum number96of selected sequences per category.

Set72of rules74define patterns that map text sequences to particular predetermined categories, each category pertaining to a particular characteristic that may be found in an interaction42. One example category relates to a characteristic of discussing Next Steps. Another example category relates to a characteristic of Humility. The set72of rules74may be organized by category, Thus, for example, rules74(a)(1) and74(a)(2) each specify a text relationship that must be present in a text sequence50to indicate the characteristic of category a (e.g., Next Steps), and rules74(b)(1) and74(b)(2) each specify a text relationship that must be present in a text sequence50to indicate the characteristic of category b (e.g., Humility).

Confidence score DB76maps rule identifiers (IDs)78to confidence scores80that assess how likely a particular text sequence50-xthat has been mapped to a particular category y by a particular rule74(y)(z) actually possesses the characteristic to which the particular category y identified by that rule74(y)(z) pertains. Thus, for example, as depicted, rule74(a)(1) has a confidence score80of 0.8, meaning that 80% of the time that rule74(a)(1) matches against a text sequence50, that text sequence50is expected to actually possess the characteristic of Next Steps. Similarly, for example, as depicted, rule74(a)(2) has a confidence score80of 0.5, meaning that 50% of the time that rule74(a)(2) matches against a text sequence50, that text sequence50is expected to actually possess the characteristic of Next Steps. As another example, as depicted, rule74(b)(1) has a confidence score80of 0.1, meaning that 10% of the time that rule74(b)(1) matches against a text sequence50, that text sequence50is expected to actually possess the characteristic of Humility. As another example, as depicted, rule74(b)(2) has a confidence score80of 0.6, meaning that 60% of the time that rule74(b)(2) matches against a text sequence50, that text sequence50is expected to actually possess the characteristic of Humility.

Match list82includes a set of matches84(depicted as matches84-A,84-B, . . . ). Each match84represents a particular rule74matching against a particular text sequence50, thereby demonstrating that that text sequence50indicates the characteristic of a particular category. Each match84-X includes a respective sequence identifier (ID)85-X, category ID86-X, and rule ID87-X. Thus, for example, as depicted, match84-A includes a sequence ID85-A that identifies text sequence50-1, a category ID86-A that identifies category a (e.g., Next Steps), and a rule ID87-A that identifies rule74(a)(1). Each match84-X may also include a score88-X drawn from the confidence score DB76, although, in some embodiments, this may be omitted as the score88-X may be obtained with simple reference to the confidence score DB76. Thus, for example, match84-A has a score88-A of 0.8 since that is the confidence score80associated with rule74(a)(1). As another example, assume that within match84-B, sequence ID85-B also identifies text sequence50-1, category ID86-B also identifies category a (e.g., Next Steps), and rule ID87-B identifies rule74(a)(2).

Category/sequence mapping90represents a set of best matches84(if any) for each possible pairing of categories and text sequences50. Thus, although matches84-A and84-B both have respective category IDs86identifying category a, since the score88-A for match84-A is 0.8 and the score88-B for match84-B is only 0.5, only match84-A is mapped to the pairing of text sequence50-1and category a. In addition, as depicted, there is no match84that is mapped to the pairing of text sequence50-2and category a because there were no matches for any of the rules74(a) of category a against text sequence50-2. As depicted, match84-C is the best match for the pairing of text sequence50-3and category a and match84-H is the best match for the pairing of text sequence50-nand category a. As further depicted, match84-D is the best match for the pairing of text sequence50-1and category b and match84-F is the best match for the pairing of text sequence50-2and category b. The pairings of category b with both text sequences50-3and50-nhave no matches.

Each per category selection92of text sequences50includes a selection of text sequences50identified by a subset of the matches84for a particular category (see category/sequence mapping90). As depicted, selection92(a) for category a includes text sequences50-1and50-n(corresponding to matches84-A and84-H, but excluding match84-C in this example). As depicted, selection92(b) for category b includes text sequence50-2(corresponding to match84-F, but excluding match84-D in this example).

There are various ways in which this selection may be done. In one embodiment, the respective text sequences50for all matches84in the category/sequence mapping90under a particular category are selected as long as the respective scores88for those matches84exceed the threshold minimum score94stored in memory64. In another embodiment, the respective text sequences50for all matches84in the category/sequence mapping90under a particular category are selected as long as the number of such matches84does not exceed the threshold maximum number96of selected sequences per category, also stored in memory64. If, however, the number does exceed the threshold maximum96, then the text sequences50from only the matches84having the highest respective scores88are included (there being the threshold maximum number96of selected text sequences50).

In some embodiments, the text sequences50within each selection92may be ranked in order from highest to lowest corresponding scores88.

Interactive analytics GUI62is configured to interactively display on the display device60the text sequences50of one or more of the selections92.

FIG. 2Adepicts an example state162of interactive analytics GUI62. As depicted, the interactive analytics GUI62displays several tabs102that allow the user32to choose between several different categories. For example, tab102(a) allows the user32to select category a, tab102(b) allows the user32to select category b, and tab102(c) allows the user32to select category c. As depicted inFIG. 2A, the user32has selected tab102(a) as the active tab104. Thus, the interactive analytics GUI62displays results for category a, displaying text sequences50identified by selection92(a) for that category (e.g., Next Steps, as depicted). As depicted, two sequences50-x,50-yare depicted in connection with category a.

FIG. 2Bdepicts an alternate example state162′ of interactive analytics GUI62in which the user32has changed the active tab104to instead be tab102(b) to indicate category b. Thus, the interactive analytics GUI62now displays results for category b, displaying text sequences50identified by selection92(b) for that category (e.g., Humility, as depicted). As depicted, only one sequences50-zis depicted in connection with category b.

FIG. 3depicts an example method200for a computing device52to allow a user32to interact with the computing device52to automatically analyze a transcription48and provide interactive feedback pertaining to interactions42between the user32and other parties36.

Method200is performed by interactive analytics application (IAA)66operating on a computing device52. It should be understood that any time a piece of software (e.g., IAA66, interactive analytics GUI code70, web server68, etc.) is described as performing a method, process, step, or function, in actuality what is meant is that a computing device (e.g., computing device52, user device34, remote device40, transcription server46, etc.) on which that piece of software is running performs the method, process, step, or function when executing that piece of software on its processing circuitry56. It should be understood that, in some embodiments, one or more of the steps or sub-steps may be omitted. Similarly, in some embodiments, one or more steps or sub-steps may be combined together or performed in a different order.

In optional step210, IAA66receives a request from a user32to interactively view categorized text sequences50for an interaction42between the user32and one or more other parties36. In other embodiments, step210may instead occur just prior to step280, while in other embodiments, step210is omitted entirely, the rest of method200being performed automatically in response to transcription server46sending the transcription48to the computing device52(which may be done, for example, immediately upon the transcription server receiving the recording44from the user device34and completing transcribing the recording44).

In step220, IAA66obtains a plurality of text sequences50(e.g., sentences, defined grammatically or by pauses) provided in a text transcription48based on a recording44of the interaction42. In some embodiments, IAA66requests that the transcription server46send it the transcription48and/or the text sequences50in response to receiving the request of step210, while, in other embodiments, transcription server46itself initiates the sending of the transcription48and/or the text sequences50upon becoming available. In some embodiments, transcription server46only sends the transcription48to the computing device52without dividing it into text sequences—in these embodiments, the computing device52itself divides the transcription48into text sequences50. In other embodiments, the transcription server46sends the individual text sequences50to the computing device52, either by sending the entire transcription48together with a list of divisions between the text sequences50or by sending the text sequences50individually.

In step230, IAA66matches each of the plurality of text sequences50against a set72of predetermined rules74arranged across multiple predetermined categories (e.g., categories a, b, . . . ), each rule74specifying a respective set of text relationships that must be present in a text sequence50to satisfy that rule74, each category pertaining to a particular characteristic (e.g., Next Steps, Humility, etc.) of the interaction42for which the rules74in that category test to determine whether the sequence50indicates the characteristic to which the category pertains, yielding a set82of matches84, each match84identifying: (a) a particular text sequence50, (b) a particular category, and (c) a particular rule74which matched the particular text sequence50, thereby demonstrating that the particular text sequence50indicates the characteristic to which the particular category pertains. The rules74may take the form of a set of at least two words and a proximity, so, in order to match a rule, all of the specified words must be present in a text sequence within the given proximity of each other. For example, rule74(a)(1) may state that the words “suggest” and “should” must appear within five words of each other in order to indicate category a relating to the characteristic of Next Steps. As another example, rule74(a)(2) may state that the words “install*” (the asterisk indicating any extension of the word “install” such as “install,” “installs,” “installed,” and “installing”) and “product” must appear within seven words of each other in order to indicate category a relating to the characteristic of Next Steps. As another example, rule74(b)(1) may state that the words “I” and “think” must appear within two words of each other in order to indicate category b relating to the characteristic of Humility. As another example, rule74(b)(2) may state that the word “I” and the phrase “don't know” must appear within three words of each other (or alternatively that the three words “I” and “don't” and “know” must appear within four words from start to finish) in order to indicate category b relating to the characteristic of Humility. In some embodiments, a rule74may also include one or more words that may not appear in proximity to the other words—if such a negative word does appear in such proximity, then that rule74will not match. For example, rule74(b)(1) may state that the word “not” may not appear within two words of the positive words “I” and “think.”

In step240, IAA66identifies confidence scores88for the set82of matches84from a DB76of confidence scores80, each confidence score88reflecting a confidence that the particular text sequence50identified by its respective match84actually possesses the characteristic to which the particular category identified by that match84pertains. In some embodiments, IAA actually stores the identified scores88as part of the matches84, while, in other embodiments, the scores80are just identified from the confidence score DB76but not stored within matches84.

In step250, for each unique sequence/category pairing, IAA66selects a match84(if any exists) that identifies that unique sequence/category pairing having a highest identified confidence score88and assigns that highest identified confidence score88to that unique sequence/category pairing. In one embodiment, IAA66creates mapping90as part of step250.

In step260, IAA66selects a plurality of the unique sequence/category pairings for display based on their respective assigned confidence scores88. In one embodiment, IAA creates selections92as part of step260.

In some embodiments, IAA66performs step260by performing sub-step262. In sub-step262, IAA66selects pairings having confidence scores88above a minimum threshold (e.g., minimum threshold score94). In one example embodiment, the minimum threshold score94is 0.2.

In other embodiments, IAA66performs step260by performing sub-step264. In sub-step264, IAA66selects up to a maximum number (e.g., threshold maximum number96) per category of pairings having the highest respective confidence scores88. In one example embodiment, the threshold maximum number96is ten.

In some embodiments, either after step260or in conjunction with step260, IAA66performs optional step270. In step270, IAA66ranks and orders the selected pairings within each category from highest to lowest respective scores88.

In some embodiments, steps220-260or220-270are performed shortly after user32makes the recording44of interaction42and transcription server46transcribes it to create transcription48, but step280is only performed after receiving a request from user32such as in step210.

In step280, IAA66causes text of the sequences50of each of the selected unique sequence/category pairings (e.g., selected within selections92) to be interactively displayed within interactive analytics GUI62on display device60, the interactive analytics GUI62allowing the user32to select a particular category of the multiple categories for display. Once the user32has selected, IAA66cause sequences50(or portions thereof) indicated to possess the characteristic to which the selected category pertains to be displayed within the interactive analytics GUI62. This may include separately displaying text of the respective selected sequences50for each category depending on which tab102is selected (seeFIGS. 2A and 2B).

In sub-step282, IAA66displays the interactive analytics GUI62directly on local display device60via UI circuitry58. In these embodiments, IAA66may directly execute interactive analytics GUI code70on processing circuitry56.

In sub-step284, IAA66sends the interactive analytics GUI code70across network38(e.g., via network interface circuitry54) for rendering by the user device34and display on remote display device60. In one such embodiment, IAA66generates the interactive analytics GUI code70(e.g., markup code such as XML, HTML, dynamic HTML, etc.) and directs web server68to send it to a web browser running on the user device34for execution.

In some embodiments, as part of step280, IAA66performs sub-step286. In sub-step286, IAA66causes the text of the selected sequences for a particular category to be displayed in the ranked order generated in step270.

In some embodiments, as part of step280, IAA66performs sub-step288. In sub-step288, instead of displaying the entirety of each text sequence50that was selected, IAA66first performs a snippeting operation on each of the selected text sequences50and then displays a snippet of each selected text sequence50. A snippet is a subset of a text sequence, which, depending on the embodiment and the particular text sequence50, may include the entirety of the text sequence or it may include less than the entirety of the text sequence.

FIG. 4depicts an example snippeting operation300in detail. Method300is performed by IAA66operating on a computing device52. It should be understood that, in some embodiments, one or more of the steps or sub-steps may be omitted. Similarly, in some embodiments, one or more steps or sub-steps may be combined together or performed in a different order.

In step310, IAA66determines whether or not the text sequence50has more than a threshold maximum number of words (e.g., more than 25 words). If it does not have more words, then it is not too long to allow the user32to easily perceive the gist of what was said. Thus, if it does not have more words, operation proceeds with step350, in which IAA66sets the snippet to be the entire text sequence50.

However, if the text sequence50does have more than the threshold maximum number of words, then operation proceeds with step320. The remaining steps of method300may best be understood with reference toFIG. 5, which depicts a text sequence50and an example sub-division of the text sequence50.

In step320, IAA66sets a base relevant phrase (BRP)202of the text sequence50to be a range of words within the sequence50from a first matching keyword of the rule74that generated it to a last matching keyword of the rule74that generated it. Thus, consider, for example, the following sentence, which has more than 25 words:

“I think that we may have had a productive meeting, and I therefore suggest that you should try installing this product to see if it serves your needs.”

Applying example rule74(a)(1) described above in connection with step230, the BRP202of this example sentence is “suggest that you should.”

In step330, IAA66determines whether or not the BRP202has at least a threshold minimum number of words (e.g., at least 15 words). If it does, then the BRP202is likely long enough to suggest the gist of the text sequence50to the user, so operation proceeds with step360in which IAA66sets the snippet to be the BRP202.

However, if the BRP is not at least as long as the threshold minimum length, then operation proceeds with step340. In step340, IAA66extends the snippet to include a prefix204and/or suffix206in addition to the BRP202. In sub-step342, IAA66extends the snippet to also include words of the sequence50before the first matching keyword up to (but not including) a previous conjunctive word or the beginning of the text sequence50. In some embodiments, various types of conjunctions may be included or excluded. In one embodiment, only coordinating conjunctions (e.g., “for,” “and,” “nor,” “but,” “or,” “yet,” “so,” etc.) are considered to the exclusion of subordinating conjunctions. Thus, since the word “and” is a conjunction, the prefix204of the example sentence is “I therefore.”

In sub-step344, IAA66extends the snippet to also include words of the sequence50after the last matching keyword up to (but not including) a next conjunctive word (again, including or excluding various types, based on the embodiment) or the end of the text sequence50. Thus, since there are no conjunctions in the example sentence after the BRP202, the suffix206of the example sentence is “try installing the product to see if it serves your needs.” Thus, the extended snippet generated by step340for the example sentence is “I therefore suggest that you should try installing this product to see if it serves your needs.”

As another example, applying rule74(a)(2) to the example sentence, the BRP202would be “installing this product,” while the prefix204is “I therefore suggest that you should try” and the suffix206is “to see if it serves your needs.”

As another example, applying rule74(b)(1) to the example sentence, the BRP202would be “I think,” while the prefix204is null and the suffix206is “that we may have had a productive meeting.”

It should be understood that there may be extra text210before and or after the snippet generated in step340.

After steps340,350, or360, operation may proceed to optional step370, which may be performed in some embodiments. In step370, IAA66removes repeated words from the snippet. Thus, for example, if the snippet is originally “I really really think that that that you may agree,” after step370it would become “I really think that you may agree.”

FIG. 6depicts an example method500that may be used for creating or auditing the confidence score DB76. Method500may be performed by IAA66or another application operating on a computing device52. It should be understood that, in some embodiments, one or more of the steps or sub-steps may be omitted. Similarly, in some embodiments, one or more steps or sub-steps may be combined together or performed in a different order.

It should be understood that method500may be performed in order to initially generate the confidence score DB76or it may be performed to audit and update the confidence score DB76during operation of IAA66. In the former case, method500operates on an initial set of text sequences50from any source (e.g., one or more previous interactions42). In the latter case, method500performs in the background during operation of method200and operates on a set of audited matches drawn from matches84from operation of method200.

Step510is performed if the rules74have not yet been applied against the set of text sequences50. Typically, when done in the audit context, the rules74have already been applied to the set of text sequences to be audited, so step510is omitted. However, in the context of creating the confidence score DB76from scratch, step510is typically performed.

In step510, computing device52applies all of the rules74of the set72against the set of text sequences (either the set of text sequences to be audited or a test set of text sequences) to generate a set of test matches or audited matches.

In step520, computing device52displays each test match or audited match to a human tester or auditor on a display device60(or this may be done across network38, such as over a web interface). This may include displaying text (possibly a snippet) of the text sequence50of test/audited match in conjunction with an indication of the characteristic to which the category associated with that match pertains.

In step530, computing device52receives input from the human tester or auditor (either from UI circuitry58or network interface circuitry54) with respect to each test match or audited match. The input indicates whether the text sequence50of each test/audited match accurately indicates the characteristic to which the category associated with that match pertains. Thus, for example, if the characteristic of the category is Next

Steps and the text sequence50is “The install of the product was defective,” then the human tester or auditor would likely indicate that the match did not accurately indicate its characteristic category.

Steps520and530may be performed either repeatedly or in parallel. Thus, in one embodiment, the human tester or auditor is presented with a long list of test/audited matches to evaluate, which he then evaluates and submits his input for all of them at once. In another embodiment, only one match is presented to the human tester or auditor at a time, the feedback is input, and then another match is displayed.

In step540, for each rule74(i)(j), the computing device52calculates a ratio of how many test or audited matches that matched that rule74(i)(j) were indicated to be accurate against a total number of test or audited matches that matched that rule74(i)(j). For example, if rule74(a)(1) matched against twenty text sequences50and the human tester or auditor indicated that sixteen of those were accurate, then the calculated ratio would be 0.8. In some embodiments, the results from auditing may be combined with the results from the original testing to create an updated ratio. In such a case, the number of accurate audited matches may be added to a previous number of accurate test matches and then that sum may be divided by a sum of the total number of test matches plus audited matches.

In step550, for each rule74(i)(j), the computing device52stores the calculated ratio for that rule74(i)(j) as the (updated) confidence score80for that rule74(i)(j) in the confidence score DB76.

In some embodiments, steps560-575may also be performed. In step560, if a rule74(i)(j) has a ratio of zero, then computing device52determines whether or not the total number of test/audited matches for that rule74(i)(j) (i.e., the denominator of the unsimplified ratio) exceeds a predetermined minimum threshold (e.g., 10 matches). If not, then the sample size is too small for an accurate accounting, so operation proceeds with step565, in which the computing device52replaces the confidence score80for that rule74(i)(j) in the confidence score DB76with a predetermined non-zero minimum confidence score. This predetermined non-zero minimum confidence score may be, for example, in the range of 0.05 to 0.15 (e.g., 0.1). In either case, operation proceeds with step570.

In step570, if a rule74(i)(j) has a ratio of one, then computing device52determines whether or not the total number of test/audited matches for that rule74(i)(j) (i.e., the denominator of the unsimplified ratio) exceeds the predetermined minimum threshold (e.g., 10 matches). If not, then the sample size is too small for an accurate accounting, so operation proceeds with step575, in which the computing device52replaces the confidence score80for that rule74(i)(j) in the confidence score DB76with a predetermined maximum confidence score that is less than one. This predetermined maximum confidence score that is less than one may be, for example, in the range of 0.85 to 0.95 (e.g., 0.9).

Thus, improved techniques have been provided to allow a user32to interact with a computer52to automatically analyze a transcript48and provide interactive feedback pertaining to interactions42between the user32and other parties36. This may be accomplished by dividing the transcript48into text sequences50, such as sentences, and matching each text sequence50against a set72of rules74that define patterns that relate text sequences50to particular characteristic categorizations. These matches84can be further scored and ranked to allow particular text sequences50to be interactively displayed to the user32in response to selection of a particular categorization.

For example, it should be understood that although various embodiments have been described as being methods, software embodying these methods is also included. Thus, one embodiment includes a tangible computer-readable medium (such as, for example, a hard disk, a floppy disk, an optical disk, computer memory, flash memory, etc.) programmed with instructions, which, when performed by a computer or a set of computers, cause one or more of the methods described in various embodiments to be performed. Another embodiment includes a computer which is programmed to perform one or more of the methods described in various embodiments.

Finally, it should be understood that all embodiments which have been described may be combined in all possible combinations with each other, except to the extent that such combinations have been explicitly excluded.