USING DIALOG-ACTIVITY SIMILARITY FOR

We want to enable users to locate desired information in spoken audio documents using not only the words, but also dialog activities. Following previous research, we infer this information from prosodic features, however, instead of retrieval by matching to a predefined finite set of activities, we estimate similarity using a vector space representation. Utterances close in this vector space are frequently similar not only pragmatically, but also topically. Using this we implemented a dialog-based query-by-example function and built it into an interface for use in combination with normal lexical search. Evaluating its utility by an experiment with four searchers doing twenty tasks each, we found that searchers used the new feature and considered it helpful, but only for some search tasks. 1. Two Views of Audio Search Most current spoken dialogue retrieval systems are based on the view that speech is essentially just noise-corrupted text [1]. They use speech recognition techniques to infer the words said, and then use text-based search techniques on the inferred transcript. However the performance of such systems is generally weak, and today audio search is not widely used. While progress is ongoing, this approach has several fundamental assumptions — for example that speech recognition is mostly accurate, that all words are in the recognizer’s vocabulary, that ambiguity, anaphor and ellipsis are rare, and that searchers can specify all words and synonyms relevant to their intent — which impose fundamental limitations. We take a different perspective on audio search. In most cases users are probably not really interested in finding words in audio archives. What people want is often information of some type, characterized in part semantically, and in part by intent [2], and in part by dialog process [3] or activity, for example recommending, answering a question, agreeing, decision-making, telling life stories, making plans, hearing surprising statements, giving advice, attempting to persuade, explaining, giving opinions and so on. Dialog activities being numerous and complex, users are unlikely to be able to explicitly specify them in formulating a query. However they can be expected to recognize the sorts of things that they are interested in when they hear them, and thus can be expected to benefit from a “more like this” function [4, 5, 6] that returns results similar to a “seed”, that is, an audio snippet used as a query. This paper presents a model of dialog-activity similarity for search. Section 2 introduces our vector space representation, We thank Olac Fuentes, David Novick, Alejandro Vega, Shreyas Karkhedkar, and Mary Harper for ideas, comments, and assistance, and the National Science Foundation for a REU supplement to Award IIS0914868. Section 3 discusses how proximity in this space provides a similarity metric, Section 4 describes an experimental investigation of the utility for search, Section 5 presents the results, and Sections 6 and 7 discuss and conclude. 2. A Vector-Space Representation of Dialog Activity Following previous research, we chose to use prosodic features as the source of information about dialog activity. As is often noted, prosody can encode information that may not be expressed, or even expressible, by lexical means. Work using prosody in search goes back at least to the observation that important words and phrases can be prosodically distinctive and that this can be used to focus search [7]. Dialog activities that prosody can reveal include interactional “hotspots” where the speakers are unusually involved [8, 9], conflicts [10], agreements on action items [11], various emotional and attitudinal states and stances [12], and dialog acts of various types [13], such as question, apology, and promise. While such work has been motivated by search, findings to date show only that prosodic information can be used to detect such regions, not that this functionality is actually useful for searchers. This may be because no finite taxonomy of dialog activities can support most search needs. We therefore chose instead to use an empirically-derived representation of dialog activities. This representation, as described in [14], is derived by the application of Principal Component Analysis to 76 local prosodic features. While using the common features pitch height, pitch range, speaking rate, and volume, this feature set is novel in being computed 1) over different windows across sixseconds of context, thus capturing significant local context, 2) for both participants in the dialog, thus capturing both speaker and listener behavior, and 3) over fixed windows, rather than being word-, syllable-, or phrase-aligned, thus better capturing dialog-activity information. These features were computed every 10 milliseconds throughout the corpus. After PCA this gave 76 dimensions, ordered by how much of the variation they explained. Upon examination [14, 15], it turned out that most of the top dimensions aligned with various aspects of dialog. These aspects were diverse, including dialog situations, transient dialog states, cooperative dialog acts, simple dialog actions, and apparent mental states. For example, there were dimensions that related to grounding, to turn-taking, to seeking and expressing sympathy, to degrees of novelty and interest, to topic shifts and closings, to emphasis, to humor versus regret, to personal versus impersonal topics, to facts, to explanations, and to opinions. We can thus refer to the space defined by these dimensions as “dialog-activity space;” in a dialog every point in time maps to a point in this 76-dimensional space. 3. Proximity and Similarity Given this vector-space model we can use proximity as a measure of similarity. Specifically we use simple Euclidian distance, over all 76 dimensions, without weighting (although of course the top dimensions exhibit higher variance and therefore affect the results more strongly). As a preliminary exploration, we selected a few seeds and examined what positions the model found as most similar. As hoped, proximity correlated with similarity: generally the closer the proximity to the seed, the more similar the regions sounded. These similarities were not just in dialog activity but frequently also in content. For example, an attempt to find information about family members across two 5-minute dialogs, by unrelated speakers, using as seed the phrase “my brother’s a trim carpenter,” using proximity with a certain threshold, gave us 14 matching regions, and of these 7 included information about family members (husbands, children), including some where there was no noun present, only the word “he. ” Interestingly, most false positives related to house construction issues, and probably could have been dealt with by enabling negative relevance feedback. By comparison, textual search on the transcripts of these two dialogs using ten family-related terms gave only 4 hits. A second interesting example was a search for complaints about the government. A little browsing turned up a complaint about a book, using this as a seed led to a complaint about a husband, using this as a seed in turn led to a complaint about the metric system, which finally led to results including a complaint about property tax, a complaint about gun laws and a complaint about public schools. 4. Experiment Design To explore the utility of this similarity metric in audio search we designed an evaluation. In general, information retrieval can be evaluated using either user-centric or technology-centric methods. Standard technology-centric metrics such as recall, precision, and refinements thereof [1, 16, 17] are useful for comparing rival implementations of an approach that is already known to be useful, but less so for evaluating novel approaches. Therefore we adopted a user-centric method, one that directly measures actual value to searchers. To support a controlled experiment, we chose to provide searchers with a set of tasks. Whereas in the examples above the seeds were found by listening to all the audio from the start, we provided the searchers with a lexical search function so they could find seeds faster. Thus, for example, if the task was to find talk about problems with pets, a searcher could first use lexical search, for example on words like pet, dog, and cat, to find possible seeds for dialog-based query-by-example searches. We framed our hypotheses in terms of the advantages of dialog-activity search as an additional function, on top of traditional word-based search. Specifically, we chose to investigate two hypotheses: A) that lexical-plus-dialog-activity search would yield more hits than lexical-only search, and B) that it would be preferred by and helpful to searchers.