Automated Team Discourse Modeling: Test of Performance and Generalization

Automated Team Discourse Modeling: Test of Performance and Generalization Peter W. Foltz 1,3 (pfoltz@crl.nmsu.edu), Melanie J. Martin 2 (mmartin@cs.csustan.edu), Ahmed Abdelali 1 (ahmed@crl.nmsu.edu), Mark Rosenstein 3 (mbr@pearsonkt.com), Rob Oberbreckling 3 (rjo@pearsonkt.com) 1. Computing Research Laboratory and Department of Psychology New Mexico State University, P.O. Box 30001, Las Cruces, NM 88003 2. Department of Computer Science California State University, Stanislaus, 801 West Monte Vista Avenue, Turlock, CA 95382 3. Pearson Knowledge Technologies 4940 Pearl East Circle, Suite 200, Boulder, CO, 80305 knowledge. The manual analysis of team communication has shown promising results, see for example, Bowers et al. (1998). This analysis, however, is quite costly, where coding for content can take upwards of 28 hours per 1 hour of tape (Emmert, 1989) and can be subjective. Thus, there is a need for techniques to automatically analyze team communications to categorize and predict performance. The ultimate goal is to be able to automatically analyze a team’s communication and incorporate that analysis into models that can provide feedback during or after training, as well as predict performance. This work extends approaches to computational analysis of language, while also providing techniques to improve modelling of teams and measuring performance. A number of AI, statistical and machine learning techniques have been applied to discourse modeling, generally for the purpose of improving speech recognition and dialogue systems. However, few have focused directly on just the content of the discourse of teams. Recent methods include decision trees (Core 1998), statistical modelling based on current utterance and discourse history (Chu-Carroll 1998), and hidden Markov models. For example, in the work by Stolcke et al., (2000), they were able to predict the tags assigned to discourse within 15% of the accuracy of trained human annotators, while Kiekel et al., (2004) developed markov models of communication patterns among team members that predicted overall performance. In this work we generate predictive models using Latent Semantic Analysis (LSA) to measure free-form verbal interactions among team members. Because LSA can measure and compare the semantic information in these verbal interactions, it can be used to characterize the quality and quantity of information expressed. LSA analysis can be used to determine the semantic content of any utterance made by a team member as well as to measure the semantic similarity of an entire team’s communication to another team. This paper extends research on automated techniques for analyzing the communication and predicting team performance using corpora of communication of teams performing simulated military missions (see Kiekel et al., 2002; Martin & Foltz, 2004). We focus on two applications of this approach in order to test how well Abstract Team communication provides a rich source of discourse, which can be analyzed and tied to measures of team performance. Our goal is to better understand and model the relationship between team communication and team performance to improve team process, develop collaboration aids, and improve the training of teams. In the present work, we use Latent Semantic Analysis (LSA) for automating the analysis and annotation of team discourse. We describe two approaches to modeling team performance. The first measures the semantic content of a team’s dialogue as a whole to predict the team’s performance. The second categorizes each team member’s statements using an established set of discourse tags and uses them to predict team performance. In three experimental settings we demonstrate the ability of these approaches to model performance and generalize to new datasets. Introduction Evaluating teams of decision-makers in complex problem-solving environments requires effective models of individual and team performance. Without these models, current methods of assessing team and group performance rely largely on global outcome metrics, which often lack information rich enough to diagnose failures or suggest improvements in team cognition or process. Modeling and assessment of teams has been hindered by the fact that the richest source of data, the verbal communication among team members, is difficult to collect and analyze. Prior attempts at annotating the content of communication have relied on tedious manual techniques or have only employed limited data such as frequencies, pattern analyses and durations of communications. With the advent of NLP, AI and machine-learning techniques that can measure the semantic content of communication discourse, novel methods for the analysis and modeling of team communication can be applied. A team’s verbal communication data provides a rich indication of cognitive processing at both the individual and the team level. This can be tied back to both the team’s and each individual team member’s abilities and