How to Analyze Verbal Protocols

How to Analyze Verbal Protocols Thora Tenbrink (t.tenbrink@bangor.ac.uk) School of Linguistics and English Language Room 331, 3rd floor New Arts, Bangor University, College Road, Bangor, Gwynedd LL57 2DG, UK Fon: +44 1248 382263; Fax: +44 1248 383267 Keywords: Think-aloud protocols; verbal data; cognitive processes; cognitive discourse analysis; linguistic structure; problem solving; complex cognition. consciously aware of the network of options (Tenbrink & Freksa, 2009) that allows for a range of linguistic choices beside their own, which emerges more clearly by considering a larger data set collected under controlled circumstances. According to previous research in cognitive linguistics and discourse analysis (e.g., van Dijk, 2008), linguistic features such as the verbal representation of semantic domains reflected in ideational networks, specific choices of prepositions, lexical omissions and elaboration, conceptual perspectives revealed by language, presuppositions, hesitation and discourse markers, and many other linguistic features indicate certain conceptual circum- stances; these are related to the current cognitive represen- tations in ways that distinguish them from other options available in the network. In particular, the chosen linguistic options reflect what speakers perceive as sufficiently relevant to be verbalized, as well as the information status assigned to the diverse parts of the verbalization. Besides building on established insights about the significance of particular linguistic choices, validating evidence for the relationship between patterns of language use and the associated cognitive processes can be gained by triangulation, i.e., the combination of linguistic analysis with other types of evidence such as memory or behavioral performance data, reaction times, eye movements, decision outcomes, or any other relevant data that can be collected in cognitively complex tasks. Objectives and Scope Cognitive science researchers are interested in a subject that is not directly accessible to observation: processes in the mind and brain, thoughts and thought processes. One way of addressing higher-level cognitive processes is to analyze verbal protocols produced along with cognitively complex tasks (Ericsson & Simon, 1993), such as problem solving or decision making. Linguistic data of this kind can be seen as an external representation of some aspects of what is going on in the mind. In particular, think-aloud protocols and retrospective reports provide procedural information that complements other data, such as decision outcomes and behavioral performance results. This tutorial explores the scope and limitations of verbal protocol analysis, and offers practical support for systematic analysis procedures. Language data can be analyzed with respect to content as well as structure. Conventionally, the focus of verbal protocol analysis lies on the content of verbal data, addressing those aspects (e.g., particular thought processes or strategies) that the speakers are themselves aware of (or 'heed', Ericsson & Simon, 1993). The content-based inspection of verbal reports, particularly if carried out by experts in the problem domain and set against a substantial theoretical background (Krippendorff, 2004), often leads to well-founded specific hypotheses about the cognitive processes involved. A detailed linguistic analysis can substantially support such content-based insights, but it can also offer further insights (e.g., Holscher et al., 2011; Tenbrink et al., 2011; Tenbrink & Seifert, 2011; Tenbrink & Wiener, 2009). Research in cognitive linguistics, psychology, discourse analysis, and psycholinguistics indicates that patterns in language are systematically related to patterns of thought (e.g., Chafe, 1998). Drawing on these insights, one focus of the tutorial is to identify types of linguistic structure that point to specific cognitive processes. This is the main idea in the method of Cognitive Discourse Analysis (CODA) (Tenbrink, 2008; Tenbrink & Gralla, 2009; Tenbrink, 2010). Some aspects of language use reflect cognitive aspects that go beyond conscious reflection by individual speakers, and that are not necessarily directly observable in linguistic content. Speakers are typically unaware of the cognitive structures that are reflected in particular ways of framing a representation linguistically. Furthermore, they are not Format and organization This tutorial is designed to cover a half day (three hours) and will be highly interactive. The tutorial will take the participants' current or intended projects as a starting point. It will be organized so as to cover the complete process of language data analysis (from initial ideas to evaluation of analysis results), including short presentations, discussion, and practical exercises where feasible. In particular, the following issues will be addressed: Motivation: How (and to what extent) can language data serve as empirical resources to address research questions in cognitive science? Data collection: What kinds of issues need to be considered in the light of actual research purposes? CODA based analysis (main part): Systematic data annotation and interpretation informed by linguistic insights. Triangulation: How can other types of empirical data complement the insights gained from language?