Lost your marbles? The puzzle of dependent measures in experimental pragmatics.

Lost your marbles? The puzzle of dependent measures in experimental pragmatics Judith Degen (jdegen@stanford.edu), Noah D. Goodman (ngoodman@stanford.edu) Department of Psychology, Stanford University Stanford, CA USA 94350 Abstract then asked to provide a binary judgment about the truth of an utterance containing a scalar item, such as Some of the B’s are in the box on the left. In metalinguistic judgment paradigms, participants are shown an utterance like Some of the B’s are in the box on the left and are then asked explicitly whether it follows that not all of the B’s are in the box on the left (Geurts & Pouscoulous, 2009). Yet neither of these depen- dent measures seems to directly measure the natural modes of language: production and interpretation. In everyday life people constantly produce and interpret language, but much more rarely adjudicate the truth of a sentence. Occasionally researchers have used measures that more directly evaluate interpretation (e.g. Goodman & Stuhlm¨uller, 2013) but some evidence—as well as much anecdotal experience—suggests that these measures may be less sensitive and more unstable. A rarely discussed but important issue in research on pragmatic inference is the choice of dependent measure for estimating the robustness of pragmatic inferences and their sensitivity to contextual manipulations. Here we present the results from three studies exploring the effect of contextual manipulations on scalar implicature. In all three studies we manipulate the salient question under discussion and the perceptual availabil- ity of relevant set sizes. The studies differ only in the depen- dent measure used: Exp. 1 uses truth judgements, Exp. 2 uses word probability ratings, and Exp. 3 uses a direct measure of sentence interpretation. We argue that the first two are effec- tively measures of production, and find they are sensitive to our contextual manipulations. In contrast the interpretation mea- sure shows no effect of context. We argue that this method- ologically troubling finding can be understood and predicted by using the framework of probabilistic pragmatics. Keywords: pragmatics; psycholinguistics; scalar implicature; QUD; methodology Methodologically, it is unsettling that implicature rates dif- fer depending on the dependent measure, and that different dependent measures are sensitive to contextual manipulations to different degrees. How is one to choose a dependent mea- sure that adequately reflects the underlying contextual effect, if there is one? We will not be able to answer this question fully here. Rather, we will present results from three exper- iments that provide the first step towards a full investigation into the sensitivity of different dependent measures to manip- ulations of context and the underlying pragmatic inference process. We manipulate the dependent measure that is used to evaluate listeners’ sensitivity to two contextual cues when they interpret utterances containing scalar items: a) the im- plicit contextual Question Under Discussion (QUD, Roberts, 2004) that interlocutors are trying to address; and b) the size of the set that the quantifier some is being used to describe. The experiments we present here are identical with the excep- tion of participants’ task. Introduction Context affects language understanding in complex and pro- found ways. Relatively small changes in the context of an utterance can radically change the interpreted meaning—or fail to entirely. For instance, take a scalar implicature: “some of the candies in that box contained nuts” will often be prag- matically strengthened to “and not all of them did”. How- ever, finding out that the speaker ate only two candies (thus lacked knowledge of the stronger situation) or has a nut al- lergy (thus cares only if any candies contain nuts) can de- crease the strength of this implicature (i.e., make it more plau- sible that all the candies had nuts). Pragmatic judgments, e.g., about scalar implicatures, are notoriously volatile, responding not only to minor aspects of the cover story, but also to the dependent measure used to probe linguistic intuitions (Geurts & Pouscoulous, 2009; Zondervan, 2010). Contradictory results from different de- pendent measures are particularly troubling to progress in the empirical study of pragmatics. Here we systematically ex- plore three dependent measures of scalar implicature, varying the context, aiming to relate them to each other and to recent formal models of pragmatics. We offer a potential explana- tion for why some measures are more sensitive to context ma- nipulation than others. Most experimental studies on scalar implicature have used either sentence verification or metalinguistic judgment paradigms to probe participants’ interpretation of utterances containing scalar items (Bott & Noveck, 2004; Geurts & Pouscoulous, 2009; Geurts, 2010; Zondervan, 2010; Degen & Tanenhaus, to appear). In sentence verification studies, participants are often shown a visual display or asked to read a story, thus establishing the facts about the world. They are Exp. 1 uses sentence verification, the most widely em- ployed measure in the literature. We will speculate that sen- tence verification is closely related to production; to evalu- ate this idea we perform Exp. 2, which uses word probability ratings as a more direct test of listeners’ expectations about speakers’ production. Finally, Exp. 3 uses sentence inter- pretation—a measure that constitutes arguably the most di- rect and natural test of understanding by allowing participants to distribute confidence over different potential states of the world in response to an observed utterance. We delay discussion of the differences between these mea- sures to the General Discussion, where we provide an ar- gument that the differences in the results of Exps. 1 and 2 vs. those of Exp. 3 are expected under a formal, probabilistic model of pragmatics.