Abstract: The strength of pragmatic inferences systematically depends on linguistic and contextual cues. For example, the presence of a partitive construction increases the strength of a so-called scalar inference: humans perceive the inference that Chris did not eat all of the cookies to be stronger after hearing "Chris ate some of the cookies" than after hearing the same utterance without a partitive, "Chris ate some cookies". In this work, we explore to what extent it is possible to learn associations between linguistic cues and inference strength ratings without direct supervision. We show that an LSTM-based sentence encoder with an attention mechanism trained on a dataset of human inference strength ratings is able to predict ratings with high accuracy (r=0.78). We probe the model's behavior in multiple analyses using corpus data and manually constructed minimal pairs and find that the model learns associations between linguistic cues and scalar inferences, suggesting that these associations are inferable from statistical input.

Introduction: An important property of human communication is that listeners can infer information beyond the literal meaning of an utterance. One well-studied type of inference is scalar inference BIBREF0, BIBREF1, whereby a listener who hears an utterance with a scalar item like some infers the negation of a stronger alternative with all:  Chris ate some of the cookies. $\rightsquigarrow $ Chris ate some, but not all, of the cookies. Early accounts of scalar inferences (e.g., BIBREF2, BIBREF1, BIBREF3) considered them to arise by default unless specifically cancelled by the context. However, in a recent corpus study, degen2015investigating showed that there is much more variability in the strength of scalar inferences from some to not all than previously assumed. degen2015investigating further showed that this variability is not random and that several lexical, syntactic, and semantic/pragmatic features of context explain much of the variance in inference strength. Recent Bayesian game-theoretic models of pragmatic reasoning BIBREF4, BIBREF5, which are capable of integrating multiple linguistic cues with world knowledge, are able to correctly predict listeners' pragmatic inferences in many cases (e.g., BIBREF6, BIBREF7). These experimental and modeling results suggest that listeners integrate multiple linguistic and contextual cues in utterance interpretation, raising the question how listeners are able to draw these pragmatic inferences so quickly in interaction. This is an especially pressing problem considering that inference in Bayesian models of pragmatics is intractable when scaling up beyond toy domains to make predictions about arbitrary utterances. One possibility is that language users learn to use shortcuts to the inference (or lack thereof) by learning associations between the speaker's intention and surface-level cues present in the linguistic signal across many instances of encountering a scalar expression like some. In this work, we investigate whether it is possible to learn such associations between cues in the linguistic signal and speaker intentions by training neural network models to predict empirically elicited inference strength ratings from the linguistic input. In this enterprise we follow the recent successes of neural network models in predicting a range of linguistic phenomena such as long distance syntactic dependencies (e.g., BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15), semantic entailments (e.g., BIBREF16, BIBREF17), acceptability judgements BIBREF18, factuality BIBREF19, and, to some extent, speaker commitment BIBREF20. In particular, we ask:  How well can a neural network sentence encoder learn to predict human inference strength judgments for utterances with some? To what extent does such a model capture the qualitative effects of hand-mined contextual features previously identified as influencing inference strength? To address these questions, we first compare the performance of neural models that differ in the underlying word embedding model (GloVe, ELMo, or BERT) and in the sentence embedding model (LSTM, LSTM$+$attention). We then probe the best model's behavior through a regression analysis, an analysis of attention weights, and an analysis of predictions on manually constructed minimal sentence pairs.

Question: What dataset did they use?
Answer: weibo-100k, Ontonotes, LCQMC and XNLI