SlowGuess/ACL_2020_OCR huggingface dataset:
Sebastian Schuster*
sebschu@stanford.edu
Yuxing Chen*
yxchen28@stanford.edu
Judith Degen
jdegen@stanford.edu
Pragmatic inferences often subtly depend on the presence or absence of linguistic features. For example, the presence of a partitive construction (of the) increases the strength of a so-called scalar inference: listeners perceive the inference that Chris did not eat all of the cookies to be stronger after hearing "Chris ate some of the cookies" than after hearing the same utterance without a partitive, "Chris ate some cookies". In this work, we explore to what extent neural network sentence encoders can learn to predict the strength of scalar inferences. We first show that an LSTM-based sentence encoder trained on an English dataset of human inference strength ratings is able to predict ratings with high accuracy  $(r = 0.78)$ . We then probe the model's behavior using manually constructed minimal sentence pairs and corpus data. We find that the model inferred previously established associations between linguistic features and inference strength, suggesting that the model learns to use linguistic features to predict pragmatic inferences.
An important property of human communication is that listeners can infer information beyond the literal meaning of an utterance. One well-studied type of inference is scalar inference (Grice, 1975; Horn, 1984), whereby a listener who hears an utterance with a scalar item like some infers the negation of a stronger alternative with all:
(1) a. Chris ate some of the cookies.
b.  $\rightsquigarrow$  Chris ate some, but not all, of the cookies.
Early accounts of scalar inferences (e.g., Gazdar 1979; Horn 1984; Levinson 2000) considered them to arise by default unless explicitly contradicted in context. However, in a recent corpus study, Degen (2015) showed that there is much more variability
in scalar inferences from some to not all than previously assumed. Degen (2015) further showed that this variability is not random and that several lexical, syntactic, and semantic/pragmatic features of context explain much of the variance in inference strength. $^{1}$
Recent Bayesian game-theoretic models of pragmatic reasoning (Goodman and Frank, 2016; Franke and Jäger, 2016) are able to integrate speaker expectations with world knowledge to predict listeners' pragmatic inferences in many cases (e.g., Goodman and Stuhlmüller 2013; Degen et al. 2015). However, to compute speaker expectations, these models require manual specification of features as well as specification of a finite set of possible utterances. Further, inference becomes intractable when scaling up beyond toy domains to make predictions for arbitrary utterances. Neural network (NN) models, on the other hand, do not suffer from these limitations: they are capable of making predictions for arbitrary utterances and do not require manual specification of features. Unlike Bayesian game-theoretic models, however, NN models have no explicit pragmatic reasoning mechanisms.
In this work, we investigate to what extent NN models can learn to predict subtle differences in scalar inferences and to what extent these models infer associations between linguistic features and
<sup>1</sup>See Section 2 for the operationalization of inference strength that we use throughout this paper and for a description of these features.
<sup>2</sup>Recent models of generating pragmatic image descriptions (Andreas and Klein, 2016; Cohn-Gordon et al., 2018) and color descriptions (Monroe et al., 2017) have overcome this issue by approximating the distributions of utterances given a set of potential referents. However, these models require a finite set of world states (e.g., several referents to choose from) and a corresponding generative model of utterances (e.g., an image captioning model) and are therefore also limited to scenarios with pre-specified world states and a corresponding generative model.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5387-5403 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
inference strength. In this enterprise we follow the recent successes of NN models in predicting a range of linguistic phenomena such as long distance syntactic dependencies (e.g., Elman 1990; Linzen et al. 2016; Gulordava et al. 2018; Futrell et al. 2019; Wilcox et al. 2019), semantic entailments (e.g., Bowman et al. 2015; Conneau et al. 2018), acceptability judgements (Warstadt et al., 2019b), factuality (Rudinger et al., 2018), negative polarity item licensing environments (Warstadt et al., 2019a), and, to some extent, speaker commitment (Jiang and de Marneffe, 2019a). In particular, we ask:
1. How well can a neural network sentence encoder learn to predict human inference strength judgments for utterances with some?
2. To what extent does such a model capture the qualitative effects of hand-mined contextual features previously identified as influencing inference strength?
To address the first question, we compare the performance of several NN models that differ in the underlying word embedding model (GloVe, ELMo, or BERT). To address the second question, we probe the best model's behavior through an analysis of predictions on manually constructed minimal sentence pairs, a regression analysis, and an analysis of attention weights. We find that the best model is able to predict inference strength ratings on a held-out test set with high accuracy  $(r = 0.78)$ . The three analyses consistently suggest that the model learned associations between inference strength and linguistic features established by previous work (Degen, 2015).
We release data and code at https://github.com/yuxingch/Implicature-Strength-Some.
2 The dataset
We use the annotated dataset collected by Degen (2015), a dataset of the utterances from the Switchboard corpus of English telephone dialogues (Godfrey et al., 1992) with a noun phrase (NP) with some. The dataset consists of 1,362 unique utterances. For each example with a some-NP, Degen (2015) collected inference strength ratings from at least 10 participants recruited on Amazon's Mechanical Turk. Participants saw both the target utterance and ten utterances from the preceding discourse context. They then rated the similarity between the original utterance like (2a) and an utterance in which some was replaced with some, but
not all like (2b), on a 7-point Likert scale with endpoints labeled "very different meaning" (1) and "same meaning" (7). Low similarity ratings thus indicate low inference strength, and high similarity ratings indicate high inference strength.
(2) a. I like - I like to read some of the philosophy stuff.
b. I like - I like to read some, but not all, of the philosophy stuff.
Using this corpus, Degen (2015) found that several linguistic and contextual factors influenced inference strength ratings, including the partitive form of, subjecthood, the previous mention of the NP referent, determiner strength, and modification of the head noun, which we describe in turn.
Partitive: (3a-b) are example utterances from the corpus with and without partitive some-NPs, respectively. Values in parentheses indicate the mean inference strength rating for that item. On average, utterances with partitives yielded stronger inference ratings than ones without.
(3) a. We [...] buy some of our own equipment. (5.3)
b. You sound like you have some small ones in the background. (1.5)
Subjecthood: Utterances in which the some-NP appears in subject position, as in (4a), yielded stronger inference ratings than utterances in which the some-NP appears in a different grammatical position, e.g., as a direct object as in (4b).
(4) a. Some kids are really having it. (5.9)
b. That would take some planning. (1.4)
Previous mention: Discourse properties also have an effect on inference strength. A some-NP with a previously mentioned embedded NP referent yields stronger inferences than a some-NP whose embedded NP referent has not been previously mentioned. For example, (5a) contains a some-NP in which them refers to previously mentioned Mission Impossible tape recordings, whereas problems in the some-NP in (5b) has not been previously mentioned.
(5) a. I've seen some of them on repeats. (5.8)
b. What do you feel are some of the main problems? (3.4)
Modification: Degen (2015) also found a small effect of whether or not the head noun of the some-NP was modified: some-NPs with unmodified head nouns yielded slightly stronger inferences than those with modified head nouns.
Determiner strength: Finally, it has been argued that there are two types of some: a weak some and a strong some (Milsark, 1974; Barwise and Cooper,
1981). This weak/strong distinction has been notoriously hard to pin down (Horn, 1997) and Degen (2015) used empirical strength norms elicited independently for each item. To this end, she exploited the fact that removing weak some from an utterance has little effect on its meaning whereas removing strong some changes the meaning. Determiner strength ratings were thus elicited by asking participants to rate the similarity between the original utterance and an utterance without some (of) on a 7-point Likert scale from 'different meaning' to 'same meaning'. Items with stronger some - e.g., (6a), determiner strength 3.3 - yielded stronger inference ratings than items with weaker some - e.g., (6b), determiner strength 6.7.
(6) a. And some people don't vote. (5.2)
b. Well, we could use some rain up here. (2.1)
The quantitative findings from Degen (2015) are summarized in Figure 4, which shows in blue the regression coefficients for all predictors she considered (see the original paper for more detailed descriptions).
For our experiments, we randomly split the dataset into a  $70\%$  training and  $30\%$  test set, resulting in 954 training items and 408 test items.
The objective of the model is to predict mean inference strength rating  $i$  given an utterance (a sequence of words)  $U = \{w_{1}, w_{2}, \dots, w_{N}\}$ . We rescale the 1-to-7 Likert scale ratings to the interval [0, 1]. Figure 1 shows the overall model architecture. The model is a sentence classification model akin to the model proposed by Lin et al. (2017). It first embeds the utterance tokens using pre-trained embedding models, and then forms a sentence representation by passing the embedded tokens through a 2-layer bidirectional LSTM network (biLSTM) (Hochreiter and Schmidhuber, 1997) with dropout (Srivastava et al., 2014) followed by a self-attention mechanism that provides a weighted average of the hidden states of the top-most biLSTM layer. This sentence representation is then passed through a transformation layer with a sigmoid activation function, which outputs the predicted score in the interval [0, 1].
4.1 Training
We used 5-fold cross-validation on the training data to optimize the following hyperparameters.
Figure 1: Model architecture.
Word embedding model: 100d GloVe (Pennington et al., 2014), 1024d ELMo (Peters et al., 2018; Gardner et al., 2018), 768d BERT-base, 1024d BERT-large (Devlin et al., 2019; Wolf et al., 2019).
Output layer of word embedding models: [1, 3] for ELMo, [1, 12] for BERT-base, and [1, 24] for BERT-large.
Dimension of LSTM hidden states: {100, 200, 400, 800}.
Dropout rate in LSTM:  $\{0.1, 0.2, 0.3, 0.4\}$ .
We first optimized the output layer parameter for each contextual word embedding model while keeping all other parameters fixed. We then optimized the other parameters for each embedding model by computing the average correlation between the model predictions and the human ratings across the five cross-validation folds.
Architectural variants. We also evaluated all combinations of two architectural variants: First, we evaluated models in which we included the attention layer (LSTM+ATTENTION) or simply used the final hidden state of the LSTM (LSTM) as a sentence representation. Second, since participants providing inference strength ratings also had access to 10 utterances from the preceding conversational context, we also compared models that make predictions based only on the target utterance with the some-NP and models that make predictions based on target utterances and the preceding conversational context. For the models using GloVe and ELMo, we pretended the conversational context to the target utterance to obtain a joint context and utterance embedding. For models using BERT, we made use of the fact that BERT had been trained to jointly embed two sentences or documents, and we obtained embeddings for the tokens in the target
utterance by feeding the target utterance as the first document and the preceding context as the second document into the BERT encoder. We discarded the hidden states of the preceding context and only used the output of BERT for the tokens in the target utterance.
Implementation details. We implemented the model in PyTorch (Paszke et al., 2017). We trained the model using the Adam optimizer (Kingma and Ba, 2015) with default parameters and a learning rate of 0.001, minimizing the mean squared error of the predicted ratings. In the no-context experiments, we truncated target utterances longer than 30 tokens, and in the experiments with context, we truncated the beginning of the preceding context such that the number of tokens did not exceed 150. Evaluation. We evaluated the model predictions in terms of their correlation  $r$  with the human inference strength ratings. As mentioned above, we optimized the hyperparameters using cross validation. We then took the best set of parameters and trained a model on all the available training data and evaluated that model on the held-out data.
4.2 Tuning results
Not surprisingly, we find that the attention layer improves predictions and that contextual word embeddings lead to better results than the static GloVe embeddings. We also find that including the conversational context does not improve predictions (see Appendix A, for learning curves of all models, and Section 6, for a discussion of the role of conversational context).
Otherwise, the model is quite insensitive to hyperparameter settings: neither the dimension of the hidden LSTM states nor the dropout rate had considerable effects on the prediction accuracy. We do find, however, that there are differences depending on the BERT and ELMo layer that we use as word representations. We find that higher layers work better than lower layers, suggesting that word representations that are influenced by other utterance tokens are helpful for this task.
Based on these optimization runs, we chose the model with attention that uses the BERT-large embeddings but no conversational context for the subsequent experiments and analyses.
4.3 Test results
Figure 2 shows the correlation between the best model according to the tuning runs (now trained on all training data) and the empirical ratings on
Figure 2: Correlation between empirical ratings and predictions of the BERT-LARGE LSTM+ATTENTION model on held-out test items.
the 408 held-out test items. As this plot shows, the model predictions fall within a close range of the empirical ratings for most of the items  $(r = 0.78)$ . Further, similarly as in the empirical data, there seem to be two clusters in the model predictions: one that includes lower ratings and one that includes higher ratings, corresponding to strong and weak scalar inferences, respectively. The only systematic deviation appears to be that the model does not predict any extreme ratings – almost all predictions are greater than 2 or less than 6, whereas the empirical ratings include some cases outside of this range.
Overall, these results suggest that the model can learn to closely predict the strength of scalar inferences. However, this result by itself does not provide evidence that the model learned associations between linguistic features and inference strength, since it could also be that, given the large number of parameters, the model learned spurious correlations independent of the empirically established feature-strength associations. To investigate whether the model learned the expected associations, we probed the model's behavior in multiple ways, which we discuss next.
5 Model behavior analyses
Minimal pair analysis. As a first analysis, we constructed artificial minimal pairs that differed along several factors of interest and compared the model predictions. Such methods have been recently used to probe, for example, what kind of
For comparison, we estimated how well the human ratings correlated through a bootstrapping analysis: We re-sampled the human ratings for each item and computed the average correlation coefficient between the original and the re-sampled datasets, which we found to be approximately 0.93.
syntactic dependencies different types of recurrent neural network language models are capable of encoding or to what extent sentence vector representations capture compositional meanings (e.g., Linzen et al. 2016; Gulordava et al. 2018; Chowdhury and Zamparelli 2018; Ettinger et al. 2018; Marvin and Linzen 2018; Futrell et al. 2019; Wilcox et al. 2019), and also allow us to probe whether the model is sensitive to controlled changes in the input.
We constructed a set of 25 initial sentences with some-NPs. For each sentence, we created 32 variants that differed in the following four properties of the some-NP: subjcethood, partitive, pre-nominal modification, and post-nominal modification. For the latter three features, we either included or excluded of the or the modifier, respectively. For example, the version in (7a) includes of the whereas the version in (7b) lacks the partitive feature. To manipulate subjcethood of the some-NP, we created variants in which some was either the determiner in the subject NP as in (7) or in the object-NP as in (8). We also created passive versions of each of these variants (9-10). Each set of sentences included a unique main verb, a unique pair of NPs, and unique modifiers. The full list of sentences can be found in Appendix C.
(7) a. Some of the (organic) farmers (in the mountains) milked the brown goats who graze on the meadows.
b. Some (organic) farmers (in the mountains) milked the brown goats who graze on the meadows.
(8) The organic farmers in the mountains milked some (of the) (brown) goats (who graze on the meadows).
(9) The brown goats who graze on the meadows were milked by some (of the) (organic) farmers (in the mountains).
(10) Some (of the) (brown) goats (who graze on the meadows) were milked by the organic farmers in the mountains.
Figure 3 shows the model predictions for the manually constructed sentences grouped by the presence of a partitive construction, the grammatical function of the some-NP, and the presence of a modifier. As in the natural dataset from Degen (2015), sentences with a partitive received higher predicted ratings than sentences without a partitive; sentences with subject some-NPs received higher predicted ratings than sentences with non-subject some-NPs; and sentences with a modified head noun in the some-NP received lower predictions than sentences with an unmodified some-NP.
Figure 3: Average model predictions on manually constructed sentences, grouped by presence of partitives, by grammatical function of the some-NP, and by presence of nominal modifiers. Semi-transparent dots show predictions on individual sentences.
All these results provide evidence that the model learned the correct associations. This is particularly remarkable considering the train-test mismatch: the model was trained on noisy transcripts of spoken language that contained many disfluencies and repairs, and was subsequently tested on clean written sentences.
Regression analysis. In the minimal pair analysis above we only investigated model predictions for three factors. As a second analysis, we therefore investigated whether the predictions of the best neural network model explain the variance explained by the linguistic features that modulate inference strength. To this end, we used a slightly simplified<sup>4</sup> Bayesian implementation of the mixed-effects model by Degen (2015) that predicted inference strength ratings from hand-mined features. We used the brms (Bürkner, 2017) and STAN (Carpenter et al., 2017) packages and compared this original model to an extended model that included both all of the predictors of the original model as well as the output of the above NN model as a predictor. For this comparison, we investigated whether the magnitude of a predictor in the original model significantly decreased in the extended model with the NN predictor, based on the reasoning that if the NN predictions explain the variance previously explained by these manually coded pre
4We removed by-item random intercepts and by-subject random slopes to facilitate inference. This simplification yielded almost identical estimates as the original model by Degen (2015).
Figure 4: Maximum a posteriori estimates and  $95\%$  -credible intervals of coefficients for original and extended Bayesian mixed-effects regression models predicting the inference strength ratings.  $* / * *$ /\*\*\* indicate that the probability of the coefficient of the original model having a larger magnitude than the coefficient of the extended model is less than 0.05, 0.01, and 0.001, respectively.
dictors, then the original predictor should explain no or less additional variance.
We approximated the probability that the magnitude of the coefficient for the predictor  $i(\beta_i)$  in the extended model including the NN predictor is smaller than the coefficient in the original model,  $P(|\beta_i^{extended}| < |\beta_i^{original}|)$ , by sampling values for each coefficient from the distributions of the original and the extended models and comparing the magnitude of the sampled coefficients. We repeated this process 1,000,000 times and treated the simulated proportions as approximate probabilities.
An issue with this analysis is that estimating the regression model only on the items in the held-out test set yields very wide credible intervals for some of the predictors-in particular for some of the interactions-since the model infers these values from very little data. We therefore performed this regression analysis (and the subsequent analyses) on the entire data. However, while we estimated the regression coefficients from all the data, we crucially obtained the NN predictions through 6-fold cross-validation (without additional tuning of hyperparameters), so that the NN model always made predictions on data that it had not seen during training. This did yield the same qualitative results as the analyses only performed on the held-out test items (see Appendix B) but it also provided us with narrower credible intervals that highlight the differences between the coefficient estimates of the two models.
Figure 4 shows the estimates of the coefficients in the original model and the extended model. We find that the NN predictions explain some or all
of the variance originally explained by many of the manually coded linguistic features: the estimated magnitude of the predictors for partitive, determiner strength, linguistic mention, subjecthood, modification, utterance length, and two of the interaction terms decreased in the extended model. These results provide additional evidence that the NN model indeed learned associations between linguistic features and inference strength rather than only explaining variance caused by individual items. This is particularly true for the grammatical and lexical features; we find that the NN predictor explains most of the variance originally explained by the partitive, subjecthood, and modification predictors. More surprisingly, the NN predictions also explain a lot of the variance originally explained by the determiner strength predictor, which was empirically determined by probing human interpretation and is not encoded explicitly in the surface form utterance. One potential explanation for this is that strong and weak some have different context distributions. For instance, weak some occurs in existential there constructions and with individual-level predicates, whereas strong some tends not to (Milsark, 1974; McNally and Geenhoven, 1998; Carlson, 1977). Since pre-trained word embedding models capture a lot of distributional information, the NN model is presumably able to learn this association.