Venue: ACL 2020

# Harnessing the linguistic signal to predict scalar inferences

Sebastian Schuster\({}^{*}\)

Stanford University

sebschu@stanford.edu &Yuxing Chen\({}^{*}\)

Stanford University

yxchen28@stanford.edu &Judith Degen

Stanford University

jdegen@stanford.edu

###### Abstract

Pragmatic inferences often subtly depend on the presence or absence of linguistic features. For example, the presence of a partitive construction (_of tthe_) increases the strength of a so-called scalar inference: listeners perceive the inference that Chris did not eat all of the cookies to be stronger after hearing "_Chris ate some of the cookies"_ than after hearing the same utterance without a partitive, "_Chris ate some cookies"_. In this work, we explore to what extent neural network sentence encoders can learn to predict the strength of scalar inferences. We first show that an LSTM-based sentence encoder trained on an English dataset of human inference strength ratings is able to predict ratings with high accuracy (\(r=0.78\)). We then probe the model's behavior using manually constructed minimal sentence pairs and corpus data. We find that the model inferred previously established associations between linguistic features and inference strength, suggesting that the model learns to use linguistic features to predict pragmatic inferences.

+
Footnote †: Equal contribution.

+
Footnote †: Equal contribution.

## 1 Introduction

An important property of human communication is that listeners can infer information beyond the literal meaning of an utterance. One well-studied type of inference is _scalar inference_[9], whereby a listener who hears an utterance with a scalar item like _some_ infers the negation of a stronger alternative with _all_:

1. [label=(0)]
2. Chris ate some of the cookies.
3. \(\leadsto\) Chris ate some, but not all, of the cookies.

Early accounts of scalar inferences (e.g., Gazdar 1979; Horn 1984; Levinson 2000) considered them to arise by default unless explicitly contradicted in context. However, in a recent corpus study, Degen (2015) showed that there is much more variability in scalar inferences from _some_ to _not all_ than previously assumed. Degen (2015) further showed that this variability is not random and that several lexical, syntactic, and semantic/pragmatic features of context explain much of the variance in inference strength.1

Footnote 1: See Section 2 for the operationalization of inference strength that we use throughout this paper and for a description of these features.

Recent Bayesian game-theoretic models of pragmatic reasoning [6, 10] are able to integrate speaker expectations with world knowledge to predict listeners' pragmatic inferences in many cases (e.g., Goodman and Stuhlmuller 2013; Degen et al. 2015). However, to compute speaker expectations, these models require manual specification of features as well as specification of a finite set of possible utterances. Further, inference becomes intractable when scaling up beyond toy domains to make predictions for arbitrary utterances.2 Neural network (NN) models, on the other hand, do not suffer from these limitations: they are capable of making predictions for arbitrary utterances and do not require manual specification of features. Unlike Bayesian game-theoretic models, however, NN models have no explicit pragmatic reasoning mechanisms.

In this work, we investigate to what extent NN models can learn to predict subtle differences in scalar inferences and to what extent these models infer associations between linguistic features and

inference strength. In this enterprise we follow the recent successes of NN models in predicting a range of linguistic phenomena such as long distance syntactic dependencies (e.g., Elman, 1990; Linzen et al., 2016; Gulordava et al., 2018; Futrell et al., 2019; Wilcox et al., 2019), semantic entailments (e.g., Bowman et al., 2015; Conneau et al., 2018), acceptability judgements Warstadt et al. (2019), factuality Rudinger et al. (2018), negative polarity item licensing environments Warstadt et al. (2019), and, to some extent, speaker commitment Jiang and de Marneffe (2019). In particular, we ask:

1. How well can a neural network sentence encoder learn to predict human inference strength judgments for utterances with _some_?
2. To what extent does such a model capture the qualitative effects of hand-mined contextual features previously identified as influencing inference strength?

To address the first question, we compare the performance of several NN models that differ in the underlying word embedding model GloVe, ELMo, or BERT). To address the second question, we probe the best model's behavior through an analysis of predictions on manually constructed minimal sentence pairs, a regression analysis, and an analysis of attention weights. We find that the best model is able to predict inference strength ratings on a held-out test set with high accuracy (\(r=0.78\)). The three analyses consistently suggest that the model learned associations between inference strength and linguistic features established by previous work Degen (2015).

We release data and code at [https://github.com/yuxingch/Implicature-Strength-Some](https://github.com/yuxingch/Implicature-Strength-Some).

## 2 The dataset

We use the annotated dataset collected by Degen (2015), a dataset of the utterances from the Switchboard corpus of English telephone dialogues Godfrey et al. (1992) with a noun phrase (NP) with _some_. The dataset consists of 1,362 unique utterances. For each example with a _some_-NP, Degen (2015) collected inference strength ratings from at least 10 participants recruited on Amazon's Mechanical Turk. Participants saw both the target utterance and ten utterances from the preceding discourse context. They then rated the similarity between the original utterance like (2a) and an utterance in which _some_ was replaced with _some, but not all_ like (2b), on a 7-point Likert scale with endpoints labeled "very different meaning" (1) and "same meaning" (7). Low similarity ratings thus indicate low inference strength, and high similarity ratings indicate high inference strength.

* I like to read _some_ of the philosophy stuff.
* I like to read _some, but not all,_ of the philosophy stuff.

Using this corpus, Degen (2015) found that several linguistic and contextual factors influenced inference strength ratings, including the partitive form _of_, subjectionod, the previous mention of the NP referent, determiner strength, and modification of the head noun, which we describe in turn.

**Partittive:** (3a-b) are example utterances from the corpus with and without partitive _some_-NPs, respectively. Values in parentheses indicate the mean inference strength rating for that item. On average, utterances with partitives yielded stronger inference ratings than ones without.

* We [...] buy _some_ of our own equipment. (5.3)
* You sound like you have _some_ small ones in the background.

**Subjecthood:** Utterances in which the _some_-NP appears in subject position, as in (4a), yielded stronger inference ratings than utterances in which the _some_-NP appears in a different grammatical position, e.g., as a direct object as in (4b).

* _Some_ kids are really having it. (5.9)
* That would take _some_ planning. (1.4)

**Previous mention:** Discourse properties also have an effect on inference strength. A _some_-NP with a previously mentioned embedded NP referent yields stronger inferences than a _some_-NP whose embedded NP referent has not been previously mentioned. For example, (5a) contains a _some_-NP in which _them_ refers to previously mentioned _Mission Impossible tape recordings_, whereas _problems_ in the _some_-NP in (5b) has not been previously mentioned.

* I've seen _some_ of them on repeats. (5.8)
* What do you feel are _some_ of the main problems? (3.4)

**Modification:**Degen (2015) also found a small effect of whether or not the head noun of the _some_-NP was modified: _some_-NPs with unmodified head nouns yielded slightly stronger inferences than those with modified head nouns.

**Determiner strength:** Finally, it has been argued that there are two types of _some_: a weak _some_ and a strong _some_ Milsark (1974); Barwise and Cooper ( 

1981). This weak/strong distinction has been notoriously hard to pin down Horn (1997) and Degen (2015) used empirical strength norms elicited independently for each item. To this end, she exploited the fact that removing weak _some_ from an utterance has little effect on its meaning whereas removing strong _some_ changes the meaning. Determiner strength ratings were thus elicited by asking participants to rate the similarity between the original utterance and an utterance without _some (of)_ on a 7-point Likert scale from 'different meaning' to 'same meaning'. Items with stronger _some_ - e.g., (6a), determiner strength 3.3 - yielded stronger inference ratings than items with weaker _some_ - e.g., (6b), determiner strength 6.7.
* [label=(6)]
* An _some_ people don't vote.
* Well, we could use _some_ rain up here.
* The quantitative findings from Degen (2015) are summarized in Figure 4, which shows in blue the regression coefficients for all predictors she considered (see the original paper for more detailed descriptions). For our experiments, we randomly split the dataset into a 70% training and 30% test set, resulting in 954 training items and 408 test items.

## 3 Model

The objective of the model is to predict mean inference strength rating \(i\) given an utterance (a sequence of words) \(U=\{w_{1},w_{2},...,w_{N}\}\). We rescale the 1-to-7 Likert scale ratings to the interval \([0,1]\). Figure 1 shows the overall model architecture. The model is a sentence classification model akin to the model proposed by Lin et al. (2017). It first embeds the utterance tokens using pre-trained embedding models, and then forms a sentence representation by passing the embedded tokens through a 2-layer bidirectional LSTM network (biLSTM) Hochreiter and Schmidhuber (1997) with dropout Srivastava et al. (2014) followed by a self-attention mechanism that provides a weighted average of the hidden states of the topmost biLSTM layer. This sentence representation is then passed through a transformation layer with a sigmoid activation function, which outputs the predicted score in the interval \([0,1]\).

## 4 Experiments

### Training

We used 5-fold cross-validation on the training data to optimize the following hyperparameters.

**Word embedding model**: 100d GloVe Pennington et al. (2014), 1024dLMo Peters et al. (2018); Gardner et al. (2018), 768d BERT-base, 1024d BERT-large Devlin et al. (2019); Wolf et al. (2019).

**Output layer of word embedding models**: \([1,3]\) for ELMo, \([1,12]\) for BERT-base, and \([1,24]\) for BERT-large.

**Dimension of LSTM hidden states**: \(\{100,200,400,800\}\).

**Dropout rate in LSTM**: \(\{0.1,0.2,0.3,0.4\}\).

We first optimized the output layer parameter for each contextual word embedding model while keeping all other parameters fixed. We then optimized the other parameters for each embedding model by computing the average correlation between the model predictions and the human ratings across the five cross-validation folds.

**Architectural variants.** We also evaluated all combinations of two architectural variants: First, we evaluated models in which we included the attention layer (LSTM+Attention) or simply used the final hidden state of the LSTM (LSTM) as a sentence representation. Second, since participants providing inference strength ratings also had access to 10 utterances from the preceding conversational context, we also compared models that make predictions based only the target utterance with the _some_-NP and models that make predictions based on target utterances and the preceding conversational context. For the models using GloVe and ELMo, we prepended the conversational context to the target utterance to obtain a joint context and utterance embedding. For models using BERT, we made use of the fact that BERT had been trained to jointly embed two sentences or documents, and we obtained embeddings for the tokens in the target

Figure 1: Model architecture.

 

utterance by feeding the target utterance as the first document and the preceding context as the second document into the BERT encoder. We discarded the hidden states of the preceding context and only used the output of BERT for the tokens in the target utterance.

**Implementation details.** We implemented the model in PyTorch Paszke et al. (2017). We trained the model using the Adam optimizer Kingma and Ba (2015) with default parameters and a learning rate of 0.001, minimizing the mean squared error of the predicted ratings. In the no-context experiments, we truncated target utterances longer than 30 tokens, and in the experiments with context, we truncated the beginning of the preceding context such that the number of tokens did not exceed 150.

**Evaluation.** We evaluated the model predictions in terms of their correlation \(r\) with the human inference strength ratings. As mentioned above, we optimized the hyperparameters using cross validation. We then took the best set of parameters and trained a model on all the available training data and evaluated that model on the held-out data.

### Tuning results

Not surprisngly, we find that the attention layer improves predictions and that contextual word embeddings lead to better results than the static GloVe embeddings. We also find that including the conversational context does not improve predictions (see Appendix A, for learning curves of all models, and Section 6, for a discussion of the role of conversational context).

Otherwise, the model is quite insensitive to hyperparameter settings: neither the dimension of the hidden LSTM states nor the dropout rate had considerable effects on the prediction accuracy. We do find, however, that there are differences depending on the BERT and ELMo layer that we use as word representations. We find that higher layers work better than lower layers, suggesting that word representations that are influenced by other utterance tokens are helpful for this task.

Based on these optimization runs, we chose the model with attention that uses the BERT-large embeddings but no conversational context for the subsequent experiments and analyses.

### Test results

Figure 2 shows the correlation between the best model according to the tuning runs (now trained on all training data) and the empirical ratings on the 408 held-out test items. As this plot shows, the model predictions fall within a close range of the empirical ratings for most of the items (\(r=0.78\)).3 Further, similarly as in the empirical data, there seem to be two clusters in the model predictions: one that includes lower ratings and one that includes higher ratings, corresponding to strong and weak scalar inferences, respectively. The only systematic deviation appears to be that the model does not predict any extreme ratings - almost all predictions are greater than 2 or less than 6, whereas the empirical ratings include some cases outside of this range.

Footnote 3: For comparison, we estimated how well the human ratings correlated through a bootstrapping analysis: We re-sampled the human ratings for each item and computed the average correlation coefficient between the original and the re-sampled datasets, which we found to be approximately 0.93.

Overall, these results suggest that the model can learn to closely predict the strength of scalar inferences. However, this result by itself does not provide evidence that the model learned associations between linguistic features and inference strength, since it could also be that, given the large number of parameters, the model learned spurious correlations independent of the empirically established feature-strength associations. To investigate whether the model learned the expected associations, we probed the model's behavior in multiple ways, which we discuss next.

## 5 Model behavior analyses

Minimal pair analysis.As a first analysis, we constructed artificial minimal pairs that differed along several factors of interest and compared the model predictions. Such methods have been recently used to probe, for example, what kind of

Figure 2: Correlation between empirical ratings and predictions of the BERT-large LSTM+Attention model on held-out test items.

 

syntactic dependencies different types of recurrent neural network language models are capable of encoding or to what extent sentence vector representations capture compositional meanings (e.g., Linzen et al.2016; Gulordava et al.2018; Chowdhury and Zamparelli2018; Ettinger et al.2018; Marvin and Linzen2018; Futrell et al.2019; Wilcox et al.2019), and also allow us to probe whether the model is sensitive to controlled changes in the input.

We constructed a set of 25 initial sentences with _some_-NPs. For each sentence, we created 32 variants that differed in the following four properties of the _some_-NP: subjecthod, partitive, pre-nominal modification, and post-nominal modification. For the latter three features, we either included or excluded _of the_ or the modifier, respectively. For example, the version in (7a) includes _of the_ whereas the version in (7b) lacks the partitive feature. To manipulate subjecthood of the _some_-NP, we created variants in which _some_ was either the determiner in the subject NP as in (7) or in the object-NP as in (8). We also created passive versions of each of these variants (9-10). Each set of sentences included a unique main verb, a unique pair of NPs, and unique modifiers. The full list of sentences can be found in Appendix C.

* Some of the (organic) farmers (in the mountains) milled the brown goats who graze on the meadows.
* Some (organic) farmers (in the mountains) milked the brown goats who graze on the meadows.
* The organic farmers in the mountains milked some (of the) (brown) goats (who graze on the meadows).
* The brown goats who graze on the meadows were milked by some (of the) (organic) farmers (in the mountains).
* Some (of the) (brown) goats (who graze on the meadows) were milked by the organic farmers in the mountains.

Figure 3 shows the model predictions for the manually constructed sentences grouped by the presence of a partitive construction, the grammatical function of the _some_-NP, and the presence of a modifier. As in the natural dataset from Degen2015, sentences with a partitive received higher predicted ratings than sentences without a partitive; sentences with subject _some_-NPs received higher predicted ratings than sentences with non-subject _some_-NPs; and sentences with a modified head noun in the _some_-NP received lower predictions than sentences with an unmodified _some_-NP. All these results provide evidence that the model learned the correct associations. This is particularly remarkable considering the train-test mismatch: the model was trained on noisy transcripts of spoken language that contained many disfluencies and repairs, and was subsequently tested on clean written sentences.

**Regression analysis.** In the minimal pair analysis above we only investigated model predictions for three factors. As a second analysis, we therefore investigated whether the predictions of the best neural network model explain the variance explained by the linguistic features that modulate inference strength. To this end, we used a slightly simplified4 Bayesian implementation of the mixed-effects model by Degen2015 that predicted inference strength ratings from hand-mined features. We used the brms Burkner2017 and STAN Carpenter2017 packages and compared this _original_ model to an _extended_ model that included both all of the predictors of the original model as well as the the output of the above NN model as a predictor. For this comparison, we investigated whether the magnitude of a predictor in the _original_ model significantly decreased in the _extended_ model with the NN predictor, based on the reasoning that if the NN predictions explain the variance previously explained by these manually coded pre

Figure 3: Average model predictions on manually constructed sentences, grouped by presence of partitives, by grammatical function of the _some_-NP, and by presence of nominal modifiers. Semi-transparent dots show predictions on individual sentences.

 

dictors, then the original predictor should explain no or less additional variance.

We approximated the probability that the magnitude of the coefficient for the predictor \(i\) (\(\beta_{i}\)) in the _extended_ model including the NN predictor is smaller than the coefficient in the _original_ model, \(P(|\beta^{extended}_{i}|<|\beta^{original}_{i}|)\), by sampling values for each coefficient from the distributions of the _original_ and the _extended_ models and comparing the magnitude of the sampled coefficients. We repeated this process 1,000,000 times and treated the simulated proportions as approximate probabilities.

An issue with this analysis is that estimating the regression model only on the items in the held-out test set yields very wide credible intervals for some of the predictors-in particular for some of the interactions-since the model infers these values from very little data. We therefore performed this regression analysis (and the subsequent analyses) on the entire data. However, while we estimated the regression coefficients from all the data, we crucially obtained the NN predictions through 6-fold cross-validation (without additional tuning of hyperparameters), so that the NN model always made predictions on data that it had not seen during training. This did yield the same qualitative results as the analyses only performed on the held-out test items (see Appendix B) but it also provided us with narrower credible intervals that highlight the differences between the coefficient estimates of the two models.

Figure 4 shows the estimates of the coefficients in the original model and the extended model. We find that the NN predictions explain some or all of the variance originally explained by many of the manually coded linguistic features: the estimated magnitude of the predictors for partitive, determiner strength, linguistic mention, subjecthood, modification, utterance length, and two of the interaction terms decreased in the extended model. These results provide additional evidence that the NN model indeed learned associations between linguistic features and inference strength rather than only explaining variance caused by individual items. This is particularly true for the grammatical and lexical features; we find that the NN predictor explains most of the variance originally explained by the partitive, subjecthood, and modification predictors. More surprisingly, the NN predictions also explain a lot of the variance originally explained by the determiner strength predictor, which was empirically determined by probing human interpretation and is not encoded explicitly in the surface form utterance.5 One potential explanation for this is that strong and weak _some_ have different context distributions. For instance, weak _some_ occurs in existential _there_ constructions and with individual-level predicates, whereas strong _some_ tends not to (Milsark, 1974; McNally and Geenhoven, 1998; Carlson, 1977). Since pre-trained word embedding models capture a lot of distributional information, the NN model is presumably able to learn this association.

Footnote 5: As explained above, Degen (2015) obtained strength ratings by asking participants to rate the similarity of the original utterance and an utterance without the determiner _some (of)_.

Figure 4: Maximum a posteriori estimates and 95%-credible intervals of coefficients for original and extended Bayesian mixed-effects regression models predicting the inference strength ratings. */*** indicate that the probability of the coefficient of the original model having a larger magnitude than the coefficient of the extended model is less than 0.05, 0.01, and 0.001, respectively.

 

Attention weight analysis.As a final type of analysis, we analyzed the attention weights that the model used for combining the token embeddings to a sentence embedding. Attention weight analyses have been successfully used for inspecting and debugging model decisions (e.g., Lee et al., 2017; Ding et al., 2017; Wiegreffe and Pinter, 2019; Vashishth et al., 2019; but see Serrano and Smith, 2019, and Jain and Wallace, 2019, for critical discussions of this approach). Based on these results, we expected the model to attend more to tokens that are relevant for making predictions.6 Given that many of the hand-mined features that predict inference strength occur within or in the vicinity of the _some_-NP, we should therefore expect the model to attend most to the _some_-NP.

Footnote 6: As pointed out by one of the reviewers, given the transformer architecture, BERT token representations are influenced by numerous tokens of the input sentence and therefore it could be that the output representation of the \(i\)-th token ultimately contains very little information about the \(i\)-th token that was input to the model. Consequently, it could be that the attention weights do not provide information about which tokens the model attends to. To rule out this possibility, we also conducted the attention weight analysis for the model using static GloVe embeddings, which always exclusively represent the input token, and we found the same qualitative patterns as reported in this section, suggesting that the attention weights provide information about the tokens that are most informative for making predictions. Nevertheless, we do want to caution researchers from blindly trusting attention weight analyses and recommend using this type of analysis only in combination with other types of analyses as we have done in this work.

To test this, we first explored whether the model attended on average more to _some_ than to other tokens in the same position. Further, we exploited the fact that in English, subjects generally occur early in a sentence. If the model attends to the vicinity of the _some_-NP, the average attention weights should be higher at early positions in utterances with a subject _some_-NP compared to utterances with a non-subject _some_-NP, and conversely for late utterance positions. We thus compared the average attention weights for each position across utterances with subject versus non-subject _some_-NPs. To make sure that any effects were not only driven by the attention weight of the _some_-tokens, we set the attention weights of the token corresponding to _some_ to \(0\) and re-normalized the attention weights for this analysis. Further, since the attention weights are dependent on the number of tokens in the utterance, it is crucial that the average utterance length across the two compared groups be matched. We addressed this by removing outliers and limiting our analysis to utterances up to length 30 (1,028 utterances), which incidentally equalized the number of tokens across the two groups. These exclusions resulted in tiny differences in the average attention weights, but the qualitative patterns are not affected.

The left panel of Figure 5 shows the average attention weight by position for _some_ versus other tokens. The model assigns much higher weight to _some_. The center panel of Figure 5 shows the average attention weight by position for subject vs. non-subject _some_-NP utterances. The attention weights are generally higher for tokens early in the utterance,7 but the attention weights of utterances with a subject _some_-NP are on average higher for tokens early in the utterance compared to utterances with the _some_-NP in non-subject positions. Both of these findings provide evidence that the model assigns high weight to the tokens within and

Figure 5: **Left: Average attention weights at each token position for _some_ and other tokens. Center: Average attention weights at each token position for utterances with subject and non-subject _some_-NPs. Right: Average attention weights of _of_–tokens in parity _some_-NPs and weights of other _of_-tokens. In the normalized cases, we take only the utterances with multiple _of_-tokens into account and re-normalize the attention weights across all _of_-tokens in one utterance. Error bars indicate 95% bootstrapped confidence intervals.**



surrounding the _some-_NP.8

Footnote 7: The regression analysis suggests that the model learned to make use of the subieothood feature and previous work on probing behavior of contextual word representations has found that such models are capable of predicting dependency labels, including subjects (e.g., Liu et al., 2019). We therefore also hypothesize that the representations of tokens that are part of a subject _some_-NP contain information about the subjethood status. This in return could be an important feature for the output layer of the model and therefore be providing additional signal for the model to attend to these tokens.

In a more targeted analysis to assess whether the model learned to use the partitive feature, we examined whether the model assigned higher attention to the preposition _of_ in partitive _some_-NPs compared to when _of_ occurred elsewhere. As utterance length was again a potential confound, we conducted the analysis separately on the full set of utterances with raw attention weights and on a subset that included only utterances with at least two instances of _of_ (128 utterances), in which we renormalized the weights of _of_-tokens to sum to 1.

Results are shown in the right panel of Figure 5. The attention weights were higher for _of_ tokens in partitive _some_-NPs, suggesting that the model learned an association between partitive _of_ in _some_-NPs and inference strength.

## 6 Context, revisited

In the tuning experiments above, we found that including the preceding conversational context in the input to the model did not improve or lowered prediction accuracy.9 At the same time, we found that the model is capable of making accurate predictions in most cases without taking the preceding context into account. Taken together, these results suggest either that the conversational context is not necessary and one can draw inferences from the target utterance alone, or that the model does not make adequate use of the preceding context.

Footnote 9: As suggested by a reviewer, we conducted post-hoc experiments in which we limited the conversational context to the preceding 2 or 5 utterances, which presumably have a higher signal-to-noise ratio than a larger conversational context of 10 preceding utterances. In these experiments, we again found that including the conversational context did not improve model predictions.

Degen (2015) did not systematically investigate whether the preceding conversational context was used by participants judging inference strength. To assess the extent to which the preceding context in this dataset affects inference strength, we re-ran her experiment, but without presenting participants with the preceding conversational context. We recruited 680 participants on Mechanical Turk who each judged 20 or 22 items, yielding 10 judgments per item. If the context is irrelevant for drawing inferences, then mean inference strength ratings should be very similar across the two experiments, suggesting the model may have rightly learned to not utilize the context. If the presence of context affects inference strength, ratings should differ across experiments, suggesting that the model's method of integrating context is ill-suited to the task.

The new, no-context ratings correlated with the original ratings (\(r=0.68\), see Appendix D) but were overall more concentrated towards the center of the scale, suggesting that in many cases, participants who lacked information about the conversational context were unsure about the strength of the scalar inference. Since the original dataset exhibited more of a bi-modal distribution with fewer ratings at the center of the scale, this suggests that the broader conversational context contains important cues to scalar inferences.

For our model, these results suggest that the representation of the conversational context is inadequate, which highlights the need for more sophisticated representations of linguistic contexts beyond the target utterance.10 We further find that the model trained on the original dataset is worse at predicting the no-context ratings (\(r=0.66\)) than the original ratings (\(r=0.78\)), which is not surprising considering the imperfect correlation between ratings across experiments, but also provides additional evidence that participants indeed behaved differently in the two experiments.

Footnote 10: The representation of larger linguistic context is also important for span-based question-answer (QA) systems (e.g., Hermann et al., 2015; Chen, 2018; Devlin et al., 2019) and adapting methods from QA to predicting scalar inferences would be a promising extension of the current model.

## 7 Conclusion and future work

We showed that despite lacking specific pragmatic reasoning abilities, neural network-based sentence encoders are capable of harnessing the linguistic signal to learn to predict human inference strength ratings from _some_ to _not all_ with high accuracy. Further, several model behavior analyses provided consistent evidence that the model learned associations between previously established linguistic features and the strength of scalar inferences.

In an analysis of the contribution of the conversational context, we found that humans make use of the preceding context whereas the models we considered failed to do so adequately. Considering the 

importance of context in drawing both scalar and other inferences in communication (Grice, 1975; Clark, 1992; Bonnefon et al., 2009; Zondervan, 2010; Bergen and Grodner, 2012; Goodman and Stuhlmuller, 2013; Degen et al., 2015), the development of appropriate representations of larger context is an exciting avenue for future research.

We also only considered the supervised setting in which the model was trained to predict inference strength. It would be interesting to investigate how much supervision is necessary and, for example, to what extent a model trained to perform another task such as predicting natural language inferences is able to predict scalar inferences (see Jiang and de Marneffe (2019b) for such an evaluation of predicting speaker commitment, and Jeretic et al. (2020) for an evaluation of different NLI models for predicting lexically triggered scalar inferences).

One further interesting line of research would be to extend this work to other pragmatic inferences. Recent experimental work has shown that inference strength is variable across scale and inference type (Doran et al., 2012; van Tiel et al., 2016). We treated _some_ as a case study in this work, but none of our modeling decisions are specific to _some_. It would be straightforward to train similar models for other types of inferences.

Lastly, the fact that the attention weights provided insights into the model's decisions suggests possibilities for using neural network models for developing more precise theories of pragmatic language use. Our goal here was to investigate whether neural networks can learn associations for already established linguistic features but it would be equally interesting to investigate whether such models could be used to discover new features, which could then be verified in experimental and corpus work, potentially providing a model-driven approach to experimental and formal pragmatics.

## Acknowledgements

We thank the anonymous reviewers for their thoughtful feedback. We also gratefully acknowledge Leyla Kursat for collecting the no-context inference strength ratings, and we thank Jesse Mu, Shyamal Buch, Peng Qi, Marie-Catherine de Marneffe, Tal Linzen, and the members of the ALPS lab and the JHU Computational Psycholinguistics group for helpful discussions.

## References

* Andreas and Klein (2016) Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP 2016)_, pages 1173-1182.
* Barwise and Cooper (1981) Jon Barwise and Robin Cooper. 1981. Generalized quantifiers and natural language. _Linguistics and Philosophy_, 4(2):159-219.
* Bergen and Grodner (2012) Leon Bergen and Daniel J. Grodner. 2012. Speaker knowledge influences the comprehension of pragmatic inferences. _Journal of Experimental Psychology: Learning, Memory, and Cognition_, 38(5):1450-60.
* Bonnefon et al. (2009) Jean-Francois Bonnefon, Aidan Feeney, and Gaelle Villejoubert. 2009. When some is actually all: Scalar inferences in face-threatening contexts. _Cognition_, 112(2):249-258.
* Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015)_, pages 632-642.
* Burkner (2017) Paul-Christian Burkner. 2017. brms: An R package for Bayesian multilevel models using Stan. _Journal of Statistical Software_, 80(1):1-28.
* Carlson (1977) Greg N. Carlson. 1977. A unified analysis of the english bare plural. _Linguistics and Philosophy_, 1(3):413-456.
* Carpenter et al. (2017) Bob Carpenter, Andrew Gelman, Matthew D. Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. Stan: A probabilistic programming language. _Journal of Statistical Software_, 76(1).
* Chen (2018) Danqi Chen. 2018. _Neural Reading Comprehension and Beyond_. Ph.D. thesis, Stanford University.
* Chowdhury and Zamparelli (2018) Shammur Absar Chowdhury and Roberto Zamparelli. 2018. RNN simulations of grammaticality judgments on long-distance dependencies. In _Proceedings of the 27th International Conference on Computational Linguistics (COLING 2018)_, pages 133-144.
* Clark (1992) Herbert H Clark. 1992. _Arenas of language use_. University of Chicago Press.
* Cohn-Gordon et al. (2018) Reuben Cohn-Gordon, Noah D. Goodman, and Christopher Potts. 2018. Pragmatically informative image captioning with character-level inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2018)_, pages 439-443 

Alexis Conneau, Guillaume Lample, Ruty Rinott, Holger Schwenk, Ves Stoyanov, Adina Williams, and Samuel R. Bowman. 2018. XNLI: Evaluating co-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018)_, pages 2475-2485.
* Degen [Degen2015] Judith Degen. 2015. Investigating the distribution of some (but not all) implicatures using corpora and web-based methods. _Semantics and Pragmatics_, 8:11-1.
* Degen et al. [2015] Judith Degen, Michael Henry Tessler, and Noah D. Goodman. 2015. Wonky worlds: Listeners revise world knowledge when utterances are odd. In _Proceedings of the 37th Annual Conference of the Cognitive Science Society (CogSci 2015)_, pages 548-553.
* Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2019)_, pages 4171-4186.
* Ding et al. [2017] Yanzhuo Ding, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Visualizing and understanding neural machine translation. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017)_, pages 1150-1159.
* Doran et al. [2012] Ryan Doran, Gregory Ward, Meredith Larson, Yaron McNabb, and Rachel E. Baker. 2012. A novel experimental paradigm for distinguishing between what is said and what is implicated. _Language_, 88(1):124-154.
* Elman [1990] Jeffrey L. Elman. 1990. Finding structure in time. _Cognitive Science_, 14(2):179-211.
* Ettinger et al. [2018] Allyson Ettinger, Ahmed Elgohary, Colin Phillips, and Philip Resnik. 2018. Assessing composition in sentence vector representations. In _Proceedings of the 27th International Conference on Computational Linguistics (COLING 2018)_, pages 1790-1801.
* Franke and Jager [2016] Michael Franke and Gerhard Jager. 2016. Probabilistic pragmatics, or why Bayes' rule is probably important for pragmatics. _Zeitschrift fur Sprachwissenschaft_, 35(1):3-44.
* Futrell et al. [2019] Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2019)_, pages 32-42.
* Gardner et al. [2018] Matt Gardner, Joel Grus, Mark Neumann, Oyvind Talfjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In _Proceedings of Workshop for NLP Open Source Software (NLP-OSS)_.
* Gazdar [1979] Gerald Gazdar. 1979. _Pragmatics: Implicature, Presupposition and Logical Form_. Academic Press.
* Godfrey et al. [1992] John J. Godfrey, Edward C. Holliman, and Jane McDaniel. 1992. SWITCHBOARD: Telephone speech corpus for research and development. In _Proceedings of the 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 92)_.
* Goodman and Frank [2016] Noah D. Goodman and Michael C. Frank. 2016. Parametric language interpretation as probabilistic inference. _Trends in Cognitive Sciences_, 20(11):818-829.
* Goodman and Stuhlmuller [2013] Noah D. Goodman and Andreas Stuhlmuller. 2013. Knowledge and implicature: Modeling language understanding as social cognition. _Topics in Cognitive Science_, 5(1):173-184.
* Price [1975] Herbert P. Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, _Syntax and Semantics, Vol. 3, Speech Acts_, pages 41-58. Academic Press, New York.
* Gulordava et al. [2018] Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2018)_, pages 1195-1205.
* Hermann et al. [2015] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In _Advances in Neural Information Processing Systems 28 (NeurIPS 2015)_.
* Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Horn [1997] Laurence Horn. 1997. All John's children are as bald as the king of France: Existential import and the geometry of opposition. In _CLS 33_, pages 155-179.
* Horn [1984] Laurence R. Horn. 1984. Toward a new taxonomy for pragmatic inference: Q-based and R-based implicature. In Deborah Schiffrin, editor, _Meaning, Form, and Use in Context: Linguistic Applications_, pages 11-4. Georgetown University Press, Washington, D.C.
* Jain and Wallace [2019] Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2019)_, pages 3543-3556.

 

Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESive? Learning IMPLature and PRESupposition. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020)_.
* Jiang and de Marneffe (2019a) Nanjiang Jiang and Marie-Catherine de Marneffe. 2019a. Do you know that florence is packed with visitors? evaluating state-of-the-art models of speaker commitment. In _Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL 2019)_, pages 4208-4213.
* Jiang and de Marneffe (2019b) Nanjiang Jiang and Marie-Catherine de Marneffe. 2019b. Evaluating BERT for natural language inference: A case study on the CommitmentBank. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 6085-6090.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _Proceedings of the 5th International Conference on Learning Representations (ICLR 2015)_.
* Lee et al. (2017) Jaesong Lee, Joong-Hwi Shin, and Jun-Seok Kim. 2017. Interactive visualization and manipulation of attention-based neural machine translation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 121-126.
* Levinson (2000) Stephen C. Levinson. 2000. _Presumptive meanings: The theory of generalized conversational implicature_. MIT press.
* Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In _Proceedings of the 5th International Conference on Learning Representations (ICLR 2017)_.
* Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. _Transactions of the Association for Computational Linguistics_, 4:521-535.
* Liu et al. (2019) Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In _Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2019)_, pages 1073-1094.
* Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018)_, pages 1192-1202.
* McNally and Van Geenhoven (1998) Louise McNally and Veerle Van Geenhoven. 1998. Refining the weak/strong distinction.
* Milsark (1974) Gary Milsark. 1974. _Existential sentences in English_. Ph.D. thesis, MIT.
* Monroe et al. (2017) Will Monroe, Robert X.D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. _Transactions of the Association for Computational Linguistics_, 5:325-338.
* Paszke et al. (2017) Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In _NeurIPS Autodiff Workshop 2017_.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNL 2014)_, pages 1532-1543.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2018)_, pages 2227-2237.
* Rudinger et al. (2018) Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2018)_, pages 731-744.
* Serrano and Smith (2019) Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019)_, pages 2931-2951.
* Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. _The Journal of Machine Learning Research_, 15(1):1929-1958.
* van Tiel et al. (2016) Bob van Tiel, Emiel van Miltenburg, Natalia Zevakhnia, and Bart Geurts. 2016. Scalar diversity. _Journal of Semantics_, 33(1):137-175.
* Vashishth et al. (2019) Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui. 2019. Attention interpretability across NLP tasks. _arXiv preprint arXiv:1909.11218_.
* Warstadt et al. (2019a) Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yinqing Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Puh Mon Hutt, Paloma Jeretic, and Samuel R. Bowman. 2019a. Investigating BERT's knowledge of language: Five analysis methods with NPIs. In _Proceedings of the 

2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019)_, pages 2877-2887.
* Warstadt et al. [2019b] Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019b. Neural network acceptability judgments. _Transactions of the Association for Computational Linguistics_, 7:625-641.
* Wiegreffe and Pinter [2019] Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019)_, pages 11-20.
* Wilcox et al. [2019] Ethan Wilcox, Roger Levy, and Richard Futrell. 2019. What syntactic structures block dependencies in RNN language models? In _Proceedings of the 41st Annual Meeting of the Cognitive Science Society (CogSci 2019)_.
* Wolf et al. [2019] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. _arXiv preprint arXiv:1910.03771_.
* Zondervan [2010] Arjen Zondervan. 2010. _Scalar Implicatures or Focus: An Experimental Approach_. LOT Publications, Utrecht.

 

Hyperparameter tuning

Figure 6 shows the learning curves averaged over the 5 cross-validation tuning runs for models using different word embeddings. As these plots show, the attention layer improves predictions; contextual word embeddings lead to better results than the static GloVe embeddings; and including the conversational context does not improve predictions and in some cases even lowers prediction accuracy.

## Appendix B Regression analysis on held-out test data

Figure 7 shows the estimates of the predictors in the original and extended Bayesian mixed-effects models estimated only on the held-out test data. We find the same qualitative effects as in Figure 4, but since these models were estimated on much less data (only 408 items), there is a lot of uncertainty in the estimates and therefore quantitative comparisons between the coefficients of the different models are less informative.

Figure 6: Correlation between each model’s predictions on valuation set and empirical means, by training epoch.

Figure 7: Maximum a posteriori estimates and 95%-credible intervals of coefficients for original and extended Bayesian mixed-effects regression models predicting the inference strength ratings on the held-out test set. */**/*** indicate that the probability of the coefficient of the original model having a larger magnitude than the coefficient of the extended model is less than 0.05, 0.01, and 0.001, respectively.

 

List of manually constructed sentences

Tables 1 and 2 show the 25 manually created sentences for the analyses described in the minimal pairs analysis in Section 5. As described in the main text, we created 16 variants of the sentence with the _some_-NP in subject position (sentences in the left column), and 16 variants of the sentence with the _some_-NP in object position (sentences in the right column), yielding in total 800 examples.

 



\begin{table}
\begin{tabular}{|p{142.3pt}|p{142.3pt}|p{142.3pt}|} \hline Some of the attentive waiters at the gallery opening poured the white wine that my friend really likes. & The attentive waiters at the gallery opening poured some of the white wine that my friend really likes. \\ \hline Some of the experienced lawyers in the firm negotiated the important terms of the acquisition. & The experienced lawyers in the firm negotiated some of the important terms of the acquisition. \\ \hline Some of the award-winning chefs at the sushi restaurant cut the red salmon from Alaska. & The award-winning chefs at the sushi restaurant cut some of the red salmon from Alaska. \\ \hline Some of the brave soldiers who were conducting the midnight raid warned the decorated generals who had served in a previous battle. & The brake soldiers who were conducting the midnight raid warned some of the decorated generals who had served in a previous battle. \\ \hline Some of the eccentric scholars from the local college returned the old books written by Camus. & The eccentric scholars from the local college returned some of the old books written by Camus. \\ \hline Some of the entertaining magicians with top hats shuffled the black cards with dots. & The entertaining magicians with top hats shuffled some of the black cards with dots. \\ \hline Some of the convicted doctors from New York called the former patients with epilepsy. & The convicted doctors from New York called some of the former patients with epilepsy. \\ \hline Some of the popular artists with multiple albums performed the fast songs from their first album. & The popular artists with multiple albums performed some of the fast songs from their first album. \\ \hline Some of the angry senators from red states impeached the corrupt presidents from the Republican party. & The angry senators from red states impeached some of the corrupt presidents from the Republic can party. \\ \hline Some of the underfunded researchers without permanent employment transcribed the recorded conversations that they collected while doing fieldwork. & The underfunded researchers without permanent employment transcribed some of the recorded conversations that they collected while doing fieldwork. \\ \hline Some of the sharp psychoanalysts in training hypnotized the young clients with depression. & The sharp psychoanalysts in training hypnoticized some of the young clients with depression. \\ \hline Some of the harsh critics from the Washington Post read the early chapters of the novel. & The harsh critics from the Washington Post read some of the early chapters of the novel. \\ \hline Some of the organic farmers in the mountains milked the brown goats who graze on the meadows. & The organic farmers in the mountains milked some of the brown goats who graze on the meadows. \\ \hline Some of the artisanal bakers who completed an apprenticeship in France kneaded the gluten-free dough made out of spelt. & The artisanal bakers who completed an apprenticeship in France kneaded some of the gluten-free dough made out of spelt. \\ \hline Some of the violent innates in the high-security prison reported the sleazy guards with a history of rule violations. & The violent innates in the high-security prison reported some of the sleazy guards with a history of rule violations. \\ \hline \end{tabular}
\end{table}
Table 1: Manually constructed sentences used in the minimal pair analyses. Sentences in the left column have a _some_-NP in subject position; sentences on the right have a _some_-NP object position.



Some of the eager managers in the company instructed the hard-working sales representatives in the steel division about the new project management tool.

The eager managers in the company instructed some of the hard-working sales representatives in the steel division about the new project management tool.

The billiard chemists in the lab oxidized the shiny metals extracted from ores.

The adventurous pirates on the boat found the valuable treasure that had been buried in the sand.

The mischievous con artists at the casino tricked the elderly residents of the retirement home.

The observutrous pirates on the boat found some of the valuable treasure that had been buried in the sand.

The mischievous con artists at the casino tricked the elderly residents of the retirement home.

The persistent recruiters at the conference hired the smart graduate students who just started a PhD as interns.

The established professors in the department supported the controversial petitions that were drafted by the student union.

The present lecturers at the conference hired the startup loaded the adjustable standing desks made out of oak onto the truck.

The unscular movers that were hired by the startup loaded some of the adjustable standing desks made out of oak onto the truck.

Some of the careful secretaries at the headquarter mailed the confidential envelopes with the bank statements.

The careful secretaries at the headquarter mailed some of the confidential envelopes with the bank statements.

The international stations in South America television the early games of the soccer cup.

The wealthy investors of the fund excessively reinventerated the successful brokers working at the large bank.

\begin{table}
\begin{tabular}{|p{142.3pt}|p{190.7pt}|} \hline Some of the enger managers in the company instructed the hard-working sales representatives in the steel division about the new project management tool. & The eager managers in the company instructed some of the hard-working sales representatives in the steel division about the new project management tool. \\ \hline Some of the brilliant chemists in the lab oxidized the shiny metals extracted from ores. & The brilliant chemists in the lab oxidized some of the shiny metals extracted from ores. \\ \hline Some of the adventurous pirates on the boat found the valuable treasure that had been buried in the sand. & The adventurous pirates on the boat found some of the valuable treasure that had been buried in the sand. \\ \hline Some of the mischievous con artists at the casino tricked the elderly residents of the retirement home. & The mischievous con artists at the casino tricked some of the elderly residents of the retirement home. \\ \hline Some of the persistent recruiters at the conference hired the smart graduate students who just started a PhD as interns. & The persistent recruiters at the conference hired some of the smart graduate students who just started a PhD as interns. \\ \hline Some of the established professors in the department supported the controversial petitions that were drafted by the student union. & The established professors in the department supported some of the controversial petitions that were drafted by the student union. \\ \hline Some of the muscular movers that were hired by the startup loaded the adjustable standing desks made out of oak onto the truck. & The muscular movers that were hired by the startup loaded some of the adjustable standing desks made out of oak onto the truck. \\ \hline Some of the careful secretaries at the headquarter mailed the confidential envelopes with the bank statements. & The careful secretaries at the headquarter mailed some of the confidential envelopes with the bank statements. \\ \hline Some of the international stations in South America televised the early games of the soccer cup. & The international stations in South America televised some of the early games of the soccer cup. \\ \hline Some of the wealthy investors of the fund excessively reinvenrated the successful brokers working at the large bank. & The wealthy investors of the fund excessively remunerated some of the successful brokers working at the large bank. \\ \hline \end{tabular}
\end{table}
Table 2: Manually constructed sentences used in the minimal pair analyses (continued).

 

Results from no-context experiment

Figure 8 shows the correlation between the mean inference strength ratings for each item in the experiment from Degen (2015) and the mean strength ratings from the new no-context experiment, discussed in Section 6.

## Appendix D

Figure 8: Mean inference strength ratings for items without context (new) against items with context (original), \(r=.68\).