Connecting degree and polarity: An artificial language learning study
Authors: Lisa Bylinina
Alexey Tikhonov
Ekaterina Garmash
Venue: EMNLP 2023 Main (2023)
Keywords: semantics
degree
polarity
artificial language learning

Abstract: We investigate a new linguistic generalisation in pre-trained language models (taking BERT Devlin et al. 2019 as a case study). We focus on degree modifiers (expressions like slightly, very, rather, extremely) and test the hypothesis that the degree expressed by a modifier (low, medium or high degree) is related to the modifier’s sensitivity to sentence polarity (whether it shows preference for affirmative or negative sentences or neither). To probe this connection, we apply the Artificial Language Learning experimental paradigm from psycholinguistics to a neural language model. Our experimental results suggest that BERT generalizes in line with existing linguistic observations that relate de- gree semantics to polarity sensitivity, including the main one: low degree semantics is associated with preference towards positive polarity.

Connecting degree and polarity: An artificial language learning study
Lisa Bylinina University of Groningen e. g. wyw.iinairuug........ nl
Alexey Tikhonov Inworld.AI, Germany
Ekaterina Garmash
Spotify, United Kingdom* a altsophegmaill com yat ga harmashhe gma com
Abstract
In this paper, we study one such connection: the problem of polarity sensitivity of degree modifiers (Isracl, 1996, 2011; Solt, 2018; Solt and Wilson, 2021). Degree modifiers are words like slightly, very, and extremely. Property A, in this case, is the degree that these words convey, defined on interval from very low to very high. For example, the degree of slightly is lower than the one of very. Property B here encodes distributional preferences of degree modifiers with respect to polarity of a sentence where they appear roughly, whether they appear exclusively in negative or affirmative sen- tences, or show no polarity preference. Polarity preferences can also be represented as a continuous property from very low (negative polarity prefer- ence) to very high (positive polarity preference), with polarity-neutral in the middle.
We investigate a new linguistic generalisation in pre-trained language models (taking BERT Devlin et al. 2019 as case study). We focus on degree modifiers (expressions like slightly, very, rather, extremely) and test the hypothesis that the degree expressed by modifier (low, medium or high degree) is related to the modi- fier's 's sensitivity to sentence polarity (whether it shows preference for affirmative or negative sentences neither). To probe this connection, we apply the Artificial Language Learning ex- perimental paradigm from psycholinguistics to neural language model. Our experimental re- sults suggest that BERT generalizes in line with existing linguistic observations that relate de- gree semantics polarity sensitivity, including the main one: low degree semantics is associ- ated with preference towards positive polarity.
Interactions between linguistic properties have been subject to experimental studies in psycholin- guistics and cognitive science. One prominent method is Artificial Language Learning (Friederici et al., 2002; Motamedi et al., 2019; Kanwal et al.. 2017; Culbertson et al., 2012; Ettlinger et al., 2014; Finley and Badecker, 2009). It has the following main ingredients:
1 Introduction
Linguistic expressions can be characterized along a variety properties: what they mean, what parts they consist of, how they combine with other ex- pressions and so on. Some of these properties are systematically related to each other. When these relations appear systematically in language after language, they can be grounds for implicational lin- guistic universals, for example, Greenberg's 's Uni- versal 37: A language never has more gender cate- gories nonsingular numbers than in the singular. (Greenberg, 1963). Here, two propertics of linguis- tic expressions are related: the grammatical number of an expression and how many gender distinctions are available for this expression. More complex generalizations may concern correlation between continuous properties A and B.
fragment of an artificial language in the form of expressions that do not belong to the language that participants are speakers of:
2. training phase, where some information about the language fragment given to the participants:
testing phase, where checked what other knowledge, beside the provided, was inferred during training.
In order to arrive at linguistic universals connect- ing A and B, the relation between these proper- ties has to be established at the level of individual languages, which not trivial for properties with complex internal structure.
Originally designed for studies with human par- ticipants, the Artificial Language Learning frame- work has also been applied to neural network-based learning models (Piantadosi et al., 2012; Carcassi et al., 2019; van de Pol et al., 2021). Replacing
Work was performed previously while at Huawei.

---

human participants with artificial learning agents allows examine the learning process in more de- tail and to make a variety of learnability statements. One important property of these experiments is that the learning agents typically come in a blank state with no prior knowledge or biases. This limits the set of linguistic questions that can be targeted by this type of experiment.
explore generalization between complex lin- guistic properties;
We use this methodology to explore the rela- tion between two linguistic phenomena, de- gree and polarity sensitivity, as represented in pre-trained LM (BERT).
We argue that, according to the experimental results, the LM in question indeed makes a connection between the degree encoded by a degree modifier and its polarity sensitivity.
The way we use the Artificial Language Learn- ing paradigm can be seen middle ground be- tween experiments with human participants and with artificial learners described above. Our ap- proach also involves an artificial language fragment and training procedure to introduce knowledge about some property A. but it uses pre-trained language model (LM) (Peters et al., 2018; Devlin al., 2019; Brown al.. 2020) the learning agent. More technically, we extend a pre-trained LM with a set of new tokens with randomly ini- tialized embeddings and perform fine-tuning on a carefully constructed synthetic dataset. The dataset is constructed in way to indirectly introduce dif- ferent values along property A for different new to- kens. Upon fine-tuning, we measure how the train- ing affected property B and how variation along depends on the values of property A introduced during training.
The paper is structured follows: gives linguistic background about degrees and po- larity. Section 3 describes the general method. Section 4, we define a synthetic dataset and the measures we use to estimate degree and polarity. Section presents the experiment. Section 6 dis- cusses our results, the limitations of our set-up and suggestions for future work.
Section 2
Background: Degree and polarity
In this section we provide background on the stud- ied linguistic properties: we describe degree as a property of degree modifiers, and polarity sensi- tivity as property of linguistic items (words) that tend to appear in certain types of contexts. We outline the relation between these two properties, as discussed in theoretical linguistic literature. We will apply our proposed method to experimentally verify the hypothesised relation.
Our study will focus on English, as represented in pre-trained LM BERT (Devlin et al., 2019). We see this as a proof of concept work that can be extended further along the cross-linguistic dimen- sion and in application to other models. Using this set-up, address the question of whether LMs encode a connection between degree semantics and polarity, therefore making generalization across two different linguistic properties.
Degree
So-called gradable adjectives describe properties that can hold to a different degree. A classic exam- ple of a gradable adjective is tall. A classic example of a non-gradable one is prime. The former, as op- posed to the latter, can be part of comparative and superlative constructions, and they combine with degree modifiers: words like slightly, very, and extremely. Examples (1)-(2) illustrate this dif- ference. We use denote all types of linguistic deviancy, including ungrammaticality as well as semantic/ pragmatic oddity:
Our approach belongs to the general area of studies using counterfactual linguistic data in NLP (Kaushik et al. 2020, 2021 a.o.); part of that sub- field that uses novel lexicon is the closest to the present paper (Thrush et al. 2020; Bylinina and Tikhonov 2022a a.o.). Our work contributes the general agenda of establishing closer connections between learning in humans and artificial neural models (Futrell et al. 2019; Wilcox et al. 2020 a.o.). The main step forward that we make with this paper is the extension of these methods to linguistic prop- erties that have complex internal structure, rather than clear morphological syntactic exponence. To sum up, we make the following contributions:
*7 is more prime than 3.
13 is the most prime number in this set. is somewhat / very / extremely prime.
Mary taller than John.
Mary the tallest person in this room. Mary somewhat / very / extremely tall.
We propose an experimental methodology to

---

For statement with a simple base form of grad- able adjective like Mary tall to be true, the property in question has to hold of the subject to some degree that is determined by linguistic and extra-linguistic contextual factors (Fara, 2000; Kennedy and McNally, 2005; Kennedy, 2007). When gradable adjective appears in combination with a degree modifier, the degree ò that makes value that depends
sequent literature). However, we will focus on the presence or absence of negation as a proxy to po- larity in the current discussion.
Like degree, we will treat polarity sensitivity as continuous property that fits into the [0, 1] interval, where 0 is very pronounced NPI, is a very pronounced PPI, with polarity-neutral expressions in the middle.
the statement true changes on the modifier. For Mary count as 'somewhat tall', her height needs to be much lower than for "extremely tall', for instance. The requirements on 8 that degree modifiers encode can be used to or- der these modifiers along a scale of degrees, for example, somewhat < extremely.
Relation between the two properties
Observations reported in linguistic literature sug- gest interaction between these two properties (Israel, 1996, 2011: Solt, 2018; Solt and Wilson, 2021). The interaction in many cases far from straightforward, but there are clear tendencies. Specifically, lower degrees associate with PPI be- haviour. English degree modifiers support this ob- servation (Solt and Wilson, 2021), as examples in (5) demonstrate. This pattern is found in other languages too (van Os, 1989; Nouwen, 2013; Ito, 2015). We will refer to modifiers of this lower de- gree range as Class v1 (low degree modifiers) later on.
In our experiments, we will map degree mod- ifiers to dense interval from 0 to I according their degree semantics (from very low to very high).
Polarity sensitivity
For certain expressions, their acceptability and/or interpretation in a context is conditioned on the polarity of this context. Expressions with distribu- tional preference' for negative contexts are called negative polarity items (NPIs). Expressions with preference towards positive contexts are called pos- itive polarity items (PPIs). For example, any is an NPI (3), while already is PPI (4). NPIs and PPIs are said be polarity-sensitive. Like degree, we treat polarity sensitivity as a continuous property on the [0,1] interval, where is a very pronounced NPI, a very pronounced PPI, with polarity- items in the middle.
(5)
The issue is fairly kind of / sort of important. "The issue isn't 't fairly somewhat / rather / kind of sort of important.
/ somewhat rather
Modifiers in the moderate range (Class v2, medium degree), to the contrary, show mild association with negative contexts (Israel, 1996). The association between negative polarity and degree modifiers from a certain range can most prominently be at- tributed to the phenomenon of 'negative strengthen- ing' (Gotzner et al., 2018; Mazzarella and Gotzner, 2021):
(3) Mary bought any books. Mary didn't buy any books.
NPI
(4) John has arrived already. John hasn't 't arrived already.
PPI
(6)
John isn't particularly smart.
While the literal meaning of (6) is compatible with John being smart quite often these types of sen- tences are used to convey stronger meaning: that John is not smart all. This is pragmatic asym- metry rather than a distributional constraint, but contributes to the interaction patterns between degree and polarity sensitivity.
Sentences that are good contexts for NPIs and PPIs are said have negative and positive polarity, re- spectively. Polarity of a sentence does not amount simply to the presence or absence of sentential negation, it is a way more complex semantic prop- erty (see Fauconnier 1975; Ladusaw 1979 and sub-
'We use the vague and permissive term 'preference' here to cover the whole spectrum of asymmetries between positive and negative contexts that an expression shows - from ungram- maticality to decreased prominence of a narrow scope reading. Gradations of polarity sensitivity will play crucial role in our discussion, but specifically for this reason we are looking for a unified way to descrihe the whole space of polarity sensitivity phenomena.
Finally, modifiers expressing very high degree (Class v3, high degree modifiers). behave like PPIs again:
This coffee is damn crazy good. ?? This coffee isn't 't damn / crazy good.

---

Existing work proposes analyses of degree mod- ification with built-in causal connection between the degree semantics of modifiers and their polar- ity profile (Israel, 1996; Solt and Wilson, 2021) even though the extent, exact shape and direction of this connection is not established yet. We use this state of affairs chance to contribute to this discussion empirically and analytically, using the method proposed below.
approach presupposes set of contexts that distri- butionally select for specific linguistic property X, denoted S(X). We obtain such contexts by mining them from data, as described in Section 4.3. Part of future work is to extend the current method it to more general case. The general structure of the synthetic dataset used to fine-tune the LM is de- scribed in Section 4.1. The dataset is also tailored to the linguistic phenomenon under investigation.
3 Method
Dataset and measures
In this section, we describe the details of a method to conduct artificial language learning experiments with pretrained LMs. Without loss of generality, we use BERT in our experiments, but other pretrained language models could be used instead.
First, we delineate a fragment of English that will be the basis of our experiment (Section 4.1): simple sentences with gradable adjective predicated over definite noun phrase (as in The pizza is good). We re-shape these sentences create diagnostic contexts for properties and (Sections 4.2, 4.3). We also use to exemplify values of A during training (Section 4.3).
We design our method to be applied linguistic hypotheses of the form A + B, where A, B are some properties in a given language. In this study, we specifically focus on the relationship between adverbial degree modification and polarity sensi- tivity. A in this context is low, medium or high degree of an adverbial modifier w, and B is nega- tive, neutral positive polarity of w. In general, we evaluate a hypothesis A(w, i) + B( by showing that if A holds according to BERT for to- ken w to an extent i, then so does to some extent j, according to BERT.
4.1 Basic set of sentences
First, we automatically identified the set of gradable adjectives and nouns to build our training samples from. We started with bert-base-uncasee
vocabulary and signed all non-subword tokens a part of speech label with the SpaCy POS tagger". We kept the top 1000 nouns. Using the CapitolWords dataset from textacy: acy*, we looked for co-occurrences of adjectives with degree modifiers somewhat, very, really, extremely, rather and picked 200 adjectives with the highest ratio of modified uses.
We use the cloze test (a task where the partici- pant is asked to recover missing language item) adapted for BERT (see Warstadt et al. 2019; Bylin- ina and Tikhonov 2022b for the cloze test LMs for polarity). The test uses BERT's probability distributions over tokens in masked positions in diagnostic contexts for property or B.
Second, we generated sentences with these nouns and adjectives using the following pattern:
To show that a hypothesis holds in general for an arbitatrary w, we:
The noung cop. adjy
where cop. PRS is either singular or plural copula in the Present tense (is or are), noun, is one of the 1000 picked nouns, and adjy is one of the 200 gradable adjectives. The procedure gave us 400k sentences like these:
PRR.PRS
(1) augment BERT's vocabulary with a set W of new tokens and randomly initialize the corre- sponding embeddings;
(2) fine-tune the corresponding embeddings dataset where the new tokens appear in con- texts that distributionally select for particular values of A:
The purpose is interesting.
The answer is simple. The environment is large.
(3) test whether the knowledge that B holds was acquired, to the extent that follows the hypoth- esised association pattern with A.
"tttpss:/buugiiggaace co/ bext-bbase-uncased
As part of Step (1), we also verify that prior to training the initialized embeddings do not show any biases w.r.t. both properties A and B. This
pittps://githube com/ wwtp///////pcoo///////// spacy-models
htttps:://ithuu// bbww de/ textacy-data

---

This 400k set varied in terms of naturalness, coher- ence and adherence lexical selectional restric- tions. To control for this, we ran the sentences through GPT-2 and kept the bottom 10k accord- ing to the assigned sentence perplexity.
Previous applications of this estimation method has shown its reliability for NPI detection (Jumelet and Hupkes, 2018; Warstadt et al., 2019; Jumelet et al., 2021; ; Bylimina and Tikhonov, 2022b). As an illustration, slightly PPI), particularty gets a score of 0.1 (is an NPT), while incredibly is PPI again with score 0.94.
slighttl gets a score of 0.99 (= is a
The construction steps above aim to output 'nat- ural' examples, based on insights from different sources (GPT-2, BERT, corpus-based statistics). Manual inspection of the resulting 10k dataset re- vealed some sentences that still sound intuitively 'weird'. We do not see this as a problem though, since the majority of sentences are natural enough.
We use this polarity estimation method get a reliable list of degree modifiers with polarity scores. For each of the 10k sentence pairs, we pick 100 tokens with highest probability in the masked po- sition for a positive sentence and 100 tokens for its negative counterpart. Then we take two unions: one of all the "positive" tokens and one for the "negative" ones. We filter these two sets to only keep tokens that appear more than 100 times in one of them. We use the resulting sets in the rest of the experiment.
The large quantity of examples in our dataset is crucial to make our experiments comparable to psycholinguistic experiments. In the latter, one sentence gives rise to multiple of observations about (roughly) one linguistic system, due to judge- ments to multiple participants with similar enough intuitions. In our setting, we have only one agent (BERT), so we compensate by increasing the num- ber of sentences.
4.3 Estimating and mining degree
To estimate polarity of tokens (Section 4.2), we relied on their patterns of occurrence in positive and negative contexts. To apply an analogous pro- cedure to degree, we need contexts that associate with various degree semantics. We propose the fol- lowing intuition. What does answer yes/no- question with gradable adjective like Is the pizza good? - depend on? certainly depends on how good the pizza is: the degree to which the property applies to the subject. Given that degree modifiers express exactly that, we can make a connection be- tween their degree value and particles that answer the degree yes/no question.
4.2 Estimating polarity
To assign polarity scores to degree modifiers, we follow the procedure in (Warstadt et al., 2019; Bylinina and Tikhonov, 2022b). We use the 10k basic sentences (Section 4.1) to build a polarity contrast set. For each sentence in the basic set, pair of sentences, one positive and one negative, with the [MASK] token in the modifier position:
The noung cop. PRS [MASK] adig. The noun, cop PRS. PRS. NEG [MASK] adjy.
We end up with 10k pairs of sentences like these: (9)
For example, we expect particles to have differ- ent distribution in the masked position in (10) as an effect of the modifier:
The reason is [MASK] simple. The zeason isn't t. [MASK] simple.
We use the generated sentence set to estimate polar- ity sensitivity pol(m) of degree modifier using the probabilities that BERT assigns to each token in its vocabulary in the masked position:
(10) - Is the pizza good?
[MASK], it is somewhat good. -[MASKJ, it is extremely good.
ESCDb((MASKI = m/ maskd
PIIMASK) = m s mskdd ]
We use this idea to mine particles that are associ- ated with low and high degree. The mined particles can be used to assess degree of the modifiers, anal- ogously to polarity measurement above. As low degree modifiers, we use somewhat and slightly; for high degree, very and extremely. We modify each of the 10k sentences to generate pairs of sen- tences like these, where MOD is one of the four modifiers of interest:
PPOS
D
(1)
where D is the 10k dataset, smasked is a sentence spac
from the dataset in the positive form, with [MASK] masked is its negative counterpart. So, we approximate polarity as the proportion of cases where token m got a higher probability in pos than in neg context.
in the degree modifier position, and syeg
Amonn the tokens that survived the filter: very, always, quite, so, realfy, too, all, actually.
Phttos://hugginggfe http::////h/nnnggga....///////

---

(11) Is the question difficult? [MASK], it is KOD difficult.
10-
0.8
As before, rum the resulting (40k) sentences through BERT and, for each sentence, we collect the top 100 tokens according to the probability of tokens in the masked position. We only keep those tokens that appear in this list 100 times or more.
polarity
The particles in the resulting list are then tested their segree-diagnosing potential, as follows.
00-
0.2
0.3
0.4
0.5
0.6
0.7
0.8
degree
We use the same procedure as for polarity: for each particle, we check in what proportion of cases the probability that BERT assigns the particle in the sentence with the high degree modifier higher than with low degree modifier. We perform this comparison for each the four pairs of high vs. low degree modifiers: very vs. somewhat, very vs. slightly, extremely vs. somewhat, extremely vs. slightly. This procedure gives us value from O to for each particle from the list, depending on the extent which it is associated with low degrees (the closer the more this holds) or high degrees (closer to 1). We fix the final set of top 10 particles that associate with low (12) degrees and with high degrees (13):
Figure I: Degree and polarity of selected English modi- fiers. Estimated as described in Sections 4.2 and 4.3.
rather,fairty merely); same for high degree (com- pletely, totally, utterly, damn, bloody) (Paradis (1997); Bennett and Goodman (2018) a.o.). There's no overlap with modifiers we used as seeds for our degree measure. Also, these practically all mod- ifiers with an undisputed degree profile discussed in the literature that are also whole BERT tokens. Our measure assigns the low degree class an aver- age score of 0.22 (min 0.11; max 0.31); average of 0.56 for the high degree class (min 0.5; max 0.62). Very, which has some intensifying effect got score of 0.45, We conclude that the measure is decent, albeit somewhat shifted to the left.
(12)
well, actually, now, but, however, still, so, why, anyway, sure
Fig. shows the distribution of polarity sensitiv- ity and degree for these modifiers (we color-code them as moderate, medium and high degree). As the scatterplot and the fitted parabola show, the ex- isting data is compatible with what is hypothesised in the linguistic literature: low degrees associate with positive polarity, while the rest is more varied- mid-range degrees gravitate towards more negative polarity somewhat, while the higher range again gravitates towards PPI behaviour.
(13) yes, oh, sir, absolutely, god, damn, remember, wow, sericusly, man
Finally, we reverse the process and now use these particles to produce a degree score for degree mod- ifiers. For each of the 10k sentences, we modify it to get 20 sentences like the following (where PRT ranges over the 20 particles in (12) and (13)):
(14)
Is the question on difficult? PAT, it is [MASK] difficult.
4.4 Degree and polarity in BERT embeddings
Comparing modifier probabilities across conditions defined by the distinction in (12) and (13) as before, we get a measure defined on the [0,1] interval that corresponds to the modifier's degree.
We use diagnostic classifiers to analyse how polar- ity sensitivity and degree semantics are represented in BERT token embeddings for degree modifiers. Using embeddings of degree modifiers as features, we fit logistic regression with LI regularization to demote non-zero coefficients for two binary classi- fication tasks: 1) token classification into 'negative' (< .5) and 'positive' (> .5) with respect to polar- ity; 2) token classification into 'low degree' (< .1, based somewhat skewed score distribution) and "high degree' (> .4).
As final step, we manually cleaned the result- ing list of 415 tokens obtained from the (MASK) get rid of syntactic junk and items whose selec- tional restrictions are too narrow, to end up with the list of 98 degree modifiers we will further use?. To validate our degree measure, we take five modifiers about which the literature agrees they introduce low or moderate degree (barely, hardly,
On folds, average accuracy for polarity on train data is 79.2%, and 74.7% on test. For degree, it's 73% and 72.3%, respectively. For each of the tasks,
"Code and data are at nttps://gg / hub..com/ weteiiiaalldeeeeeeeeeeeeeeeeeeee deegree_mod fiers
al altsoph/artiiiie

---

we find the most important part of the embedding that is responsible for the distinction, by taking co- ordinates that have non-zero coefficients in at least four of the folds. We found 20 important coordi- nates for polarity and 13 for degree. There was no overlap between these coordinates, indicating no representational overlap between polarity and de- gree at the level of token embeddings. If it turns out that the model encodes the dependency between the two properties, it would be on a level other than embeddings directly.
Before training
After training
degrec
polarity
degree
polarity
0:48, 0.06
0.42, 0.24
0.18, 0.02
0.99, 0.03
V2
0.50, 0.06
0.43, 0.21
0.40, 0.02
0.00, 0.00
v3
0.48, 0.06
0.39, 0.18
0.83, 0.02
0.85, 0.26
Baselines
random
0.52, 0.06
0.38, 0.20
0,41, 0.09
0.83, 0.30
untrained
0.50, 0.06
0.39, 0.20
0.42, 0.08
0.00,
Table 1: Estimates of polarity and degree of new to- kens before and after training. Each pair of numbers represents mean and standard deviation. v1, v2, v3 represent polarity and degree statistics for the new mod- ifiers (low, medium, high) from our main experiment.
5 Experiment
This section describes how we teach BERT a new system of degree modifiers. Section 5.1 describes how we introduced new tokens into BERT's vocab- ulary by using particles that signal the properties we wish to teach BERT. Section 5.2 provides the details of the fine-tuning procedure and the experi- mental results.
picked particle corresponding to the degree class of the modifier (n = number id):
(15)
Is the reason simple? [prt_vi], it is [mod_vl_n] simple.
Is the reason simple? [prt_v2], it is [mod_v2_n] simple. Is the reason simple? [prt_v3]
it is [mod_ v2_n] simple.
5.1 Mining contexts for new degree modifiers We partition the existing degree modifiers into three same-sized groups, based on the degree scale re- gion they belong (according degree estimation procedure in Section 4.3): moderate, medium, high (or, vl, v2 and v3, respectively). The groups are three color-coded vertical divisions in Fig. 1. We use the identified groups to instantiate three classes of new degree modifiers. For each of the groups, we mine degree-region-specifi particles, using the procedure in Section 4.3, The resulting sets of 3-way diggnssig particles are:
5.2 Fine-tuning BERT new tokens
We split the dataset into training and validation parts with 0.85:0.15 ratio. Then we randomly mask 15% of tokens in the resulting dataset and fine-tune BERT for the task of masked token prediction. We use the same type of pretrained BERT model as in the previous steps. We use the Adam optimization algorithm with decoupled weight decay regulariza- tion (Kingma and Ba, 2014; Loshchilov and Hutter, 2017) and learning rate of 5e-5. 5. We use the batch size of 32 and fine-tune the model for three epochs. For the training, we freeze all weights except for the very first layer of token embeddings."
v1: alternatively, myself, similarly, accordingly, otherwise, however, al alternately, likewise, conversely, er, although, thus, nevertheless, nonethel ess, still, hence
We compare our method against two baselines:
random baseline: 99 randomly initialized
v2: yes, once, naturally, evidently, eventually, not, surely, nowadays, however, someday, fortunately, here, presumably, ideally, accordingly, hopefully
tokens are trained in contexts with particles randomly chosen from any of the three sets (v1, v2 and v3);
untrained baseline: 99 new tokens to be ran- domly initialized before the training phase, but not fine-tuned.
v3: god, gods, goddess, danmit, christ, goddamn, jesus, fucking, holy, kate, damn, skyle, lord, princess, love, daddy
Upon fine- tuning, the three groups of tokens form three clusters, as shown in Fig. 2. Tokens that
For each of the three groups, we instantiate 33 new modifiers. Then, for each sentence in the 10K set, we generate vl sentence, a v2 and a v3. The sen- tences are of the same question-ande answer form as in Section 4, and in each of them we insert a randomly
"This decision is based on the intuition that learning new words in an artificial language learning setting shouldn't 't lead to deep changes in prior linguistic knowledge of native language for realistic learner.

---

0.6
10
0.0
polarity
0.5
polarity
0.6
0.00
0.45
0.40
0.5
04
06
0.3
0.5
08
degree
degree
Figure 2: Target new tokens before (left) and after fine-tuning (right).
100
0.077
0.5
0.066
polarity
0.044
0.23
0.02
0001
0.00
02
03
04 0.5
0.6
025
0.30
0.35
0.40
0.45
0.50
0.5
0.00
degree
degree
Figure 3: Baselines: contexts randomly mixed during training (left) and untrained tokens (right)
belong groups vl and v3 cluster in the PPI re- gion, medium-degree tokens (v2) show NPI-like be- haviour. This is generally in line with linguistic ob- servations described in Scctions and 4. The two baselines (Figure 3), as expected, do not show pro- nounced degree profiles - but develop non-ram non-random polarity behaviour. The random baseline gravitates towards positive polarity, while the untrained base- line shows NPI behaviour. Means and standard deviations for degree and polarity before and after training are listed in Table 1.
of degree contexts is bound to contain a majority of PPT-associated diagnostic particles (this holds for both low and high degree, that is, 2/3 of datapoints). So, the model has prevailing evidence to treat ran- dom baseline items as PPIs. Untrained baseline is more interesting in this respect: new tokens that did not appear in the training dataset at all develop NPI behaviour. We do not know what leads to this, but, at the level of observation, general shift in the direction of lower polarity scores for the whole lexicon might be some artefact of our training pro- cedure. this is true, the very low polarity scores that we see for some items should be interpreted as actually corresponding to somewhat higher scores. We leave exploration of this effect to future work.
6 Discussion and future work
6.1 Interpretation of the experimental results
We saw that the training organized the new tokens into three clusters. First, we observe that the tokens develop low, medium or high degree behaviour, as intended by dataset construction. This means that our procedure conveyed degree information to the model. Furthermore, polarity scores upon training show that the three groups generally follow the hypothesis from Section 2 and analysis from Section 4.3: low and high degrees lead to PPI be- haviour, while medium degrees are associated with negative polarity.
6.2 Limitations and future work
Summing up Sections 5.2 and 6.1, our results are compatible with existing linguistic observations concerning the relation between degree and polar- ity. However, the biggest question our approach is how much we trust the obtained results in making conclusions about natural language. We could gain insight on this question by reproduc- ing the experiment with human participants. The experiment with artificial LMs could serve as a pre- liminary step to polish the underlying hypothesis and the setup for the human experiment. We leave to future work as well.
What is somewhat surprising though is how strong the association with negative polarity is for medium degrees. Here, looking at our baselines might provide hint towards an explanation. The random baseline develops PPI behaviour: this is not particularly surprising given that a random pool
Another question whether there is a reliable way to introduce property A without leaking in- formation about property B the training data.

---

Admittedly, the simple procedure we follow does not take specifie precautions to convincingly show this did not happen. We hope that the version of the experiment that we present here will serve as a starting point for future work developing methods to address this question or recycling existing tools from other types of experiments.
Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In NAACL-HLT.
Marc Ettlinger, Ann Bradlow, and Patrick CM Wong- 2014. Variability in the learning complex mor- phophonology. Applied psycholinguistics, 35(4):807- 831.
7 Conclusion
Delia Fara. 2000. Shifting sands: An interest-relative theory of vagueness. Philosophical topics, 28(1):45- 81.
We introduced method to assess the connection between two linguistic properties as encoded by pre-trained neural LMs. We applied this method an observation in linguistic semantics: the rela- tion between degree and polarity sensitivity. We found that the experimental results are in line with the generalizations from the linguistic literature, indicating validity of our approach and pointing in the direction of BERT making the generalization in question. We hope that this set-up can be applied to other types of models (trained on languages other than English, or multilingual) and other linguistic generalizations, both within individual languages and cross-lngguisicallyy
Gilles Fauconnier. 1975. Polarity and the scale princi- ple. In Proceedings of Chicago Linguiste Suciety 11, pages 188-99 99.
Sara Finley and William Badecker. 2009. Artificial language learning and feature- generalization. Journal of Memory and Language, 61(3):423-43)
Angela D Friederici, Karsten Steinhauer, and Erdmut Pfeifer. 2002. Brain signatures of artificial language processing: Evidence challenging the critical period hypothesis. Proceedings of the National Academy of Sciences, 99(1 (9(1):529-53) 534.
Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 32 42, Minneapolis, Minnesota. Association for Computational Linguisties.
References
Erin D Bennett and Noah Goodman. 2018. Extremely costly intensifiers are stronger than quite costly ones. Cognition, 178:147-111
Nicole Gotzner, Stephanie Solt, and Anton Benz. 2018. Scalar diversity, negative strengthening, and adjecti- val semantics. Frontiers in Psychology, 9:1659.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.1 14165.
Joseph H Greenberg. 1963. Some universals of gram- mar with particular reference to the order of mean- ingful elements. Universals of language, 2:73-113.
Lisa Bylinina and Alexey Tikhonov. 2022a. The driv- Experiments with
Michael Israel. 1996. Polarity sensitivity as lexical semantics. Linguistics and Philosophy, 19(6)619- 666.
ing forces of polarity-sensitivigy multilingual pre- trainraind neural language models. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44.
Michael Israel. 2011.
The grammar of polarity: Prag- matics, sensitivity, and the logic of scales, volume 127. Cambridge University Press
Lisa Bylinina and Alexey Tikhonov. 2022b. Transform- ers in the loop: Polarity in neural models of language. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pages 6601-6610
Satoshi Ito. 2015. Bias and Prosody Japanese Neg- alive Polar Questions. Ph.D. thesis, Cornell Univer- sity.
Jaap Jumelet, Milica Denic. Jakub Szymanik, Dieuwke Hupkes, and Shane guage models use monotonicity to assess NPI licens- ing, CORR, abs/2105.13818.
Fausto Carcassi, Shane Steinert-Thin Threlkeld, and Jakub Szymanik. 2019. The emergence of monotone quanti- fiers via iterated learning. In Proceedings of the 41st Annual Meeting of the Cognitive Science Society.
2021. Lan-
Jaap Jumelet and Dieuwke Hupkes. 2018. Do lan- guage models understand anything? on the ability of Istms to understand negative polarity items. BlackboxxNIP @ EMNLP.
Jennifer Culbertson, Paul Smolensky, and Géraldine Legendre. 2012. Learning biases predict word order universal. Cognition, 122(3):306-329.

---

Jasmeen Kanwal, Kenny Smith, Jennifer Culbertson, and Simon Kirby. 2017. Zipf"s law of abbreviation and the principle of least effort: Language users opti- mise miniature lexicon for efficient communication. Cognition, 165:45-52.
Stephanie Solt. 2018. Not much: On the variable polar- ity sesitivity of 'much' words cross- masasuiiiiiticallly In Proceedings of Sinn und Bedeutung. volume 23.
Tristan Thrush, Ethan Wilcox, and Roger Levy. 2020. Investigating novel verb learning in bert: Selectional preference classes and alternation- generalization. In Proceedings of the Third Black- boxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. pages 265-2755
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes differ- ence with aannterracuuall--agmmeeet data. In Inter- national Conference on Learning Representations.
syntactic
Divyansh Kaushik, Amrith Sethur, Eduard Hovy, and Zachary C. Lipton. 2021. Explaining the efficacy of counterfactually augmented data.
Iris van de Pol, Paul Lodder, Leendert van Maanen, Shane Steinert-Threlkel Threlkeld, and Jakub Szymanik. 2021. Quantifiers satisfying semantic universals are simpler. In Proceedings of the 43rd Annual Meeting of the Cognitive Science Society.
Christopher Kennedy. 2007. Vagueness and grammar: The semantics of relative and absolute gradable ad- jectives. Linguistics and philosophy, 30(1):1 -45.
Charles van Os. 1989. Aspekte der Intensivierung im Deatschen, volume 37. Narr Tübingen.
Christopher Kennedy and Louise McNally. 2005. Scale structure, degree modification, and the semantics of gradable predicates. Language, pages 345-381.
Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Ha- gen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, al. 2019. Investigating bert's ' knowledge of language: Five analysis methods with npis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCN) pages 2877 2887.
Diederik Kingma and Jimmy 2014. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980.
William A Ladusaw. 1979. Polarity sensitivity as in- herent scope relations. Ph.D. thesis, Austin, TX: University of Texas at Austin.
Ethan Wilcox, Peng Qian, Richard Futrell, Ryosuke Kohita, Roger Levy, and Miguel Ballesteros. 2020. Structural supervision improves few-shot learning and syntactic generalization in neural language mod- els. Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), pages
Tyya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint arXiv:1711051001.
Diana Mazzarella and Nicole Gotzner. 2021. The polar- ity asymmetry of negative strengthening: dissociat- ing adjectival polarity from facethreatening potential. Glossa: journal of general linguistics, 6(1).
Online. Association for Computational Linguistics.
Yasamin Motamedi, Marieke Schouwstra, Kenny Smith, Jennifer Culbertson, and Simon Kirby. 2019. Evolv- ing artificial sign languages in the lab: From im- provised gesture to systematic sign. Cognition, 192:103964.
Rick Nouwen. 2013. Best nogal aardige middenmot- ers: de semantiek van graadadverbia van het midden- bereik. Nederlandse Taalkunde, 18(2)200 22244214..
Carita Paradis. 1997. Degree modifiers of adjectives in spoken British English. Lund University Press.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zeulemoyer. 2018. Deep contextualized word repre- sentations. In Proc. of NAACL.
Steven T Piantadosi, Joshua B Tenenbaum, and Noah D Goodman. 2012. Bootstrapping in language of thought: A formal model of numerical concept learn ing. Cognition, 123(2):199-117
S. Solt and E.C. Wilson. 2021. M-modifiers, attenuation and polarity sensitivity. Proceedings of Sinn und Bedeulung, volume 25.


Reviews:
[{"review_id": "n9wLqvM5AL", "reviewer": "Reviewer_zJs2", "summary": "", "questions": "1. If you expand the proxies for polarity to cues that are not just the presence of negation, what knowledge would you derive further? \n\n### Follow-up: It would be more helpful if you could find relevant citations on the other types of contexts in \"Looking at other types of contexts, ... contexts with indirect polarity marking.\".\n\n2. The example (6) and its explanation aren't particularly clear to me. Are you suggesting that modifiers with moderate-to-high range don't pattern with negation? If so, together with (5), are you saying only mid-range modifiers pattern with negation? Could you provide another example? \n\n### This part is solved.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful to the reviewer for the positive comments! There is not much material for rebuttal here, since we seem to be on the same page about the aspects of the work that make it strong, as well as the weaker sides of how we present it (the paper does need streamlining, and thanks to the reviewers, we now see the exact points where this is needed).\n\nAnswers to the questions:\n\n1. If you expand the proxies for polarity to cues that are not just the presence of negation, what knowledge would you derive further?\n\nThat's a great question. Polarity (negative vs. positive) can be analysed in different ways. A superficial notion of polarity could just be the presence or the absence of overt negation. Or it could be a deeper grammatical notion that encompasses the presence vs. absence of negation, but also, additionally, shows up in other contexts marked by other, more subtle, polarity clues (quantifiers 'no' vs. 'some', 'few' vs. 'many' and so on -- a very long list, in fact, arguably, infinite). Looking at negation only, we cannot disentangle which notion of polarity is at play in the cases we study. Looking at other types of contexts, we could see whether there is a qualitative difference between the results we report here and results on contexts with indirect polarity marking. That would give us a clearer idea on the underlying generalisations that the models encode.\n\n2. The example (6) and its explanation aren't particularly clear to me. Are you suggesting that modifiers with moderate-to-high range don't pattern with negation? If so, together with (5), are you saying only mid-range modifiers pattern with negation? Could you provide another example?\n\nSection 2 is definitely not a presentational success on our part, thanks for pointing it out! The distilled message of this section is simply that polarity and degree interact in a non-trivial way. It should say instead that the linguists know that the relation is kind of parabolic, with the lower and high end on the degree scale being positive polarity items, the middle actually having a tendency of combining with (and being interpreted in the scope of) negation to the extent other degree groups don't. There's much more to say about it and the reasons behind it (for instance, extreme modifiers, expressing very high degrees, are 'expressives' in the technical sense of the word: always getting the highest possible scope and thus avoiding being embedded under anything, including negation -- see, for instance, a meta-linguistic feel that a sentence like 'That van is not fricking huge' gives) -- but we can't spend too much space on this. What we should definitely do is a better job mapping the degrees of the modifiers we discuss in Section 2 to the degrees we work with later in the experimental part (v1, v2, v3). Low-to-moderate is v1, moderate-to-high is v2. We should also add a very high degree modifier as an example. \n\nOverall, with the help of the reviewers we identified the weak points in how we present our set-up and results (Section 5 is another example of what should be improved) and we will introduce the respective changes in the new versions of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Besides the above-mentioned contributions, the clear presentation of the paper could lead the readers to a good learning experience: the presentation of the linguistic facts is clear, concise and to the point; the mining and experiment procedure was stated clearly with great potential for replication and extension.\n\nThe authors have good knowledge of existing research in linguistics and NLP/NLU. They are able to synthesize the insights and delineate a clear picture of their contribution.\n\nThe overall high-level research question is appealing and worthwhile -- we are not only interested in the evaluation of the models' current abilities but also in the generalization ability as well as the learning question. I think the methodology the authors used in this paper could be applied to more case studies to bring us closer to the major learning question.", "reasons_to_reject": "I don't have particular reasons to reject this paper, but I would appreciate learning more about why the authors picked this specific phenomenon against the high-level research question.", "questions_to_authors": "1. If you expand the proxies for polarity to cues that are not just the presence of negation, what knowledge would you derive further? \n\n### Follow-up: It would be more helpful if you could find relevant citations on the other types of contexts in \"Looking at other types of contexts, ... contexts with indirect polarity marking.\".\n\n2. The example (6) and its explanation aren't particularly clear to me. Are you suggesting that modifiers with moderate-to-high range don't pattern with negation? If so, together with (5), are you saying only mid-range modifiers pattern with negation? Could you provide another example? \n\n### This part is solved.", "paper_topic_and_main_contributions": "The authors presented a technique to evaluate the correlation between the degree of modifiers and their polarity distribution as represented by pre-trained LLMs. The outcomes of our experiments aligned with established linguistic findings, confirming the soundness of their approach and suggesting the potential for BERT to make generalizations. There are several potential contributions: (1) they provided another case of linguistic evaluation, confirming LLMs' good linguistic knowledge even when the knowledge is quite subtle and not staying at the morphological syntactic level; (2) they provided novel techniques for evaluation -- in addition to common practices in psycholinguistic research, their mining procedure could be used in further studies and is quite inspiring; (3) the final fine-tuning and generalization procedure could also attract more attention in the field to the question of learning in general.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In the Equation (1), there is s^{mskd}_{AFF}; in the following explanation, this symbol becomes s^{masked}_{pos}. It is necessary to keep the symbols consistent.\n\n2. The presentation of Section 5 could be improved. It seems that there is a logic disconnection between Section 5 and the previous parts.\n\n3. More details would be helpful for understanding at the end of Section 5. How were these new modifiers selected? How to compute their corresponding polarity?"}}, {"review_id": "0vfCmkSJg1", "reviewer": "Reviewer_tJwV", "summary": "", "questions": "How was the process of adding new tokens to the model vocabulary carried out?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed analysis of the paper and for pointing out its weaker aspects that might need attention from our side.\n\nAnswer to the question the reviewer poses:\n\nQuestion: How was the process of adding new tokens to the model vocabulary carried out?\nAnswer: \nWe use standard techniques to introduce new tokens, as provided by Huggingface API: https://huggingface.co/docs/transformers/internal/tokenization_utils#transformers.SpecialTokensMixin.add_tokens\nTechnically, the process consists of several steps:\n * add the new token to token2id map in the tokenizer\n * resize the embedding tensor in the model by adding a corresponding number of columns\n * initialize them by random weights\nThere are more complex strategies (see for example, arxiv:2112.14569), but we used the currently-standard practice.\nThe random initialization of these weights is stated in the paper twice (L100-101, L287-289), the code to replicate the our whole experimental pipeline, including the new tokens addition, can be found in the anonymized github repo that we provide a link to in the paper.\n\n\n\nComments on Reasons To Reject:\n\n-- Some parts of the paper are complicated, and several steps in the methodology are not adequately explained. For instance, the process of introducing new tokens into BERT's vocabulary needs clearer discussion. The authors should provide a more explicit description of how these new tokens were added to the model's vocabulary, especially concerning how the weights associated with the selected degree modifiers were initialized.\n\nWe agree that some parts of the paper would benefit from streamlining and better presentation, thanks to the reviewers' comments! We will make sure to describe what we did in a clearer way. But also want to re-iterate that the random initialization of weights is stated in the paper twice, and the code for new token initialization is available on GitHub, with the repository linked in the paper. We would like to note that a Reproducibility score 2 in the situation when there is a GitHub repository with the whole set of ingredients to reproduce the results, seems like a bit of a stretch even if some descriptions of the method in the text could use a bit of restructuring.\n\n-- The results discussion section appears to be brief and lacking in-depth analysis. The paper would benefit from a more concise description of the methodology, allowing more space for analyzing and discussing the obtained results. A clearer presentation of the results would enrich the paper and stimulate further thought and interpretation. I would also suggest to restructure the paper as a short paper.\n\nIt would be great to have more space to discuss the results! Doing it by shortening the methodology would contradict the previous suggestion, however, requesting higher level of detail in the methodology part. In any case, none of this would be possible if the paper is restructured as a short paper, so the requests seem internally contradictory.\n\n\n-- As I wrote previously, Section 2 is interesting for what concerns the description of studies related to degree and polarity sensitivity, but it lacks context from more NLP-oriented studies. Including related work on similar approaches applied to study language model competencies would strengthen the paper's background and demonstrate its relevance within the broader NLP research.\n\nWe would love to learn about other NLP-oriented work with similar focus and experimental methods, and we will happily include citations/discussions of this work in future versions of the paper.\n\n-- While it is not necessary to test a phenomenon on an extensive range of models, studying interpretability in language models could benefit from testing the approach on a few additional models. Incorporating at least one or two other models and evaluating performance on different languages would bolster the claims made in the paper.\n\nWe agree that adding more models and more languages can enrich the conclusions we draw. We disagree that this is a reason for rejection.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-- The application of the \"Artificial Language Learning\" paradigm from psycholinguistics to study linguistic generalizations in pre-trained language models is a compelling approach. This methodology adds value to the paper by offering new insights into how language models handle degree modifiers and their relation to sentence polarity.\n\n-- The authors provide a comprehensive and detailed description of the studies conducted on degree modifiers and polarity sensitivity in Section 2. This helps readers understand the context and motivation for their experiments.", "reasons_to_reject": "-- Some parts of the paper are complicated, and several steps in the methodology are not adequately explained. For instance, the process of introducing new tokens into BERT's vocabulary needs clearer discussion. The authors should provide a more explicit description of how these new tokens were added to the model's vocabulary, especially concerning how the weights associated with the selected degree modifiers were initialized.\n\n-- The results discussion section appears to be brief and lacking in-depth analysis. The paper would benefit from a more concise description of the methodology, allowing more space for analyzing and discussing the obtained results. A clearer presentation of the results would enrich the paper and stimulate further thought and interpretation. I would also suggest to restructure the paper as a short paper.\n\n-- As I wrote previously, Section 2 is interesting for what concerns the description of studies related to degree and polarity sensitivity, but it lacks context from more NLP-oriented studies. Including related work on similar approaches applied to study language model competencies would strengthen the paper's background and demonstrate its relevance within the broader NLP research.\n\n-- While it is not necessary to test a phenomenon on an extensive range of models, studying interpretability in language models could benefit from testing the approach on a few additional models. Incorporating at least one or two other models and evaluating performance on different languages would bolster the claims made in the paper.", "questions_to_authors": "How was the process of adding new tokens to the model vocabulary carried out?", "paper_topic_and_main_contributions": "The paper investigates the sensitivity of a pre-trained language model on degree modifiers and sentence polarity. In particular, the authors hypothesize that the degree expressed by a modifier (low, medium, or high degree) is related to the modifier's sensitivity to sentence polarity. To test this hypothesis, they apply the Artificial Language Learning experimental paradigm from psycholinguistics to the BERT model. The experimental results suggest that BERT generalizes in accordance with linguistic observations, associating degree semantics with polarity sensitivity, particularly low degree semantics being associated with positive polarity sensitivity.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9ZtsQL3WNr", "reviewer": "Reviewer_DQ6w", "summary": "", "questions": "1. Around L359 it is mentioned that BERT acts as only one \"psycholinguistic subject\". Could this be addressed by running your experiments on a larger suite of LMs, for example the 25 MultiBERT models?\n2. Are the standard deviations in Table 1 over multiple seeds of finetuning?\n3. I genuinely do not understand how your finetuning setup would lead to the v2 group to result in a polarity score of 0.0 (Table 1). What is different about these tokens to result in such a drastically different outcome as the v1/v3 groups? I see you address this a bit in Section 6, but I don't think it's fair to leave this interpretation open for future work.\n4. You state in section 2 the the connection between degree and polarity is somewhat linear (low-degree+PPI, high-degree+NPI), but in Figure 1 and around L622 you state that your parabolic finding is in line with prior linguistic research. Do I interpret this wrong, or is there a contradiction here?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the deep, friendly and fair discussion. This is exactly the kind of discussion we were hoping for when working on this!\n\nAnswers to the Reviewer's questions:\n\n1. Around L359 it is mentioned that BERT acts as only one \"psycholinguistic subject\". Could this be addressed by running your experiments on a larger suite of LMs, for example the 25 MultiBERT models?\n\nThat's a great idea, actually. That'd be a pretty computationally heavy experiment, but we think it's worth it.\n\n2. Are the standard deviations in Table 1 over multiple seeds of finetuning?\n\nYes.\n\n3. I genuinely do not understand how your finetuning setup would lead to the v2 group to result in a polarity score of 0.0 (Table 1). What is different about these tokens to result in such a drastically different outcome as the v1/v3 groups? I see you address this a bit in Section 6, but I don't think it's fair to leave this interpretation open for future work.\n\nThe shortest answer is: We don't know. A slightly longer answer is: That's kind of the point -- we expected the parabolic outcome with respect to polarity, and that's what we find. Why such an exaggerated pattern? Can it be that the token embeddings for this group of modifiers under-trained and thus closer to untrained baseline? Here initializing the embeddings non-randomly would give a hint, per reviewer's helpful suggestion (when we conducted the experiments, the 2023 paper wasn't yet out)! Maybe we have time to test it for the longer version of this paper, if accepted. (In fact, we found the comments in this review so helpful and on-point that we would've been happy to continue further investigations in collaboration with the reviewer, but the anonymity..) \n\n4. You state in section 2 the the connection between degree and polarity is somewhat linear (low-degree+PPI, high-degree+NPI), but in Figure 1 and around L622 you state that your parabolic finding is in line with prior linguistic research. Do I interpret this wrong, or is there a contradiction here?\n\nIt's true that clarity of Section 2 fell victim of dramatic shortening of the text before submission! Basically, it does not state the direction or linearity of the relation between degree and polarity, now it just says that there are some interactions that linguists noticed. In fact, it should say it is parabolic, as linguists know. We could also have done a better job mapping the degrees of the modifiers we discuss in Section 2 to the degrees we work with later in the experimental part (v1, v2, v3). Low-to-moderate is v1, moderate-to-high is v2. Overall, the reviews showed us weak points in how the paper is structured and some things are explained. We will streamline the paper and do a better job at presentation, if accepted.\n\nRe other suggestions: \n\n1. Missing references. We know this work (the first paper was actually cited in the penultimate draft which was shortened and the paper didn't make it to the final version..) and will cite it in the next version of the paper for sure, especially when given the extra page.\n\n2. Parallel study with human participants: a great idea, something that we also want to do -- waiting for the right funding moment.\n\n3. \"At L491 it is stated that the max degree score is 0.62, but in Fig. 1 the degree axis ranges to 0.8.\" At L491, the max degree score refers to the hand-picked sanity-check subset of modifiers, not the whole set. Will make this more clear in the text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Strong background on linguistic principles that makes it well-suited for a venue like EMNLP.\n- Exciting usage of LMs as linguistic learning agents, which allows for testing of complex linguistic hypotheses in a controlled environment. The setup described in this paper will definitely inspire other linguistics-minded NLP researchers.", "reasons_to_reject": "- The paper could be structured a bit more clearly, which makes it hard to follow at times. E.g., the setup described in Section 3 only plays a role three pages later, and it took me a while to figure out how everything relates together.\n- The main results present a somewhat confusing pattern, which is not fully addressed by the analysis either.", "questions_to_authors": "1. Around L359 it is mentioned that BERT acts as only one \"psycholinguistic subject\". Could this be addressed by running your experiments on a larger suite of LMs, for example the 25 MultiBERT models?\n2. Are the standard deviations in Table 1 over multiple seeds of finetuning?\n3. I genuinely do not understand how your finetuning setup would lead to the v2 group to result in a polarity score of 0.0 (Table 1). What is different about these tokens to result in such a drastically different outcome as the v1/v3 groups? I see you address this a bit in Section 6, but I don't think it's fair to leave this interpretation open for future work.\n4. You state in section 2 the the connection between degree and polarity is somewhat linear (low-degree+PPI, high-degree+NPI), but in Figure 1 and around L622 you state that your parabolic finding is in line with prior linguistic research. Do I interpret this wrong, or is there a contradiction here?", "paper_topic_and_main_contributions": "The paper investigates the connection between two linguistic properties: polarity-sensitivity and degree modification. Polarity sensitive items are words like *any*, which can only occur within a negative context (I didn't/*did buy any books). Degree modifiers are words like *slightly* and *very*. The connection between these two properties is that low-degree modifiers tend to support positive polarity items (*The issue is fairly important*), whereas high-degree modifiers tend to support negative polarity items (*John isn't particularly smart*).\n\nThis connection is investigated in the context of LMs, in particular BERT. BERT is employed as a learning agent here: via an elaborate experimental setup, the authors attempt to see how the connection between polarity<>degree manifests itself within BERT, and how BERT generalises to novel tokens for which it has no prior knowledge about degree or polarity.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Denić, Milica, Vincent Homer, Daniel Rothschild, and Emmanuel Chemla. (2021). \"The influence of polarity items on inferential judgments.\" Cognition\n- Diagnostic Classifiers (Section 4.4) are not cited, cite e.g. Hupkes et al. or Belinkov here.", "Typos_Grammar_Style_and_Presentation_Improvements": "## Suggestions\n- I would suggest using DeBERTa instead of BERT, which is generally speaking a better model (but I can see that the exact model choice is not vital at all for the merit of the paper)\n- In prior work it has been shown that using randomly initialised embeddings may be suboptimal for pretrained LMs, and that it is better sample from the manifold of the existing embeddings instead. See for example Papadimitriou & Jurafsky (2023), section 5.1.2: https://arxiv.org/pdf/2304.13060.pdf\n- It would be nice to report the results of the experiment in 4.2 somewhere (small table or in appendix). Also: is it possible to evaluate the BERT scores with some human data from elsewhere? I don't know if such a resource exist, but I find this experiment interesting in itself already.\n- If I understand section 4 correctly, the experiments therein show rather the extent to which the polarity<>degree hypothesis holds within vanilla BERT. I would emphasize this a bit more, the introduction made me expect that all experiments would focus on artificial language learning.\n- The parabolic relationship between degree<>polarity in Fig. 1 came as a surprise to me, I suggest explaining the expected behaviour a bit more clearly in Section 2 already (see also my Q4).\n- At L491 it is stated that the max degree score is 0.62, but in Fig. 1 the degree axis ranges to 0.8. Is the information at L491 correct, or should I not interpret these numbers as being the same?\n- Section 4.4 does not add a lot to the story in my opinion, and could be shortened.\n\n## Style\n- Represent the mean,std tuples in Table 1 as mean ± std instead. (I personally prefer using a slightly smaller font for the std values also to make the mean values stand out a bit more, since those are most important here).\n\n## Typos\n- l27: relation -> relations\n- l440: tested *for* their degree-diagnosing\n- l636: *The* untrained baseline\n- l658: We leave *this open* to future work\n- l678: indicating *the* validity"}}]

Author rebuttal:

Meta-review: The paper introduces a method to evaluate the relationship between degree expressed by modifiers and their sensitivity to sentence polarity. The authors draw on the artificial language learning paradigm from psycholinguistics and apply it to BERT, finding that the model’s generalization patterns align with linguistic observations relating degree and polarity sensitivity.

The reviewers express that the paper is clear and presents a meaningful synthesis between linguistics and NLP. The reviewers also express that the methodological setup is novel, interesting, inspiring. The reviewers raised some issues pertaining to clarity, much of which has been addressed in discussion.

My main hesitation, also mentioned in some of the reviews, is the focus of the paper’s experiments solely on the BERT model, which imposes some limitations on the generality and impact of the conclusions. However, based on the merits and interest of the methodology, and the interdisciplinary value, I recommend that the paper be accepted at least to Findings, and potentially to the main conference.
Decision: Accept-Main