Venue: NAACL 2021

# Scalar Adjective Identification and Multilingual Ranking

Aina Gari Soler

Universite Paris-Saclay

CNRS, LISN

91400, Orsay, France

aina.gari@limsi.fr &Marianna Apidianaki

Department of Digital Humanities

University of Helsinki

Helsinki, Finland

marianna.apidianaki@helsinki.fi

###### Abstract

The intensity relationship that holds between scalar adjectives (e.g., _nice_ < _great_ < _wonderful_) is highly relevant for natural language inference and common-sense reasoning. Previous research on scalar adjective ranking has focused on English, mainly due to the availability of datasets for evaluation. We introduce a new multilingual dataset in order to promote research on scalar adjectives in new languages. We perform a series of experiments and set performance baselines on this dataset, using monolingual and multilingual contextual language models. Additionally, we introduce a new binary classification task for English scalar adjective identification which examines the models' ability to distinguish scalar from relational adjectives. We probe contextualised representations and report baseline results for future comparison on this task.

## 1 Introduction

Scalar adjectives relate the entities they modify to specific positions on the evoked scale (e.g., goodness, temperature, size): A _worderful view_ is nicer than a _good view_, and one would probably prefer a _delicious_ to a _tasty meal_. But not all adjectives express intensity or degree. Relational adjectives are derived from nouns (e.g., _wood_\(\rightarrow\)_wooden_, _chemistry_\(\rightarrow\)_chemical_), have no antonyms and serve to classify nouns (e.g., _a wooden table_, _a chemical substance_) McNally and Boleda (2004). The distinction between scalar and relational adjectives is an important one. Identifying adjectives that express intensity can serve to assess the emotional tone of a given text, as opposed to words that mostly contribute to its descriptive content. Additionally, estimating the intensity of a scalar adjective is useful for textual entailment (_wonderful_\(\models\)_good_ but _good_\(\not\models\)_wonderful_), product review analysis and recommendation systems, emotional chatbots and question answering de Marneffe et al. (2010).

Work on scalar adjectives has until now evolved around pre-compiled datasets de Melo and Bansal (2013); Taboada et al. (2011); Wilkinson and Oates (2016); Cocos et al. (2018). Reliance on external resources has also restricted research to English, and has led to the prevalence of pattern-based and lexicon-based approaches. Recently, Gari Soler and Apidianaki (2020) showed that BERT representations Devlin et al. (2019) encode intensity relationships between English scalar adjectives, paving the way for applying contextualised representations to intensity detection in other languages.1

Footnote 1: de Melo and Bansal (2013) discuss the possibility of a pattern-based multilingual approach which would require the translation of English patterns (e.g., “X but not Y”) into other languages.

In our work, we explicitly address the scalar adjective identification task, overlooked until now due to the focus on pre-compiled resources. We furthermore propose to extend scalar adjective ranking to new languages. We make available two new benchmark datasets for scalar adjective identification and multilingual ranking: (a) scal-rel, a balanced dataset of relational and scalar adjectives which can serve to probe model representations for scalar adjective identification; and (b) multiscale, a scalar adjective dataset in French, Spanish and Greek. In order to test contextual models

\begin{table}
\begin{tabular}{c|p{170.7pt}}  & DeMelo \\ en & dim ¿ gloomy ¡ dark ¡ black \\ fr & terme ¡ sombre ¡ fönce! Rö nii es sombiróts¿ c tebnrooss \\ el & zqu¿ go¿ i yx? o¿ t¿ u¿ vz? o¿ x q¿ vz? o¿ x q¿ vz? o¿ \\ \end{tabular}
\end{table}
Table 1: Example translations from each dataset. “\(||\)” indicates adjectives at the same intensity level (ties).



on these two tasks, the adjectives need to be seen in sentential context. We thus provide, alongside the datasets, sets of sentences that can be used to extract contextualised representations in order to promote model comparability. We conduct experiments and report results obtained with simple baselines and state-of-the-art monolingual and multilingual models on these new benchmarks, opening up avenues for research on sentiment analysis and emotion detection in different languages.2

Footnote 1: [https://github.com/ainagari/scalar_adjs](https://github.com/ainagari/scalar_adjs).

## 2 The Datasets

### The MULTI-SCALE Dataset

We translate two English scalar adjective datasets into French, Spanish and Greek: deMelo consists of 87 hand crafted half-scales3(de Melo and Bansal, 2013) and Wilkinson contains 12 full scales (Wilkinson and Oates, 2016). We use the partitioning of Wilkinson into 21 half-scales proposed by Cocos et al. (2018). In what follows, we use the term "scale" to refer to half-scales.

Footnote 3: A full scale (e.g., {_hideus > ugly_, pretty < beautiful < gorgeous_} can be split into two half scales which contain antonyms, often expressing different polarity {_hideus > ugly_} and {_pretty < beautiful | < gorgeous_}.

The two translators have (near-)native proficiency in each language. They were shown the adjectives in the context of a scale. This context narrows down the possible translations for polysemous adjectives to the ones that express the meaning described inside the scale. For example, the Spanish translations proposed for the adjective _hot_ in the scales {_warm < hot_} and {_flavorful < zesty < hot || spicy_} are _caliente_ and _picante_, respectively. Additionally, the translators were instructed to preserve the number of words in the original scales when possible. In some cases, however, they proposed alternative translations for English words, or none if an adequate translation could not be found. As a result, the translated datasets have a different number of words and ties. Table 1 shows examples of original English scales and their French, Spanish and Greek translations. Table 2 contains statistics on the composition of the translated datasets.

In order to test contextual models on the ranking task, we collect sentences containing the adjectives from OSCAR (Suarez et al., 2019), a multilingual corpus derived from CommonCrawl. French, Spanish and Greek are morphologically rich languages where adjectives need to agree with the noun they modify. In order to keep the method resource-light, we gather sentences that contain the adjectives in their unmarked form.

For each scale \(s\), we randomly select ten sentences from OSCAR where adjectives from \(s\) occur. Then, we generate additional sentences through lexical substitution. Specifically, for every sentence (context) \(c\) that contains an adjective \(a_{i}\) from scale \(s\), we replace \(a_{i}\) with \(\forall~{}a_{j}\in s\) where \(j=1...|s|\) and \(j\neq i\). This process results in a total of \(|s|*\) 10 sentences per scale and ensures that \(\forall~{}a\in s\) is seen in the same ten contexts. For English, we use the ukWaC-Random set of sentences compiled by Garf Soler and Apidianaki (2020) which contains sentences randomly collected from the ukWaC corpus (Baroni et al., 2009).

### The SCAL-REL Dataset

scal-rel contains scalar adjectives from the deMelo, Wilkinson and Crowd (Cocos et al., 2018) datasets (i.e. 79 additional half-scales compared to multi-scale). We use all unique scalar adjectives in the datasets (443 in total), and subsample the same number of relational adjectives, which are labelled with the pertainym relationship in WordNet (Fellbaum, 1998). There are 4,316 unique such adjectives in WordNet, including many rare or highly technical terms (e.g., _birefringent_, _anphylactic_).4 Scalar adjectives in our datasets are much more frequent than these relational adjectives; their average frequency in Google Ngrams (Brants and Franz, 2006) is 27M and 1.6M, respectively. We balance the relational adjectives set by frequency, by subsampling 222 frequent and 221 rare adjectives. We use the mean frequency of the

\begin{table}
\begin{tabular}{r l|r r}  & & \# unordered pairs & \# adjectives \\ \hline \multirow{5}{*}{**Searches**} & en & 548 (524) & 339 (293) \\  & fr & 590 (567) & 350 (303) \\  & es & 448 (431) & 313 (275) \\  & el & 557 (535) & 342 (295) \\ \cline{2-3} \cline{2-3}  & en & 61 (61) & 59 (58) \\ \cline{2-3}  & fr & 67 (67) & 61 (60) \\ \cline{2-3}  & es & 59 (59) & 58 (56) \\ \cline{2-3}  & el & 68 (68) & 61 (58) \\ \end{tabular}
\end{table}
Table 2: Composition of the translated datasets. In parentheses, we give the number of unique adjectives and pairs.

 

4,316 relational adjectives in Google Ngrams as a threshold.5 We propose a train/dev/test split of the scal-rel dataset (65/10/25%), observing a balance between the two classes (scalar and relational) in each set. To obtain contextualised representations, we collect for each relational adjective ten random sentences from ukWaC. For scalar adjectives, we use the ukWaC-Random set of sentences (cf. Section 2.1).

Footnote 5: Nine scalar adjectives from our datasets are also annotated as pertainyms in WordNet (e.g., _skimmy_, _microscopic_) because they are denominal. We consider these adjectives to be scalar for our purposes since they clearly belong to intensity scales.

## 3 Multilingual Scalar Adjective Ranking

### Methodology

**Models** We conduct experiments with state-of-the-art contextual language models and several baselines on the multi-scale dataset. We use the pre-trained cased and uncased multilingual BERT model Devlin et al. (2019) and report results of the best variant for each language. We also report results obtained with four monolingual models: bert-base-uncased Devlin et al. (2019), flaubert_base_uncased Le et al. (2020), bert-base-spanish-wwm-uncased Canete et al. (2020), and bert-base-greek-uncased-v1 Koutsikakis et al. (2020). We compare to results obtained using fastText static embeddings in each language Grave et al. (2018).

For a scale \(s\), we feed the corresponding set of sentences to a model and extract the contextualised representations for \(\forall~{}a\in s\) from every layer. When an adjective is split into multiple BPE units, we average the representations of all wordpieces (we call this approach "WP") or all pieces but the last one ("WP-1"). The intuition behind excluding the last WP is that the ending of a word often corresponds to a suffix with morphological information.

**The**diffvec **method** We apply the adjective ranking method proposed by Gari Soler and Apidianaki (2020) to our dataset, which relies on an intensity vector (called \(\overrightarrow{Occ}\)) built from BERT representations. The method yields state-of-the art results with very little data; this makes it easily adaptable to new languages. We build a sentence specific intensity representation (\(\overrightarrow{Occ}\)) by subtracting the vector of a mild intensity adjective, \(a_{mid}\) (e.g., _smart_), from that of \(a_{ext}\), an extreme adjective on the same scale (e.g., _brilliant_) in the same context. We create a \(dVec\) representation from every sentence available for these two reference adjectives, and average them to obtain the global \(\overrightarrow{Occ}\) for that pair. Gari Soler and Apidianaki (2020) showed that a single positive adjective pair (diffvec-1 \((+)\)) is enough for obtaining highly competitive results in English. We apply this method to the other languages using the translations of a positive English (\(a_{mid}\), \(a_{ext}\)) pair from the Crowd dataset: _perfect-good_.6 Additionally, we learn two dataset specific representations: one by averaging the \(\overrightarrow{Occ}\)'s of all (\(a_{ext}\), \(a_{mid}\)) pairs in Wilkinson that do not appear in deMelo (diffvec-wk), and another one from pairs in deMelo that are not in Wilkinson (diffvec-dm). We rank adjectives in a scale by their cosine similarity to each \(\overrightarrow{Occ}\): The higher the similarity, the more intense the adjective is.

Footnote 6: fr: parfait-bon, es: perfecto-bueno, ell: \(\tau\hat{c}\beta\xi\xi\xi\xi^{0}

 

Greek due to low coverage.8

Footnote 8: Only 47% of the Greek adjectives have a BabelNet entry, compared to 95.7% and 88.9% for Spanish and French. All English adjectives are present in WordNet.

### Evaluation

We use evaluation metrics traditionally used for ranking evaluation de Melo and Bansal (2013); Cocos et al. (2018): Pairwise accuracy (p-acc), Kendall's \(\tau\) and Spearman's \(\rho\). Results on this task are given in Table 3. Monolingual models perform consistently better than the multilingual model, except for French. We report the best wordpiece approach for each model: WP-1 works better with all monolingual models and the multilingual model for English. Using all wordpieces (WP) is a better choice for the multilingual model in other languages. We believe the lower performance of WP-1 in these settings to be due to the fact that the multilingual BPE vocabulary is mostly English-driven; this naturally results in highly arbitrary partitionings in these languages (e.g., ES: _fantasticio_\(\rightarrow\) fantastico; EL: YYYYYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYYZQYYZQYYYZQYYZQYYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYYZQYZQYYZQYZQYYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQYZQZQYZQYZQYZQYZQYZQZQYZQYZQYZQYZQZQYZQYZQYZQZQYZQYZQYZQZQYZQYZQYZQZQYZQYZQZQYZQYZQZQYZQYZQZQYZQYZQZQYZQYZQZQYZQYZQZQYZQYZQZQYZQZQYZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQYZQZQZQYZQZQZQYZQZQZQYZQZQZQYZQ 

reported for English, which shows that there is room for improvement in new languages.

## 4 Scalar Adjective Identification

For each English adjective in the scal-rel dataset, we generate a representation from the available ten sentences (cf. Section 2.2) using the bert-base-uncased model (with WP and WP-1). We experiment with a simple logistic regression classifier that uses the averaged representation for an adj-rep as input and predicts whether it is scalar or relational. We also apply the diffvec-1 \((+)\) method to this task and measure how intense an adjective is by calculating its cosine with \(\overrightarrow{dev}\). The absolute value of the cosine indicates how clearly an adjective encodes the notion of intensity. In Figure 1, we show two scalar adjective vectors with negative and positive cosine similarity to \(\overrightarrow{dev}\), and another vector that is perpendicular to \(\overrightarrow{dev}\), i.e. describing a relational adjective for which the notion of intensity does not apply.10 We train a logistic regression model to find a cosine threshold separating scalar from relational adjectives (dv-1 \((+)\)). Finally, we also use as a feature the cosine similarity of the adjective representation to the vector of "_good_", which we consider as a prototypical scalar adjective (proto-sim).

Footnote 10: To draw a parallel with gender debiasing, this value would reveal words’ bias in the gender direction Bolukbasi et al. (2016), regardless of the gender (male or female).

The best BERT layer is selected based on the accuracy obtained on the development set. We report accuracy on the test set. The baseline classifiers only use frequency (freq) and polysemy (sense) as features. We use these baselines on scal-rel because the WordNet pertainyms included in the dataset are rarer than the scalar adjectives. The intuition behind the sense baseline explained in Section 3.1 also applies here.

Results on this task are given in Table 4. The classifier that relies on adj-rep BERT representations can distinguish the two types of adjectives with very high accuracy (0.946), closely followed by fastText embeddings (0.929). The dv-1 \((+)\) method does not perform as well as the classifier based on adj-rep, which is not surprising since it relies on a single feature (the absolute value of the cosine between \(\overrightarrow{dev}\) and adj-rep). Comparing adj-rep to a typical scalar word (proto-sim) yields better results than dv-1 \((+)\). The sense and freq baselines can capture the distinction to some extent. Relational adjectives in our training set are less frequent and have fewer senses on average (2.59) than scalar adjectives (5.30). A closer look at the errors of the best model reveals that these concern tricky cases: One of the four misclassified scalar adjectives is derived from a noun (_microscopic_), whilst five out of eight wrongly classified relational adjectives can have a scalar interpretation (e.g., _sympathetic, imperative_). Overall, supervised models obtain very good results on this task. scal-rel will enable research on unsupervised methods that could be used in other languages.

## 5 Conclusion

We propose a new multilingual benchmark for scalar adjective ranking, and set performance baselines on it using monolingual and multilingual contextual language model representations. Our results show that adjective intensity information is present in the contextualised representations in the studied languages. We also propose a new classification task and a dataset that can serve as a benchmark to estimate the models' capability to identify scalar adjectives when relevant datasets are not available. We make our datasets and sentence contexts available to promote future research on scalar adjectives detection and analysis in different languages.

Figure 1: Illustration of two scalar adjectives that are close to \(\overrightarrow{dev}\) and to its opposite (which represents low intensity). The red vector describes a relational adjective that is perpendicular to \(\overrightarrow{dev}\).

 



## Acknowledgements

This work has been supported by the French National Research Agency under project ANR-16-CE33-0013. The work is also part of the FoTran project, funded by the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement N* 771113). We thank the anonymous reviewers for their valuable suggestions.

## References

* Baroni et al. (2009) Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky wide web: a collection of very large linguistically processed web-crawled corpora. _Journal of Language Resources and Evaluation_, 43(3):209-226.
* Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In _Advances in Neural Information Processing Systems 29_, pages 4349-4357. Barcelona, Spain.
* Brants and Franz (2006) Thorsten Brants and Alex Franz. 2006. Web 1T 5-gram Version 1. In _LDC2006TI3D Philadelphia, Pennsylvania. Linguistic Data Consortium_.
* Canete et al. (2020) Jose Canete, Gabriel Chaperon, Rodrigo Fuentes, and Jorge Perez. 2020. Spanish Pre-Trained BERT Model and Evaluation Data. In _PML4DC at ICLR 2020_.
* Cocos et al. (2018) Anne Cocos, Skyler Wharton, Ellie Pavlick, Marianna Apidianaki, and Chris Callison-Burch. 2018. Learning Scalar Adjective Intensity from Paraphrases. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 1752-1762, Brussels, Belgium. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Fellbaum (1998) Christiane Fellbaum, editor. 1998. _WordNet: An Electronic Lexical Database_. Language, Speech, and Communication. MIT Press, Cambridge, MA.
* Soler and Apidianaki (2020) Aina Gari Soler and Marianna Apidianaki. 2020. BERT knows punta cana is not just beautiful, it's gorgeous: Ranking scalar adjectives with contextualised representations. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7371-7385, Online. Association for Computational Linguistics.
* Geurts (2010) Bart Geurts. 2010. _Quantity implicatures_. Cambridge University Press.
* Grave et al. (2018) Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Koutsikakis et al. (2020) John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2020. GREEKERT: The Greeks visiting Sesame Street. In _11th Hellenic Conference on Artificial Intelligence_, pages 110-117.
* Le et al. (2020) Hang Le, Loic Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Alauzen, Benoit Crabbe, Laurent Besacier, and Didier Schwab. 2020. FlauBERT: Unsupervised Language Model Pre-training for French. In _Proceedings of The 12th Language Resources and Evaluation Conference_, pages 2479-2490, Marseille, France. European Language Resources Association.
* de Marneffe et al. (2010) Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2010. "Was It Good? It Was Provocative." Learning the Meaning of Scalar Adjectives. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, pages 167-176, Uppsala, Sweden. Association for Computational Linguistics.
* McNally and Boleda (2004) Louise McNally and Gemma Boleda. 2004. Relational adjectives as properties of kinds. Colloque de Syntaxe et Semantique a Paris.
* Melo and Bansal (2013) Gerard de Melo and Mohit Bansal. 2013. Good, Great, Excellent: Global Inference of Semantic Intensities. _Transactions of the Association for Computational Linguistics_, 1:279-290.
* Navigli and Ponzetto (2012) Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. _Artificial Intelligence_, 193:217-250.
* Quarez et al. (2019) Pedro Javier Ortiz Suarez, Benoit Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In _7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7)_. Leibniz-Institut fur Deutsche Sprache.
* Taboadg et al. (2011) Maite Taboadg, Julian Brooke, Milan Tofiloski, Kimberly Voll, and Manfred Stede. 2011. Lexicon-based methods for sentiment analysis. _Computational Linguistics_, 37(2):267-307

Bryan Wilkinson and Tim Oates. 2016. A Gold Standard for Scalar Adjectives. In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)_, pages 2669-2675, Portoroz, Slovenia. European Language Resources Association (ELRA).
* Zipf [1945] George Kingsley Zipf. 1945. The meaning-frequency relationship of words. _Journal of General Psychology_, 33(2):251-256.

## Appendix A Comparison of Wordpiece Selection Methods

Table 3 of the main paper contains results of the diffvec method with the best approach for selecting wordpieces (WPs) for each model. In Table 5, we present results obtained using the alternative approach for each model and language:

* for all monolingual models and the multilingual model for English, Table 5 contains results obtained with the WP approach;
* for the multilingual models in the other languages, we show results with WP-1.

The best approach was determined by comparing their average scores across the different methods. Some configurations improve, but they yield overall worse results per model, especially in Spanish. Differences between WP and WP-1 are generally more pronounced in the multilingual models than in the monolingual models 



\begin{table}
\begin{tabular}{l l|c c c|c c c|c c c|c c c}  & \multicolumn{4}{c}{EN} & \multicolumn{4}{c}{FR} & \multicolumn{4}{c}{ES} & \multicolumn{4}{c}{EL} \\  & \multicolumn{3}{c}{**Mono WP**} & \multicolumn{3}{c}{**Mono WP**} & \multicolumn{3}{c}{**Mono WP**} & \multicolumn{3}{c}{**Mono WP**} \\ \hline  & & p-acc & \(\tau\) & \(\rho_{avg}\) & p-acc & \(\tau\) & \(\rho_{avg}\) & p-acc & \(\tau\) & \(\rho_{avg}\) & p-acc & \(\tau\) & \(\rho_{avg}\) \\ \hline \multirow{2}{*}{**MV-1**} & DV-1 (+) & **.664\({}_{9}\)** & **.463\({}_{9}\)** & **.531\({}_{9}\)** & **.617\({}_{3}\)** & **.384\({}_{3}\)** & **.406\({}_{3}\)** & **.652\({}_{9}\)** & **.367\({}_{9}\)** & **.390\({}_{9}\)** & .546\({}_{8}\) & .201\({}_{8}\) & .215\({}_{8}\) \\  & DV-wk & .557\({}_{9}\) & .246\({}_{9}\) & .284\({}_{6}\) & .517\({}_{1}\) & .170\({}_{1}\) & .140\({}_{1}\) & .645\({}_{10}\) & .353\({}_{10}\) & .313\({}_{10}\) & **.557\({}_{2}\)** & **.226\({}_{2}\)** & **.240\({}_{2}\)** \\ \cline{2-11}  & DV-1 (+) & .852\({}_{7}\) & .705\({}_{7}\) & .766\({}_{1}\) & .612\({}_{7}\) & .262\({}_{1}\) & .215\({}_{6}\) & **.763\({}_{8}\)** & **.525\({}_{8}\)** & **.755\({}_{6}\)** & .632\({}_{8}\) & .312\({}_{8}\) & .256\({}_{8}\) \\  & DV-dm & **.918\({}_{6}\)** & **.836\({}_{6}\)** & **.839\({}_{6}\)** & .627\({}_{2}\) & .292\({}_{2}\) & .392\({}_{2}\) & .746\({}_{6}\) & .492\({}_{6}\) & .658\({}_{6}\) & **.779\({}_{1}\)** & **.617\({}_{11}\)** & **.663\({}_{11}\)** \\ \hline \end{tabular} 
\begin{tabular}{l l|c c c|c c c|c c c|c c c}  & \multicolumn{4}{c}{**Multi WP**} & \multicolumn{4}{c}{**Multi WP-1**} & \multicolumn{4}{c}{**Multi WP-1**} & \multicolumn{4}{c}{**Multi (unc) WP-1**} \\ \hline \multirow{2}{*}{**MV-1**} & DV-1 (+) & .588\({}_{4}\) & .301\({}_{4}\) & .312\({}_{4}\) & .549\({}_{7}\) & .239\({}_{7}\) & .276\({}_{7}\) & .589\({}_{3}\) & .229\({}_{3}\) & .234\({}_{1}\) & .524\({}_{9}\) & .153\({}_{9}\) & .171\({}_{9}\) \\  & DV-wk & .516\({}_{5}\) & .153\({}_{11}\) & .198\({}_{5}\) & .490\({}_{2}\) & .113\({}_{2}\) & .134\({}_{7}\) & .603\({}_{12}\) & .268\({}_{12}\) & .287\({}_{12}\) & .521\({}_{6}\) & .146\({}_{6}\) & .186\({}_{6}\) \\ \hline \multirow{2}{*}{**MV-2**} & DV-1 (+) & .820\({}_{7}\) & .639\({}_{7}\) & .667\({}_{3}\) & .612\({}_{3}\) & .262\({}_{3}\) & .362\({}_{3}\) & .746\({}_{4}\) & .492\({}_{4}\) & .608\({}_{4}\) & .647\({}_{9}\) & .358\({}_{9}\) & .369\({}_{9}\) \\  & DV-dm & .885\({}_{7}\) & .770\({}_{7}\) & .834\({}_{7}\) & **.687\({}_{7}\)** & **.412\({}_{7}\)** & **.435\({}_{3}\)** & .661\({}_{10}\) & .322\({}_{10}\) & .447\({}_{6}\) & .662\({}_{6}\) & .388\({}_{6}\) & .444\({}_{6}\) \\ \end{tabular}
\end{table}
Table 5: Results of diffvec (dv) methods with contextualised representations derived from monolingual and multilingual models for each language, using an alternative approach to selecting wordpieces (WP, WP-1) than the one used for the results reported in Table 3. For all languages but Greek, the multilingual model is cased.