Variance Matters: Detecting Semantic Differences without Corpus/Word Alignment
Authors: Ryo Nagata
Hiroya Takamura
Naoki Otani
Yoshifumi Kawasaki
Venue: EMNLP 2023 Main (2023)
Keywords: Semantic difference
semantic shift
word vectors
variance
concentration parameter
TL;DR: Detecting Semantic Differences without Corpus/Word Alignment

Abstract: In this paper, we propose methods for discovering semantic differences in words appearing in two corpora. The key idea is to measure the coverage of meanings of a word in a corpus through the norm of its mean word vector, which is equivalent to examining a kind of variance of the word vector distribution. The proposed methods do not require alignments between words and/or corpora for comparison that previous methods do. All they require are to compute variance (or norms of mean word vectors) for each word type. Nevertheless, they rival the best-performing system in the SemEval-2020 Task 1. In addition, they are (i) robust for the skew in corpus sizes; (ii) capable of detecting semantic differences in infrequent words; and (iii) effective in pinpointing word instances that have a meaning missing in one of the two corpora under comparison. We show these advantages for historical corpora and also for native/non-native English corpora.

Variance Matters: Detecting Semantic Differences without Corpus/Word Alignment
Ryo Nagata Konan University, Japan
Hiroya Takamura
AIST, Japan
nagata-emm1p2223 @m1 hyo..hh.gou.ac. jp.
hokkamura.hiroya@aiiiiei. go.
Naoki Otani
Yoshifumi Kawasaki
Tokyo University of Foreign Studies, Japan
University of Tokyo, Japan
otani@tufs.ac. ac.
ykawasaki@g. ecc. w-t.kyo..cocc.ooo ac.
Abstract
vector becomes. We show that this property of con- textualized word vectors can be quantified as kind of variance of their distribution and that a good indicator of semantic differences.
this paper, we propose methods for discov- ering semantic differences in words appearing two corpora. The key idea is to measure the coverage of meanings of word in cor- pus through the norm of its mean word vec- tor, which is equivalent to examining kind of variance of the word vector distribution. The proposed methods do not require alignments between words and/or corpora for comparison that previous methods do. All they require are to compute variance (or norms of mean word vectors) for each word type. Neverthe- less, they rival the best pest-performing system in addition, they are
The major approach to semantic difference de- tection, which is based on non-contextualized word vectors such as Word2vee (Mikolov et al., 2013), has several limitations and thus is not always appli- cable to any corpora will be discussed in detail in Sect. 5. Above all, many non-contextualized word vector-based methods require some sort of corre- spondence between two corpora for comparison (c.g., word alignment). The task is, however, to find words that do not correspond well in terms of their meanings, and thus it is more natural not to as- sume any correspondence in advance, as Aida et al. (2021) point out. For example, it is not straight- forward all align words between native and non-native English corpora. Besides, most previ- ous methods are computationally costly and are not suitable for targeting all words in a large corpus.
the SemFval-22200 Task robust for the skew in corpus sizes; (ii) capa- ble of detecting semantic differences in infre- quent words; and (iii) effective in pinpointing word instances that have meaning missing in one of the two corpora under comparison. We show these advantages for historical corpora and also for native/non-native English corpora.
1 Introduction
In contrast, the proposed methods do not assume any correspondence between corpora. All they re- quire are to compute the mean of contextualized word vectors and its variance (or its norm) for each word type. They do not require training or hyper- parameter search unlike previous methods. Never- theless, they rival the best-performing system in the SemEval-2020 Task (Schlechtweg al., 2020). They are even effective in corpus pairs whose sizes are considerably different and also infrequent words. addition, they are capable of pinpoint- ing word instances that have a meaning missing in one of the two corpora under comparison. For instance, in Sect. 3, they reveal that near is one of the most semantically different words between the native and non-native sub-corpora (approxi- mately 10,000 and 100.000 words, respectively) of ICNALE (Ishikawa, 201 I); its most representa- tive instance out the 11 near occurrences in the native portion is "ir has near inpossible," which is
In this paper, we propose method for detecting semantic or usage differences? in words and also method for extracting their typical instances in con- text based on variance of contextualized word vec- tors, which bring out various applications: e.g., his- torical linguistics (Hamilton et al., 2016) (e.g., dis- covering words that have acquired a meaning) and second language acquisition research (McEnery et al., 2019) (c.g., words and their meanings that non-native speakers do not use as native speakers). The key observation is that the more meanings word covers in a corpus, the shorter its mean word
'The source codes are available httpp:///ithub com/ nagata-ga gi thub/vmf http///m_aaaanngeeeee www.mnnnnnnncc.. aheaning_changee change_detecton FFollowing the convention in the literature, we use the term semantic difference rather abstractly to refer to differences in meaning and usages. In this paper, semantic meanings of linguistic units include not only lexical meanings. but also contextual meanings, discourse functions, and grammatical functions.

---

interpreted almost; this usage does not appear at all in the 267 instances of near in the non-native sub-corpus.
(a) Only one meaning
(b) Opposite mennings
(c) Orthogonal meanings
The contributions of this paper are three-fold: (i) We show for the first time that norms of the mean contextualized word vectors are good indicator of semantic differences; (ii) We show that (i) is math- ematically equivalent examining variance of the word vector distribution; (iii) We actually reveal words that have semantic differences with their rep- resentative instances in 1800s/2000s English and also native/non-natie English.
Word voctor
Mean word vector
Figure I: Intuitive Illustration for Mean Norms.
word type is represented by two opposite vectors as in Fig. I (b), which should cover much wider meanings. In this case, its mean vector becomes the zero-vector with the zero norm. Other cases in between would give norm between zero and one. For instance, two orthogonal vectors result in the mean word vector whose norm is Fig. (c)².
2 Methods
We describe two methods, one for detecting words that have semantic differences two corpora and one for extracting their representative instances. So far, we have used the term word abstractly to mean both word type and word token. Hereafter, for better understanding, we will distinguish between the two; will use the term word type to refer word types and the term word instance to refer word tokens in context, which we assume is whole sentence.
v2 as in
The observations so far suggest that the wider meanings word type covers in given corpus, the shorter its mean word vector becomes. This prop- erty of word vectors is the basis of the proposed methods.
To discuss formally, we will introduce the fol- lowing symbols. We will denote a word vector by æ. Recall once again that |x| = for all æ. We will also denote the mean vector of æ and its norm by E and respectively (i.e., N 1 Σi= and
2.1 Detecting Semantic Differences
To begin with, let us first note that the similarity between two words (tokens or types) are conven- tionally measured by the cosine similarity between their word vectors (hereafter, for simplicity, word vectors will refer to contextualized ones unless oth- erwise noted). This is equivalent measuring the word similarity based only the directions of word vectors, or to assuming that all word vectors are normalized so that their norms become one. We follow this convention, hereafter.
1 = x= where refers to the number of word instances of that word type in given corpus). We will denote the two corpora for comparison by S and T' (source and target", respectively): for exam- ple, Is refers to the norm of the mean word vector of a word type obtained from the source corpus.
With this notation, the straight forward imple- mentation of the above idea for measuring semantic differences would be taking the ratios b / for all word types appearing in two corpora; larger values of this indicate larger semantic differences (wider and narrower meanings in the source and target corpora, respectively)..
Under this condition, any word vector appears on the unit hypersphere. As a special case of this, when the dimension of word vectors is two, word vectors appear on the unit cirele as in the dashed arrows (vectors) in Fig. 1.
In the proposed method, we use its extended ver- sion as our score function, which we call coverage; we define the coverage as
We now examine the norm of the mean word vector for various cases. An extreme case would be that word type is always used in the exact same context, and thus with the same meaning. Its word vectors uppear at the same point the unit hypersphere in Fig. (a). Then, its mean vector is always identical the original word vectors, and thus its norm is also always one; recall all word vectors are normalized so that their norms equal one. The other extreme case would be that a
1 (11 - 12)
c(S, log
(1)
1s(1- (2)
the reason for which we will explain in Subsect. 2.3.
orthogonal vectors produces along the diagonal line with a norm of /2, and thus the norm of the mean word vector y/
Addition
"Source and target corpora would, for example, he native and non-native English corpora.

---

For the time-being, let us just notice that in the coverage, the norms IT and ls are respectively weighted by 1/ and l. which are based on their counterpart. Also, note that taking the log does not affect rankings by the ratio and that, at the same time, positive and negative values indicate narrowing and broadening meanings of the word type, respectively.
UT
E TT
Figure 2: Illustration for Difference of Mean Vectors.
'he procedure for detecting word types having semantic differences is follows:
for which we will give the mathematical back- ground in Subsect. 2.3.
Input: source and target corpora S, T Output: a list of words sorted order of coverage
The procedure for extracting word instances hav- ing meaning which not, or seldom used in the target corpus as follows:
1. Vectorize all word instances in S and T
Input: source and target corpora S, T; a target word type w
2. For each word type, compute its mean vectors Es and Ty, and then its norms ly and by
Output: a list of word instances in S sorted in or- der of the representativeness
3. Sort the word types by the coverage defined by Eq. (1) in descending order
4. Output the sorted list
1. For w, compute the mean word vectors æs and Øy from S and T. respectively
2.2 Extracting Typical Word Instances
2. For each word instance of w in S, compute the representativeness defined by Eq.(3)
We now turn our focus to extracting word instances having a meaning which is not, or seldom ever, used in the target corpus. For this, we once again consider the illustrative unit circle shown in Fig. 2. Fig. 2 exemplifies two mean vectors Iş and Fy of word type obtained from the source and target cor- pora, respectively. Intuitively, the word instances (or their word vectors) that we are looking for now are those that are distant from the mean word vec- tor for the target corpus (to make sure that their meanings are not, or seldom, used in it) and also that are near the mean word vector for the source corpus (to make sure that their meanings are indeed used in it). The dashed arrow as shown in Fig. 2 would be example of this.
3. Sort the word instances by the representative- ness in descending order
4. Output the sorted list
The list obtained by swapping S and T is also helpful to investigate where the meaning difference comes from.
2.3 Mathematical background
We now give mathematical background the pro- posed methods. Specifically, we show that the two score functions assume the von Mises- Fisser distri- bution behind word vectors (see the study (Baner- jee et al., 2005) for the details of the distribution).
Fortunately, a difference of the two mean word vectors (i.e., Øs Iτ) will facilitate satisfying these conditions. Fig. 2 illustrates that the word vector øy in the source corpus is similar to Eg Tr. This corresponds to taking:
The von Mises-Fisher distribution is probabil- ity density function for the random d-dimensional unit vector x. defined
f(x: μ,, k) Zy exp
(4)
[gg- FT T
The parameters μs (1/=1) 1) and KS (x 2 0) are respectively the mean direction and concentration parameter. The constant ž is the normalization constant depending on k. The distribution can be regarded as akin to the isotropic Gaussian distribu- tion of the hypersphere. It is commonly used to process directional data as in the present paper.
cos([T - TT, x)
(2)
Eg Fy x
Noting ||0||| = and that |æs- - æri is constant with respect to x, Eq. (2) reduces to (z.- Ty)² x. As in Subsect. 2. we will adjust this cosine- based function by 1 I3' and 1 - 12. to define another score function called representativeensss as
In our case, the unit vector x of the von Mises- Fisher distribution is the word vector æ. It fol- lows that the word vector æ distributes isotropically
1
1
((,,,TT) = 1
Ug
UT) x,
(3)
[
12

---

around the mean direction j with the concentra- tion K. Then, is interpreted as the concentration word meanings of the corresponding word type. Conversely, the reciprocal 1/K can be seen as a kind of variance of word meanings.
3 Evaluation
3.1 Data and Conditions
We detect word types having semantic differences and extract their word instances using the proposed methods to evaluate their effectiveness. Specifi- cally, we compare the following two corpus pairs for three purposes: 1800s/2000s English for a historical analysis; citrv//noonnativve English for comparisons between native/non-native speakers of English and between non-native speakers. We use the cleaned version (Alatrash et al., 2020) of COHA (Davies, 2012) for the first one and IC- NALE (Ishikawa, 2011) for the second and third ones, Table shows their sizes.
Considering this, we define the coverage as log(Nr/ /sg). To examine the ratio log(sT ( /ng),
one needs estimate s. Banerjee et al. (2005)
show a simple approximate solution of its maxi-
mum likelihood estimate is:
ldd 12) K ~
(5)
1 1-
where is the norm of the mean vector as defined in Subsect. 2.1 while denotes the dimension of the unit vector x. Then the ratio is approximated to
COHA provides texts published between the 1820s and the 2010s. Accordingly, we use the texts in the corresponding periods. In COHA, 5% of ten consecutive tokens every 200 are replaced by due to copyright regulations. We exclude sen- tences containing this special token from our analy- sis. They also contain a wide variety of fixed labels such as citation information as in Produced from puge scans provided by Internet archive. These inevitably make the norm of the mean word vector longer for the words. Also, they can be noise in that words would not appear in the corpora (e.g., Internet in the 1800s). Similarly, proper names often collocate with fixed contexts such as movic scripts (e.g., John: Yes, is.). We exclude these noisy word types and proper nouns from the sorted list of word types".
bp (d 2
RT log
1-1
log
KS
(o(d (2)
Using d >>>> I, it is further approximated
I 12) 1s(1 (27))
KT
log
log
KS
which is equivalent to our score function coverage. For the representativeness (Eq. (3)), we can show that it is equivalent examining the Log Likeli- hood Ratio (LLR) of the probability density func- tion, which compares how probable the given is in the two corpora. It is given by
In addition, we the SemEval-2020 Task dataset (Schlechtweg al., 2020), which is also based on COHA, to evaluate performance in se- mantic difference detection quantitatively. We eval- uate the proposed detection method in its English sub-task 17, the binary classification task of decid-
EKg exp (SSHL Tx)) x:)
LLR.
log
2nT exp (KTHTT)
log ERs + (KSUs KTHTT) T (8)
"We manually removed such words by consulting their typical word instances from the shown Table and Table the following sections.
The maximum likelihood estimate of μ is given by f = (Banerjee al., 2005). Here, note that only the second term in the second line matters with respect to æ. Then, putting Eq. (5) into the second term results in
"The official dataset provides original and lemmatized texts. The target word instances are tagged only in the latter. We semi-au automatically tagged the original texts by aligning the two and used them in this experiment.
) x.. The approximations d 12 ~ d I2 for d > 12 and d >>>> 13g give the score function representativeness. Note the coarse approximation = I would give the naive score functions originally introduced in Subsect. 2.1 and 2.2.
d 1- Fg d 12 1-2
Corpus
# tokens
COHA 1800s
111,048,657
COHA 2000s
68,678,659
ICNALE Native
97,899
ICNALE Non-native
986,764
Forr example, d = 1024 when hhert-largeuuncssedd" is used as vectorizer while E 1].
Table Sizes of Corpora for Evaluation.

---

Table 3 shows the top ten word types having wider meanings in the 2000s (source) than in the 1800s (target) corpus; for space limitation, the table for the opposite combination is shown in Appendix A. Note that "S,:" and "Ti:" in the representative in- stance column denote that the corresponding word ing which words have semantic difference out of the 37 larget words. Since the proposed detection method designed to predict whether meanings of a word type are narrowing or broadening, we
Method
Accuracy
Proposed
0.730
Rother al. (2020)
0.730
Aida al. (2021)
0.676
take m)x()r<65) min(&r.(Ks) to perform this binary classifica- tion (the larger the value, the greater the difference, whether narrowing or broadening, word type has); we use the K K-means++ algorithm with K 2 to determine the threshold for the binary classifica- tion.
Mean direction-
0.622
Mean direction & coverage- based
0.702
Table Semantic Difference Detection Accuracy in Sub-task of SemEval-2020 Task 1.
These two processes make it difficult to apply the Rother al. (2020)'s method to all word types in a large corpus. While Aida al. (2021)'s method re- quires less computation. it performs worse than the proposed detection method. Somewhat unexpect- edly, the methods that exploit the mcan directions do not perform as well as the proposed method depending only on the concentration parameter k (or equivalently variance). The results suggest that semantic differences are reflected more in variance than in the mean direction. We will discuss this point detail in Sect.
ICNALE consists of essays written by native and non-native speakers of English. As non-native sub-corpus, we use the essays labelled as either China, Indonesia, Japan, Korea, Taiwan, and Thai- land. The essay topics are written on either (a) It is important for college students to have a part-lime job. or (b) Smoking should be completely banned at all the restaurants in the country. This means that the essay topics common to the native and non- native sub-corpora, while the former is ten times smaller than the latter, as shown in Table 1.
The other conditions in this evaluation are as fol- lows. In all corpora, we only target tokens that oc- cur more than ten times. We use as word vectors the outputs of the final layer of "bert-largecased" (De- vlin et al., 2019) for the SemEval sub-task and "bert-largeuuncasedd for the other tasks in the Hug- ging Face implementations. We only target tokens that are not split into multiple sub-words and that consist only of alphabetic letters.
3.3 Comparing 2000s to 1800s English
instances are respectively the ith word instance in the lists obtained by the representativeness from the source and target corpora.
3.2 Semantic Difference Detection
Table 3 reveals the following two categories of word types": potential semantic shift and differ- ences in Part-Of-Speech (POS). We describe them in detail in this order below.
We begin by evaluating detection accuracy in the sub-task of the SemEval-22200 Task Table shows detection accuracy of the proposed detection method together with that of the following four methods for comparison: the berf-performing sys- tem (Rother et al., 2020) in the SemEval shared task; Aida et al. (2021)'s method, which is a non- alignment-based method that can be applicable to all word instances in given corpus the proposed method; Mean -base:::-bbaed:: 1 (os(Zs, Brr is used as score function instead of (1)( Mean direction & coverage- based: the average of
Potential semantic shift: systemic in Table 3 is typical example. It frequently used in body- related sense as in systemic circulation in both sub-corpora. In contrast, the representative word instance systemic injustices only appears in the 2000s. This usage should be relatively new con- cept, probably introduced in the 21st century. A similar argument applies to a kind of technical term political spectrum in the 2000s, which is system that classifies different political positions in rela- tion to one another.
1 1- c-cos(Es, T and the coverage used as a score function.
turns out that despite its simplicity, the pro- posed detection method rivals the Semlival best- performing system that involves much more com- plicated processes - dimensionality reduction and clustering - on top of contextualized word vectors.
The other examples are as follows (the following interpretations in brackets are all in 2000s to I 1800s
cchmmps is an exception. It is often used as a French word as in Champs Elysees in the 1800s while it means champions in the 2000s.

---

c(..))
Word type
fs
fT
Representative instance
0.93
systemic
210
24
S1: systemic injustices / Ti: systemic circulation
0.91
dynamo
77
39
S3: teen dynamo daughter Ti: motor or reversed dynamo
0.83
conversions
36
S1: conversions into theaters / (religious) conversions
0.83
trigger
1222
337
S2: can trigger widespread problems / (T2: drew the trigger
0.81
strikers
17
205
S1 Pjanic playing behind the strikers / Th: the ulacrity of the strikers
0.78
grille
100
11
S1: bar & grille / T2: open the grille
0.76
rotating
333
29
Sj: have been rotating / Ti: the rotating sphere
0.73
champs
82
60
Sy: national champs / T1: Champs Elysees
0.73
spectrum
618
272
S1 political spectrum / Ty: prismatic spectrum
0.72
norm
446
15
S1 wider than the norm at Olympic/ / T1: nor of correct English
Table 3: Top Ten Word Types Having Wider Meanings in 2000s (Source) than 1800s (Target) in COHA. fs and fr: word frequencies in S and T, respectively.
order): dynamo (an energetic person vs. a ma- chine), conversions (physical vs. religious), strik- ers (a position in football vs. people who are on strike), grille (an cating place/a cooking tool vs. metal bars/a gate), and norm (average vs. stan- dard model). While it is difficult to attest them from the information available, these word types, at least, are used in wider meanings in the 2000s.
reveals word types used an idiomatic phrase or a phrasal verb that seldom appear in the non-native sub-corpus, including fall into place, in place (as in effective), and hold down job (as in manage to keep the job). The non-native writers often use place to refer to physical locations while the native speakers also it metaphorically, including the idiomatic phrases. The word hold appears more than 100 times in the non-native sub-corpus, but never collocates with down, suggesting that most non-native speakers do not use or know the phrasal verb that native speakers use (four out of the 16 instances of hold appear in the phrasal verb in the native sub-corpus). Instead, they often use it transitive verb as in hold a job, which also fre- quently appears in the native sub-corpus.
Differences in POS: trigger and rotating fall into this category. The former is used often as a noun (a part of gun) in the 1800s while it is used also as a verb to mean to cause. This can also be interpreted as meaning shift. The latter is used mainly as an attributive adjective in the 1800s while also as verb in the 2000s. Differences in POS like these are reflected in a large value of c(S, T).
Idiomatic phrases also play the opposite role. Specifically, the non-native speakers repeatedly use concerned and course in the idiomatic phrases as shown in Table 4. Surprisingly, they use the idiom of course 87% of the time while the native speakers often use course to mean a set of classes, which decreases the relative frequency of the idiomatic phrase (63%). Although strictly, they not id- iomatic phrases, the non-native speakers use first and third in fixed phrase. Surprisingly again, for instance, the first 886 word instances of first (sorted by r(x T, S)) are actually First, ... It should be emphasized that the nationalities of the non-native speakers range over six countries and nevertheless, fixed expressions like these are common to them. This is beyond the scope of this paper, but it would be interesting to investigate the reasons behind this.
3.4 Comparing Native to Non-Native English
Table 4 shows the top 12 word types having wider meanings in the native sub-corpus (source) in IC- NALE, which follows the same format as Table 3; the table for the opposite combination is available in Appendix B. Table 4 reveals the following three major reasons for the semantic differences: influ- ence from essay prompt, idiomatic phrases, and dif- ferences in construction and part-of-speech (POS). Influence from essay prompt: Simply, many of the non-native speakers use one of the essay prompts Smoking should be completely banned at all the restaurants in the country, as it is (and its partial phrases). To be precise, the entire phrase appears 39 times and only once in the non-native and native sub- cub-corpora, respectively. This natu- rally makes the contexts of completely and country rather fixed in the non-native sub-corpus, resulting in their large norms of their mean word vectors. Idiomatic phrases: More interestingly. Table 4
Differences in construction and POS: These are represented by near and knowledge. The for- mer only appears 11 times in the native sub-corpus and its representative instance is the one that can

---

((,
Word type
fs
fr
Representative instance
0.61
completely
48
1662
Ti: ESSAY PROMPT
0.54
near
11
267
S2: it has become near impossible
0.50
country
48
1707
T1: ESSAY PROMPT
0.46
concerned
11
113
T1: as far as 'm concerned
0.45
third
11
348
Ty: Third, ...
0.39
period
13
115
Sj: Period!
0.38
first
87
1512
T1: First,
0.37
course
46
489
T1: Of course
0.37
place
67
1764
S1: put ban in place / Ss: fall into place
0.36
taking
34
461
S1: taking away TT: taking part time job
0.34
hold
16
111
S1 hold down a job
0.34
knowledge
16
574
S1: it is common knowledge that smoking and passive smoking kill people
Table 4: Top 12 Word Types Having Wider Meanings in Native (Source) than Non-native (T'rrget) English in ICNALE. fs and fr: word frequencies in S and T, respectively.
responding foreign word and that does not exist in native English), c.g., meet; (ii) influence from the sound system of the mother tongue, e-g., low for infrequent instances. This is true for the other example knowledge, which appears only 16 times in the native sub-corpus. Out of the 16, according to r(æ, T),, the top two representative word in- stances of this consist of a that-clause describing the formal subject it as shown in Table 4. This usage seldom appears in the non-native sub-corpus; far we checked manually, only two out of the 574 instances had this construction
be replaced with almost shown in Table 4. Man- ual investigation reveals that this usage does not appear all in the non-native portion. This is an example of the robustness of the proposed methods
for low and law; and (iii) meaning specialization caused by loan words transliterated from English, c.g., rest.
These results together with those in Subsect. 3.4 suggest that the proposed methods might be use- ful for language learning assistance. An example would be an application to feedback comment gen- eration (Nagata, 2019), which is the task of gen- erating hints or explanatory notes that facilitate language learners. One might be able to find word instances by the detection method and to output pre-defined feedback comments to the results.
3.5
Comparison between Chinese and Japanese Learners of English
We now turn our interest the comparison between non-native speakers of English. Here, we compare the Chinese and Japanese sub-corpora in ICNALE for we understand the two mother tongues at least to some extent.
4 Discussion
The evaluation results Sect. show that the con- centration parameter or reciprocal, which is interpreted as a kind of variance, is a good indica- tor of semantic differences. The proposed detec- tion method, which solely relies on the ratio of k, achieves the best detection accuracy as shown in Subsect. 3.2. Besides, the differences shown in Ta- ble 3, Table 4, and Table 5 are mostly interpretable and some of them are indeed differences of mean- ing; after having seen few extracted representative word instances, we were able to tell, in most cases, where the difference(s) came from.
Table 5 and Table 6 show the top word types having wider meanings in the Chinese and Japanese sub-corpora, respectively, which follow the same format as Table It is difficult exam- ine the results because of the lack of the authors" knowledge about the two languages and the En- glish teaching systems in the respective countries. Keeping in this mind, the results are summarized in the following three points (the details are shown in Appendix C): (i) calque (English words used with a meaning that is transferred from the cor-
Interestingly, the results in the SemEval-2022 sub-task suggest that semantic changes are flected more in the concentration (or variance) than the mean direction μı; the methods that exploit the mean direction do not perform as well as the k- based proposed method. It requires more in-
We first searched the non-native sub-corpus for the pattern knowledge that, obtaining 22 instances. We then looked into them, finding that 19 cases were used in a relative clause and one in an erroncous construction.

---

T),
Word type
Js
JT
Representative instance
0.66
alone
26
52
S1: let alone
0.62
rest
23
23
Si: rest of cigarettes Ti: take a rest
0.60
gradually
38
S1: ...solved gradually/ Ti:: decreasing gradually
0.59
value
30
Sy: ... value the advantages T1: the value of money
0.54
second
268
330
Th: Second,
0.53
contact
19
17
S1: early contact with society Th: to contact with other people
0.49
mect
71
75
Sg: those restaurants may meet some difficulties
Table 5: Top Seven Word Types Having Wider Meanings in Chinese (Source) than Japanese (7 (Targett English in ICNALE. fs and fr: word frequencies in S and T, respectively.
T(
Word type
fs
fr
Representative instance
0.94
etc
43
14
S5: clothes and books
0.73
contain
12
11
S1: public place contain restaurants
0.68
word
18
112
Ti: In a word
0.62
taking
27
295
Th: ... taking a part time job can
0.55
whose
13
23
S1: ... students whose families are
0.55
low
12
15
Ty: the low of prohibitring smoking
0.54
becoming
15
37
Sg: when becoming university student
Table 6: Top Seven Word Types Having Wider Mcanings in Japanese (Source) than Chinese (Target) English in ICNALE. Is and fr: word frequencies and T, respectively.
vestigations to confirm this argument, but a pos- sible explanation is as follows. Semantic broad- cning/narrowing normally occurs when mean- ing gradually emerges/disappears from the existing meaning(s). Unlike this, the change only in direc- tion (not in variance) would have to assume that meaning emerges and another disappears to the same degree simultaneously or that a meaning is completely lost and a new one has emerged. Both cases are not so likely in a relatively short period of time as in our 1800s-2000s case.
sis). They solely rely on an off-the-shelf language model (BERT in our case), which is a large advan- tage in terms of implementation and development.
Fortunately, K, or norms of mean word vectors are stable with respect to word frequency as shown in Appendix D; norms of the mean vector become almost constant with frequency of five stability of the norm enables the proposed meth- ods discover semantic differences in infrequent instances. should be emphasized that only one word instance would be enough to proof that a word type has a certain meaning, as in the near example in Table 4 (while the opposite case does not hold).
This
One of the advantages of the proposed detection method is that it can predict whether the mean- ing of target word is narrowing or broadening as illustrated in Subsect. 2.1. Namely, positive and negative values of (1) indicate narrowing and broadening, respectively. Although it would be difficult to evaluate this property of the proposed method quantitatively, since there is publicly available dataset in this regard, qualitatively have seen actual broadening/narrowingg word in- stances in Subsect.3.3 to Subsect. 3.5.
It should also be emphasized that its robustness for the low frequency problem comes from the use of contextualized word vectors via large language model. Even if the source and target corpora are small, the obtained word vectors should be statis- tically reliable considering the language model is trained on a large corpus. In contrast, the previ- ous methods based on non-contextualized word vectors inevitably suffer from the low frequency problem because non-contextualize word vectors are learned from the input corpora.
Another advantage is that the proposed methods are computationally efficient. They require train- ing fine-tuning unlike the previous approaches discussed in Sect.5. Besides, they have almost no paramerssreeters except for the threshold for word frequency (word instances whose frequency is more than this threshold are the target of analy-
As we have discussed, the proposed methods are simple and efficient, but the same time effective in discovering semantic differences. All these nice properties come from the assumption of the von Mises-Fisher distribution behind word vectors. As

---

we will discuss in Sect. although this assump- tion has its limitations theoretically, it works well practically as we have seen in Sect.
word vectors avoid making such assumptions. For this, is applicable to any pair of corpora. At the same time, suffers from the bias corpus sizes and the low frequency problem.
5 Related Work
Some researchers contextualized word vec- tors for semantic difference detection. Hu et al. (2019); Giulianelli al. (2020); Rother et al. (2020); Kobayashi al. (2021) automatically group contextualized word vectors to predict word meanings and then compare the results to detect semantic differences. Predicting word meanings is itself another difficult task. Also, it is costly to train classifier or conduct clustering for every single word type found in corpora.
Linguists (c.g., Fujimura et al. (2013); McEnery et al. (2019)) often use frequency-based methods to discover differences in words between two corpora. Because they only consider superficial frequency counts, it requires more sophisticated methods to conduct deeper analyses into semantic differences.
The use conteeettxtuleeed word vectors is the major approach to semantic difference detec- tion. For diachromic analysis, Kim et al. (2014) propose setting word vectors obtained from the pre- vious time to initial word vectors of the next. For the same purpose, Kulkarni et al. (2015); Hamil- ton et al. (2016) propose methods for discover- ing semantic differences by aligning words in two corpora. These alignment-baan based methods make a strong assumption that words are linearly aligned between two corpora, which does not necessarily hold in any corpus pair (e.g., native and non-native English). Takamura et al. (2017); Kawasaki et al. (2022) extend this approach to discover semantic differences across languages while their methods require a word-alignmen dictionary.
Recently, Aida and Bollegala (2023) have pro- posed a method exploiting the variance of word vector distribution. Their method models the distri- bution by the Gaussian distribution and uses both the mean vector and the covariance matrix. The use of the Gaussian distribution has an advantage in that is not isotropic unlike the von-Mises Fisher distribution. On the other hand, their method is computationally costly. Besides, it requires that words instances should be similar in number in the two corpora for comparison. To avoid the problem, they use sampling to have word instance sets of a similar size, which is not necessary in our methods.
Yao et al. (2018) avoid the problem in word alignment by learning word vectors and align- ment simultaneously. Their method has sensitive hyper-paranee parerrparameerrs that need to be tuned, which re- sults in a complex combinatorial optimization prob- lem (Aida al., 2021). Dubossarsky al. (2019) propose a method for detecting semantic differ- ences by simultancously optimizing multiple word vectors. While this method does not require lin- ear transformations or extensive hyper-parameter search, it requires a list of target words, which is not realistic in practical uses. Aida et al. (2021) extends Dubossarsky et al. (2019)'s method by opti- mizing multiple context vectors together with mul- tiple word vectors. These non-alignment-bur based methods, however, still make the assumption that word vectors and/or context vectors are close to each other two corpora. The task of detecting semantic differences is to find words that are not aligned well in terms of their meanings in two cor- pora, and thus methods that do not require such assumptions are preferable.
Conclusions
In this paper, we have proposed using the concen- tration parameter (or variance), which can be cal- culated from the norm of mean word vectors, to detect semantic differences with their representa- tive instances. The proposed methods do not re- quire assumptions concerning words and corpora for comparison that the previous methods do. The only assumption that word vectors follow the von Mises- Fisher distribution. This enables the pro- posed methods to be applied to corpus pairs such as native and non-native English corpora where the assumptions of the previous methods do not hold. Also, they are simple and efficient in that they do not require training nor extensive hyper- hyper-paramen parameter search. With these methods, we have actually dis- covered semantic differences in historical corpora and also in native and non-native English corpora. We have shown that they are effective even for in- frequent word instances and also for corpora whose sizes are considerably different.
Gonen et al. (2020) propose a method based on nearest neighbors obtained by non-contextullized

---

7 Limitations
Ethical Statement
We do not foresee any considerable risks associated with the present work. We only use well established datasets for the purposes for which they were de- signed, which will likely not cause ethical concerns that did not already exist for these data. Neverthe- less, it should be emphasized that the methods pre- sented in this paper may return results containing noise. Languages are used differently by different people, and attempts measure changes in lan- guage inevitably simplify the diversity of uses. As such, any work applying the methods to measure se- mantic change should be aware of their limitations and proceed carefully.
It should be emphasized that semantic differences found in two corpora do not necessarily mean that the word type has acquired/lost meaning (for his- torical analyses) the writers do not know/cannot use the missing meaning(s) (for non-native speak- ers). It would require further investigations to con- firm these arguments. Rather, the proposed meth- ods are suitable for obtaining new hypotheses about semantic differences in words or for supporting a hypothesis one already has.
As described in Subsect. 2.3, the proposed meth- ods assume the von-Mises Fisher distribution be- hind word vectors. This inevitably assumes that the distribution of word vectors unimodal and isotropic. can be multimodal and/or anisotropic. A more sophisticated modeling (e.g.. a mixture of the von-Mises Fisher distribution (Banerjee et al.. 2005)) might achieve further improvements in per- formance.
Acknowledgements
The authors thank all anonymous reviewers and program chairs for their valuable comments. We also thank Hui-Syuan Ych for her comments on Chinese English. This work was partly supported by JSPS KAKENHI Grant Numbers JP22K12326 and IP23K12152 This work was partly conducted by using computational resource of AI Bridging Cloud Infrastructure (ABCI) provided by National Institute of Advanced Industrial Science and Tech- nology (AIST).
The use of the von-Mises Fisher distribution in- evitably discards norms of individual word vectors. This does not necessarily mean that norms of word vectors are not important for handling word mean- ings; they might encode some important aspects of word meanings. Detecting semantic differences is one thing; encoding word meanings in vectors is another. What this paper has shown is that the con- centration parameter (or variance) of the von-Mises Fisher distribution is effective in semantic differ- ence detection. More investigations are necessary to reveal what is encoded in vector norms.
References
Taichi Aida and Danushka Bollegala. 2023. Unsuper- vised semantic variation prediction using the distri- bution of sibling embeddings. In Findings of the As- sociation for Computational Linguistics: ACL 2023, pages 6868-6882, Toronto, Canada. Association for Computational Linguistics.
Another limitation is that the proposed method assumes that the form of a word is constant as its meaning(s) change. In reality, word forms change well meanings. This limitation is common to the previous methods. It challenging problem to predict changes in forms and meanings simulta- neously.
Taichi Aida, Mamoru Komachi, Toshinobu Ogiso, Hi- roya Takamura, and Daichi Mochihashi. 2021. A comprehensive analysis of PMI-based models for measuring semantic differences. In Proceedings of the 35th Pacific Asia Conference Language, In- formation and Computation, pages 21-31, Shanghai. China. Association for Computational Lingustics.
The use of a large language model to obtain word vectors implicitly assumes that it models the target language well. For such analyses as we conducted in this paper, the assumption should hold at least some extent. However, it would not hold if the target language differs considerably from the language (data) on which the language model is trained. An example would be ancient languages (or ancient corpora).
Reem Alatrash, Dominik Schlechtweg, Jonas Kuhn, and Sabine Schulte im Walde. 2020. CCOHA: Clean cor- pus of historical American Tinglish. In Proc. of the 12th Language Resources and Evaluation Confer- ence, pages 6958-6966.
Arindam Banerjee, Inderjit S. Dhillon, Joydeep Ghosh, and Suvrit Sra. 2005. Clustering the unit hyper- sphere using von Mises-Fisher distributions. Journal of Machine Learning Research, 6(46):

---

Mark Davies. 2012. Expanding horizons in historical linguistics with the 400-million word corpus of his- torical American English. Corpora, 7(2):
Yoshifumi Kawasaki, Maälys Salingre, Marzena Karpin- ska, Hiroya Takamura, and Ryo Nagata. 2022. Re- visiting statistical laws of semantic shift in Romance cognates. In Proceedings the 29th International Conference on Computational Linguistics, pages 141- 151, Gyeongju, Republic of Korea. International Committee Computational Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Yoon Kim, Yi-I Chiu, Kentaro Hanaki, Darshan Hegde, and Slav Petrov. 2014. Temporal analysis of lan- guage through neural language models. In Proceed- ings of the ACL 2014 Workshop on Language Tech- nologies and Computational Social Science, pages 61-65, Baltimore, MD, USA. Association for Com- putational Linguistics.
Haim Dubossarsky, Simon Hengehen, Nina Tahmasebi, and Dominik Schlechtweg. 2019. Time-out: Tem-
poral referencing for robust modeling of lexical se- mantic change. In Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 457-470, Florence, Italy, Association for Computational Linguistics.
Kazuma Kobayashi, Taichi Aida, and Mamoru Komachi. 2021. Analyzing semantic changes in Japanese words using BERT. In Proceedings of the 35th Pa- cific Asia Conference on Language, Information and Computation, pages 270-280, Shanghai, China. As- sociation for Computational Lingustics.
Itsuko Fujimura, Shoju Chiba, and Mieko Ohso. 2013. Lexical and grammatical fcatures of spoken and writ- ten japanese in contrast: exploring lexical profiling approach to comparing spoken and written corpora. In Proc. of the 7th GSCP International Conference. Speech and Corpora, pages 393-398.
Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detec- tion of linguistic change. In Proceedings of the 24th International World Wide Web Conference, pages 625-635.
Mario Giulianelli, Marco Del Tredici, and Raquel Fer- nández. 2020. Analysing lexical semantic change with contextualised word representations. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 3960- 3973. Online. Association for Computational Lin- guistics.
Tony McEnery, Vaclav Brezina, Dana Gablasova, and Jayanti Banerjee. 2019, Corpus linguistics, learner corpora, and SLA: Employing technology to analyze language use. Annual Review of Applied Linguistics. 39:74-92.
Tomas Mikolov, 1lya Sutskever, Kai Chen, Greg Cor- rado, and Jeffrey Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. In Advances in Neural Information Processing Sys- tems 26, pages 31 11-11999.
Hila Gonen, Ganesh Jawahar, Djamé Seddah, and Yoav Goldberg. 2020. Simple, interpretable and stable method for detecting words with usage change across corpora. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 538-555, Online. Association for Computa- tional Linguistics.
Ryo Nagata. 2019. Toward a task feedback comment generation for writing learning. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPP-JCNILP), pages 3206-3215, Hong Kong, China. Association for Computational Linguistics.
William L. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statisti- cal laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Pupers), pages 1489-1501, Berlin, Germany. Association for Com- putational Linguistics.
David Rother, Thomas Haider, and Steffen Eger. 2020. CMCE at SemFvall-2000 task I: Clustering on man- ifolds of contextualized embeddings to detect his- torical meaning shifts. In Proceedings of the Four- leenth Workshop on Semantic Evaluation, pages 187- 193, Barcelona (online). International Committee for Computational Linguistics.
Renfen Hu, Shen Li, and Shichen Liang. 2019. Di- achronic sense modeling with deep contextualized word embeddings: An ecological view. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 3899-3908, Flo- rence, Italy, Association for Computational Linguis- tics.
Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 task 1: Unsupervised lexical semantic change detection. In Proceedings of the Fourteenth Workshop Semantic Evaluarion, pages 1-23, Barcelona (online). International Committee for Computational Linguistics.
Shinichiro Ishikawa. 2011. A new horizon in learner
corpus studies: The aim of the ICNALE project, pages 3-11. University of Strathclyde Publishing, Glasgow.

---

Michael Swan and Bernard Smith. 2001, Learner En- glish, 2nd edition. Cambridge University Press, Cam- bridge.
in the new corpus. It refers something rebound- ing physically in the 1800s, while also refers to an action in basketball in the 2000s; accord- ing to Wikipedia" basketball was first played in 1891 and thus this usage did not exist in the middle 1800s or earlier. This new usage has become the majority, which has made the others relatively less frequent. We cannot tell if this is a true semantic narrowing or for some other reason(s), for example, register differences between the 1800s and 2000s corpora. Still, the above discussion explains why rebounds is detected as semantic narrowing in the 2000s although it actually gained a new meaning.
Hiroya Takamura, Ryo Nagata, and Yoshifumi Kawasaki. 2017. Analyzing semantic change in Japanese loanwords. In Proceedings of the 15th Con- ference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Pa- pers, pages 1194-1204, Valencia, Spain. Association for Computational Linguistics.
Zijun Yao, Yifan Sun, Weicong Ding, Nikhil Rao, and Hui Xiong. 2018. Dynamic word embeddings for evolving semantic discovery. Proceedings of the 11th ACM International Conference on Web Search and Data Mining, pages 673-681.
B Comparing Non-Native to Native English
A Comparing 1800s to 2000s English Corpora
Table shows the top 12 word types having wider meanings in the non non-native sub-corpora (source) in ICNALE, which follows the same format as Ta- ble 4 (Subsect. 3.4). Table 8 reveals the following four major reasons why the word types have wider meanings in the non-native sub-corpus: spelling and grammatical errors; unusual collocations; dif- ferences in POS; idiomatic phrases. We describe their details in this order below.
Table 7 shows the list of top ten word types having wider meanings in the 1800s (source) than in the 2000s (target) corpus. It follows the same format Table 3 (Subsect. 3.3). The word types are classi- fied into the following two categories: transcription errors and potential semantic shift.
Transeription errors: They are flagged as semantic difference because they are seemingly in- correctly transcribed as in the representative word instance of whore, which should be where; other ex- amples include teen for been, coma for come, and luna for tune?. Transcription errors increase the variation in the contexts of a word type and shorten its mean word vector. It is crucial to remove tran- scription errors to conduct accurate analyses. This is especially true for historical corpora. The eval- uation results suggest that the proposed detection method may be used detect transcription errors. Potential semantic shift: All other word types in Table 7 exhibit a potential semantic difference. Although it would be difficult to prove their mean- ing differences from the available information, they are at least all interpretable. A typical example is pregnant. In the 1800s, means both having a child and rich. In the 2000s, both meanings are still valid, but the ratio has shifted heavily towards the former. From the word instance list sorted by representativeness. (i.c., (3)), we estimate that 99% of the 2,290 instances are of the former sense in the 2000s. The other examples are: quanium (a eating place).
Spelling and grammatical errors: Spelling and grammatical errors increase the variety of contexts of a word type. For example, form often appears as a spelling error of from and thus the superficial word type appears in the contexts of the verb and noun (form) and also of the preposition (from). This makes the mean vector of the word type shorter. Similarly, hope (with direct object noun) and re- sponsible (as a noun or a verb) are used incorrectly. Unusual collocation: Some words are used in unusual collocations (e.g., I'll introduce interesting job to mean recommend). Similarly, the adjective great is often used with negative words such as harm and damage. This usage is not quite incorrect. but is rare in the native sub-corpus where it is used in a positive sense such as in great idea. Other examples include section, and staff.
Differences in POS: Some words are used with a POS that seldom, or never, appears in the native sub-corpus. For example, essential is only used as predicative adjective as in It is essential for in the native sub-corpus, while it is also used as an attributive adjective and noun in the non-native sub-corpus. Similarly, pretty is used as an adverb
unit quantum in physics), and diner (meal vs an
Interestingly, while rebounds acquires a new meaning in the 2000s, its meanings are narrowing
1tttps:: htapp:://en.wweee www.wwwwiidd..... a.rg org/wiki /Bikkk/aakkbal.... Ac- cessed on 5th, May, 2023.

---

7(
Word type
fs
fr
Representative instance
1.19
whore
57
483
S1: the rear, whore was found
1.06
rebounds
18
342
S2: the wave rebounds / Ti: 9.2 rebounds
1.01
teen
44
849
S1: has already teen shown
0.96
hitter
40
303
S₁ with hitter feelings / :i: power hitter
0.92
recession
32
539
S4: the recession of cliffs/ Ti: After the recession began, presidents Bush
0.90
pregnant
343
2290
S1: pregnant sense / T; she was pregnant.
0.88
tuna
13
464
S1: we tuna ourselves with the peoples
0.86
coma
30
345
S2: must have coma from god.
0.85
quantum
43
901
Sy: the usual quantum of abuse / T22 a quantum physicist
0.84
diner
14
635
S3: If diner was an apple 'T: afternoon shift at the diner.
Table Top Word Types Having Wider Meanings in 1800s (Source) than 2000s (Target) in COHA. fs and fri word frequencies in S and T, respectively.
()
Word type
Js
fT
Representative instance
0.76
form
92
21
S1: learn form the books / Ti: some form of
0.72
degree
70
39
S2: some degree S3: to some degree
0.64
pretty
44
28
Ss: pretty clothes
0.62
hope
195
24
Sy: must hope money from their parent
0.61
worry
94
27
Sy: a lot worry / S5 their worry
0.55
section
85
11
S3: important section of students / Ti: smoking section
0.53
great
429
67
S1: great harm / S2: greal damage
0.52
responsible
127
44
S1: a man who responsible S3: their responsible
0.51
staff
45
S1: a concert staff S4: a part time staff
0.51
yes
68
13
S1: we say yes,
0.48
essential
86
13
S1: essential university students / S2: biochemistry essential
0.48
introduce
11
13
Sj: I'll introduce interesting job S2; introduce set
Table 8: Top 12 Word Types Having Wider Meanings in Non-native (Source) than Native (Target) English (ICNALE). fs and fr: word frequencies in and respectively.
and also as an adjective. It is not clear why the non- native speakers English use these words with wider POSs, which would be a research question for second language acquisition research.
pears only once out of 52 occurrences in JPN. These differences are reflected their concen- tration parameters.
2. CHN, rest is used as both to relax as in take a rest and remaining things as in the rest of the time while in JPN, it is only used in the first sense. This might be due to the Japanese loan word resuto transliterated from rest. It only refers the first sense. Takamura al. (2017) empirically show that Japanese loan words are used in differently, often limited meanings, from their English counterparts. This might be the case"l.
Idiomatic phrases: The non-native speakers often use the idiomatic phrases to some degree and in some degree, which never appear in the native sub-corpus. In contrast, the native speakers it mean a college qualification, which also appears in the non-native sub-corpus. This is similar to concerned and course in Table 4.
Details of Comparison between Chinese and Japanese Speakers of English
3. In CHN, gradually collocates with various verbs while in JPN, it does with limited verbs. mostly with increase, decrease, and expand in the progressive form.
The results of the comparison is shown in Table 5 and Table 6 in Subsect. Our interpretation of the results are as follows (CHN and JPN refer the Chinese and Japanese sub-corpora, respectively). Words having wider meanings in CHN: 1. alone appears in various contexts including the idiomatic phrase let alone in CHN. It ap-
value is used a noun and verb in CHN
"The argument here will be only fully valid after having examined the Chinese case. That is, Chinese may have a loan word such as the Japanese resuto and may have limited meanings. This is heyond the authors' ability.

---

while the verb usage never appears (out of 38 occurrences) in JPN. This might be the influence from the transliterated loan word baryu just as in the "rest/resuto" case above. 5. second frequently used in the phrase Second, in both CHN and JPN, but the ratio is higher in JPN (68% and 72% in CHN and JPN, respec- tively). In CHN, it appears in wider varieties of contexts in second hand smoke/smokingg. 6. contact are used as both a noun and verb in CHN while (often with an erroncous preposition (c.g., con- tact with); 12 time out of 17) in JPN.
0.005
Differece from previous norm
0,004
2.001
30
Number of word occurrences
frequently used as a verb
Figure 3: Relationship between Number of Word Oc- currences and Differences in Two Consecutive Norms.
6. low is used to mean both low and law in JPN. This is a possible mother tongue interference reflecting the fact that the two sounds pro- nounced as one sound in Japanese (Swan and Smith, 2001).
7. In CHN, meet is used to mean to face and to run into as in meet difficulty and *meet problem. These usages never appear in JPN; rather it is used in the more standard to sense, which also appears in CHN. This might be an influence from the Chinese word fi), which translated into to face and to run into as well as to meet. As a result, erroneous phrases such as the above two appear in CHN.
7. becoming is often used with more and more in the representative instance in CHN. In contrast, is used with more flexible context including in the progressive with omission as when becoming a university student. JPN, which never appears in CHNE (out 37 instances). This might reflect that Chinese learners of English have difficulty with pro- gressive aspect (Swan and Smith, 2001).
Words having wider meanings in JPN:
1. etc appears only the sentence end with a comma (e.g., hall, church, etc. EOS) in CHN while it appears everywhere in a sentence in JPN. To be precise, it uppears after various noun phrase as in the representative word in- stance. This is a possible mother tongue inter- ference; the Japanese word nado correspond- ing to "etc" can appear after almost any noun phrase without a comma.
D Relationship between Word Frequency and Vector Norms
We investigated the relationship between word fre- quency and vector norms. Specifically, we cal- culated norms of the mean word vector for each occurrence of each word type and then the differ- ences between the two consecutive values of the norms.
2. Grammatical error, correctly containing, which only appears in JPN
3. In CHN, word frequently appears in the phrase in word (and the like) (101 times out of 1 112), which shortens the mean vector and makes its meanings narrower than in JPN. In contrast, it is used in other contexts in JPN.
Fig. shows the results where the horizontal and vertical axes denote the number of occurrences of word types and the norm differences averaged over all word types, respectively. Fig. 3 shows that after around five occurrences, the average norm differ- ence becomes almost zero, meaning that the norm of the mean vector is almost constant. Considering this, setting the frequency threshold a small value such as ten gives stable vector norms (and thus sta- ble values of the concentration parameter). This stability of the norm enables the proposed meth- ods to discover semantic differences in infrequent instances.
4. taking appears frequently in the phrase taking part-time job CHN, which is a possible interference from &
corresponding to take (K% is used with job to mean to work or to do job literally in Chinese). The phrase taking part-time job, mostly as subject, are so frequent that it makes its mean word vector short (and narrower meanings) in CHN. 5. whose appears frequently in the phrase stu- dents whose family are CHN, which makes the mean word vector short. The reasons are not clear.


Reviews:
[{"review_id": "ye6gVGuSqN", "reviewer": "Reviewer_Nr3G", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Nr3G,\n\nWe would like to express our thanks for the review and the useful comments. We would be grateful if you could read and check the following responses.\n\n-------\n# Reviewer’s Comment 1:\n>This work has only done experiments and reported results in English corpora, it would be very interesting to see some analysis in \n> different languages or across languages, especially in the non-native English users with their mother tongue languages, and explore \n> further potential in using variance metrics in second language acquisition research.”\n\n\n# Response 1:\nWe appreciate the suggestion. Following the advice, we have done comparisons between different non-native English users. As a result, we were able to find meaning/usage differences; some of them are possibly ascribed to mother tongue interference. The details are shown below, but they are summarized in three points: (i) influence from the sound system of the mother tongue, (ii) meaning specialization caused by words transliterated from English, (iii) calque (English words used with a meaning that is transferred from the corresponding foreign word and that does not exist in native English). It is difficult to attest them because of the lack of our knowledge about the languages and the English teaching systems in the respective countries. However, the results suggest that the proposed methods are likely beneficial for further investigations in second language acquisition as the reviewer suggests.\n\n\nThe details of the additional experiment are as follows. \n- Data:\n ICNALE six non-native sub-corpora: Chinese, Indonesia, Japanese, Korea, Thai, Taiwanese English data abbreviated as CHN, IND, JPN, KOR THA, TWN, respectively, hereafter.\n\n- Method:\nWe have mainly compared CHN and JPN for we understand the two mother tongues to some extent. Specifically, we have extracted words having wider meanings/usages for the combination and have extracted their representative word instances just as in the experiments we described in the original manuscript. We have investigated top seven words for CHN and JPN. We compared them with other sub-corpora in some cases.\n\n- Results: The two tables below with their explanations.\n\nTable: Words having wider meanings in JPN (w.r.t CHN)\n-------\n|TopN| Word type |  Representative instance|\n| ---- | ---- | ---- |\n|1| etc| study, club activity , part time jobs _etc_ all things are a part of college life.|\n|2.|\tcontain| public place _contain_ restaurants\n|3|word| in a _word_ (in CHN)\n|4|taking| _taking_ a part-time job (in CHN)\n|5|whose| students _whose_ family are (in CHN)\n|6|low| the _low_ of prohibiting smoking\n|7|becoming| when _becoming_ a university student\n\n\nCorresponding interpretation and explanation\n1. “etc” appears only in the sentence end with a comma (e.g., hall, church, etc. EOS) in CHN (similarly in IDN and TWH) while it appears everywhere in a sentence in JPN. To be precise, it appears after various noun phrase as in the representative word instance above. This is a possible mother tongue interference; the Japanese word “nado” corresponding to “etc” can appear after almost any noun phrase without a comma. \n2. Grammatical error, correctly “containing”, which only appears in JPN\n3. In CHN, “word” frequently appears in the phrase “in a word” (and the like) (101 times out of 112), which shortens the mean vector and makes its meanings narrower than in JPN. In contrast, it is used in other contexts in JPN. \n4. “taking” appears frequently in the phrase “taking a part-time job” in CHN, which is a possible interference from 接受 corresponding “to take” (接受 is used with “job” to mean “to work” or “to do a job” literally in Chinese). The phrase “taking a part-time job” so frequently  that it makes its mean word vector short (and a narrower meanings in CHN). \n5. “whose” appears frequently in the phrase “students whose family are” in CHN\n6. “low” is used to mean both “low” and “law” in JPN. This is a possible mother tongue interference reflecting the fact that the two sounds pronounced as one sound in Japanese (Swan and Smith, Learner English, Cambridge University Press, 2001). Similarly, the same error appears in KOR, which is detected compared to CHN by the proposed methods. This agrees the fact that Korean learners of English a similar problem in English vowels (Ibid.) \n7. “becoming” is used in the progressive form in JPN as in “when becoming a university student.”\n\n\nTable: Words having wider meanings in CHN (w.r.t JPN)\n-------\n|TopN| Word type |  Representative instance|\n| ---- | ---- | ---- |\n|1|alone| let _alone_\n|2|rest| take a _rest_ (in JPN)\n|3|gradually| is decreasing _gradually_ (in JPN)\n|4|value| Companies _value_ work experience\n|5|second| _Second_, (in JPN)\n|6|contact| early _contact_ with society\n|7|meet| _meet_ a similar problem\n\nCorresponding interpretation and explanation\n1. “alone” appears in various contexts including the idiomatic phrase “let alone” in CHN. The distribution is similar in JPN, but the idiomatic phrase appears only once out of 52 occurrences. These differences are reflected in the concentration parameters. \n2. “rest” is used in two meanings (“to relax” as in “take a rest” and “remaining things” as in “the rest of the time”) in CHN. In contrast, it is only used to mean “to relax” in JPN. This might be a possible mother tongue interference; influence from the transliterated Japanese “resuto” that only refers to the first meaning. Because of this, most of the Japanese learners in ICNALE might have only known the word “rest” with the first meaning and have used it frequently.\n3. In CHN, “gradually” collocates with various verbs while in JPN, it  is used frequently with “increase”, “decrease”, and “expand” in the progressive form as in the above example. \n4. “value” is used as a noun and a verb in CHN while the verb usage never appears (out of 38 occurrences) in JPN. This might be a possible mother tongue interference just as in the “rest/resuto” transliteration influence above; “value” is transliterated as “baryu” in Japanese, which is only used as a noun. \n5. “second” is frequently used in the phrase “Second, “ in both CHN and JPN, but the ratio is higher in JPN (68% and 72% in CHN and JPN, respectively). In CHN, it appears in wider varieties of contexts as in “second hand smoke/smoking”. \n6. “contact” used as both a noun and a verb in CHN while it is frequently used as a verb (often with an erroneous preposition (i.e., contact with); 12 time out of 17) in JPN.\n7. In CHN, “meet” is used as “face” and “run into” as in “face a difficulty” and “run into a problem”. These usages never appear in JPN; rather it is used in the more standard “to see” sense, which also appears in CHN. This might be a influence from the Chinese word 遇倒, which is translated into “to meet”, “to face”, and “to run into”.\n\n# Reviwer’s Comment 2\n> As mentioned in the limitation section, that the usage of a large language model to obtain word vectors implicitly assumes that it \n> models the target language as well. The methods rely on a monolingual language model trained on certain time period of data can be\n> limited in detecting semantic shifts across a wider period of time. The current approach can be improved and this topic can be explored\n> further.:\n\n# Response 2\nAs the reviewer points out, the limitation concerning language modeling is one of the challenges we have to attack in the future work. Luckily, the first comment and this second comment gave us a chance to rethink the limitation. Now we feel that the limitation has been somewhat relaxed thanks to the comments. \n\nThe current language model, BERT, is trained on English Wikipedia and BookCorpus, both of which should consist of rather recent texts. Wikipedia might contain webpages citing texts published in the 1800s or earlier, but the majority should be those written more recently. Although the details of BookCorpus are not available, the corresponding paper (Zhu et al., 2015) says that it consists of unpublished books, which suggests that they were written again recently. To sum up, BERT is mainly trained on recent texts and a relatively small amount from the 1800s or earlier. Nevertheless, the experimental results in 3.2 and 3.3 (in the original manuscript) show that the BERT-based proposed methods are effective in detecting words with semantic differences in the 1800s. This supports the hypothesis that language models can work well for texts in different time periods as long as the time difference is within a certain amount such as 100-200 years as in the present case.\n\nA similar argument can be made about the comparison between native and non-native English data. Again, BERT is mainly trained on native English data and not on non-native one; of course, Wikipedia (and probably BookCorpus, too) should contain texts written by non-native speakers of English, but the vast majority are of native speakers. Still, the BERT-based proposed methods are effective in native and non-native English comparison. Besides, we have just seen that they are also effective in the non-native-non-native comparisons (between non-native speakers of English with different mother tongues) as shown in the first response. These results support the argument that language models work well (with the proposed methods) on texts of the target language as long as the difference between the training and target corpora is not too large.\n\nFrom these, we now speculate that language models do not even have to model the target sub-language well in the sense that they predict masked words or following words when they are used as a part of the proposed methods. All they should have to do is to map word instances to vectors of different directions. Even when they are given an anormal context such as an ancient sentence or an unnatural, erroneous sentence, they only have to map it to a vector with a direction different from the others, which is not so hard to believe.\n\nHaving said that, this is only our speculation. We need more investigations to confirm this argument. We do not know how much difference in time period or language uses language models can afford when they are used with the proposed methods. These should be included in our future work.\n\nWe will include the above discussion in the conclusion section and include the new challenges in our future work. The reviewer’s comments and the discussion concerning them have deepened our understanding of the proposed methods and in turn have enriched the contents of the manuscript. We really appreciate your thoughtful suggestions and expertise.\n\n\n------\n\nMany thanks again for the careful review. We will revise the manuscript following the advice.\n\nBest regards, \n\nAuthors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed methods present a straightforward and effective approach for detecting semantic differences and the mathematical background is illustrated in certain detail. \n2. Use-cases are given with both quantitative and qualitative analyses. \n3. The codes are provided and the work is reproducible.\n4. The work can be beneficial in research in other disciplines, such as second language acquisition.", "reasons_to_reject": "1. This work has only done experiments and reported results in English corpora, it would be very interesting to see some analysis in different languages or across languages, especially in the non-native English users with their mother tongue languages, and explore further potential in using variance metrics in second language acquisition research. \n2. As mentioned in the limitation section, that the usage of a large language model to obtain word vectors implicitly assumes that it models the target language as well. The methods rely on a monolingual language model trained on certain time period of data can be limited in detecting semantic shifts across a wider period of time. The current approach can be improved and this topic can be explored further.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes methods for measuring semantic differences in words between two corpora, contributing to computationally-aided linguistic analysis. \n\nUsing a pre-trained large language model, e.g., BERT, new methods are used to examine the coverage of the meanings of words, through the norm of the mean word vectors across meanings. The proposed methods are easy to implement and without pre-training any language models, and do not require alignments between words or corpora for comparison. The proposed methods excel in semantic difference detection tasks compared to previous works, reveal potential semantic shift and difference in POS using diachronic datasets. Furthermore, the differences between native and non-native English usages are revealed using the proposed methods, in terms of different aspects, which can be beneficial for further investigation in second language acquisition. \n\nThe paper also discusses the limitations of the proposed methods, specifically the presumption of using von Mises-Fisher distribution and also other limitations for applications.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "thl9fnAVLq", "reviewer": "Reviewer_3y58", "summary": "", "questions": "A: In a sketchy algorithm given in section 2.2 you write that the xs-xt should be counted. But the definition of representativeness  contains weights here. ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 3y58,\n\nWe would like to express our thanks for the thorough review. We really appreciate the effort and time.\n\n As pointed out in the question field,\n\n> In a sketchy algorithm given in section 2.2 you write that the xs-xt should be counted. \n> But the definition of representativeness contains weights here.”\n\nthe algorithm in 2.2 was erroneous. It should be:\n\n- Error: For w, compute the difference mean word vector $\\bar{x}_S − \\bar{x}_T$\n- Correct: For w, compute the mean word vectors $\\bar{x}_S$ and $\\bar{x}_T$\n\nand the rest are unchanged. We will correct the error in the revised manuscript.\n\nMany thanks again for the careful review.\n\nBest regards, \n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method is relatively simple, but very effective. The authors presented the mathematical basis of the proposed measures. An extended qualitative analysis was conducted with some interesting observations about the English language itself.", "reasons_to_reject": "I do not see any", "questions_to_authors": "A: In a sketchy algorithm given in section 2.2 you write that the xs-xt should be counted. But the definition of representativeness  contains weights here. ", "paper_topic_and_main_contributions": "The authors present a new method for checking whether the same words occurring in two different corpora have the same or different meanings. The method is relatively simple because it uses pre-trained contextual word vectors and is based on counting the norm of the mean vectors. The intuition is that if a word has more meanings (more boundary meanings), the mean vector is shorter. A suitable coefficient for comparing norms in two corpora was defined and called \"coverage\".  An additional task that was solved was to identify a representative occurrence of a word for a meaning that does not occur in the other corpus.  The solution is based on counting the cosine similarity between a vector representing a particular occurrence of a word and a vector representing the difference between the weighted mean norms. Experiments were conducted on two pairs of corpora: English 1800s/2000s (COHA) and native and non-native speakers of English (ICNALE).  SMEval-2020 Task 1 data (from COHA) was used for quantitative evaluation.  The results obtained were very good, at the level obtained by much more computationally demanding methods.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E9tXyCblLg", "reviewer": "Reviewer_TMju", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer TMju,\n\nWe would like to express our thanks for the review and the useful information. As pointed out, our explanations are insufficient in some parts of the original manuscript, which makes readers confused. To make it clearer, let us add some more information.\n\nFirst of all, we thank the reviewer for letting us know about the citations. Both are related to our work and we will cite them in the revised manuscript adequately.\n\n------\n# Reviewer's comment 1\n> Although two-dimensional vectors are specified, it is unclear if this is always the case.\n\n# Response 1\n(We might be misunderstanding this comment). We suppose that our explanation was confusing. We used the two-dimensional vector case just as an example because we can easily visualize the situations as in Figure 1 and Figure 2 (in the original manuscript). The arguments about the concentration parameter and the length of mean vectors hold for three- or more dimensional vectors. That is, if a word appears in more varieties of contexts (thus, more meanings), its mean word vector gets short. This is shown by Eq.(5) --- concentration parameter $k \\approx \\frac{l(d-l^2)}{(1-l^2)}$ --- where $l$ and $d$ denote the length of the mean word vector and the dimensionality of the word vector, respectively. Since we did not explicitly state this fact, it was not clear if the arguments hold for higher dimensions. We will add the explanation in the revised manuscript.\n\n# Reviewer's comment 2\n>  Words that are formally identical but have different meanings ought to be connected to various vectors.\n\n# Response 2\n(We might also be misunderstanding this comment). Again we suppose that our explanation seemed to be insufficient on this point, too. As the reviewer suggests, words that are formally identical are mapped to different vectors depending on their surrounding contexts. If their contexts are diverse, the directions of the corresponding vectors are also diverse. Then, summing vectors with diverse directions results in a relatively short summed vector and in turn in a short mean word vector as visualized in Figure 1. We will make this point clear in the revised manuscript.\n\n# Reviewer's comment 3\n> the assumption that the form of a word is constant as its meaning(s) change. \n\n# Response 3\nThis is probably the largest limitation in the proposed methods. It is also one of the largest limitations in all previous methods. Considering these facts, we will include the limitation as future work in 6. Conclusions.\n\n------\n\nMany thanks again for the careful review. We will revise the manuscript following the advice.\n\nBest regards, \n\nAuthors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach might be useful to compare diachronic corpora and detect periods of creation for some document collections. It could also be applied to detect texts developed by non-native speakers and even discriminate between texts developed by speakers with different mother tongues.", "reasons_to_reject": "It goes without saying that vectors rely on specific texts, and that some degree of similarity between collections of texts can be measured. The authors stated a few limitations on the suggested approach themselves: the assumption that the form of a word is constant as its meaning(s) change.\nAlthough two-dimensional vectors are specified, it is unclear if this is always the case. Words that are formally identical but have different meanings ought to be connected to various vectors.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a method for detecting semantic or usage differences and a method for extracting their typical instances in context based on the variance of contextualised word vectors. The approach should be effective in corpus pairs whose sizes are considerably different and also in infrequent words. The authors propose two methods: one for detecting words that have semantic differences in two corpora and one for extracting the representative instances.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "https://aclanthology.org/P19-1321/\nhttps://paperswithcode.com/paper/learn-interpretable-word-embeddings\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents simple yet efficient methods that can be used to compare word usage across different contexts (historical, native vs. non-native etc.). The methodology is sound and uncomplicated and the results are compelling. 

Work that identifies simple techniques that allow us to capture significant linguistic properties and better understand the structure of human language will make a useful contribution to this track.
Decision: Accept-Main