Comparing semantic and morphological analogy completion in word embeddings
Authors: Haopeng Xie
Venue: Submitted to ICLR 2023 (2023)
Keywords: machine learning
computational linguistics
word embeddings
morphemes
semantic relationship
analogy completion

Abstract: Word embeddings have prompted great excitement in the NLP community due to their capacity for generalization to unforeseen tasks, including semantic analogy completion. Features such as color and category relationships have been examined by previous work, but this is the first research considering the morphological relationships encoded in word embeddings. We construct several natural experiments examining analogy completion across word stems modified by affixes, and find no evidence that Word2Vec, glove, and fasttext models encode these morphological relationships. We note that a special case of this problem is part-of-speech transformation, and note that the lack of support for part-of-speech analogies is surprising in the context of other successful cases of semantic inference using word embeddings.

COMPARING SEMANTIC AND MORPHOLOGICAL ANAL- OGY COMPLETION IN WORD EMBEDDINGS
Anonymout mouy authors Paper under double-blind review
ABSTRACT
Word embeddings have prompted great excitement the NL.P community due their capacity for generalization to unforeseen tasks, including semantic analogy completion. Features such as color and category relationships have been exam- ined by previous work, but this is the first research considering the morphological relationships encoded word embeddings. We construct several natural experi- ments examining analogy completion across word stems modified by affixes, and find no evidence that Word2 glove, and fasttext models encode these morpho- logical relationships. We note that special of this problem is part-of-speech transformation, and note that the lack of support for part-of-speech analogies is surprising in the context of other successful cases of semantic inference using word embeddings.
INTRODUCTION
Prior work (Fejzo et al., 2018) has shown that compounding morphological learning is key to lan- guage acquisition and literacy development in children. When learn new language, we able to quickly memorize known words and derive unknown words through our knowledge of root words and morphological features. This ability enables language learners efficiently and accurately comprehend the meanings of enormous words, but can computers do the same?
This introduction reviews prevailing approaches to modeling semantic relationships between words, as well as some prior work on morphological and analogical analysis. Then, methods for quantifying the ability for word embeddings represent semantic morphological features are proposed in the form of results are summarized.
tasks. Experiments are run on three common word embeddings and
1.1 PRIOR WORK
Early popular methods for Natural Language Processing (NLP) relied on simplistic models trained over large datasets, such the N-gram for statistical language modeling. However, although such models outperform many more sophisticated counterparts, a limitation is the size of obtainable data for training, which impedes accurate representations over large corpus of words. As the corpus size increases, the computational resources required to handle the learning with this simplistic ar- chitecture also accumulate.
1.1.1 WORD EMBEDDING MODELS
In 2013, a group led by Google scientist Tomas Mikolov (Mikolov al., 2013a) proposed a novel continuous vector representation method along with two neural network models built for the task: Continuous Bag-of-Worde (CBOW) Model and Continuous Skip-gram Model. Mikolov's model considered not only the proximity but also the similarity between related words in the vector space, and multitude of degrees of linearity is encoded into the word embedding. With this property, using simple algebraic operations of the vector can formulate computer generate analogies comparable to those humans. Pairwise semantic relationships within a subset of words can also be represented using the linear properties of the embedding space.

---

In subsequent publication, Mikolov (Mikolov et al., 2013b) proposed several optimization meth- ods, such as subsampling the Skip-gram Model. To encode idiomatic information, simple phrases composed of individual words were also trained separate entries. The optimized model could train on 100 billion words a day, with a total data size of billions compared to a few millions of words in other models.
Word embedding has become a central topic in the field of Computational Linguistics and Natural Language Processing. The Word2vec model (Mikolov et al., 2013a), utilizes Continuous Bag-of- words (CBOW), which calculates nearby context words but disregards any additional syntactical information. Due to its simplifying nature, CBOW tends to rely on multiplicity rather than syntax. (Mikolov et al., 2013a)
GloVe (Pennington al., 2014) another word embedding model, first published by Jeffry Pen- nington and his team from Stanford University in 2014. Like Word2 GloVe utilizes similar unsupervised process, but the difference is that combines local word context information and global word co-occurrence statistics.
In contrast to Word2Vec and Geo FastText (Joulin et al., 2016) by Facebook Al Research (FAIR) is based on a supervised sentence classification task. The task structure encourages word embed- dings that can be added together to derive sentence-level representation. It can train on large datasets relatively quickly and run with less computational consumption than prior models.
1.1.2 CONTEXT-SENSITIVE SEQUENTIAL MODELS
Some more recent approaches to general NLP tasks involve context-sensitive word-embeddings ing the Transformer Model, first introduced 2017 Vaswani et al. (2017). Similar to Recurrent Neural Networks (RNN), Transformer Models are designed process sequential input, but the key difference that it utilizes the "Attention Mechanism", which weights the context vectors accord- ing to their estimated importance and forms connections within sentence. This method enhances encoding of long-term dependencies and improves the overall performance the high efficiency of the Transformer model, several highly general pre-trained models have been designed, such as BERTDevlin et al. (2018), GPT-3Brown et al. (2020), and ELMOPeters et al. (2018), for variety of tasks with fine-tuning. These context-sensitive embeddings are more expres- sive than the static word-en word-embedding models that map each word (or idiomatic phrase) to a single vector, regardless of context, but they are much harder study semantically as any metrics for word similarity become necessarily contextual.
the model. Due to
1.1.3 SEMANTIC EMBEDDINGS FROM MORPHOLOGICAL FEATURES
Several approaches have been proposed to specifically encode morphological features into word embedding models. One method character-based learning on word segmentsCao Rei (2016), which divides words into segments, treating morphemes separately. However, the character- model performed worse on semantic similarity and semantic analogy tasks than traditional word- based models.
Another is vector representation by composing characters using bidirectional LSTMs(Ling et al., 2015). This model excels in morphologically rich languages, languages that indicate part of speech by adding affixes rather than changing the position in sentence, but learning complicated rules for how characters link together is inefficient on larger corpora.
1.1.4 WORD EMBEDDING ANALOGIES
There is also some prior work examining vector-based analogies in word embedding space. For example, one research paper (Bolukbasi et al., 2016) examines the semantic quality of gender as a difference vector in embedding space for the w2vNEWS embedding (a variant of Word2 Vec). The paper focuses specifically on gender bias word representation and use, and uses similar methods to this paper explore the semantic representation of gender.
2

---

1.2 CONTRIBUTIONS
The contributions of this work two-fold. First, systematic experiments around morphological analogy completion reveal that Word2Vec, gloVe, and FastText word embeddings all fail to capture the same quality of semantic information around rreaning-modifyinn prefixes (such as un-, re-, or anti-) as between oft-touted "common-sense" examples. Furthermore, these models fail to capture the relationships encoded morphologically by part-of-speech modifying suffixes (such as -tion, ing, or -ly). Second, broad statistical examination of word-pair distance-vectors corroborates the absence of morphological features these commonly-usee word-embedding models.
2
METHODS
This work consists primarily of exploratory experiments on pre-trained language models. This sec- tion describes the models, datasets, and cmapoyy-completion tasks used for this work, well as the methods of statistical analysis used to examine the semantic representation of morphological features.
2.1 PRETRAINED MODELS
This work uses three popular pre-trained word-embedding models: Word2 (Mikolov et al., 2013a), GloVe (Pennington et al., 2014), and FastText (Bojanowski al., 2016). Each of these pre-trained models represent words (and some idiomatic phrases) as a single vector. This work uti- lized pre-trained variants with 300-length vectors for each model. These word-embedding models static, context-insensitive, in that the same word in different sentences always receives the same embedding. This means that words with multiple highly distinet meanings (polysemy) are sometimes ill-represented, but has the advantage that the relationship between words across all contexts can be modeled as the difference between those two vectors.
2.2 ANALOGY COMPLETION
Word embeddings achieve analogy completion with algebraic vector operations. For example, in the classic case of "man is to king as woman is to queen", we can obtain the word "queen" by adding the vector of "woman" to "king" and subtracting the vector of "man". This operation allows for pairwise analogy for any given three words, by searching for the word whose embedding is nearest to the predicted vector. The capacity for word embeddings to solve these kinds of analogy- completion problems (without explicitly training for them) has been compelling demonstration of the depth and versatility of the word embedding models. As demonstrated in table 2.2 using 10 hand-crafted examples, Word2Vec (by way of example) appears accurately represent variety of semantic features. It is noteworthy that of the 10 examples, are perfectly completed, has a plausible but unintended answer, and only has unreasonable answer.
Table I: Examples of Analogy Completion with Semantic Information using Word2Vec
Parent Analogy
Target Word Computed Answer Success
king as woman queen student learn as teacher teach
man
queen
queen
Yes
teach narrow
teach
Yes
big small as wide narrow
narrow aunt aunt
Yes
man woman as uncle aunt
Yes
little big as dwarf:: giant
giant dwarfs
No
boy: girl man woman
woman woman
Yes
high: low as up down
down down
Yes
paris: : france tokyo japan
japan
japan
Yes
brother: : sister as grandson granddaughter
granddaughter
granddaughter
Yes
food eat toy: play
play Legos No

---

2.3 MOST-COMMON WORD DATASETS
The experiments in this paper used public dataset of most common words from Google's Trillion Word Corpus pre-processed by Google's N-gram models. This dataset contains the first ten thou- sand most common words calculated from 1,024,908,2611 ,299 words of text analyzed by Google's Al research teams. To examine morphological feature representation in embedding space, a list of the 22 most common prefixes and 30 suffixes was also used.
A dataset of word pairs consisting of root word and an affix-root compound was generated for each affix. The most common words were initially filtered for the presence of the affix morpheme, and then were further filtered to those with valid roots when truncating the affix. Note that some valid pairs were omitted by the procedure because the affix changed the spelling of the root component (e.g. happy happily), and some pairs that were generated were semantically implausible (e.g. member: remember). In total 6730 pairs of words madix-modified and root words) were generated. This will be called the "affix-pairs dataset"..
2.4 MORPHOLOGICAL (AFFix) ANALOGY COMPLETION
A large number of pairwise morphological analogy completion tasks were generated by selecting two words with the same affix and their corresponding root words (say, a: b and e: d), then applying algebraic vector operations (b c) and comparing the resulting vector to the closest words in the embedding space.
Grouped morphological analogy completion tasks were also created by taking the arithmetic mean of the difference vectors for each affix, and then testing whether each modified word could be predicted by added the mean difference vector the root word.
2.5 DISTRIBUTIONS OF DISTANCE VECTORS
To further explore morphological feature encoding in word embeddings, the distribution of differ- ences (and differences of differences) of word-pair vectors is analyzed. In particular, this distribution compared to the distribution differences between random pairs of words selected uniformly from the 10000 most common words. If morphological features are well represented by the em- bedding space, the norms of the difference vectors (between the root word and the modified word) should be tightly clustered (more tightly than the differences between random words). Furthermore, the difference between the word-pair difference vectors (the differences between "modification vec- tors") should be close to
(In other words, if the same kind of semantic analogy exists for morphological features as for other semantic features, the difference vectors between root words and the words modified by the same affix should be very similar.)
EXPERIMENTAL RESULTS
This section considers, in-turn, structural similarities in the morphological relationships encoded by each of the word embeddings.
tasks, distributions of difference vectors, and
3.1 Do WORD EMBEDDINGS SOLVE PAIRWISE MORPHOLOGICAL ANALOGIES?
No. Across word pairs using the same affix, no tested pair word-pairs resulted in success- ful analogy completion in Word2Vec, GloVe, FastText. Pairs of word-pairs were randomly sampled for each affix and embedding, and word-pairs with the most similar difference vectors (as measured by Euclidean norm) were also explicitly tested. The 10 pairs of word pairs (from the affix- pairs dataset) with most similar difference vectors in Word2Vec are summarized Table 3.1. This result stands in stark contrast to the non-morphological semantic analogies, where many previously untested analogies are completed successfully.
"https://www. schol lasti http://wwcclaa.....oooooooooa//a wttp./////nneeeeeeeee////aaaiiie http:t/teeeeeeeeeee/etetteeeee/e htt.eeehhhheeeeeeeepeeeeeeeeee pesssnnplansmmiigreeeeeeee wwwess-i--ooo///eeeeeeeee-e

---

Table 2: Examples of Analogy Completion with Morphological Information
Parent Analogy
Target Word
Computed Answer
Success
fixed fix as fined: fine
fine
fining
No
presented; : present as allowed allow
allow
permitted
No
supported: support signed: sign
sign
signing
No
informed: inform as worked work
work
working
No
sorted as covered cover
cover
kind
No
intended: intend as viewed : view extended extend owned: own played: play as liked like returned: return as blamed blame passed pass ended: end
view
see
No
owns
No
like
loved
No
blame
blames
No
end ending
No
3.2 DO WORD EMBEDDINGS SOLVE AVERAGE MORPHOLOGICAL ANALOGIES?
No. Across all affixes and the three considered embeddings, there not single instance of successful analogy completion using the average difference vector for a given affix. This outcome generally agrees with the pairwise analogy result, but further demonstrates that the pairwise result not merely a consequence of unbiased noise.
3.3 Do AFFIX WORD-PAIR DIFFERENCES CLUSTER MORE TIGHTLY THAN RANDOM PAIRS?
Unclear. As seen Figure 3.3, the difference vectors for "ed" (selected as an example due to its comparatively small average distribution of difference vectors) have a different distribution shape but have comparable range the differences of random pairs of words. No affix under consideration demonstrated a substantially narrower distribution.
3.4 Do DIFFERENCES OF DIFFERENCE VECTORS CLUSTER NEAR ZERO?
No. As shown Figure 3.3, differences between word-modification vectors for the suffix "-ed" have a mean well away from zero.
As shown in Figure 3.4, the mean difference between modification vectors for pairs involving the same affix was well away from zero for all affixes considered and in many cases exceeded the difference between random words.
3.5 Do DIFFERENT WORD EMBEDDINGS ENCODE MORPHOLOGICAL FEATURES STRUCTURALLY?
Yes. Although this result is surprising in contrast the other results in this paper, the three word embeddings considered show a high correlation between the rank order of affixes by difference of modification vectors, shown Table 3.5. Examples such as "tion", "ious", and "de" have high mean norms compared to others such as "s", "ed"", and "ty" for all models. This suggests that while some affixes better represented than others, these patterns throughout all three models. Note that GloVe and Word2Vec have the most similar structural trends, which likely a result of their more similar training procedure and data (relative Fest.ext).
mostly consistent
Although this kind of structure is not predictive in the sense of enabling compoyycooplettin, it does show that morphological features are reflected in the topology of the embedding space. For example, across all of the embeddings, pluralization is the most analogous operation, perhaps because words and their plurals have very similar meanings begin with.
3.6 SUMMARY
These experiments demonstrate that morphological features with semantic content are generally not represented in commonly used word embeddings, at least not to the same extent that non- morphological semantic qualities, such gender, age, location, opposites seem be encoded.
5

---

Randomly Chosen Words
300
200
100
50
5
7
Norm
"ed" Word-Pair Differences
160
120
Occurrence
100
80
20
0
15
2.0
255
3.0
355 Norm
4.0
4.5
5.0
5.5
Differences of "-ed" Word-Pair Differences
50000
40000
Occurrence
30000
20000
0
Norm
Figure 1: From top to bottom: the distribution of (Euclidean) norms of differrnnce vectors between pairs of words chosen uniformly from the 10000 most common; the distribution of norms of dif- ference vectors for root words and root words plus "ed"; the distribution of norms of differences between pairs of difference vectors for word-pairs from the "-ed" set. All numbers from Word2Vec.
At the same time, there appear to be structural similarities (relating to the spread of modification vectors) between embeddings produced by different methods. Returning to the motivating question of the paper, extant word embeddings do reflect some properties of morphological features, but do not generalize across morphological analogies as humans do.

---

Word2Vec
4.0
35
mg
0Js
ment
est
non
under
By
on
Super less mid
en
ative
mis
Inter
Ive
Ible
over
y
im
em
trans
lal
dis
pro
toon
fore
ant
Morphemes
Fasttext
2.0
15
cd
ous
ress
Ive
ion
trans
under
sub
inter
ablie
(al
over
Ibl
foze
im
by
ition
super
anti
Itve
in ran
tton
gre
mdd
em
Morphemes
Glove
Norm
c0
mg
mont
non
es at om
est less
ion
sble mis
fty
sub
im
en
over
ive
Super inter mi trans ative
y Ible
aal em
ition
pro itie tion
Morphemes
Figure 2: Mean Euclidean norm of differences between word-modification vectors for three word
embeddings are plotted for each affix from the suffix and prefix list.
Table 3: Pairwise Spearman's 's's Rank Coefficient for Word2Vee, FastText, and GeoVe.
Model Names
Coefficient
Word2 FastText
0.66449
Word2 Vec GloVe
0.88337
FastText + GloVe
0.58122
4 DISCUSSION
Here, the limitations and implications the work are discussed. While the experiments conducted confined to common English words, three static word-embedding models, and analogy comple- tion experiments, they are very suggestive that morphological features may not well-represented by fairly modern word-embedding approaches.

---

4.0.1 LIMITATIONS
One limitation of this work is the simplification of the morphological model considered. We ex- cluded words with modified root when combined with an affix due to the complexity of such modifi- cation in English and the difficulty of its inclusion into the experiment. If we were able to encompass all words with affix regardless transformation, we would acquire a larger and more representa- tive morphological model, although this seems unlikely to alter the qualitative outcomes of these experiments.
The scope of experiment is also limited three commonly-used word-embedding models: Word2Vec, FastText, and GloVe. To gain a better understanding of word embedding as a whole, an examination of other models of different characteristics and publication times is likely to yield a more well-rounded result. Furthermore, all embedding models considered here are context- less, meaning that there is no sentence-level information handle challenges such polysemy. It is much more difficult to construct rigorously quantitative models. but the authors observe anecdotally that naive initial
experiments for
sensixt-eensitive transformer-basee attempts failed to elicit clear evidence of morphological feature encoding.
4.0.2 IMPLICATIONS
This work at once reproduces oft-touted instances of semantic analogy completion (for example, woman:queen::mnnkingg and yet also shows that many morphological relationships that reflect deep semantic features of human language do not have comparable representation in Word2Vec, FastText, and GloVe. Furthermore, statistical analysis of word-pair ristance-vectorn suggests that the words in question are not near to each other in embedding space, and their difference vectors do not share common direction.
This is surprising and unfortunate result. Without the ability to quantitatively represent the effect of morphological modification of words, it will be difficult develop language models that reproduce humans' ability to generate and understand novel words, as well as to generalize from much smaller sets of textual examples.
In fact, this shortcoming may be natural consequence of the Continuous- of- tation. Since this approach estimating word relationships based on frequency of proximity largely ignores the relationship of words that do not co-occur, and many morphologically-modife are either semantically or grammatically incompatible (for example, because of verb-tense agree- ment), morphological feature relationships may be relatively unconstrained by the model training objective.
Bag-off Words Represen-
words
5 CONCLUSION
Our experiments show that morphological features are at best indirectly represented in common CBOW word embeddings. Both the empirical distributions of embedding-space distances and per- formance on analogy completion tasks suggest this conclusion.
The questions of why morphological features are under- redeesreereeette remains unanswered. Is this just a pitfall of CBOW- embeddings? How could we address this problem? Does this suggest, by contrast, a pattern in human acquisition of language?
Methods that explicitly utilize morphological learning might significantly improve the efficiency and accuracy of word embedding models and provide better solution to many NLP problems. Such models could learn from a much smaller set of root words compared to Word2vec, which treats root words and its derivatives as individual entries. A reduction in the complexity of the model can decrease the computational resources required and increase the accuracy of the embedding. Mimicking natural language learning of human more closely, word embedding models that contain morphological learning could potentially lead to revolutionary performance zero-shot NLP tasks, including those with novel words.
8

---

REFERENCES
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information, 2016. URL https:
/atas://axxte //tp//aaavv.////////a//e///aaaaa org/ aa///bbs///10/ 04606.
Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam T Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embed- dings. In D. Lec, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https:: http::://proceediggs..eee neurips c4 Pacpeaaar...ccc-eee.....e
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020. URL https:/ //trx/aa.vvvv/// org/abss 2005 14165.
Kris Cao and Marek Rei. joint model for word embedding and word morphology, 2016. URL https://arxiv: org/ /twrg//b/////////////////////a// 10../666.0000 02601.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert:
Pre-
training of deep bidirectional transformers for language understanding, 2018. URL
pitps::/aarxe wtt./orggg//a/oooeeeeeoo
abs 04805.
Anila Fejzo, Alain Desrochers, and Hélène Deacon. The acquisition of derivational morphology in children. pp. 112-132, 04 2018. doi: 10.4524/9783115229144
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification, 2016. URL http:: /ttp::///axxvv...oee /abs///////e 607. 759.
Wang Ling, Tiago Luis, Luis Marujo, Ramón Fernandez Astudillo, Silvio Amir, Chris Dyer, Alan W. Black, and Isabel Trancoso. Finding function form: Compositional character models for open vocabulary word representation, 2015. URL https: atxiv.iiiiv.///.v www.aiii..g//a/..e. 1008 02096.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint 1301.378. 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. tributed representations of words and phrases and their compositionality, 2013b.
Dis-
URL
https: arxiv. wwwp.xiwwwww..o///toot/to/ org/abss/////oooooooog.o.o... 1310. 4546.
Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word rep- resentation. In Proceedings of the 2014 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 1532-1544 Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/W1D1141112.. URL http:: http::s:s///cllaahhllllogo/ 1111:///owooooooo.//1.o0/ wwwp/orgg/11./. -126...
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Ken- ton Lee, and Luke Zettlemoyer. Deep contextualized word representations, 2018. URL
https://arxil: org/ org/abs/////ooooooooogooo..e.cs/ 1101.0502666
Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. 2017. doi:
10-4850/ARXIV..70600333


Reviews:
[{"review_id": "s8hIaPnn39", "reviewer": "Reviewer_L7qJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the morphological relationships encoded in word embeddings learned from traditional approaches such as Word2Vec, GloVe, and FastText. The authors used the analogical reasoning task as an instrument to investigate morphological information. The experiments conducted on 6,730 pairs of words (-affix) revealed that the word embeddings failed the morphological analogy task. A series of morphological investigations showed that there was no consistent trend.", "summary_of_the_review": "I do not recommend the paper to ICLR 2023 because the paper lacks a significant literature review and the impact of the paper is not relevant to the current NLP research.", "strength_and_weaknesses": "Strengths:\n1. The paper presented a series of investigations and reveal interesting insights about word embeddings.\n\nWeaknesses:\n1. Since most of the NLP methods have moved beyond standalone word embeddings, this makes the insight revealed in this paper not as relevant. The authors could try to conduct a similar study on contextual embedding approaches.\n2. What is being captured in word embeddings could depend on the context size (shorter context, more syntactic), this is an essential aspect to include in the study.\n3. The key insight from the study is not quite novel. There were several works augmenting morphological information to the word embeddings (https://aclanthology.org/N15-1140.pdf or https://arxiv.org/abs/1407.1687). Perhaps, the authors could try to study these methods.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "nFmV0e9KZh", "reviewer": "Reviewer_CtMu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims at revisiting study on analogy capabilities of word2vec models. More specifically, morphological task is studied with Glove, fasText and word2vec (CBOW) word embeddings. Contrary to examples suggested in the original word2vec paper, word2vec model performs poorly on a line of morphological tasks. ", "summary_of_the_review": "Reasonable questions to address, but (i) has a limited novelty, and (ii) do not put work in context of more modern approaches such as Transformers with BPE approach to morphology. ", "strength_and_weaknesses": "Strengths: \n\n- Paper represents a clear useful nugget of research answering a few research questions. It allows for better understanding of word2vec capabilities. These may be useful for building some morphological analysers. \n\nWeaknesses: \n\n- Little novelty. Paper clearly adds some knowledge about word embeddings, but I would expect a more large scale study from a full paper. \n\n- The static word embeddings we gaining their momentum about 10 years ago and since then Transformers and other deep pre-trained language models addressed the problem of morphology in a different way i.e. through BPE encoding. Authors do not mention or put in context of these more modern development their research. Maybe its a good way to improve the paper though larding the scope and adding more models to consider (ask questions how they deal with morphology).  One opportunity is to to extract contextualised word embedding from these models and add to comparison. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CYg7Yebfo1t", "reviewer": "Reviewer_uwiD", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The semantic understanding of word representation is often established through the analogy completion task where vector arithmetics such as King - man + woman = Queen example is notoriously used to establish the static embedding models like Word2Vec learn semantic relationships. Learning word morphology is fundamental to learning a language and the authors conjecture that the static embedding models don’t capture this knowledge by formulating analogy completion across word stems modified by affixes. The examples like sorted: sort as covered: cover fail. The authors also analyzed the distribution of distance vectors to see if morphological features are well represented in the embedding space. \n", "summary_of_the_review": "The authors used an off-the-shelf method to question the existence of morphological knowledge in the static embedding models. Their approach is very limited for the claim made in the paper, especially when previous research has shown through other methods that these models do learn word morphology. The paper does not have methodical novelty, and the arguments are weak. Also the paper does not experiment with or discuss state-of-the-art to draw any comparative analysis. It is not clear how the community can benefit from this paper in any sense.  ", "strength_and_weaknesses": "Weaknesses\n\nWhile it is interesting to note that analogy completion does not function the same way for morphology in static models, I think it is still too strong to claim that these models do not encode morphological relationships. Previous work has shown that representation from static embedding such as word2vec, GloVe etc. capture morphology, by training classifiers from the feature vectors towards the task of morphological tagging. A lot of work on probing linguistic properties for static and contextualized embedding have shown that the models capture word morphology during earlier epochs of training. For example see\n\nUnderstanding the learning dynamics of word2vec\n\nAnother recent work is from Musil et al (2019) where they use unsupervised clustering of word2vec vectors to capture derivational relationships. \n\nI could not understand why the methodology can’t be applied to contextualized embedding? Studying deep transformer pLMs would be much more interesting where previous research has shown that lower layers learn morphological features whereas the higher and upper layers capture semantic properties using probes. The earlier layers predominantly reconstruct word structure which is disintegrated from subword segmentation. I am hopeful that applying the method in the earlier layers of pLMs would reveal interesting findings.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "9GOBmHOqB5", "reviewer": "Reviewer_REfa", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors aim to study the ability of pre-trained word embeddings in representing morphological relations between words. The authors constructed a dataset of words with affixes and their roots. The authors conduct experiments on word analogy tasks concerning word affixes and explore the distribution of the difference of the word vectors of word pairs.", "summary_of_the_review": "The paper requires significant improvements in all key aspects to meet the bar for publication. I suggest that the authors start with a thorough survey and review of existing work and then perhaps rethink the approach and the story. ", "strength_and_weaknesses": "## Strengths\nNone\n\n## Weaknesses\n1. Related works are severely under-surveyed. Pos tagging and morphological segmentations (e.g. Ruokolainen et al., 2016) are two popular tasks in NLP. A number of works have examined the abilities of word embedding in capturing morphological information (e.g. Botha et al., ICML 2014; Cao & Rei, 2016; Cotterell & Schütze, TACL 2018; Jinman et al., Findings 2020), or POS-tagging (e.g. Ling et al., NAACL 2015; Pinter et al., EMNLP 2017; Zhao et al., EMNLP 2018). Just name a few. The authors mention or discuss none of the existing investigations. I suggest the authors read some of the surveys, e.g. Yang et al. (2018), recent evaluations of word embedding models (e.g. Wang et al., 2019; Ghannay et al., LREC 2016), or try search on Google Scholar using keywords like word2vec, word embedding, syntax, morphology, pos tagging.\n2. The descriptions of previous works (word2vec, GloVe and FastText) are misleading. One main difference between word2vec and GloVe is that word2vec uses localized losses similar to stochastic gradient descent whereas GloVe uses matrix factorization. FastText uses a similar objective as word2vec but added on a bunch of tricks for performance and efficiency.\n3. Important details about dataset missing, affecting the validity and reproducibility. In Sec 2.3, the list of chosen affixes is not provided. The criteria and the list of chosen roots is not described. It is not clear if the authors take careful steps beyond simple string matching to make sure if the selected \"affix pairs\" are true morphological conjugates, rather than mere look-alikes.\n4. There exists carefully crafted datasets for word affixes, e.g. Lazaridou et al. (ACL 2013). The authors do not discuss or justify the need or benefit of their new dataset.\n5. Important details about experiment setting missing, affecting the validity and reproducibility. In Section 3, the authors do not specify which versions of pre-trained w2v, glove, and fasttext are used. The authors only mention that they use the 300d version. However, each of the embedding methods has many publicly available checkpoints trained over different corpus, in different (combination of)  languages, and with different hyperparameters. Some of the hyperparameters have been shown to drastically change the downstream performance of the word vectors,  e.g subword size of fastText (Bojanowski et al., TACL 2017, Table 4), corpus domain (Lat et al., 2016), number of negative examples (Levy et al., 2014), window size. Window size has especially been shown to affect the syntax-semantics trade-off in learned embeddings.\n6. Experiment support is weak for the claims. The authors heavily rely on anecdotal examples and study of a single case rather than quantitative analysis. In Table 1, 2, the authors use hand-picked examples to show that word embedding is able to complete semantic analogy but not syntactic analogy, where top-k accuracy or the error/correlation between the predicted and the correct word vector could be more convincing. In Fig 1, the authors use histogram to show the distribution of norms of word vectors related to \"-ed\", where numeric statistics (mean, variance, etc) of all cases could be more convincing. \n7. Findings are trivial or known. The ability of word embeddings on word analogy (Sec 2.2 and Table 1) is known knowledge and has been reported in many previous works. Search works that use WordSim353 (Finkelstein et al., 2001), s RareWord (Luong et al., 2013), Card-660 (Pilehvar et al., 2018) and other similar word analogy benchmark. Sec 3.5, it is known that w2v vector norms correlated to word rank.\n8. Dubious claims. See LPs regarding P8.\n\n## Localized points (LPs)\n1. Use \\citet{} for \"Smith et al. (2022)\" style citation and \\citep{} for  \"(Smith et al., 2022)\" style citation.\n2. P4, Footnote 2. Link to the common prefixes not accessible.\n3. P4, Sec 2.3. Do you manually inspect and exclude \"implausible\" instances? Finding true morphological affixes is much trickier than string matching. For example, \"replacement\" can be analyzed as replace/ment but not re/placement.\n4. P5, Sec 3. Where is Figure 3.3, Figure 3.4 and Table 3.5? They probably mean Figure 1 (middle), Figure 1 (bottom), Figure 2. Please fix this. Maybe use \\label and \\ref?\n5. P5, Sec 3.3. Could report variance for all affixes instead of just showing a plot for one affix.\n6. P5, Sec 3.4. I guess the authors wanted to say the distribution of difference vector has a large variance. Computing the variance would be much clearer.\n7. P8, \"Furthermore, statistical analysis of word-pair distance-vectors suggests that the words in question are not near to each other in embedding space, and their difference vectors do not share a common direction.\" Why should they be near? The evidence in this paper (Figure 1 middle) indeed show that the vector for \"[root]ed\" is indeed closer to \"[root]\" than random word pairs.\n8. P8, \"Without the ability to quantitatively represent the effect of morphological modification of words, it will be difficult to develop language models that reproduce humans’ ability to generate and understand novel words, as well as to generalize from much smaller sets of textual examples.\" We have seen a plethora of evidence of the two.\n\n## References\n1. Two/Too Simple Adaptations of Word2Vec for Syntax Problems (Ling et al., NAACL 2015), https://aclanthology.org/N15-1142/\n2. Compositional Morphology for Word Representations and Language Modelling. Jan Botha, Phil Blunsom Proceedings of the 31st International Conference on Machine Learning, PMLR 32(2):1899-1907, 2014. https://proceedings.mlr.press/v32/botha14.html\n3. A Joint Model for Word Embedding and Word Morphology (Cao & Rei, 2016), https://aclanthology.org/W16-1603/\n4. Ryan Cotterell, Hinrich Schütze; Joint Semantic Synthesis and Morphological Analysis of the Derived Word. Transactions of the Association for Computational Linguistics 2018; 6 33–48. doi: https://doi.org/10.1162/tacl_a_00003\n5. PBoS: Probabilistic Bag-of-Subwords for Generalizing Word Embedding (Jinman et al., Findings 2020), https://aclanthology.org/2020.findings-emnlp.53/\n6. Mimicking Word Embeddings using Subword RNNs (Pinter et al., EMNLP 2017), https://aclanthology.org/D17-1010/\n7. Generalizing Word Embeddings using Bag of Subwords (Zhao et al., EMNLP 2018), https://aclanthology.org/D18-1059/\n8. Li, Yang, and Tao Yang. \"Word embedding for understanding natural language: a survey.\" Guide to big data applications. Springer, Cham, 2018. 83-104. https://link.springer.com/chapter/10.1007/978-3-319-53817-4_4\n9. Teemu Ruokolainen, Oskar Kohonen, Kairit Sirts, Stig-Arne Grönroos, Mikko Kurimo, Sami Virpioja; A Comparative Study of Minimally Supervised Morphological Segmentation. Computational Linguistics 2016; 42 (1): 91–120. doi: https://doi.org/10.1162/COLI_a_00243\n10. Wang, Bin, et al. \"Evaluating word embedding models: methods and experimental results.\" APSIPA transactions on signal and information processing 8 (2019). https://www.cambridge.org/core/journals/apsipa-transactions-on-signal-and-information-processing/article/evaluating-word-embedding-models-methods-and-experimental-results/EDF43F837150B94E71DBB36B28B85E79\n11. Word Embedding Evaluation and Combination (Ghannay et al., LREC 2016), https://aclanthology.org/L16-1046/\n12. Compositional-ly Derived Representations of Morphologically Complex Words in Distributional Semantics (Lazaridou et al., ACL 2013), https://aclanthology.org/P13-1149/\n13. Piotr Bojanowski, Edouard Grave, Armand Joulin, Tomas Mikolov; Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics 2017; 5 135–146. doi: https://doi.org/10.1162/tacl_a_00051\n14. Lai, Siwei, et al. \"How to generate a good word embedding.\" IEEE Intelligent Systems 31.6 (2016): 5-14. https://ieeexplore.ieee.org/abstract/document/7478417, https://arxiv.org/abs/1507.05523\n15. Levy, Omer, and Yoav Goldberg. \"Neural word embedding as implicit matrix factorization.\" Advances in neural information processing systems 27 (2014).  https://proceedings.neurips.cc/paper/2014/hash/feab05aa91085b7a8012516bc3533958-Abstract.html\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject