Self-supervised Post-processing Method to Enrich Pretrained Word Vectors
Authors: Hwiyeol Jo
Venue: EMNLP 2023 Findings (2023)
Keywords: Retrofitting
Word Embedding
Word Semantics
TL;DR: We propose an extension to a retrofitting approach (especially, extrofitting) for semantically enriching word vectors that does not require external lexical constraints.

Abstract: Retrofitting techniques, which inject external resources into word representations, have compensated for the weakness of distributed representations in semantic and relational knowledge between words. However, the previous methods require additional external resources and strongly depend on the lexicon. To address the issues, we propose a simple extension of extrofitting, self-supervised extrofitting: extrofitting by its own word vector distribution. Our methods improve the vanilla embeddings on all of word similarity tasks without any external resources. Moreover, the method is also effective in various languages, which implies that our method will be useful in lexicon-scarce languages. As downstream tasks, we show its benefits in dialogue state tracking and text classification tasks, reporting better and generalized results compared to other word vector specialization methods.

Self-supervised Post-processing Method to Enrich Pretrained Word Vectors
Hwiyeol Jo NAVER, Search US fiiyeolj@gmailn
Abstract
applied to any kinds of pretrained word vectors. and (3) retrofitting can make word vectors special- ize a specific task.
Retrofitting techniques, which inject external resources into word representations, have com- pensated for the weakness distributed rep- resentations in semantic and relational knowl- edge between words. However, the previous methods require additional external resources and strongly depend the lexicon. To ad- dress the issues, we propose a simple extension extrofitting, self-supervised extrofitting: ex- trofitting its word vector distribution. Our methods improve the vamilla embeddings on all of word similarity tasks without any ex- ternal resources. Moreover, the method is also effective in various languages, which implies that our method will be useful in lexicon-scaree scarce languages. As downstream tasks, we show its benefits dialogue state tracking and text clas- sification tasks, reporting better and general- ized results compared to other word vector spe- cialization methods.'
The previous studies focusing on explicit retrofitting have used manually defined or learned functions to make synonyms close and antonyms distant (see the details in $4). As result, their approaches strongly rely external resources.
Furthermore, we agree that making synonyms close together is somewhat reasonable, even though the synonyms have different nuances in some texts. However, other kinds of relations, such antonyms, should be further investigated. For in- stance, love and hate are generally grouped as antonyms. Most of the previous studies have made the words distant from each other, but the words definitely share the meaning of emotion in their rep- resentations. We thus conjecture that the methods are not generalized well.
In this paper, propose word vector enrich- ment based on extrofitting (Jo and Choi, 2018):
1
Introduction
Static word vectors are still widely used in natural language tasks despite the recent trends of contex- tualized models. For example, in wide study of Dialogue State Tracking (DST) (Feng et al., 2021), contextualized models performed worse than static word embeddings. It seems reasonable that the con- textualized models do not perform well in such lack of context, which implies that static embed- dings are still useful.
Self-supervised extrofitting that extends ex- trofitting for enriching word vectors without using external semantic lexicons. This method can resolve limitation of post-processing ap- proaches, which requires well well-defined seman- tic lexicons. We highlight its usefulness in (relatively) lexicon-scarce languages.
We report the effects of word vector post- processing on several downstream tasks show the generalization of the word vectors. Our methods consistently improve model per- formances the fundamental tasks. In con- trast, other post- post-processing methods degrade the performance.
To make better word vectors, we focus on retrofitting ideas (also called word vector post- processing), which injects the semantic information from external resources by modifying the values of pretrained word vectors (Faruqui al. (2015); Mrkšić et al. (2016a); inter alia). The benefits of post-processing methods are that (1) the methods can reflect additional resources into the word vec- tors without re- re-training, (2) the methods can be
2 Preliminary
Extrofitting Extrofitting (Jo and Choi, 2018) ex- pands word embedding matrix by concatenating
hattp://githin hwtp:/ttu....ommwiieeteeeee com /omm/hiieoloo/Selfeee.

---

W with rw:
is vocabulary and the y-axis is (unknown) seman- tic dimension. Thus, as researchers have use the decomposition of the term-document matrix to get the term-vectors in LSA, the decomposition of the term-semantic matrix can be considered to represent term-vectors (US) and semantic-vectory (S V.). This intuition corresponds to the traditional methods in vector embedding, verified in Levy and Goldberg (2014); Allen et al. (2019).
Expand(W.c) c)
if w otherwise
WWe @ Tw
Hw
where Pw is the mean value of elements in word vector w. L denotes semantic lexicons, and de- notes the same class (synonym pairs). In other words, Expand makes an additional dimension per the word vector, and fill with the same value if the pairs are synonym.
With the idea, we first decompose word embed- dings matrix W to make latent representations as follows:
Next, Trans calculates transform matrix given the matrix W:
W2 = U
As described above, we can get term-vectors, which are word representations in k-dimensional latent space, by computing UnS. Adjusting the dimen- sion of latent space in $2), we calculate seman- tically related words using cosine similarity. For every word, we calculate its cosine similarity to all other words in the vocabulary. If the similarity exceeds a predetermined threshold, we group the words. Note that a group can contain only single word. This process is repeated iteratively for each ungrouped word until all words are clustered. We use the set of semantically related words (c') as the class (synonym pairs, c) of extrofitting instead of semantic lexicons:
Trans(W, c)
TUT Ed - M14 4) UI =argmax NUT E. Ex(x Mc) - Ho
where x is word vector, is class. The overall average of x is μ,, and the class average in class i is denoted by μú: This formula finds transform matrix U, which minimizes the variance within the same class and maximizes the variance between different classes. Each class is defined as the index of synonym pairs.
In the end, Extrofitting is formulated as follows:
Ex,ro(W,c)
SelfExtro W. c')
=Trans(Expand(W.c.cc Expand(xpp
=Trans(ExpandWW.' c'), "'EExxdp.WWW c')
Latent Semantic Analysis (LSA) LSA (Lan- dauer and Dumais, 1997) has been used to extract the relation of data through latent variables. LSA is based on Singular Value Decomposition (SVD), which decomposes a matrix as follows:
To sum up, we use the principle of vanilla ex- trofitting (Jo and Choi, 2018), which utilizes LDA algorithms to group in-domain instances (in this case, word vectors). Simultaneously, the algorithm pushes out-domain instances apart. Instead of us- ing pre-defined lexicons for extrofitting, the idea of LSA to get semantically related words. Even if the number of extracted word pairs are small or the words are not meaningful, the pro- cess of making out-domain instances far can make better representations.
USVT.
where S is diagonal matrix with singular values, and U and V are the orthogonal eigenvectors. We can select top-k singular values to represent matrix A in k-dimensional latent space. Then U and are re-defined as U₃= E Nok and V E RIxN re- spectively, with diagonal matrix S E Rlxx When we use LSA for topic modeling, A is defined as a term-document matrix. Then, U&S& & and S.V are considered as term vectors and document vectors in the k-dimensional latent space, respectively.
In the experiments, we start with pretrained GloVe (Pennington et al., 2014) (if we do not men- tion it explicitly) and set a threshold that determines whether a pair of words are semantically related. We use a high threshold (0.9 of cosine similarity) since type Il error is rather better than type error.
3 Self-supervised Extrofitting
Related Works
We consider the word embedding matrix as the term-document matrix; The x-axis of the matrix
The first successful post-processing approach was Retrofitting (Faruqui et al., 2015), which modi-

---

GloVe (glove. 42B)
fastText (wiki nnws)
W(5)
(())
RW
ME
SE
SL
SV
W(s)
W(r)
RW
ME
SE
SL.
SV
Raw
.39
.74
.57
.37
.22
.82
.62
.51
.80
.64
.44
.35
SelfExtro, 100
.73
.50
.84
.66
.51
.39
.81
.70
.56
.82
.66
.50
.41
SelfExtro, k=200
.79
.72
.49
.84
.66
.51
.39
.81
.70
.57
.22
.65
.51
.42
SelfExtro, 300
.78
.69
.48
.84
.64
.50
.37
.81
.71
.57
.83
.64
.51
.42
DordNeettEExtro
.80
.74
.49
83
.66
.49
.36
.78
.67
.51
.80
.63
.50
.40
vecmap
.53
.72
.52
.35
.20
.81
.62
.49
.80
.59
.44
.34
Table Spearman's correlation of self-supervised extrofitted pretrained word embeddings. & denotes the dimension of latent space that is used to extract semantically related words (see $3). Ablation studies in threshold are presented in Appendix A.4.
fied word vectors by weighted averaging the word vectors with semantic lexicons. Extending from the simple idea, Counter-fitting (Mrkšić et al., 2016a) used both synonym pairs to collect word vectors and antonym pairs to make word vec- tors the gene from one another. Next, Para- gram embeddings (Wieting et al., 2015) used synonyms and negative sampling to collect the word vectors. Borrowing attract-terms from the Paragram embeddings and adding repel-terms, Attract-Repel (Mrkšić et al., 2017) injected lin- guistic constraints into word vectors through prede- lin-
scores and the cosine similarity of the word vector pairs.
We use 6 datasets in English: WordSim353
(W(s) for similarity and W(r) for relation) (Finkel-
stein et al., 2001), RareWord (RW) (Luong et al.,
2013), MEN-3K (ME) (Bruni et al., 2014), SemEval al., 2017), SimLex- (SE) Gamacho-Colladee
999 (SL) (Hill et al., 2015), and Vim
(SV) (Gerz al., 2016).
Results. In Table 1, self-supervised extrofitting (SelfExtro) improves the performance on all the word similarity datasets when compared with the popular pretrained word vectors, GloVe (Penning- ton et al., 2014) and fast (Bojanowski et al., 2016)². The result implies the pretrained word vec- tors can enriched by our method, which means does not require any semantic lexicons to make better embeddings. In addition, Sel fExtro shows better performances with the original extrofitting (+ (+WordNet) on most of the evaluations.
fined cost function with mono-/cross-ling guistic constraints. Explicit Retrofitting (ER- CNT) (Glavaš and Vulić, 2018) directly learned mapping functions of linguistic constraints with deep neural network architectures. They then used the functions retrofit the word vectors. Post- Specialization (Vulié et al., 2018; Ponti et al., 2018) resolved the problem of the previous mod- els that only updates words external lexicons. Some works have used cross-lingual resources to get further semantic information (Vulić et al., 2019; Kamath et al., 2019).
We also investigate the extracted semantic in- formation (see Appendix A.2). The extracted in- formation hardly seems synonyms, but we can see that some similar words are grouped. As ex- trofitting affects all the word vectors, every word can benefit from the other words being enriched. In other words, although the number of extracted information is small, this simple extension makes extrofitting fully utilize its advantages.
While the previous methods utilized text-level resources, vecmap (Artetxe et al., 2018) used other word embeddings external resources. 2 For fair comparisons, we input the same word vectors for source and target embeddings of the method. Recently, BERT-based model LexFit (Vulić et al., 2021) was derived, but the method requires external resources.
Qualitative examples of self-supervised ex- trofitting are presented in Table 2. Although the representations lose similarity scores, the similar words become diverse and reasonable. Additional qualitative examples shown in Appendix A.5. Our proposed Sel SeltEx is also potentially use- ful when semantic lexicon resources are scarce, such as in many non-English languages. In Ta-
5 Experiment 1: Word Similarity Tasks
Settings. Word similarity datasets consist of word pairs with human-rated similarity scores be- tween the words and models calculate Spearman's 's correlation (Daniel, 1990) between the similarity
Wo selected unsupervised version of vecmap because it performs hetter than identical version.
wword2yec could also he considered as hase model, but it takes too much time due to its large vocabulary size.

---

Cuc Word
Method
Top-19 10 Nearest Nords(Cosine Similarity Score)
loved(,7745), i(.7338), (wves..3111), know lovingt. 7263),
Vanilla
really(.7196), (lways(.7193) want(,1122). (712), think(.7110)
love
lovedK.7122), loving(.67344 loves(6648)) adore(.6343 passion(.6333)
+SelfExtro
luv(.6326), hope( 6256). i( ((.620)) want(.6209) hate(.681))
dont(,7318), stupid(.7193) hates(. hat0s, think[, why(.6943)
Vanilla
love(,6928), bating(,6922 (927), hated(.686) shit(, (hit(,6847, know( (6055
hate
hating(.6707), hatred( 6650), donN, 6580J, matcs(.6529)
+SelfExtro
(islike(.644))
despise( 6309), hated(, 6306), stupidr (266)),6666....... think( (ilkk),6244)... love(.6181)
alive(.64 6478), gone(,6450), love(.638) never(.6267) againi .6249).
Vanilla
yours( .6238, lifc(.6171), alone(.6153) anymore(.6129), always((6093)
forever
(iernally(66297), alive(.581a (5814), yours(.5711) life(.5532),, (mymoro(.5524))
+SelfExtro
(iernity(.5448) eternall (5577), gone(.5447), permanently(.5444 againl.5414)
8053), living,. .7134), things(,686)), way(.6852), mind(.6844)
Vanilla
(h79), much( much(.6733), love(.6719), because(.6716)
work(.6706
life
(.ve(,.801),, living(.65.8) journey(.60444 things(.5920). (ietime(.5892)) happiness(.574p9 love(.5743), way(.5728)
(erryda(.60338) mind(,5922)
+Sel.fExtro
Table 2: List of top-10 nearest words of cue words in different post-processing methods. We report cosine similarity scores of words-love, hate. forever, and life. Underline and bold indicate the difference between the list from vanilla vectors and sule-supe extrofitted vectors, respectively.
CZ
IT((/r)
RU(s/r)
GF(s/r)
JGA with
JGA with A-R Lexicon
Model (Resource)
fastText
240
.377
.188
.28
.215
.335
.169
WordNet
k=109
450
.453
370
525
.411
.425
.261
GloVe
7986
k=200
443
444
.359
545
426
.431
.261
Retrofitting
.792(.03)
.793(01)
k=300
.451
.456
339
.399
.426
.449
.271
Paragram(Syn)
.670(05)
.657(.03
Extro(Syn)
821(.03)
.800.01)
Table 3: Spearman's 's correlation of self-supervised ex- trofitting on WordSim datasets in 4 languages: Czech (CZ), Italian (IT), Russian (RU), and German (GE), k denotes extracted semantic information from fastText in k-dimensional latent space.
Counter- (Stt-AAAn))
.65((01)
.630(.3)
(Syn-Aat)
.671(.04
.67((03)
vecmap SelfExtro(-)
.772(01)
.825(.02)
Table 5: Joint goal accuracy and its standard deviation in WOZ 2.0 datasets. Despite the fact that used the original Github code and data, the result is different from their report. Furthermore, our settings show different trends from what they reported.
CY
ET
FI
FR
HE
PL
RU
ES
FT
.187
213
455
.314
.371
2283
305
385
k= 180
2001
223
.45
340
400
306
334
425
k-200
.209
215
.63
.399
.399
.299
306
.23
k=300
.144
219
.664
.388
.02
.005
.308
.421
claimed that the performance on the task is related to word vector specialization in that the model has to identify the flow of dialogue with only a few words, which seems reasonable. Thus, we use the NBT and check the model performances with our post-processed embeddings. Refer to the papers cited above for the details of DST.
Table 4: Spearman's correlation of self-supervised ex- trofitting on Multi-SimLex datasets in languages: Welsh (CY), Estonian (ET), Finnish (FI), French (FR),
Hebrew (HE), Polish (PL), Russian (RU), and Spanish (ES). Other languages cannot be found in fast.ext.
ble and Table 4, we use pretrained fastText for WordSim datasets in 4 languages and Multi- SimLex (Vulié et al., 2020) in 8 languages, respec- tively. Sel fExtro significantly increases the per- formance in all the languages.
Table 5 shows the performance of DST in Wizard-of-Oz 2.0 dataset (Wen et al., 2017). The results show that the semantic specialization (e.g.. Attract-Rese Repuel does not increase the performance. In contrast, Extro and Sel fExtro show better performance than vanilla GloVe.
Results.
6 Experiment 2: Dialogue State Tracking
We additionally experiment with the lexicons (both synonyms and antonyms) included in the Github of Attract-Repel*. It shows only a little
Settings. Previous works (Mrkšić al., 201 6a, 2017; Vulić et al., 2019) showed that word vector post-processing useful for dialogue state tracking (DST) (Young et al., 2010). Using Neural Belief' Tracker (NBT) (Mrkšić et al., 2016b), they have
httpps: www.ttt.........eeeeeeeeeeee.ete cow/wmrksic/////ccccoc.coommcmce neural belief- tracker httpp: http:::///ttte thub.

---

Freezed Vectory
DBpedia
Vahoo(Up)
Yahoo(Low)
Yelp
AGNeww
IMDB
GloVe
98.31
71.97s0.76
49.35x0.51
61.5550.35
90.66s0.79
87,47+1.98
Retrofit(Syn)
81.9245.11
44.72x2.77
22.31
48.87a0.40
82.04=1.499
63.40+3.53
PSragram(Sym)
86.88=0.36
62.37=0.36
40.25=0.19
50.04=0.20
80.16=0.14
74.77+0.43
(Xyr)(Syn)
98,52=0.05
72.94=0.10
50.01+0.39
62.70=0006
91.36=0029
89.40=0.20
Counter-fit(Syn-AAn
73.26=1.29
59.70=0.19
37.80=0.08
47.97=0.45
76,47=1.45
66.54=0.48
Mut-RepellSSynAnn
87.52=0.12
62.51=0.41
39.75+1.15
50.08=0.29
79.80 19.80=0.93
70.31
(Syt-ptc()ym-Ant)
76.78=0.79
59.45=0.15
37.16+0.35
49.1620.64
76.06=0.62
66.11=0.79
vecmap
98.01=0.27
68.99=0.68
4..57=1.14
62.2220.25
89.87=0.88
83.80=1.33
Sel.fExtro(-)
98.44=0.05
72.41=0.26
49.79=0.37
62.97=0.18
90.93=0.11
89.36=0.41
Trainable Vectors
DBpedia
Yahoo(Up)
Yahoo(Low)
Yelp
AGNews
IMDB
GloVe
+0.61
73.45=0.70
49.30#0.61
63.01=0.42
91.54:0.45
88.82+0,55
(Strofit(Syn)
98.16=00.05
6..15=1.73
13.14+1.31
61.7440.33
19621
82.77+1.27
Paragram(Syn)
98. 18.14:0.04
66.32=0.27
42.85+0.43
61.96:0,22
89.91=0.19
83.14+1.15
(xtro(Syn)
00.61
73.50=0.19
50.43+0.30
63.2240.19
91.78=0.29
89.74+0.28
Counter (Synn--An)
98.07=0.11 10
63.74=0.53
40.83+0.20
61.89=0.17
90.09=1.23
83.78+1.84
Att -e-RRepe(SynnAnt)
98.1 0.007
66.05=0.42
42.68+0.43
61 80:40.0
90.05
83.74 1,30
PSyt
98. 0.00.5
03.88=0.36
40,49+0,51
61 03
90,48=0.38
84,54
vecmap(-) SelfExtro(-)
=0.27
68.99=0.68
44.57+1.14
62.22+0.25
89.87=0.88
83.80
98.58
73.02=0.15
50.05+00.33
63.25
01.61
80.98
Table 6: 10 times average accuracy initialized with the post-processed word vectors. We experiment with our methods in 2 different settings: fixed word vectors and trainable word vectors.
performance gain; we thus guess that the difference in DST performance comes from lexical resources or fine-tuning NBT rather than specialization.
the original GloVe. It affects the performance of the fundamental tasks, which are largely degraded. On the other hand, our method enriches the word vector in general, resulting in marginal but certain improvements.
7 Experiment 3: Text Classification
Contextual representations provide substantial improvements in NLP tasks, but the representa- tions lack semantic information due to their nature. Since the context and the semantics are different types of information that can enrich representations, we believe our approaches might further improve the contextual representations.
Settings. We experiment with our methods in 2 different settings: fixed word vectors and trainable word vectors. When the word vectors are fixed, can evaluate the usefulness of the word vectors per se. When the word vectors are trainable, we can see the improvement of the model performance in a conventional training setting. The dataset and classifier are described in Appendix A.3.
Conclusion
We report the performances in Table 6. The classifier initialized with Sel fExtro outper- forms the vanilla GloVe and performs on par with the original extrofitting. On the other hand, al- though the previous word vector post-procet prosesrocsssing methods specialize well on domain-specific task, the approaches failed to be generalized; they show large performance deterioration despite the added general semantic information.
Results.
We develop a self-supervised retrofitting model that enriches word vectors without semantic lex- icons. The method utilizes its own distribution of word vectors get better representations. In the Exp. we show that our method can im- prove the performance on word similarity tasks and present qualitative examples. plied to other pretrained word vectors, resulting in better performances on all the word similar- ity datasets. Sel fExtro also has potential advan- tages in lexicon-scarce languages. In the Exp. 2 and 3, we presented the effect of post-processing methods on downstream tasks. Our method shows marginal but certain improvements, while other post post-processed word vectors largely degrade the performances, which seems the result of losing generalization.
can be ap-
8 Diseussion
The specialized word vectors do not warrant bet- ter performance. We conjecture that the methods trade off catastrophic forgetting against semantic specialization even though the previous methods successfully specialize their vectors into given lexi- cons, losing the pretrained information encoded in

---

10 Limitations
cross mooss-liggul mappings of word embeddings. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational I Linguistics (Volume 1: Long Papers), pages 789-798.
Usefulness compared with contextual embed- dings Contextual embeddings are recently domi- nant NLP tasks, whereas static word embeddings have become less frequently used. However, it does not mean static word embeddings are not useful. Although we can assume that the amount of pre- training dataset and training resources are similar, the cost of inference is much cheaper at static word embeddings. Furthermore, static word embeddings perform better when task lacks enough context (e.g.. word similarity tasks). It will be interesting future work to retrofit contextual embeddings, but it is out of scope in this paper.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vec- tors with subword information. arXiv preprint arXiv:160.004606
Elia Bruni, N Tram, Marco Baroni, et al. 2014. Multi- modal distributional semanties. The Journal of Artifi- cial Intelligence Research, 49:1-47.
Jose Camacho-Collade Mohammad Taher Pilehvar, Nigel Collier, and Roberto Navigli. 2017. Semeval- 2017 task 2: Multilingual and cross-lingual semantic word similarity. In Proceedings of the I 1Ith Interna- lional Workshop on Semantic Evaluation (SemEval- 2017), pages 15-26.
The use of antonyms Although we believe that there are no definitely opposite meanings of the word (c.g., love and hate share the sense of emo- tion), previous works (see $4) that utilize antonyms showed significant improvement in word similar- ity tasks. However, compared to the methods, self-super extrofitting explicitly considers syn- onyms only, but implicitly expects antonyms to be distant while maximizing the variances of in- ouss//ut-offclaas word vectors. Also, the process of self-supervised extrofitting makes it hard to in- corporate other kinds of word relations.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task I: Semantic textual similarity-ulliiinuua and cross-lingual focused evaluation. arXiv preprint arXiv:1788.00055.
Ming-Wos Mei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008, Importance of semantic repre- sentation: Dataless classification. In AAAI, volume 2, pages 830-835.
Wayne W Daniel. 1990. Spearman rank correlation coefficient. Applied nonparametric statistics, pages 358-365.
The method is only a simple linear projection Both extrofitting and self-supervised extrofitting use linear projections in enriching vectors, follow- ing a traditional NLP method. The linear model might not be the best to reflect word relations in vector spaces, but we believe that it a simple yet effective method, as we still calculate lots of things (e.g., distance) in linear way.
Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar. Chris Dyer, Eduard Hovy, and Noah A Smith. 2015. Retrofitting word vectors to semantic lexicons. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 1606-1615.
Yue Feng, Yang Wang. and Hang Li. 2021. A sequence- to to-sequence approach to dialogue state tracking. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers). pages 1714- 1725.
Acknowledgement
The author would like to thank all the reviewers in several rounds of submission, a total of years. Lastly, I am grateful to Alice Lee for her help in qualitative analysis.
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Ey- tan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedings of the 10th in- ternational conference on World Wide Web, pages 406-414, ACM.
References
Carl Allen, Ivana Balazevic, and Timothy Hospedales. 2019. What the vec? towards probabilistically grounded embeddings. In Advances Neural In- formation Processing Systems, pages 7465-7475.
Daniela Gerz, Ivan Vulić, Felix Hill, Roi Reichart, and Anna Korhonen. 2016. Simverb-3500: A large-scale evaluation set of verb similarity. In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pages 2173-2182.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised

---

Goran Glavaš and Ivan Vulić. 2018. Explicit retrofitting of distributional word vectors. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), vol- ume 1, pages 34 45.
Proceedings of the 49th annual meeting of the associ- ation for computational linguisties: Human language pechnologies-volure 1, pages 142-150. Association for Computational Linguistics.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-44)
Richard HR Hahnloser, Rahul Sarpeshkar, Misha A Ma- howald, Rodney J Douglas, and H Sebastian Seung. 2000. Digital selection and analogue amplification coexist in a cortex-inspires silicon circuit. Nature, 405(6789):947.
Nikola Mrkšić, Diarmuid O Séaghdha, Blaise Thom- son, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsiiet Wen, and Steve Young, 2016a. Counter- fitting word vectors linguistic con- straints. arXiv preprint arXiv: 660300082..
Felix Hill, Roi Reichart, and Anna Korhonen. 2015. Simlex-999 Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguis- tics, 41(():665-695 695.
Nikola Mrkšić, Diarmuid O Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 6b. Neu- ral belief tracker: Data-driven dialogue state tracking. arXiv preprint 1rXiv:1606.03:: 1606.03777
Hwiyeol Jo and Stanley Jungkyu Choi. 2018. Ex- trofitting: Enriching word representation and its vec- tor space with semantic lexicons. arXiv preprint arXiv:1804.0783
Nikola Mrkšić, Ivan Vulié, Diarmuid Ó Séaghdha, Ira Leviant, Roi Reichart, Milica Gašić, Anna Korhonen, and Steve Young. 2017. Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the Association for Computational Linguistics, 5:309- 324.
Aishwarya Kamath, Jonas Pfeiffer, Edoardo Maria Ponti, Goran Glavaš, and Ivan Vulić. 2019. Special- izing distributional vectors of all words for lexical entailment. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepLANLP-2019) pages 72.83.
Roberto Navigli and Simone Paolo Ponzetto, 2012. Ba- belnet: The automatic construction, evaluation and application of wide-coverage multilingual semantic network. Artificial Intelligence, 193:217-250.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 1746-1751.
Jeffrey Pennington, Richard Socher, and Christopher Manning, 2014, Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532- 1543.
Diederik P Kingma and Jimmy Ba. 2014. Adam: method for stochastic optimization. arXiv preprint
Thomas Landauer and Susan T Dumais. 1997. solu- tion to plato's 's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological review, 104(2):211.
Edoardo Maria Ponti, Roi Reichart, Anna Korhonen. and Ivan Vulić. 2018. Isomorphic transfer of syn- tactic structures in cross- nlp. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1531-
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al. 2015, Dbpedia-a a large-scale, mul- tilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167-195.
Ivan Vulié, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, et al. 2020. Multi-simlex: large-scale evaluation of multi- lingual and crosslingual lexical semantic similarity. Computational Linguisties. 46(4):847-88) 897.
Omer Levy and Yoav Goldberg. 2014. Dependency- based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 302- 308.
Ivan Vulié, Goran Glavaš, Nikola Mrkšić, and Anna Korhonen. 2018 Post-specialisation: Retrofitting vectors of words unseen in lexical resources. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chupter of the Association for Computational Linguistics: Human Language Téchnologies, Volume (Long Papers), pages 516-527
Minh-Thang Luong, Richard Socher, and Christopher Manning. 2013. Better word representations with re- cursive neural networks for morphology. In Proceed- ings of the Seventeenth Conference on Computational Natural Language Learning, pages 104-113.
Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, and Goran Glavaš. 2021. LexFit: Lexical fine-tuning of pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 1 11hh International Joint
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In

---

Conference on Natural Language Processing (Vol- ume Long Papers), pages 5269-5283, Online. As- sociation for Computational Linguisties.
Ivan Vulié, Simone Paolo Ponzetto, and Goran Glavaš, 2019. Multilingual and cross- lingual graded lexical entailment. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 4963 4974.
Tsung- Hisinn-innnnnn Wen, David Vandyke, Nikola Mrkšić, Mil- ica Gasic, Lina M Rojas Barahona, Pei-Hao Su, Ste- fan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the Turo- pean Chapter of the Association for Computational Linguistics: Volume Long Papers, pages 438-449.
John Wieting, Mohit Bansal, Kevin Gimpel, Karen Livescu, and Dan Roth. 2015. From paraphrase database compositional paraphrase model and back. Transactions of the Association for Compu- tational Linguistics, 3:345-358.
Steve Young, Milica Gašić, Simon Keizer, François Mairesse, Jost Schatzmann, Blaise Thomson, and Kai Yu. 2010. The hidden information state model: A practical framework for pomdp-base based spoken dia- logue management. Computer Speech & Language, 24(2):150-174:
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in neural information pro- cessing systems, pages 649-657.

---

A Appendix
views (Zhang et al., 2015), AGNews, and IMDB (Maas et al., 2011). We utilize Ya- hoo!Answer dataset for 2 different tasks, which are classifying upper-level categories and classify- ing lower-level categories, respectively. We use all the words tokenized by space as inputs. The data information is described in Table 8.
A.1 Data Resources
A.1.1 Pretrained word vectors
Pretrained word vectors include words composed of n-dimensional float vectors. One of the major pretrained word vectors we used is Vl (Pen- nington et al., 2014) glove.42B.300 many word embedding algorithms and pretrained word vectors have been suggested, GloVe is still being used as a strong baseline on word similarity tasks (Cer et al., 2017; Camacho-Collane 2017). We also use fastText (Bojanowski et al., 2016), and Paragram (Wieting et al., 2015) as re- sources and baseline of self- supervsee extrofitting,
Even though
Classifier. We build TexCCNN (Kim, 2014) rather than use classifier based on Bag-of- of-Words (BoW), as Faruqui al. did, in order to process word se- quences. The classifier consists of 2 convolutional layers with the channel size of 32 and 16, respec- tively. We adopt the multi-channel approach, im- plementing 4 different sizes of kernels- 2, 3, 4, and 5. We concatenate them after every max-pooling layer. The learned kernels go through an activa- tion function, ReLU (Hahnloser al., 2000), and max-pooling. We set the dimension of word em- bedding to 300, optimizer to Adam (Kingma and Ba, 2014) with learning rate 0.001, and use early- stopping if validation accuracy does not increase over 5 epochs.
et al.,
A.1.2 Semantic Lexicon
As an external semantic lexicon, we use Word- Net (Miller, 1995), which consists of approxi- mately 150,000 words and 1 115,000 synsets pairs. We use Faruqui et al.'s WordNetar lexicon, com- prised of synonyms, hypernyms, and hyponyms. Some recent works built their own or used large lex- icons like BabelNet (Navigli and Ponzetto, 2012), but in order to observe the effect of post-processing algorithm rather than the power of lexicons, we use the (relatively) small lexicon.
A.4 Ablation Study in Threshold
When the latent dimension is 300, following the best performance:
For fair comparisons, we replace the previous works' lexicon with WordNetall lexicon. If the models require antonyms, we use the antonyms pairs, which are uploaded in their Github.
A.5 Qualitative Examples
The additional qualitative examples are presented in Table 12.
A.2
Example of Extracted Relation
Threshold
Grouped Words
points225 points203 points 166 points253
17:20:18 09:22:30
10:22:49 07:33:04
2 .90
review 10/15/2012 1/1//03
review 1/30/2012 review 12/18/2012 !!!
1.99
January lebruary
range emailicon linkicon
Table 7: Examples of extracted related word groups. This results are extracted from 100 dimensional latent space GloVe.
A.3 Further Details in Experiment 3
Datasets. We use 5 classification datasets; DB- pedia ontology (Lehmann et al., 2015), Ya- hoo!Answers (Chang et al., 2008)6, YelpRe-
hnttps://cogcompppeet upenn. www.w.eweeaea.a....eeeeee edu/page/ resource_vis wvee//8 Note that Chang et al. (20n8) said the dataset has 20 top-level categories but actually it has 3
duplicated top-level categories hecause of emoos.

---

DBpedia
Yahoo(Up)
Yahoo(Low)
AGNews
Yelp
IMDB
#Train
560,000
133,703
133,703
120,000
650,000
25,000
#Test
70,000
23,595
23,595
7,600
50,000
25,000
#Class
14
17
280
4
2
#Vocab
626,717
154,142
154,142
66,049
198,625
47,113
Table 8: The data information used in text classification. YahooAnswer dataset is used for different tasks, which are to classify upper-level categories classify lower-level categories, respectively. The vocabulary size can slightly different due to the predefined special tokens such none and out-of--occaularry
Word Similarity Task
W(s)
W(r)
RN
NE
SE
SL
SV
GloVe
.70
.57
.39
.74
.57
.37
.28
threshold=.955
.76
-58
.42
.28
threshold=.99
.64
.50
.37
threshold=. 50
.38
.75
.56
.39
.23
Table Spearman's correlation of self sulf-sup extrofitted pretrained word embeddings according to threshold.
Model (threshol))))
DST
GloVe
.798
SelfExtro .95)
800
(l.fExtro(thresiin
.825
SelfExtro Selffxtro(threeholdd
.811
Table 10: Joint goal accuracy according threshold
Freezed Vectors
DBpedia
Yahoo(Up)
Yahoo(Low)
Yelp
AGNews
IMDB
GloVe
98.31
71.97
49.35
61.55
90.66
87.47
Sel 95)
98.35
71.61
49.07
61.01
90.78
87.23
Sth
98.44
72.41
49,79
62.97
90,93
89.36
fel 50)
72.74
50.80
90.94
88.33
Trainable Vectors
DBpedia
Yahoo(Up)
Yahoo(Low)
Yelp
AGNews
IMDB
GloVe
98.61
73.45
49.30
63.01
91.54
Sel fExtro 95)
98.42
72.08
49.17
61.83
91.03
86,80
SelfExtro(thrre
98.58
73.02
50.05
63.25
91.61
89.98
SellfExroothhe (thr=.
98.58
73.35
50.66
62.64
91.67
88.64
Table 11: Text classification accuracy according threshold

---

Cue Word
Nethod
Top- Nearest Words(Cosine Similarity Score)
S000 8394), soonol 79338), s00000(.7715), soono0o 7359), s00nonon(, (6844)
Vanilla
haha( (5744, hahah( damn( (6247, omg( omg(16244) hahaha(.62))
500
S000 8196) 30000(.7743) s00000l. (376), s000000l, 7304), 500000000,68533
*elffExtro
500000000( 6342), 5000000000 6314), 50000000000(.600.
too0(.5967), 500000000000
intricato(?7244), claborately(.6236 extravagant,6223, complicated(.6089)
Vanilla
lav1sh( (ormal((5643), sophisticated(,56.9
dctailed( ormatc(.5619), simplc(.5566)
elaborate
intricato(.7002) claborately(66329) extravagant, comppi///oo.mmmtitttted.d..o.. complicated((.555)
SelfExtro
ornate(.5586), lav3sh(.5533), grandiose(
formal(.5326) detailed(.5235) fancifulk
thankfulness( (111,, generosity( kindiness( appreciation(.6660)
Vanilla
compassion( 6695), admiraaiion,66677 grateful(
thankful(,633) 6385), heartfein 6366), sympathy(.611)
gratitude
foinkfulness(766433 generosity(.00) kindness(,6975) appreccctto (ppreciation..69.4
+ExtfExtro
admiration( (833)) compaasionn,66335 gratefulness( (6444),
ivverence,6366 thankful( (3313,,, sympathy(66305
(xsultate(.5262)) deum(,4763), exultatel 4545). motet(,4054) cxcelsiss[3777).
Vanilla
deo(.3721), cuntata(.3651), alleluia( (3255). stabat(,3558), laudamus(
jubilate
cxsultato(_511)) cxultatei 44449) dcum(-4269), motci(_4093), dco(.3703),
Selffxxro
choir(.3670) excelsis(.3662)) mozart((3594] stabat(.3591) cantata(,3532)
overjoyed( ecstatic(.7045), Uhrilled(665766 exhilarated
Vanilla
gratified(,64311 enthused(,621 62114)) saddenedí,607 (675)).
Habbergastedd dishcartened( (5599))) giddy(.5984)
elated
overjoyed(, 8001)) ecstatic(, 7424), thrilled(.7320),, excited(,68%
+Sel
exhilarated(, 6758). delighted gratified(,6690) r556)) enthused( (5564), dismayed(,6344)
disaccharide( (44)ca),rdd(6647)),, monosacchardess(.11111
oligosaccharide(.99)) 4911))
Vanilla
galactoset 4686), S-carbon( 4609), 1-aaccyylluccosaminee42332
mono- saccharide
saccharide( (4223), sucrose(.4215) mannose( (4196), disaccharides(
disaccharide (6329), monosaccharides, 4888). (4130saccharide(44444)..
+SelfExtro
5-carbon( (6117), galactose( 44144,, 1--accyygllccosaminee427777 sucrose(44206), disaccharides(
carbohydratee.4045 moieties(.4033)
excel,,5686), recovery(.5457), expori(.5688) forecast(.5222) forecasts(.5060)
Vanilla
contacts(.5025) powerpoint( (5055) (5555), exchange(.5011) microsoft( (9444), import(.4816)
outlook
excel(.3145), (napssevere(.5054) icalendargooolee449555 exp8))(_4883),
fExtro
contacts( 4877), recovery(.4753) advisorieshourlyy46733
mailbox((4661) vcard(,4612), thunderbird (4599)
foolish( (2229), futile(.6 (104), se3fish vainly(.5242). fruitless(.5133)
Vanilla
thy(.4981). arrogant( (976)), useless(_4844), (8833), thou((4830)
vain
futiler 6273), foolish(,6011 (679), vainly(.S454), fruitless(.3338),
+Setroxtro
selfish(.5143) (seless(_5044), pointless((49) 49100)
hopeless(44866) attempt(.4797) presumptuous( (4780)
prioritise( (00rit prioritizing(,7423 priorities(6322) prioritising(.5995))
Vanilla
evaluate(_59555 (5950), strategize( 59333),, proactively(.866)
amalyze(. 58000, assess(.5766 (564), adlettif(..5333)
prioritize
prioritise 8322), prioritziig(.7433)) prioritee(66300) prioritisin((6222)
#Extro
proactively(
Strategize(.5999) evaluate(.5944 (5546).
analyze(.5919) prioritized( manage(.5783)
nomads(.49 4911) drifter( 4735),, nomadici (5744),,(4454)), (223)) 2e6(,4
Vanilla
muvo2(_4054) gypsy (:4054), jukebox((3997) vulcan(.3906) adventuren(
nomad
nomads(.5254). (991),) (ommadic,44411), muvo2(,4631), vagahond(
+SelfExtro
traveler( .246), traveller( 16) zen( (55)) jukebox(.411000 wandererr, 4073)
Srictions(_5299) ncar(.4985), road( (936)) intcrsection( creck(.4911).
Vanilla
highway( exit( 48533, bridge(.4811) rd(.9779), jet(.4722)
junction
junctions(_580) jct(.5099), highway(,4788 motorway(4788) road(.4744),
+SelfExtro
exit(,47 htw.4733).ww.477777 hwy(,4710) rd(44667), intersections 46288) heafford 4596)
peat( (3660. hraady.5090), reed(,4998) 4998), brown(, (4963), 4945).
Vanilla
greent 4920), woods(.4877 pine(.4850) (840), campbell( 4824),
moss
lichen(,5628), sphagnum, 5463), spagnumí (ang)um(.4577) peatl 5323), lichenst, 3264), mosses( 5224). (934), welker 4852). pine(.4846)
fExtro
Table 12: List of 10p nearest words of cue words in different post-processing methods. We report cosine similarity scores of random words. Underline and bold indicate the difference between the list from vanilla vectors and self-supervised extrofitted vectors, respectively.


Reviews:
[{"review_id": "r6cTwihyug", "reviewer": "Reviewer_FCUs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your critical comments (especially [Weak1]). It will help clarify the idea behind this work.\nWe hope this rebuttal could convince you that this paper can be much improved until camera-ready with the help of your comments.\n\n[Weak1] The authors don't convey any intuitions about why this method works well\n\n- We agree that we needed to clarify the explanations in the main page. The explanations were scattered and briefly summarized in the Limitation Section.\n- We want to clarify that the intuition is:\n  - LDA algorithms group in-domain instances (in this case word vectors), together.\n  - Simultaneously, they push out-domain instances apart, which aligns with the principle of vanilla extrofitting.\n  - Therefore, even though we have only a small amount of lexicons to determine in/out-domain, the process of \"making out-domain instances far\", in particular, could result in better representations.\n  - To do this, we combine extrofitting with a simple SVD trick in order to get lexicons.\n\n[Weak2] The intrinsic evaluation presents results that are far from the state of the art\n- First of all, we appreciate your understanding that \"That doesn't mean the method is not useful\"\n- Since we do not have any training process and external resources to fit on SimLex, it seems natural that the performance of our method is behind the state-of-the-art. Furthermore, the goal of this method is to get better-generalized representations.\n- However, extending this method for vector specialization could be an interesting future work.\n\n[Minor1] Footnote 2: \"it performs better than another version\" - please be specific, which other version?\n- Thank you for the correction. vecmap has identical version and unsupervised version, so it refers identical version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There are multiple experiments to evaluate intrinsic and extrinsic quality, and the overall picture is that the proposed method has some benefits:\n- It is a nice result that the self-supervised method can achieve similar quality to the lexicon-driven extrofitting method; as the authors observe this can be helpful in languages where there are limited lexical resources.\n- While the authors tried and failed to reproduce previous results on the DST dataset (I appreciate their honesty in stating this), I note the SelfExtro results are better than those reported in the original Attract-Repel paper.\n\nPractitioners who have a need for non-contextual word embeddings (e.g. for efficiency reasons) may find this approach a useful one.", "reasons_to_reject": "- The authors don't convey any intuitions about why this method works well. Unlike other specialisation methods, the approach in this paper doesn't introduce any new information that's not already present in the vectors. For example, is it necessary to perform SVD-based dimensionality reduction on vectors that are already highly compressed?\n\n- The intrinsic evaluation presents results that are far from the state of the art, e.g. the SimLex-999 homepage indicates that the best score in 2016 was 0.76, compared to 0.50 reported here. That doesn't mean the method is not useful, but there is some important context missing.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper describes an approach to semantic specialisation of word vectors, i.e. an optimisation method that can be applied to off-the-shelf word vectors in order to improve their usefulness for downstream tasks. The approach described here (\"self-supervised extrofitting\") is based on the extrofitting method of Jo and Choi (2018) but differs from that method and most prior methods by not requiring a manually curated lexicon to group words by similarity. The authors show that despite not using a lexicon, the resulting word vectors have comparable or better quality to those derived with the extrofitting method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "p. 2 typo \"have use the decomposition\"\n\nFootnote 2: \"it performs better than another version\" - please be specific, which other version?"}, "extra_text": {}}, {"review_id": "onT9hHlrh9", "reviewer": "Reviewer_fe3o", "summary": "", "questions": "Question A: Are the results presented in the paper are statistically significant?\nQuestion B: Are there any other metrics used for text classification other than the accuracy? If yes, how do the results change?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your interest in this paper and for letting us know its weaknesses. We hope this rebuttal answers the questions.\n\n[Weak1 & Question A] it's not mentioned in the paper whether the results are statistically significant\n- Since the results of self-extrofitting make consistent vector representations, we do not need to measure the significance test in intrinsic tasks.\n- However, the results of the other downstream tasks can be different because of weight initialization; we thus denote every standard deviation.\n- As you can see through the reports, most of the results are meaningful.\n\n[Question B] Are there any other metrics used for text classification other than the accuracy? If yes, how do the results change?\n- F1 metrics could be used for text classification, but the trends are the same with accuracy since the text classification datasets are balanced (except for YahooAnswers)\n- In the case of YahooAnswers, it also showed the same trends.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The approach in the paper will be used for languages where the resources are less\n- The approach doesn't depend on the presence of external semantic lexicons\n- The results presented in the paper can be easily reproducible", "reasons_to_reject": "- Even though the results are better or comparable to the existing approaches, it's not mentioned in the paper whether the results are statistically significant", "questions_to_authors": "Question A: Are the results presented in the paper are statistically significant?\nQuestion B: Are there any other metrics used for text classification other than the accuracy? If yes, how do the results change?", "paper_topic_and_main_contributions": "- The paper proposes a new approach called self-retrofitting of word vectors without depending on external lexicons and also presented the results on languages where the resources are less. \n- Instead of using the external semantic lexicons to find the related words, they use the latent semantic analysis method to find the related words and applying the standard retrofitting approach on top of it. \n- The experiments are conducted across different tasks of word similarity, dialogue state tracking and text classification and obtained better results\n- The paper is written well and the related work is comprehensive that compares and contrasts with this work\n- The limitations of the work are also mentioned in detail\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line no 119: \"researchers have used\" instead of \"researchers have use\""}, "extra_text": {}}, {"review_id": "9zOTGSciOC", "reviewer": "Reviewer_dUyB", "summary": "", "questions": "Did you perform any kind of significance testing in your experiments? If not, why?\n\nTable 6 presents groups of words that share a minimum cosine similarity in the latent space. Is this similarity compared against some anchor word embedding (i.e. every vword in the group is at least the minimum similarity with the anchor word)? Or are all pairs within that group a minimum similarity with each other? If the former, what are the anchor words for each of those groups? If the latter, how did you find these groups?\n\nThe example groups given in Table 6 seem to largely consist of non-words (i.e. may of them appear to be the result of bad text sanitation in the original GloVe embeddings) and doesn't give confidence in the extracted word groups. Did you perform any qualitative analysis on the extracted groups?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for letting us know the concerns with too brief explanations of the results.\nWe hope this rebuttal could convince you that this paper can be improved significantly until camera-ready with the help of your comments.\nPlease kindly understand that this is mainly due to the page limit; we hope you could relieve the concerns through this response.\n\n[Weak1]The provided discussion of experimental results is very surface level\n- In the current version of this paper, we attempted to present empirical results as much as possible rather than investigate the word vectors with a closer look, which can appear to be cherry-picked\n- We will add more reasons/explanations, like qualitative analysis. Please see the appendix of this rebuttal below.\n\n[Question 1] significance testing in your experiments\n- The experiments (except for intrinsic evaluation) include standard deviations, and the results are statistically meaningful.\n- In the case of word similarity tasks, self-extrofitting generates consistent representations in the same setting.\n\n[Question 2] How to group the synonym words\n- \"are all pairs within that group a minimum similarity with each other?\" is right.\n- To be specific, the processes are:\n  - For every word (referred to as an anchor word), we calculate its cosine similarity to all other words in the vocabulary.\n  - If the similarity exceeds a predetermined threshold, we group the words.\n    - In this process, a group can contain only a single word.\n  - This process is repeated iteratively for each ungrouped word until all words are assigned to groups.\n    - It means the next anchor word will not assign to existing group; it always generates new group.\n- It's important to note that the order of anchor word selection doesn't affect the final groups due to the symmetric nature of cosine similarity.\n\n[Question 3] Quality of extracted word groups\n- When we observe the extracted word groups, most are similar to Table 6. (e.g., timestamp, the name of months, numbers etc.)\n- Although the extracted word groups may not seem linguistically meaningful at first glance, they serve as a crucial mechanism for the effect of self-extrofitting.\n  - By clustering seemingly unimportant words and scattering the other words together, we facilitate the formation of more meaningful connections between words, ultimately enhancing the vector representations.\n- Due to this advantage, we claim that the extracted word groups itself do not need to be meaningful.\n- If you are interested in the quality of the result, please refer to the Appendix of this rebuttal.\n\n[Appendix]\n- We do not want to say that this only one example of qualitative analysis means our method is superior, rather to say the result is meaningful, so please see it as a reference:\n- For example, even if we used the extracted word groups (which seems linguistically meaningless), the similar words to 'love' in the vector spaces are as follows:\n    - Vanilla glove (glove.42B.300d.txt):\n\n    [('loved', 0.7745193839073181), ('i', 0.733807384967804), ('loves', 0.7310654520988464), ('know', 0.7285589575767517), ('loving', 0.7262834310531616), ('really', 0.7195589542388916), ('always', 0.7193377017974854), ('want', 0.7191829681396484), ('hope', 0.712653636932373), ('think', 0.7109541893005371)]\n    - SelfExtro:\n\n    [('loved', 0.715188205242157), ('loving', 0.6733922362327576), ('loves', 0.6488849520683289), ('adore', 0.6347748637199402), ('passion', 0.6332944631576538), ('luv', 0.6325735449790955), ('hope', 0.6255802512168884), ('i', 0.6250261068344116), ('want', 0.6209121346473694), ('hate', 0.6181126832962036)]\n    - Although the representations lose similarity scores, the list of close words become diverse and reasonable; we want to claim it as better-generalized representation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- relatively simple way to improve performance of pretrained word embeddings\n- language independent\n- reduces need for external resources", "reasons_to_reject": "The provided discussion of experimental results is very surface level and doesn't provide the reader with little insight into how the proposed method enhances performance and what the limitations or pitfalls of this approach might be.", "questions_to_authors": "Did you perform any kind of significance testing in your experiments? If not, why?\n\nTable 6 presents groups of words that share a minimum cosine similarity in the latent space. Is this similarity compared against some anchor word embedding (i.e. every vword in the group is at least the minimum similarity with the anchor word)? Or are all pairs within that group a minimum similarity with each other? If the former, what are the anchor words for each of those groups? If the latter, how did you find these groups?\n\nThe example groups given in Table 6 seem to largely consist of non-words (i.e. may of them appear to be the result of bad text sanitation in the original GloVe embeddings) and doesn't give confidence in the extracted word groups. Did you perform any qualitative analysis on the extracted groups?", "paper_topic_and_main_contributions": "This paper introduces a new method for self-supervised extrofitting that doesn't require an external semantic resource. This makes this method applicable to any pretrained embedding set for any language, even if that language does not have readily available semantic resources. The authors test their approach on word similarity, dialogue state tracking, and text classification tasks and show comparable (if not superior) performance to traditional extrofitting methods which do require an external semantic resource.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "S1P2 Line 035 - \"any kinds\" -> any kind\n\nSome issues with column width in table 7\n\nForcing A4 into two columns would result in less wasted space"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a word on semantic specialisation of word vectors, based on the extrofitting method, without requiring any external lexicon or manually curated resources. The proposed method improves its baseline on various benchmarks and languages, showing the quality of the approach. 
After author rebuttal, the reviewers, in general, agree that the work is simple, language independent, and it improves the performance of vanilla word vectors without the need for an external lexicon. They appreciated the author response, however, the paper would need a better explanation of the intuition behind the proposed method, so the clarity of the paper should be improved, as it reads more like an incremental work.
Decision: Accept-Findings