Word Sense Induction with Knowledge Distillation from BERT
Authors: Anik Saha
Alex Gittens
Bulent Yener
Venue: ICLR 2022 Submitted (2022)
Keywords: word embeddings
sense embeddings
word sense induction

Abstract: Pre-trained contextual language models are ubiquitously employed for language understanding tasks, but are unsuitable for resource-constrained systems.  Noncontextual word embeddings are an efficient alternative in these settings. Such methods typically use one vector to encode multiple different meanings of a word, and incur errors due to polysemy. This paper proposes a two-stage method to distill multiple word senses from a pre-trained language model (BERT) by using attention over the senses of a word in a context and transferring this sense information to fit multi-sense embeddings in a skip-gram-like framework. We demonstrate an effective approach to training the sense disambiguation mechanism in our model with a distribution over word senses extracted from the output layer embeddings of BERT. Experiments on the contextual word similarity and sense induction tasks show that this method is superior to or competitive with state-of-the-art multi-sense embeddings on multiple benchmark data sets, and experiments with an embedding-based topic model (ETM) demonstrates the benefits of using this multi-sense embedding in a downstream application.


Under review as a conference paper at ICLR 2022
WORD SENSE INDUCTION WITH KNOWLEDGE DISTIL- LATION FROM BERT
Anonymous authors Paper under double-blind review
ABSTRACT
tre-trained contextual language models are ubiquitously employed for language understanding tasks, but are unsuitable for srource-constrained systems. Non- contextual word embeddings are an efficient alternative in these settings. Such methods typically use one vector to encode multiple different meanings and incur errors due to polysemy. This paper proposes two-stage method to distill multiple word senses from a pre-trained language model (BERT) by using attention over the senses of word in context and transferring this sense information to fit multi-sense embeddings in a skip- caap-gram-iike framework. We demonstrate an effec- tive approach to training the sense disambiguation mechanism in model with distribution over word senses extracted from the output layer embeddings of BERT. Experiments the contextual word similarity and sense induction tasks show that this method is superior to or competitive with state-of-the-art multi-sense embed- dings on multiple benchmark data sets, and experiments with an embedding-baid based topic model (ETM) demonstrates the benefits of using this multi-sense embedding in a downstream application.
word,
INTRODUCTION
While modern deep contextual word embeddings have dramatically improved the state-of-the-art in natural language understanding (NLU) tasks, shallow noncontextual representation of words are more practical solution in settings constrained by compute power latency. In single-sense embeddings such word2vec or GloVe, the different meanings of word are represented by the same vector, which leads to the meaning conflation problem in the presence polysemy. [Arora al. (2018) showed that in the presence of polysemy, word embeddings are linear combination of the different sense embeddings. This leads to another limitation in the word embeddings due to the triangle inequality of vectors (Neelakantan et al.) 2014). For apple, the distance between the words similar to its two meanings, i.e. peach and software, will be less than the sum of their distances to apple. To account for these and other issues that arise due to polysemy, several multi- embeddings model have been proposed to learn multiple vector representations for each word.
There
major components for multi-sense embedding model to be used in downstream tasks: (1) sense selection to find the correct sense of a word in context (2) sense representation for the different meanings of a word. Two types of tasks are used evaluate these two distinct abilities of a word sense embedding model: word sense induction and contextual word similarity respectively. In word sense induction, a model has to select the correct meaning of a word given its context. For contextual word similarity, the model must be able to embed word senses with similar meaning close each other in the embedding space. Prior works on multi-sense embeddings have been shown to be effective on these two tasks simultaneously (e.g. AdaGram (Bartunov et al.) 2016) works well on word sense induction but not on contextual word similarity). We propose a method to distill knowledge from contextual embedding model, BERT (Devlin al. 2018), for learning word sense model that can perform both tasks effectively. Furthermore, we show the advantage of sense embeddings word embeddings for training topic models.
Pre-trained language models like BERT, GPT RRafford et al. 2019). RoBERTa (Liu et al. 2019 etc. have been immensely successful in multiple natural language understanding tasks, but the computational requirements and large latencies of these models inhibits their use in low-resource systems. Statie word embeddings sense embeddings have significantly lower computational

---

Under review as a conference paper at ICLR 2022
requirements and lower latencies compared these large transformer models. In this work, use knowledge distillation (Hinton et al.) 2013)) to transfer the contextual information in BERT for improving the learning of multi-sense word embedding models. Although there are several prior works on distilling knowledge to smaller transformer models, to the best of our knowledge there has been no work using knowledge distillation improve the training of word or sense embeddings using pre-trained language models.
The contributions of this paper are:
A two-stage method for knowledge distillation from BERT to static word sense embeddings
A sense embedding model with state-of-the-art simultaneous performance sense selection and sense induction.
iii
A demonstration of the increased efficacy of using these multi-sense embedding in lieu of single sense in embeddings in a downstream task, namely embedded topic modeling (Dieng et al. 2020
RELATED WORK
Here we describe number of previous models trained to represent different meanings of words using the word2vee framework. Prior works on model distillation using BERT are also discussed.
Reisinger Mooney (010)) was the first to learn vector representations for multiple senses of word. They extracted feature vectors of word in different contexts from the context words and clustered them to generate sense clusters for word. The cluster centroids are then used as sense embeddings or prototypes. |Huang et al. (2012) similarly clustered the context representations of a word using spherical k- --mans,, and learned sense embeddings by training a neural language model.
Neelakantan et al. (2014) extended the Skip-gram model to learn the cluster centers the sense embeddings simultaneously. At each iteration, the context embedding is assigned to the nearest cluster center and the sense embedding for that cluster is used to predict the context word. A non-parametric method is employed to learn an appropriate number of clusters during training.
Tian et al. (2014) defined the probability of a context word given center word as mixture model with word senses representing the mixtures. They developed an EM algorithm for learning the
embeddings by extending the skip-gram likelihood function. Li & Jurafsky (2015) used the Chinese Restaurent Process to learn the number of senses while training the sense embeddings. They evaluated their embeddings on number of downstream tasks but did not find a consistent advantage in using multi-sense embeddings over single-sense embeddings.
represented words multi-modal Gaussian distributions where the different modes correspond to different senses of a word. To denote the similarity between two word distributions, they used the expected likelihood kernel replacing cosine similarity for vectors.
Athiwaratkun & Wilson 2017
Bartunov et al. (0166) proposed a non-parameteric Bayesian extension to the skip-gram model to learn the number of senses of words automatically. They used the Dirichlet Process for infinite mixture modeling by assuming prior probabilities over the meanings of a word. Due to the intractibility of the likelihood function, they derived a stochastic variational inference algorithm for the model.
Grzegorezyk & Kurdziel (2018) learned sense disambiguation embeddings to model the probability of different word senses context. They used the gumbel-softmax method to approximate the categorical distribution of sense probabilities. To learn the number of senses of word, they added an entropy loss and combined it with a pruning strategy.
Lee & Chen (2017) proposed the first sense embedding model using a reinforcement learning framework. They learned only sense embeddings and an efficient sense selection module for assigning the correct sense to word in context.
Hinton et al. (2015) demonstrated the effectiveness of employing a large pre-trained teacher network to guide the training of a smaller student network and named this method model distillation. Recently, this technique has been used to train many efficient transformer models from BERT. Sun et al. (2019): Sanh et al. (2019); Diao al. (2019); [Sun et al. (2020) have trained smaller BERT models by distilling knowledge from pre- trained BERT models down to fewer layers. However, there seems to be no prior work improving word or sense embeddings by using pre-trained language models like BERT.
2

---

Under review as a conference paper at ICLR 2022
sense
p(w, = w/
Bi
Sigmoid [8+1 M/()
Softmax (c d
c
Mean
k senses
A
context word
context word
center word W/
W
Figure I: Sense embedding model
3 MODEL
We first describe our word sense embedding model. Then the details of extracting knowledge from the contextual embeddings in BERT are discussed.
3.1 SENSE EMBEDDING MODEL
We have dataset D (wi, ........ wN), a sequence of N words from a vocabulary V. For word w; in our vocabulary, we assume K senses and for the k-th sense, we learn a sense embedding, v and disambiguation embedding, d We also have global embedding g: for each word ut:
Following the word2vec framework to train the sense embeddings, we slide window of size 2 + 1 (center word and A words to its left and right) over the dataset. In context window C' = { [u;= (U;-A,,.. ......
Wi we define the conditional probability of context word given the Wi..... , center word, Wt as
p(w,+1/w,)
Σ p(w++ wy wt)p )p(w: wtlw, Ius, C) k=1
(1)
where, / E A- A).1
Here the probability context word, W3+4 given a sense embedding. wf is p(w w, w o o(g v(t)
where (Ix) (1 + exp(-z))- is the logistic sigmoid function.
The sense disambiguation vector selects the more relevant sense based on the context. The probability of sense, of word, w; in context C' is defined similar to the dot-product attention mechanism.
edp c))
p(w; = w ww C)
(3)
ΣK
exp (d c))
where, c is the context embedding for context C' defined by the average of the global embeddings of the context words
c 2A
gi+
(4)
I=E0
Iterative sense disambiguation Instead of using the global embeddings of the context words, we can use their sense embeddings and sense probabilities obtain context embedding with more contextual information. The steps are following:
Step 1 Using the context embedding from Eq. compute the sense probability p(w++ =
w w wett, C`) for each word in the context (1 (, +

---

Under review as a conference paper at ICLR 2022
Step 2 Replace the global embeddings in Eq. 4 with the weighted average of the embeddings for each context word
k 144 p(w;-1 = w/ss/us4 C)
(5)
2A
k=
Iy0
Similar to the skip-gram model (Mikolov et al., 2013), we employ the negative sampling method reduce the computational cost of softmax over huge vocabulary. The loss function for the multi-sense embedding model is as follows
Lisense = - Σ
+ Σ log - ppwp:://w/pprs [w))]
(6)
W.DD
j=1
L40
where, is the number of negative samples and W4+1j is the j-th negative sample for the context word W1+--
3.2 TRANSFER KNOWLEDGE FROM BERT
We use the contextual embeddings from BERT to improve our sense embedding model in 2 steps. First, we learn the sense embeddings as a linear decomposition of the BERT contextual embedding in a context window. Second, the sense embeddings from the first step are used for knowledge distillation to our sense embedding model.
3.2.1 BERT SENSE MODEL
Let, the sense embeddings of word w; trained from BERT is {u: u and the contextual embedding for word, w; is b; embedding from the output layer of BERT. Since BERT uses WordPiece tokenization, some words are divided into smaller tokens. So each word has multiple 'wordpiece' tokens and output embeddings. To get the contextual embedding b, from these token embeddings, we take the mean of all token output embeddings for word. Similar to Eq. the probability of sense, of word, uy in context is
p(w; = w wes C) exp(id) CBERT))
K Σ k= exp( CBERT))
where, context embedding
A
CBERT
b
2A
We want the weighted sum of the sense embeddings to be similar to the contextual embedding from BERT, So, the loss function is
K
2
LIERT
Σ
Σ
p(w; = w We, C)uf b,
(8)
W:DD
3.2.2 KNOWLEDGE DISTILLATION FROM BERT
Here we combine our multi-sense embedding model and the sense embeddings from BERT to transfer the contextual information. To be precise, we want to improve the sense disambiguation module (3) of the multi-sense embedding model using BERT. The key idea of knowledge distillation is that the output from complex network contains more fine grained information than a class label. So we want the student network to produce similar probabilities as the teacher network. Hinton et (2015) suggested using a "softer" version of the output from the teacher network using a temperature T over the output logits 2;:
exp(z,/T)
pT
Σ, exp(2:/TT
4

---

Under review as a conference paper at ICLR 2022
where, is the input.
The loss for the knowledge transfer is the combination of the original loss (Eq. 6 and the cross- entropy loss between the sense disambiguation probability from the sense embedding (Eq. 3 model and the BERT model (Eq. 7
L Lvease
(9)
where
Luanster -T
Prenso (w, = w C) Log(PERr)(( w,/w,, C))
(10)
is the cross-entropy loss between the distilled sense distribution and the BERT sense distribution. We multiply this loss by T because the magnitudes of the gradients in the distillation loss scale by 1/7".
EXPERIMENTS
We conduct both qualitative and quantitative experiments on our trained sense embedding model.
4.1 DATASET AND TRANING
We trained our models on the April 2010 snapshot of Wikipedia (Shaoul Westbury 2010) for fair comparison with the prior multi-sense embedding models. This corpus has over 2 million documents and about 990 million words. We also used the same vocabulary set as Neelakantan et al. (2014).
We selected the hyperparameters of our models by training them small part of the corpus and using the loss metric on a validation set. For training, we set the context window size to A 5. We use 10 negative samples for each positive (word, context) pair. We do not apply subsampling like word2vec since the BERT model was pre-trained on sentences without subsampling. We learn sense embeddings for each word with dimension of 300 for comparison with prior work. We use the Adam (Kingma & Ba 2014) optimizer with a learning rate of 0.001.
The base model of BERT is used as the pre-trained language model. The PyTorch transformers (Wolf et al, 2019) library is used for inference through the BERT-hase model. With a batch size of 2048, takes 10 hours to train the sense embedding model on the wikipedia corpus for one epoch in a Tesla V100 GPU. The sense embedding models with and without knowledge distillation are trained for 2 epochs over the full Wikipedia dataset. The weight for the knowledge distillation loss (Eq. 9 a is selected from (0.1, 0.5, 1.0). Temperature parameter T is set to 4 by tuning over the range [2,6].
4.2 NEAREST NEIGHBOURS
The nearest neighbours of words in context are shown in Table D Given the context words, the context embedding is computed using Eq. [5] and sense probability distribution computed with Eq. The most probable sense is selected and then the cosine distance is computed with the global embeddings of all words in the vocabulary. Table Jalso shows the values of the sense probability for the word in context. The sense probabilities are close to in most of the cases. This indicates that the model learned a narrow distribution over the senses and it selects a single sense most of the time.
3
4.3 WORD SENSE INDUCTION
This task evaluates the model's ability to select the appropriate sense given a context. assign sense word in context, we take the words in its context window and compute the context embedding using Eq. 5 Then the sense probability from Eq. Biives the most probable sense for that context.
We follow Bartunov et al. (2016) for the evaluation dataset and metrics. Three WSI datasets are used in our experiments. The SemEval-2000 dataset is from SemEval-2007 Task 2 competition (Agirre & Soroa 2007). contains 27232 contexts for 100 words collected from Wall Street Journal (WSJ) corpus. The SemFxal-2010 dataset is from the SemFyal-2210 Task 14 competition (Manandhar &
5

---

Under review as a conference paper at ICLR 2022
Context
Sense:
Nearest Neighbours
Proh.
macintosh is family of computers from apple
1 : 0.99
apple, amstrad, macintosh, commodore, dos microsoft, ipod, announced, software, mac caple, strawberry, cherry, blueberry mill. plant, cogeneration, factory, mw species, plant, plants. lichens, insects thistle, plant, gorse, salvia, leaves depolarization. contractile, hematopoietic, follicular, apoptosis
apple software includes mac
2: 0.963
apple good for health
3: 0.999
the power plant is closing
3: 1.0
the science of plant life the seed of plant the cell membranes of an organism
2: 1.0
1: 0.99
3: 1.0
almost everyone has cell phone now prison cell is not pleasant place
0.99
phone, cell, phones, mobile, each prison, room, bomb, arrives, jail
1 0.99
Table 1: Top 5 nearest neighbours of words in context based on cosine similarity. We also report the index of the selected sense and its probability in that context.
Klapaftis 2009) which contains 8915 contexts for 100 words. Finally, the SemEval-288 2012 dataset comes from the SemEval-2013 Task 13 (Jurgens & Klapaftis 2013) with 4664 contexts for 50 words.
For evaluation, cluster the different contexts of word according the model's predicted sense. To evaluate the clustering of a set of points, we take pairs of points in the dataset and check if they in the same cluster according to the ground truth. For the SemEval competitions, two traditional metrics were used: F-score and V-measure. But there are intrinsic biases in these methods. F-score will be higher for cluster with fewer centers while V-measure will be higher for clusters with large number of centers. So Bartunov et al. (2016) used ARI (Adjusted Rand Index) as the clustering evaluation method since does not sufter from these problems.
We report the ARI scores for MSSG (Ncelakantan et al. 2014). MPSG (Tian et al. 2014) and AdaGram (Bartunov et al., 2016) from Bartunov et (2016). Scores for disambiguated skip-gram -ipp-gramm are collected from Grzegorczyk & Kurdziel
We used the trained embeddings published by MUSE Lee & Chen (2017 7) with Boltzmann method for this task. Single-sense models like skip-gram cannot be used in this task since the task is to assign different senses to different contexts of a word. Contextual models like BERT give an embedding for a word given its context, but this embedding is not associated with specific discrete meaning of the word. Therefore BERT models are not suitable for comparison on this task.
MUSE shows strong performance in word similarity tasks (section 4.4 but fails to induce meaningful senses given a context. The Disambiguated Skip-gram model was the best model on SemEval- and SemEval-2011 datasets. Our sense embedding model performs better than MSSG and MPSG models but there is a large gap with the state-of-the-art. We can see a significant improvement in ARI score (1.25x 2.25x) the 3 datasets with knowledge distillation from BERT.
Model
SE-2007
SE-2010
SE-2013
MSSG
0.048
0.085
0.033
MPSG
0.044
0.077
0.033
AdaGram
0.069
0.097
0.061
MUSE
0.0009
0.01
0.006
Disamb. SG
0.077
0.117
0.045
SenseEmbed
0.053
0.076
0.054
BERTSenS: Sense
0.179
0.179
0.155
BERTeDEmmee
0.145
0.144
0.133
Table 2: ARI (Adjusted Rand Index) scores for word sense induction tasks. All models have dimension of 300. The Adagram and Disambiguated skip-gram models learn variable number of senses while all other models learn fixed number of senses per word. All baseline results except MUSE were collected from Grzegorezyk Kurdziel (2018). SenseEmbed Sense embedding model without distillation, BERTŠense Sense embedding model (Sec. 3.2.1) BERTKDEmbed Sense embedding with distillation
4.4 CONTEXTUAL WORD SIMILARITY
Here, the task is to predict the similarity of the meaning of words given the context they were used. We use the Stanford Contextual Word Similarity (SCWS) dataset (Huang et al. 2012). This dataset

---

Under review as a conference paper at ICLR 2022
contains 2003 word pairs with their contexts, selected from Wikipedia. Spearman's 's rank correlation is used to measure the degree of agreement between the human similarity scores and the model's similarity scores.
We use the following methods for predicting similarity of two words in context for a sense embedding model following RRiisngerr & Mooney (20T0): AvgSimC and MaxSimC. The AvgSimC measure weighs the cosine similarity of all combinations of sense embeddings words with the probability of those senses given their respective contexts.
AvgSimC(ur, w2)
p(w w wi/wi, w1, C)p(w2 w C) (1) og))
(11)
Here, the cosine similarity between sense word and sense of word 2 is multiplied by the probability of these 2 senses given their contexts.
The MaxSimC method selects the best sense for each context based on their probabilities and measures their cosine similarity.
MaxSimC(w1, w2) =cos(ui) cos(vi, un))
(12)
where
K arg max maxp = w1/ w1, (0100 C) and j = arg maaxppuss uy W2, C)
(13)
For comparison with baselines on this task, we use the single-sense skip-gram model (Mikolov et al. 2013) with 300 and 900 dimensions since word sense model has 3 embeddings per word with 300 dimensions each. We use the contextual models (BERT, DistilBERT) by selecting the token emhedding from the output layer for the word in context window of 10 words the left and right.
We report the score for the best performing model from each method in Table 3 For MUSE, the best model (Boltzmann method) is used. | Our sense embedding model (SenseEmbed) is competitive in AvgSimC measure whereas the knowledge distillation method significantly improves the MaxSimC score. We infer from this result that the improvement in sense induction mechanism helped learn better embeddings for each word sense. The best performing multi-sense embedding model's performance is very close the 900 dimensional skip-greee gra- embeddings (69.3 vs 68.4). The best performing multi-sense model considering the average over the 2 measures is MUSE with an average of 68.2. But does not do well on the word sense induction task. Both contextual embedding models (BERT and DistilBERT (Sanh et al.) 2019)) obtained lower scores than our sense embedding model.
Model
AvgSimC
MaxSimC
Skip-Graid (300D)
65.2
65.2
Skip-Gram (900D)
68.4
68.4
MSSG
69.3
57.26
MPSG
65.4
63.6
AdaGram
61.2
53.8
MUSE
68.8
67.6
BERT-base
64.26
64.26
DistilBERT-base
66.51
66.51
SenseEmbed
67.6
54.7
BERTSense
62.6
64
BERTKDEmbed
67.2
Table 3: Spearman rank correlation score 100) the SCWS dataset. Skip-gram is the only single-sense embedding model. All other embedding models are 300 dimensional. Skip-gram results reported in Bartunov al. (206)), MUSE score is obtained using their published model weights and for other models the results are from the respective papers.
4.5 EFFECTIVENESS OF KNOWLEDGE DISTILLATION
We can use the intermediate sense embeddings {u,, .,, u from the BERT Sense model (Sec. 3.2.1 word sense induction and contextual word similarity tasks. Since the BERT sense embed- dings are trained using the output embeddings from BERT, we can use them as context embedding for sense induction.
https:: thw.t com ww..bb//MUSH we used trained embeddings published here

---

Under review as a conference paper at ICLR 2022
In the knowledge distillation method, train sense embedding model to mimick the sense distribution from the BERT Sense model. Since the WSI task evaluates the predicted sense of the model, this is the ideal task to quantify the quality of knowledge transfer from BERT to the sense embedding model. The BERT Sense model is compared with the sense embedding model trained with knowledge distillation in Table 4 As the BERT Sense model is trained with MSE (Mean Squared Error) loss, it's s sense embeddings have the same dimensionality as BERT (768), From Table 4 we see that the 300 dimensional sense embedding model can obtain 75% to 88% ARI scores without using the 12 layer transformer model with 768 dimensional embeddings.
Model
SE-2007
SE-2010
SE-2013
BERTSSense 0.195
0.163
0.155
BERTKDEmbed
0.145
0.144
0.133
Table Word sense induction using the BERT Sense model (768D) and knowledge distilled sense embedding model (300D). ARI (Adjusted Rand Index) scores are reported.
Model
AvgSimC
MaxSimC
BERTSense
62.6
64
BERTKDEmbed
68.6
67.2
Table 5: Comparison of BERT sense model with sense embedding trained with knowledge distillation. Spearman rank correlation (p x 100) scores are reported the SCWS dataset.
We also compare the BERT sense model with the sense embedding model on the contextual word similarity task in Table 5 The lower correlation score from BERT Sense model indicates that these embeddings are not better at sense representation although they learn to disambiguate senses. The loss function in Eq. g helps the BERT Sense model to extract contextual sense information in BERT output embeddings, but it is not optimal for training sense embeddings. The sense embedding model combined with the knowledge gained from the BERT sense model learns to do both i) sense selection and ii) sense representation effectively.
4.6 EMBEDDED TOPIC MODEL
As sense embeddings help discern the meaning a word in context, prior work has shown the benefits of using them to perform topic modeling in the embedding space (Dieng et al. (022).. We explore the further benefits of using embeddings which model the fact that one token can have multiple meanings. Given a word embedding matrix p where the column P. is the embedding of word v, Embedded Topic Model (ETM) follows the generative process:
1. Draw topic proportions 0g ~ LO,
2. For each word n in the document:
(a) Draw topic assignment zdn Cat(Of))
(b) Draw the word Wan ~ softmax(p" (
Here, 0% is the topic embedding for topic k. These topic embeddings are used to obtain per-topic distribution over the vocabulary of words.
We experimentally compare the performance of ETM models trained using skip-gram word em- beddings, our sense embeddings and MSSG (Ncelakantan al. (014) embeddings. Three senses were trained for each word in both of the multi-sense models. A 15K document training set, 4.9K test set, and 100 document validation set are sampled from the Wikipedia data set and used for the experiments. During training using our sense embeddings and the MSSG multi-sense embeddings, the sentences are represented using a "bag-of-sense" representation where each word is replaced with its most probable sense selected using context window of 10 words; with the skip-gram ofg-os-words embeddings, bag-of- words representations are used.
For evaluation, measured the document completion perplexity, topic coherence, and topic diversity for different vocabulary sizes. Document perplexity measures how well the topic model can be used
Using code published by the authors of the original ETM model; see https:///g wt.ggttub.comm adjidieng//RMM
8

---

Under review as a conference paper at ICLR 2022
to predict the second half of a document, given its first half. Topic coherence measures the probability of the top words each topic to occur in the same documents, while topic diversity is a measure of the uniqueness of the most common words in each topic. Higher topie coherence and topic diversity indicate that a model performs better at extracting distinct topics; the product of the two measures is used measure of topic quality:
topic quality topic coherence topic diversity
We follow (Dieng al. 2020) in reporting the ratio of perplexity and topic quality measure of the performance the topic model: a lower ratio is better.
We set the minimum document frequency to [5, 10, 30, 100] to change the vocabulary size. ETM is initialized with pre-trained sense embeddings and then fine tuned along with the topic embeddings. We followed the hyperparameters reported in Dieng et al. (2020) train all models. Since use bag-of-sense model, ETM provides probability distribution over the senses. We sum the probability for the different senses of word obtain the probability for word. In Figure we show the ratio of the document completion perplexity (normalized by vocabulary size) and the topic quality.
ETM
ETM (mssg)
LDA
Quality No---almllee Parplexity
10000
20000
30000
40000
50000
60000
70000
80000
Vocabulary
Figure 2: Comparison of word embeddings and sense embeddings for topic modeling with different
vocabulary sizes. Here, ETM (sense) ETM with BERTKDEmbed, ETM (mssg) ETTM with MSSG.
We see that both MSSG and BERTKDEmbed ETM models obtain lower perplexity-quality ratio for higher vocabulary sizes compared to the single-sense (skip-gram) ETM model. This confirms that multi-sense embedding models produce Embedded Topic Models with higher performance. BERTKDEmbed ETM also outperforms the other multi-sense ETM model.
CONCLUSION
We present a multi-sense word embedding model for representing the different meanings of a word using knowledge distillation from pre-trained language model (BERT). Unlike prior work on model distillation from contextual language models, we focus on extracting non-contextual word senses; this is accomplished an unsupervised manner using an intermediate attention- based sense disambiguation model. We demonstrate tfrr-oofthhe-at results on three word sense induction data sets using fast and efficient sense embedding models. This is the first multi-sense word embedding model with strong performance on both the word sense induction and contextual word similarity tasks. As downstream application, we show that the sense embedding model increases the performance of the recent embedding- based based topic model (Dieng et al. 2020). In future. we will explore the benefits of using pre-trained multi senti- embeddings as features for other downstream tasks, and tackle the task of tuning the number of word meanings for each individual word.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Encko Agirre and Aitor Soroa. SemEval-2007 task 02: Evaluating word sense induction and discrim- ination systems. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEvall2007), pp. 7-12, Prague, Czech Republic, June 2007. Association for Computational Linguisties. URL https::///// www. wew.wb...........c wwt.ee...rggaatoooogg//////
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. Linear algebraic structure of word senses, with applications polysemy. Transactions of the Association for Computational Linguistics, 6:483-495, 2018. doi: (a622/tacla00033444 CRLhttps:: /www
sel.web.orrg/ammhhooor//// 1034
Ben Athiwaratkun and Andrew Wilson. Multimodal word distributions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1645-1656, 2017.
Sergey Bartunov, Dmitry Kondrashkin, Anton Osokin, and Dmitry Vetrov. Breaking sticks and ambiguities with adaptive skip-gram. In artificial intelligence and statistics, pp. 130-138, 2016.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling embedding spaces. Transac- tions of the Association for Computational Linguistics, 8:439-453, 2020.
Karol Grzegorezyk and Marcin Kurdziel. Disambiguated skip-gram model. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1445-1454, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/D18-111 174. URI.tttps:/ /wwww//www/.//////////////.// a e web.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. URI. http:://rx///i
org/ orr//bs// 503..... 02531
Eric H Huang, Richard Socher, Christopher D Manning, and Andrew Y Ng. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers- Volume 1, pp. 873-882. Association for Computational Linguistics, 2012.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tenybert: Distilling bert for natural language understanding. arXiv preprint arXiv: 1909:1999.1135/1 2019.
David Jurgens and Ioannis Klapaftis. SemEval-2011 task 13: Word sense induction for graded and non-graded senses. In Second Joint Conference Lexical and Computational Semantics ( (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). pp. 290-299, Atlanta, Georgia, USA, June 2013. Association for Computational Linguistics.
URL https:: / www. www.eb. wtt.:/ewworrg..nnttteee wtt.o//atoooogg/o111111..1 2049
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.69800 2014.
Guang-He Lee and Yun-Nung Chen. Muse: Modularizing unsupervised sense embeddings. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 327-337, 2017.
Jiwei Li and Dan Jurafsky. Do multi- sense embeddings improve natural language understanding? In Proceedings of the 2015 Conference on Empirical Methods Natural Language Processing, pp. 1722-1732, 2015.
Yinhan Liu. Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
10

---

Under review as a conference paper at ICLR 2022
Suresh Manandhar and Ioannis Klapaftis. SemEval-2010 task 14: Evaluation setting for word sense induction disambiguation systems. In Proceedings of the Workshop on Semantic Evalua- tions: Recent Achievements and Future Directions (SEW-2009), pp. 117-122, Boulder, Colorado,
June 2009. Association for Computational Linguistics. URL https: www ac web. org/
anthoologgy/N00222211
Tomas Mikolov, Iya Sutskever, Kai Chen, Greg S Corrado. and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111-3119, 2013.
Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. Efficient non- parametric estimation of multiple embeddings per word in vector space. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1059-1069, 2014.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Joseph Reisinger and Raymond J Mooney. Multi Multi-prototype vector-space models of word meaning. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp. 109-117. Association for Computational Linguistics, 2010.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 01 108, 2019.
Cyrus Shaoul and Chris Westbury. The westbury lab wikipedia corpus. 2010.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: ptt.:/988..99555 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert compact agno-tgi bert for resource- limited devices. arXiv preprint SKKV:22004.02884, 2020.
Fei Tian, Hanjun Dai, Jiang Bian, Bin Gao, Rui Zhang, Enhong Chen, and Yin Liu. A probabilistic model for learning multi-prototype word embeddings. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pp. 151-160, 2014.
Thomas Wolf, L Debut, V Sanh, J Chaumond, C Delangue, A Moi, P Cistac, T Rault, R Louf, M Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abv/1910.03771, 2019.


Reviews:
[{"review_id": "uF3zVon4Frj", "reviewer": "Reviewer_npDw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: Their paper is easy to follow. Surveys on the area are quite decent in the related work section. An idea of obtaining multi-sense word embedding is novel. In the word sense induction task, their method exhibits better performance than conventional ones.\n\nWeakness: \nResults of their proposed method in contextual word similarity tasks do not outperform existing methods.\nReaders cannot know the method’s uniqueness. In particular, semantic similarity task is not supporting this method’s superiority. Discussion of comparison between their proposed method and existing methods is not provided. \nAnother concern is that authors are visible through the footnote link to their GitHub account.\n\nMinor point\nWhy is BERTKDEmbed highlighted in Table2?", "summary_of_the_paper": "The authors proposed a method to obtain suitable multi-sense word embedding based on skip-gram and BERT distillation. In my opinion, from technological point of view, their idea is very novel. They also incorporated their word embedding into embedded topic modeling. ", "summary_of_the_review": "The authors build a new type of obtaining multi-sense word embedding is novel. Although the model is just comparable to existing methods in contextual word similarity tasks, the model and derivatives show potential for both the word sense induction task and the embed topic model. Thus, I found this paper is marginally significant.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hs3sS17kXKh", "reviewer": "Reviewer_xBJd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well-written and the proposed methodology is clear and straightforward. The models introduced in this work demonstrate state-of-the-art results. The authors compare with a wide range of previous approaches thoroughly described in Related Work.\n\nMy biggest concern with the current version of the paper is a low methodological novelty: the approach fairly similar to the previously proposed approaches and is not introducing some principally new ideas or mathematical methods. Also, the authors acknowledge that the presented distillation approach is already used for the downstream tasks.\n\nAnother concern is that, the results presented in the paper are not compared with other participants of SemEval tasks (and based on the SemEval metrics). Concerning comparison to other systems, I also suggest to compare to another simple approach for obtaining static sense embedding out of word embedding by Pelevina et al. (2017).\n\nIt would be also important to explain in more detail why the authors chose Topic Modelling as the extrinsic evaluation task. The choice of the exact three senses (k=3) is also not justified. Another concern is about the corpora the models are trained on. The authors insist that training models on the April 2010 snapshot of Wikipedia is for fair comparison with the prior multi-sense embedding models. At the same time they apply the base BERT model which is pretrained on the later version of Wikipedia.  \n", "summary_of_the_paper": "The paper studies the problem of word sense induction using knowledge of the large pre-trained model BERT. The authors propose a two-stage method to distill multiple word senses by using attention over the senses of a word in a context and transferring this sense information to fit multi-sense embeddings in a skip-gram-like framework.\n\nThis paper has the following contributions: a two-stage method for knowledge distillation from BERT to static word sense embeddings, a sense embeddings model and application of such embedding models to the topic modelling task. Topic modelling is used as an extrinsic evaluation. \n", "summary_of_the_review": "The paper proposes fairly straightforward, yet effective approach based on BERT. Intinsic evaluation setup and the structure of the paper is more or less identical to that of Bartunov et al. (2016). The section about the extrinsic evaluation of the proposed method should be extended (adding more details about the topic modelling), but preferably another use-case of the proposed approach. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_ZznDpSjZhv", "reviewer": "Reviewer_2C3z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe writing of the paper is easy to follow. And the paper makes a good attempt to leverage the pre-trained models. \n\nWeakness:\n\n1. One of the main weaknesses is that the results are not significant enough. In Table 3, the performance of the proposed model on both criteria is worse than the existing models. And the paper didn't compare with all existing sense models (see missing citations). In Figure 2, most of the improvements are brought by the ETM model, the benefits of using sense embeddings seem to be very marginal. The paper should also compare with more existing sense embeddings rather than just MSSG. The main improvements that the proposed model has compared to previous sense embeddings are on SemEval tasks, which are not supervised given that contextualized embeddings and pre-trained models perform well on these tasks themselves. The motivation of having sense embeddings is less strong (https://arxiv.org/abs/1909.10430, https://aclanthology.org/2020.coling-main.107.pdf). And the proposed distilled sense embedding performs worse than the BertSense itself. There's no reason for using the distilled embeddings rather than the BERT embeddings themselves. \n\n2. Another main weakness is the novelty of the technique of this paper. Equations 1,2,3,4 have been commonly used in the previous work started by Šuster et al. (2016). The distillation technique is also not novel. \n\nMinor weakness:\n\n1. SemEval is most commonly evaluated by Precision and Recalls, while in this paper, the ARI is used. It's better to add more explanation of what ARI is and why it's used.\n2. Missing citations:\n\nQiu et al. \"Context-dependent sense embedding.\" Association for Computational Linguistics (ACL), 2016\nŠuster et al. (2016). Bilingual learning of multi-sense embeddings with discrete autoencoders. In NAACL\nGuo et al. \"Which Evaluations Uncover Sense Representations that Actually Make Sense?.\" Proceedings of the 12th Language Resources and Evaluation Conference. 2020.\n", "summary_of_the_paper": "This paper proposes a sense embedding model that distills the sense induction probability conditional on context from pre-trained BERT. Experimental results on the word similarity tasks show that it is able to perform well on the word sense induction task and achieve a balance between the contextual and non-contextual word similarity tasks. An experiment on one downstream task, topic modeling, shows that the Embedded Topic Model with the proposed BERT distilled sense embedding is able to decrease the perplexity marginally. ", "summary_of_the_review": "The technique adopted in this paper is simple and not novel. The improvements are not significant. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vie8GbEKPv0", "reviewer": "Reviewer_oDq7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents a multi-sense word embedding model for polysemy words by distilling knowledge from the contextual word embedding model (BERT). The authors first incorporate an attention mechanism over the senses of a word into a skip-gram-like framework. Then, BERT-enhanced sense embeddings are used to improve the skip-gram framework by knowledge distillation. In experiments, the authors confirm the effectiveness of the sense embedding model on three tasks: word sense induction, contextual word similarity, and embedded topic model.\n\nOne limitation of the proposed method is the assumption that one word has a fixed number (three in the paper) of word senses, restricting its effectiveness and generalization in downstream tasks. I wonder why the authors chose three and how this would influence the WSI task if a word has more or less sense than 3 in reality?\n\nThere have been studies to perform word sense disambiguation with pre-trained models and sense inventory [1,2,3]. Without comparison and analysis of these critical works, it is not easy to verify the technical contribution of this paper. This is important since the proposed technical design (e.g., the attentional skip-gram model) is simple, though I do not see simplicity as a shortcoming.\n\nThe performance improvement on the contextual word similarity task looks marginal. Regarding the word induction task, BERT-based models perform much better than non-BERT-based models. The authors are encouraged to do some quantitative analysis on the time-consuming on BERTSense and BERTKDEmbed. Meanwhile, comparing further with some other existing distilled (or compressed) BERT models here will highlight BERTKDEmbed's effectiveness and efficiency. \n\nI find the overall training process is somewhat hard to follow. The clarity of the whole method can be improved, and some detailed design seems not well-motivated. For example, why are two individual embeddings d_i and v_i used in sense representation and disambiguation, respectively? In some similar works [3], coordinating sense and context embeddings is enough to perform disambiguation. In Sec. 3.2.1, which parameters are trainable? Is D^i_k here shared with the D^i_k  in Sec. 3.1?\n\n[1] GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge. EMNLP 2019 \n\n[2] Analysis and Evaluation of Language Models for Word Sense Disambiguation. Computational Linguistics 2021\n\n[3] Leveraging Human Prior Knowledge to Learn Sense Representations. ECAI 2020\n", "summary_of_the_paper": "This paper proposes a two-stage method to distill word sense knowledge from BERT into a skip-gram-like framework. The authors show that the proposed method performs well on sense selection and induction (or representation). They also verify the efﬁcacy on the downstream task of embedded topic modeling.", "summary_of_the_review": "The effectiveness and generalization of the proposed method are somewhat limited due to the fixed sense number. The novelty seems not strong, and missing high-related works make this paper difficult to position in the research background. The clarity also has room for improvement.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "I8zjatSbfBb", "reviewer": "Reviewer_pvxe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\nThis work is able to leverage the contextual knowledge of BERT to improve the performance of non-contextual word sense induction.\nThe performance of the proposed methods on downstream sense induction and word similarity is comparable to those of existing SoTA methods; The distilled sense embedding model yields lower topic perplexity in embedded topic modeling.\nThe combined worse sense and BERT distillation loss force the resulting distilled model to be performant in both word sense disambiguation and induction.\n\nConcerns\nThe authors propose static multisense embedding distillation as a solution to the latency concerns that emerge from running large-scale BERT models. There are no experiments showing the computational overhead of their proposed method.\nA set of nearest neighbor results shown in sec4.2 are used to suggest that the sense disambiguation network largely selects a single word sense. A stronger justification of this claim would be to show the distribution of sense probabilities computed over all examples rather than in selected examples.\n\nSuggestions & Nit:\nWould like to see a Related Work discussion of contextual word sense induction models (especially those using BERT-style transformers).\nhttps://arxiv.org/pdf/1909.08358.pdf\nhttps://arxiv.org/pdf/1908.07245.pdf \nTable 2/4: Bolding of the proposed method is misleading as it is not the most performant model. In contrast, highest accuracy is highlighted in Table 3.\n", "summary_of_the_paper": "Single-sense word representations which collapse multiple word meanings into a single vector representation are often used in settings where large model inference is impractical. The authors propose a method for extracting noncontextual multiple sense embeddings from transformer-based encoder models via knowledge distillation.\n\nThe author’s model predicts likelihoods for the different senses of a target word as the combination of a learned disambiguation vector for each sense and a context embedding constructed from a sliding window over the surrounding words (a-la skipgram). The output multi-sense embedding is the probability weighted sum of the individual sense embeddings. A distillation loss learning against the BERT sense likelihoods is used to improve disambiguation.\n", "summary_of_the_review": "Overall Recommendation: Lean Accept\nThe paper proposes a new noncontextual model for handling multiple word senses by constructing a sense embedding model with improved disambiguation learned via BERT knowledge distillation. As compared to existing work, this work is able to both perform word sense induction while also having strong performance on word similarity. This suggests that the model is able to both disambiguate and also learn strong underlying representations.\n\nI would like to see my above concerns addressed by the authors. For example, it would be beneficial to report the latency of model inference and parameter counts (as relevant) for the proposed sense model, BERT, and other baselines averaged over the test/val set in the word sense induction and word similarity tasks. Likewise re:Concern #2, I would like to see a histogram over the top word sense probabilities. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates a technique for projecting contextual embeddings into static embeddings. Neither the technique is ver novel, nor are the empirical results very strong. While the reviewers did not engage in a discussion, the area chair does not see this paper reaching the quality bar of the conference.
Decision: Reject