Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the problem of word sense induction using knowledge of the large pre-trained model BERT. The authors propose a two-stage method to distill multiple word senses by using attention over the senses of a word in a context and transferring this sense information to fit multi-sense embeddings in a skip-gram-like framework.

This paper has the following contributions: a two-stage method for knowledge distillation from BERT to static word sense embeddings, a sense embeddings model and application of such embedding models to the topic modelling task. Topic modelling is used as an extrinsic evaluation. 

main_review: Overall, the paper is well-written and the proposed methodology is clear and straightforward. The models introduced in this work demonstrate state-of-the-art results. The authors compare with a wide range of previous approaches thoroughly described in Related Work.

My biggest concern with the current version of the paper is a low methodological novelty: the approach fairly similar to the previously proposed approaches and is not introducing some principally new ideas or mathematical methods. Also, the authors acknowledge that the presented distillation approach is already used for the downstream tasks.

Another concern is that, the results presented in the paper are not compared with other participants of SemEval tasks (and based on the SemEval metrics). Concerning comparison to other systems, I also suggest to compare to another simple approach for obtaining static sense embedding out of word embedding by Pelevina et al. (2017).

It would be also important to explain in more detail why the authors chose Topic Modelling as the extrinsic evaluation task. The choice of the exact three senses (k=3) is also not justified. Another concern is about the corpora the models are trained on. The authors insist that training models on the April 2010 snapshot of Wikipedia is for fair comparison with the prior multi-sense embedding models. At the same time they apply the base BERT model which is pretrained on the later version of Wikipedia.  

summary_of_the_review: The paper proposes fairly straightforward, yet effective approach based on BERT. Intinsic evaluation setup and the structure of the paper is more or less identical to that of Bartunov et al. (2016). The section about the extrinsic evaluation of the proposed method should be extended (adding more details about the topic modelling), but preferably another use-case of the proposed approach.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a method to obtain suitable multi-sense word embedding based on skip-gram and BERT distillation. In my opinion, from technological point of view, their idea is very novel. They also incorporated their word embedding into embedded topic modeling. 
main_review: Strength: Their paper is easy to follow. Surveys on the area are quite decent in the related work section. An idea of obtaining multi-sense word embedding is novel. In the word sense induction task, their method exhibits better performance than conventional ones.

Weakness: 
Results of their proposed method in contextual word similarity tasks do not outperform existing methods.
Readers cannot know the method’s uniqueness. In particular, semantic similarity task is not supporting this method’s superiority. Discussion of comparison between their proposed method and existing methods is not provided. 
Another concern is that authors are visible through the footnote link to their GitHub account.

Minor point
Why is BERTKDEmbed highlighted in Table2?
summary_of_the_review: The authors build a new type of obtaining multi-sense word embedding is novel. Although the model is just comparable to existing methods in contextual word similarity tasks, the model and derivatives show potential for both the word sense induction task and the embed topic model. Thus, I found this paper is marginally significant.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a sense embedding model that distills the sense induction probability conditional on context from pre-trained BERT. Experimental results on the word similarity tasks show that it is able to perform well on the word sense induction task and achieve a balance between the contextual and non-contextual word similarity tasks. An experiment on one downstream task, topic modeling, shows that the Embedded Topic Model with the proposed BERT distilled sense embedding is able to decrease the perplexity marginally. 
main_review: Strength:

The writing of the paper is easy to follow. And the paper makes a good attempt to leverage the pre-trained models. 

Weakness:

1. One of the main weaknesses is that the results are not significant enough. In Table 3, the performance of the proposed model on both criteria is worse than the existing models. And the paper didn't compare with all existing sense models (see missing citations). In Figure 2, most of the improvements are brought by the ETM model, the benefits of using sense embeddings seem to be very marginal. The paper should also compare with more existing sense embeddings rather than just MSSG. The main improvements that the proposed model has compared to previous sense embeddings are on SemEval tasks, which are not supervised given that contextualized embeddings and pre-trained models perform well on these tasks themselves. The motivation of having sense embeddings is less strong (https://arxiv.org/abs/1909.10430, https://aclanthology.org/2020.coling-main.107.pdf). And the proposed distilled sense embedding performs worse than the BertSense itself. There's no reason for using the distilled embeddings rather than the BERT embeddings themselves. 

2. Another main weakness is the novelty of the technique of this paper. Equations 1,2,3,4 have been commonly used in the previous work started by Šuster et al. (2016). The distillation technique is also not novel. 

Minor weakness:

1. SemEval is most commonly evaluated by Precision and Recalls, while in this paper, the ARI is used. It's better to add more explanation of what ARI is and why it's used.
2. Missing citations:

Qiu et al. "Context-dependent sense embedding." Association for Computational Linguistics (ACL), 2016
Šuster et al. (2016). Bilingual learning of multi-sense embeddings with discrete autoencoders. In NAACL
Guo et al. "Which Evaluations Uncover Sense Representations that Actually Make Sense?." Proceedings of the 12th Language Resources and Evaluation Conference. 2020.

summary_of_the_review: The technique adopted in this paper is simple and not novel. The improvements are not significant.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Single-sense word representations which collapse multiple word meanings into a single vector representation are often used in settings where large model inference is impractical. The authors propose a method for extracting noncontextual multiple sense embeddings from transformer-based encoder models via knowledge distillation.

The author’s model predicts likelihoods for the different senses of a target word as the combination of a learned disambiguation vector for each sense and a context embedding constructed from a sliding window over the surrounding words (a-la skipgram). The output multi-sense embedding is the probability weighted sum of the individual sense embeddings. A distillation loss learning against the BERT sense likelihoods is used to improve disambiguation.

main_review: Strengths
This work is able to leverage the contextual knowledge of BERT to improve the performance of non-contextual word sense induction.
The performance of the proposed methods on downstream sense induction and word similarity is comparable to those of existing SoTA methods; The distilled sense embedding model yields lower topic perplexity in embedded topic modeling.
The combined worse sense and BERT distillation loss force the resulting distilled model to be performant in both word sense disambiguation and induction.

Concerns
The authors propose static multisense embedding distillation as a solution to the latency concerns that emerge from running large-scale BERT models. There are no experiments showing the computational overhead of their proposed method.
A set of nearest neighbor results shown in sec4.2 are used to suggest that the sense disambiguation network largely selects a single word sense. A stronger justification of this claim would be to show the distribution of sense probabilities computed over all examples rather than in selected examples.

Suggestions & Nit:
Would like to see a Related Work discussion of contextual word sense induction models (especially those using BERT-style transformers).
https://arxiv.org/pdf/1909.08358.pdf
https://arxiv.org/pdf/1908.07245.pdf 
Table 2/4: Bolding of the proposed method is misleading as it is not the most performant model. In contrast, highest accuracy is highlighted in Table 3.

summary_of_the_review: Overall Recommendation: Lean Accept
The paper proposes a new noncontextual model for handling multiple word senses by constructing a sense embedding model with improved disambiguation learned via BERT knowledge distillation. As compared to existing work, this work is able to both perform word sense induction while also having strong performance on word similarity. This suggests that the model is able to both disambiguate and also learn strong underlying representations.

I would like to see my above concerns addressed by the authors. For example, it would be beneficial to report the latency of model inference and parameter counts (as relevant) for the proposed sense model, BERT, and other baselines averaged over the test/val set in the word sense induction and word similarity tasks. Likewise re:Concern #2, I would like to see a histogram over the top word sense probabilities.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a two-stage method to distill word sense knowledge from BERT into a skip-gram-like framework. The authors show that the proposed method performs well on sense selection and induction (or representation). They also verify the efﬁcacy on the downstream task of embedded topic modeling.
main_review: This paper presents a multi-sense word embedding model for polysemy words by distilling knowledge from the contextual word embedding model (BERT). The authors first incorporate an attention mechanism over the senses of a word into a skip-gram-like framework. Then, BERT-enhanced sense embeddings are used to improve the skip-gram framework by knowledge distillation. In experiments, the authors confirm the effectiveness of the sense embedding model on three tasks: word sense induction, contextual word similarity, and embedded topic model.

One limitation of the proposed method is the assumption that one word has a fixed number (three in the paper) of word senses, restricting its effectiveness and generalization in downstream tasks. I wonder why the authors chose three and how this would influence the WSI task if a word has more or less sense than 3 in reality?

There have been studies to perform word sense disambiguation with pre-trained models and sense inventory [1,2,3]. Without comparison and analysis of these critical works, it is not easy to verify the technical contribution of this paper. This is important since the proposed technical design (e.g., the attentional skip-gram model) is simple, though I do not see simplicity as a shortcoming.

The performance improvement on the contextual word similarity task looks marginal. Regarding the word induction task, BERT-based models perform much better than non-BERT-based models. The authors are encouraged to do some quantitative analysis on the time-consuming on BERTSense and BERTKDEmbed. Meanwhile, comparing further with some other existing distilled (or compressed) BERT models here will highlight BERTKDEmbed's effectiveness and efficiency. 

I find the overall training process is somewhat hard to follow. The clarity of the whole method can be improved, and some detailed design seems not well-motivated. For example, why are two individual embeddings d_i and v_i used in sense representation and disambiguation, respectively? In some similar works [3], coordinating sense and context embeddings is enough to perform disambiguation. In Sec. 3.2.1, which parameters are trainable? Is D^i_k here shared with the D^i_k  in Sec. 3.1?

[1] GlossBERT: BERT for Word Sense Disambiguation with Gloss Knowledge. EMNLP 2019 

[2] Analysis and Evaluation of Language Models for Word Sense Disambiguation. Computational Linguistics 2021

[3] Leveraging Human Prior Knowledge to Learn Sense Representations. ECAI 2020

summary_of_the_review: The effectiveness and generalization of the proposed method are somewhat limited due to the fixed sense number. The novelty seems not strong, and missing high-related works make this paper difficult to position in the research background. The clarity also has room for improvement.
review_title: null

Meta-review: This paper investigates a technique for projecting contextual embeddings into static embeddings. Neither the technique is ver novel, nor are the empirical results very strong. While the reviewers did not engage in a discussion, the area chair does not see this paper reaching the quality bar of the conference.
Decision: Reject