How does BERT address polysemy of Korean adverbial postpositions -ey, -eyse, and -(u)lo?

Abstract: The present study reports computational accounts of resolving word-level polysemy in a lesser-studied language—Korean. Postpositions, which are characterized as multiple form-function mapping and thus polysemous, pose a challenge to automatic analysis and model performance in identifying their functions. In this study, we devised a classification model by employing BERT and introduces a computational simulation that interactively demonstrates how a BERT model simulates human interpretation of word-level polysemy involving Korean adverbial postpositions -ey, -eyse, and -(u)lo. Results reveal that (i) there is an inverse relationship between the classification accuracy and the number of functions that each postposition manifests, (ii) the model performance is affected by the corpus size of each function, and (iii) the performance gradually improves as the epoch proceeds.

Reviews and author responses:

This paper builds a BERT classifier for word-level polysemy of Korean language. Adverbial positions such as -ey, eyes, and –(u)lo are involved in the investigation. There are three findings during the experiments, if there are more functions then the classification accuracy is lower, corpus size is sensitive and more epochs are preferred. It is currently difficult for me to evaluate the technical novelty or methodology novelty of this paper. I could find less baseline systems of this target problem. The target question is more alike a specific NLP field of Korean laugange.

Detailed questions and comments:
1.	Figure 1, possibly adding English annotations to these Korean sentences is a better choice?
2.	Are there any other methods investigated for comparison with BERT based models?
3.	Any details of BERT’s detailed configurations, parameter sizes, hardware used?
4.	Any downstream applications of using the trained bert models?
 Strong:
1.	Detailed bert classification construction and analysis for Korean word-level polysemy;
2.	Rich examples with figures.
Weak:
1.	This is more alike an experiment report of reusing existing BERT network for a specific language related classification task;
2.	The three findings are kind of general in deep learning domain and currently it is difficult to learn novel discoveries from them.


The paper presents a study of training a BERT-style model on the task of function identification for 3 different Korean adverbial postpositions(ey, eyse and (u)lo). The paper presents a new dataset for the task, show that training a BERT model performs well and also introduce a method to visualise BERT embeddings as the training proceeds. The authors show that the model performs better for postpositions which have fewer associated functions (classes for classification), performs better for classes that occur more frequently in data and that model performance improves with additional epochs.
 Strengths of the paper:
* The paper studies the identification of functions of 3 different Korean postpositions. The task is novel and it is good to see an analysis on a lesser studied language. 
* The paper reports good performance on the task by leveraging BERT

Weaknesses of the paper:
1. I think there is a misalignment between the title of the paper and the paper itself. The title "How does BERT address polysemy of Korean Adverbial Postpositions", in my opinion, indicates that the study is about if polysemy is well characterised by (multilingual)BERT embeddings and if it is a consequent of the MLM style training. However, the paper presents the analysis on the embeddings of a classification model trained to identify function classes. Any well trained classifier would have separation of feature embeddings for different classes, so it is not surprising that a fine-tuned BERT model also does. But the analysis fails to answer the question if this is also inherently present in the non finetuned BERT embeddings (i.e if BERT addresses polysemy of Korean Adverbial Postpositions and if yes, then how)
2. While the authors have presented examples associated with different functions, it would be good to actually define what the functions denote. I am still not sure about what some of the functions (eg: criterion / final state etc.) mean. If space is a concern, then the definitions as well as examples can be shifted to an Appendix section.
3. The dataset created in the paper assumes presence of only one postposition and predicate. This also limits the proposed approach: there is no clear way of handling sentences where the same postposition can occur multiple times, associated with different functions. In my opinion, this limitation is artificial: there is no language specific reason for this constraint.
4. The frequency of some of the classes is quite low (eg: for the AGENT function for ey, the number of samples is 47). If the testing set is 10% of this, then the number of samples in the test set with this class is ~5. This may not be a good indicator of performance.
5. The authors indicate that they used fixed values for hyperparameters (lr 2e-5, batch size 32 etc.). However, hyperparameter tuning is very dataset specific: some tuning (instead of using fixed values) might help improve performance 
6. Since a new dataset is introduced, the methodology presented in previous work should be trained on the new dataset to provide a fair comparison against the proposed method. In the current version, this crucial detail is not present.
7. The issues around under-represented classes not achieving good performance is a classic case of class imbalance, and is well studied in literature. The authors can look into the relevant literature to address the issue (eg: a quick search results in [1,2]. Might be good place to start)
8. The posited findings that having multiple functions (classes) for classification make the problem harder, increased training helps improve performance and under-represented classes have worse accuracies are trends that would occur commonly for any learning task, and in their current form don't provide any new insight about the specific task.
9. The T-SNE visualizations of embeddings as the training progresses does not provide any insight: a well trained classifier should map feature representations of different classes into separable clusters as training progresses. In my opinion, an in depth analysis of the embedding space of an unsupervised BERT model (i.e not trained on the classification task) vis a vis the task would be more insightful and instrumental in answering how BERT handles polysemy for postpositions.

[1] Mateusz Buda, Atsuto Maki, and Maciej A Mazurowski. A systematic study of the class
imbalance problem in convolutional neural networks. Neural Networks, 106:249–259, 2018
[2] Dong, Qi, Shaogang Gong, and Xiatian Zhu. "Imbalanced deep learning by minority class incremental rectification." IEEE transactions on pattern analysis and machine intelligence 41.6 (2018): 1367-1381. While the task is novel and it is good to see analysis on less well studied languages, this paper in its current form needs additional iterations in terms of technical contributions for the paper to be accepted. Additionally, there are concerns related to the research question, experimental setup and baselines (please see the Main Review above) that should be addressed.

The technical essence of the paper is to use BERT to perform classification and provide basic visualization of the change of contextual word embeddings during training. The classification problem concerns polysemy in Korean: for each adverbial postposition (e.g. -eyse), classify the use of the postposition in a given sentence into one of the available functions (e.g. LOC or SRC), where the number of classes range from two to eight.

The contributions of this paper are as follows: the paper studies a relatively understudied language, Korean; the authors made progress on a very specific instance of polysemy classification problem by using contextual word embeddings compared to the previous approach that uses traditional static word embeddings (albeit this claim is not supported by the experiments); and the authors created a visualization tool for the word embeddings and model performance. The contribution of the paper is very marginal from both NLP and HCI perspectives.

From a NLP perspective, the proposed method is to use BERT in replacement of static word embeddings. Apart from the main technical contribution being the mere adaptation of the existing technique (which is not necessarily a bad thing), the paper fails to provide any meaningful comparison between the proposed method and previous baselines or other alternatives (although they imply that their method is better than previous methods, saying “the model performance reported in the previous studies is unsatisfactory”). Furthermore, the three findings from case studies (Section 4) basically restate the common knowledge about training any ML model and do not provide any new insights from the study. Most importantly, it appears that their data creation and experiments do not follow standard practices in NLP: the annotated data do not have a validation set; the model was fine-tuned for 50 epochs despite the small size of the dataset and the convergence around 20th epoch, etc. These are taken as signals that the experiments are not performed in a correct manner. 

From a HCI perspective, the visualization system presented in the paper does not include any supporting evidence for its usefulness and effectiveness (either using standalone metrics through user study or in comparison to other existing visualization tools). The system contains t-SNE visualization as well as a few model performance metrics such as accuracy, which closely resembles TensorBoard. More explanations and justifications regarding the validity and contribution of the proposed visualization system are required. 

Lastly, the specific instance of Korean polysemy classification (-ey, -eyse, and -(u)lo) is an interesting subject of study, but is likely to be too narrow to fully demonstrate and justify the effectiveness of the proposed method. In addition, as of now, the classification problem only considers from two to eight classes, and  the method heavily depends on the manual annotation of data, which are both not ideal.

Overall, I would recommend improving upon the depth and breadth of the subject (studying 100+ instances of such polysemy and integrating insights/intuitions into the model training) and connecting the findings about the model (when it performs well vs. poorly) to observed human behavior/perception/understanding for such phenomenon from linguistics -- maybe NLP conferences are a more suitable venue for this. Also, more discussion about related areas such as word sense disambiguation and lexical substitution is expected from this work. The contribution of the paper is very marginal from both NLP and HCI perspectives, and is not well-supported by the experiments.

This paper presents a BERT-based classification to predict a function of an adverbial postposition. Furthermore, this paper presents a visualization system to interpret the predicted results. Pros:
+ This paper introduces three Korean adverbial postpositions (-ey, -eyse, and –(u)lo) with examples.

+ This paper constructs a human-annotated dataset for this research.

Concerns:
- The key concern about the paper is the lack of description about the usefulness and the setting of this study. Despite the paper mentioning that there is a gap between English and Korean, there is no description about why we need to solve polysemy of the Korean adverbial postpositions. Furthermore, since this paper does not specify the reason why only three adverbial postpositions -ey, -eyse, and –(u)lo are chosen, this paper should clarify why those are selected.

- Another key concern is the lack of model novelty used in this paper. This paper adopts BERT. 
That is, this paper lacks rigorous experimentation about the proposed method. 

- This paper uses the [CLS] token to predict adverbial postposition’s function. How about using the adverbial postposition tokens compared to [CLS] token? In other words, an ablation study on network would have been very useful. 

- This paper extracts sentences that have only one postposition and predict. This seems that this paper makes the task simple, but there is no justification for this. 

- According to Table 1, the annotated data is quite imbalanced. However, this paper uses accuracy as the evaluation metric. At least, this paper should have used the F1-measure as the evaluation metric.

- All experiments are done without the validation dataset. Showing the performances per epoch is meaningless since we do not know test data in advance. Furthermore, there is no point in showing the average performance of epoch 1, 10, 20, 30, 40, 50.

- The claim that the average classification accuracy is at a satisfactory level considering previous reports has not been clearly addressed because the dataset used in this paper and the ones used in previous studies are not the same.

- On the same note, the findings of this paper is not compared with other methods. Without experimental or mathematical comparison with the other published methods, it can't to prove the superiority of the proposed method. 

- Explaining visualization system textually is unintuitive. It would be better to convert this paper to a system demonstration paper. Predicting a function of adverbial postposition is interesting topic, however the method used in this paper is too naïve and the evaluation settings are incorrectly set.

Meta-review: This paper presents a study on identifying word-level polysemy in Korean based on a language-specific BERT model.  It is always very good to see language-specific studies especially in non-English scenarios.  However, as the reviewers point out, there could be significant improvements to the experimental design of the paper (see comments from reviewer eqSx) and also the HCI aspects of the visualization setup (see comments from reviewer MV2W) to transform it to an acceptable ICLR paper.

Decision: Reject