Year: 2021

Abstract:
The present study reports computational accounts of resolving word-level polysemy in a lesser-studied language—Korean. Postpositions, which are characterized as multiple form-function mapping and thus polysemous, pose a challenge to automatic analysis and model performance in identifying their functions. In this study, we devised a classification model by employing BERT and introduces a computational simulation that interactively demonstrates how a BERT model simulates human interpretation of word-level polysemy involving Korean adverbial postpositions -ey, -eyse, and -(u)lo. Results reveal that (i) there is an inverse relationship between the classification accuracy and the number of functions that each postposition manifests, (ii) the model performance is affected by the corpus size of each function, and (iii) the performance gradually improves as the epoch proceeds.

heading: 1 INTRODUCTION
text: Polysemy, one type of ambiguity, occurs when one form delivers multiple, and yet related, meanings/functions and vice versa (Glynn & Robinson, 2014). Traditional word-embedding models showed an unsatisfactory level of performance in polysemy interpretation. This is due to the technical nature of these models: they are static in that a single vector is assigned to each word (Desagulier, 2019; Ethayarajh, 2019; Liu et al., 2019a). To overcome this issue, recent studies have proposed a contextualized word-embedding model which considers neighborhood information about a polysemous word on the basis of sequences of words around the target word. Various models have been suggested for this task, such as Embeddings from Language Models (Peters et al., 2018), Generative Pre-Training (Radford et al., 2018), and Bidirectional Encoder Representations from Transformer (BERT; Devlin et al., 2018). Among these models, BERT shows the best performance in many language tasks such as translation, classification, and question-answering (e.g., Devlin et al., 2018; Tang et al., 2019).
Despite a good deal of research on BERT in English, very few studies have investigated BERT-based polysemy interpretation in languages that are typologically different from English. We turn our attention to Korean, an agglutinative Subject–Object–Verb language in which multiple postpositions or affixes with dedicated forms and meanings are attached to the stem of nominals or predicates. A postposition is a function word providing grammatical information to words it is attached (Sohn, 1999). It normally involves many-to-many associations between form and function; as such, a postposition is polysemous (Choo & Kwak, 2008).
Several studies have used word-embedding models to capture and tease apart the different meanings/functions of Korean postpositions (e.g., Bae et al., 2014; 2015; Kim & Ock, 2016; Lee et al., 2015; Mun & Shin, 2020; Shin et al., 2005). However, the model performance reported in the previous studies is unsatisfactory, with the accuracy ranging from 0.621 (Bae et al., 2014) to 0.837 (Kim & Ock, 2016). One possible reason for this unsatisfactory performance is that they did not consider contextual information. Against this background, the current study employs BERT for the same kind of classification task for Korean postpositions. BERT produces contextual embeddings, and this characteristic may help us to create a better classification system for postpositions. Still unclear is the particular reason for BERT’s superior performance over the others. In order to further understand how BERT recognizes the word-level polysemy, we propose a BERT-based visualization system in addressing polysemy interpretation of three adverbial postpositions, -ey, -eyse, and -(u)lo,
which are frequently used and documented in the previous studies (e.g., Cho & Kim, 1996; Jeong, 2010; Nam, 1993; Park, 1999; Song, 2014).
2 KOREAN ADVERBIAL POSTPOSITIONS: -ey, -eyse, AND -(u)lo
In order to determine the number of functions of each postposition, this study considers the major functions of these postpositions which are frequently attested in the Sejong dictionary: eight for -ey, two for -eyse, and six for -(u)lo (Shin, 2008). -ey involves the following functions: agent (AGT), criterion (CRT), effector (EFF), final state (FNS), goal (GOL), instrument (INS), location (LOC), and theme (THM).
(1) -ey as AGT (agent)
가두
katwu street
진출이
cinchwul-i go.out-NOM
경찰에
kyengchal-ey police-AGT 저지되었다. ceci-toy-ess-ta. stop-PSV-PST-DECL
‘By going out to the street was stopped by the police.’
(2) -ey as CRT (criterion)
영호는
Yenghuy-nun Yenghuy-TOP 20만원에 20manwen-ey 200,000 won-CRT 모니터를 monithe-lul moniter-ACC 낙찰했다. nakchalhay-ss-ta. sell-PST-DECL
‘Yenghuy sold the monitor (to a bidder) for 200,000 won.’
(3) -ey as EFF (effector)
문들이
mwun-tul-i door-PL-NOM
거센
keseyn strong
바람에
palam-ey wind-EFF
모두
motwu all 건들댄다. kentultay-n-ta. sway-PRS-DECL
‘The doors all sway by the strong wind.’
(4) -ey as FNS (final state)
김교수는
kimkyoswu-nun professor.Kim-TOP
조교에
cokyo-ey assistant-FNS
박군을
park-kwun-ul Park-Mr-ACC 추천했다. chwuchenhay-ss-ta. recommend-PST-DECL
‘Professor Kim recommended Park as an assistant.’
(5) -ey as GOL (goal)
철수가
Chelswu-ka Chelswu-TOP
던진
tenc-i-n throw-CST-PRS
칼이
khal-i knife-NOM
땅바닥에
ttangpatak-ey ground-GOL 내리꽂혔다. naylyekkoc-hi-ess-ta. stick-PSV-PST-DECL
‘The knife thrown by Chelswu stuck to the ground.’
(6) -ey as INS (instrument)
그
ku That
어린
eli-n young-REL
소년은
sonye-nun boy-TOP
화롯불에
hwalospwul-ey fire-INS
손을
son-ul hand-ACC
녹이고
nok-i-ko melt-CST-and 있었다. iss-ess-ta. be-PST-DECL
‘The young boy was using the fire to warm his hands.’
(7) -ey as LOC (location)
그는
ku-nun He-TOP
온종일
oncongil all day
서재에
secay-ey study.room-LOC
파묻혀
phamwut-hi-e bury.in-PSV-PRS 지낸다. cinay-n-ta. be-PRS-DECL
‘He is buried in his study room all day.’
(8) -ey as THM (theme)
현대인들은
hyentayin-tul-un modern.people-PL-TOP
모두
motwu all
참된
chamtoy-n true-REL
지식에
cisik-ey knowledge-THM 허기져있다. hekicye-iss-ta. hungry-PRS-DECL
‘All modern people are hungry for true knowledge.’
-eyse has only two functions, location (LOC) (9) and source (SRC) (10). -eyse manifests fewer functions than -ey (Choo & Kwak, 2008). However, frequency is equally high compared to that of -ey (e.g., Cho & Kim, 1996; Song, 2014).
(9) -eyse as LOC (location)
철수는
Chelswu-nun Chelswu-TOP
서울에서
sewul-eyse seoul-LOC 태어났다. thayena-ss-ta. born-PST-DECL
‘Chelswu was born in Seoul.’
(10) -eyse as SRC (source)
광부들이
kwangpwutul-i miner-PL-NOM
바다에서
pata-eyse sea-SRC
석유를
sekyu-lul oil-ACC 뽑아올린다. ppopaoll-i-n-ta. pull-CST-PRS-DECL
‘Miners pull oil from the sea.’
-(u)lo engages in six functions: criterion (CRT), direction (DIR), effector (EFF), final state (FNS), instrument (INS), and location (LOC) (Shin, 2008).
(11) -(u)lo as CRT (criterion)
적당한
cektangha-n appropriate-REL
시간
sikan time
간격으로
kankyek-ulo interval-CRT 배차되었다. paycha-toy-ess-ta. arrange-PSV-PST-DECL
‘It was arranged at appropriate time intervals.’
(12) -(u)lo as DIR (direction)
범인은
pemin-un criminal-NOM
어두운
etwuwun dark
골목으로
kolmok-ulo alley-DIR 달아났다. talana-ss-ta. flee-PST-DECL
‘The criminal fled into a dark alley.’
(13) -(u)lo as EFF (effector)
환자가
hwanca-ka patient-NOM
위암으로
wiam-ulo stomach.cancer-EFF
매우
maywu very
괴로워하고
koyloweha-ko suffer-and 있습니다. iss-supni-ta. be-HON-DECL
‘The patient is suffering greatly due to stomach cancer.’
(14) -(u)lo as FNS (final state)
그는
ku-nun He-TOP
대표
tayphyo representative
강사로
kangsa-lo lecturer-FNS 초빙되었다. choping-toy-ess-ta. invite-PSV-PST-DECL
‘He was invited as a representative lecturer.’
(15) -(u)lo as INS (instrument)
전선이
censen-i wire-NOM
연줄로
yencwul-lo connection.wire-INS 감겼다. kam-ki-ess-ta. wind-PSV-PST-DECL
‘The wire wound around with the connection wire.’
(16) -(u)lo as LOC (location)
경찰이
kyengchal-i police-NOM
피해자를
phiuyca-lul suspect-ACC
검찰로
kemchal-lo prosecution-LOC 압송했다. apsonghay-ss-ta. transport.do-PST-DECL
‘The police transported the suspect to the prosecution.’

heading: 3 METHODS

heading: 3.1 CREATING INPUT CORPUS
text: The Sejong primary corpus1, the representative corpus in Korean, does not code the information about the functions of postpositions directly in each sentence (which is necessary for model training). We thus annotated a portion of the original corpus data manually. For this purpose, we extracted sentences that have only one postposition and predicate. This also allowed us to control for additional confounding factors which might have interfered with the performance of our model. We then extracted 5,000 sentences randomly for each postposition from the initial dataset.
1Sejong corpus is available at:https://www.korean.go.kr
Three native speakers of Korean annotated each postposition for its function in this 15,000-sentence corpus. Fleiss’ kappa scores showed that the annotators’ outcomes were almost identical: 0.948 (- ey), 0.928 (-eyse), and 0.947 (-(u)lo). We further excluded instances which showed disagreement among the annotators. The final corpus consisted of 4,715 sentences for -ey, 4,853 sentences for -eyse, and 4,708 sentences for -(u)lo. Table 1 presents the detailed by-function frequency list of the three postpositions 2.

heading: 3.2 CREATING TRAINING AND TEST SETS
text: We pre-processed the data in consideration of how BERT works (we used the original BERT model for this task). First, we added [CLS] (‘classification’; indicating the start of a sentence) before a sentence and [SEP] (‘separation’; indicating the end of a sentence) after a sentence to indicate where the sentence starts and ends. These indicators made it possible for the BERT model to recognize a sentence boundary in a text, allowing the model to learn word meaning while considering intersentential variations. Second, we made a separate column (‘Label’) to indicate the intended function of each postposition in each sentence (Figure 1). We then split the corpus into two sub-sets, one with 90 per cent of the corpus for the training and with the remaining 10 per cent of the corpus for the testing.

heading: 3.3 DEVELOPING BERT CLASSIFICATION MODEL
text: We set the parameters related to BERT training such as batch size (32), epoch (50), seed (42), epsilon (0.00000008), and learning rate (0.00002), as advised by McCormick (2019). We then employed a pre-trained language model in order to obtain high accuracy of outcomes; for this purpose, we used a Korean BERT model (KoBERT; Jeon et al., 2019). Before the actual BERT training, we transformed the input data into three embedding types—token embeddings, position embeddings, and segment embeddings (cf., Devlin et al., 2018)—in the following ways.
First, for the token embedding, we used KoBertTokenizer for the sentence tokenization (the maximum number of tokens for each sentence was set to 128). Second, we converted each token into numeric values indicating unique indices of the tokens in the vocabulary of KoBERT for the position embedding. Third, for the segment embedding, we converted the number of tokens of each sentence into 128 numeric values using 0 (i.e., not existed) or 1 (i.e., existed). The labels of the data indicating the intended function of each postposition in the sentence were stored separately.
After this transformation step, we proceeded to the model training as follows. We first loaded KoBERT through the function BertForSequenceClassification from transformers (Wolf et al., 2019).
2Our corpus is available at: we will propose a URL
Next, we fine-tuned the pre-trained model by using the training set, with a view to reducing loss values and updating the learning rate for better classification accuracy of the model. We then loaded the testing set to evaluate whether the fine-tuned model successfully recognized the intended functions of each postposition in each sentence. In this part, the rates of accuracy for each function and the total accuracy rate were calculated by comparing the intended function of each postposition in each test sentence with the classified function of each postposition via the BERT model. Lastly, we employed t-distributed Stochastic Neighbor Embedding (t-SNE; Maaten & Hinton, 2008) for dimension reduction of classification embeddings from the postposition per epoch. In addition, to statistically confirm the changes of sentence-level embedding outcomes by each epoch, we performed densitybased clustering (Sander et al., 1998). These outcomes were fed into the visualization system, which we outline next.

heading: 3.4 DEVELOPING VISUALIZATION SYSTEM
text: We designed a visualization system with JavaScript, HTML, and CSS environments, using the test set under the two-dimensional distribution. For the interface of this system, we created three areas showing model performance: a distributional map for sentence-level embeddings, accuracy/loss charts relating to the model, and graphs for the density-based clustering. To manipulate visualization outcomes, Figure 2(a) provides options to select the postpositions and checkboxes to highlight and tracking interesting sentences according to the index number or the function of these postpositions. The distributional map as in Figure 2(b) presents the relationship between the sentences with the selected postposition (represented as dots) involving different functions (represented as colors). A slider at the bottom of the map allows for changing the epochs; the patterns of clustering change as the slider moves. Each dot shows the details of the sentence (e.g., an index of the selected sentence, the intended function used in the sentence, the original sentence) once the mouse pointer is located on the dot. The right side of the system as in Figure 2(c) provides users with various information about the model performance: overall accuracy, by-function accuracy, and loss rates in the classification task by epoch. This section also provides accuracy rates of each function by hovering around the mouse pointer onto the specific-colored lines. The bar chart at the bottom right side of the system presents the number of clusters produced by the model. This chart also provides a hovering function, providing the actual number of clusters per epoch. The particular hovering activity is interlocked with the density cluster view, located at the bottom left of the system, by presenting the clustering results according to the selected epoch.

heading: 4 RESULTS: THREE CASE STUDIES
text: In order to reports the BERT model performance of classifying the functions of postpositions and assess how our visualization system works, we conducted three case studies.

heading: 4.1 DOES THE NUMBER OF FUNCTIONS FOR A POSTPOSITION AFFECT MODEL PERFORMANCE?
text: The presenting tables (Tables 2-4) show the classification accuracy of the BERT model for each postposition. The result show that the BERT model performed better for -eyse, which has only two functions (SRC and LOC), than for the other two postpositions (-ey and -(u)lo). The average classification accuracy for -ey, -eyse and -(u)lo is a satisfactory level of accuracy considering previous reports (Bae et al., 2014; Kim & Ock, 2016).
To statistically explore the classification by postposition/epoch, we performed a two-way ANOVA. As Table 5 shows, there is no statistical significance in the accuracy across the postpositions but significant difference in the accuracy across the epochs. This indicates the general tendency that model performance improved in proportion to the number of epochs (see also case study 3).
Note. * < .05
We conducted additional by-postposition pairwise comparisons through a two-sample t-test. As Table 6 shows, the model performance in -eyse is significantly better than in the other two postpositions. Considering the different number of functions (e.g., two for -eyse, six for -(u)lo, and eight for -ey), this finding indicates an inverse relationship between the classification accuracy and the number of functions that each postposition manifests.
Note. *** < .001

heading: 4.2 DO THE ASYMMETRIC PROPORTIONS OF THE FUNCTIONS IN EACH POSTPOSITION AFFECT THE MODEL PERFORMANCE?
text: The answer is they do. The average classification accuracy of each function for -ey is the highest for LOC (0.947) and the lowest for AGT (0.041); for -eyse, it is the highest in LOC (0.948) and the lowest in SRC (0.535); for -(u)lo, it is the highest in DIR (0.938) and the lowest in LOC (0.106) (Tables 2-4). As for the occurrences of individual functions per postposition, LOC for -ey, LOC for -eyse, and DIR for -(u)lo account for the larger portion of the entire corpus than other functions (see Table 1). This finding thus indicates that the model performance is affected by the asymmetric proportions of the functions comprising the use of each postposition.

heading: 4.3 HOW DOES THE BERT MODEL CLASSIFY SENTENCES BASED ON THE POSTPOSITIONS’ FUNCTIONS AS THE EPOCH PROGRESSES?
text: Our visualization system showed that the model was able to recognize the functions of each postposition as the epoch progressed. For -ey, all of the sentences were divided into two groups when the epoch was one, but as the epoch progressed, the sentences were divided into three in Epoch 7, four in Epoch 12, and five in Epoch 15. For -eyse, the number of clusters was one when the epoch was one, and there were two clusters when the epoch was nine. For -(u)lo, the number of clusters increased, starting from one (Epoch 1) to three (Epoch 4), five (Epoch 12), and six (Epoch 46). In particular, for -(u)lo, there are two interesting findings. First, in Epoch 12 (Figure 3), a cluster of EFF (the function with low-frequency occurrences in the data) emerged. This finding indicates that the BERT can identify functions at a satisfactory level, even though they are relatively infrequent, as long as there are sufficient epochs provided. Second, interestingly, LOC could not form a designated cluster in the end. Highlighting and zooming into the individual instances of LOC (Figure 4), we found that many of the LOC instances (11 out of 15) belonged to the DIR group. This is due to (i) the low frequency of LOC in the data and (ii) the semantic closeness between DIR and LOC—they relate to a location and are often difficult to distinguish one from another. This finding indicates that there are still some limitations in regard to the identification of functions given the above complications.

heading: 5 CONCLUSION
text: In this study, we note three major findings. First, there is an inverse relation between the classification accuracy and the number of functions of each postposition. Second, the model is affected by the corpus size of each function. Third, the model can identify the intended functions of a postposition as the epoch progresses, even though the corpus size of a function is small. However, despite these findings, our BERT model still seems to be affected by the scarcity of input and/or semantic closeness between the items, limiting its performance in the given task to some extent. We believe our visualization system will contribute to extending the current understanding of how BERT works for language tasks (particularly in non-English settings).
The findings of this study should be further verified by incorporating more postposition types that have similar degrees of polysemy that three adverbial postpositions demonstrate, which we plan to pursue next. Researchers will also benefit from considering other contextualized word-embedding models such as Generation Pre-trained Transformer 3 (Brown et al., 2020) or a Robustly Optimized BERT Pretraining Approach (Liu et al., 2019b) to better ascertain the advantage of BERT in this kind of task.

Review summaries:
This paper builds a BERT classifier for word-level polysemy of Korean language. Adverbial positions such as -ey, eyes, and –(u)lo are involved in the investigation. There are three findings during the experiments, if there are more functions then the classification accuracy is lower, corpus size is sensitive and more epochs are preferred.

The paper presents a study of training a BERT-style model on the task of function identification for 3 different Korean adverbial postpositions(ey, eyse and (u)lo). The paper presents a new dataset for the task, show that training a BERT model performs well and also introduce a method to visualise BERT embeddings as the training proceeds. The authors show that the model performs better for postpositions which have fewer associated functions (classes for classification), performs better for classes that occur more frequently in data and that model performance improves with additional epochs. 

The technical essence of the paper is to use BERT to perform classification and provide basic visualization of the change of contextual word embeddings during training. The classification problem concerns polysemy in Korean: for each adverbial postposition (e.g. -eyse), classify the use of the postposition in a given sentence into one of the available functions (e.g. LOC or SRC), where the number of classes range from two to eight.  The contributions of this paper are as follows: the paper studies a relatively understudied language, Korean; the authors made progress on a very specific instance of polysemy classification problem by using contextual word embeddings compared to the previous approach that uses traditional static word embeddings (albeit this claim is not supported by the experiments); and the authors created a visualization tool for the word embeddings and model performance.