How does BERT address polysemy of Korean adverbial postpositions -ey, -eyse, and -(u)lo?

1 INTRODUCTION . Polysemy , one type of ambiguity , occurs when one form delivers multiple , and yet related , meanings/functions and vice versa ( Glynn & Robinson , 2014 ) . Traditional word-embedding models showed an unsatisfactory level of performance in polysemy interpretation . This is due to the technical nature of these models : they are static in that a single vector is assigned to each word ( Desagulier , 2019 ; Ethayarajh , 2019 ; Liu et al. , 2019a ) . To overcome this issue , recent studies have proposed a contextualized word-embedding model which considers neighborhood information about a polysemous word on the basis of sequences of words around the target word . Various models have been suggested for this task , such as Embeddings from Language Models ( Peters et al. , 2018 ) , Generative Pre-Training ( Radford et al. , 2018 ) , and Bidirectional Encoder Representations from Transformer ( BERT ; Devlin et al. , 2018 ) . Among these models , BERT shows the best performance in many language tasks such as translation , classification , and question-answering ( e.g. , Devlin et al. , 2018 ; Tang et al. , 2019 ) . Despite a good deal of research on BERT in English , very few studies have investigated BERT-based polysemy interpretation in languages that are typologically different from English . We turn our attention to Korean , an agglutinative Subject–Object–Verb language in which multiple postpositions or affixes with dedicated forms and meanings are attached to the stem of nominals or predicates . A postposition is a function word providing grammatical information to words it is attached ( Sohn , 1999 ) . It normally involves many-to-many associations between form and function ; as such , a postposition is polysemous ( Choo & Kwak , 2008 ) . Several studies have used word-embedding models to capture and tease apart the different meanings/functions of Korean postpositions ( e.g. , Bae et al. , 2014 ; 2015 ; Kim & Ock , 2016 ; Lee et al. , 2015 ; Mun & Shin , 2020 ; Shin et al. , 2005 ) . However , the model performance reported in the previous studies is unsatisfactory , with the accuracy ranging from 0.621 ( Bae et al. , 2014 ) to 0.837 ( Kim & Ock , 2016 ) . One possible reason for this unsatisfactory performance is that they did not consider contextual information . Against this background , the current study employs BERT for the same kind of classification task for Korean postpositions . BERT produces contextual embeddings , and this characteristic may help us to create a better classification system for postpositions . Still unclear is the particular reason for BERT ’ s superior performance over the others . In order to further understand how BERT recognizes the word-level polysemy , we propose a BERT-based visualization system in addressing polysemy interpretation of three adverbial postpositions , -ey , -eyse , and - ( u ) lo , which are frequently used and documented in the previous studies ( e.g. , Cho & Kim , 1996 ; Jeong , 2010 ; Nam , 1993 ; Park , 1999 ; Song , 2014 ) . 2 KOREAN ADVERBIAL POSTPOSITIONS : -ey , -eyse , AND - ( u ) lo In order to determine the number of functions of each postposition , this study considers the major functions of these postpositions which are frequently attested in the Sejong dictionary : eight for -ey , two for -eyse , and six for - ( u ) lo ( Shin , 2008 ) . -ey involves the following functions : agent ( AGT ) , criterion ( CRT ) , effector ( EFF ) , final state ( FNS ) , goal ( GOL ) , instrument ( INS ) , location ( LOC ) , and theme ( THM ) . ( 1 ) -ey as AGT ( agent ) 가두 katwu street 진출이 cinchwul-i go.out-NOM 경찰에 kyengchal-ey police-AGT 저지되었다 . ceci-toy-ess-ta . stop-PSV-PST-DECL ‘ By going out to the street was stopped by the police. ’ ( 2 ) -ey as CRT ( criterion ) 영호는 Yenghuy-nun Yenghuy-TOP 20만원에 20manwen-ey 200,000 won-CRT 모니터를 monithe-lul moniter-ACC 낙찰했다 . nakchalhay-ss-ta . sell-PST-DECL ‘ Yenghuy sold the monitor ( to a bidder ) for 200,000 won. ’ ( 3 ) -ey as EFF ( effector ) 문들이 mwun-tul-i door-PL-NOM 거센 keseyn strong 바람에 palam-ey wind-EFF 모두 motwu all 건들댄다 . kentultay-n-ta . sway-PRS-DECL ‘ The doors all sway by the strong wind. ’ ( 4 ) -ey as FNS ( final state ) 김교수는 kimkyoswu-nun professor.Kim-TOP 조교에 cokyo-ey assistant-FNS 박군을 park-kwun-ul Park-Mr-ACC 추천했다 . chwuchenhay-ss-ta . recommend-PST-DECL ‘ Professor Kim recommended Park as an assistant. ’ ( 5 ) -ey as GOL ( goal ) 철수가 Chelswu-ka Chelswu-TOP 던진 tenc-i-n throw-CST-PRS 칼이 khal-i knife-NOM 땅바닥에 ttangpatak-ey ground-GOL 내리꽂혔다 . naylyekkoc-hi-ess-ta . stick-PSV-PST-DECL ‘ The knife thrown by Chelswu stuck to the ground. ’ ( 6 ) -ey as INS ( instrument ) 그 ku That 어린 eli-n young-REL 소년은 sonye-nun boy-TOP 화롯불에 hwalospwul-ey fire-INS 손을 son-ul hand-ACC 녹이고 nok-i-ko melt-CST-and 있었다 . iss-ess-ta . be-PST-DECL ‘ The young boy was using the fire to warm his hands. ’ ( 7 ) -ey as LOC ( location ) 그는 ku-nun He-TOP 온종일 oncongil all day 서재에 secay-ey study.room-LOC 파묻혀 phamwut-hi-e bury.in-PSV-PRS 지낸다 . cinay-n-ta . be-PRS-DECL ‘ He is buried in his study room all day. ’ ( 8 ) -ey as THM ( theme ) 현대인들은 hyentayin-tul-un modern.people-PL-TOP 모두 motwu all 참된 chamtoy-n true-REL 지식에 cisik-ey knowledge-THM 허기져있다 . hekicye-iss-ta . hungry-PRS-DECL ‘ All modern people are hungry for true knowledge. ’ -eyse has only two functions , location ( LOC ) ( 9 ) and source ( SRC ) ( 10 ) . -eyse manifests fewer functions than -ey ( Choo & Kwak , 2008 ) . However , frequency is equally high compared to that of -ey ( e.g. , Cho & Kim , 1996 ; Song , 2014 ) . ( 9 ) -eyse as LOC ( location ) 철수는 Chelswu-nun Chelswu-TOP 서울에서 sewul-eyse seoul-LOC 태어났다 . thayena-ss-ta . born-PST-DECL ‘ Chelswu was born in Seoul. ’ ( 10 ) -eyse as SRC ( source ) 광부들이 kwangpwutul-i miner-PL-NOM 바다에서 pata-eyse sea-SRC 석유를 sekyu-lul oil-ACC 뽑아올린다 . ppopaoll-i-n-ta . pull-CST-PRS-DECL ‘ Miners pull oil from the sea. ’ - ( u ) lo engages in six functions : criterion ( CRT ) , direction ( DIR ) , effector ( EFF ) , final state ( FNS ) , instrument ( INS ) , and location ( LOC ) ( Shin , 2008 ) . ( 11 ) - ( u ) lo as CRT ( criterion ) 적당한 cektangha-n appropriate-REL 시간 sikan time 간격으로 kankyek-ulo interval-CRT 배차되었다 . paycha-toy-ess-ta . arrange-PSV-PST-DECL ‘ It was arranged at appropriate time intervals. ’ ( 12 ) - ( u ) lo as DIR ( direction ) 범인은 pemin-un criminal-NOM 어두운 etwuwun dark 골목으로 kolmok-ulo alley-DIR 달아났다 . talana-ss-ta . flee-PST-DECL ‘ The criminal fled into a dark alley. ’ ( 13 ) - ( u ) lo as EFF ( effector ) 환자가 hwanca-ka patient-NOM 위암으로 wiam-ulo stomach.cancer-EFF 매우 maywu very 괴로워하고 koyloweha-ko suffer-and 있습니다 . iss-supni-ta . be-HON-DECL ‘ The patient is suffering greatly due to stomach cancer. ’ ( 14 ) - ( u ) lo as FNS ( final state ) 그는 ku-nun He-TOP 대표 tayphyo representative 강사로 kangsa-lo lecturer-FNS 초빙되었다 . choping-toy-ess-ta . invite-PSV-PST-DECL ‘ He was invited as a representative lecturer. ’ ( 15 ) - ( u ) lo as INS ( instrument ) 전선이 censen-i wire-NOM 연줄로 yencwul-lo connection.wire-INS 감겼다 . kam-ki-ess-ta . wind-PSV-PST-DECL ‘ The wire wound around with the connection wire. ’ ( 16 ) - ( u ) lo as LOC ( location ) 경찰이 kyengchal-i police-NOM 피해자를 phiuyca-lul suspect-ACC 검찰로 kemchal-lo prosecution-LOC 압송했다 . apsonghay-ss-ta . transport.do-PST-DECL ‘ The police transported the suspect to the prosecution . ’ 3 METHODS . 3.1 CREATING INPUT CORPUS . The Sejong primary corpus1 , the representative corpus in Korean , does not code the information about the functions of postpositions directly in each sentence ( which is necessary for model training ) . We thus annotated a portion of the original corpus data manually . For this purpose , we extracted sentences that have only one postposition and predicate . This also allowed us to control for additional confounding factors which might have interfered with the performance of our model . We then extracted 5,000 sentences randomly for each postposition from the initial dataset . 1Sejong corpus is available at : https : //www.korean.go.kr Three native speakers of Korean annotated each postposition for its function in this 15,000-sentence corpus . Fleiss ’ kappa scores showed that the annotators ’ outcomes were almost identical : 0.948 ( - ey ) , 0.928 ( -eyse ) , and 0.947 ( - ( u ) lo ) . We further excluded instances which showed disagreement among the annotators . The final corpus consisted of 4,715 sentences for -ey , 4,853 sentences for -eyse , and 4,708 sentences for - ( u ) lo . Table 1 presents the detailed by-function frequency list of the three postpositions 2 . 3.2 CREATING TRAINING AND TEST SETS . We pre-processed the data in consideration of how BERT works ( we used the original BERT model for this task ) . First , we added [ CLS ] ( ‘ classification ’ ; indicating the start of a sentence ) before a sentence and [ SEP ] ( ‘ separation ’ ; indicating the end of a sentence ) after a sentence to indicate where the sentence starts and ends . These indicators made it possible for the BERT model to recognize a sentence boundary in a text , allowing the model to learn word meaning while considering intersentential variations . Second , we made a separate column ( ‘ Label ’ ) to indicate the intended function of each postposition in each sentence ( Figure 1 ) . We then split the corpus into two sub-sets , one with 90 per cent of the corpus for the training and with the remaining 10 per cent of the corpus for the testing . 3.3 DEVELOPING BERT CLASSIFICATION MODEL . We set the parameters related to BERT training such as batch size ( 32 ) , epoch ( 50 ) , seed ( 42 ) , epsilon ( 0.00000008 ) , and learning rate ( 0.00002 ) , as advised by McCormick ( 2019 ) . We then employed a pre-trained language model in order to obtain high accuracy of outcomes ; for this purpose , we used a Korean BERT model ( KoBERT ; Jeon et al. , 2019 ) . Before the actual BERT training , we transformed the input data into three embedding types—token embeddings , position embeddings , and segment embeddings ( cf. , Devlin et al. , 2018 ) —in the following ways . First , for the token embedding , we used KoBertTokenizer for the sentence tokenization ( the maximum number of tokens for each sentence was set to 128 ) . Second , we converted each token into numeric values indicating unique indices of the tokens in the vocabulary of KoBERT for the position embedding . Third , for the segment embedding , we converted the number of tokens of each sentence into 128 numeric values using 0 ( i.e. , not existed ) or 1 ( i.e. , existed ) . The labels of the data indicating the intended function of each postposition in the sentence were stored separately . After this transformation step , we proceeded to the model training as follows . We first loaded KoBERT through the function BertForSequenceClassification from transformers ( Wolf et al. , 2019 ) . 2Our corpus is available at : we will propose a URL Next , we fine-tuned the pre-trained model by using the training set , with a view to reducing loss values and updating the learning rate for better classification accuracy of the model . We then loaded the testing set to evaluate whether the fine-tuned model successfully recognized the intended functions of each postposition in each sentence . In this part , the rates of accuracy for each function and the total accuracy rate were calculated by comparing the intended function of each postposition in each test sentence with the classified function of each postposition via the BERT model . Lastly , we employed t-distributed Stochastic Neighbor Embedding ( t-SNE ; Maaten & Hinton , 2008 ) for dimension reduction of classification embeddings from the postposition per epoch . In addition , to statistically confirm the changes of sentence-level embedding outcomes by each epoch , we performed densitybased clustering ( Sander et al. , 1998 ) . These outcomes were fed into the visualization system , which we outline next .

Summary: The paper presents a study of training a BERT-style model on the task of function identification for 3 different Korean adverbial postpositions(ey, eyse and (u)lo). The paper presents a new dataset for the task, show that training a BERT model performs well and also introduce a method to visualise BERT embeddings as the training proceeds. The authors show that the model performs better for postpositions which have fewer associated functions (classes for classification), performs better for classes that occur more frequently in data and that model performance improves with additional epochs.