Diversifying language models for lesser-studied languages and language-usage contexts: A case of second language Korean
Authors: Hakyung Sung
Gyu-Ho Shin
Venue: EMNLP 2023 Findings (2023)
Keywords: Multilinguality
DEI
NLP applications
L2 Korean
Morpheme parsing/tagging
TL;DR: This study explores the applicability of existing morpheme parsers/taggers to lesser-studied languages and contexts (L2 Korean) by training a neural-network model on diverse L2 Korean data and evaluating its parsing/tagging performance.

Abstract: This study investigates the extent to which currently available morpheme parsers/taggers apply to lesser-studied languages and language-usage contexts, with a focus on second language (L2) Korean. We pursue this inquiry by (1) training a neural-network model (pre-trained on first language [L1] Korean data) on varying L2 datasets and (2) measuring its morpheme parsing/POS tagging performance on L2 test sets from both the same and different sources of the L2 train sets. Results show that the L2 trained models generally excel in domain-specific tokenization and POS tagging compared to the L1 pre-trained baseline model. Interestingly, increasing the size of the L2 training data does not lead to improving model performance consistently.

Diversifying Language Models for Lesser-studied Languages and Language-usage Contexts: A Case of Second Language Korean
Hakyung Sung
Gyu-Ho Shin Department of Linguistics
Department of Linguistics University of Oregon hsungêuoregon.co edu
University of Illinois Chicago ghshinYuic.
Abstract
that the types and characteristics of language-usage data affect the way in which specific inquíries are investigated properly on the basis of those data (Biber, 1993). Given this aspect, may be the case that currently available parsing/tagging models, ex- clusively trained and evaluated on L1 data, do not work reliably and optimally for L2 data (cf. Kyle et al., 2022; Lan al., 2023; Sung and Shin, 2023). Another concern is the commitment to Diver- sity, Equity, and Inclusion (DEI) within research practice. Notably, there is a strong bias to a lim- ited range of languages (e.g., (L2-)Engglish) and language-usage contexts (e.g.. general-purpose L1 usage) (Bender al., 2021). This sampling bias poses threat to DEI in the field, as well as weak- ening the generalizability of implications from pre- vious findings to wider situations. This calls for researchers' attention to lesser-studied languages and language-usage contexts.
This study investigates the extent to which cur- rently available morpheme parsers/taggers ap- ply to lesser-studied languages and language- usage contexts, specifically focusing on second language (L2) Korean. We pursue this inquiry by (I) training a neural-network model (pre- trained on first language [LI] Korean data) on varying L2 datasets and (2) measuring its mor- pheme parsing/tagging performance on L2 test sets from both the same and different sources of L2 training sets. The results show that the L2 trained models generally excel in L2 domain- specific parsing and tagging tasks compared to the Ll pre-trained baseline model. Interest- ingly, increasing the size of L. training data does not lead to improving model performance consistently.
1 Introduction
Computational accounts for language science are gaining momentum in bringing together theories and data on second language (L2) research, offer- ing promising directions toward interdisciplinary collaboration (e.g., Chapelle, 2001; Kyle, 2021;
With this background, we investigate how well the current morpheme parsers/taggers cater to L2 Korean, a lesser-studied language and language- usage context. In particular, we explore a poten- tial boost in performance when an Ll pre-traines pra-trained model is additionally trained on L2 data with vary- ing sizes, annotation schemes, and data cleaning processes. To preview, the results demonstrate that (I) models fine-tuned with L2 data outperform the LI pre-trained baseline when evaluated on test sets from the same domain and (2) simply increasing the size of the L.2 training data does not invariably lead to improved the fine-tuned models' perfor- mance.
Meurers and Dickinson, 2017; Thewissen, 2013).
This often involves leveraging NLP techniques to address learner language characteristics in sizable L2 corpora. Specifically, automatic processing of L2 data, including part-of-speech (POS) tagging, has become crucial better understand the char- acteristies of learner language (e.g., Bestgen and Granger, 2014; Biber et al., 2016; Granger and Bestgen, 2017).
Despite the emerging trend, we identify two ma- jor concerns in the field. One lies in the research practice that applics the currently available NLP tools, predominantly trained on first language (L1) and
This paper is structured as follows: We discuss the importance of morphological analysis in Ko- rean studies and how morphological analyzers are applied in L2-Korean research ($2). Next, out- line experiments including datasets and evaluation metrics ($3). We delve into two subsequent experi- ments: the first involves fine-tuning an L1 model using the full scope of the L2 datasets ($4), while
data, directly L2 data. Con- sidering the fact that no single corpus perfectly captures all aspects of how language is used (Meur- ers and Dickinson, 2017). it is reasonable to think

---

the second focuses on rigorous data cleaning in one of the L2 datasets ($5). Comprchensive analyses of morpheme tokenization and POS tagging accuracy for both experiments are presented. Finally, we summarize our findings and propose future direc- tions ($6).
2.2 Application of morphological analyzers in L2-Korean research
Research L2 Korean is increasingly lever- aging automatic morphological analyzers (i.e., parsers/taggers) This includes, but is not limited to, lexico- sarico-graamatical feature analysis, automatic writing evaluation, and text analysis.
2 Background
2.1 Morphological analysis of Korean
Lexico-grammatical feature analysis: early studies, researchers investigated Korean particles as core lexico-grammatical feature using automatic morpheme parser/tagger. For example, Nam and Hong (2014) investigated -2-eoreann learners' spoken production by com- paring their use of particles across proficiency levels using a POS tagger (Utagger). Similarly, Dickinson et al. (2010) and Lee et al. (2016) incorporated an automatic POS tagger with a proposed automatic error-detection scheme to detect L2-Korean learners' non-prototypical use of particles. Building this, Kim al. (2016) explored the grammatical patterns in L2-Korean learners' written productions. They tagged morphemes in the target corpus and examined the resulting grammatical structures. Meanwhile, Jung (2022) and Shin and Jung (2022) analyzed the distribution of particle use in L.- KorKan textbooks, by creating pipeline that automatically extracts (using UDpipe tagger) the target particles.
Korean has unique features such as its agglutinative nature and Sobjjctt-Obect--e Verb word order (but relatively flexible due overt case-min mareeig via grammatical markers and context-dependent opera- tion). Additional complexity is added by the use of speech levels (e.g., honorifics reflecting formality) and particles/affies that sometimes lead to formal changes of root words to which they are attached (Sohn, 1999).
At the heart of these complexities involving Ko- rean is the identification of morpheme- the small- est meaningful unit in a language. Specifically, a word (dubbed an cojeol in Korean) often com- prises a combination of several morphemes, each of which carries distinctive meanings and/or functions. To illustrate, while h chayk "book" consists of one single morpheme, 사 z chaykcang "bookshelf" in- volves two different morphemes, each of which has distinctive meaning: al] chayk "book"+22 + cang "shelf, case". more complex example can be found in a predicate. Take H. Et poyessta "was seen" as an example. This verb comprises a verb root H- po- "to see", followed by (1) a passive- voice suffix -o]- -i, (2) a past-tense marker -1]- -ess-, and (3)= a declarative ending -t -ia. On top of this combination, involves re-syllabification of the passive-voice suffix and the past-tense marker, resulting in formal change the given morphemes (i.e. -i- + -ess- + -yess-).
Automatic writing evaluation: Lim al. (2022) developed an automated system for evaluating L2- L22-orean writing. This system utilized a transformer- based multilingual model alongside LoBRRaa Considering morphemes as one of the important measurable complexity features in the production data, an automatic POS tagger was employed. The developed tool was then applied evaluate the writing proficiency of the L2-Korean learners. More recently, Hwang (2023) proposed a linguistic analysis tool that measures morphological complexity of I.2-Korean production data based on an automatic morpheme tagger (Kkma). By automatically parsing mor- pheme and calculating I.2 assessment indices, this study suggested its potential to model L2 Korean development.
Given the intricacy its morpheme composition, automatic analysis of Korean language hinges cru- cially upon accurate and appropriate identification of morphemes. Indeed, previous studies have high- lighted the importance of morpheme-level pars- ing and tagging of Korean, implementing various approaches (c.g., morpho-syntactic rule-based ap- proach, probabilistic method, grapheme-level ap- proach, conditional random field method) to handle the composition of Korean morphemes (e.g., Choi et al., 2016; Han and Palmer, 2004; Lee and Rim, 2005, 2009; Na, 2015).
Text analysis: Cho and Park (2018) applied four distinct automatic morphological analyzers (Kkma, Okt, Hannanum, and Komoran) to 16

---

essays penned by L2-Korean learners. To evaluate the text similarity of these essays, they utilized the TF-IDF (Term Frequency-Inverse Document Frequency) method, incorporating extracted morphemes as a key feature.
Second, the most effective strategy for L2- dataset development, which would yield an optimal L2 model after training, has not been fully iden- tified. This process, particularly the creation of datasets with gold annotations, often requires con- siderable resourees and time (Snow et al., 2008). Specifically in L2 studies, to the best of our knowl- edge, there is research determining whether the primary focus in developing an I.2 dataset for fine- tuning an LI pre-trained model should be on in- creasing the size of the L.2 training data or on im- proving its quality. These two questions invite the need for further investigation, which motivates the current study.
2.3 Evaluation of morphological analyzers on L2-Korean data
A recent finding points to a considerable gap in performance of currently available Korean mor- phological analyzers when applied to I.2-Korean texts, necessitating further research to improve their reliability, Sung and Shin (2023) evaluated the performance of four publicly available Korean morphological analyzers (Stanza, Trankit, Kkma, and Komoran) on L2-Korean written texts. Re- sults showed a reduced performance on the 1.2 data compared to the L1 data across all investigated an- alyzers. Specifically, Stanza, which demonstrated the highest performance on the L1/L2 datasets (F1l: 0.92 [tokenization); 0.93 [POS tagging]), exhibited reduced performance on the L2 dataset (F1: 0.89 [tokenization]; 0.86 JPOS tagging]). Upon a de- tailed examination model performance based on by-tag accuracy, it was observed that the POS tags associated with predicatss-specfffcalty V (Verb), VA (Adjective), and VX (Auxiliary Verb)-showed lower accuracy compared to other tags. Overall, the finding highlights potential challenges involv- ing the use of deep-learning models, trained exclu- sively on L1 data, to process L2-Korean corpora.
Experimental Setup
3.1 Dataset
We used two L.2-Korean and one LI-Korean corpora which include morpheme annotations based on the Sejong tag set (Kim et al., 2007; Appendix A).
L2-NIKL: The National Institute of Korean
Language (NIKL) compiled comprehensive L2- L2-Korean learner corpus", which consists of written (27,299) and spoken (2,541) texts from years 2015 to 2020. They documented and provided details on learner proficiency in Korean and each participant's 's L1. The corpus is accessible in raw txt files and XML files (with individual morpheme and/or error annotations; see Appendix B as example), though some data is exclusively in raw format. Data access requires permission?.
2.4 L2 domain-specifie model development
Although previous studies have consistently shown that fine-tuning L1 pre-trained models with L2 data improves the accuracy of both tokenization and POS tagging for L2 data (Berzak et al., 2016; Kyle et al., 2022; Sung and Shin, 2023), there are two key questions unresolved with respect to develop- ing I.2 domain-specitn models. First, it is unclear how the models perform in zero-shot scenarios with unseen L2 data (i.e., L2 test sets not sourced from the same origin L2 training data'), which is crucial factor for enhancing the model's 's reliability and robustness (Choi and Palmer. 2012).
L2-KLM: The L2 Korean Learner Morpheme (KLM) corpus consists of 600 L.2-Korean written texts, each annotated at the morpheme level. These texts are distributed across six proficiency levels of Korean, with 100 texts per level (Sung and Shin, 2023). The morpheme annotations were manually constructed and underwent comprehensive cross- validation by trained human annotators. The data in this corpus is in CoNLL-U format, adhering to the Universal Dependencics (UD) formalism (Nivre et al., 2020).
IIn this context, "the same origin" refers to learner data from similar settings or conditions, including factors such as the data-collection prompt and/or the nature of the task assigned to L2 Korean learners for language production. This consideration was informed by previous learner corpus studies that show that the task's 's prompt and instructions can markedly influence learner output (Alexapoulou et al., 2017).
Phttps: //corpus. korean.go.kk go.
'For more information on this dataset's creation, refer to the official manual from the institute:
ht https: / http///koreenenggee wwwoorranggo..e. wwo.kr/ffott////ooo trrooo/pppottaaaa//
reportDataVice www.ttaawaawwd......... id==277ssearchhrdeeryyeess

---

The Google Korean Universal De- pendency Treebank (UD Korean GSD)4 was utilized as an LI reference corpus (Chun et al., 2018; McDonald et al., 2013). This dataset contains roughly 6,000 sentences which were collected from online blogs and news outlets written by Korean native speakers. For evaluation. we used 989 sentences from the UD Korean GSD test set.
LI-GSD:
Fl: measures instances of exact matches between predicted tokens/tags and the gold standard, but only when the parsed mor- pheme count matches the gold-annotated count within an eojeol. This metric aggre- gates true positives, false positives, and other counts across the dataset, leading to micro- Fl calculation.
By-tag accuracy: measures correct predic- tions for each POS tag using micro-FI scores.
3.2 Model training and evaluation
We employed Stanza (Qi et al., 2020) (version 1.4.2, Korean-GSt package) for model training and eval- uation, which is pre-trained on the UD Korean GSD treebank. It provides a unified pipeline for Korean- specific tasks such as tokenization (based on an eojeol), lemmatization (based on a morpheme), and POS tagging (including both UPOS [Universal POS] and XPOS |treebank-specifit POS]). For Ko- rean data, lemmatization and XPOS tagging corre- sponds to morpheme parsing and morpheme POS tagging, respectively; therefore, these two tasks were utilized. Stanza also offers separate pipeline for model training tailored for processing CoNLL- U formatted f
4
Experiment 1
The goal of Experiment was to leverage the siz- able L2-NIKL dataset to the fullest extent, despite some data loss during the data pre-processing stage.
4.1 Data pre-processing
There were several issues in handling the L2-NIKL corpus in its original form. These included format inconsistencies across some files (c.g., files that in- cluded only the title of the text without any content), the sentences written by non-Korean characters. and misalignment of sentence-ending punctuation marks and their corresponding POS tags
We selected a batch size of 5000 for training the POS tagger and 500 for the lemmatizer. Prelimi- nary experiments revealed that a larger batch size of 5000 most effective for training the POS tagger, whereas the performance for morpheme tokenization remained consistent across different batch sizes (cf. Zhou al., 2015). In order to op- timize the training process, we further used Adam as an optimization algorithm.
For the first issue, we extracted files that con- tained only a single sentence, typically title, and excluded those. For the second issue, we identi- fied sentences with non Korean characters, such Chinese or English alphabets, using the Unicode specification script and excluded them. For the last issue, we treated sentence- punctuation marks as distinct tokens, placing them the end of each sentence with their respective POS tags. Further- more, we replaced all instances of the SYMBOL tag with the SF tag.
To construct and evaluate the model, we ran- domly split each L.2 dataset into three subsets (80% for training: 10% for validation; 10% for test).
After resolving these issues, we extracted indi- vidual tokens and their corresponding POS tags, and restructured the entire file format comply with the CoNLL-U format, while preserving the original metadata". For convenience, we dubbed the L.2NIKLL dataset used in Experiment as L2- NIKL- ((riin(r) to differentiate it from the dataset used in Experiment 2.
3.3 Evaluation metrics
For robust evaluation, we measured the model's performance using the following metrics:
Perfeet score (PS): measures the accuracy in morpheme parsing by comparing the number of morphemes within an eojeol. It evaluates the model's 's ability to predict the exact num- ber of morphemes in alignment with the gold standard.
'ased on the UD convention, sentence final punctuation marks should ideally been tokenized separately. However, they were instead attached to sentence- final functional morphemes (e.g.. L L++"." "). with POS tags that are not found in the Sejong tag set (c.g.. a period (".") was often incorrectly tagged as SYMBOL instead the correct SF tag.)
http:s///unverrslleeeeerreeeeree ko_ bo_gsd/index. html
" "Refer to
https: //gi wtt..t..bcooo/...... main/NIKL coor-s---------- process for the related codes.
"Korean/KLM
www.//a--oopusstteeeeeeeeeeeaeee httt-corpussttrteee /tree/
Phttps: / http:///ggttt..t.tttaatttrrinone httpu///.s.ssstooooo///aaaaaaaaa

---

Meanwhile, we recognized the need to address the three tags (NF, NV, NA) present in the L2-KLM corpus. Upon reviewing the tags used in the L2- NIKL and LI-GSD datasets, we noticed that these two datasets rarely used the NF (Undecided noun) or NV (Undecided verb) tags. Instead, they they predominantly employed the NA (Undecided) tag to annotate unknown words, most of which originated from spelling errors. We thus merged the NF and NV tags into the NA tag for the L2-KLMM dataset for the following training and cross-dataset evaluation.
Tag
L2-NIKL-O
L2-KLM
LI-GSD
NNG
68,223 (22.40)
25,279 (19.48)
28,360 (29.53)
VV
26,882 (8.83)
10,196 (7.86)
6,561 (6.83)
EC
22,370 (7.35)
8,558 (6.59)
7,885 (8.21)
EF
17,604 (5.78)
7,537 (5.81)
3,030 (3.16)
JKB
15,185 (4.99)
6,362 (4.90)
4,956 (5.16)
ETM
14,862 (4.88)
6,676 (5.14)
4,923 (5.13)
MAG
13,807 (4.53)
4,303 (3.32)
2,427 (2.53)
JX
13,338 (4.38)
5,399 (4.16)
3,946 (4.11)
JKO
10,316 (3.39)
4,716 (3.63)
3,185 (3.32)
JKS
10,270 (3.37)
4.128 (3.18)
2,527 (2.63)
NNB
9,723 (3.19)
4,739 (3.65)
4,004 (4.17)
VA
9,476 (3.11)
3,375 (2.60)
1,790 (1.86)
4.2 Data exploration
XSV
7,965 (2.62)
3,267 (2.52)
3,489 (3.63)
Table 1 shows the basic statistics of sentences and morphemes from each dataset used in Experiment
NP
7,436 (2.44)
2,135 (1.65)
589 (0.61)
VX
5,322 (1.75)
3,222 (2.48)
1,918 (2.00)
s Table 2 presents the top 15 POS tags frequently occurring in each dataset".
Total
252,779 (83.01)
99,892 (76.98)
79,590 (82.88)
Table 2: Frequencies (proportions %) of the top 15 POS tags the datasets used Experiment
L2-
L2- KLM
L1-
NIKL-
GSD
Training
Metric
L2-NIKL-O
L2-KLM
O
TOK
POS
TOK
POS
# sentStotal
28,849
7,527
4,400
L2-NIKL-O
94.32
95.10
89.64
92.01
# sentswritten
15,773
7,527
4,400
PS Abest
11.07
J5.46
13.18
# sentsspoken 13,076
0
0
95.65
93.07
88.61
86.52
Fl Abest
J6.70
15.51
# morphemes 304,501
129,784
96,028
L2-KLM
92.26
93.40
95.10
95.19
Table 1: Descriptive statistics of the datasets used in Experiment
PS Abest
42.06
J2.77
93.60
88.88
95.31
92.03
Fl Abest
42.05
14.19
LI-GSD
93.61
96.17
92.28
92.47
PS Abest
J0.71
J2.82
J2.72
4.3 Evaluation and results
95.19
88.02
92.26
87.44
Table 3 presents overall PS and Fl of the morpho- logical analyzers for the L.2-NIKLOO and L2-KLLM test sets. Figure displays the by-tag accuracy for the top 15 POS tags analyzed in the two L2 test sets.
F1 Abest
40.46
45.05
J3.05
14.59
Table 3: Comparison of overall PS and (in %) out of 100 for morphological analyzers on the 1.2-NIKI-O and L-LKLM test sets. TOK refers to morpheme tok- enization (lemmatization in Stanza), while POS refers morpheme POS tagging. Bold numbers indicate the best score. The symbol Abest illustrates the difference in performance between the best score and the respective score. A downwards arrow shows decrease in perfor- mance compared to the best score.
We note three major findings. First, when the L2-trained models were tested on the data from
*While we did not train a model using LI-GSD, we in- cluded information about the 1.1-GDD training set for refer- ence. This was because the pipeline that we utilized in this study was pre-trained with this dataset.
"Tag starting with S, such as SF (sentence- final punctu- ation marks, e.g.. periods). SN (numbers), and SL (foreign characters). were excluded they were not deemed important for this study.
the same domain as their training, they overall out- performed the L1-GSD baseline model in both L2- NIKL-O and L2- KLM test sets. This is in line with previous findings suggesting that domain-speciff data enhance model performance because of the models' familiarity with patterns found in the target data (e.g.. Toutanova et al., 2003; Giménez and Marquez, 2004; Shen et al., 2007). Specif- ically, the L2 models seem adept at identifying basic words uncommon in the Ll dataset. This is
10 "'deally. each morpheme should be assigned to its corre- sponding POS tag, so the PS for TOK and POS should he identical in every instance. In reality, however, the language model in Stanza (and probably other models in other pipelines) has such inconsistencies inherently. To illustrate, the word y -l-# y pwal-man-cokk-he morphemes # El- + but tagged with only three POS tags 'NNG+XSA+ETM investigation point in the present study, future reseurch should seck to find why the original language models manifest the issue and how it can be handled properly.
"unsatisfied" is parsed into four
While this issue is not the focal

---

task seemed to be heavily influenced by the pres- ence/absence of sentence-final punctuation marks in the L2-NIKL-O dataset. Specifically, the LI- GSD model performed poorly with the sentence- final morpheme (EF) when handling the L.2NIKII- O test set (as in Figure la). This observation prompted more detailed examination of the L2- NIKL-O dataset and its POS-tagging outcomes. We observed that the EF tag was frequently mis- labeled as EC (Ending, connecting), particularly when the sentence-final punctuation mark was ab- sent in the L2-NIKL-O test set. In the L2-NIKL-O training set, a similar issue arose with certain sen- tences, notably those from spoken sources, lacking sentence-fing final punctuation marks.
100
50-
60-
50-
20
L1-650
ANG bc
jsu ZIN MAG
JNO
Jto his VA. XtV VX.
Tags
(a) Test set: L2-NIKI-OO
50
The last two findings suggest that thorough data cleaning, as seen in the creation of the hand-crafted L2-KLLM corpus, crucial when fine-tuning an L2 model. In other words, the most promising L2 model would be something that does not for- get what it has learned from the L1 training data (Kirkpatrick et al., 2017) and is exposed to robust representations of L2 data. This led to the next stage of this study, which involved further clean- ing of the L2-NIKL-O dataset and additional train- ing/evaluation of the L2 model assess its robust- ness.
60-
LI-05D
20
12HLM
12-M8i--
KNG
BC
jen ETM MAG
|8O
Ve Mk HMn
Tags
(b) Test set: L2-KLM
Figure 1: By-tag performance of top 15 POS tags in
L2-NIKL-O and L2-KLM test sets
Experiment 2
supported by the frequent tagging of the noun (NNG) (see Table 2) and the modest increase in F1 for this tag with L2 models (see Figure 1). Furthermore, I.2 models may better recognize morpheme com- binations prevalent in L2 datasets. This possibly underscores both the Korean language's aggluti- native nature in which a word can have multiple morphemes and the fact that (L2-Korean) learn- ers often employ simpler graimmaiaaamtic struc- tures (Lim et al., 2022; Hwang, 2023) in writing. Second, asymmetry was observed in the model's zero-shot performance on unseen L2 data, Both the L.2-NIL--O and L.2-KLM models dis- played a decline in PS and Fl on the unseen L2 test sets compared to the attested L2 test sets (from the same domain as their training). However, the gap in score reduction" 11 was larger for the L2-NIKL-O model than the L2-KLM model.
The goal of Experiment 2 was address potential issues inherent in the L2-NIKL-O dataset, which may have aggravated the performance of the model. Therefore, we undertook further data-cleaning as outlined below.
5.1 Data cleaning
First, we excluded sentences without sentence-fing final punctuation marks. We then eliminated one-token responses from the spoken data due to their repeti- tive nature and potential to introduce noise during model tuning. Examples include the words like vil ney "yes" or oo ah "ah (interjection)). Lastly, we omitted sentences with morphemes tokenized from eojeols but missing their corresponding POS tags. Given the likely unreliability and incorrect tagging of these sentences, their inclusion could diminish fine-tuning quality.
Third, model performance in the POS-tagging
For convenience, we dubbed the L2-NIKL dataset in Experiment as I.2-NIKL-CCleaned) differentiate it from the L2-NIKL-O dataset used in Experiment 1.
"I In most instances, A best represents the difference be- tween a score obtuined from a domain-specific test set (which is from the same domain as their training) and a score from zero-shot performance.

---

5.2 Data exploration
Tag
L2-NIKL-C
L2-NIKL-O
NNG
45,444 (22.51)
68,223 (22.40)
Figure visualizes the length of sentences in each dataset, determined by the count of cojeols per sen- tence. The L2-NIKL-C dataset included a smaller number of short sentences than the L2-NIKL-O dataset result of the additional data-cleaning process.
VV
18,562 (9.19)
26,882 (8.83)
EC
14,576 (7.22)
22,370 (7.35)
EF
14,231 (7.05)
17,604 (5.78)
JKB
10,621 (5.26)
15, 15,185 (4.99)
ETM
10,131 (5.02)
14,862 (4.88)
MAG
8,693 (4.31)
13,807 (4.53)
JX
8,909 (4.41)
13,338 (4.38)
JKO
7,796 (3.86)
10,316 (3.39)
JKS
7,358 (3.64)
10,270 (3.37)
Numbee df
NNB
6,667 (3.30)
9,723 (3.19)
VA
6,643 (3.29)
9,476 (3.11)
k00
XSV
5,843 (2.89)
7,965 (2.62)
NP
3,858 (1.91)
7,436 (2.44)
VX
4,355 (2.16)
5,322 (1.75)
Total
173,687 (86.08)
252,779 (83.01)
Table 5: Frequencies (proportions %) of the top POS tags in the datasets used in Experiment 2
Figure 2: Visualization of sentence lengths in each dataset
In parallel to Table 1 and 2, Table 4 shows the basic statistics of sentences and morphemes from the Experiment 2 datasets. Table illustrates the 15 most frequent POS tags in each dataset. Overall, the size of the L2-NIKL-C dataset was approxi- mately half of the L2-NIKL dataset while keep- ing the POS-tag distributions compatible with each other.
tests. Yet, the gap in score reduction was largely similar between the two models. Notably, it was somewhat surprising that the model fine-tuned on 1.2-KLM surpassed the L1-GSD baseline model in POS tagging for the unseen L2-NIKL-C test set. This indicates that models tuned on L.2 data might be more effective in handling L2 test data, even in a zero-shot scenario, compared to models exclusively trained on L1 data.
L2-NIKL-
L2-NIKL-
C
O
Training
Motric
L2-NIKL
L22KLLM
# sentstotal
14,682
28,849
TOK
POS
TOK
POS
sentSwritten
12,973
15,773
I.-NIKKI-C
PS
97.67
98.00
91.64
92.07
Abest
43.46
43.12
# sentSspoken 1,709
13,076
F1
98.63
96.91
91.13
86.45
# morphemes 201,920
304,501
Fl Abest
14.18
15.58
1.2KLLM
PS
94.01
94.45
95.10
95.19
Table 4: Descriptive statistics of the datasets used in Experiment
PS Abest
43.66
43.55
Fl
95.26
92.14
95.31
92.03
Fl Abest
43.37
J4.77
L1-GSD
PS
94.45
94.34
92,28
92.47
5.3 Evaluation and results
Abest
43.22
43.66
42.82
J2.72
Fl 95.15
88.67
92.26
87.44
Table 6 presents the comprehensive PS and Fl of the morphological analyzers evaluated on the L2- NIKL-C and L2-KLM test sets. Figure 3 shows by-tag accuracy for the top 15 POS tags as analyzed in the L2-NIKL-C test set. Consistent with the findings from Experiment 1, models tuned on L2 data outperformed the oth- ers when tested within the same training domain. Additionally, both the L22NIKL-CC and L2-KLM models saw a decrease in PS and Fl zero-shot
Fl Abest
43.48
J..24
13.05
14.59
Table 6: Comparison of overall PS and Fl (in %) out of 100 for morphological analyzers on the L2-NIKL-CC and L2-KLM test sets
A detailed examination of Figure 3 reveals that the by-tag accuracy exhibited larger stability in the 12-KLM model than the LI-GSD baseline model. This is particularly noticeable for ETM (Ending, de-

---

terminative), JX (Postposition, auxiliary), and NP (Pronoun) tags. Conversely, the L2-KLM model demonstrates somewhat reduced performance on VA (Adjective) and VX (Verb, auxiliary) tags. This is attributable to the robust and fine-grained anno- tation scheme employed in the KLM annotation process (cf. Sung and Shin, 2023, p.3), which treats predicate-related morphemes important piece of morpho-syntactic knowledge in acquiring Korean.
Training
Metric
L1-GSD
L1-GSD POS
LI-GSD
PS
94,40
95.04
PS Ahest
F1
94.51
93.65
FI Abest
L2-KLM
PS
86.97
88.92
PS Abest
17.43
J6.12
F1
88.58
83.40
F1 Abest
J.5.93
110.25
L2-NIKL-C
PS
88.05
89.51
PS Abest
|6.35
J5.53
F1
87.71
81.15
Abest
16.80
112.50
L2-NIKL-O
PS
83,24
88,96
PS Abest
111.16
J6.08
90
F1
81.41
80.96
80-
F1 Abest
113.10
413.00
(c
Table 7: Comparison of overall PS and Fl (in %) out of 100 for morphological analyzers the LI-GSD (reference) test set
60-
50-
30
exhibit asymmetry and variability in zero-shot eval- uations. Lastly, augmenting the size of the training dataset does not always lead to enhancing the per- formance of fine-tuned model. The findings of this study thus confirm domain -opaai-specifficty light of data processing and highlight the importance of input quality, including rigorous annotations (con- ducted by human annotators as in the case the L2-K 1.2-KLM KLM) or cleaning (as exemplified in the case of the L2-NIKL-C) when fine-tuning L.2 models on the basis of existing L. L.-oriieted, pre-trained models.
LI-G5D
20
10
ETM MAG
XSM
Tags
Figure 3: By-tag performance of top 15 POS tags in the I.2-NIKI-C test set
Taken together, the findings of Experiments 1 and 2 suggest that merely increasing the size of training data does not necessarily enhance perfor- mance for fine-tuned L2 pipelines. Instead, the quality of the data, which encompasses its annota- tion and cleaning, plays an important role. Specif- ically, even though the L2-NIKL-O training set (304,501 tokens) was larger than that of the L2- KLM model (129,784 tokens), the L2-NIKL-O model did not outperform the I.2- KLM model in zero-shot tests, including those the LI test set (Table 7). In similar vein, the L.2NIKL--C model, with its training set of approximately half the size (201,92) 200,92 tokens) of the I.2-NIKL-O model, did not lag behind performance. The performance rather improved after we cleaned the NIKL dataset for training, especially in terms of tokenization.
Effectively augmenting computational resources for lesser-studied languages and language-usage contexts requires meticulously engineered and vali- dated data processo pipelines. This is done not only by expanding target L.2 corpora with gold annotations but also by refining existing tagging schemes. For example, in our study, the I.2-NIKL. corpus, while robust in quantity, lacks clear guide- lines on whether annotations were manually cre- ated from scratch or generated with the assistance of automatic taggers Building upon the find-
"Accordiig to the official NIKL guideline', Korean- specific morpheme analyers (i.c.. Kkma, Komoran. ETRI, Khaii,, K/WI, Mecab-Korean, Utagger) were evaluated terms of their performance in order to facilitate the creation of sizable L2-Korean morpheme annotations (p. 188). However, remains unclear NIKL utilized semi-automatic annota- tions, which involve automatic analysis followed by human verification, or they adopted different approuches to this task. This ambiguity becomes problematic, especially when address- ing tagging errors made by L2 learners. Ll-pre-trained models often face difficulties in identifying these errors, mostly by incorrecly assigning them one of the most frequent POS tags
6 Conclusion and Future Works
Through the two experiments above, we have re- vealed three key aspects with regard constructing 1.2 models. First, models fine-tuned on 1.2 data can outperform an LI baseline when evaluated within the same domain. Second, model performance can

---

ings of this study, researchers are encouraged to consider improving the quality of gold annotations for more reliable and robust results in automatic handling of L2 data.
or presentation of the research, including plagia- rism, deliberate violation of anonymity, citation, review, or duplicate submission policies, falsify- ing results, or misrepresentation. Our research is expected to broaden the horizon of language sci- ence at the interface of technology, currently being limited to handful of the world's over 7000 lan- guages (Joshi et al., 2020), which contributes to promoting more inclusive research practice toward diverse languages and language-usen usage contexts.
Beyond the primary focus of this study on mor- pheme tokenization and POS tagging, extending the scope of investigation toward such tasks de- pendency parsing would offer a holistic evalua- tion of automatic analyses on learner corpora. We believe this extension will contribute to not only promoting a sense of DEI in research but also fa- cilitating AI biteracy-by improving researchers' understanding of how computational algorithms operate and how these can be upplied to research purposes appropriately. Moreover, pursuing this research direction would advance NLP applications in educational contexts, offering valuable resources on language learning and teaching.
References
Theodora Alexopoulou, Marije Michel, Akira Mu- rakami, and Detmar Meurers. 2017. Task effects on linguistic complexity and accuracy: A large- scale learner corpus analysis employing natural lan- guage processing techniques. Language Learning, 67(S1) (P(S1))188 10:1
Emily M Bender, Timmit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM confer- ence on fairness, accountability, and transparency, pages 610-623.
Limitations
Although we rre-processed the L2-NIKL data fol- lowing the outlined steps, we acknowledge that there may be different, and potentially more opti- mal, methods for data pre-processing. Additional experiments would yield a more comprehensive report on model performance, along with strengths and weaknesses of the model. The L2-NIKL data is substantially restricted in its use because of the major drawbacks inherent in the data themselves (e.g., unclear data-collection process, little control for topics and prompts, imbalanced distribution of learners' LI backgrounds). Furthermore, since the L2-NIKL data cannot be shared without per- mission, this limits the replicability of this study, deviating from Open Science practices.
Yevgeni Berzak, Jessica Kenney, Carolyn Spadine, Jing Xian Wang, Lucia Lam, Keiko Sophie Mori, Sehastian Garza, and Boris Katz. 2016. Universal dependencies for learner english. arXiv preprint 16Xiv:165
Yves Bestgen and Sylviane Granger. 2014. Quantifying the development of phraseological competence 12 english writing: An automated approach. Journal of Second Language Writing, 26:28-41.
Douglas Biber. 1993. Representativeness in corpus de- sign. Literary and linguistic computing, 8(4):243- 257.
Douglas Biber, Bethany Gray, and Shelley Staples. 2016. Predicting patterns of grammatical complexity across language exam task types and proficiency levels. Ap- plied Linguistics, 37(5)663966)) 6688.
Ethics Statement
We used two publicly available datasets (L1-GSD and I.2-KLM) and one dataset with permission required (L2-NIKL). For the two open-access datasets, we ensured that all potentially identifying information such as names and other personal de- tails was removed. For the permission-only dataset, we strictly followed all stipulated guidelines to re- spect the interests of the data providers. Other than that, we believe that there is no substantial ethical issue with the research presented in this study. There is no dishonesty in the execution
Computer applications in Carol A Chapelle. 2001.
second language acquisition. Cambridge University Press.
S. Cho and Y. Park. 2018. Characteristics of korean language writing by students the university of sheffield (kofean:hheffield tayhakkyo hankwuke hak- supcauy cakmwun thukseng pwunsek). Cakmwun- yenkwu [Korean writing association]. 38: 179-172.
Jihun Choi, Jonghem Youn, and Sang goo- Lee. 2016. A grapheme-level approach for constructing a korean morphological analyzer without linguistic knowledge. In 2016 IEEE International Conference on Big Data (Big Data), pages 3872-3879. IEEE
(c.g., NNG). This poses another hurden for human annotators to identify and tag these errors during the verification process.

---

Jinho D Choi and Martha Palmer. 2012. Fast and robust part-of-speech tagging using dynamic model selec- tion, In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Pupers), pages 363-367.
Kieran Milan, John Quan, Tiaoo Ramalho, Ag- nieszka Grabska-Barwinee et al. 2017. Over- coming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):35113326.
Jayeol Chun, Na-Rae Han, Jena D Hwang, and Jinho D Choi. 2018. Building universal dependency tree- banks in korean. In Proceedings of the Eleventh International Conference Language Resources and Evaluation (LREC 2018).
Kristopher Kyle. 2021. Natural language processing for learner corpus research. International Journal of Learner Corpus Research, 7(1):1-16.
Kristopher Kyle, Masaki Eguchi, Aaron Miller, and Theodore Sither. 2022. A dependency treebank of spoken second language English. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022). pages 39-45, Seattle, Washington. Association for Computational Linguistics
Markus Dickinson, Ross Israel, and Sun-Hee Lee. 2010. Building korean web corpus for analyzing learner language. In Proceedings of the NAACL HLT 2010 Sixth Web As Corpus Workshop, pages 8-16.
Jesús Giménez and Lluis Marquez. 2004. Svmtool: A general pos tagger generator based on support vector machines. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (44)
Ge Lan, Xiaofei Pan, Yachao Sun, and Yuan Lu. 2023, Part of speech tagging of grammatical features related to 12 chinese development: A case analysis of stanza in the 12 writing context. Frontiers psychology. 14:508.
Sylviane Granger and Yves Bestgen. 2017. Using coll- grams to assess 12 phraseological development: A replication study. Language, Learners and Levels: Progression and Variation, pages 385-408.
Do-Gil Lee and Hae Chang Rim. 2005. Probabilis- tic models for korean morphological analysis. In Companion Volume to the Proceedings of Conference including Posters/Demos and tutorial abstracts.
Chung-Hye Han and Martha Palmer. 2004. A morpho- logical tagger for korean: Statistical tagging com- bined with corpus- morphological rule applica- tion. Machine Translation, 18:275-297.
Do-Gil Lee and Hae-Chang Rim. 2009. Probabilis- tic modeling of korean morphology. IEEE trans- actions on audio, speech, and language processing. 17(5)99459955.
Haerim Hwang. 2023. Development of morphological diversity second language korean: An nip analysis using the korean morphological richness analyzer. Available SSRN 4334736.
Sun-Hee Lee, M Dickenson, and Ross Israel. 2016. Challenges of learner corpus annotation: Focusing on korean learner language analysis (kolla) system. Language facts and perspectives, 38:221-251.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the nlp world. arXiv preprint arXiv:2004.09095.
Kyung Tae Lim, Jayoung Song, and Jungyeul Park. 2022. Neural automated writing evaluation for korean 12 writing. Natural Language Engineering, page 1-23,
Ryan McDonald, Joakim Nivre, Yvonne Quirmbach- Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Os- car Täckström, et al. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 92-97.
Boo Kyung Jung. 2022. The nature of 12 input: Anal- ysis of textbooks for learners of korean second language. Korean Linguistics, 18(2):182-208,
H. Kim, B. Kang, and J. Hong, 2007, 21 ^l 7] All t A क प of 7 & - A): 28 ] t 전 a (21st century sejong modern korean corpora: Results and expectations). श H. // o| क a) $ # H. to E म (Proceedings of the Human and Language Technology). pages 311 311-316.
Detmar Meurers and Markus Dickinson. 2017. Ev- idence and interpretation in language learning re- search: Opportunities for collaboration with compu- tational linguistics. Language Learning, (7(6)): 95.
JY. Kim, YH. Park, MJ. Kim, HN. Kim, SK. Choi, JH. Suh, and YJ Kwak. 2016. A study of devel- oping usage searcher of grammar pattern in the ko- rean learner's 's writing corpus (korean: hankwuke hak- supcauy cakmwun malmwungchilul hwalyonghan mwunhyeng yonglyey kemsaykki kaypal yenkwu). Teaching Korean as a Foreign Language, 44:131- 156.
Seung- Hoon Na. 2015. Conditional random fields for korean morpheme segmentation and pos tagging. ACM Transactions on Asian and Low-Resource Lan- guage Information Processing (TALLIP), 14(3):1-16.
YJ. Nam and UP. Hong. 2014. Towards a corpus- based approach to korean as a second language (ko- rean: 1.2lsseuy hankwuke cayenpalhwa khophesuuy
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu,

---

kwuchwukkwa hwalyong). The Journal the Hu- manities for Unification, 57:193-22 220.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajié, Christopher D Manning, Sampo Pyysalo, Scbastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal dependencies v2: An evergrowing multilingual treebank collection. arXiv preprint arX4::2004. 10643.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. arXiv preprint arXiv:2003.0708.
Libin Shen, Giorgio Satta, and Aravind Joshi. 2007. Guided learning for bidirectional sequence classifica- tion. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 760-767.
Gyu-Ho Shin and Boo Kyung Jung. 2022. Input- output relation in second language acquisition: Textbook and learner writing for adult english-speaking be- ginners of korean. Australian Review of Applied Linguistics, 45(3):347-370.
Rion Snow, Brendan O' O'connor, Dan Jurafsky, and An- drew Y Ng. 2008. Cheap and fast-but is it good? evaluating non-expert annotations for natural lan- guage tasks. In Proceedings of the 2008 conference on empirical methods in natural language processing, pages 254-263
Ho-Min Sohn. 1999. The Korean language. New York, NY: Cambridge University Cambridge University Press.
Hakyung Sung and Gyu-Ho Shin. 2023. Towards 12- friendly pipelines for learner corpora: A case of writ- ten production by 12- korean learners. In Proceedings of the 18th Workshop Innovative Use NLP for Building Educational Applications (BEA 2023).
Jennifer Thewissen. 2013. Capturing 12 accuracy de- velopmental patterns: Insights from an error-tagged efl learner corpus. The Modern Language Journal,
97(S1):77-101.
Kristina Toutanova, Dan Klein, Christopher D Man- ning, and Yoram Singer. 2003. Feature-rich part-of- speech tagging with cyclic dependency network. In Proceedings of the 2003 Human Language Technol- Conference of the North American Chapter of the Association for Computational Linguistics, pages 252-259.
Hao Zhou, Yue Zhang, Shujian Huang, and Jiajun Chen. 2015. A neural probabilistic structured prediction model for transition-based dependency parsing. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joini Conference Natural Language Processing (Volume 1: Long Pupers), pages 1213- 1222.

---

A Sejong tag set
B L2-NIKL morpheme annotation
Table 8 provides a description of the Sejong tag set used in this study (Kim et al., 2007).
Figure provides an example of the L2-NIKL. mor- pheme annotation in an XML file.
doreen serrem Sarpat-
Tag
Description
sir soreititi esriinge rncemntt
NNG
Noun, common (H 5
x4)
NNP
Noun, proper ( T8 4b)
Apr2 (bbloooeeaaaa
NNB
Noun, common bound (2) & od 4
NR
Numeral (++
NP
Pronoun (rf 0 A/)
VV
Verb, main ( */)
430%
targe fran/
VA
Adjective (0] 0 A})
VX
Verb, auxiliary (H.33. 51+
otapi walyyadrge "eaigesty 20s- idsegaace 2hherin
VCP
Copular, positive ( 7 2| (1)
VCN
Copular, negative ( 3 7| (2)
cores welynehheer pep-tge memmmnneithheeeeen
Vearce
MM
Determiner ( + b X
morp frawdg terge wnlpmeeeeeee/nntiittt per 'M6" iiteeaecerssooopppp
MAG
Common adverb ( [ t 11)
MAJ
Conjunctive adverb ( 4 [111)
<immar tor (1"
IC
Exclamation (2) (2/ 2+)
oery ieee Inreveratiato
JKS
Postposition, nominative ( 4++
eorph 1ee indlycffoprrktttrii pes sabretamee Tollwark
JKC
Postposition, complement ( 2 X++
49tth
JKG
Postposition, prenominal () 2 Ex+
JK
Postposition, objectival (_ (+)
woreh cntvedlaanee sal wess tabuueee
anaisralisum pnucue uduepesin Coyytarr
JKB
Postposition, adverbial (# A+)
JK
Postposition, vocative ( 2| {+)
Figure 4: An example of the L2-NIKL morpheme anno-
JQQ
Postposition, quotative (?] 2| (1+)
tation
JC
Postposition, conjunctive (1 X*+
UX
Postposition, auxiliary (H (
EP
Ending, prefinal (1) of = of Pl)
EF
Ending, closing ( _ o ml)
EC
Ending, connecting [1] _ o u])
ETN
Ending, nounal _d c - A u))
ETM
Ending, determinitiie - o L
Zy ul))
XPN
Prefix, nounal (2) d 23 1
XSN
Suffix, noun derivative ( (1d) X} TH
XSV
Suffix, verb derivative (5 1 A -)
XSA
Suffix, adjective derivative
( 0 ^}
XR
Root ))
NA
Undecided [ 4
SF
Period, Question, Exclamation (1) R
SE
Ellipsis ( FF
SS
Quotation, Bracket, Dash (m) -t. )
SP
Comma, Colon, Slash 2 H. Z5, =
SO
Hyphen, Swung Dash ( o] H, H))
SW
Symbol [/
SH
Chinese characters (7))
SL
Foreign characters (2) 70))
SN
Number (5 x7)
Table Sejong tag set

---

C Hyperparameter values
Table and Table 10 provide information about the hyperparameter settings for lemmatization (mor-
pheme tokenization) and POS tagging trainings.
Hyperparameter
Selected
Num hidden units
200
Embedded vector space
50
Number of layers
1
Dropout rate of layers Beam size
0.5
1
Attension type
soft
Optimization algorithm
Adam
Learning rate
0.001
Numepoch
60
Batch size
50
Max grad norm
5.0
Table 9: Hyperparameters used for lemmatization
Hyperparameter
Selected
Batch size
5000
Max training steps
50000 (32, 64, 128, 50000)
Learning rate
3e-3
Optimization algorithm
Adam
Max grad norm
5.0
Table 10: Hyperparameters used for POS tagging


Reviews:
[{"review_id": "Zk5VSaBXD0", "reviewer": "Reviewer_prvE", "summary": "", "questions": "A - Line 215, it would be useful to expand a bit on what is meant by \"same origin\" (same speakers?) and why it creates this uncertainty (I assume this is because some L2 errors vary a lot from one speaker to another, so the dataset leaks with respect to the typology of errors? Is that all?).\n\nB - Line 301: Unclear how \"Perfect score\" is defined. \"comparing the number\" is dubious (does not seem to match what is done in the provided reference): only the number, or their boundaries as well? And how is this defined when using Perfect score on the POS task? This is important information that should be clear in the text itself (not relying solely on the reference).\n\nC - Line 305: Unclear how the F1 score is defined. Instances=the morphemes? What is the difference between \"parsing is perfect\" and \"tokens align exactly with the golds\" (the metric seems tautological)? And the phrasing looks more as an accuracy than an F1 (what do precision and recall mean, with that definition?). For the POS task, is it micro-F1 or macro-F1?\n\nD - Line 326: is it \"consist\" (only non-Korean) or \"contain\" (some non-Korean)? Line 320 seemed to imply \"contain\"... For instance, if a Korean sentence happens to contain the name of a foreign entity, written in Latin script, is the sentence discarded? This warrants a bit more discussion, at least for more clarity on what has been done, and preferably to explain why it is the right way to do that (or why it is not ideal and only chosen for practical reasons, but it is still OK and does not hurt representativeness because of this or that reason).", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer 1**\n\nWe sincerely appreciate Reviewer 1 for the many helpful comments and suggestions. We have addressed each comment below and highlighted the changes we plan to make in the revised manuscript. We’ve reordered the suggestions/questions by grouping related comments.\n\n---\n\n**1. (Suggestion):** Line 581, \"expected to ameliorate research biases\", I think this could be expanded to put more light on how this contribution is beneficial from that ethical point of view (because it is!).\n\n  **Response:** To better (and more straightforwardly) articulate this point, we have revised the sentence in line 581 as follows:\n\n  “Our research is expected to broaden the horizon of language science at the interface of technology, currently being limited to a handful of the world’s over 7000 languages (Joshi et al., 2020), which contributes to promoting more inclusive research practice toward diverse languages and language-usage contexts”.\n\n---\n\n**2. (Shortcoming):**  There is a discrepancy between the initial framing of the paper (title, introduction...) which is about lesser-studied languages and language-usage contexts, and the actual content of the paper which is solely about morphological analysis of L2-Korean. Is it a paper about low-resourced languages/domains (and Korean is just a case study to expand upon), or is it a paper about morphological analysis? If the former, it would be really useful for instance to add a new section \"Discussion\" before the conclusion, in order to discuss how that case study can be generalized to the targeted (broader) problem, which findings are new and reusable for low-resourced languages/domains in general... based on specific facts, and well-articulated.\n\n**Response:**  Our focus was predominantly on L2 Korean as a case study. While the broader context of lesser-studied languages and language-usage contexts served as an important background of this research, the core motivation was to delve into the intricacies of morphological analysis pertaining to L2 Korean. **In other words, our primary research aim was not to generalize our findings to other low-resource languages or domains. Instead, we sought to provide an empirical study on L2 Korean which, we believe, may offer insights for both researchers in computational linguistics aiming for diversification in their model across languages / domains and those in L2 learning-teaching leveraging computational tools.**\n\n**Meanwhile, we may modify the title and introduction in a way that better aligns with the content.** A potential title would be: “Morphological Analysis of L2 Korean: Experimental Insights into Developing Domain-Specific Model”. The revised introduction will emphasize the paper’s focus on morphological analysis of L2 Korean rather than its broader context of low-resource languages and domains. We would also elaborate on potential contributions on lesser-studied languages and language-usage contexts in the Ethics Statement section.\n\n---\n\n**3. (Shortcoming; Lines 470-473):** yes, matching the domain of training and test data is at the core of machine learning, and using in-domain data to fine-tune a model trained on out-of-domain data is the basis of many domain adaptation approaches, so \"implies the possibility that may...\" is awkward. What is the real conclusion here, what is the discovery?\n\n**Response:**  We acknowledge that the original phrasing was ambiguous and did not straightforwardly convey our findings. We intended to claim that the experiment demonstrated the effectiveness of using in-domain data to fine-tune a model initially trained on out-of-domain data. **Based on the suggestion, we have revised this part in the following way to provide a more straightforward interpretation of our results.**: “This suggests that a model trained on L2 data is more effective for handling L2 test data than a model solely trained on L1 data.”\n\n---\n\n**4. (Shortcoming; Line 372):** this is certainly ill-phrased, because \"patterns found in the data\" is really what machine learning is supposed to do in general... We do want to recognize patterns, otherwise there is nothing to learn. Question is: are those appropriate patterns, are they informative for that language/domain, is the dataset leaking, etc. This should be rephrased and expanded to better delineate what phenomena are specifically targeted here.\n\n**Response:** We admit that our initial wording was overly generic and did not clearly delineate the specific phenomena that we wanted to highlight. **The followings are two major aspects considered in our study.** \n\n**First, the L2-domain specific model learned basic words that are rarely found in the L1 training dataset.** As shown in Table 2, noun (tagged as NNG) emerged as the most frequently identified POS tag in the datasets, and the F1 scores for predicting this tag elevated with the L2 models (i.e., L2-NIKL-o; L2-KLM), when compared to the baseline model (Figure 1). \n\n**Second, the L2-domain specific model may have been adept at identifying prevalent morpheme combination patterns unique to the L2 datasets.** As elaborated in §2, the Korean language has agglutinative nature in which words very frequently encompass multiple morphemes. These morphemes in the datasets are combined with a plus (+) symbol, illustrated in a sentence “나+는 (*na+nun*; I+subjective case marker) 학교+에 (*hak-kyo+ey*; school+oblique case marker) 가+요 (*ka+yo*; go+sentence final marker)” which translates to “I go to school” in English. Given that L2-Korean learners tend to use simpler lexico-grammatical structures in Korean (e.g., Lim et al., 2022; Hwang, 2023), we hypothesized that morpheme combinations in L2 datasets would differ from those in the L1 dataset. **We have thus revised Lines 369-374 as follows:** \n\n“This is in line with previous findings suggesting that domain-specific data can enhance model performance because of the models’ familiarity with patterns found in the target data (e.g., Toutanova et al., 2003; Giménez and Marquez, 2004; Shen et al., 2007). In this study, two patterns emerged as significant. First, the L2 models may effectively detect basic words that are rarely present in the L1 dataset. This is supported by the results that (i) a noun (tagged as NNG) was the most frequently identified POS tag in the datasets (see Table 2) and (ii) the F1 scores predicting this NNG tag elevated with the L2 models (see Figure 1). Second, the L2 models may excel in discerning common morpheme combinations that more frequently appeared in the L2 datasets.  What is particularly evident is the Korean language’s agglutinative nature (see §2.1) where a word can comprise multiple morphemes marked with a plus (+) symbol in the dataset (e.g., 나+는 학교+에 가+요 *na+nun hak-kyo+ey ka+yo* “I go to school”). Given that L2-Korean learners tend to use less sophisticated lexico-grammatical structures (Lim et al., 2022; Hwang, 2023), we postulate a variation in morpheme combinations between native and L2 Korean speakers.”\n\n---\n\n**5. (Question, Line 215):** it would be useful to expand a bit on what is meant by \"same origin\" (same speakers?) and why it creates this uncertainty (I assume this is because some L2 errors vary a lot from one speaker to another, so the dataset leaks with respect to the typology of errors? Is that all?).\n\n**Response:** **By \"same origin\", we meant data originating from comparable settings or conditions.** This includes, but is not limited to, factors such as the data-collection prompt or the nature of the task assigned to participants (in this case, L2 Korean learners). For example, in learner corpus studies, the prompt and input of a task (given to a learner to ask to produce language) may significantly influence the language output (Alexopoulou et al., 2017). **We have elaborated on this term in the revised manuscript as footnote:**\n\nIn this context, “same origin” refers to learner data from similar settings or conditions, including factors such as the data-collection prompt and/or the nature of the task assigned to L2 Korean learners for language production. This consideration was informed by previous learner corpus studies that show that the task’s prompt and instructions can markedly influence learner output (see Alexopoulou et al. (2017) for more details).\n\n---\n\n**6. (Question, Line 326):** is it \"consist\" (only non-Korean) or \"contain\" (some non-Korean)? Line 320 seemed to imply \"contain\"... For instance, if a Korean sentence happens to contain the name of a foreign entity, written in Latin script, is the sentence discarded? This warrants a bit more discussion, at least for more clarity on what has been done, and preferably to explain why it is the right way to do that (or why it is not ideal and only chosen for practical reasons, but it is still OK and does not hurt representativeness because of this or that reason).\n\n**Response:** Our original intention was to filter out sentences from the L2-NIKL dataset that **contained** any non-Korean alphabets, predominantly Chinese characters. This decision was made after observing a subset of L1-Chinese L2-Korean learners who wrote their texts entirely in Chinese, which did not fit our research purposes. However, we now understand that, in certain contexts, the presence of non-Korean characters such as foreign names or technical terms can be relevant even if these terms are written in a non-Korean script. **Given these considerations, we have modified the wording in Line 326 to “contain” and provided a more detailed explanation in the revised manuscript to clarify our decisions and their implications.**\n\n---\n**7. (Evaluation metrics):**\n\n**7-1. (Shortcoming):** Lack of clarity on the evaluation metrics (see questions below), which renders the numbers quite hard to interpret.\n\n**7-2. (Question, Line 301):** Unclear how \"Perfect score\" is defined. \"comparing the number\" is dubious (does not seem to match what is done in the provided reference): only the number, or their boundaries as well? And how is this defined when using Perfect score on the POS task? This is important information that should be clear in the text itself.\n\n**Response:** **“Perfect score” emphasizes the need for perfect alignment between the predicted and gold standard annotations. For a sentence to be perfectly predicted in light of tokenization and POS tagging, every word (or *eojeol* in Korean) in the sentence should match the number of morphemes indicated in the gold standard.** This ensures accuracy in both the tokenization of each morpheme and its subsequent POS tagging (Note that we provided the codes used for this evaluation in the supplemental materials; ‘evaluation.ipynb’). \n\nGiven the agglutinative nature of the Korean language in which a single word often contains multiple morphemes, our priority was to ensure that morphological analyzers identified the same number of morphemes as indicated in the gold annotations. **If a word, according to the gold standard, is expected to be segmented into three morphemes but the morphological analyzer divides it into only two, we considered the word mis-parsed even though two morphemes are correctly tagged in alignment with three gold POS annotations.**\n\nWhile we aimed to explore how previous Korean tokenization/parsing models tackled this issue, we were unable to identify a relevant reference. In this regard, “Perfect score” outlined by Ramen et al. (2022) resonated with our objectives as it similarly addresses the ramifications of imperfect tokenization, especially in languages without spaces (e.g., Thai). The definition in the paper, suggesting that it \"reflects the % of times an example is parsed perfectly,\" closely aligns with our approach. \n\n**To provide clearer insight in the manuscript, we have explicitly elucidated both the motivation behind and the definition of “Perfect score” within our study's context. We have also stressed the significance of counting morphemes before evaluating the F1 scores for tokenization and POS tagging. In addition, we have incorporated examples for readers’ understanding.**\n\n**7-3. (Question, Line 305):** Unclear how the F1 score is defined. Instances=the morphemes? What is the difference between \"parsing is perfect\" and \"tokens align exactly with the golds\" (the metric seems tautological)? And the phrasing looks more as an accuracy than an F1 (what do precision and recall mean, with that definition?). For the POS task, is it micro-F1 or macro-F1?\n\n**Response:** \n- In our study, each **\"instance\"** is referred to as an individual morpheme.\n- What we intended by **\"perfect parsing\"** was about our model's ability to predict the exact number of morphemes for a token, consistent with the gold standard. Meanwhile, the term **\"exact alignment with golds\"** refers to the precise match of both the sequence and each morpheme’s tokenization/POS tag with its counterpart in the gold standard. To consider a token as perfectly parsed, it must satisfy both these conditions. (Lines 304-307: “F1 score measures the instances in which predicted tokens/tags align exactly with the golds, but only when the parsing is perfect for the given eojeol.”)\n- Our methodology involves aggregating all true positives, false positives, and so on, across the entire dataset and then calculating precision, recall, and F1, making it a **micro-F1 calculation**.\n\n---\n\n**8. (Other stylistic, grammatical, formatting issues):**\n\n**8-1. (Adding an outline):**  At the end of §1 it would be good to have some teaser on the outline, so that the readers can identify more directly where to find the information they seek. This can be short, for instance in the style of \"After introducing the specificities of L2-Korean (§2), we present in §3...\".\n\n**Response:** **We have added an additional paragraph at the end of §1:**\n“This paper is structured as follows: We discuss the importance of morphological analysis in Korean studies and application of morphological analyzers in L2-Korean research (§2). Next, we outline an experiment including datasets and evaluation metrics (§3). We delve into two subsequent experiments: the first involves fine-tuning an L1 model using the full scope of the L2 datasets (§4), while the second focuses on rigorous data cleaning in one of the L2 datasets (§5).  Comprehensive analyses of morpheme tokenization and POS tagging accuracy for both experiments are presented. Finally, we summarize our findings and propose future directions (§6).\n\n**8-2. (Changing titles):** Title of §3: this section is not really about a method, it discusses data, metrics... basically, the experimental conditions. A more usual title for a section like this would be \"Experimental setup\". Title of §3.2: this section is also about evaluation, so titling it \"Model training and evaluation\" would be more appropriate. Titles of §4 and §5: any way to have a more explicit title than \"Experiment N\"? Something that actually summarizes the purpose or conditions of that experiment. That would be helpful to understand the outline, but also in other places of the paper that make references to those experiments, it would certainly help to understand what is referred to and why.\n\n**Response:** We have updated the title of §3 from \"Method\" to **\"Experimental setup\"** and the title §3.2 from \"Model training\" to **\"Model training and evaluation\"**. We have also revised the titles of §4 and §5 by **adding subtitles** to \"Experiment N\". For §4, the title is now **\"Experiment 1: Maximizing Utility of the L2-NIKL dataset\"**, and for §5, **\"Experiment 2: Refining and cleaning L2-NIKL Dataset\"**. We believe these subtitles to encapsulate the goals of each experiment, which are summarized at the start of the respective experiment sections (§4 Lines 312-314; §5 Lines 413-417). \n\n**8-3: (Line 298):** missing word after \"performance\"? \"model performance measure was undertaken\"?\n\n**Response:** We have added \"measurement\" after the word \"performance\": \"(Line 298) For robust evaluation, model performance measurement was undertaken using the following metrics:\"\n\n\n**8-4 (Line 366):** (and same for lines 458, 471, and probably others): It would be better to use \"tuned\" rather than \"trained\" in order to better distinguish models trained from scratch from models fine-tuned on that data. A phrasing like \"the models trained on L2 data\" seems to refer to models which have never seen any L1 data.\n\n**Response:** For improved clarity, we have replaced \"trained\" with \"tuned\" in the revised manuscript. \n\n\n**8-5. (Table formatting):** Table 3 (and same for Table 4): The way to read the table is really not obvious (why that many numbers in the same cell), the answer is only given in the caption and part of it is given only implicitly (by means of putting Perfect in parentheses). I would suggest reducing the first column (maybe rotating the labels? or just reducing fontsize?) so that a short second column can be added to indicate the meaning of each line, i.e. adding for instance \"F1\" / \"\\Delta_{best}\" / \"PS\" / \"\\Delta_{best}\" vertically just on the right of \"L2-NIKL-o Train\".\n\n**Response:** To ensure better readability, we have revised both tables in the following way (e.g., Table 3): \n\n| Training | Metric | L2-NIKL-o TOK | L2-NIKL-o POS | L2-KLM TOK | L2-KLM POS |\n|----------|--------|---------------|---------------|------------|------------|\n| L2-NIKL-o | PS | **94.32** | 95.10 | 89.64 | 92.01 |\n| | PS Δbest | - | ↓1.07 | ↓5.46 | ↓3.18 |\n| | F1 | **93.27** | **92.52** | 84.12 | 84.87 |\n| | F1 Δbest | - | - | ↓9.09 | ↓6.03 |\n| L2-KLM | PS | 92.26 | 93.40 | **95.10** | **95.19** |\n| | PS Δbest | ↓2.06|↓2.77| - | - |\n| | F1 | 90.26 | 87.35 | **93.21** | **90.90** |\n| | F1 Δbest | ↓3.01 | ↓5.17 | - | - |\n| L1-GSD | PS | 93.61 | **96.17** | 92.28 | 92.47 |\n| | PS Δbest | ↓0.71 | - | ↓2.82| ↓2.72 |\n| | F1 | 92.36 | 87.52 | 88.98 | 85.37 |\n| | F1 Δbest | ↓0.91 | ↓5.00 | ↓4.23 | ↓5.53 |\n\n---\n\n**9. (Punctuation/Capitalization/Grammatical issues)** \n\n**Response:** We thank the reviewer for their keen eyes on these matters. \n\n- Footnote 9: issues with parentheses and commas. After \"S\": missing comma + remove parenthesis. Before \"SN\": missing comma\n(Revised sentence in Footnote 9: “We excluded tags beginning with 'S', such as SF (sentence-final punctuation marks, e.g., periods), SN (numbers), and SL (foreign characters), as they are not deemed critical for this study.”\n- Table 3's caption (and same for Table 4's caption): missing final dot\n- Line 378 (and same for line 462): \"Perfect\" with capital P would render the sentence more legible.\n- Line 442: \"presents shows\" --> \"shows\"\n\n---\n**10. References (to be included)**\n\nThe following studies are mentioned in the Rebuttal stage but were not included in the submitted manuscript:\n\n- Alexopoulou, T., Michel, M., Murakami, A., & Meurers, D. (2017). Task effects on linguistic complexity and accuracy: A large‐scale learner corpus analysis employing natural language processing techniques. *Language Learning*, 67(S1), 180-208\n\n-  Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020). The state and fate of linguistic diversity and inclusion in the NLP world. *arXiv preprint arXiv:2004.09095*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Lesser-studied languages are a known important topic, but this paper further expands it by introducing the consideration of language-usage context. This triggers new ideas, and could lead to expanding the views of many researchers. Also, the paper itself offers a lot of context (§2 provides an interesting review of L2-Korean and the associated challenges and applications), and the experimental work is conducted with rigorous analysis. In particular, it is appreciable to see how the authors performed error analysis and examined detailed results to better understand what phenomena affected the performance of the models (around line 390, line 482...).\n\nSuggestion: Line 581, \"expected to ameliorate research biases\", I think this could be expanded to put more light on how this contribution is beneficial from that ethical point of view (because it is!).", "reasons_to_reject": "The main two shortcomings are:\n- a) Lack of clarity on the evaluation metrics (see questions below), which renders the numbers quite hard to interpret.\n- b) There is a discrepancy between the initial framing of the paper (title, introduction...) which is about lesser-studied languages and language-usage contexts, and the actual content of the paper which is solely about morphological analysis of L2-Korean. Is it a paper about low-resourced languages/domains (and Korean is just a case study to expand upon), or is it a paper about morphological analysis? If the former, it would be really useful for instance to add a new section \"Discussion\" before the conclusion, in order to discuss how that case study can be generalized to the targeted (broader) problem, which findings are new and reusable for low-resourced languages/domains in general... based on specific facts, and well-articulated.\n\nThere are also a few places where the authors make claims or conclusions that appear rather void:\n- Line 470-473: yes, matching the domain of training and test data is at the core of machine learning, and using in-domain data to fine-tune a model trained on out-of-domain data is the basis of many domain adaptation approaches, so \"implies the possibility that may...\" is awkward. What is the real conclusion here, what is the discovery?\n- Line 372: this is certainly ill-phrased, because \"patterns found in the data\" is really what machine learning is supposed to do in general... We do want to recognize patterns, otherwise there is nothing to learn. Question is: are those appropriate patterns, are they informative for that language/domain, is the dataset leaking, etc. This should be rephrased and expanded to better delineate what phenomena are specifically targeted here.", "questions_to_authors": "A - Line 215, it would be useful to expand a bit on what is meant by \"same origin\" (same speakers?) and why it creates this uncertainty (I assume this is because some L2 errors vary a lot from one speaker to another, so the dataset leaks with respect to the typology of errors? Is that all?).\n\nB - Line 301: Unclear how \"Perfect score\" is defined. \"comparing the number\" is dubious (does not seem to match what is done in the provided reference): only the number, or their boundaries as well? And how is this defined when using Perfect score on the POS task? This is important information that should be clear in the text itself (not relying solely on the reference).\n\nC - Line 305: Unclear how the F1 score is defined. Instances=the morphemes? What is the difference between \"parsing is perfect\" and \"tokens align exactly with the golds\" (the metric seems tautological)? And the phrasing looks more as an accuracy than an F1 (what do precision and recall mean, with that definition?). For the POS task, is it micro-F1 or macro-F1?\n\nD - Line 326: is it \"consist\" (only non-Korean) or \"contain\" (some non-Korean)? Line 320 seemed to imply \"contain\"... For instance, if a Korean sentence happens to contain the name of a foreign entity, written in Latin script, is the sentence discarded? This warrants a bit more discussion, at least for more clarity on what has been done, and preferably to explain why it is the right way to do that (or why it is not ideal and only chosen for practical reasons, but it is still OK and does not hurt representativeness because of this or that reason).", "paper_topic_and_main_contributions": "This paper presents a mostly experimental contribution, focusing on morphological analysis of L2 Korean and its specificities compared to L1 Korean (hence a lesser-studied language + a lesser-studied language-usage context). Two sets of experiments are described:\n- one about fine-tuning an L1 model with L2 data and studying the impact of train-test domain discrepancy even within L2 data,\n- and another one that is more oriented towards data quality, cleaning annotation artefacts and where the authors analyze the impact of that cleaning on domain mismatch effects.\n\nThe authors also intend to share the script for cleaning that data, which by itself is a contribution as it can facilitate future research on L2 Korean and its morphological analysis.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- At the end of §1 it would be good to have some teaser on the outline, so that the readers can identify more directly where to find the information they seek. This can be short, for instance in the style of \"After introducing the specificities of L2-Korean (§2), we present in §3...\".\n- Title of §3: this section is not really about a method, it discusses data, metrics... basically, the experimental conditions. A more usual title for a section like this would be \"Experimental setup\".\n- Title of §3.2: this section is also about evaluation, so titling it \"Model training and evaluation\" would be more appropriate.\n- Line 298: missing word after \"performance\"? \"model performance measure was undertaken\"?\n- Titles of §4 and §5: any way to have a more explicit title than \"Experiment N\"? Something that actually summarizes the purpose or conditions of that experiment. That would be helpful to understand the outline, but also in other places of the paper that make references to those experiments, it would certainly help to understand what is referred to and why.\n- Line 366 (and same for lines 458, 471, and probably others): It would be better to use \"tuned\" rather than \"trained\" in order to better distinguish models trained from scratch from models fine-tuned on that data. A phrasing like \"the models trained on L2 data\" seems to refer to models which have never seen any L1 data.\n- Footnote 9: issues with parentheses and commas. After \"S\": missing comma + remove parenthesis. Before \"SN\": missing comma.\n- Table 3 (and same for Table 4): The way to read the table is really not obvious (why that many numbers in the same cell), the answer is only given in the caption and part of it is given only implicitly (by means of putting Perfect in parentheses). I would suggest reducing the first column (maybe rotating the labels? or just reducing fontsize?) so that a short second column can be added to indicate the meaning of each line, i.e. adding for instance \"F1\" / \"\\Delta_{best}\" / \"PS\" / \"\\Delta_{best}\" vertically just on the right of \"L2-NIKL-o Train\".\n- Table 3's caption (and same for Table 4's caption): missing final dot\n- Line 378 (and same for line 462): \"Perfect\" with capital P would render the sentence more legible.\n- Line 442: \"presents shows\" --> \"shows\"\n- Better to reference the annexes in the main text (explicit pointers from the sections they provide more details for), so that the reader knows immediately while reading the main text when additional information is available elsewhere."}}, {"review_id": "vZxgqeQwbv", "reviewer": "Reviewer_qhvV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer 2**\n\nWe sincerely appreciate the Reviewer 2’s feedback and concern.\n\n---\n\n**1. (Feedback/concern):** My main concern about this paper is that the motivation is somewhat unclear to me even after reading Section 2. Concretely, the paper did not explain why it chose L2 Korean to investigate the generalizability of models. It would be better to use some examples or statistics that can show the different natures of L1 and L2 data.\n\n  **Response:** \n  **Our primary focus centers upon L2 Korean studies. However, we believe this research would appeal to computational linguists who prioritize a sense of Diversity, Equity, and Inclusion in research because L2 Korean represents one of the lesser-studied languages and language-usage contexts in the field.**\n\n  To provide clarity for Section 2, we elaborated on the motivation for our study. We began by explaining the agglutinative nature of Korean, characterized by its morphological intricacies (Section 2.1). These complexities are often amplified when encountered by L2 learners. Meanwhile, a growing number of L2-Korean studies have started utilizing automatic morphological parsers/taggers, predominantly in quantitative research (Section 2.2). However, when existing models – trained exclusively on L1 datasets – are applied to L2 Korean data, they have shown notable discrepancies in performance (Section 2.3). These computational challenges, combined with the growing demand on L2-specific models from (applied) linguistics, underscored the need for models tailored to the L2-Korean domain. Furthermore, we drew attention to the ongoing research efforts in L2 English and Chinese in Section 1 (Lines 47-48: Kyle et el., 2022; Lan et al., 2023 – Upon reviewing this part, we realized it might be beneficial to specify that the studies by Kyle et al. (2022) and Lan et al. (2023) are related to L2 English and L2 Chinese, respectively).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper shows an interesting result: increasing the size of training data does not always help to enhance performance. The result can be an essential instruction for developing L2 Korean data.", "reasons_to_reject": "My main concern about this paper is that the motivation is somewhat unclear to me even after reading Section 2. Concretely, the paper did not explain why it chose L2 Korean to investigate the generalizability of models. It would be better to use some examples or statistics that can show the different natures of L1 and L2 data.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper mainly investigates the generalizability of the morphological parsers/taggers on second-language (L2) Korean datasets. The results underline the importance of the quality of training data; the model trained on cleaned training data outperforms the one trained on original training data even though the size of cleaned training data is half of the original size.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MQunH59dBc", "reviewer": "Reviewer_i3cX", "summary": "", "questions": "- Have you considered the effect of the native language of the L2 speakers? This probably has an effect on the characteristics of L2 sentences.\n- I do not really understand the benefits of L2 models when tested out-domain, as they perform evenly or worse than the baseline. What are the advantages in this case, if any?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reviewer 3**\n\nWe sincerely appreciate the Reviewer 3’s insightful questions and suggestion regarding the title style. We have addressed each comment in the sections below:\n\n---\n\n**1. (Question):** Have you considered the effect of the native language of the L2 speakers? This probably has an effect on the characteristics of L2 sentences.\n\n  **Response:** \n  **The influence of the native language on L2 writing is undeniably significant in L2 research, but it was not the central point of our current study.** This was because we were primarily interested in how currently available language models work with L2 data processing in a broad sense. Together with providing detailed background information (e.g., proficiency level, L1 background, etc.) on L2 speakers from both datasets (i.e., L2-NIKL; L2-KLM) in the appendices, we have mentioned that future research (with a more sizeable dataset) should seek to pursue this issue thoroughly.\n\n---\n\n**2. (Question):** I do not really understand the benefits of L2 models when tested out-domain, as they perform evenly or worse than the baseline. What are the advantages in this case, if any?\n\n  **Response:** While it might appear at first glance that L2 models perform similarly or even worse than the baseline in some instances (as in Table 3), **it is important to highlight that we observed improvements in certain POS tags, as depicted in Figures 1 and 3. Moreover, our data from Lines 466-473 revealed an intriguing trend such that the models trained on the L2-KLM dataset surpassed the performance of the L1-GSD baseline model in POS tagging of the out-domain (unseen) L2-NIKL-c test set (also illustrated in Table 6)**. This suggests that a model fine-tuned with L2 data might be better equipped to handle out-domain L2 data than that solely reliant on L1 data (the baseline model). Thus, our findings underscore the potential necessity of fine-tuning current L1 models to L2-specific domains. Further emphasizing this point, our discussions in Section 6 (Lines 514-521) advocate for a heightened focus on input quality, emphasizing the imperative for rigorous annotations and thorough data cleaning. We believe that such measures are integral for developing L2 models that benefit overarching L2 studies, not just those confined to L2 Korean.\n\n---\n\n**3. (Style):** The titles in English are usually written capitalising all words except for short articles, prepositions, and coordinating conjunctions. For example, the title of the paper should be: \"Diversifying Language Models for Lesser-Studied Languages and Language-Usage Contexts: A Case of Second Language Korean.”\n\n  **Response:** We have made the necessary changes to ensure that the title adheres to the convention.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Code will be public.\n- Complete and exhaustive experimental setup: extensive experiments and detailed discussion.\n- In-depth data treatment, both analysis and cleaning.", "reasons_to_reject": "- None", "questions_to_authors": "- Have you considered the effect of the native language of the L2 speakers? This probably has an effect on the characteristics of L2 sentences.\n- I do not really understand the benefits of L2 models when tested out-domain, as they perform evenly or worse than the baseline. What are the advantages in this case, if any?", "paper_topic_and_main_contributions": "The authors present a study about morpheme POS tagging and parsing of L2 Korean speakers. They conclude that models trained on L2 data can outperform models trained on L1 data when tested on the same domain. Second, more data is not always better, as cleaned training sets of L2 speakers provide a better result than the original training set with more samples. Third, performance is not bidirectional when testing on unseen L2 data from different domains.\n\nOverall, this paper is complete and proposes an interesting and solid study about Korean language learners text.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The titles in English are usually written capitalising all words except for short articles, prepositions, and coordinating conjunctions. For example, the title of the paper should be: \"Diversifying Language Models for Lesser-Studied Languages and Language-Usage Contexts: A Case of Second Language Korean.”"}}]

Author rebuttal:

Meta-review: This article describes some interesting work on the quality of morpho-syntactic analysis of learner sentences in Korean, a new task that raises many interesting questions. The reviewers all emphasised the quality and level of detail of the analyses. Many suggestions were made by the reviewers to improve the quality of the presentation. Given the number of modifications requested, I think a second round of reviews is necessary
Decision: Accept-Findings