Does Incomplete Syntax Influence Korean Language Model? Focusing on Word Order and Case Markers
Authors: Jong Myoung Kim
Young-Jun Lee
Yong-Jin Han
Ho-Jin Choi
Sangkeun Jung
Venue: COLM (2024)
Keywords: syntax, word order, postposition, case marker, syntactically-incomplete data
TL;DR: We confirmed the syntactic flexibility of Korean, examined if LLMs capture this feature, and applied it as a data augmentation method to enhance performance.

Abstract: Syntactic elements, such as word order and case markers, are fundamental in natural language processing. Recent studies show that syntactic information boosts language model performance and offers clues for people to understand their learning mechanisms. Unlike languages with a fixed word order such as English, Korean allows for varied word sequences, despite its canonical structure, due to case markers that indicate the functions of sentence components. This study explores whether Korean language models can accurately capture this flexibility. We note that incomplete word orders and omitted case markers frequently appear in ordinary Korean communication. To investigate this further, we introduce the Syntactically Incomplete Korean (SIKO) dataset. Through SIKO, we assessed Korean language models’ flexibility with incomplete syntax and confirmed the dataset’s training value. Results indicate these models reflect Korean’s inherent flexibility, accurately handling incomplete inputs. Moreover, fine-tuning with SIKO enhances the ability to handle common incomplete Korean syntactic forms. The dataset’s simple construction process, coupled with significant performance enhancements, solidifies its standing as an effective data augmentation technique. The SIKO will become accessible post-publication.

Published as a conference paper at COLM 2024
Does Incomplete Syntax Influence Korean Language Model? Focusing on Word Order and Case Markers
Jong Myoung Kim!
Young-Jun Lee'
Yong-jin Han
Sangkeun Jung' Ho-Jin Choil
School of Computing, KAIST
School of Computer Science and Engineering, Kyungpook National Univeristy The Division of Computer Convergence, Chugnam National University
grayapple@kaist.acc kr
yjhan@sejong knu ac
yj2961 hojine
@kaist ac.kr
hugmanOcnu.co ww..muac...... kr
Abstract
Syntactic elements, such as word order and case markers, are fundamental in natural language processing. Recent studies show that syntactic infor- mation boosts language model performance and offers clues for people to understand their learning mechanisms. Unlike languages with a fixed word order such as English, Korean allows for varied word sequences, despite its canonical structure, due case markers that indicate the functions of sen- tence components. This study explores whether Korean language models can accurately capture this flexibility. We note that incomplete word orders and omitted case markers frequently appear in ordinary Korean communi- cation. To investigate this further, we introduce the Syntactically Incomplete Korean (SIKO) dataset. Through SIKO, assessed Korean language mod- els' flexibility with incomplete syntax and confirmed the dataset's ss training value. Results indicate these models reflect Korean's inherent flexibility, accurately handling incomplete inputs. Moreover, fine-tuning with SIKO enhances the ability to handle common incomplete Korean syntactic forms. The dataset's s simple construction process, coupled with significant perfor- mance enhancements, solidifies its standing as an effective data augmenta- tion technique. We make our source code and dataset publicly available,
1 Introduction
Syntactic information is crucial for humans to understand sentence meanings accu- rately (Chomsky 2002). One prominent syntactic element, word order, has been a central topic in recent Language Model (LM) research. Studies have explored various aspects of word order: its significance to LMs (Pham et al.) al. Sinha et al. (2020), methods to improve model performance via word order adjustments (Sinha et al. 2021)),)) and insights into how LMs progressively learn human language (Abdou et al, 2202)).
Korean follows canonical Subject-Ojject-Vent (S-O-V) word order. However, the presence of case markers, type of postposition, allows Korean to exhibit relatively flexible word or- der (Lee & Im 1997 Yeon & Brown 2013 |ohn (205). Case markers denote the grammatical roles of constituents within sentence, like their function as subject or object. Conversely, due to its canonical word order, Korean often experiences omissions of case markers. Our aim is to verify whether this flexibility is reflected in Korean LMs.
"Corresponding author.

---

Published as a conference paper at COLM 2024
Figuree exemplifies the flexibility of Korean sentence, "4 E " in terms of word order and case markers. Even with word order changes (Arrow a), the case markers '7H' and g identify '4 and " subject and object, respectively. Conversely, in S-O-V order (Arrow b), subject and object are identifiable by position, even without case markers. Of course, the freedom from case markers and word order comes with limitations. Sentences lacking case markers or with rearranged word order might appear unnatural or have the potential to be interpreted differently from the original sentence. More detailed information about Korean word order and structure is as follows:
Korean typically follows an SOV (Subject-Ohject-Ver) order.
Major reorderings occur among full phrases (e.g., noun phrase arguments) and usually take place before the verb.
Moving the verb to an earlier position is much rarer.
Splitting a noun phrase argument (e.g,, placing an adjectival modifier before the noun) also much rarer.
Naturally dropping case markers is more common with subjects than with objects (or vice versa, depending on the context).
Word order shuffle
4
|
(The butterfly is drinking nectar.)
(a)
"L
7
- D
V
S
O-marker
S-marker
subject butterfly
Case marker exists (meaning preserved)
case marker for subject
verb:
(b)
"
D O
drinking
Case marker delete
object: nectar
V
S-O-V canonical order (meaning preserved)
case marker for object
complete sentence
incomplete sentences
Figure 1: An example of Korean syntax flexibility: a) Case markers enable word order variability, b) They can be omitted in canonical sequences.
We aimed to determine if the syntactic flexibility inherent to the Korean language is mirrored in Korean LMs. Leveraging these unique linguistic characteristics, our objective was to improve the performance of Korean LMs. To this end, we established a methodology for generating Syntactically Incomplete Korean (SIKO) data and subsequently constructed comprehensive dataset. Further, we evaluated the effectiveness of the SIKO dataset as resource for fine-tuning LMs, focusing its practical applicability in enhancing language processing capabilities.
We designed various experiments using the SIKO data. We employed four fundamental tasks used for assessing LM performance: Text Classification (TC), Natural Language Inference (NLI), dialogue topic classification, and dialogue summarization. For the experiments, we used the well-established KLUE benchmark (Park et al. 2021) and the AI-hub datafreeeesse by national authorities. This compilation includes a wide range of Korean language expres- sions, from news headlines to conversational texts, thereby expanding our experimental scope to cover the extensive usage of Korean. Additionally, we examined the characteristics and performance across three different Large Language Models (LLMs) with varying sizes and structures, to uncover both commonalities and unique aspects in processing syntactic phenomena.
2It is a data platform operated by the Korea National Information Society Agency (NIA). It collects and processes artificial intelligence data in six fields and makes it publicly available. https: //www aihub. kr/
2

---

Published as a conference paper at COLM 2024
In our study, we observed that expressions featuring syntactic incompleteness prevalent in real-world Korean language applications. We were able to confirm through inference experiments using SIKO data that LLMs responding to such syntactic incompleteness. Our analysis reveals that the SIKO dataset surpasses representative data augmentation meth- ods in efficacy, especially for fine-tuning purposes. Significantly, it demonstrates superior capability in improving the handling of inputs that are syntactically incomplete, common phenomenon in typical Korean usage.
The contributions of our study are as follows:
This study is the first to explore syntactically incomplete Korean sentences.
We constructed the Syntactically Incomplete Korean (SIKO) dataset.
Through downstream testing with SIKO, we verified that the Korean LLM exhibits flexibility when dealing with syntactically incomplete inputs.
Models tuned using SIKO demonstrated improved performance handling incom- plete data commonly found in Korean.
Related works
Syntactic Information In neural network-based Natural Language Processing (NLP) research, syntactic information has been a consistently explored topic. Further emphasizing its significance, Vanderwende & Dolan (2005) addressed a considerable number of problems in the PAACCAL challenge using solely syntactic information (Dagan et al., 2005). The question of whether these LMs genuinely understand the syntax human language and align their learning outcomes with human comprehension remains subject of continued interest (Sinha et al. 2020
Word Order Word order a prominent element of syntactic information. While research asserting the lack of importance of word order in input data has been presented (Pham et al. 2020), consistent efforts are also being made in studies attempting to enhance performance through training on data with disrupted word order (Sinha et al., 2021). Furthermore, endeavors persist to comprehend how LMs learn human language by utilizing models trained on data with shuffled corpus (Abdou et al.) 2022), Korean, while possessing formal word order, exhibits relative flexibility from it due to the presence of case markers (Lee & Im 1997). We seek to determine whether the Korean LM also reflects this flexibility.
Incomplete Sentences for data augmentation Recent research has demonstrated success in enhancing model robustness and performance by utilizing noisy or incomplete data. Consistency learning, performed by /Zhou et al. (2021) with perturbed data and Gao et al. (0022) with dropout-applied incomplete data, is one approach. Data augmentation methods, such the repetition method employed by Wu et al. (0222 and the replacement of low-informatiin words proposed by Xie et al. 22200, have also shown promise. General and simple data augmentation methods like Easy data augmentation (EDA) An easier data augmentation (AEDA) have already become widely used techniques (Wei & Zou 2019 Karimi et al. 2021). We aimed to augment data by considering the flexibility of word order and case markers as form of noise.
3 Syntactic Flexibility in Korean
We explored the frequency of case marker omissions and word order changes in practical Korean usage. For this investigation, we took 1,000 samples each from two tasks within the KLUE benchmark and conversational data from Al-hub. We assessed the presence of omitted case markers and deviations in word order, and determined whether such phenomena led to ambiguity or challenges in interpreting the meaning of the sentences.
Additionally, we restored omitted case markers and corrected word order in 300 samples for each task. This process allowed us to identify the number of case marker positions,
3

---

Published as a conference paper at COLM 2024
Case of Syntactical Completeness
of samples
KLUE-TC
KLUE-NLI
Al-Hub Dialogue
Complete Case Marker
1,000
6.60%
54.30%
7.70%
Case Marker Omitted & No Translation Issues
1,000
93.40%
43.70%
79.30%
Marker Omnitee Translation Issues Present
1,000
0%
13.00%
Average Number of Case Marker Positions
300
4.23
7.15
8.86
Case Marker Omission Rate
300
80.10%
21.37%
54.25%
Canonical Word Order
1,000
88.00%
95.70%
65.00%
Non-Canonical Word Order No Translation Issues
1,000
12.00%
4.30%
35.00%
Non-Canonical Word Order & Translation Issues Present Word Order Correction Rate (for Non- Canonical
1,000
0%
0.00%
0.00%
Word Order)
300
45.35%
31.10%
28.98%
Table 1: Analyzing Case Marker Omission and Word Order Variations in Korean with KLUE- TC, KLUE-NLI, and Al-hub Dialogue Datasets. Case Marker Omitted, -on-Canonical Word Order: Instances with omitted case markers non-standard word order. Translation Issues Present: Ambiguous or uninterpretable sentences. Number of Case Marker Positions: The number of sentence case markers after restoration. Case Marker Omission Rate: The ratio of omitted to total case markers. Word Order Correction Rate: The word-level edit distance ratio of correction effort total words.
SIKO case
Description
Example
original data
Unprocessed data
CMjdl
Remove markers
Shufsem.Prso
Rearrange sentence order while maintaining meaning using chatGPT
19
H
Shufsem Pressr Mael
Remove case markers from Shufsem.Press Presry data
Shufsem:Non Non.Prr
Randomly shuffle the sentence order
Shufsem Non Presey &c
Remove case markers from Shufsem. Non.Preery data
Table 2: An example of SIKO data from Figure 1. Shufsem. Presrv and Shufsem. Non. might yield identical results. Red denotes the subject and its marker, blue the object and its marker, and green the predicate. The signifies deleted case markers.
assess omission rates across datasets, and evaluate the scale of word order modifications. The findings are showcased in Table [ All tasks described in this section were manually performed, and details about the individuals involved are provided in the Experiment Settings Section (5.4).
The rate of case marker omission case markers to the total number of case markers after their restoration. The word order correction rate determined by taking the word-level edit distance between the original data and the corrected word order data, then dividing by the total word count in the data.
calculated by comparing the number of omitted
From this analysis, we ascertained that the omission of case markers and alterations in word order are common in ordinary Korean usage. Specifically, case marker omission occurred in 97.4% of KLUE-TC instances and non-canonical word orders were found 35% of Al-hub dialogues. The scale of these phenomena is underscored by their extensive occurrence: an average omission rate of 80.1% for case markers in KLUE-TC and an average word order correction rate of 45.35% in KLUE-TC.
Notably, even with such syntactical incompleteness, the sentences largely retained their clar- ity of meaning. In dialogue data, there were instances where interpretation was challenged due to case marker omissions. However, it's important to note that these were extracted from conversations, potentially lacking some contextual or background information. This investigation reaffirms that while Korean showcases syntactic flexibility, it remains bounded to ensure semantic clarity.
4
SIKO: Syntactic Incomplete Korean dataset
In Section 3
we demonstrated Korean's extensive syntactic flexibility and its frequent occurrence actual use. We aimed to determine if Korean LMs reflect this variability. To this end, we generated datasets with deliberate syntactic incompleteness. This approach occasionally led to reductions in syntactic detail and interpretability.
4

---

Published as a conference paper at COLM 2024
We randomly sampled 22,000 instances from three data types (KLUE-TC, KLUE-NLI, Alhub- dialogue) and developed five variants SIKO data. Of these, 2,000 instances (and their SIKO derivatives) served the test set in Section 6.1 while 20,000 (with their SIKO derivatives) were used for fine-tuning in Section 6.2 Additional details on data statistics, prompts, and reviewer information are documented in Appendix
4.1 Case Markers Deletion (CMael)
We removed all case markers from the data to thoroughly eliminate the syntactic information they provide. While partial omission of case markers is common in Korean, complete removal is rare and generally limited to specific contexts, such as news headlines. We crafted this data by analyzing sentence structures with a morphological analyzer and stripping out the components that correspond to case markers.
Morphological analysis is a fundamental task in Korean NLP and crucial for Korean EDA. The method of removing case markers is as cost-efficient as EDA in terms of data generation. The morphological analyzer we used was developed in-house for the services of the company I am affiliated with. In terms of general performance, it has recorded an F1 score of 98. 98.8261n1 in evaluations conducted on sampled news article sentences.
4.2 Semantic Preserving Shuffling (Shufsem Presrv)
As mentioned earlier, Korean permits certain variations in word order without altering the meaning. We changed the word order in our dataset while preserving the sentence's 's meaning. These sentences may not conform to typical Korean patterns. Initial reordering was conducted by ChatGPT, followed by human reviewers ensuring semantic integrity.
4.3 Semantic Non-Preserving Shuffling (Shufsem.Non.Peee Non..resrv
We generated data by shuffling word order using Python's Random library, a method distinct from EDA's simple two-word swap, as it involves rearranging words across the entire sentence. This process may diverge significantly from typical Korean structures, potentially compromising syntactic integrity and original meaning.
4.4 Mixed Application (Shufsem.Prese &CMdel and Shufsem Non.. &Cl
We created datasets where we simultaneously applied word order changes and case marker deletion. We employed two distinct methods to change the word order, and then removed case markers from the altered data. This led the most pronounced reduction in syntactic information throughout our experiments.
Experiment Setting
5.1 Datasets
5.1.1 KLUE Benchmark
The KLUE benchmark, inspired by GLUE (Wang et al. al.] 2018), stands premier dataset in Korea. For our study on syntactically incomplete data, we omitted tasks that the answer resides within the input text, such machine reading comprehension, due the challenges of uniformly altering word order in both questions and answers. Thus, we selected two tasks from the available eight. The KLUE dataset comprises Training, Validation, and Test segments; our study solely employed the Training subset with accessible labels.
Text Classification (KLUE-TC) The aim of this task is to identify the subject or theme from a given text snippet. The dataset consists of 45,680 news headline titles. It's common for Korean news headlines to minimize the of case markers. We classified the news

---

Published as a conference paper at COLM 2024
headlines into one of seven categories based on their content: 'politics', 'economy', 'society', 'culture', world', 'T/scieece', or 'sports'.
Natural Language Inference (KLUE-NLD) The objective of NLI is determine the rela- tionship between premise and hypothesis. The dataset, formulated in standard literary style, consists of 23,000 entries. Based on given premise, an NLI model categorizes the hypothesis as either entailment (true), contradiction (false), or neutral (undeterminable))
5.1.2 AI-Hub Korean Dialogue Dataset
The AI-Hub dataset, known for its variety in dialogue types such as daily conversations and debates, consists of 279,99 ,992 dialogues primarily in the colloquial style typical of messaging platforms. Dialogues usually feature 2 to 5 participants and average 11.27 exchanges per conversation, with each dialogue annotated with topic and summary.
Text Classification (Dial-TCC This task involves classifying the topic of a given con- versation into one of nine categories: 'Personal and Relationships', 'Beauty and Health', 'Commerce (Shopping)', 'Current Affairs and Education', Food and Drink', Leisure', PPro- fessions', 'Residential and Living', and 'Events'.
Summarization (Dial-Sum) This task requires generating summary of provided con- versation, utilizing the same data as the TC task.
5.2 Models
PKO-T5 The PKO-T5 (Park) 2022), released by PAUST, is T5 (Raffel et al. 2020) based Korean sequence-to-sequenec model using Byte Pair Encoding (BPE) for out-of-vocabulary words. It's trained on sources like Namuewiki, Wikipedia, and Modu Corpus for T5's unsu- pervised task. PAUST offers three model sizes: small, base, and large. For our experiments, we primarily used the base model and examined the effects of scaling up to the large model.
Ko-GPT-Trinity 1.2B The Ko-GPT-Trinity developed by SK telecom, adapts the GPT-3 architecture (Brown et al. 2020). was trained on the extensive "ko-DAT" dataset, compris- ing 35 billion tokens over 72,000 iterations, using a masked autoregressive LM approach with cross-entropy loss for evaluation. We examined its syntactic flexibility in a causal LM framework, where the model's size, like selection of T5-large, was crucial.
chatGPT ChatGPT OPenAT 2022) by OpenAl is a GPT variant tailored for coherent conversational responses. Sourced from a vast internet corpus, excels dialogue gen- eration across multiple languages. We employed this model for generating semantically preserved word order alteration data and measuring zero-shot performance on syntacti- cally incomplete inputs. In-Context Learning(ICL) was not used; the tasks were performed purely in zero-shot mode. The prompts and detailed information used are recorded the Appendix B.3.2
5.3
Implementation Details
We conducted fine-tuning and generation experiments for syntactic flexibility using an NVIDIA A100 40GB GPU. We employed the Adam optimizer with decoupled weight Hutter (2017)) and set learning rate of 5e-5
decay Kingma & Ba (2014); Loshchilov with warm-up period spanning 3 epochs. Some of the experiments were conducted in zero-shot manner, without any tuning or training. Detailed library versions and additional information available in the Appendix E.1
Phtps///www.stelecmm.cmm/

---

Published as a conference paper at COLM 2024
T54aase
TS-Trge
KLUE-IC
KLLENNI
DGaFTC
Dial-Sum
KLUE-IC
KLUBNNII
Dial-IC
Dial-Sum
inconplleenness type
Mae-ft
Mac-ii
ROUGE-1
ROUGE-2
Mac-11
Mac-11
ROUGE1
ROUGBE2
ordirary Korean
85.1
78.3
73.4
76.8
06.3
85.8
80.0
69.8
79.6
46.1
CMal
-42
73.1.-a3)
76-5
442.111
N5.7
7955-84)
685
785
44.2-199
Shufeon Presyy
84.0 10
75.5 14
72.8 0s)
29:
45. 04)
-19
73.8 422
202
46.0. a.1
ShufsanPreer & CMad
83.9
742, 742.-13:
-222
78.
4577
85.3
75..
705
43.6.
Shufson
84.0
71.0
73.3
29.3gm
-37.-11
54.6
73.
20.6uu0
77.9 77.9-127
44.1-220
Nan
(4-2
(2.7
79:0
4344
72.1(-79
--44
-2.6
KO-GPT
chatGPT Dial-IC
KLUE-IC KLUE-NLI Dal-TC
Dial-Sum
KLUE-IC
KLUE-NLI
Dial-Sum
incompleteness type
Mae-fl
Acc.
Maefll
ROUGB-]
ROUGB-2
Mac-f1
Acc.
Mac-fl
ROUGE-1
ROUGB-2
ordirary Korean
82.8
22.4
67.
72.0
417
80.9
574
39.
26.8
6.5
CMul
-0.
-022
7100-1))
-0:5
-7.0
40.
28.3pm
6.6
ShufownThrer
82.5
67.6
68.2.s
72.
01.6
7927-123
52.5; 4.9
3977mw
27.
6.0 45)
Shufenn.herr CMaa
-2
65.3,-211
68.3
720
395
80.0-15)
51.3--1)
-5.4--14)
62.
Shufcon Naw Peenr
822 08)
68.
704
39. 120
78.7 122
38.; 16.50
38.8 -3)
248
-1
4. 11)
Shufoon Non Powee & CMss
81A
53.7 (87)
68.9
71.8
38.5
78.9 21
020) 154)
39.7
26.1
5.6
Table 3: When SIKO data is input into a model pre-trained or fine-tuned on standard Korean data, we observe the processing outcomes. We report scores for SIKO test cases and their variation from standard Korean cases as score The performance for TC was measured using Macro-F1, for NLI using Accuracy, and for summarization using ROUGE-1 and ROUGE-2.
5.4 Human-Constructed Data
In this study, multiple human-centric tasks such evaluation, restoration, and correction were conducted. In Section 3 they identified the presence omitted case markers or altered word order in general Korean usage and performed the task of restoring omitted case markers or word order to calculate their scale. In Section they reviewed and corrected the greaning-preserve meaning was indeed preserved. Two people directly handled these tasks. Subsequent to their efforts, another individual, termed the reviewer, carried out checks and verifications. All participants brought to the table minimum of 18 months' experience in creating and evaluating data for NLP. More detailed information on the task guidelines, training methods, and other aspects provided to these individuals is documented in the Appendix B.3.1
word order change data generated by ChatGPT to ensure that the
Experiments
6.1 Experiment 1: The Influence of Incomplete Syntax in Inference
We explored how LMs process inputs missing case markers or with non-standard word orders, selecting 2,000 samples from four tasks' training data as our test set. Using the SIKO approach from Section [ The remaining ordinary data was divided in a 9:1 ratio for training and validation during fine-tuning. For TC and NLI tasks with class labels, our sampling preserved original class distributions. We evaluated the inference performance of a LM fine-tuned on typical data with syntactically incomplete test set. We also attempted zero-shot inference with chatGPT to note processing differences. (Unfortunately,, T5 and KO-GPT were unable to correctly ana- lyze the task under zero-shot conditions. Therefore, we compared the inference performance using fine-tuned models.)
we created five types of syntactically incomplete test sets.
6.1.1 Result
Table 3 presents our experimental results. Notably, case marker deletion (CM&el) barely affected task performance. However, tasks reacted differently to word order changes, with the NLI task being particularly sensitive. Except for NLI, alterations in word order, whether semantically preserved (Shufsem.Presm) Pressv)) or not (Shufsem.Non Presry), showed minimal perfor- mance influence. This indicates that grammatical variations minimally affect understanding in contexts like news headlines or informal speech, or that LLMs can effectively interpret their meaning.
7

---

Published as a conference paper at COLM 2024
BASELINE
$280
Score
Ang 19te
not-Aup:
duplication
repeitiinn
AR3M
CMai
Shoffooe Powe
Stulseehhrr
Shaffee Sax
Shulees Pror CMa4
0.1
506p5)
83.3 433
$57
844
$36
83.
63.5
KLUE-TC
0.5
0.5
6
82.7
822
625
935
822
2msl
545
84.0pm
8433pm
634
534
825
835
62.9pm
51.5
631
(300g
500gpm
82 (07
K3
#55
803
$23
803
k38
613
52%
KLUE-NLI
812
825
818
833
81.
6355
822
82.3
$23
825
523
S1A
520gg
820
943
13
835
833
56-1g-
695
696
(0772
Dal TC
InA00 F1
69.6
693
680
70
697
0.1
770
27-kaa
787
778k.))
77.
7K (
0.5
26.3
772
222
778
77.444
77.
784
22.
787a7
78huo
273
283
750
77A
77.
75-
77.5
(k00
Dial Sum
77.5
m72
785pm
282
782
275
788
773
784
u35
15kpm
435mm
154
16700
412
434
444pa)
447mm
443
452
4533
437
443
442
435
411m5
453
453
443
458
442 Am
440
466
440gz)
467
443
433gm
4388gd
484,1.
Table 4: Data tuning results using SIKO data were compared to representative augmenta- tion strategies across transformer architectures for downstream performance. Tests varied augmentation ratios and were repeated five times, with results in meansid format.
This pattern persisted with larger models (e.g,, t5-large) and different architectures (e.g., KO- GPT). In a zero-shot context, chatGITT showed similar performance trends despite lacking prior information. Without fine-tuning, chatGPT's performance struggling especially in summarization tasks by generating verbose sentences, leading to low Rouge scores.
pertoraas lagged behind other models,
6.2 Experiment 2: Training with SIKO
To assess the SIKO impact on training, we initially restricted our fine-tuning data to 20,000 instances per task, due to data construction and validation capabilities. As described in Section 4 we generated five SIKO data variants by incorporating 10,, 30%, 50% and 100% of ordinary instances. We fine-tuned models with blend of 20,000 ordinary instances and SIKO data, comparing performance to other augmentation methods to establish its training efficacy. Performance to validate SIKO as an augmentation technique. Following Yoo et al. (2021), this procedure was repeated five times analyze average performance and variability. This verification method was employed to avoid sampling bias.
evaluated on ordinary test set (excluding SIKO modifications)
6.2.1 Baselines: Augmentation
We compared its performance using the following four representative data augmentation methods, with detailed descriptions of these methods documented in the appendix D
Duplicate - -Tiis method simply adds the sampled data to the tuning data without any processing. It serves as a baseline, highlighting performance improvements attributable solely to increased data volume.
Repetition - Introduuced by Wu et al. 2022), this approach duplicates randomly selected word within the data.
EDA Suggested Wei & Zou 2019), this technique involves four probabilistic actions ap- plied to the data: synonym insertion, synonym substitution, random deletion, and position swapping.
AEDA
As presented by Karimi et al. (2021), this method involves the probabilistic insertion of special characters into the data.
8

---

Published as a conference paper at COLM 2024
KLUE-TC
KL.
Dialogue TC
Macro-F1
Acc.
Macro-F1
completeness cases
whole
Complete case marker
Incomplete case marker
Camplete case marker
Incomplete case marker
Complete case marker
Incomplete case marker
rate
100.0
6.6
93.4
100.0
54.3
45.7
100.0
7.7
92.3
baseline
nor-augmented
81.6
94.0
80.7
79.7
79.7
79.6
64.6
68.4
64.3
duplication
82.3
94.0
81.5
82.4
84.8
79.6
68.4
71.5
68.2
repetition
83.5
96.0
82.6
81.4
83.0
69.2
71.7
EDA
82.0
92.2
81.3
81.4
81.7
81.1
68.9
71.5
68.7
STKO
CMad
84.4
98.0
83.4
83.0
83.4
82.6
69.7
70.7
69.7
Shufsem Preere
84.2
98.0
81.8
72.3
Shufgem
84.0
100.0
82.9
82.0
83.9
79.8
69.4
71.9
69.2
Table 5: 50% augmentation test results with T5-base by case marker completeness. CMael method enhances performance on incomplete case marker data.
KLUE-TC
KLUE-NLI
Dialogue TC
Macro-F1
Acc.
Macro-I1
Complete word order
Incomplete word order
Complete order
Incomplete word order
Complete word order
Incomplete word order
completeness
whole
whole
whole
rate
100.0
88.0
12.0
100.0
95.7
4.3
100.0
65.0
35.0
baseline
not-augmenled
81.6
81.3
83.5
79.7
80.7
56.9
64.6
64.9
64-4
duplication
82.3
82,9
78.0
82.4
83.6
56.5
68.4
70.2
67.5
ropetition
74.6
63.3
70.3
EDA
82.0
82.4
78.9
81.4
82.2
65.5
68.9
70.4
68.2
SIKO
CMail 84.4
84.4
84.5
83.0
84.0
60.7
69.7
70.7
69.2
Shufsem Pheerv
84.2
83,9
86.0
81.8
82.4
68.1
70.2
71.1
69.7
ChofsennNon.PPeae
82.0
82.7
69.6
Table 6: 50% Augmentation test results with T5-base by word order completeness. Shuf based methods boost performance on data with altered Word Order.
6.2.2 Result
Table presents our fine-tuning experiment results, highlighting consistent performance improvements with SIKO data in most scenarios. Notably, deleting case markers and altering word order (Shufsem.* dMdee) resulted in inconsistent performance changes, from minor to major improvements, likely from losing syntactic information. Due to the use of syntactically noisy data, overly high SIKO data proportions (e.g., 100% augmentation ratio) led to reduced gains in performance.
Integrating insights from Section we evaluated SIKO's impact on performance. Table details results from 50% augmentation experiment by case marker usage completeness, while Table 6 focuses on sentence order completeness. Models fine-tuned on CMadl data exhibited high performance in handling incomplete case markers. Similarly models adjusted with Shufsem. data demonstrated superior performance in scenarios with altered sentence order. Interestingly, while KLUE-TC showed large performance improvements with perfect case marker usage, this appears to be due to the rare occurrence of such usage (6.6%), suggesting that the performance changes were significantly influenced by few test cases.
Section 3 highlightee the prevalence of syntactically incomplete sentences in Korean, with SIKO effectively enhancing model performance on such sentences. As Section elaborates, strategies like CMdel and Shufsem. Pres. provide cost-efficient data augmentation alter- natives EDA, demonstrating consistent performance improvements and establishing their reliability. To achieve a precise analysis, we focused our attention on test sets with unambiguously defined correct answers, as seen Tables and We excluded the sum- mary test set from this comparison because Rouge score could be influenced by the length or expressiveness of the generated sentence. Furthermore, for a streamlined comparison, omitted mixed-generation methods and AEDA due to its conceptual overlap with EDA. A comprehensive table that includes these excluded components can found in the Appendix C

---

Published as a conference paper at COLM 2024
Furthermore, to evaluate the effects models beyond the T5-base, we ran experiments with 50% augmented data on both the GPT-3 based SKT-Kogpt-trintt model and the T5-large model. While the boost in performance from augmentation was subtle, consistent pattern emerged. More detailed results and analysis are recorded in the appendix C.1
7 Discussion
In this study, we established that Korean's syntactic flexibility are indeed mirrored in LMs. We also demonstrated that data generated via the SIKO method enhances comprehension of ordinary Korean. The scope of our research was intentionally limited to Korean, constrained by our linguistic expertise and the experimental scope. Yet, it's noteworthy that numerous languages, including Japanese, certain Turkic languages, Mongolian, and Hungarian, pos- sess case markers or exhibit flexibility in sentence structure, like Japanese, Russian, and Latin. We anticipate that our findings will significantly benefit LMs developed for these languages.
Our methodology leveraged training data infused with syntactic noise to boost model performance. By experimenting with various rates data augmentation, we ascertained the necessity for moderation in augmentation to avoid detrimental effects. Future studies will aim to pinpoint the optimal augmentation level, evaluate potential drawbacks, and assess the method's 's utility across different tasks.
Conclusion
We sought to determine whether Korean LMs reflect the inherent syntactic flexibility of the Korean language, especially regarding case marker usage and word order. Our initial findings confirmed the frequent use of incomplete case markers and word orders in Korean through publicly available benchmark datasets. To delve deeper, we introduced the Syntac- tically Incomplete Korean (SIKO) dataset. Through SIKO, it became evident that Korean LMs can flexibly handle syntactically incomplete inputs. Fine-tuning with SIKO enhances this capability to process common incomplete syntactic forms in Korean. Moreover, the dataset's low construction cost, coupled with significant performance enhancements, solidifies its standing as an effective data augmentation technique.
Acknowledgement
This work was supported by Institute of Information & communications Technology Plan- ning & Evaluation (ITPP) grant funded by the Korea government(MSIII [No.2022-0-00641, XVoice: Multi-Modas Voice Meta Learning]
This work has supported by the National Research Foundation of Korea(NRF) grant funded by the Korea govemnmen((MSTT/NN. 2022R1F1A1071047),
10

---

Published as a conference paper at COLM 2024
References
Mostafa Abdou, Vinit Ravishankar, Artur Kulmizev, and Anders Sogaard. Word order does matter and shuffled language models know it. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6907-6919, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint, 2020.
Noam Chomsky. Syntactic structures. Mouton de Gruyter, 2002.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entail- ment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Lincertainty Visual Object Classification, and Recognizing Tex- tual Entailment, MLCW"05, pp. 177-190, Berlin, Heidelberg, 2005. Ppringgr-Veelag. ISBN 3540334270, doi: 0..1007711173777 9. URL https://doi.org wt..oor//....//..... 1000///00/00/
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings, 2022.
Akbar Karimi, Leonardo Rossi, and Andrea Prati. Aeda: an easier data augmentation technique for text classification. Xrv preprint ar arXiv:2108.13230 2021.
Diederík P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint ar Xri::1412.698)) 2014.
Ikseob Lee and Hongpin Im. Korean Grammar Theory, volume hakyounsa, Seoul, 1997.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint arXit:1111.05101, 2017.
OpenAT. Chatgpt, 2022. URL http://bbeea.opppa...ooo.m
Available at:
httpp:/ httttss//bbet.opeaa..oomm /sigup//
Dennis Park. pko-t5: Paust korean t5 for text-to-text unified framework, 5 2022. URL https://github. https//b..-..ooeuuusottteta/aott
Sungjoon Park, Jihyung Moon, Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, et al. Klue: Korean language understanding evaluation. arXiv preprint ar arXiv:2105.09680, 2021.
Thang M Pham, Trung Bui, Long Mai, and Anh Nguyen. Out of order: How important is the sequential order of words in sentence in natural language understanding tasks? ar Xiv preprint ar drXiv:2012.15188 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. http: http:///innr.ooo/pppree.ee arggppapers/v21/200-00 html
Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau, and Adina Williams. Unnatural language inference. arXiv preprint Xiv:2110 000000, 2020.
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. Xiv preprint ar arXiv:2104.06644 2021.
Ho-min Sohn. Korean language in culture and society. University of Hawaii press, 2005.
11

---

Published as a conference paper at COLM 2024
Lucy Vanderwende and William B Dolan. What syntax contribute in the entailment task. In Machine Learning Challenges Workshop, pp. 205-216. Springer, 2005.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Xiv preprint ar Xrio:1884.00766) 2018.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arv preprint ar 1rXi::1901.11990 2019.
Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. ESimCSE: Enhanced sample building method for contrastive learning of unsupervised sen- tence embedding. In Proceedings of the 29th International Conference on Computational Linguis- fics, pp. 3898-33833 3907, Gyeongju, Republic of Korea, October 2022. International Committee 2022..00....22 coling-1.34
on Computational Linguistics. URL
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems, 33:6256-6268, 2020. 2020.
Jaehoon Yeon and Lucien Brown. Korean: A comprehensive grammar. Routledge, 2013.
Kang Min Yoo, Dongju Park, Jaewook Kang, Sang-Woo Lee, and Woomyeong Park. GptSmix: Leveraging large-scale language models for text augmentation. arXiv preprint arXit:2104.08826, 2021.
Wangchunshu Zhou, Qifei Li, and Chenle Li. Learning from perturbations: Diverse and informative dialogue generation with inverse adversarial training, ar arviv preprint arXim:2105.1517,, 2021.
12

---

Published as a conference paper at COLM 2024
A Licenses
We list the licenses of each source dataset that we utilized in the creation of SIKO.
KLUE: Creative Commons Attrbaation-SarreAliks 4.0 International License.
Al-hub dialogue: This data is modifiable, but redistribution is not allowed.
B
SIKO and Resources for Construction
KLUE-TC
KLUE-NLI
DIALOGUE
# of total data
45,680
23,000
279,992
# of training data for 4.3.1
43,680
20,998
277,992
# of test data for 4.3.1
2,000
2,000
2,000
# of training data for 4.3.2
20,000
20,000
20,000
# of test data for 4.3.2
2,000
2,000
2,000
average of length
27.37
45.41/24.92
120.62
average of words # of category
6.61
10.74/5.84
27.87
average of speaker number
2.07
average of turns
11.27
average of length in each turn
10.7
average of words in each turn
2.47
Table Benchmark data information
Training Data
Validation Data
Test Data
CMdel
18000
2000
2000
Shufsem. Presrv
18000
2000
2000
Shufsem. Shufsem.Presse Presry & CM
18000
2000
2000
Shufsem.Nee Non. Non..roorv
18000
2000
2000
Shufsem. Coa.sSemm....P... Pre.rresvv & CMdel 18000
2000
2000
Table Generated SIKO Data Statistics
B.1 SIKO statistics
Tables and record the statistics of each benchmark dataset used for this experiment and the stanitics of the generated SIKO data, respectively.
B.2
Evaluation Task Information
B.2.1 KLUE Benchmark
The KLUE (Korean Language Understanding Evaluation) benchmark is a comprehensive suite of tasks designed to evaluate and advance the capabilities of language models specif- ically in understanding the Korean language. It includes variety of tasks such as text classification, named entity recognition, semantic textual similarity, natural language in- ference, and question answering. These tasks cover wide range of linguistic phenomena and challenges that are representative of real-world language use in the Korean context. The creation of KLUE aims to foster the development of AI models that better understand and process the Korean language, contributing to the global progress natural language processing technologies.
13

---

Published as a conference paper at COLM 2024
B.2.2 AI-hub Data
Al-hub is a comprehensive data platform in South Korea, operated under the auspices of the National Information Society Agency (NIA). Its primary mission is to facilitate advancements in artificial intelligence by offering an extensive repository of high-quality datasets. These datasets span wide range of fields, including language, voice, image, and more, catering to diverse Al research and development needs. The platform serves as critical resource for developers, researchers, and businesses engaged in AT, providing them with the necessary data to train, test, and refine their models and applications. By fostering innovation and supporting the Al ecosystem Korea, Al-hub plays a pivotal role in driving the country's technological advancement and competitiveness in the global Al landscape.
B.3 Data Generation and Inspection
B.3.1 Human Inspection
This experiment involved total of three human inspectors. They are individuals contracted on an annual basis by our company to build NLP data, generate natural language data, create tagged corpora, or evaluate model performance. These professionals have over 18 months of experience and hold at least a bachelor's 's degree. The team consisted of two workers and one reviewer, where the workers performed the tasks below, and the reviewer checked and corrected their outputs.
We requested them to undertake the following two tasks:
Review of word order alteration data We constructed draft data for semantically pre- served word order alterations using chatGPT. The workers reviewed the results ensure that 1) the meaning was preserved, and 2) all words from the original sentence were retained. either condition was not met, the workers would reconstruct themselves.
Statistical survey on the use of syntactically incomplete Korean To compile the statistics in T workers checked 1) the case of omitted case markers, 2) the restoration of omitted case markers, 3) the case markers in sentences with incorrect word order, 4) the generation of sentences with correct word order, and 5) the determinability of interpretation.
B.3.2 chatGPT
We utilized chatGPT in constructing semantically preserved word order alteration data. translates the prompt we used into English. Figure Soture@documentt the actual code and version information used. Figure 3 displays the result when the prompt is entered into the Web UT, featuring the same example sentence as in Figure])
Table 9
version
gepplo--ranker-gtäbburre engine' (May 2023 version). Please change the word order of the following Input sentence without altering its meaning, Rach word in the sentence be maintained is.
Provide short
prompt
Input sentence: (DATA]
What is the altered sentence?
ERS 2 9 2
22 of E
# 2 Ho Fh o oj
Korean prompt
4 [DATA]
1 2
Table 9: ChatGPT information. Version Translated Prompt
14

---

Published as a conference paper at COLM 2024
C Additional Experiments
C.1
Augmentation Result using Large Model
Furthermore, to evaluate the effects on models beyond the T5-base, we ran experiments with 50% augmented data on both the GPT-3 based SKT-Kogpt-trinity model and the T5- large model. While the boost in performance from augmentation was subtle, a consistent pattern emerged. When contrasting the outputs of models that were fine-tuned without any data augmentation, models with more parameters consistently surpassed the T5-base. This observation suggests that larger models inherently possess greater syntactic flexibility than their smaller counterparts. The fact that both large and small models exhibit nearly identical performance ceilings when fine-tuned with augmented data supports this interpretation.
C.2
Omitted Experimental Results
Table and table are the full versions of Table and table
D Baseline: Augmentation Methods
Duplicate This method simply adds the sampled data the tuning data without any processing. It serves as baseline, highlighting performance improvements attributable solely to increased data volume.
Repetition Introduced by Wu al. 2022), this approach duplicates a randomly selected word within the data.
EDA Easy Data Augmentation (EDA) is set of straightforward techniques aimed at hancing natural language processing (NLP) models by augmenting textual data. Introduced by Wei & Zou (2019), EDA improves model performance through four operations: synonym replacement, random insertion of synonyms, random swapping of words, and deletion of words in sentences. These methods increase the diversity of training data by slightly altering sentences, thereby helping models become more robust and generalize better. EDA especially valuable for projects with limited labeled data, offering an effective way to expand datasets without significantly altering their original meaning. We used the "koedaff E library to generate EDA and AEDA data, and this library references the Korean WordNet.
AEDA As presented by Karimi et al. (2021), A lighter Easier Data Augmentation, is data augmentation technique for NLP that focuses simplifying and streamlining the augmentation process. Contrary to its predecessor, EDA, which involves synonym replacement, random insertion, swapping, and deletion, AEDA simplifies the augmentation by predominantly using random insertion of punctuation marks into text data. This method aims to enhance model performance by diversifying the training data with minimal changes, thereby maintaining the semantic integrity of the original text while introducing syntactic variations. AEDA's approach is benefícial for improving the robustness of NLP models, particularly in scenarios with limited training data, by offering straightforward and effective means augment text data.
E Implementation Details
E.1 Development Environment
Table 13 shows our development environments. We conducted fine-tuning and generation experiments for syntactic flexibility using an NVIDIA A100 40GB GPU Score. We employed
http:://gginbbb.om///rvvnn/////
15

---

Published as a conference paper at COLM 2024
the Adam optimizer with decoupled weight decay (Kingma & Ba 2014 Loshchilov & Hutter (2017)) and set a learning rate of 5e-5 with a warm-up period spanning 3 epuchs.
BASELINE
SINO
aug
duplication
repetition
Shuffe Fresce CMai
Shulsen-See.Poo CMMl
KLUB-TC
83
8L4
KLUE-NLI
81.
82.5
(..
82.5g30
838
64.723)
60.3
(3:5)
68.
08.61
69.0g7)
6900g.
70.3
(8.3
ROUGF-1
778
782
79.8
79
29-7
78.8pz)
786
ROUGE22
0.5
(2.
(1.
45.
44.7
48.9
45.
45.0
40.0
43.0
10e exparsion
BASELINE
SIKO
aug
duplication
repeition
CM2i
Shufsen
Shufle Precx
KLUB-TT
macro-Ri
0.5
838
858
83.9p
882
(4.
640
83-3g33)
KLUE-NLI
828
826
838
682
69/722
70.0
89,
(2)
780
7
77.8044
78.2
78.9
77.1
788
78.0
ROUGF-2
44.9
41.9pu
44.2
45.2
42.8gg
43.
43.5p4
40-7
44.2
Results 50%
the KO-GPT
Table 10: Results from the 50% augmentation test with T5-large and GPT3, Despite size and structure differences, methods using SIKO data consistently improved performance, echoing the T5-base results.
KLUE-TC
KLUU--III
Dialogue TC
Macro-FI Fl
Acc.
Macro-FI Fl
completeness cases
whole
Complete word order
Incomplete word order
whole
Complete word order
Incomplete word order
whole
Complete word order
Incomplete
word order
rale
100.0
4.3
100.0
65.0
baseline
duplication
82.3
62.9
78.0
82.4
83.6
56.5
68.4
70.2
67.5
repetition
83.5
84.7
74.6
81.4
82.3
63.3
69.2
70.3
68.6
AEDA
81.2
82.3
72.8
60.7
69.1
69.9
EDA
82.0
82.4
78.9
81.4
82.2
65.5
68.9
70.4
68.2
owrs
CMdel
84.4
84.4
84.5
83.0
84.0
60.7
69.7
70.7
69.2
Shuffem.Preee
64.2
63.9
86.0
81.8
82.4
68.1
70.2
71.1
69.7
Shubsem Prerr & CMad
Prere
83.5
83.7
81.8
81.7
82.4
65.5
67.7
69.5
66.7
Shufsem Non.
Shu fuem CMas
83.3
83.2
83.7
81.1
81.7
68.8
69.1
70.5
68.4
Table 11: Table analyzing the performance of models trained on augmented data in relation to word order completeness. The model augmented with Shufsem.Presrr demonstrates superior performance in test cases where word order is incomplete.
16

---

Published as a conference paper at COLM 2024
KLUB-TC
KLUENLI
Dialogue
Macro-F1
Ace.
Macro-FI
completeness
whole
Complete Marker
Incomplete Marker
whole
Complete Marker
Incomplete Marker
whole
Complete Marker
Incomplete Case Marker
rate
100.0
6.5
93.4
100.0
54.3
45.7
100.0
7.7
92.3
baseline
81.6
94,0
8117
79.7
79.7
79.6
64.6
68.4
64.3
duplication
repetitton
83.5
96.0
82.6
81.4
$3.0
79.6
69.2
71.7
69.0
AEDA
81.2
98.0
80.0
81.8
83.6
79.6
69.1
70.6
69.0
EDA
82.0
92.2
81.3
81.4
$1.7
81.1
68.9
71.5
68.7
CMaa
844
98.0
834
83,0
83.4
82.6
69.7
70.7
69.7
Shufsen
84.2
98.0
83.2
81.8
82.1
81.3
70.2
72.3
70.0
Shufser Peeare
CMad
Shufsem Non.Peese Pheare
84.0
100.0
82.9
82.0
83.9
79.8
69.4
71.9
69.2
Shufe
CMdd
83.3
98.0
82.2
81.1
81.1
81.1
69.1
70.5
69.0
Table 12: Table analyzing the performance of models trained on augmented data in relation to case marker completeness. The model augmented with CMael demonstrates superior performance in test cases where case markers are incomplete.
python Ver.
3.8.0
PyTorch Ver.
2.0.0+cul17
transformer Ver.
4.30.1
epoch
batch size
16
learning rate
5.00e-05
Table 13: Development environments
17

---

Published as a conference paper at COLM 2024
messagesteader =..
{"role": "system", "content": "You are a helpful assistant")
def inputMaker(eenttene))
ret []
prompt www.
48 2 2|1iフ म 인 o 말ेक 2 4 Eo コQE 1f
92 2 बा
2
28: 0
2 2
format (sentence)
aeteap ( ("role": "user", "content": "()". format(promppt)
return ret
def wtt..cll.cccrrrp(ttn....
response = openai. www.aaoottteeeeeiooceeeeeee
engine= pogine==deppl "deluyrrankee--ptttttuoo""
messaleseessseeeleedde + turn #t messageTail
return response
Figure 2: ChatGPT prompt and version information
18

---

Published as a conference paper at COLM 2024
0 _ 2| / 인 o म
인 2 = コH _
인 OF Tं
4 E O.CC.
2
E J. _ レ C.
Figure 3: Example of chatGPT prompt usage
19


Reviews:
[{"review_id": "LAc8g9J9Hq", "reviewer": "Reviewer_51wH", "summary": "- This paper investigates the performance of LLMs for incomplete syntactic Korean dataset.\n- This paper finds that Korean LLM exhibits flexibility when dealing with syntactically incomplete inputs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to express our gratitude for your deep understanding and interest in our research.\n\nAs you mentioned, conducting experiments on various languages would have increased the generalizability and value of our research. We have also investigated languages with similar features, such as Japanese (which uses case markers), and languages with relatively free word order like Turkish, Magar, and Kazakh.\n\nHowever, we believed that grammatical knowledge was necessary for preparing and analyzing the experiments, so we limited the scope of our research to our native language, Korean. If given the opportunity, we plan to expand the application of syntactically incomplete data to more languages in future research.\n\nOnce again, thank you for your understanding and valuable advice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The findings in this paper could be generalized to other languages that has similar characteristics, such as Japanese.", "reasons_to_reject": "- Lack of experiments. Specifically, more experimental results with other languages make clearer the generalizability of this paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rnZLi0dqyd", "reviewer": "Reviewer_8Me2", "summary": "The paper analyzes the effect of incomplete syntactical structures in Korean sentences for LLMs. The paper focuses on two forms of incomplete syntax, which are omitting case markers and changing word order. In addition, a new syntactically incomplete Korean dataset (SIKO) was compiled and this dataset is also used for fine-tuning the LLMs. Two Korean LLMs were used. The experiments showed that LLMs are flexible in handling the incomplete syntactic structures.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First, we would like to express our gratitude for your deep understanding and interest in our research.\n\nWe appreciate your point that the deletion of case markers and changes in word order may not be methodologically novel, and we agree to a certain extent. However, we wanted to share with other researchers that even such familiar and simple methods can contribute to performance improvement. We will prepare a comprehensive rewriting to make our research more easily understandable to other researchers.\n\nResponse to Questions:\n\nThank you for pointing out the deficiencies related to formatting. We believe these are crucial elements for enhancing the clarity of our research. We will definitely incorporate the feedback you provided.\n\nThe morphological analyzer we used was developed in-house for the services of the company I am affiliated with. In terms of general performance, it has recorded an F1 score of 98.826 in evaluations conducted on sampled news article sentences.\n\nIt seems we did not provide sufficient explanation when using the term \"ICL.\" This term was used to indicate that the task was performed in a zero-shot manner with no contextual information provided. (In-context learning)\n\nThe human evaluators you mentioned were involved in the tasks described in Section 3 (Table 1) and Section 4.2. In Section 3, they identified the presence of omitted case markers or altered word order in general Korean usage and performed the task of restoring omitted case markers or word order to calculate their scale. In Section 4.2, they reviewed and corrected the meaning-preserved word order change data generated by ChatGPT to ensure that the meaning was indeed preserved.\n\nOnce again, thank you for your understanding and valuable advice.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper analyzes the effect of syntactically incomplete sentences on LLM, which can be regarded as one of the first studies in this area.\n- The paper compiles a new syntactically incomplete Korean dataset.", "reasons_to_reject": "- The paper conducts an experimental analysis, but there is no novelty in methodogical aspects of the models used.\n- Some explanations in the paper are not clear and there are some problems with the language use.", "questions_to_authors": "- Please use \"section\" instead of \"chapter\".\n- Section 4.1: Please specify the accuracy of the morphological analyzer used and also state how much error occurred in the process of stripping out the case markers.\n- \"Mixed Application(\" -> \"Mixed Application (\"\n- Page 6: What is ICL?\n- Section 5.4: \"multiple human-centric tasks such as evaluation, restoration, and correction were conducted.\". What is the reason of these tasks in the paper?\n- \"Two workers\". Not a good phrase. May be \"Two people\" or something like.\n- \"tables 5 and 6\" -> \"Tables 5 and 6\"", "extra_scores": {}}, {"review_id": "KWYbPEwwJT", "reviewer": "Reviewer_tuTs", "summary": "This paper introduces a Korean data set to test out how syntactic variation found in Korean speech and certain common writing styles (dropping of case markers and non-SOV word order) affects LLM tasks.  In addition to the data set itself, the authors bench mark on 3 Korean LLMs across 2 datasets and 4 tasks, looking at fine-tuned and non-fine tuned models.  \n\nAlthough such syntactic variation is common, as the authors show as part of the dataset construction, the LLMs perform much better when fine-tuned with such data than when used out of the box.\n\nAs the authors point out, in addition to providing a dataset for Korean, this type of syntactic variation is common in many languages and so the findings and the approach to creating the dataset are applicable well beyond Korean.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we sincerely thank you for your deep understanding and interest in our research. Due to the rebuttal length limitation (2500 characters), we apologize for not being able to explain each point in detail.\n\n1. Knowledge of Korean Grammar:\nThe information you shared is essential for understanding our research. We will include this in the revised version.\n\n2. Licenses and Usage Permissions:\nAll data used is anonymized and publicly available for research. Dialogue data cannot be redistributed in modified form, so only non-dialogue data from the SIKO dataset will be made public. We will re-examine all licenses and permissions.\n\n3. Additional Information:\nWe will ensure that the revision includes details on the usage of SIKO data, table metrics, cases without tuning, references for Korean grammar, the thesaurus used in EDA, and the Korean version of the prompts used, as you pointed out.\n\n4. Responses to Specific Questions:\nMany questions arose due to our insufficient explanations. We will clarify these points in the revision.\n\n\"Different meanings\": Refers to potential misinterpretation of the author's intent.\n\nOmitted case markers: Investigating the role of omitted case markers is indeed interesting. We will include this in the revision as adding new research results during the review period is prohibited.\n\n\"Including Restore\": We restored all omitted case markers in 300 samples and then investigated the rate of omitted case markers. \"Including Restore\" indicates that the number of case marker positions includes the restored case markers.\n\nCorrection rate unit: Based on words, evaluated by human evaluators with simple and specific guidelines for consistency.\n\nMeaning of \"rshuf\": Refers to \"randomly shuffled data\" mentioned earlier. We missed this detail. Korean word order is relatively free, but the verb position tends to stay fixed. Our changes include cases where the verb is not in the final position, especially in random shuffles and meaning-preserving word order changes (e.g., chat conversations).\n\nVerification method: To avoid sampling bias, we performed five different samplings and expansions for each expansion rate, conducted tuning, and measured performance. We will investigate data variation.\n\nLastly, we deeply appreciate your valuable feedback, which has improved our paper. Your advice included many points we had not considered, providing valuable information for other researchers. We will ensure all suggestions are reflected in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Highlights major issue between standard language and language in actual usage and how this affects LLM-based tasks.\n- Quantifies the issue.\n- Provides straight-forward way to create a data set to fine-tune existing LLMs.\n- Provides the data set.\n- Bench-marks on 3 models, 2 datasets, and 2 tasks.\n- Findings and solution are very broadly applicable outside of English.", "reasons_to_reject": "- Minor rewriting needed to make the Korean facts clearer to those without linguistic knowledge of Korean.\n- Needs proofreading.", "questions_to_authors": "Please proofread the final paper, paying special attention to spacing around parentheses and to determiner placement. Also, references to \"chapter\" should say \"section\".\n\nComments in order of the paper:\n\nMake clear the license of the dataset being made available and whether any code will be made available with it (e.g. code for its creation in case others want to replicate this for another language or evaluation code).\n\nThe overall description of Korean may need to be a bit clearer for those who know nothing about Korean syntax, which will be most of the readers.  Early on make clear (perhaps near where Fig 1 is discussed and repeated in Sec 3) that:\n- Korean is by default SOV\n- The main reorderings are among full phrases (e.g. NP arguments) and occur before the verb\n- Moving the verb earlier is much rarer\n- Splitting apart an NP argument (e.g. fronting an adjectival modifier but not the noun) is much rarer\n- Natural dropping of case markers is more common with subjects than objects (or vice versa or the same, whichever is true)\n\nFor all the tables of results, indicate statistical significance, or lack thereof, between the best system and the next best and between the best system and the baseline.\n\np1: It is not clear what is meant by \"different meanings\".  Is this argument structure different (e.g. if the original Korean was \"dog cat bit\" and the new order was \"cat dog bit\" both with no case markers, then the subject and object are swapped) or discourse structure different (e.g. the first NP will be interpreted as a topic).\n\np2: For the conversational corpus, have the participants all agreed to have their data used? This is about the existing corpus, but it would be good to make clear that all ethical considerations were met in creating the corpora.  To a lesser extent, for the non-conversational corpus, are all copyright issues respected?\n\nTable 1: It would be interesting to know the case marker drop rate for subjects vs objects.  This would be easy to calculate on the 300 sentences and could be added as a footnote.\nIt would be good to know the % of sentences which have both non-canonical word order and case marker dropping (as opposed to one or the other).\nWhat is \"Including Restore\"?\n\nSec 3: when calculating the correction rate, were the words counted or the constituents (e.g. NPs)?  Although more of a hassle to compute, the constituent rate would be more indicative since the paper is really about word order changes of entire constituents such that a single word constituent (\"dog\") counts the same as a complex on (\"large brown dog with spots\").\n\nTable 2: what is \"rshuf\"? Presumably the non-V-final orders are much rarer; this should be mentioned earlier. Also, the example has the object still with the verb, which maintains the VP (if Korean has one, which I realize is a major discussion), was the V-subj-obj order also used?\n\nSec 4: Make clear that created this much data to allow fine-tuning. 22K sentences are not needed for eval.\n\nTable 3: In the caption mention that the metrics are \"Macro F1\", \"Accuracy\" and \"ROUGE\"  (I think ROUGE is spelled with all caps, but please check).\n\nsec 5.3: Mention that you also tried the LMs out of the box without fine-tuning.\n\nsec 6.2: Was this 5-fold cross-validation or simply presenting the data in different orders to see the variants. State briefly (due to space) how much variance there was.\n\nSec 7: Although not required, if you have references for the word order and case marking facts for these languages, include them.  This is, admittedly, all well know in the linguistic community, but less well known to many NLP practitioners.\n\nTable 9: Show the Korean version as well. Perhaps merge this with Fig 3.\n\nApp D: EDA does require a way to get synonyms.  Most languages have this and it is even easier with LLM supported languages, but it does add complexity, especially if the substitution requires inflecting the form.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduced a benchmark in Korean that focuses on syntactic incompleteness in the language; finetuning on this dataset greatly improves the performance of LLMs highlighting the importance of syntax in Korean. Overall, reviewers liked this paper but had some concerns on writing quality and provided many constructive suggestions to help make the paper better. The paper contributes a useful dataset and provides a concrete improvement to language technologies beyond English.

[At least one review was discounted during the decision process due to quality]
Decision: Accept