Does Incomplete Syntax Influence Korean Language Model? Focusing on Word Order and Case Markers

Authors: jong myoung kim
young-jun  lee
yong-jin  han
sangkeun  jung
ho-jin  choi
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-21 06:57:59

Abstract: Syntactic elements, such as word order and case markers, are fundamental in natural language processing. Recent studies show that syntactic information boosts language model performance and offers clues for people to understand their learning mechanisms. Unlike languages with a fixed word order such as English, Korean allows for varied word sequences, despite its canonical structure, due to case markers that indicate the functions of sentence components. This study explores whether Korean language models can accurately capture this flexibility. We note that incomplete word orders and omitted case markers frequently appear in ordinary Korean communication. To investigate this further, we introduce the Syntactically Incomplete Korean (SIKO) dataset. Through SIKO, we assessed Korean language models’ flexibility with incomplete syntax and confirmed the dataset’s training value. Results indicate these models reflect Korean’s inherent flexibility, accurately handling incomplete inputs. Moreover, fine-tuning with SIKO enhances the ability to handle common incomplete Korean syntactic forms. The dataset’s simple construction process, coupled with significant performance enhancements, solidifies its standing as an effective data augmentation technique. The SIKO will become accessible post-publication.

Main paper: 
 introduction: 
 Syntactic information is crucial for humans to understand sentence meanings accurately (Chomsky, 2002) . One prominent syntactic element, word order, has been a central topic in recent Language Model (LM) research. Studies have explored various aspects of word order: its significance to LMs (Pham et al., 2020; Sinha et al., 2020) , methods to improve model performance via word order adjustments (Sinha et al., 2021) , and insights into how LMs progressively learn human language (Abdou et al., 2022) .Korean follows a canonical Subject-Object-Verb (S-O-V) word order. However, the presence of case markers, a type of postposition, allows Korean to exhibit relatively flexible word order (Lee & Im, 1997; Yeon & Brown, 2013; Sohn, 2005) . Case markers denote the grammatical roles of constituents within a sentence, like their function as a subject or object. Conversely, due to its canonical word order, Korean often experiences omissions of case markers. Our aim is to verify whether this flexibility is reflected in Korean LMs.Figuree 1 exemplifies the flexibility of a Korean sentence, "나비가 꿀을 마신다," in terms of word order and case markers. Even with word order changes (Arrow a), the case markers '가' and '을' identify '나비' and '꿀' as subject and object, respectively. Conversely, in S-O-V order (Arrow b), subject and object are identifiable by position, even without case markers. Of course, the freedom from case markers and word order comes with limitations. Sentences lacking case markers or with rearranged word order might appear unnatural or have the potential to be interpreted differently from the original sentence. More detailed information about Korean word order and structure is as follows:• Korean typically follows an SOV (Subject-Object-Verb) order.• Major reorderings occur among full phrases (e.g., noun phrase arguments) and usually take place before the verb. • Moving the verb to an earlier position is much rarer.• Splitting a noun phrase argument (e.g., placing an adjectival modifier before the noun) is also much rarer. • Naturally dropping case markers is more common with subjects than with objects (or vice versa, depending on the context). We aimed to determine if the syntactic flexibility inherent to the Korean language is mirrored in Korean LMs. Leveraging these unique linguistic characteristics, our objective was to improve the performance of Korean LMs. To this end, we established a methodology for generating Syntactically Incomplete Korean (SIKO) data and subsequently constructed a comprehensive dataset. Further, we evaluated the effectiveness of the SIKO dataset as a resource for fine-tuning LMs, focusing on its practical applicability in enhancing language processing capabilities.We designed various experiments using the SIKO data. We employed four fundamental tasks used for assessing LM performance: Text Classification (TC), Natural Language Inference (NLI), dialogue topic classification, and dialogue summarization. For the experiments, we used the well-established KLUE benchmark (Park et al., 2021) and the AI-hub data2 released by national authorities. This compilation includes a wide range of Korean language expressions, from news headlines to conversational texts, thereby expanding our experimental scope to cover the extensive usage of Korean. Additionally, we examined the characteristics and performance across three different Large Language Models (LLMs) with varying sizes and structures, to uncover both commonalities and unique aspects in processing syntactic phenomena.In our study, we observed that expressions featuring syntactic incompleteness are prevalent in real-world Korean language applications. We were able to confirm through inference experiments using SIKO data that LLMs are responding to such syntactic incompleteness. Our analysis reveals that the SIKO dataset surpasses representative data augmentation methods in efficacy, especially for fine-tuning purposes. Significantly, it demonstrates superior capability in improving the handling of inputs that are syntactically incomplete, a common phenomenon in typical Korean usage.The contributions of our study are as follows:• This study is the first to explore syntactically incomplete Korean sentences.• We constructed the Syntactically Incomplete Korean (SIKO) dataset.• Through downstream testing with SIKO, we verified that the Korean LLM exhibits flexibility when dealing with syntactically incomplete inputs.• Models tuned using SIKO demonstrated improved performance in handling incomplete data commonly found in Korean. 
related works: 
 Syntactic Information In neural network-based Natural Language Processing (NLP) research, syntactic information has been a consistently explored topic. Further emphasizing its significance, Vanderwende & Dolan (2005) addressed a considerable number of problems in the PASCAL RTE challenge using solely syntactic information (Dagan et al., 2005) . The question of whether these LMs genuinely understand the syntax of human language and align their learning outcomes with human comprehension remains a subject of continued interest (Sinha et al., 2020) .Word Order Word order is a prominent element of syntactic information. While research asserting the lack of importance of word order in input data has been presented (Pham et al., 2020) , consistent efforts are also being made in studies attempting to enhance performance through training on data with disrupted word order (Sinha et al., 2021) . Furthermore, endeavors persist to comprehend how LMs learn human language by utilizing models trained on data with shuffled corpus (Abdou et al., 2022) . Korean, while possessing a formal word order, exhibits relative flexibility from it due to the presence of case markers (Lee & Im, 1997) . We seek to determine whether the Korean LM also reflects this flexibility.Recent research has demonstrated success in enhancing model robustness and performance by utilizing noisy or incomplete data. Consistency learning, as performed by Zhou et al. (2021) with perturbed data and Gao et al. (2022) with dropout-applied incomplete data, is one approach. Data augmentation methods, such as the repetition method employed by Wu et al. (2022) and the replacement of low-information words proposed by Xie et al. (2020) , have also shown promise. General and simple data augmentation methods like Easy data augmentation (EDA) or An easier data augmentation (AEDA) have already become widely used techniques (Wei & Zou, 2019; Karimi et al., 2021) . We aimed to augment data by considering the flexibility of word order and case markers as a form of noise. 
syntactic flexibility in korean: 
 We explored the frequency of case marker omissions and word order changes in practical Korean usage. For this investigation, we took 1,000 samples each from two tasks within the KLUE benchmark and conversational data from AI-hub. We assessed the presence of omitted case markers and deviations in word order, and determined whether such phenomena led to ambiguity or challenges in interpreting the meaning of the sentences.Additionally, we restored omitted case markers and corrected word order in 300 samples for each task. This process allowed us to identify the number of case marker positions, assess omission rates across datasets, and evaluate the scale of word order modifications.The findings are showcased in Table 1 . All tasks described in this section were manually performed, and details about the individuals involved are provided in the Experiment Settings Section ( 5.4).The rate of case marker omission was calculated by comparing the number of omitted case markers to the total number of case markers after their restoration. The word order correction rate was determined by taking the word-level edit distance between the original data and the corrected word order data, then dividing by the total word count in the data.From this analysis, we ascertained that the omission of case markers and alterations in word order are common in ordinary Korean usage. Specifically, case marker omission occurred in 97.4% of KLUE-TC instances and non-canonical word orders were found in 35% of AI-hub dialogues. The scale of these phenomena is underscored by their extensive occurrence: an average omission rate of 80.1% for case markers in KLUE-TC and an average word order correction rate of 45.35% in KLUE-TC.Notably, even with such syntactical incompleteness, the sentences largely retained their clarity of meaning. In dialogue data, there were instances where interpretation was challenged due to case marker omissions. However, it's important to note that these were extracted from conversations, potentially lacking some contextual or background information. This investigation reaffirms that while Korean showcases syntactic flexibility, it remains bounded to ensure semantic clarity. 
siko: syntactic incomplete korean dataset: 
 In Section 3, we demonstrated Korean's extensive syntactic flexibility and its frequent occurrence in actual use. We aimed to determine if Korean LMs reflect this variability. To this end, we generated datasets with deliberate syntactic incompleteness. This approach occasionally led to reductions in syntactic detail and interpretability.We randomly sampled 22,000 instances from three data types (KLUE-TC, KLUE-NLI, AIhubdialogue) and developed five variants of SIKO data. Of these, 2,000 instances (and their SIKO derivatives) served as the test set in Section 6.1, while 20,000 (with their SIKO derivatives) were used for fine-tuning in Section 6.2. Additional details on data statistics, prompts, and reviewer information are documented in Appendix B.We removed all case markers from the data to thoroughly eliminate the syntactic information they provide. While partial omission of case markers is common in Korean, complete removal is rare and generally limited to specific contexts, such as news headlines. We crafted this data by analyzing sentence structures with a morphological analyzer and stripping out the components that correspond to case markers.Morphological analysis is a fundamental task in Korean NLP and is crucial for Korean EDA. The method of removing case markers is as cost-efficient as EDA in terms of data generation. The morphological analyzer we used was developed in-house for the services of the company I am affiliated with. In terms of general performance, it has recorded an F1 score of 98.826 in evaluations conducted on sampled news article sentences.As mentioned earlier, Korean permits certain variations in word order without altering the meaning. We changed the word order in our dataset while preserving the sentence's meaning. These sentences may not conform to typical Korean patterns. Initial reordering was conducted by ChatGPT, followed by human reviewers ensuring semantic integrity.4.3 Semantic Non-Preserving Shuffling (Shuf Sem.Non.Presrv )We generated data by shuffling word order using Python's Random library, a method distinct from EDA's simple two-word swap, as it involves rearranging words across the entire sentence. This process may diverge significantly from typical Korean structures, potentially compromising syntactic integrity and original meaning.4.4 Mixed Application (Shuf Sem.Presrv &CM del and Shuf Sem.Non.Presrv &CM del )We created datasets where we simultaneously applied word order changes and case marker deletion. We employed two distinct methods to change the word order, and then removed case markers from the altered data. This led to the most pronounced reduction in syntactic information throughout our experiments. 
klue benchmark: 
 The KLUE benchmark, inspired by GLUE (Wang et al., 2018) , stands as a premier dataset in Korea. For our study on syntactically incomplete data, we omitted tasks that the answer resides within the input text, such as machine reading comprehension, due to the challenges of uniformly altering word order in both questions and answers. Thus, we selected two tasks from the available eight. The KLUE dataset comprises Training, Validation, and Test segments; our study solely employed the Training subset with accessible labels.The aim of this task is to identify the subject or theme from a given text snippet. The dataset consists of 45,680 news headline titles. It's common for Korean news headlines to minimize the use of case markers. We classified the news headlines into one of seven categories based on their content: 'politics', 'economy', 'society', 'culture', world', 'IT/science', or 'sports'.The objective of NLI is to determine the relationship between a premise and a hypothesis. The dataset, formulated in standard literary style, consists of 23,000 entries. Based on a given premise, an NLI model categorizes the hypothesis as either entailment (true), contradiction (false), or neutral (undeterminable).The AI-Hub dataset, known for its variety in dialogue types such as daily conversations and debates, consists of 279,992 dialogues primarily in the colloquial style typical of messaging platforms. Dialogues usually feature 2 to 5 participants and average 11.27 exchanges per conversation, with each dialogue annotated with a topic and summary.Text Classification (Dial-TC) This task involves classifying the topic of a given conversation into one of nine categories: 'Personal and Relationships', 'Beauty and Health', 'Commerce (Shopping)', 'Current Affairs and Education', 'Food and Drink', 'Leisure', 'Professions', 'Residential and Living', and 'Events'.This task requires generating a summary of a provided conversation, utilizing the same data as the TC task.The PKO-T5 (Park, 2022) , released by PAUST, is a T5 (Raffel et al., 2020) based Korean sequence-to-sequence model using Byte Pair Encoding (BPE) for out-of-vocabulary words. It's trained on sources like Namuewiki, Wikipedia, and Modu Corpus for T5's unsupervised task. PAUST offers three model sizes: small, base, and large. For our experiments, we primarily used the base model and examined the effects of scaling up to the large model.The Ko-GPT-Trinity3 , developed by SK telecom, adapts the GPT-3 architecture (Brown et al., 2020) . It was trained on the extensive "ko-DAT" dataset, comprising 35 billion tokens over 72,000 iterations, using a masked autoregressive LM approach with cross-entropy loss for evaluation.We examined its syntactic flexibility in a causal LM framework, where the model's size, like our selection of T5-large, was crucial.chatGPT ChatGPT (OpenAI, 2022) by OpenAI is a GPT variant tailored for coherent conversational responses. Sourced from a vast internet corpus, it excels in dialogue generation across multiple languages. We employed this model for generating semantically preserved word order alteration data and measuring zero-shot performance on syntactically incomplete inputs. In-Context Learning(ICL) was not used; the tasks were performed purely in zero-shot mode. The prompts and detailed information used are recorded in the Appendix B.3.2.We conducted fine-tuning and generation experiments for syntactic flexibility using an NVIDIA A100 40GB GPU. We employed the Adam optimizer with decoupled weight decay ( Kingma & Ba (2014); Loshchilov & Hutter (2017) ) and set a learning rate of 5e-5 with a warm-up period spanning 3 epochs. Some of the experiments were conducted in a zero-shot manner, without any tuning or training. Detailed library versions and additional information are available in the Appendix E.1. Table 3 : When SIKO data is input into a model pre-trained or fine-tuned on standard Korean data, we observe the processing outcomes. We report scores for SIKO test cases and their variation from standard Korean cases as score diff . The performance for TC was measured using Macro-F1, for NLI using Accuracy, and for summarization using ROUGE-1 and ROUGE-2.In this study, multiple human-centric tasks such as evaluation, restoration, and correction were conducted. In Section 3, they identified the presence of omitted case markers or altered word order in general Korean usage and performed the task of restoring omitted case markers or word order to calculate their scale. In Section 4.2, they reviewed and corrected the meaning-preserved word order change data generated by ChatGPT to ensure that the meaning was indeed preserved. Two people directly handled these tasks. Subsequent to their efforts, another individual, termed the reviewer, carried out checks and verifications.All participants brought to the table a minimum of 18 months' experience in creating and evaluating data for NLP. More detailed information on the task guidelines, training methods, and other aspects provided to these individuals is documented in the Appendix B.3.1. 
experiment 1: the influence of incomplete syntax in inference: 
 We explored how LMs process inputs missing case markers or with non-standard word orders, selecting 2,000 samples from four tasks' training data as our test set. Using the SIKO approach from Section 4, we created five types of syntactically incomplete test sets. The remaining ordinary data was divided in a 9:1 ratio for training and validation during fine-tuning. For TC and NLI tasks with class labels, our sampling preserved original class distributions. We evaluated the inference performance of a LM fine-tuned on typical data with syntactically incomplete test set. We also attempted zero-shot inference with chatGPT to note processing differences. (Unfortunately, T5 and KO-GPT were unable to correctly analyze the task under zero-shot conditions. Therefore, we compared the inference performance using fine-tuned models.) Table 4 : Data tuning results using SIKO data were compared to representative augmentation strategies across transformer architectures for downstream performance. Tests varied augmentation ratios and were repeated five times, with results in mean std format.This pattern persisted with larger models (e.g., t5-large) and different architectures (e.g., KO-GPT). In a zero-shot context, chatGPT showed similar performance trends despite lacking prior information. Without fine-tuning, chatGPT's performance lagged behind other models, struggling especially in summarization tasks by generating verbose sentences, leading to low Rouge scores.To assess the SIKO impact on training, we initially restricted our fine-tuning data to 20,000 instances per task, due to data construction and validation capabilities. As described in Section 4, we generated five SIKO data variants by incorporating 10%, 30%, 50%, and 100% of ordinary instances. We fine-tuned models with a blend of 20,000 ordinary instances and SIKO data, comparing performance to other augmentation methods to establish its training efficacy. Performance was evaluated on a ordinary test set (excluding SIKO modifications) to validate SIKO as an augmentation technique. Following Yoo et al. (2021) , this procedure was repeated five times to analyze average performance and variability. This verification method was employed to avoid sampling bias.We compared its performance using the following four representative data augmentation methods, with detailed descriptions of these methods documented in the appendix D.Duplicate -This method simply adds the sampled data to the tuning data without any processing. It serves as a baseline, highlighting performance improvements attributable solely to increased data volume.Repetition -Introduced by Wu et al. (2022) , this approach duplicates a randomly selected word within the data.EDA -Suggested by Wei & Zou (2019) , this technique involves four probabilistic actions applied to the data: synonym insertion, synonym substitution, random deletion, and position swapping.AEDA -As presented by Karimi et al. (2021) , this method involves the probabilistic insertion of special characters into the data. Integrating insights from Section 3, we evaluated SIKO's impact on performance. Table 5 details results from a 50% augmentation experiment by case marker usage completeness, while Table 6 focuses on sentence order completeness. Models fine-tuned on CM del data exhibited high performance in handling incomplete case markers. Similarly, models adjusted with Shuf Sem. data demonstrated superior performance in scenarios with altered sentence order. Interestingly, while KLUE-TC showed large performance improvements with perfect case marker usage, this appears to be due to the rare occurrence of such usage (6.6%), suggesting that the performance changes were significantly influenced by a few test cases.Section 3 highlighted the prevalence of syntactically incomplete sentences in Korean, with SIKO effectively enhancing model performance on such sentences. As Section 4 elaborates, strategies like CM del and Shuf Sem.Non.Presrv provide cost-efficient data augmentation alternatives to EDA, demonstrating consistent performance improvements and establishing their reliability. To achieve a precise analysis, we focused our attention on test sets with unambiguously defined correct answers, as seen in Tables 5 and 6 . We excluded the summary test set from this comparison because Rouge score could be influenced by the length or expressiveness of the generated sentence. Furthermore, for a streamlined comparison, we omitted mixed-generation methods and AEDA due to its conceptual overlap with EDA. A comprehensive table that includes these excluded components can be found in the Appendix C.2.Furthermore, to evaluate the effects on models beyond the T5-base, we ran experiments with 50% augmented data on both the GPT-3 based SKT-Kogpt-trinity model and the T5-large model. While the boost in performance from augmentation was subtle, a consistent pattern emerged. More detailed results and analysis are recorded in the appendix C.1. 
discussion: 
 In this study, we established that Korean's syntactic flexibility are indeed mirrored in LMs.We also demonstrated that data generated via the SIKO method enhances comprehension of ordinary Korean. The scope of our research was intentionally limited to Korean, constrained by our linguistic expertise and the experimental scope. Yet, it's noteworthy that numerous languages, including Japanese, certain Turkic languages, Mongolian, and Hungarian, possess case markers or exhibit flexibility in sentence structure, like Japanese, Russian, and Latin. We anticipate that our findings will significantly benefit LMs developed for these languages.Our methodology leveraged training data infused with syntactic noise to boost model performance. By experimenting with various rates of data augmentation, we ascertained the necessity for moderation in augmentation to avoid detrimental effects. Future studies will aim to pinpoint the optimal augmentation level, evaluate potential drawbacks, and assess the method's utility across different tasks. 
conclusion: 
 We sought to determine whether Korean LMs reflect the inherent syntactic flexibility of the Korean language, especially regarding case marker usage and word order. Our initial findings confirmed the frequent use of incomplete case markers and word orders in Korean through publicly available benchmark datasets. To delve deeper, we introduced the Syntactically Incomplete Korean (SIKO) dataset. Through SIKO, it became evident that Korean LMs can flexibly handle syntactically incomplete inputs. Fine-tuning with SIKO enhances this capability to process common incomplete syntactic forms in Korean. Moreover, the dataset's low construction cost, coupled with significant performance enhancements, solidifies its standing as an effective data augmentation technique. 
 
 Appendix: We list the licenses of each source dataset that we utilized in the creation of SIKO.• KLUE: Creative Commons Attribution-ShareAlike 4.0 International License.• AI-hub dialogue: This data is modifiable, but redistribution is not allowed. The KLUE (Korean Language Understanding Evaluation) benchmark is a comprehensive suite of tasks designed to evaluate and advance the capabilities of language models specifically in understanding the Korean language. It includes a variety of tasks such as text classification, named entity recognition, semantic textual similarity, natural language inference, and question answering. These tasks cover a wide range of linguistic phenomena and challenges that are representative of real-world language use in the Korean context. The creation of KLUE aims to foster the development of AI models that better understand and process the Korean language, contributing to the global progress in natural language processing technologies.AI-hub is a comprehensive data platform in South Korea, operated under the auspices of the National Information Society Agency (NIA). Its primary mission is to facilitate advancements in artificial intelligence by offering an extensive repository of high-quality datasets. These datasets span a wide range of fields, including language, voice, image, and more, catering to diverse AI research and development needs. The platform serves as a critical resource for developers, researchers, and businesses engaged in AI, providing them with the necessary data to train, test, and refine their models and applications. By fostering innovation and supporting the AI ecosystem in Korea, AI-hub plays a pivotal role in driving the country's technological advancement and competitiveness in the global AI landscape.This experiment involved a total of three human inspectors. They are individuals contracted on an annual basis by our company to build NLP data, generate natural language data, create tagged corpora, or evaluate model performance. These professionals have over 18 months of experience and hold at least a bachelor's degree. The team consisted of two workers and one reviewer, where the workers performed the tasks below, and the reviewer checked and corrected their outputs.We requested them to undertake the following two tasks:Review of word order alteration data : We constructed draft data for semantically preserved word order alterations using chatGPT. The workers reviewed the results to ensure that 1) the meaning was preserved, and 2) all words from the original sentence were retained. If either condition was not met, the workers would reconstruct it themselves.Statistical survey on the use of syntactically incomplete Korean : To compile the statistics in Table 1 , workers checked 1) the case of omitted case markers, 2) the restoration of omitted case markers, 3) the case markers in sentences with incorrect word order, 4) the generation of sentences with correct word order, and 5) the determinability of interpretation.We utilized chatGPT in constructing semantically preserved word order alteration data. Table 9 translates the prompt we used into English. Figure 2 documents the actual code and version information used. Figure 3 displays the result when the prompt is entered into the Web UI, featuring the same example sentence as in Figure 1 . prompt Please change the word order of the following Input sentence without altering its meaning. Each word in the sentence be maintained as is.Provide a short answer. C.1 Augmentation Result using Large Model Furthermore, to evaluate the effects on models beyond the T5-base, we ran experiments with 50% augmented data on both the GPT-3 based SKT-Kogpt-trinity model and the T5large model. While the boost in performance from augmentation was subtle, a consistent pattern emerged. When contrasting the outputs of models that were fine-tuned without any data augmentation, models with more parameters consistently surpassed the T5-base. This observation suggests that larger models inherently possess greater syntactic flexibility than their smaller counterparts. The fact that both large and small models exhibit nearly identical performance ceilings when fine-tuned with augmented data supports this interpretation.Table 12 and table 11 are the full versions of Table 5 and table 6 .Duplicate -This method simply adds the sampled data to the tuning data without any processing. It serves as a baseline, highlighting performance improvements attributable solely to increased data volume.Repetition -Introduced by Wu et al. (2022) , this approach duplicates a randomly selected word within the data.EDA -Easy Data Augmentation (EDA) is a set of straightforward techniques aimed at enhancing natural language processing (NLP) models by augmenting textual data. Introduced by Wei & Zou (2019) , EDA improves model performance through four operations: synonym replacement, random insertion of synonyms, random swapping of words, and random deletion of words in sentences. These methods increase the diversity of training data by slightly altering sentences, thereby helping models become more robust and generalize better. EDA is especially valuable for projects with limited labeled data, offering an effective way to expand datasets without significantly altering their original meaning. We used the 'koeda4 ' library to generate EDA and AEDA data, and this library references the Korean WordNet.AEDA -As presented by Karimi et al. (2021) , A lighter Easier Data Augmentation, is a data augmentation technique for NLP that focuses on simplifying and streamlining the augmentation process. Contrary to its predecessor, EDA, which involves synonym replacement, random insertion, swapping, and deletion, AEDA simplifies the augmentation by predominantly using random insertion of punctuation marks into text data. This method aims to enhance model performance by diversifying the training data with minimal changes, thereby maintaining the semantic integrity of the original text while introducing syntactic variations. AEDA's approach is beneficial for improving the robustness of NLP models, particularly in scenarios with limited training data, by offering a straightforward and effective means to augment text data.Table 13 shows our development environments. We conducted fine-tuning and generation experiments for syntactic flexibility using an NVIDIA A100 40GB GPU 8core. We employed the Adam optimizer with decoupled weight decay ( Kingma & Ba (2014); Loshchilov & Hutter (2017) ) and set a learning rate of 5e-5 with a warm-up period spanning 3 epochs. It is a data platform operated by the Korea National Information Society Agency (NIA). It collects and processes artificial intelligence data in six fields and makes it publicly available. https://www. aihub.or.kr/https://www.sktelecom.com/https://github.com/toriving/KoEDA 


Decision: This paper introduced a benchmark in Korean that focuses on syntactic incompleteness in the language; finetuning on this dataset greatly improves the performance of LLMs highlighting the importance of syntax in Korean. Overall, reviewers liked this paper but had some concerns on writing quality and provided many constructive suggestions to help make the paper better. The paper contributes a useful dataset and provides a concrete improvement to language technologies beyond English.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: - This paper investigates the performance of LLMs for incomplete syntactic Korean dataset.
- This paper finds that Korean LLM exhibits flexibility when dealing with syntactically incomplete inputs.
strength_weakness: strength_weakness: - Lack of experiments. Specifically, more experimental results with other languages make clearer the generalizability of this paper.
paper_summary: paper_summary: The paper analyzes the effect of incomplete syntactical structures in Korean sentences for LLMs. The paper focuses on two forms of incomplete syntax, which are omitting case markers and changing word order. In addition, a new syntactically incomplete Korean dataset (SIKO) was compiled and this dataset is also used for fine-tuning the LLMs. Two Korean LLMs were used. The experiments showed that LLMs are flexible in handling the incomplete syntactic structures.
strength_weakness: strength_weakness: - The paper conducts an experimental analysis, but there is no novelty in methodogical aspects of the models used.
- Some explanations in the paper are not clear and there are some problems with the language use.
questions: questions: - Please use "section" instead of "chapter".
- Section 4.1: Please specify the accuracy of the morphological analyzer used and also state how much error occurred in the process of stripping out the case markers.
- "Mixed Application(" -> "Mixed Application ("
- Page 6: What is ICL?
- Section 5.4: "multiple human-centric tasks such as evaluation, restoration, and correction were conducted.". What is the reason of these tasks in the paper?
- "Two workers". Not a good phrase. May be "Two people" or something like.
- "tables 5 and 6" -> "Tables 5 and 6"
paper_summary: paper_summary: This paper introduces a Korean data set to test out how syntactic variation found in Korean speech and certain common writing styles (dropping of case markers and non-SOV word order) affects LLM tasks.  In addition to the data set itself, the authors bench mark on 3 Korean LLMs across 2 datasets and 4 tasks, looking at fine-tuned and non-fine tuned models.  

Although such syntactic variation is common, as the authors show as part of the dataset construction, the LLMs perform much better when fine-tuned with such data than when used out of the box.

As the authors point out, in addition to providing a dataset for Korean, this type of syntactic variation is common in many languages and so the findings and the approach to creating the dataset are applicable well beyond Korean.
strength_weakness: strength_weakness: - Minor rewriting needed to make the Korean facts clearer to those without linguistic knowledge of Korean.
- Needs proofreading.
questions: questions: Please proofread the final paper, paying special attention to spacing around parentheses and to determiner placement. Also, references to "chapter" should say "section".

Comments in order of the paper:

Make clear the license of the dataset being made available and whether any code will be made available with it (e.g. code for its creation in case others want to replicate this for another language or evaluation code).

The overall description of Korean may need to be a bit clearer for those who know nothing about Korean syntax, which will be most of the readers.  Early on make clear (perhaps near where Fig 1 is discussed and repeated in Sec 3) that:
- Korean is by default SOV
- The main reorderings are among full phrases (e.g. NP arguments) and occur before the verb
- Moving the verb earlier is much rarer
- Splitting apart an NP argument (e.g. fronting an adjectival modifier but not the noun) is much rarer
- Natural dropping of case markers is more common with subjects than objects (or vice versa or the same, whichever is true)

For all the tables of results, indicate statistical significance, or lack thereof, between the best system and the next best and between the best system and the baseline.

p1: It is not clear what is meant by "different meanings".  Is this argument structure different (e.g. if the original Korean was "dog cat bit" and the new order was "cat dog bit" both with no case markers, then the subject and object are swapped) or discourse structure different (e.g. the first NP will be interpreted as a topic).

p2: For the conversational corpus, have the participants all agreed to have their data used? This is about the existing corpus, but it would be good to make clear that all ethical considerations were met in creating the corpora.  To a lesser extent, for the non-conversational corpus, are all copyright issues respected?

Table 1: It would be interesting to know the case marker drop rate for subjects vs objects.  This would be easy to calculate on the 300 sentences and could be added as a footnote.
It would be good to know the % of sentences which have both non-canonical word order and case marker dropping (as opposed to one or the other).
What is "Including Restore"?

Sec 3: when calculating the correction rate, were the words counted or the constituents (e.g. NPs)?  Although more of a hassle to compute, the constituent rate would be more indicative since the paper is really about word order changes of entire constituents such that a single word constituent ("dog") counts the same as a complex on ("large brown dog with spots").

Table 2: what is "rshuf"? Presumably the non-V-final orders are much rarer; this should be mentioned earlier. Also, the example has the object still with the verb, which maintains the VP (if Korean has one, which I realize is a major discussion), was the V-subj-obj order also used?

Sec 4: Make clear that created this much data to allow fine-tuning. 22K sentences are not needed for eval.

Table 3: In the caption mention that the metrics are "Macro F1", "Accuracy" and "ROUGE"  (I think ROUGE is spelled with all caps, but please check).

sec 5.3: Mention that you also tried the LMs out of the box without fine-tuning.

sec 6.2: Was this 5-fold cross-validation or simply presenting the data in different orders to see the variants. State briefly (due to space) how much variance there was.

Sec 7: Although not required, if you have references for the word order and case marking facts for these languages, include them.  This is, admittedly, all well know in the linguistic community, but less well known to many NLP practitioners.

Table 9: Show the Korean version as well. Perhaps merge this with Fig 3.

App D: EDA does require a way to get synonyms.  Most languages have this and it is even easier with LLM supported languages, but it does add complexity, especially if the substitution requires inflecting the form.

Comments: comment: We deeply appreciate your profound understanding and encouragement.
Dear Reviewer,

We deeply appreciate your valuable feedback and suggestions, which have provided us with an opportunity to thoroughly review and improve our research. Your encouragement has also been a great source of strength for us.

Thank you very much.
Thank you for your comment.

I agree that the experiment with a single language, Korean, is suffice as a first-step research paper.
I hope that you and followers of this paper will add experimental results through the experiments with other languages in the future.
I thank the authors for the clarifications.