Analyzing Syntactic Generalization Capacity of Pre-trained Language Models on Japanese Honorific Conversion
Year: 2023

Abstract:
Using Japanese honorifics is challenging because it requires not only knowledge of the grammatical rules but also contextual information, such as social relationships. It remains unclear whether pre-trained large language models (LLMs) can flexibly handle Japanese honorifics like humans. To analyze this, we introduce an honorific conversion task that considers social relationships among people mentioned in a conversation. We construct a Japanese honorifics dataset from problem templates of various sentence structures to investigate the syntactic generalization capacity of GPT-3, one of the leading LLMs, on this task under two settings: fine-tuning and prompt learning. Our results showed that the fine-tuned GPT-3 performed better in a context-aware honorific conversion task than the prompt-based one. The fine-tuned model demonstrated overall syntactic generalizability towards compound honorific sentences, except when tested with the data involving direct speech.

heading: 1 Introduction
text: The correct use of Japanese honorifics is difficult because it requires both the knowledge of grammatical rules (i.e., verb conjugation) and contextual information (i.e., social relationships among the speaker, the hearer, and the people mentioned in a conversation) (Harada, 1976). We expect this syntactic and pragmatic ability for pre-trained large language models (LLMs), as they have shown high performance on natural language tasks (Brown et al., 2020, Ouyang et al., 2022, inter alia). However, it remains unclear whether LLMs can handle Japanese honorifics in a similar manner to humans, based on sentence structures and social context.
Several studies proposed datasets of Japanese honorifics for classification (Liu and Kobayashi, 2022; Someya and Oseki, 2022) and generation (Matsumoto et al., 2022). Liu and Kobayashi
(2022) introduced a task in which a model takes an honorific sentence as input and classifies its honorific level or the types of honorifics used in the sentence. Someya and Oseki (2022) provided a Japanese acceptability classification dataset called JCoLA. In JCoLA, subject honorifics are categorized as sub-categories of subject-verb agreement tasks. However, these datasets aim to evaluate the syntactic performance of language models, and they do not analyze their pragmatic ability to understand honorifics by considering social relationships behind sentences. Matsumoto et al. (2022) introduced an evaluation dataset for an honorific conversion task in which the input was a non-honorific sentence, and the output was an honorific sentence. Matsumoto et al. (2022) mentioned the necessity of considering the information on social relationships among people in honorific conversion but did not clarify how such information should be processed in the task. In summary, the existing benchmark datasets of Japanese honorifics focus on either syntactic or pragmatic knowledge required for honorific understanding, not both (Appendix A). Additionally, none of these existing studies discusses the generalization capacity toward various syntactic structures of honorific sentences.
In this research, we introduce a new honorific conversion task that uses information on person’s social relationships as additional input. In Matsumoto et al. (2022)’s proposed honorific conversion, the input was only a non-honorific sentence. In our task, social relationships are expressed as a sentence and concatenated into an input sentence (Section 2). This enables us to analyze whether LLMs could consider information on social relationships when executing honorific conversion. We also construct a dataset to investigate the syntactic generalizability of LLMs for this honorific task. We create hand-crafted templates and generate problems for the task by filling in the placeholders (Section 3). By focusing on the syntactic generalization
40
capacity, we analyze how flexibly LLMs can apply the grammatical rules of honorifics. Using our dataset, we then fine-tune and evaluate the performance of GPT-3 on the task (Section 4). Additionally, we evaluate the models using zero-shot learning to determine how well these models perform for honorific conversion using the prompt-based method. Our experiments indicate that the finetuned GPT-3 successfully generalizes to sentences with more complex structures, such as scrambling, but not to those involving direct speech. We also
show that the model with prompt learning demonstrates much lower performance than that with finetuning.
Our dataset will be publicly available at https: //github.com/ynklab/japanese_honorifics.

heading: 2 Task Overview
text: Japanese Honorifics Japanese honorifics are based on various linguistic phenomena (Council for Cultural Affairs, 2007; Nihongo Kijutsu-bunpou Kenkūkai, 2009); some have grammatical rules of conjugation. We target Subject Honorifics (SH) and Object Honorifics (OH). As shown in Table 1, these honorifics are applied depending on the grammatical position of sensei (a teacher) so that the speaker can express respect or deference towards the teacher. SH is applied to the predicate when agent has a higher social status than the speaker, and OH is applied when the patient has a higher social status than the speaker.
Honorific Conversion The existing research proposed honorific conversion (Matsumoto et al.,
2022). We extend this task to include sentences explaining social relationships as input. In the upper example of Figure 1, the speaker is talking about supervisor Yamada’s actions, so the verb uketoru (receive) should be converted into the subject honorific form. In the lower example, the speaker and Yamada are in a casual relationship because they are colleagues; therefore, the model should output the same sentence as the input without honorific conjugation.
Syntactic Generalization We focus on the models’ syntactic generalization ability to capture whether models can flexibly use honorific rules. In this paper, syntactic generalization refers to a model’s ability to use honorific rules for not only simple syntactic structures but also complex syntactic structures (see Figure 2).

heading: 3 Dataset Construction
text: We construct a Japanese honorific dataset by manually creating problem templates and filling their placeholders with vocabulary using dictionaries to evaluate LLMs’ performance on the honorific task. Our dataset construction method is shown in Figure 3. We take this approach instead of automatically collecting data from corpora for two reasons. First, it is difficult to create sentence data with complex structures, such as scrambling, from corpora in a controlled manner. This possibly makes it easier for a model to do honorific conversion than when the information is implicit. Second, we need to prepare controlled settings for social relationship information to evaluate whether LLMs utilize it in honorific conversion; however, such information does not appear explicitly in the corpora.
The second problem is related to the fact that words in argument positions are often dropped in Japanese, especially in dialogue sentences. (1) and
(2) display a conversation between a junior worker and their boss.
(1) A boss asks a question to a junior Okashi-tte snack-TOP, mada still nokotteru remain ? ?
‘Are there any snacks left?’
(2) The junior answers ϕi ϕ nokori remained wa TOP itadakimashitai had-OH
‘I/We had them all.’
In this conversation, two ambiguous points must be clarified to determine the honorific relationship behind the conversation. The first point is that the junior answers with object honorifics to show deference in (2), but the target of deference is ambiguous without additional context. If snacks are something the boss originally brought to their office, the boss is the target of the junior’s deference. However, if the snacks are prepared by some third person with a higher rank of job position than the junior, the deference must be towards them instead of the boss. The second point is that the subject is dropped in (2) (pro-drop), but we cannot determine whether ϕ refers to the junior or to a group of workers containing the junior. Considering such language-specific phenomena, we take a templatebased approach instead of a corpus-based approach for dataset construction.

heading: 3.1 Templates
text: We create 39 problem templates based on the literature on Japanese linguistics (Council for Cultural Affairs, 2007; Nihongo Kijutsu-bunpou Kenkūkai, 2009). A graduate student with a linguistic background created the templates by consulting a linguistics researcher. Each problem template has three elements for generating input and output data for honorific conversion: the relationship template, sentence template, and honorific type.
Relationship Templates Relationship templates represent social relationships among a speaker, a person who makes an action (agent), and one who is the target of the action (patient) in an equationlike format. For example, speaker=actor<target means that the speaker and actor do not use honorifics for each other and should use honorifics for the target.
Sentence Templates Sentence templates have placeholders for person’s names and verbs. Based on their structural complexities, we prepare two types of sentence templates: SIMPLE and COMPLEX. SIMPLE is a template that has one clause and S(O)V structure, and COMPLEX is a template that has more complex syntactic structures: scrambling (SC), center embedding (CE), direct speech (DS), and indirect speech (IS). The first two structures change the argument positions within a sentence, potentially posing challenges for the model in capturing subject-verb agreement. The last two are related to honorific application, depending on whether the sentence has quotation marks (brackets) or not (see Appendix B). A COMPLEX template may contain multiple structures (e.g., IS & CE). See Appendix B for further details.

heading: 3.2 Problem Generation
text: We create problem data for training and evaluating models by filling in placeholders of the templates for verbs and person’s names. From the relationship template, context sentences are generated that explain the social relationships between the speaker and the people mentioned in the input sentence. In addition, from the sentence template, we create an incorrect or non-honorific sentence and a correct honorific sentence. The verb conjugates according to the honorific type given when its placeholder is being filled. We used 23 verbs and 19 names in this experiment. We chose the verbs which are commonly used in daily conversation. We also avoid verbs such as nusumu(steal) because honorifics cannot usually be applied to disrespectful actions. Regarding the names of people, we used the 19 most common family names in Japan in 20221. Finally, a set of the following data is generated from each problem template: context sentences, an incorrect or non-honorific sentence, and a correct sentence.
1https://myoji-yurai.net/prefectureRanking. htm

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: We evaluate GPT-3 models on the proposed honorific conversion task under two different experimental settings: fine-tuning and prompt learning. Despite the general expectation of the superior performance of fine-tuning compared to zero-shot prompt learning, no prior research has aimed to evaluate the performance of LLM on honorific conversion in a prompt-based method. Thus, we compare the scores of these two methods to validate whether the same goes true for honorific conversion. For the two settings accordingly, we use davinci (Brown et al., 2020) and text-davinci-003 (Ouyang et al., 2022), which are available in the OpenAI API (see Appendix C for details including hyperparameter settings).
Fine-tuning We fine-tune two models that differ in the training dataset’s size to measure how much data are needed to generalize the problems. SIMPLE_TRAIN is used for training and SIMPLE_TEST and COMPLEX_TEST for evaluation. 3_times is a model trained with 117 problems we prepare by generating three data from each problem template, and in the same way, 7_times is trained with 273 problems. Although our dataset has relatively little data, we consider it enough for the experiments because the minimum dataset size for fine-tuning GPT-3 is “a few hundred.”2 As shown in Figure 1, the input is a concatenation of condition sentences and an incorrect sentence, and the output is a proper honorific sentence.
Prompt Learning GPT-3 is known for zero-shot learning, solving some tasks given only a natural language description as a prompt. In addition to the input text used for fine-tuning, we include a task description in the input prompt (see Appendix E).
Evaluation We manually calculate the percentage of correct sentences generated by a model. In this experiment, we regard the output as correct if the verb conjugates to one of the possible honorific forms. We also ignore mistakes unrelated to verb conjugation (e.g., adding a comma in a natural position). We create test datasets using the same problem templates and vocabulary as the training datasets. SIMPLE_TEST contains 108 examples, and COMPLEX_TEST has 408 examples (see Appendix D).
2https://platform.openai.com/docs/guides/ fine-tuning/preparing-your-dataset

heading: 4.2 Results
text: Table 3 shows the scores under all settings of our experiments on the honorific conversion task. Overall, the fine-tuning scores surpass those of the promptbased method.

heading: 4.2.1 Fine-tuning
text: The scores plummeted when the models were tested with COMPLEX_TEST compared to SIMPLE_TEST. When we increased the data size, the scores increased in most cases, except when tested for problems with direct speech sentences. In Table 2, the model failed to convert a direct speech sentence (Takahashi-san ga “Kimura ga okaerininaru” to ossharu). The verb kaeru should not be in a subject honorific form (okaerininaru) because Takahashi does not use honorifics for Kimura, given their relationships. However, if the brackets (quotation marks in Japanese, see Appendix B) are removed, the sentence (Takahashi-san ga Kimura ga okaerininaru to ossharu) becomes an indirect speech sentence and thus becomes proper honorifics. Based on this characteristic, we suppose that the model applied the same honorific knowledge as indirect speech to direct speech, ignoring the role of brackets.

heading: 4.2.2 Prompt Learning
text: The scores were relatively higher when tested with SIMPLE_TEST than with COMPLEX_TEST, but the scores under all of our settings were lower than 25%. We found that the models transferred non-
honorific sentences to polite forms in almost all cases by simply changing the last letters of the verbs that end -suru into -shimasu instead of applying SH or OH. This conversion is possibly caused by our prompt instructing the models to “convert to the proper honorific sentence,” which may include polite forms too. To validate whether the models use contextual information, we need to construct a prompt that can differentiate SH and OH from polite speech because polite forms are less restricted to social relationships.

heading: 5 Conclusions and Future Work
text: In this paper, we introduced an honorific conversion task that requires not only syntactic knowledge but also pragmatic knowledge, such as social relationships among people. We constructed a Japanese honorific dataset using problem templates created manually and evaluated the syntactic generalization capacity of GPT-3 models on the task using our dataset. The experiments showed that the finetuned models could solve problems with simple structures but failed to generalize to problems with more complex structures, particularly with direct speech. Regardless of the sentence structure, the prompt-based models did not successfully solve the problems with our current prompt setting.
In future work, we plan to expand our dataset to include more diverse Japanese honorific expressions, such as predicates other than verbs or honorific prefixes attached to nouns. For the promptbased experiments, we evaluated the models using zero-shot learning. It would be valuable to test them using few-shot learning by including simple examples in the prompts.
We conducted experiments by explicitly providing information about social relationships. We will also continue to seek data construction methods to extract such information from the corpora, although we did not apply these corpora-based methods in this paper.
Limitations
We discuss two limitations of this research in this section. First, this research focuses on Japanese honorifics with grammatical rules of verb conjugation, which we can judge whether the honorific conversion is correct based on social relationships and sentence structures created in a controlled manner. Japanese honorifics have more expressions based on linguistic phenomena that we did not include in our templates, such as noun honorifics (e.g., ofutagata, a polite and formal way of saying “the two people”). Creating templates for noun honorifics requires more detailed settings because they are based on information on context other than social relationships. Second, GPT-3 is the only language model evaluated on our honorific conversion task. This research aims to analyze how capable the well-known, high-performing GPT-3 is of generalizing Japanese honorific sentences and not to explore which existing LLM can achieve the best performance in honorific conversion.

heading: Acknowledgements
text: We thank three anonymous reviewers for their helpful comments and suggestions. We would also like to thank Anirudh Reddy Kondapally, Tomoki Sugimoto, Tomoya Kurosawa, and the other laboratory members for their helpful advice. This work was supported by JSPS KAKENHI Grant Number JP20K19868.

heading: A Existing Datasets
text: Table 4 shows examples of existing Japanese honorifics datasets.

heading: B Templates
text: Table 5 shows examples of templates we created. Table 6 shows examples of indirect and direct speech in Japanese.

heading: C Model Details
text: davinci is the largest model among the ones provided for fine-tuning, and text-davinci-003 is trained by reinforcement learning on human feedback and aimed at being used with prompt learning.
Hyperparameters For fine-tuning GPT-3, n_epochs is set to 2. For text generation, we set max_tokens to 50 and temperature to 0.
Pre-training Data of GPT-3 GPT-3 can input and output Japanese texts because some of its pre-training datasets (Common Crawl, WebText) contain Japanese texts, although the proportion of Japanese texts is not clarified.

heading: D Test Dataset
text: Within the COMPLEX_TEST dataset, 156 data have center embedding, 252 for scrambling, 160 for indirect speech, and 160 for direct speech. Scrambling and center embedding can not appear in one problem data; the same goes for indirect and direct speech.

heading: E Prompt Example
text: Figure 4 shows an example of our prompt used for prompt learning.
以下の文はあなたの発言です。人物間の敬語の 条件を踏まえて、敬語が不十分かそれらを誤って 使っている場合は正しい敬語に変換してくださ い。 (The following sentence is your speech. Given the condition of usage of honorifics between people, convert the sentence to the proper honorific one if it contains wrong or insufficient honorifics.) === 敬語の条件:あなたは田中に敬語を使います。 (Condition: You use honorifics for Tanaka.) 田中が受け取る (Tanaka receives) ->
Correct output: 田中がお受け取りになる (Tanaka receives-SH)
Figure 4: An example of the prompt used for zero-shot learning. The bold text is a task description.

heading: Future Work
text: In future work, we plan to expand our dataset to include more diverse Japanese honorific expressions, such as predicates other than verbs or honorific prefixes attached to nouns. For the promptbased experiments, we evaluated the models using zero-shot learning. It would be valuable to test them using few-shot learning by including simple examples in the prompts. We conducted experiments by explicitly providing information about social relationships. We will also continue to seek data construction methods to extract such information from the corpora, although we did not apply these corpora-based methods in this paper. Limitations
We discuss two limitations of this research in this section. First, this research focuses on Japanese honorifics with grammatical rules of verb conjugation, which we can judge whether the honorific conversion is correct based on social relationships and sentence structures created in a controlled manner. Japanese honorifics have more expressions based on linguistic phenomena that we did not include in our templates, such as noun honorifics (e. g. , ofutagata, a polite and formal way of saying “the two people”). Creating templates for noun honorifics requires more detailed settings because they are based on information on context other than social relationships. Second, GPT-3 is the only language model evaluated on our honorific conversion task. This research aims to analyze how capable the well-known, high-performing GPT-3 is of generalizing Japanese honorific sentences and not to explore which existing LLM can achieve the best performance in honorific conversion