Generating Dialog Responses with Specified Grammatical Items for Second Language Learning
Year: 2023

Abstract:
This paper proposes a new second language learning task of generating a response including specified grammatical items. We consider two approaches: 1) fine-tuning a pre-trained language model (DialoGPT) by reinforcement learning and 2) providing a few-shot prompt to a large language model (GPT-3). For reinforcement learning, we examine combinations of three reward functions that consider grammatical items, diversity, and fluency. Our experiments confirm that both approaches can generate responses including the specified grammatical items and that it is crucial to consider fluency rather than diversity as the reward function.

heading: 1 Introduction
text: The use of dialog systems for language learning has attracted attention. Many studies have introduced dialog systems as training partners for language learners and verified their effectiveness. According to previous studies (Kim, 2016; Tegos et al., 2014; Ruan et al., 2019), the advantages of using dialog systems in language education include: they can be used regardless of time, i.e., are more available for learners, they can be easily integrated into chatbased applications that many people are familiar with, i.e., are more user-friendly, and they can be adapted to each learner using various information from chit-chat, i.e., are more supportive.
Needless to say, experiencing a substantial amount of production is critical in language acquisition. Nagata et al. (2020) showed that even a very primitive rule-based chatbot like ELIZA has the potential to increase learner’s sentence production. Their experiments also revealed that learners adopted words that appeared in the chatbot’s responses, suggesting that the expressions used by the dialog system had a positive impact on learners and that the system was effective in helping them learn unfamiliar words.
Considering these results, we propose a task of generating a response including the specified grammatical items. Here, grammatical items refer to such as to the present perfect, subjunctive, and relative clauses. Usually, they are gradually covered in a language learning course, typically through a school curriculum. Such responses can naturally expose learners to a variety of uses of a specific item and can give them experience of how to use the item in a variety of topics and situations, based on their own past experiences evoked in the conversation. In turn, we expect the learners to use the exposed constructions in their own production more as the exposed uses are linked tightly to their memories by encountering usage examples through dialog based on their own experiences.
The proposed task is formalized as follows. Given C = [c1, c2, ..., cn], a dialog context that is a sequence of n utterances between two interlocutors (the system and the learner), and I , a set of grammatical items specified to be included, the task is to generate r, a natural response that follows cn, on the condition that r includes an expression corresponding to each item i ∈ I . To the best of our knowledge, this is the first work to tackle this generation task for language learning.
To generate text that satisfies particular conditions, Lin et al. (2021) propose using auxiliary modules to guide pre-trained language models. Keskar et al. (2019) propose training language models with control code. Since these methods are based on supervised learning, they require annotated datasets. However, there is a lack of large labeled dialog datasets for grammatical items.
In this paper, we examine two approaches for generating responses containing the specified grammatical items without a large labeled dataset: 1) RL-based generation: fine-tuning a pre-trained language model using reinforcement learning (RL), and 2) Prompt-based generation: providing a large language model with prompt text with a task in-
184
struction and a few examples. The experiments confirm both approaches are promising.

heading: 2 Related Work

heading: 2.1 Dialog systems and adaptation in language learning
text: According to Xiao et al. (2023), there are three main uses for dialog systems in language learning.
One way is language learning through general communication. As one of the educational applications of dialog systems, there is a growing body of research on introducing dialog systems in second language learning through free interaction with dialog systems. Alexa (Moussalli and Cardoso, 2020; Dizon, 2017; Dizon and Tang, 2020) and Google Assistant (Tai, 2022) were used. In most studies, learners favorably accepted the system as a dialog partner.
Another way is task-based language learning. The introduction of a dialog system into a task allows for more content-focused learning. Tasks can be varied, such as asking for the time of day at a particular location or ordering at a coffee shop (Wu et al., 2020; Timpe-Laughlin et al., 2020). Learners are allowed to interact and receive feedback throughout the task, which contributes to second language acquisition.
The third way is language learning based on structured pre-programmed dialog. To create a dialog on a specific topic, researchers design their system, rather than adapting a general dialog system. Many studies have been conducted with children. Some had three to six-year-olds learn to read through questions (Xu et al., 2021a,b), and had nine-year-olds answer their questions (Lee and Jeon, 2022). Another related survey is (Huang et al., 2022).
A further related area is user adaptation to difficulty in language tutoring. Pandarova et al. (2019) worked on predicting the difficulty of fill-in-theblank questions in which the words to be entered were specified.
Our study proposes a new task not addressed in these studies and provides new insights into methods for this task.

heading: 2.2 Reinforcement Learning
text: Reinforcement learning is a machine learning framework that acquires an optimal action policy based on non-instantaneous evaluations given by a reward function for a set of actions. By considering
the output tokens as actions, language generation can be treated as a reinforcement learning problem. Given an appropriate reward function, policy gradient methods such as REINFORCE (Williams, 1992) can fine-tune a pre-trained generative neural language model without a training dataset. In this paper, we adopt self-critical sequence training (SCST) (Rennie et al., 2017). SCST is proposed for image caption generation and is known for its simplicity and effectiveness.
The design of the reward function varies from task to task, but unlike the loss function in supervised learning, it allows the use of nondifferentiable functions including the evaluation metrics used in text generation tasks such as BLEU and ROUGE (Paulus et al., 2018; Wu et al., 2018; Narasimhan et al., 2016).
Language generation based on deep learning generally uses cross-entropy as the loss function, which means that the objective function and the evaluation measure will be different. By incorporating the evaluation measure in the reward function, the gap can be alleviated.

heading: 2.3 Large-scale Pre-trained Language Model
text: In recent years, many researchers have studied methods for controlling the output of generative language models by providing prompts containing task instructions and examples as input (Li et al., 2022; Reynolds and McDonell, 2021; Dou et al., 2022).
In particular, GPT-3 (Brown et al., 2020) has achieved significant performance comparable to or better than other fine-tuned models in CoQA and TriviaQA in few-shot settings.

heading: 3 Method
text: For the sake of simplicity, in this paper, we assume context C contains only the immediately previous utterance (n = 1). We also limit the number of specified items to 1 (|I| = 1).

heading: 3.1 RL-based generation
text: For simplicity again, we train a different model for each grammatical item. In applications, we assume the models are to be switched given a learner’s need. For example, when a learning partner chatbot finds that the learner tends to make errors with a particular item, the chatbot can increase the frequency of opting the generation model for the item than the vanilla generation model.
We consider three sub-functions for the reward, Rg for inclusion of grammatical items, which is the main objective, Rd for greater diversity, and Rf for higher fluency. The latter two are to mitigate learning bias towards including grammatical items. When only Rg is used, the model easily starts to exploit a fixed utterance against any input context. We will examine several combinations of these functions in our experiment in the next section.
Reward on grammatical items Let Fi(s) ∈ [0, 1] be a soft classifier that evaluates whether a given sentence s contains a specified grammatical item i. When we train a response generation model for item i, we set Rg(s) = Fi(s).
For Fi(s), we use BERT (Devlin et al., 2019). We obtain hidden representation h[CLS] of the [CLS] token from the final layer of a pre-trained BERT model. Fi(s) is formulated as follows: Fi(s) = σ(w
⊤h[CLS] + b), where σ() is the sigmoid function and w, b are the learnable parameters. In training, the BERT model is not frozen and fine-tuned together with the parameters.
Although Fi(s) is trained in a supervised manner, the necessary data for this training is much more affordable than that for training a generation model. We will revisit this point in the next section.
Rewards on diversity and fluency We use Distinct-N (Li et al., 2016), an n-gram based diversity metric, as Rd. As Rf , we use the likelihood of the output r conditioned on the input, i.e., the dialog context C. The likelihood is computed by a pre-trained dialog model.

heading: 3.2 Prompt-based generation
text: In the same way with the RL-based approach, we prepare a prompt template for each item i. The templates are to be switched by applications.
Figure 1 shows a prompt template used in this study, which consists of an instruction indicating what the task is, some examples (called shots) and a query at the end. < c > in Figure 1 is replaced with an input context utterance. Given an input prompt, a left-to-right generative language model outputs a sentence r that follows the prompt.

heading: 4 Experiment
text: We verified the effectiveness of both RL-based and prompt-based approaches using three items in the SCoRE corpus (Chujo et al., 2015): the present perfect, relational clause, and subjunctive.

heading: 4.1 Datasets
text: In accordance with the assumption of n = 1, we extracted only the first utterance pair of each dialog from the Daily Dialog corpus1 (Li et al., 2017) to compose our dataset. The first utterance of each pair was used as a context C, and the second was used as a reference (used for analysis purposes). We split the pairs into three subsets: 10,618 for training, 500 for development, and 1,000 for test.
We used the SCoRE corpus to build Fi(·). We built a classifier for each of the three items above. Appendix A gives the details of the SCoRE dataset, classifier training, and performance. Note that the required data for training here need not be dialog data and can be much smaller than that for supervised training of a dialog language model.

heading: 4.2 Evaluation metrics
text: We used three metrics for our evaluation. First, we defined the function δi(s), which returns 1 or 0 for sentence s by using Fi(s) with a threshold of 0.5.
As the first metric, we introduced G-ratio to measure the capability of the model to generate responses that include the specified grammatical item. G-Ratio indicates the percentage of outputs containing the item and can be automatically measured by using δi(s).
Considering our aim of exposing learners to various uses of grammatical items in dialog, the model should be able to return diverse responses. We adopted Distinct-N (N=2) as the second metric.
Finally, we defined GOAL (Grammar Oriented Average Likelihood), which measures the fluency of only the generated sentences that contain the specified item using the output likelihood based on
1https://huggingface.co/datasets/daily_dialog
a dialog language model Pm as follows:
HTi = {s ∈ GTi |δ(s) = 1},
GOAL(HTi ;Pm) =
∑ s∈HTi Pm(s|c(s))
|HTi | ,
where GTi the set of the generated responses given test set T in terms of item i, and HTi is the set of responses in GTi that Fi(·) evaluated as containing the grammatical item. c(s) denotes the input context for output s.

heading: 4.3 Experimental setups
text: For the RL-based approach, we used DialoGPT (Zhang et al., 2020), a GPT-2 based dialog language model trained on a Reddit corpus, as the initial model in SCST, the main body of Rf , and Pm. For decoding, we used top-k sampling (Fan et al., 2018) (k = 50). The model was evaluated every 10 batches using the development data, and training was stopped with a patience of 3. As training progressed , the number of sentences containing the target grammatical item increased, but many similar sentences were generated, resulting in a loss of diversity. Therefore, as we observed a tradeoff between G-Ratio and diversity, we adopted the product of the two as an indicator of early stopping.
For the prompt-based approach, we used GPT-3 davinci. We set the sampling temperature to 1 for GPT-3. Other settings are detailed in Appendix B.

heading: 4.4 Evaluation
text: For the RL-based approach, ten sentences were generated using beam search with a beam width of 10 for each test case. Out of the ten, the sentence with the highest likelihood and the specified item is chosen as the output. If no sentence included the item, the first one was chosen. We compare the following five combinations of the reward functions: Rg, Rg +Rd, Rg ×Rd, Rg +Rf , and Rg ×Rf .
For the prompt-based approach, ten sentences were generated thorough the web API using a prompt for each test case, from which one was picked as above. We compared the following five variations, which combines 0, 1, and 3 task examples (called shots) and with/without task instructions: instr., 1-shot, 3-shots, instr.+1-shot, and instr.+3-shots. For example, “instr.” means 0-shot with instructions. “1-shot“ means 1-shot without instructions. “instr.+3-shot“ means 3-shot with instructions.
All metrics were applied to 1,000 outputs.

heading: 5 Results
text: Table 1 shows the results for each grammatical item. Example outputs are shown in Appendix C. Rg × Rf showed the highest GOAL for the present perfect and the subjunctive, while Rg +Rf showed the highest GOAL for the relative clause.
The RL-based approach successfully improved G-Ratio in all cases. Although the Dist.-2 values got lower than before training (Baseline), this was expected in advance as the result of introducing a grammatical constraint in generation.
In the RL-based approach, a higher Dist.-2 tended to be obtained with the fluency reward function Rf than with the diversity reward function Rd except for the subjunctive, suggesting that the effect of Rd was limited. The reasons for this may be as follows. Even if sentences with a high Dist.-2 are more likely to be generated, it does not necessarily reflect the diversity of the model overall, and if the input sentences in the batch are similar, Dist.-2 in the output will naturally decrease, but the current reward function does not fully take this into account. In addition, taking fluency into account suppresses the abuse of fixed patterns (fixed patterns increase Rg but decrease diversity). For all grammatical items tested, GOAL improved when the reward function for fluency, Rf , was applied.
In the prompt-based approach, G-Ratio tended to be higher for inputs with both task instruction and shots. However, 3-shots sometimes gave worse results than 1-shot. This suggests that task instructions should be included in the input, but that increasing the number of shots may add noise or unintended bias to the language model, making it more difficult to obtain the desired output.
Comparing the two approaches, the promptbased one demonstrated higher diversity than the RL-based one, and a comparable G-Ratio. Though the GOAL scores for the RL-based approach were higher than those for the prompt-based approach, we must note that GOAL is favorable to the RLbased approach that, in this paper, uses the same DialoGPT model as GOAL. As far as we manually compared the concrete responses from GPT-3 and DialoGPT for a small number of randomly picked cases, we did not find significant differences.

heading: 6 Discussion
text: Even though we want to expose more instances of a particular item to a learner, it is not natural to include the item in every dialog response. Therefore,
we do not need to pursue 100% for G-Ratio. We presented GOAL as a primary metric candidate for the proposed task. However, as noted in the previous section, it is not reliable when one wants to compare two results based on different language models. Taking the similarity to the reference sentences into account is one direction to mitigate this issue. Another strategy is combining GOAL with reference-free unsupervised dialog evaluation methods using follow-ups such as FULL (De Bruyn et al., 2022). Unlike GOAL, these evaluation methods do not measure the likelihood of the target utterances directly; they, however, still rely on a particular language model. A simple way to make this issue easier would be an ensemble approach using multiple language models or majority voting.
Considering the high diversity and the nature of training-free, so far the prompt-based approach seems to be advantageous, assuming the availability of a huge pre-trained model such as GPT-3. However, the RL-based approach may have merits in terms of its fine-grained, delicate, and implicit control than the prompt-based approach. (Besides, DialoGPT and GPT-2 did not work in the promptbased approach. See Appendix C.)

heading: 7 Conclusion
text: We have proposed a new task of generating a response including the specified grammatical items for language learners. We examined two approaches and found that both are feasible.
Future directions include the expansion of the grammatical items. To push this task to practical use, locating appropriate places in conversations to include the items is also important.
This paper aimed to increase learners’ exposure to specific grammatical items, but another inter-
esting direction is generating preceding utterances that encourage or facilitate learners to use specific grammatical items in their next utterances.

heading: A Classifier for grammatical items
text: We used a classifier that determines whether a grammatical item is included or not as a reward function for RL. The structure of the classifier is as described in §3.1, where the input sentences to be judged are estimated to determine whether they contain grammatical items or not by a linear layer and a sigmoid function based on the embedding of BERT’s [CLS] tokens.
The classifier requires a dataset for training. However, the required data need not be interactive, and can be smaller than for supervised learning of a language model. When data is not available, regular expression-based classification can be used as a substitute.
In this section, we describe the dataset used to train the classifier and the settings. The performance of the classifier is compared with rule-based classification using regular expressions. The regular expressions were created on the basis of the CEFR-J regular expression list (Ishii and Tono, 2018).
A.1 SCoRE Corpus
The SCoRE corpus, in which grammatical items are manually assigned to sentences, was used to train the classifier. Therefore, the grammatical items were those included in the SCoRE corpus. The SCoRE corpus contains approximately 20 grammatical items, and Table 2 shows the number of data corresponding to the grammatical items used in this study. For example, in the subjunctive, I wish, if I were, if + verb past tense, if + had + verb past participle, etc. are included in the data.
In addition to positive examples with the target grammar item, negative examples without the item are required to train the classifier. Therefore, for the negative examples, we use sentences in the SCoRE corpus that are assigned grammatical items that are not the target ones.However, if all sentences that do not have the target grammar item are used as negative examples, there is a possibility that unsuitable data will be included, and the proportion of unsuitable data will be greatly biased. We constructed a dataset for training by extracting data from the negative examples in the dataset in such a way that there is no bias in the number of positive examples.
From the data obtained, 80% was split into training data and 20% into test data. Finally, for the
present perfect, the training data and test data were 1,222 and 306, respectively, and for the relational clause and hypothetical, the training data and test data were 1,977 and 495, respectively.
A.2 Hyperparameter for training the classifier
We used BERT (bert-large-uncased) to set the initial values for the classification model. Parameters were optimized by AdamW during training. The learning rate was set to 2e−5 and the coefficient of L2 regularization to 1e−2. The batch size was set to 10 and the number of epochs was set to 10. In this experiment, the classifier is the model that performed best on the test data.
A.3 Classification Performance Table 3 shows the classification performance of the classifiers for each grammar item. The evaluation was conducted using the percentage of correct answers between the correct and predicted labels as the evaluation measure. In the experiment, the BERT-based classifier was used as the reward function for the other items because BERT had better classification performance than the regular expression.

heading: B Hyperparameter in the experiment
text: In top-k sampling in SCST, we set k to 50. For Distinct-N in Rd, N = 2. The parameters were optimized by AdamW during training, with a learning rate of 2e−5 and a coefficient of L2 regularization of 1e−2. The minimum output length was set to 10 in order to properly compute Distinct-N. The batch size was set to 10, with a maximum of 1100 iterations. For GPT-3, we set engine to davinci, max_tokens to 20, temperature to 1, n to 10, and stop to "\n".

heading: C Examples
text: In this section, we provide generated sentences of compared methods. First, we discuss additional smaller models we experimented with in addition to the GPT-3. Next, we show samples of outputs for two inputs for several RL-based and prompt-based methods.
C.1 Other Models in the Prompt-based Approach
We also tested the performance of GPT-2 and DialoGPT in the same settings as GPT-3. Table 4 shows the results. Comparing the performance of the three models in terms of G-Ratio, GPT-3, which has the largest model size, shows the best performance, while GPT-2 tends to perform better than DialoGPT. In GOAL, GPT-3 showed consistently high, but DialoGPT also showed high values in some settings. Note, however, that DialoGPT was used in the GOAL calculations and is a favorable indicator for this model. Also, GPT-2 and DialoGPT did not seem to produce higher quality responses than GPT-3, as far as we could visually confirm. (See Appendix C.2) Therefore, GPT-3 is superior to the other models in terms of both the G-Ratio and GOAL value, regardless of the grammatical items, and in terms of the quality of the response sentences.
C.2 Samples Table 5, 6 show examples of output in the present perfect tense with different input contexts. Compared with the Daily Dialog corpus and DialoGPT, after learning, the response sentences are in the present perfect tense, and the responses of the method that performed well in our experiments are not too broken to be used as a dialog response. However, some of the methods showed unstable output, such as repetition of similar sentences or very few words.

heading: I ’ve never

heading: I have not

heading: Future Work
text: Future directions include the expansion of the grammatical items. To push this task to practical use, locating appropriate places in conversations to include the items is also important. This paper aimed to increase learners’ exposure to specific grammatical items, but another inter-
esting direction is generating preceding utterances that encourage or facilitate learners to use specific grammatical items in their next utterances