## 4 Experiment and Discussion
### Designing and Evaluating Experimental Models
The simulated conversation was used for model training with reinforcement learning, which performs better as the length of the conversation increases. However, the model quickly converged at turn 5. Although BLEU and ROUGE scores are widely used for evaluating dialogue quality, they are mainly designed for comparing individual sentences rather than entire dialogues, and can become noisy if the data comparison is too long. Moreover, in reinforcement learning, the reward function reflects the success of a model in achieving its goals. Our two reward functions are based only on word embeddings and not on the \begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline  & **BLEU 1** & **BLEU 2** & **BLEU 3** & **BLEU 4** & **ROUGE Precision** & **ROUGE Recall** & **ROUGE Emeasure** \\ \hline
**BERT (Baseline)** & 0.452 & 0.328 & 0.269 & 0.207 & 0.315 & 0.279 & 0.279 \\ \hline
**BERT LC** & & & & & & \\
**(only left-context)** & 0.458 & 0.345 & 0.287 & 0.226 & 0.332 & 0.302 & 0.301 \\ \hline
**BERT RC** & & & & & & \\
**(only right-context with 1 turn)** & 0.466 & 0.357 & 0.297 & 0.235 & 0.337 & 0.313 & 0.309 \\ \hline
**BERT FC** & & & & & & & \\
**(bi-directional context with 1 turn)** & 0.488 & 0.367 & 0.308 & 0.246 & 0.345 & 0.325 & 0.320 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Summarization results of different models. sentence level. Thus, they may not be strong enough to capture contextual information in long conversations. We recognize this as a reason why RL algorithms may not be effective beyond a certain threshold, which is determined by the length of the simulated dialogue (MDP). We also compare the proposed methods with the relevant current methods (shown in Table 4). HRED [55] is a hierarchical model based on the encoder-decoder. The model used previous responses and encoded all the past information to generate a probable next token. Their experiments demonstrate that their model improves upon previous models. [PERSON] [56] build their model using emotion states and capturing expressions from predefined emotions. They show that their proposed model can generate contextually and emotionally appropriate responses. Recently, a pre-trained dialogue generation model, PLATO [57], has been used based on hidden vectors to determine the inherent features. The model used attention mechanisms to combine contextual information and characteristic features. It showed an improvement in comparison with previous studies. \begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **BLEU 1** & **BLEU 2** & **BLEU 3** & **BLEU 4** & **ROUGE Precision** & **ROUGE Recall** & **ROUGE Fmeasure** \\ \hline BERT FC (**1 turn**) & 0.488 & 0.367 & 0.308 & 0.246 & 0.345 & 0.325 & 0.320 \\ \hline BERT FC (**3 turns**) & 0.492 & 0.374 & 0.316 & 0.256 & 0.362 & 0.336 & 0.334 \\ \hline BERT FC (**5 turns**) & 0.518 & 0.403 & 0.348 & 0.289 & 0.389 & 0.371 & 0.366 \\ \hline BERT FC (**7 turns**) & 0.500 & 0.390 & 0.337 & 0.281 & 0.381 & 0.366 & 0.362 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Model performance using different lengths of simulated conversation. Figure 5: Chart showing the performance of experiments measured by BLEU score (**a**) and ROUGE score (**b**). Figure 6: Chart showing the performance with different lengths of simulated conversation measured by BLEU score (**a**) and ROUGE score (**b**). These current studies are shown in Table 4. Our proposed model showed a significant improvement, especially in higher BLEU score indices such as BLEU2, BLEU3, and BLEU4. This indicates that the use of context in our model helped generate more accurate and coherent responses. Thus, our proposed model can potentially improve the quality of automated conversation systems. Moreover, these results demonstrate superior performance compared to those of previous studies. While the BLEU index decreases significantly for earlier models in these studies as we compare BLEU2, BLEU3, and BLEU4 to BLEU1, our model still produces results that are not too far off.