# CGM: Copy Mechanism GPT with Mask for Ellipsis and Anaphora Resolution in Dialogue
## 4 Experimental Results
### Experimental Results and Analysis
#### 4.3.2 IT Domain Utterance Restoration Experiments  
Table 4 shows the performance results. We conducted experiments in a single domain to verify the restoration effect of the proposed method. We compared the performance by changing each element. Using the resolved information, \(R\), we achieved higher performance compared with using the simple history \(R^{-}\), and adding special characters to distinguish each sentence's improved performance across all metrics. Additionally, considering BERTScore and the SimCSE score confirmed that the resolved sentences effectively preserved the original intent expressed by the user. In particular, the CGM-based model that applied the copying mechanism achieved higher performance in BLEU and ROUGE than the baseline, as it operates by directly copying the input tokens for output. In the restaurant reservation data as well, although the proposed method showed slightly lower performance in terms of \begin{table}
\begin{tabular}{l c c c c c c c c c c} \hline \hline  & & \multicolumn{3}{c}{**IT**} & \multicolumn{3}{c}{**Restaurant Reservation**} \\ \hline
**Model** & **BLEU** & **Rouge-L** & **F1** & **simeCSE** & **BERTScore** & **BLEU** & **Rouge-L** & **F1** & **simeCSE** & **BERTScore** \\ \hline \(R^{-}P\) & 0.3677 & 0.2435 & 0.0371 & 0.872 & 0.8832 & 0.8017 & 0.3482 & 0.1433 & 0.9598 & 0.9551 \\ \(RP\) & 0.3894 & 0.3776 & 0.1118 & 0.890 & 0.8972 & 0.8016 & 0.3827 & 0.1251 & 0.9577 & 0.9547 \\ \hline Baseline & 0.4180 & 0.3622 & 0.1254 & 0.8776 & 0.9657 & **0.8222** & 0.4177 & 0.1457 & 0.9646 & **0.9629** \\ PC-CPT & 0.4315 & 0.6340 & 0.4204 & 0.8985 & 0.9072 & 0.8218 & 0.4443 & 0.1867 & **0.9674** & 0.9636 \\ \hline CGM & **0.4319** & **0.4599** & 0.2066 & **0.9040** & 0.9070 & 0.8216 & 0.4276 & 0.7148 & 0.9638 & 0.9580 \\ CGM + HardPrompt B & 0.4260 & 0.4335 & **0.2087** & 0.9004 & 0.9048 & 0.8160 & **0.4473** & **0.3077** & 0.9626 & 0.9578 \\ CGM + HardPrompt B & 0.4260 & 0.4295 & 0.1985 & 0.8997 & 0.9081 & 0.8157 & 0.4373 & 0.1695 & 0.9604 & 0.9558 \\ CGM + 0/1 Mask only & 0.4216 & 0.3238 & 0.0011 & 0.8699 & **0.9006** & 0.8178 & 0.4100 & 0.1424 & 0.9644 & 0.9614 \\ CGM + Sep/Nice & 0.4158 & 0.4348 & 0.1782 & 0.9007 & 0.9039 & 0.8108 & 0.4343 & 0.1560 & 0.9646 & 0.9675 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Experimental result of utterance restoration in a single domain. Bold represents the best performance in each evaluation indicator. sentence fluency and intent preservation, it demonstrated high performance in Rouge-L and F1, which indicate restoration capability. Table 5 presents examples of each model's result. In the current utterance, \(P_{i}\), \"But what if there's an accident because of the robot?\" the virtual product 'AI delivery robot' was substituted with 'robot', yet it can be observed that the 'AI delivery robot' was effectively restored by referencing the previous history, \(R_{i}\). Table 6 shows the resolution performance for each utterance type. In most performance metrics, CGM-based models applying the copy mechanism recorded high performance. For anaphora and ellipsis types, which require accurately copying necessary keywords from previous time points, CGM showed relatively high performance by copying even detailed keywords, as shown in Table 7, unlike other CGM-based models that only copy higher-level concept keywords. While other models simply connected the product name and the current question, \(P_{i}\), and failed to replace 'the first product' with the product name, CGM was able to find 'the first product' from the answer, \(R_{i}\), and correctly restore it. Furthermore, as shown in Table 8, CGM can accurately convey the exact meaning of the user's utterance by copying only 'card payments', excluding other unnecessary words. On the other hand, for the 'Others' type, CGM recorded lower performance compared with other models. This is because, in the current copy mechanism structure for GPT, it is not possible to assign concentrated weights to specific parts of the given input information, so it failed to find the correct parts to copy.