# CGM: Copy Mechanism GPT with Mask for Ellipsis and Anaphora Resolution in Dialogue
## 1 Introduction  
GPT (Generative Pre-trained Transformer) has achieved great success in the field of natural language processing. This model demonstrates exceptional performance in various text generation tasks, producing innovative results in several applications such as machine translation, text summarization, and question answering. The success of GPT has become a significant milestone showcasing the potential of large language models (LLMs).  
However, in environments where LLMs cannot be used, sLLMs (small LLMs) in the form of GPT often struggle to achieve desired results with zero-shot learning alone [1]. We aim to introduce a copying mechanism to GPT, which is a method that preserves keywords found in the input while having limited training data. As far as we know, encoder-decoder transformer models like T5 [2] have achieved excellent results in applications requiring specific contexts by using a copying mechanism, but there have been no instances of its application in GPT.  
We validated the capabilities of the copying mechanism for GPT through tasks involving ellipsis and anaphora resolution in conversation. Ellipsis and anaphora processing are among the challenging problems in natural language processing. In human conversations, as shown in Table 1, information is often explicitly omitted, or substitutes are used based on already shared context, avoiding the repetition of previous dialogue content. In Table 1, User1 uses anaphora ('\(\mathbf{\mathcal{I}}\) (it)') at the end of the conversation. While humans can recognize omitted information as the original information through contextual clues in such phenomena, it is very difficult for machines to interpret this appropriately [3].  
Previous studies have addressed ellipsis and anaphora resolution in ways similar to coreference resolution [4] or machine reading comprehension [5]. However, in a question-and-answer environment with a mixture of various domains, it is necessary to reference the content from the same domain as the sentences where ellipsis and anaphora occur, making simple coreference resolution insufficient. Particularly in spoken dialogue, two additional factors must be considered. The first is processing time. Among existing methods, the mention pair approach [6] generates all possible candidates for classification, which requires a significant amount of time. The second is contextual consideration. It is essential to reference only the appropriate contextual content in dialogue to handle ellipsis and anaphora.  
In this paper, we conducted experiments on ellipsis and anaphora resolution through prompt modification based on whether the history and current information share the same context in a conversational environment with context switching, as well as through a copying mechanism for GPT. We demonstrated that the copying mechanism in GPT shows significant performance improvements not only in the training domain but also in domains outside of training.  
The contributions of this paper are as follows:  
* We improved the performance of ellipsis and anaphora resolution by introducing a copying mechanism to the GPT structure.
* We proposed a method for prompt modification in conversational environments where context switching occurs.
* We demonstrated that the proposed method shows significant performance improvements not only in the training domain but also in domains outside of training ([[https://github.com/cwnu-airlab/CGM](https://github.com/cwnu-airlab/CGM)]([https://github.com/cwnu-airlab/CGM](https://github.com/cwnu-airlab/CGM)), accessed on 14 December 2024).