# Discriminator-Enhanced Knowledge-Distillation Networks
## 2 Related Work  
In recent years, neural networks have achieved remarkable success in generating natural language text. Among various text generation tasks, question-answering completion is a common one that is employed by web search engines to suggest the most frequently used completions to users, a process called most popular completion (MPC) [8]. In the QAC task, both the accuracy and the inference speed of the model are critical factors that need to be considered, yet few prior studies have addressed these two factors simultaneously. In the following sections, we will review the methods for enhancing accuracy and the compression techniques in the QAC task. Traditional methods for QAC accomplish the task in two steps. The first step involves extracting candidates to increase recall. MPC [8] expands each query to a high-dimensional feature vector and uses cosine similarity to find the top N candidates from historical queries with the same prefix. Other approaches generate query suggestions from additional corpora to offer more accurate recommendations [9; 10]. The second step involves re-ranking the candidates with additional features to increase precision [11]. Several kinds of additional information have been exploited for this task, including context or session information [8; 12], personalization [8; 13], time/popularity sensitivity [8; 14; 15], user behaviors [16; 17; 18; 19], and click-through logs [20]. To overcome traditional methods' insufficient feature extraction ability, neural network methods have been adopted in recent years. Neural language models can seamlessly incorporate additional features, such as user ID embeddings to model personalization [21; 22]. Time-aware [11] and spelling-error-aware models [23] have also been developed under this framework to increase the accuracy rate. Deep neural networks, such as ELMo [24] and BERT, learn high-quality, deep contextualized word representations using deep neural models that can be directly integrated into various tasks for performance boosting. However, these models contain millions of parameters, making their application in practice difficult when computational resources are limited. Knowledge distillation [25] is a promising framework for reducing the size of models. Several methods have been developed to leverage linguistic knowledge to make the distillation process more informative. [PERSON] et al. proposed a method, where the student layer can learn from multiple consecutive teacher layers to obtain more comprehensive supervision information [26]. A soft-attention mechanism is used to determine the impact of different layers on the learning process. Similarly, [PERSON] et al. developed three types of loss functions that encourage the effective transfer of linguistic knowledge from teacher layers to the student model [27]. The losses are applied to different parts of the network, including the embedding layer, the transformer layer, and the prediction layer. The ultimate goal of these knowledge-distillation methods is to enable the student model to accurately mimic the characteristics of the teacher model, such as its hidden representations [28], output probabilities [25], or even the generated sentences themselves [29]. In contrast to the previous approaches, this paper proposes a novel approach to simultaneously address both critical and challenging problems by introducing a discriminator into the knowledge-distillation framework. Notably, our framework offers an easy extension to the existing distillation methods mentioned above. In this section, we present the details of Dis-KD. The overall architecture of our model is illustrated in Figure 1, which primarily comprises two components:  
Our model comprises two main components: (1) a conventional knowledge-distillation framework based on GPT-2, which is known for its superior ability to model the relationships among words and its expressive power, and (2) a simple LSTM network in the discriminator that evaluates the quality of the generated sequence.