# Discriminator-Enhanced Knowledge-Distillation Networks
## 3 Experiments
### Dataset and Evaluation Metric  
Our experiments were conducted on the AOL dataset [31], following the protocol delineated in [32]. In particular, the dataset comprises 17,521,031 training queries, 1,521,971 validation queries, and 1,317,632 test queries. Notably, the test dataset includes 670,810 seen queries and 646,822 unseen queries, with the latter constituting almost half of the test data. In our context,'seen' data refer to test data that have appeared in the training dataset, while 'unseen' data signify test data that have never been featured in the training output. Itshould be noted that this arrangement does not induce tag leakage issues in QAC, owing to its distinct task definition. In our experiment, if the query 'ask.com' is included in the test set, its presence or absence in the training dataset classifies it as either seen or unseen. Our primary task is to utilize the input subword to generate a complete query. For instance, during the model testing process, if a user enters the subword 'ask', the expected completion is 'ask.com', which may be generated from various candidate words, such as 'ask jeeves' or 'ask.com'. Several queries starting with 'ask' can be found in the training data. We expect the model to rank candidate words in accordance with our anticipated order. To gauge the concordance of such expectations, we employ the mean reciprocal rank (MRR) metric, which is discussed in further detail later. The AOL dataset comprises five distinct variables for examination: AnonID, Query, QueryTime, ItemRank, and ClickURL. These variables can inform diverse analytical models. Table 1 illustrates a representative data sample. Nonetheless, for this study, we limit our focus solely to the query column as input to our analytical model. For instance, 'ask.com' is the data used for model training. In our study, the BPE (byte pair encoding) tokenizer was utilized to preprocess the language model input as proposed by [PERSON] et al. [33]. Specifically, the input text \"ask.com\" was tokenized into three subwords, namely \"ask\", \".\", and \"com\". Our model requires the input text to satisfy a minimum subword criterion. For example, in the case of \"ask.com\", the input text used to test our model is \"ask\", rather than \"as\" or \"a\". This criterion was consistently applied to the input data in the test set. We used the model to generate ten candidate words to calculate the mean reciprocal rank (MRR) [34]. We show the output of the model in Table 5 and compare the output of other models. The dataset used in our experiments was accumulated over three months, with the final week used for testing and the preceding week used for validation. During data preprocessing, we removed non-ASCII characters and transformed uppercase characters into lowercase. We implemented a technique to replace multiple consecutive spaces with a single space, and excised any leading or trailing spaces. Additionally, we identified and consolidated duplicates submitted by the same user for an identical query that was adjacently located. To ensure the rigor of our analysis, we also filtered out queries shorter than three characters or longer than forty. These preprocessing steps were instrumental in enhancing the quality of our dataset, thereby bolstering the validity of our subsequent analysis. Detailed information is in Table 2:  
In the evaluation of the QAC system, we consider two key metrics: accuracy and time. Accuracy reflects the degree to which the model's suggestions match the user's intended queries, and better performance in this regard leads to a more satisfactory user experience. To quantify the auto-comple \begin{table}
\begin{tabular}{l l l l l} \hline
**Anon ID** & **Query** & **Query Time** & **Item Rank** & **Click URL** \\ \hline
142 & rentdirect.com & 1 March 2006 07:17:12 & & \\
217 & ask.com & 31 March 2006 14:31:10 & 1 & www.ask.com \\
993 & myspace.co & 1 March 2006 12:13:36 & & \\ \hline \end{tabular}
\end{table} 
Table 1: Dataset examples.