Pseudo Labeling and Negative Feedback Learning for Large-scale Multi-label Domain Classification	In large-scale domain classification, an utterance can be handled by multiple domains with overlapped capabilities. However, only a limited number of ground-truth domains are provided for each training utterance in practice while knowing as many as correct target labels is helpful for improving the model performance. In this paper, given one ground-truth domain for each training utterance, we regard domains consistently predicted with the highest confidences as additional pseudo labels for the training. In order to reduce prediction errors due to incorrect pseudo labels, we leverage utterances with negative system responses to decrease the confidences of the incorrectly predicted domains. Evaluating on user utterances from an intelligent conversational system, we show that the proposed approach significantly improves the performance of domain classification with hypothesis reranking.

Shortlister Model ::: Leveraging Negative Feedback	During the model training, irrelevant domains could be top predicted, and regarding them as additional target labels results in wrong confirmation bias , which causes incorrect model training. To reduce the side effect, we leverage utterances with negative responses in order to discourage the utterances' incorrect predictions. This setting can be considered as a multi-label variant of Positive, Unlabeled, and Biased Negative Data (PUbN) learning . We obtain training utterances from log data, where utterances with positive system responses are used as the positive train set in Equation DISPLAY_FORM6 and DISPLAY_FORM10 while the utterances with negative responses are used as the negative train set in Equation DISPLAY_FORM14. For example, AnimalSounds is a (positive) ground-truth domain for “a monkey sound” because the system response to the utterance is “Here comes a monkey sound” while it is a negative ground-truth for “a dragon sound” as the response is “I don't know what sound a dragon makes”. Previous work ,  excludes such negative utterances from the training set.

Therefore, we are given only one ground-truth for each training utterance. In order to improve the classification performance in this setting, if certain domains are repeatedly predicted with the highest confidences even though they are not the ground-truths of an utterance, we regard the domains as additional pseudo labels. This is closely related to pseudo labeling  or self-training , , . While the conventional pseudo labeling is used to derive target labels for unlabeled data, our approach adds pseudo labels to singly labeled data so that the data can have multiple target labels. Also, the approach is related to self-distillation, which leverages the confidence scores of the non-target outputs to improve the model performance , . While distillation methods utilize the confidence scores as the soft targets, pseudo labeling regards high confident outputs as the hard targets to further boost their confidences. We use both pseudo labeling and self-distillation in our work. Pseudo labels can be wrongly derived when irrelevant domains are top predicted, which can lead the model training with wrong supervision. To mitigate this issue, we leverage utterances with negative system responses to lower the prediction confidences of the failing domains.

We find that it is more effective to explicitly demote the prediction confidences of the domains resulted in negative responses if they are top ranked. It is formulated as a loss function: where $j$ denotes the index corresponding to the negative ground-truth domain. We demote the confidences of the negative ground-truths only when they are the highest so that the influence of using the negative ground-truths is not overwhelming.

Using self-distillation $((3)\rightarrow (4) \;\text{and}\; (5)\rightarrow (6))$ shows increased F-1 score by increasing recall and decreasing precision, but the improvements are not significant. One issue is that pseudo labeling and self-distillation are contrary since the former encourages entropy minimization ,  while the latter can increase entropy by soft targeting the non-target labels. More investigation of self-distillation along with the proposed pseudo labeling would be future work. Table TABREF22 shows examples of derived pseudo labels from model (6). It demonstrates that the domains capable of processing the utterances can be derived, which helps more correct model training.

Shortlister Model ::: Deriving Pseudo Labels	We hypothesize that the outputs repeatedly predicted with the highest confidences are indeed correct labels in many cases in multi-label PU learning setting. This approach is closely related to pseudo labeling  or self-training , ,  in semi-supervised learning since our model is supervised with additional pseudo labels, but differs in that our approach assigns pseudo labels to singly labeled train sets rather than unlabeled data sets. We derive the pseudo labels when the following conditions are met: Maximally $p$ domains predicted with the highest confidences that are higher than the confidence of the known ground-truth. Domains predicted with the highest confidences for $r$ times consecutively so that consistent top predictions are used as pseudo labels. For the experiments in Section SECREF5, we use $p$=2 and $r$=4, which show the best dev set performance. Those derived pseudo labels are used in the model training as follows:  where $\tilde{y}$ denotes an $n$-hot vector such that the elements corresponding to the original ground-truth and the additional pseudo labels are set to 1.

Hypothesis Reranking Model	Figure FIGREF20 shows the overall architecture of the hypothesis reranker that is similar to . First, we run intent classification and slot filling for the $k$ most confident domains from the shortlister outputs to obtain additional information for those domains . Then, we compose $k$ hypotheses, each of which is a vector consists of the shortlister confidence score, intent score, Viterbi score of slot-filling, domain vector, intent vector, and the summation of the slot vectors. On top of the $k$ hypothesis vectors, a BiLSTM is utilized for representing contextualized hypotheses and a shared feed-forward neural network is used to obtain final confidence score for each hypothesis. We set $k$=3 in our experiments following . We leverage the given ground-truth and the derived pseudo labels from the shortlister at the epoch showing the best dev set performance as target labels for training the reranker. We use hinge loss with margin 0.4 as the loss function. One issue of the hypothesis reranking is that a training utterance cannot be used if no ground-truth exist in the top $k$ predictions of the shortlister.

Shortlister Model ::: Self-distillation	Knowledge distillation has been shown to improve the model performance by leveraging the prediction confidence scores from another model or from previous epochs , , . Inspired by , we utilize the model at the epoch showing the best dev set performance before the current epoch to obtain the prediction confidence scores as the soft target. The self-distillation in our work can be formulated as follows:  where $\tilde{o_i}$ denotes the model output at the epoch showing the best dev set performance so far. Before taking sigmoid to obtain $\tilde{o_i}$, we use 16 as the temperature to increase the influence of distillation , which shows the best dev set performance following .

Shortlister Model	Our shortlister architecture is shown in Figure FIGREF3. The words of an input utterance are represented as contextualized word vectors by bidirectional long short-term memory (BiLSTM) on top of the word embedding layer . Then, the concatenation of the last outputs of the forward LSTM and the backward LSTM is used to represent the utterance as a vector. Following  and , we leverage the domain enablement information through attention mechanism , where the weighted sum of enabled domain vectors followed by sigmoid activation is concatenated to the utterance vector for representing a personalized utterance. On top of the personalized utterance vector, a feed-forward neural network followed by sigmoid activation is used to obtain $n$-dimensional output vector $o$, where the prediction confidence of each domain is represented as a scalar value between 0 and 1. Given an input utterance and its target label, binary cross entropy is used as the baseline loss function as follows:  where $o$, $y$, and $n$ denote the model output vector, the one-hot vector of the target label, and the number of total labels. We describe other proposed loss functions in the following subsections.

Introduction	Domain classification is a task that predicts the most relevant domain given an input utterance . It is becoming more challenging since recent conversational interaction systems such as Amazon Alexa, Google Assistant, and Microsoft Cortana support more than thousands of domains developed by external developers , , . As they are independently and rapidly developed without a centralized ontology, multiple domains have overlapped capabilities that can process the same utterances. For example, “make an elephant sound” can be processed by AnimalSounds, AnimalNoises, and ZooKeeper domains. Since there are a large number of domains, which are even frequently added or removed, it is infeasible to obtain all the ground-truth domains of the training utterances, and domain classifiers for conversational interaction systems are usually trained given only a small number (usually one) of ground-truths in the training utterances. This setting corresponds to multi-label positive and unlabeled (PU) learning, where assigned labels are positive, unassigned labels are not necessarily negative, and one or more labels are assigned for an instance , . In this paper, we utilize user log data, which contain triples of an utterance, the predicted domain, and the response, for the model training.

Model Overview	We take a hypothesis reranking approach, which is widely used in large-scale domain classification for higher scalability , . Within the approach, a shortlister, which is a light-weighted domain classifier, suggests the most promising $k$ domains as the hypotheses. We train the shortlister along with the added pseudo labels, leveraging negative system responses, and self-distillation, which are described in Section SECREF3. Then a hypothesis reranker selects the final prediction from the $k$ hypotheses enriched with additional input features, which is described in Section SECREF4.

This is problematic in the multi-label PU setting since correct domains can indeed exist in the top $k$ list but unknown, which makes the training utterance less useful in the reranking. Our pseudo labeling method can address this issue. If correct pseudo labels are derived from the shortlister's top predictions for such utterances, we can use them properly in the reranker training, which was unavailable without them. This allows our approach make more improvement in hypothesis reranking than shortlisting.

For example, if a system response of a domain for an input utterance is “I don't know that one”, the domain is regarded as a negative ground-truth since it fails to handle the utterance. Evaluating on an annotated dataset from the user logs of a large-scale conversation interaction system, we show that the proposed approach significantly improves the domain classification especially when hypothesis reranking is used , .

Experiments ::: Datasets	We utilize utterances with explicit invocation patterns from an intelligent conversational system for the model training similarly to  and . For example, given “ask {AmbientSounds} to {play thunderstorm sound}”, we extract “play thunderstorm” as the input utterance and Ambient Sounds as the ground-truth. One difference from the previous work is that we utilize utterances with positive system responses as the positive train set and the dev set, and use those with the negative responses as the negative train set as described in Section SECREF11. We have extracted 3M positive train, 400K negative train, and 600K dev sets from 4M log data with 2,500 most frequent domains as the ground-truths. Pseudo labels are added to 53K out of 3M in the positive train set as described in Section SECREF7. For the evaluation, we have extracted 10K random utterances from the user log data and independent annotators labeled the top three predictions of all the evaluated models for each utterance so that we can correctly compute nDCG at rank position 3.

Conclusion	We have proposed deriving pseudo labels along with leveraging utterances with negative system responses and self-distillation to improve the performance of domain classification when multiple domains are ground-truths even if only one ground-truth is known in large-scale domain classification. Evaluating on the test utterances with multiple ground-truths from an intelligent conversational system, we have showed that the proposed approach significantly improves the performance of domain classification with hypothesis reranking. As future work, combining our approach with pure semi-supervised learning, and the relation between pseudo labeling and distillation should be further studied.

Experiments ::: Experiment Results	Table TABREF21 shows the evaluation results of the shortlister and the hypothesis reranker with the proposed approaches. For the shortlisters, we show nDCG$_3$ scores, which are highly correlated with the F1 scores of the rerankers than other metrics since the second and third top shortlister predictions contribute the metric. We find that just using the pseudo labels as the additional targets degrades the performance (2). However, when both the pseudo labels and the negative ground-truths are utilized, we observe significant improvements for both precision and recall (5). In addition, recall is increased when self-distillation is used, which achieves the best F1 score (6). Each of utilizing the negative feedback $((1)\rightarrow (3) \;\text{and}\; (2)\rightarrow (5))$ and then additional pseudo labels $((3)\rightarrow (5) \;\text{and}\; (4)\rightarrow (6))$ show statistically significant improvements with McNemar test for p=0.05 for the final reranker results.

Experiments	In this section, we show training and evaluation sets, and experiment results.

Shortlister Model ::: Combined Loss	The model is optimized with a combined loss function as follows:  where $\alpha ^t=1-0.95^t$ and $t$ is the current epoch so that the baseline loss is mainly used in the earlier epochs while the pseudo labels and self-distillation are more contributing in the later epochs following . $\beta $ is a hyperparameter for utilizing negative ground-truths, which is set to 0.00025 showing the best dev set performance.

Question: How do they decide by how much to decrease confidences of incorrectly predicted domains?
Answer: The confidence of the incorrectly predicted domain is decreased only when it is highest among all predictions.