Pseudo Labeling and Negative Feedback Learning for Large-scale Multi-label Domain Classification

Abstract: In large-scale domain classification, an utterance can be handled by multiple domains with overlapped capabilities. However, only a limited number of ground-truth domains are provided for each training utterance in practice while knowing as many as correct target labels is helpful for improving the model performance. In this paper, given one ground-truth domain for each training utterance, we regard domains consistently predicted with the highest confidences as additional pseudo labels for the training. In order to reduce prediction errors due to incorrect pseudo labels, we leverage utterances with negative system responses to decrease the confidences of the incorrectly predicted domains. Evaluating on user utterances from an intelligent conversational system, we show that the proposed approach significantly improves the performance of domain classification with hypothesis reranking.

Domain classification is a task that predicts the most relevant domain given an input utterance BIBREF0. It is becoming more challenging since recent conversational interaction systems such as Amazon Alexa, Google Assistant, and Microsoft Cortana support more than thousands of domains developed by external developers BIBREF3, BIBREF2, BIBREF4. As they are independently and rapidly developed without a centralized ontology, multiple domains have overlapped capabilities that can process the same utterances. For example, “make an elephant sound” can be processed by AnimalSounds, AnimalNoises, and ZooKeeper domains.
Since there are a large number of domains, which are even frequently added or removed, it is infeasible to obtain all the ground-truth domains of the training utterances, and domain classifiers for conversational interaction systems are usually trained given only a small number (usually one) of ground-truths in the training utterances. This setting corresponds to multi-label positive and unlabeled (PU) learning, where assigned labels are positive, unassigned labels are not necessarily negative, and one or more labels are assigned for an instance BIBREF5, BIBREF6.
In this paper, we utilize user log data, which contain triples of an utterance, the predicted domain, and the response, for the model training. Therefore, we are given only one ground-truth for each training utterance. In order to improve the classification performance in this setting, if certain domains are repeatedly predicted with the highest confidences even though they are not the ground-truths of an utterance, we regard the domains as additional pseudo labels. This is closely related to pseudo labeling BIBREF7 or self-training BIBREF8, BIBREF9, BIBREF10. While the conventional pseudo labeling is used to derive target labels for unlabeled data, our approach adds pseudo labels to singly labeled data so that the data can have multiple target labels. Also, the approach is related to self-distillation, which leverages the confidence scores of the non-target outputs to improve the model performance BIBREF11, BIBREF12. While distillation methods utilize the confidence scores as the soft targets, pseudo labeling regards high confident outputs as the hard targets to further boost their confidences. We use both pseudo labeling and self-distillation in our work.
Pseudo labels can be wrongly derived when irrelevant domains are top predicted, which can lead the model training with wrong supervision. To mitigate this issue, we leverage utterances with negative system responses to lower the prediction confidences of the failing domains. For example, if a system response of a domain for an input utterance is “I don't know that one”, the domain is regarded as a negative ground-truth since it fails to handle the utterance.
Evaluating on an annotated dataset from the user logs of a large-scale conversation interaction system, we show that the proposed approach significantly improves the domain classification especially when hypothesis reranking is used BIBREF13, BIBREF4.
We take a hypothesis reranking approach, which is widely used in large-scale domain classification for higher scalability BIBREF13, BIBREF4. Within the approach, a shortlister, which is a light-weighted domain classifier, suggests the most promising $k$ domains as the hypotheses. We train the shortlister along with the added pseudo labels, leveraging negative system responses, and self-distillation, which are described in Section SECREF3. Then a hypothesis reranker selects the final prediction from the $k$ hypotheses enriched with additional input features, which is described in Section SECREF4.
Our shortlister architecture is shown in Figure FIGREF3. The words of an input utterance are represented as contextualized word vectors by bidirectional long short-term memory (BiLSTM) on top of the word embedding layer BIBREF14. Then, the concatenation of the last outputs of the forward LSTM and the backward LSTM is used to represent the utterance as a vector. Following BIBREF2 and BIBREF17, we leverage the domain enablement information through attention mechanism BIBREF18, where the weighted sum of enabled domain vectors followed by sigmoid activation is concatenated to the utterance vector for representing a personalized utterance. On top of the personalized utterance vector, a feed-forward neural network followed by sigmoid activation is used to obtain $n$-dimensional output vector $o$, where the prediction confidence of each domain is represented as a scalar value between 0 and 1.
Given an input utterance and its target label, binary cross entropy is used as the baseline loss function as follows:
where $o$, $y$, and $n$ denote the model output vector, the one-hot vector of the target label, and the number of total labels. We describe other proposed loss functions in the following subsections.
We hypothesize that the outputs repeatedly predicted with the highest confidences are indeed correct labels in many cases in multi-label PU learning setting. This approach is closely related to pseudo labeling BIBREF7 or self-training BIBREF8, BIBREF9, BIBREF10 in semi-supervised learning since our model is supervised with additional pseudo labels, but differs in that our approach assigns pseudo labels to singly labeled train sets rather than unlabeled data sets.
We derive the pseudo labels when the following conditions are met:
Maximally $p$ domains predicted with the highest confidences that are higher than the confidence of the known ground-truth.
Domains predicted with the highest confidences for $r$ times consecutively so that consistent top predictions are used as pseudo labels.
For the experiments in Section SECREF5, we use $p$=2 and $r$=4, which show the best dev set performance. Those derived pseudo labels are used in the model training as follows:
where $\tilde{y}$ denotes an $n$-hot vector such that the elements corresponding to the original ground-truth and the additional pseudo labels are set to 1.
During the model training, irrelevant domains could be top predicted, and regarding them as additional target labels results in wrong confirmation bias BIBREF19, which causes incorrect model training. To reduce the side effect, we leverage utterances with negative responses in order to discourage the utterances' incorrect predictions. This setting can be considered as a multi-label variant of Positive, Unlabeled, and Biased Negative Data (PUbN) learning BIBREF20.
We obtain training utterances from log data, where utterances with positive system responses are used as the positive train set in Equation DISPLAY_FORM6 and DISPLAY_FORM10 while the utterances with negative responses are used as the negative train set in Equation DISPLAY_FORM14. For example, AnimalSounds is a (positive) ground-truth domain for “a monkey sound” because the system response to the utterance is “Here comes a monkey sound” while it is a negative ground-truth for “a dragon sound” as the response is “I don't know what sound a dragon makes”.
Previous work BIBREF21, BIBREF22 excludes such negative utterances from the training set. We find that it is more effective to explicitly demote the prediction confidences of the domains resulted in negative responses if they are top ranked. It is formulated as a loss function:
where $j$ denotes the index corresponding to the negative ground-truth domain. We demote the confidences of the negative ground-truths only when they are the highest so that the influence of using the negative ground-truths is not overwhelming.
Knowledge distillation has been shown to improve the model performance by leveraging the prediction confidence scores from another model or from previous epochs BIBREF11, BIBREF12, BIBREF17. Inspired by BIBREF17, we utilize the model at the epoch showing the best dev set performance before the current epoch to obtain the prediction confidence scores as the soft target. The self-distillation in our work can be formulated as follows:
where $\tilde{o_i}$ denotes the model output at the epoch showing the best dev set performance so far. Before taking sigmoid to obtain $\tilde{o_i}$, we use 16 as the temperature to increase the influence of distillation BIBREF11, which shows the best dev set performance following BIBREF17.
The model is optimized with a combined loss function as follows:
where $\alpha ^t=1-0.95^t$ and $t$ is the current epoch so that the baseline loss is mainly used in the earlier epochs while the pseudo labels and self-distillation are more contributing in the later epochs following BIBREF23. $\beta $ is a hyperparameter for utilizing negative ground-truths, which is set to 0.00025 showing the best dev set performance.
Figure FIGREF20 shows the overall architecture of the hypothesis reranker that is similar to BIBREF4. First, we run intent classification and slot filling for the $k$ most confident domains from the shortlister outputs to obtain additional information for those domains BIBREF0. Then, we compose $k$ hypotheses, each of which is a vector consists of the shortlister confidence score, intent score, Viterbi score of slot-filling, domain vector, intent vector, and the summation of the slot vectors. On top of the $k$ hypothesis vectors, a BiLSTM is utilized for representing contextualized hypotheses and a shared feed-forward neural network is used to obtain final confidence score for each hypothesis. We set $k$=3 in our experiments following BIBREF4. We leverage the given ground-truth and the derived pseudo labels from the shortlister at the epoch showing the best dev set performance as target labels for training the reranker. We use hinge loss with margin 0.4 as the loss function.
One issue of the hypothesis reranking is that a training utterance cannot be used if no ground-truth exist in the top $k$ predictions of the shortlister. This is problematic in the multi-label PU setting since correct domains can indeed exist in the top $k$ list but unknown, which makes the training utterance less useful in the reranking. Our pseudo labeling method can address this issue. If correct pseudo labels are derived from the shortlister's top predictions for such utterances, we can use them properly in the reranker training, which was unavailable without them. This allows our approach make more improvement in hypothesis reranking than shortlisting.
In this section, we show training and evaluation sets, and experiment results.
We utilize utterances with explicit invocation patterns from an intelligent conversational system for the model training similarly to BIBREF4 and BIBREF17. For example, given “ask {AmbientSounds} to {play thunderstorm sound}”, we extract “play thunderstorm” as the input utterance and Ambient
Sounds as the ground-truth. One difference from the previous work is that we utilize utterances with positive system responses as the positive train set and the dev set, and use those with the negative responses as the negative train set as described in Section SECREF11. We have extracted 3M positive train, 400K negative train, and 600K dev sets from 4M log data with 2,500 most frequent domains as the ground-truths. Pseudo labels are added to 53K out of 3M in the positive train set as described in Section SECREF7.
For the evaluation, we have extracted 10K random utterances from the user log data and independent annotators labeled the top three predictions of all the evaluated models for each utterance so that we can correctly compute nDCG at rank position 3.
Table TABREF21 shows the evaluation results of the shortlister and the hypothesis reranker with the proposed approaches. For the shortlisters, we show nDCG$_3$ scores, which are highly correlated with the F1 scores of the rerankers than other metrics since the second and third top shortlister predictions contribute the metric. We find that just using the pseudo labels as the additional targets degrades the performance (2). However, when both the pseudo labels and the negative ground-truths are utilized, we observe significant improvements for both precision and recall (5). In addition, recall is increased when self-distillation is used, which achieves the best F1 score (6). Each of utilizing the negative feedback $((1)\rightarrow (3) \;\text{and}\; (2)\rightarrow (5))$ and then additional pseudo labels $((3)\rightarrow (5) \;\text{and}\; (4)\rightarrow (6))$ show statistically significant improvements with McNemar test for p=0.05 for the final reranker results.
Using self-distillation $((3)\rightarrow (4) \;\text{and}\; (5)\rightarrow (6))$ shows increased F-1 score by increasing recall and decreasing precision, but the improvements are not significant. One issue is that pseudo labeling and self-distillation are contrary since the former encourages entropy minimization BIBREF25, BIBREF7 while the latter can increase entropy by soft targeting the non-target labels. More investigation of self-distillation along with the proposed pseudo labeling would be future work.
Table TABREF22 shows examples of derived pseudo labels from model (6). It demonstrates that the domains capable of processing the utterances can be derived, which helps more correct model training.
We have proposed deriving pseudo labels along with leveraging utterances with negative system responses and self-distillation to improve the performance of domain classification when multiple domains are ground-truths even if only one ground-truth is known in large-scale domain classification. Evaluating on the test utterances with multiple ground-truths from an intelligent conversational system, we have showed that the proposed approach significantly improves the performance of domain classification with hypothesis reranking.
As future work, combining our approach with pure semi-supervised learning, and the relation between pseudo labeling and distillation should be further studied.

Figure and table captions:
Fig. 1. Shortlister architecture: an input utterance is represented as a concatenation of the utterance vector from BiLSTM and the weighted sum of domain enablement vectors through domain enablement attention mechanism. Then, a feed-forward neural network followed by sigmoid activation represents the n-dimensional output vector.
Fig. 2. Hypothesis Reranker Architecture: Each hypothesis consists of scores and vectors of domain, intent, and slots. Then, BiLSTM and a feed-forward neural network are used to represent contextualized hypothesis confidence scores.
Table 1. Evaluation results on various metrics (%). pseudo, neg feed, and self dist denote using derived pseudo labels, negative feedback, and self-distillation, respectively.
Table 2. Examples of additional pseudo labels.

Questions about this paper:
By how much do they improve on domain classification?
Which dataset do they evaluate on?
How do they decide by how much to decrease confidences of incorrectly predicted domains?

Answers:
F1 is improved from 80.15 to 80.50 and from 80.71 to 81.69 of Shortlister and Hipothesis Reranker models respectively.
The dataset was created by extracting utterances from the user log data from an intelligent conversational system.
The confidence of the incorrectly predicted domain is decreased only when it is highest among all predictions.