Regression-Free Model Updates for Spoken Language Understanding
Year: 2023

Abstract:
In real-world systems, an important requirement for model updates is to avoid regressions in user experience caused by flips of previously correct classifications to incorrect ones. Multiple techniques for that have been proposed in the recent literature. In this paper, we apply one such technique, focal distillation, to model updates in a goal-oriented dialog system and assess its usefulness in practice. In particular, we evaluate its effectiveness for key language understanding tasks, including sentence classification and sequence labeling tasks, we further assess its effect when applied to repeated model updates over time, and test its compatibility with mislabeled data. Our experiments on a public benchmark and data from a deployed dialog system demonstrate that focal distillation can substantially reduce regressions, at only minor drops in accuracy, and that it further outperforms naive supervised training in challenging mislabeled data and label expansion settings.

heading: 1 Introduction
text: Machine learning models that are deployed in realworld applications typically require regular updates to accommodate data distribution shifts or changes to the output label space. The retraining process, even if it leads to stable or improved overall accuracy, can result in different sample-level predictions due to its stochastic nature. In an application setting, that in turn can change (or even break) specific functionalities. A key requirement for model updates in real-world applications is therefore to minimize regressions in user experience.
For classification models, Yan et al. (2021) formalized this requirement as minimizing the number of negative flips of a model, defined as the number of previously correct classifications that turn incorrect for a new model. Previous work proposed several methods towards that goal (Shen et al., 2020; Yan et al., 2021; Zhao et al., 2022; Träuble et al.,
2021) that rely on knowledge distillation, model ensembling or Bayesian learning.
In this work, we focus on model update-caused regressions in goal-oriented dialog systems, and in particular on updates of spoken language understanding models. In real-world dialog systems, a negative flip would mean that a request that was previously correctly understood is now interpreted as a different intent (or with different slots) and therefore leads to a regression in user experience.
While previous work explored the reduction of negative flips on various tasks, spoken language understanding remains unexplored (see section 2). We therefore apply focal distillation (Yan et al., 2021), the most applicable existing technique, to this use case. Moreover, the use in a real-world goal-oriented dialog system raises additional questions that we address. Specifically, we study the following:
Effectiveness for DC and IC: We test focal distillation on domain classification (DC) and intent classification (IC), two key tasks in spoken language understanding, using public data as well as internal datasets from a real-world dialog system.
Applicability to SL: We further test the effectiveness for slot labeling (SL), a sequence labeling task that requires an extension of focal distillation to handle tasks with token-level supervision.
Repeated Model Updates: We simulate multiple iterations of retraining with focal distillation to study its long-term effect, in particular, whether the coupling of new and old model via distillation restricts the model’s ability to learn new features.
Noisy Labels: Finally, we also study the effect of mislabeled data. In the presence of annotation errors, focal distillation bears the risk that it enforces prediction consistency on samples that have supposedly correct classifications in the old model, but are actually mislabeled, preventing the new model to predict the true correct label.
We run extensive experiments for DC, IC and
538
SL tasks on SLURP (Bastianelli et al., 2020), a public benchmark, and internal datasets from our real-world goal-oriented dialog system. We find that focal distillation is effective for DC and IC and reduces negative flips by up to 30% relative at no or only marginal decreases in accuracy. For SL, a naive application as a token-level loss is effective as well and brings 8% relative reduction on average. When simulating repeated retraining over time, focal distillation can restrict the model’s ability to learn new labels, but this can be remedied by warm-starting the model with the previous model’s weights. Finally, we also show that focal distillation is beneficial even under annotation errors, and can be made even more robust by adding noise-awareness to the loss.

heading: 2 Related Work
text: Enabling regression-free model updates is a relatively recent line of research. Shen et al. (2020) first studied it for computer vision problems with the goal of learning backwards-compatible image representations. Yan et al. (2021) introduced the notion of negative flips for classification tasks and coined the minimization of them as positive-congruent training. They proposed focal knowledge distillation, a variant of traditional teacher-student distillation (Hinton et al., 2015), and model ensembling as techniques to achieve positive-congruent training. Zhao et al. (2022) continued this line of work by extending and combining the distillation and ensembling ideas into a single method called ELODI. With a slightly different focus, namely accepting or rejecting the predictions of a new model rather than training it, Träuble et al. (2021) proposed a Bayesian approach to reduce negative flips. In our work, we focus on Yan et al.’s (2021) focal distillation method as it is most applicable to our real-world use case where we cannot afford the use of model ensembles because of their computation, storage and latency overhead.
Xie et al. (2021) first applied the methods to NLP tasks. They found that negative flips are also prevalent during model updates for NLP tasks and demonstrated mitigations with distillation and ensembling methods in line with the earlier work. Concurrent to our work, Cai et al. (2022) extended positive-congruent training ideas to structured prediction tasks like parsing, which require extensions such as sequence distillation (Kim and Rush, 2016) or reranking. Also concurrent to our work,
Schumann et al. (2023) introduced an importanceweighted interpolation method that they find to outperform focal distillation on intent classification benchmarks. We plan to incorporate their findings in our future work.
Continual learning (also known as incremental learning, sequential learning or lifelong learning) is closely related to our work (McCloskey and Cohen, 1989; Silver and Mercer, 2002; Biesialska et al., 2020). While we focus specifically on avoiding negative flips, continual learning is more general and studies continuous training of models on evolving data and tasks, with a particular focus on avoiding catastrophic forgetting. The latter is a challenge if data for previously learned features is no longer available; it is however less relevant for our application scenario, a real-world goal-oriented dialog system, with ongoing user interactions covering all features.

heading: 3 Methods
text: Application Scenario Spoken language understanding models are a core component in many goal-oriented dialog systems. They map a natural language request to a machine-readable meaning representation that the system can act upon to fulfill the request. In our experiment setup, this is modelled as a combination of domain classification (DC), intent classification (IC) and slot labeling (SL). Consider the example Play Michael Jackson. DC recognizes this request as a Music request, IC detects a PlayMusic intent and SL identifies Michael Jackson as Artist slot, whereas Play does not represent a slot in this case.
Negative Flips Let x ∈ X be a model input (e.g. an utterance), y ∈ Y its ground truth label (e.g. an intent label) and p(y|x) a model that can be used to predict ŷi = argmaxy p(y|xi). A negative flip occurs if a new model incorrectly predicts a sample that the previous model predicted correctly, i.e. if ŷnewi ̸= yi and ŷoldi = yi. The negative flip rate (NFR) measures the fraction of samples where a correct prediction turns incorrect between two models in a dataset with size N . Yan et al. (2021) define it as
NFR = 1
N
N∑
i=1
1(ŷnewi ̸= yi ∧ ŷoldi = yi) (1)
Focal Distillation (FD) Focal distillation, as introduced by Yan et al. (2021) and illustrated in Figure 1, aims to reduce negative flips by minimizing
the loss
LCE(ŷnew, y) + λLFD(pnew(y|x), pold(y|x)) (2) whereLCE denotes the standard cross entropy (CE) loss between new model and ground truth and LFD is the additional focal distillation (FD) loss term that discourages negative flips, with a trade-off parameter λ. This loss term is formally defined as
LFD = −F(x, y) · D(pnew, pold) F(x, y) = α+ β · 1(ŷold = y),
(3)
where D is a distance between the output distributions of the new and old model, and F(x, y) is a “filtering” function. It applies a weight α to all samples in the training set and an additional weight β to the samples correctly predicted by the old model. When α = 1 and β = 0, focal distillation reduces to ordinary distillation. When α = 0 and β > 0, we are only applying the distillation objective to the training samples predicted correctly by the old model.
In their work, Yan et al. (2021) experiment with two choices for D: Kullback-Leibler (KL) divergence between temperature-scaled p(y|x) and mean-squared error (MSE) between pre-softmax logits. Since the latter performed better in their experiments, we adopt it. Hence, D is defined as
D(pnew, pold) = 1 K
K∑
j=1
(znewj (x)−zoldj (x))2 (4)
where zj(x) is element j of the K-dimensional pre-softmax logit vector for x.
FD for Slot Labeling In slot labeling, a sequence of labels has to be predicted instead of just a single label as in DC or IC. Naturally, there are two options to apply distillation in that case: either apply the loss independently to each token or use the reference model’s sequence-level decision for supervision. For the latter, Wang et al. (2020) proposed multiple techniques. In this work, we resort to the simpler token-level distillation for now and leave sequence-level distillation for future work.
We compute the FD loss for each token j in the sequence i with length M as
LFDTok = − M∑
j=1
F(xj , yj)D Ä pnewj , p old j ä (5)
Notice that this formulation works both if the models perform token-level decisions, and if they perform sequence-level decisions. In the latter case, when training the new model, the token-level FD loss is summed to the sequence-level loss.
FD with Noisy Labels FD biases the new model towards the old model’s predictions when those predictions are correct. To discern correct predictions we rely on accurate labels. However, real-world data is often noisy. Therefore we investigate the combination of FD with label noise detection. We experiment with Area Under the Margin (AUM), a method suggested by Pleiss et al. (2020). The method leverages the observation that mislabeled data hurts generalization, and thus monitors the training dynamics to define the margin of a sample. The margin M at epoch t of sample (x, y) mea-
sures how much larger the assigned logit is than the largest other logit. Let zt(x) ∈ Rc be the logit vector of sample (x, y) at epoch t. Then M at t is
M (t)(x, y) = z(t)y (x)−maxk ̸=y z(t)k (x) (6)
where logit z(t)k corresponds to class k. The first term corresponds to the assigned logit, while the second is the largest other logit. If a sample is mislabelled, the assigned logit tends to receive weaker gradient updates due to the tension between generalization from similar, correctly labeled samples and memorization of the sample itself. For instance, an utterance that is semantically similar to others labelled as PlayMusic, but is incorrectly labelled as GetWeather, results in the model predicting the true class with more confidence (higher logit) and assigning lower logit (confidence) to the incorrect label. As a consequence, a correctly labeled sample will have a larger margin than a mislabeled sample in expectation. Each sample’s margin is measured during training and averaged over all epochs T :
AUM(x, y) = 1
T
T∑
t=1
M t(x, y) (7)
We then use this measure as an additional term in the FD objective to re-weight the FD loss contributions of mislabeled samples:
LFDAUM = − g(AUM(x, y))︸ ︷︷ ︸ noise-aware weight F(x, y)D(pnew, pold)︸ ︷︷ ︸ standard FD loss (8) where g(·) simply rescales AUM into [0, 1].

heading: 4 Experimental Setup
text: We run experiments on both public and internal data. For our experiments on public data, we use SLURP (Bastianelli et al., 2020), an English multi-domain dataset for NLU spanning across 18 domains, 60 intents and 55 slot types (ca. 16,000 utterances). In addition, we present results on our internal datasets for English and German. These datasets comprise live traffic utterances, deidentified and anonymized for privacy reasons, then annotated to enable supervised training. For the internal datasets, the number of slot types is domainspecific. In the experiments for SL we employ three domain-specifc internal datasets, referred to as INTG, INT-M and INT-S, that have 88, 101 and 35 slot types, respectively. Results on public data are averaged across 5 seeds, while we only train once on
internal data. In our experiments we examine two settings: (i) A data update scenario, in which we only update the training data leaving the model architecture unchanged. In this scenario, 50% of the samples are left out when training the old model, while the complete dataset is used when training the new model, either with the baseline approach or FD. (ii) A label introduction scenario, in which we gradually introduce a new label in the dataset, training n models in sequence on datasets in which we uniformly increase the support for that label. For implementation details, we refer the reader to Appendix C. Across all experiments, we compare FD with the baseline approach of simply retraining the model on the whole training data, without any additional signal from the previous model. All models employ a BERT-based (Devlin et al., 2018) architecture: a pre-trained encoder extracts contextualized semantic word embeddings, then fed either to a Multi-Layer Perceptron (MLP) in case of DC and IC, or to a Conditional Random Field (CRF) (Lafferty et al., 2001; Lample et al., 2016) in case of SL, to obtain either sequence-level or word-level predictions. We experiment also with the introduction of warm start for the new model, i.e. the model’s weights are initialized with those of the previous model. See Appendix A for more details.

heading: 5 Experimental Results
text: We present results for each of the research questions raised in the introduction.
Is FD effective for DC and IC? Results on public data are displayed in Table 1. FD reduces NFR for both DC and IC in similar magnitude, by 27.64% for DC and by 33.79% for IC, while only decreasing accuracy slightly by 0.75% for DC and 0.44% for IC. Results on internal data are shown in Table 2. On internal data, we can only disclose relative changes to baseline, no absolute metrics. For experiments on German data we use the full dataset and the production model. While we consider all domains for DC, we only consider intents within a single domain for IC since training is expensive and time-consuming. For English we only use 10% of the full training set and a surrogate model from Huggingface to speed up the experiments (see Appendix A). Also on internal data, FD reduces NFR for both DC and IC. Again accuracy is only slightly reduced, for DC on English data we even see a slight increase in accuracy. NFR reduces less significantly on German data, which can be explained by the fact that the training set for the old
and the new model are the same and negative flips only stem from randomness in training. For the English dataset, we simulate an increase in training data, as explained in Section 4.
Can FD be used for token-level SL? As mentioned above, we employ models with a CRF layer for SL, able to make structured predictions. However, we experiment with token-level FD (see Equation (5)) that takes as input the token logits directly, instead of the top-scoring label path from the CRF. Therefore, the CRF layer of the student model is not affected by the additional distillation objective. Results on public data are shown in Table 3. With only a slight decrease in accuracy of -0.46%, NFR can be reduced by 7.93%. Results on internal data are reported in Table 4. Here we see an even larger reduction of 15.94% in NFR on the INT-G dataset, while a less significant reduction is observed on the other datasets: -3.66% on INT-M and -2.86% on INT-S. For all datasets, changes in accuracy are negligible. We conclude that FD on token-level SL reduces NFR without harming accuracy.
Does repeated FD restrict learning? Figure 2 reports the comparison of the baseline approach with FD in the label introduction scenario (spanning 5 iterations). We can observe how FD does not seem to negatively influence the overall accuracy over time of the model; on the contrary, the additional loss term seems to be moderately beneficial in helping the model learn the task compared to the baseline. The approach also behaves well in reducing the overall NFR of the new model. Interestingly, standard CE is slightly superior in absolute terms with respect to FD in learning the new label distribution. However, the gap remains fixed over time, therefore FD is not hindering the ability of
the model to learn, but only introducing an initial delay. Remarkably, FD is able to reduce regression on the newly introduced label already with a handful of samples, and consistently remains lower than the baseline on the NFR metric. Interestingly but not surprisingly, warm start helps both approaches in both metrics, with respect to the non-warm start alternative. This suggests that, in general, warm start is a useful strategy for retaining model performance during an update. However, it is clear from the results how FD benefits more from warmstart than the baseline, in terms of both accuracy improvement and NFR reduction. Further results for this setting (and the specular one of gradual removal of a label) are reported in Appendix E.
Can FD cope with noisy labels? In order to verify the extent to which FD coupled with AUM is capable of dealing with increasing level of noise we experiment both on the public SLURP dataset as well as on the internal English dataset, and we also test the approach on specific versions of those datasets manipulated to artificially introduce varying levels of noise: 20%, 40%, 60%. The algorithm used to generate noise, together with a study of how AUM is able to detect it, is reported in Appendix D. Results on the internal dataset and SLURP are reported in tables 5 and 6, respectively. Overall we observe that integrating AUM into FD does not lead to significant improvement over vanilla FD. We be-
lieve the reason behind the lack of improvement is twofold: first, there might be a more effective way to integrate the AUM signal into the FD objective; secondly, the models trained with the baseline, especially on internal data, already exhibit low NFR, therefore there is little margin for improvement. On the other hand, FD with AUM is not detrimental, neither on original nor the noisy datasets: when the level of label noise is significant, AUM helps FD recovering its performance; when the label noise if less present (if at all), AUM does not significantly decrease FD performance.

heading: 6 Conclusions
text: In this paper, we presented an extensive set of experiments to evaluate the effectiveness of focal distillation to reduce negative flips in a real-world goal-oriented dialog system. We found the technique to be effective in DC, IC and SL with only minor accuracy drops. When used repeatedly over multiple updates, the effect remains while still allowing the model to learn new labels. In addition, the method is also robust to labeling errors in the training data. As future work, we plan to extend our experiments to alternative techniques for negative flip reduction, in particular those proposed concurrent to our work, and to experiment with potentially more powerful sequence-level distillation for slot labeling.
Limitations
A first limitation of our contribution stems from the fact that to compute the focal distillation term in the loss, predictions from the old model are required. This additional stream of information will therefore cause a slight increase in the required computational power. In this work, we only experimented with FD based on mean-squared error between pre-softmax logits as that approach yielded best results in the paper our experiments are based on, leaving experiments using FD with Kullback-Leibler divergence between temperature-scaled softmax outputs for future research. Due to inference time limitations in a production setting, we did not investigate the reduction of negative flips with ensembles either. Finally, we have not tested more principled approaches for NER distillation and focused on tokenlevel distillation leaving sequence-level distillation for future work.

heading: A Experiment Details
text: The experiments have been run on p3.16xlarge EC2 instances1, equipped with eight NVIDIA Tesla V100 GPUs2. As optimization framework, PyTorch (Paszke et al., 2019) (version 1.10.0) has been used, along with PyTorch Lightning (Falcon, 2019) (version 1.8.6) for easier development and faster experimental iterations.
Across all the experiments on English corpora, the encoder is based on pre-trained BERT (Devlin et al., 2018) models from HuggingFace (Wolf et al., 2019), with their weights unfrozen during training, hence allowing their fine-tuning. For DC and IC experiments on German corpora, a custom pretrained tiny-bert (Jiao et al., 2020) is used. All models feature a two-layer, fully-connected MLP mapping word or sentence embeddings into labelspace. Additionally, the model for SL employs a CRF layer to make structured predictions about the label sequence, taking estimated label-label transition probabilities into account. For the SL experiments, we obtain subword token-level embeddings on the English corpora by summing the hidden states of the last 3 layers of the encoder. On the German corpora, the model considers the last hidden states. In the latter, word-level embeddings (aligned with slot labels) are obtained performing an average subword pooling, i.e. for each input text token we take the average embedding of all its corresponding subword tokens. In the former, the last subword token embedding is considered.
For models trained with Focal Distillation, we follow the authors’ suggestion in Yan et al. (2021) and set α = 1, β = 5 and λ = 1 for all experiments. Table 7 reports the hyperparameters used to train the models across all the experiments.
All models are trained to convergence using early stopping, monitoring model performance on a held-out validation set as convergence condition.

heading: B Data Update Scenario
text: Over time, the data available to train a predictive model can change for various reasons. In a supervised learning setting, one simple reason may be the acquisition of more labeled data: human annotators review existing unlabeled instances and assign labels to them, enlarging the training corpus.
1https://aws.amazon.com/ec2/instance-types/ p3/
2https://www.nvidia.com/en-us/data-center/ v100/
In this work, we refer to this event as a data update, and study the impact of applying FD in this scenario.
The results presented in section 5 examine in particular a scenario in which the amount of available training data is doubled for the new model. This is realized by simply training the old model on 50% of the overall training dataset, then training the new model (with either the baseline or FD) on 100% of the training samples.

heading: C New Label Scenario
text: Another possible reason for a change in the training data is the addition of data supporting new classes. New classes appearing in the training dataset of an already deployed model may be the result of the definition of a new downstream feature that the model has to support. In this work, we refer to this event as a label introduction, and study the impact of applying FD in this scenario.
Usually, data supporting a new feature is not readily available, but rather comes in batches as human annotators work to provide new labeled data based on the feature definition. For this reason, in this work we study the impact of FD on a gradual introduction of a new label. In particular, the scenario is implemented as follows: (i) a label is chosen for scenario simulation and completely removed from the dataset, i.e. all the samples belonging to that class are removed; (ii) a schedule for introducing the label in the following n “re-
leases” of the dataset is stated. For simplicity, we assume the rate at which newly labelled data becomes available is constant over time, and therefore the schedule simply dictates that a fixed amount of labelled data is reintroduced at each iteration. To do so, data pertaining to the removed labels is evenly partitioned in n batches, and the i-th dataset is simply the union of the previous dataset in the sequence and the i-th batch.
In this work we set the number of releases to n = 5. We run experiments on the SLURP dataset for Domain Classification using the qa domain and for Intent Classification using the news_query intent. This choice reflects two competing needs: on one hand, we want to reflect the observed reality of new features not becoming the predominant classes in the dataset in terms of data, even after a long time; on the other hand, to report statistically significant results we need more than a handful of samples to be removed. As a result, we choose labels that are neither the prevalent classes nor the scarcest, but are averagely represented.
When a label is introduced for the first time, we set the MSE loss in FD to zero for samples with the new label as the previous model cannot provide useful information for those. That means we zero out the logits for the new label coming from the old model by concatenating a zero tensor to the logits coming from the old model. As a result, the contribution to the MSE loss in FD is zero for the new label, falling back to only Cross Entropy loss for samples with the new label.

heading: D Area Under the Margin and Noise Generation Procedure
text: Pleiss et al. (2020) introduce the concept of Area Under the Margin (AUM), and demonstrate its ability to identify mislabelled samples in syntheticallymislabeled versions of popular Computer Vision datasets, such as CIFAR10. Their approach makes no assumption about the specific task under consideration, but only draws on the insight that a neural network’s training dynamics contain salient signals about noisy data and generalization. In this work, however, before testing the interaction of AUM with FD in a noisy data setting we test the ability of AUM of spotting noise in our Natural Language Understanding (NLU) setting to begin with. To do so, we repeat the synthetically-mislabeled experiment on our datasets.
Algorithm 1 Label noise generation Input: true labels Y , noise level nl ∈ [0, 1] Output: assigned labels Ỹ
1: N ← | Y | 2: L← {y | y ∈ Y } 3: Nflip ← ⌈nl ·N⌉ 4: for i = 0→ Nflip do 5: yi ← sample an item uniformly at random
from Y without replacement 6: L̃← {y | y ∈ L ∧ y ̸= yi} 7: ỹi ← sample a label uniformly at random
from L̃ 8: change yi into ỹi in Y 9: end for
Table 8 reports the noise levels estimated in the synthetically-mislabeled datasets. Pleiss et al. (2020) introduce threshold samples, purposefully mislabeled samples belonging to an extra class, to identify a AUM upper bound that isolates mislabeled data (see algorithm 2). In particular, they establish that the 99th percentile of threshold AUM values separates correctly- and mislabeled data. Notice that this mechanism would introduce additional complexity for coupling the AUM approach with FD, since we do not wish for the extra class to be present in the output distribution of the new model trained with FD. Therefore, beside testing vanilla AUM in the NLU setting, we test whether simply observing the sign of the AUM values is a satisfying proxy metric of the true AUM metric. Synthetic noise is injected using algorithm 2 for the former, and algorithm 1 for the latter.
We can see how standard AUM is able to estimate the noise level quite accurately, with an average (absolute) estimation error of 1.71%. The simpler variant is less competitive in estimating noise levels, reporting an average estimation error of 3.70%. Interestingly, the variant consistently overestimates noise levels for the SLURP dataset in the IC setting, exhibiting a sensitivity to the ratio between label space dimension and dataset size. Indeed, moving from the DC task to the IC task, the number of samples remains constant but the label space nearly triples in dimension. While the same holds roughly true also for the INT-G dataset, its size is considerably larger than SLURP. We hypothesize this is due to the approach having to rely on fewer samples to observe training dynamics, leading to a less informative metric computation.
Algorithm 2 Label noise generation when using threshold samples
Input: true labels Y , noise level nl ∈ [0, 1] Output: assigned labels Ỹ , threshold samples Ī
1: N ← | Y | 2: L← {y | y ∈ Y } 3: Nflip ← ⌈nl ·N⌉ 4: Nthreshold ← ⌈N · (| L |+ 1)⌉ 5: ȳ ← (| L |+ 1) ▷ new class for threshold
samples 6: Ī ← { } 7: for i = 0→ Nflip do 8: yi ← sample an item uniformly at random
from Y without replacement 9: L̃← {y | y ∈ L ∧ y ̸= yi}
10: ỹi ← sample a label uniformly at random from L̃ 11: change yi into ỹi in Y 12: end for 13: for i = 0→ Nthreshold do 14: yi ← sample an item uniformly at random
from Y without replacement 15: change yi into ȳi in Y 16: Ī ← Ī ∪ {i} 17: end for

heading: E Additional Results
text: Figure 3 reports results for repeated application of FD to the IC task, in contrast with the DC task reported in fig. 2.
Figures 4 and 5 present instead results for a specular setting to the “label introduction” one, in which we gradually remove data supporting a label.
In figs. 6 to 9 we investigate the influence of model size on repeatedly applying FD. In particular, a tiny-bert encoder is used. Results suggest that FD becomes detrimental when the older model does not have sufficient “capacity” to accurately provide a distillation signal for the new model

heading: Future Work
text: As future work, we plan to extend our experiments to alternative techniques for negative flip reduction, in particular those proposed concurrent to our work, and to experiment with potentially more powerful sequence-level distillation for slot labeling. Limitations
A first limitation of our contribution stems from the fact that to compute the focal distillation term in the loss, predictions from the old model are required. This additional stream of information will therefore cause a slight increase in the required computational power. In this work, we only experimented with FD based on mean-squared error between pre-softmax logits as that approach yielded best results in the paper our experiments are based on, leaving experiments using FD with Kullback-Leibler divergence between temperature-scaled softmax outputs for future research. Due to inference time limitations in a production setting, we did not investigate the reduction of negative flips with ensembles either. Finally, we have not tested more principled approaches for NER distillation and focused on tokenlevel distillation leaving sequence-level distillation for future work