SlowGuess/NAACL_2021_OCR huggingface dataset:
5 to 12 newly added classes varying and then suddenly drops.
- Predictable performance. Both CCFI and iCaRL display linear patterns in overall performance. It means the possibility to predict and estimate class-incremental learning performance, which is a preferable feature in many applications. But iCaRL starts at a lower accuracy and drops much faster than CCFI, probably because it discards the neural network and tunes to the simple k-nearest neighbors algorithm as the final classifier. This phenomenon also confirms that CCFI can enjoy the excellent performance of neural network classifiers and overcome its drawback of catastrophic forgetting.
4.2.1 Different Exemplar Size
To provide insight into the working mechanism of models capable of continual learning, we conduct experiments by varying the exemplar set's size with the number of new classes fixed. Figure 3 shows the model performance under the effect of different exemplar sizes by changing the observable ratios of old data.
- Overall pattern. CCFI continues to beat baselines with obvious advantages in performance. Especially, CCFI can achieve high accuracy with a minimal amount (e.g.,  $1\%$ ) of old data, although all methods can obtain performance improvement by increasing the ratio of old observable data. A dramatic performance gain can also be observed from all models when the observable ratio of old data increases from zero to non-zero values. This phenomenon further confirms that our experimental setting with limited access to old data is practically useful.
- Consistent improvement. CCFI, CoNDA, and iCarL obtain consistent improvements when increasing the ratio of old observable data. How
(a) 73-domain dataset.
(b) 150-class dataset.
Figure 3: Effect of Differential Exemplar on the 73-domain dataset and 150-class dataset. For the 73-domain dataset, the observable ratio of old data is changed from zero to  $5\%$ , while the number of new classes is fixed: 68 domains are used for initial training, and five domains are used as the additional new classes. For the 150-class dataset, the observable ratio of old data is changed from zero to  $10\%$ , while the number of new classes is fixed: 90 domains are used for initial training, and 30 domains are used as the additional new classes.
ever, the fixed feature method doesn't get apparent benefits with more old data. This phenomenon indicates more observations of old data are not the guarantee for better performance. And it further confirms the necessity of developing continual learning methods that can effectively utilize the information learned from exemplars.
Our proposed CCFI outperforms all the state-of-the-art methods. To provide further insights into its working mechanism, additional experiments are conducted to examine individual aspects of CCFI.
4.3.1 Dataset and Experimental Setting
In order to avoid the occasionalities introduced by data and model complexity, components are examined on a synthetic data set by simple neural networks with fixed weight initialization.
Specifically, we generate a synthetic dataset of 10 completely separable classes, and each class includes 1,000 examples. As the setting for continual learning, we use six classes for initial training, and four classes as additional new classes for retraining. The neural network used in this section is a simple network with two fully-connected layers. The first layer is served as a feature extractor, which is fixed after the initial training. The second layer is used as a classifier that can be tuned during retrain. To ensure other parts won't affect the component to be validated, the neural networks are initialized with the same weight matrix generated by a fixed random seed. With these settings, the results can best reflect the true contribution of components.
4.3.2 Dynamical Weight Consolidation
First, we analyze the effectiveness of the dynamical weight consolidation component. Figure 4 plots the consolidation loss (second part in Equation 8b) of model using traditional fixed weight and our proposed dynamical weight consolidation. Several observations can be made as follows.
- Fixed weight with big value. When the weight  $(\lambda$  in Equation 8b) is set by a big value (e.g.,  $10^{25}$  in Figure 4a), the consolidation loss is hard to converge and suffers from fluctuations.
- Fixed weight with small value. Oppositely, if the weight is initialized with a relatively small value (e.g.,  $10^{2}$  in Figure 4b), the consolidation loss is too small to be effective. In fact, as can be observed from Figure 4b, under the small weight setting, the consolidation loss even experiences an increase first before it slowly decreases. The increase in consolidation indicates that the neural network tends to sacrifice consolidation loss to lower the overall loss. Furthermore, this phenomenon happens when the new model modifies the important parameters learned by the original model, which are supposed to be kept with the least changes for the continual learning purpose.
- Dynamical weight consolidation. In contrast to the unstable performance of the traditional method, as shown in Figure 4c, consolidation loss converges fast and stable by using our proposed dynamical weight consolidation.
4.3.3 Fisher Information Sampling
The second set of experiments validate whether Fisher information sampling is indeed beneficial to the overall performance, compared with using
(b)
Figure 4: Consolidation loss under different weight strategies. Figure 4a: Fixed weight strategy initiated by a big value  $10^{25}$ . Figure 4b: Fixed weight strategy initiated by a small value 100. Figure 4c: Dynamic weight strategy using weight calculated by Equation 9.
(a) Sampling rate  $0.5\%$
(b) Sampling rate  $1\%$
(c) Sampling rate  $2\%$
Figure 5: Accuracy of the old task during retraining. Blue lines denote the accuracy using exemplars randomly selected, while red lines reflect the performance of exemplars generated by Fisher sampling.
(d) Sampling rate  $5\%$
randomly selected examples.
To examine how much improvement can be obtained by Fisher sampling alone, we remove the weight consolidation component in this section. Thus the results reported here are outputs of the simple two-layer model by using exemplars during retraining. From another point of view, these results show the amount of information the exemplars carrying from the original model.
Figure 5 plots the accuracy of the old task during the retraining process. Although the network is retrained with only a small set of old data, the accuracy is computed over all old data to fully examine the quality of exemplars. Since the classes in synthetic data are fully separable, the accuracy will be  $100\%$  eventually. Therefore, the quality of exemplars is demonstrated by the converging speed. A faster converge speed provided by an exemplar set is of great significance in three aspects:
- Better computational efficiency. With the same amount of old data for retraining, the most
obvious benefit indicated by the faster converge speed is, the better computational efficiency since the model requires less retraining time.
- Less "damage" to original model. A faster converge speed indicates less "damage" to the original model. All weight consolidation schemes act like "Buffers" for the old parameter. With these schemes, old parameters will slow down their changes when new tasks come. To best cope with the consolidation schemes, good exemplars should achieve comparable good performance with fewer retraining epochs, since more retraining epochs mean that the new model has modified more parameters from the original network.
- More information of original dataset. As mentioned above, under the synthetic data is fully separable, the accuracy will be  $100\%$  eventually. In this case, a faster speed can be "converted" to more information, as experiments with more data always require fewer epochs to reach the states of convergence. For example, as shown in Figure 5, much more epochs are needed under sampling rate  $0.5\%$  than that of  $1\%$ .
Figure 5 shows, exemplars generated by Fisher sampling can achieve much faster converge speed than randomly selected exemplars, which proves Fisher sampling alone can contribute contribution effectively to the overall performance.
This paper proposes a hyperparameter-free model called CCFI for continuous domain classification. CCFI can record information of old models via exemplars selected by Fisher information sampling, and conduct efficient retraining through dynamical weight consolidation. The comparison against the existing models reveals CCFI is the best performer under various experimental environments, without additional efforts in hyperparameter searching.
classifier	Finetune	Feature Extraction	CoNDA	CCFI
Initial training	Retrain	Initial training	Retrain	Initial training	Retrain	Initial training	Retrain
One-layer	0.9671	0.7232	0.9671	0.8556	0.9566	0.9116	0.9614	0.9426
Two-layer	0.9592	0.2387	0.9592	0.6495	0.9522	0.9366	0.9541	0.9408
Table 1: Effect of classifier layer number on the 150-class dataset. 90 out of 150 classes are used for initial training, and then additional data of 10 domains plus  $10\%$  of old data (90-class) are used for retraining.
classifier	Finetune	Feature Extraction	CoNDA	CCFI
Initial training	Retrain	Initial training	Retrain	Initial training	Retrain	Initial training	Retrain
One-layer	0.9561	0.6322	0.9561	0.2839	0.9349	0.4375	0.9587	0.7378
Two-layer	0.9422	0.0751	0.9422	0.1193	0.9314	0.6868	0.9472	0.7134
Table 2: Effect of classifier layer number on the 73-domain dataset. 43 out of 73 classes are used for initial training, and then additional data of 10 domains plus  $1\%$  of old data (43-class) are used for retraining.
The general statistics of the 150-class dataset  ${}^{1}$  and 73-domain dataset are listed below.
- 150-class dataset: balanced dataset with 150 intents that can be grouped into 10 general domains. Each intent has 100 training queries, 20 validation, and 30 testing queries.
- 73-domain dataset: imbalanced dataset with 73 domains. Each domain contains 512 examples on average. However, this dataset is highly imbalanced that the largest domain includes 1,771 examples, while the smallest domain only has 27 examples.
In both datasets, we split examples of each class into  $90\%$  for training,  $5\%$  for validation, and  $5\%$  for testing. All classification accuracy results are evaluated on the test set.
A.2 Implement Details
Specific settings. In our implement of CoNDA, we pick up hyperprometer  $\Delta_{pos} = 0.5$  and  $\Delta_{neg} = 0.3$ . The fixed-feature method freezes 12 layers of BERT after the initial training. Only the parameters in the new classifier layer are allowed for tuning, which in a way provides the function of continual learning. Fine-tune method can modify parameters in all 12 layers of BERT, which can be viewed as the network with little prevention of catastrophic forgetting.
General settings. Adam optimizer is used in all learning processes, and the learning rate is always set to be 0.00001. All runs are trained on 4 V100 GPUs with a batch size of 32. Our exam-
```bash
ple code can be found at: https://github.com/tinghua-code/CCFI
A.3 Multi-layer Classifier Results
To examine the effect of classifier layer number (amount of retrainable parameters), we run experiments under two frameworks. The first framework is the same as the one used in the main experimental part, which consists of a 12-layer BERT feature extractor and a one-layer linear classifier. The second framework keeps the BERT feature extractor unchanged and adds one more layer to the classifier. The results are listed in Table 1 and 2, and several observations can be made as follows.
- CCFI still remains the best performer. Our proposed CCFI model produces good performance regardless of the number of layers in the classifier. This phenomenon further confirms its effectiveness and stability.
- CoNDA is the second-best performer in both frameworks. Notably, the retraining performance of CoNDA increases when we increase the number of layers.
- BERT finetune and feature extraction method become worse when increasing the number of layers. These two baselines are sensitive to the structure of the classifier, which indicates the superficial variations of pre-trained models are not enough for continual learning.
- One-layer classifier works well enough with BERT. As can be seen from Table 1 and 2, the initial training results of all methods degrade when increasing the number of classifier layers. Therefore, we report the results based on a one-layer linear classifier in the main body of the paper.
$^{1}$ https://github.com/clinc/oos-eval
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014. Decaf: A deep convolutional activation feature for generic visual recognition. In International conference on machine learning, pages 647-655.
Robert M French. 1999. Catastrophic forgetting in connectionist networks. volume 3, pages 128-135. Elsevier.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 580-587.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799.
Young-Bum Kim, Dongchan Kim, Joo-Kyung Kim, and Ruhi Sarikaya. 2018a. A scalable neural shortlisting-eranking approach for large-scale domain classification in natural language understanding. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 16-24.
Young-Bum Kim, Dongchan Kim, Anjishnu Kumar, and Ruhi Sarikaya. 2018b. Efficient large-scale neural domain classification with personalized attention. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2214-2224.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. volume 114, pages 3521-3526. National Acad Sciences.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105.
Anjishnu Kumar, Arpit Gupta, Julian Chan, Sam Tucker, Bjorn Hoffmeister, Markus Dreyer, Stanislav Peshterliev, Ankur Gandhi, Denis
Filiminov, Ariya Rastrow, et al. 2017. Just ask: building an architecture for extensible self-service spoken language understanding.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1311-1316, Hong Kong, China. Association for Computational Linguistics.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. volume 521, pages 436-444. Nature Publishing Group.
Han Li, Jihwan Lee, Sidharth Mudgal, Ruhi Sarikaya, and Young-Bum Kim. 2019. Continuous learning for large-scale personalized domain classification. In Proceedings of NAACL-HLT, pages 3784-3794.
Zhizhong Li and Derek W Hoiem. 2016. Learning without forgetting. In Computer Vision-14th European Conference, ECCV 2016, Proceedings, pages 614-629. Springer-Verlag.
David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pages 6467-6476.
Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In *Psychology of learning and motivation*, volume 24, pages 109-165. Elsevier.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014. CNN features off-the-shelf: an astounding baseline for recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 806-813.
Gokhan Tur and Renato De Mori. 2011. Spoken language understanding: Systems for extracting semantic information from speech. John Wiley & Sons.
Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3987-3995. JMLR.org.