SlowGuess/NAACL_2022_OCR huggingface dataset:
We evaluate our model on response generation using the MultiWoZ-2.0 dataset (Budzianowski et al., 2018). It contains six domains (Attraction, Hotel, Restaurant, Booking, Taxi and Train) and seven DA intents ("Inform, Request, Select, Recommend, Book, Offer-Booked, No-Offer"). Following the setting in (Mi et al., 2020), the original train/validation/test splits of MultiWoZ are used. The detailed stats of the datasets are in Table 3.
We used the implementation in (Mi et al., 2020) and compare RMR_DSE to their proposed ARPER, an exemplar-replay based method. Since RMR is a generic regularized algorithm, it can be integrated to any framework by replacing either the optimizer or revising the loss function. Our comparisons to ARPER are made from two aspects: with and without exemplars.
For evaluation metrics, slot error rate (SER) and Bleu4 are used. Again, we only report bar figures with SER scores in the main pages while leaving full scores in the Appendix.
We also report metrics for two settings in LLL
following (Mi et al., 2020):
$$
\Omega_ {a l l} = \frac {1}{T} \sum_ {i = 1} ^ {T} \Omega_ {a l l, i} \Omega_ {f i r s t} = \frac {1}{T} \sum_ {i = 1} ^ {T} \Omega_ {f i r s t, i}
$$
where  $T$  is the total number of LLL tasks;  $\Omega_{all,i}$  is the average test performance on all the previous tasks after the  $i^{th}$  task has been learned;  $\Omega_{first,i}$  is performance on the first task after the  $i^{th}$  task has been learned. The former measures the test accuracy of all of the test data for tasks seen to the  $i^{th}$  point while the latter is about the model's retention of the first task.
Following (Mi et al., 2020), two Seq2seq language generation models, conditional variational encoder (CVAE) and semantic conditioned LSTM (SCLSTM) are used as the generation models. We evaluate the following LLL settings:
- Finetune: This is finetuning the model trained from the previous domain using data for the current domain.
- Full: This is using the data from all the domains.
- ARPER: We run ARPER following the setting of the original paper (Mi et al., 2020).
- EWC: ARPER without exemplars is EWC.
We make two comparisons with (Mi et al., 2020) based on exemplar numbers. The first one is when RMR_DSE and ARPER do not use exemplars. The reason for this comparison is that one of advantages of RMR_DSE is that it does not need extra storage to recover previous tasks. In addition, ARPER is equivalent to using EWC when no exemplars are used.
Figure 5 and Figure 6 show the results using different methods, where the red (CVAE) and blue (SCLSTM) bars are results of RMR_DSE, and the yellow (CVAE) and green (SCLSTM) bars are those of ARPER. We can see RMR_DSE obtains lower SER (higher Bleu4 as well, see appendix) results than ARPER in most cases. Table 4 shows that without exemplars, ARPER obtain even poorer results than Finetune while RMR_DSE achieves significantly better results than ARPER and Finetune in both  $\Omega_{all}$  and  $\Omega_{first}$ . Regarding the two seq2seq models, SCLSTM and CVAE, there are some different patterns when using RMR_DSE.
Table 2: Examples of the generated paraphrases by BART and RMR_DSE on QTW data setting.
SOURCE	BART	RMR_DSE	TARGET
Why is German Shepherd/Great Pyrenees mix coveted among breeders?	Why is German Shepherd/Great Pyrenees mix coveted from browns?	Why is German Shepherd/Great Pyrenees mix coveted among breeders?	Why is German Shepherd/Great Pyrenees mix coveted among breeders?
eyeing trump, Obama takes new action to ban arctic drilling	president Obama takes new action to ban drilling	Obama takes new action to ban arctic drilling	please save the earth mr. president . Obama takes new action to ban arctic drilling
death toll in 6.5 - magnitude earthquake in Indonesia's aceh province increase to at least 52	a 6.5 earthquake in kills at least 26 people @cnn	death toll in 6.5 - magnitude earthquake in aceh province increase to at least 52	powerfull quake kills dozens at least 25 people were killed in an earthquake that struck indonesia's aceh province
pipeline 150 miles from dakota access protests leaks gallons of oil	the new york times pipeline 150 miles from dakota access pipeline .	pipeline 150 miles from dakota access leaks gallons of oil	of oil, or gallons, have leaked from the pipeline
described by many critics as more about " exploring the meaning of human life " or " the hollow existence of the American western suburbs ", the feature film itself has explicitly defied categorization by even the anonymous filmmakers	Described by many critics as more about the meaning of human life " or " the existence of the American suburbs ", the film has explicitly defied categorization by the anonymous filmmakers .	described by many critics as more about " exploring the meaning of human life " or " the hollow existence of the American suburbs ", the film itself has explicitly defied categorization by even the anonymous filmmakers .	Described by many as about " the meaning of life " or " the hollow existence the American suburbs ", the film has defied categorization by even the filmmakers .
Table 3: Stats for full MultiWoZ-2.0 Dataset.
Domain Stats of MultiWoZ-2.0 data
domain	Attraction	Hotel	Restaurant	Booking	Taxi	Train
total	8,823	10,918	10,997	8,154	3,535	13,326
Intent Stats of MultiWoZ-2.0 data
Intent	total	Involved Domains
Inform	28,700	Attraction, Hotel, Restaurant, Taxi, Booking
Request	7,621	Attraction, Hotel, Restaurant, Taxi, Booking
Select	865	Attraction, Hotel, Restaurant, Taxi
Recommend	3,678	Attraction, Hotel, Restaurant, Taxi
Book	4,525	Booking, Taxi
Offer-Booked	2,099	Train
No-Offer	1,703	Attraction, Hotel, Restaurant, Taxi
Table 4: Average performance of continually learning 6 domains on MultiWoZ-2.0 with zero exemplars.
Ωall	Ωfirst
Methods	SER	bleu4	SER	bleu4
Finetune	64.46	36.1	107.27	25.3
ARPER on CVAE	63.54	36.00	102.87	19.22
ARPER on SCLSTM	66.87	35.64	100.56	21.09
RMR_DSE on CVAE	51.92	39.49	68.56	25.67
RMR_DSE on SCLSTM	48.79	39.86	57.18	30.32
The second comparison is made by using 250 exemplars, the same setting for ARPER as described in (Mi et al., 2020). In this setting, we also incrementally deploy MR_DSE and RMR_DSE in both CVAE and SCLSTM. In (Mi et al., 2020), using 250 exemplars for computing Fisher information matrix boosted the performances to a large degree. For equal comparison, Fisher information matrix is also utilized in MR_DSE and RMR_DSE in the
Figure 5: Slot error rate (SER) results for  $\Omega_{all}$  when zero exemplar is used for all the methods. CVAE RMR_DSE and SCLSTM RMR_DSE obtained lower SER than CVAE ARPER and SCLSTM ARPER respectively.
- computation of the loss function. Yet, we do not use it when updating the parameter importance. As shown in Table 5, MR_DSE and RMR_DSE achieve better results than ARPER in all metrics for both CVAE and SCLSTM models when we use 250 exemplars. We can also see that consistent with the results in the QTW setting, RMR_DSE always outperforms MR_DSE. These results illustrate the advantage of RMR_DSE over ARPER through the entire continual learning process.
Two additional observations can be summarized here. The first is that the results with exemplars obtain better  $\Omega_{\text{first}}$  than  $\Omega_{\text{all}}$ . This is consistent with the original paper and may indicate diverse tasks increase the difficulty of handling all the tasks. The
Figure 6: Slot error rate (SER) results for  $\Omega_{\text{first}}$  when zero exemplar is used for all the methods. CVAE RMR_DSE and SCLSTM RMR_DSE obtained lower SER than CVAE ARPER and SCLSTM ARPER respectively.
Table 5: Average performance of continually learning 6 domains on MultiWoZ-2.0 with 250 exemplars. Best performance excluding "Full" are in bold in each column.
Ωall	Ωfirst
Methods	SER	bleu4	SER	bleu4
ARPER on CVAE	5.24	58.3	2.97	62.1
ARPER on SCLSTM	5.97	56.7	3.59	61.3
MR_DSE on CVAE	4.68	59.8	2.81	62.7
MR_DSE on SCLSTM	4.95	59.9	2.60	63.2
RMR_DSE on CVAE	4.52	59.8	2.02	63.5
RMR_DSE on SCLSTM	4.38	60.3	2.12	63.6
Full	4.26	59.9	3.60	61.6
second is that if we compare both Table 4 and Table 5, we can find that AEPER severely relies on exemplars while RMR_DSE does not. This sufficiently showcases RMR_DSE functions with less need of data storage.
In this section, we provide some additional analyses and observations of this work. First, we take a closer look at the domain shift estimation (DSE) and why and how it works. Also, we will see what problems the current DSE framework has and whether we can make improvements on it. In Figure 7 we present three groups of embedding drifts to illustrate the intuitions behind the model. The ones on the left part are the TSNEs (the first two dimensions) of 10,000 embeddings generated with Quora model on Twitter data (upper) and with Twitter model on Twitter data (lower). In the middle are those of 10,000 embeddings generated with Twitter models on Wiki data (upper) and with Wiki model on Wiki data (lower). The right part are still the
Figure 7: upper left: Embedding generated with Quora Model on Twitter data of 10000 samples. Lower left: Embedding generated with Twitter Model initialized with Quora model on Twitter data of 10000 samples. Upper middle: Embedding with Twitter Model on Wiki data. Lower middle: Embedding with Wiki Model initialized with Twitter model on Wiki data. Upper right: Embedding with Twitter Model on Wiki data. Lower right: Embedding with Wiki Model initialized with Twitter model on Wiki data.
Wiki data, but the number is only 3,000. We can see clear density differences between embeddings generated by older models (upper) and newer models (lower) although their value ranges are quite similar. Hence, our meanshift algorithm can make up such differences. However, the value range similarity also partly explains why DSE does not play a big role in the performance improvements.
This may give us some hints that the embedding learning may need improvements. Right now, we only deploy label smooth cross entropy loss in our whole framework. This loss function focuses more on the label differences (vocabulary distribution in natural language generation work). A natural extension is the addition of deep contrastive learning loss. Further, current DSE cares more about the sentence embeddings. However, the decoder in our framework uses beam search on token levels. Hence, algorithms considering both sentence and token level's distribution should help the shift estimation.
In this work, we introduce RMR_DSE, a generic LLL framework for addressing forgetting in seq2seq language generation learning. Our experimental results have shown that it outperformed state-of-the-art method by a large margin in two neural seq2seq language generation tasks, paraphrase generation and dialog response generation. Future work includes applying RMR_DSE to diverse generation tasks and generation network structures.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. 2018. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pages 139-154.
Saket Anand, Sushil Mittal, Oncel Tuzel, and Peter Meer. 2013. Semi-supervised kernel mean shift clustering. IEEE transactions on pattern analysis and machine intelligence, 36(6):1201-1215.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. Multiwoz-a largescale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Kumar Dokania, Philip H. S. Torr, and Marc'Aurelio Ranzato. 2019. Continual learning with tiny episodic memories. CoRR, abs/1902.10486.
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020.
Cyprien de Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. 2019. Episodic memory in lifelong language learning. Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada.
Mihail Eric, Rahul Goel, Shachi Paul, Adarsh Kumar, Abhishek Sethi, Peter Ku, Anuj Kumar Goyal, Sanchit Agarwal, Shuyang Gao, and Dilek Hakkani-Tur. 2019. Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020.
Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-135.
Tianxing He, Jun Liu, Kyunghyun Cho, Myle Ott, Bing Liu, James R. Glass, and Fuchun Peng. 2021. Analyzing the forgetting problem in pretrain-finetuning of open-domain dialogue response models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19 - 23, 2021, pages 1121-1133. Association for Computational Linguistics.
Ferenc Huszár. 2018. Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences, page 201717042.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526.
Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. 2017. Learning from noisy labels with distillation. In ICCV.
Zhizhong Li and Derek Hoiem. 2018. Learning without forgetting. IEEE Trans. Pattern Anal. Mach. Intell., 40(12):2935-2947.
David Lopez-Paz and Marc'Aurelio Ranzato. 2017. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA.
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul A. Crook, Bing Liu, Zhou Yu, Eunjoon Cho, and Zhiguang Wang. 2020. Continual learning in task-oriented dialogue systems. CoRR, abs/2012.15504.
Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In *Psychology of learning and motivation*, volume 24, pages 109-165. Elsevier.
Fei Mi, Liangwei Chen, Mengjie Zhao, Minlie Huang, and Boi Faltings. 2020. Continual learning for natural language generation in task-oriented dialog systems. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020.
Lorenzo Pellegrini, Gabriele Graffieti, Vincenzo Lomonaco, and Davide Maltoni. 2019. Latent replay for real-time continual learning. IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020, Las Vegas, NV, USA, October 24, 2020 - January 24, 2021.
Mark Bishop Ring et al. 1994. *Continual learning in reinforcement environments*. Ph.D. thesis, University of Texas at Austin, Austin, Texas 78712.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P Lillicrap, and Greg Wayne. 2019. Experience replay for continual learning. Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. 2017. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 2990-2999.
Wenbo Wang, Yang Gao, He-Yan Huang, and Yuxiang Zhou. 2019. Concept pointer network for abstractive summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3067-3076.
Yigong Wang, Zhuoyi Wang, Yu Lin, Latifur Khan, and Dingcheng Li. 2021a. Cifdm: continual and interactive feature distillation for multi-label stream learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2121-2125.
Zhuoyi Wang, Yuqiao Chen, Chen Zhao, Yu Lin, and Latifur Khan. 2021b. Clear: Contrastive-prototype learning with drift estimation for resource constrained stream mining. In Proceedings of The Web Conference.
Zirui Wang, Sanket Vaibhav Mehta, Barnabás Póczos, and Jaime Carbonell. 2020. Efficient meta lifelong-learning with limited memory. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020.
Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Herranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. 2020. Semantic drift compensation for class-incremental learning. In CVPR, pages 6982-6991.
Friedemann Zenke, Ben Poole, and Surya Ganguli. 2017. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 3987-3995. PMLR.
A.1 Domain Order Permutation
Due to page limit, we put tables with detailed evaluations (both on current data, see Table 6 and on previous datasets, see Table 7), including bleu4, rougeL, meteors on QTW settings in appendix. In the main page, we only show figures of meteor metrics. From Table 6, we can see that RMR_DSE takes the lead in almost all metrics in three tasks. Similarly, from Table 7, we can see that RMR_DSE has less forgetting rates than all other models as well.
Besides QTW setting, we also had run other two combinations including TQW and QWT setting. The results are basically consistent with QTW setting and can reach similar conclusion. The detail results are in Table 8 and Table 9.
A.2 Metrics Details
Throughout the paper, we use those evaluation metrics that have been widely used in the previous work to measure the quality of the paraphrases. In general, BLEU measures how much the words (and/or n-grams) in the machine generated summaries appeared in the human reference summaries. Rouge measures how much the words (and/or n-grams) in the human reference summaries appeared in the machine generated summaries. Specifically, we use the library from HuggingFace to compute BLEU scores and py-rouge² to compute ROUGE scores. As BLEU and ROUGE could not measure the diversity between the generated and the original sentences, we follow unsupervised paraphrasing methods and adopt meteor to measure the diversity of expression in the generated paraphrases by penalizing copying words from input sentences. The introduction of Slot error rate, Ωall and Ωfirst can be seen in the data setting of MultiWoz2.
A.3 Bleu4 scores for MultiWoZ-2.0 dataset
Due to page limit, we put figures of Bleu4 for MultiWoZ-2.0 with zero exemplars in appendix as well. From all experiments, we can see that RMR_DSE achieves consistently better results than ARPER. Without doubt, ARPER is a strong baseline. Its adaptive EWC enables ARPER to update parameters discriminatively. However, RMR_DSE can update parameters more differentially with its memory aware penalty mechanisms.
Figure 8: Results for Bleu4 of  $\Omega_{all}$  when zero exemplar is used for all different methods. CVAE RMR_DSE and SCLSTM RMR_DSE obtained higher Bleu4 than CVAE ARPER and SCLSTM ARPER respectively.
Figure 9: Results for Bleu4 of  $\Omega_{\text{first}}$  when zero exemplar is used for all different methods. CVAE RMR_DSE and SCLSTM RMR_DSE obtained higher Bleu4 than CVAE ARPER and SCLSTM ARPER respectively.
A.4 Packages Used for Implementation
The relevant packages that we use in the implementation and their corresponding versions are as following: python==3.6.6, fairseq==1.0, torch==1.4.0, CUDA==10.2, tensorboard==1.10.0, numpy==1.14.5, scipy==1.1.0, NLTK==3.4.5 and scikit-learn==0.21.3.
A.5 Parameter Update Analysis of RMR_DSE on Different Network Structures
In our experiments, we apply RMR_DSE to different network structures, involving BART, CVAE and SCLSTM. Therefore, we need to set up quite different values for their hyperparameters as shown in Table 10. Specifically, four hyperparameters, including update frequency (update_freq, how often to update  $\Pi$ ), regularization coefficient  $reg_{\lambda}$  (the proportion of  $\Pi$ ), anneal weights ( $anneal_w$ , how
<sup>1</sup>https://huggingface.co/metrics/sacrebleu
$^{2}$  https://pypi.org/project/py-rouge/
Table 6: Results of model evaluations on QTW setting
Quora Test	Twitter Test	Wiki Test
Models	BLEU-4	ROUGE-L	METEOR	BLEU-4	ROUGE-L	METEOR	BLEU-4	ROUGE-L	METEOR
Quora-trained	36.98	58.19	60.76	2.12	6.13	5.49	4.51	11.21	12.13
Twitter-trained	3.18	11.46	9.01	36.47	47.49	45.57	4.60	9.76	7.50
Wiki_data-trained	22.38	43.44	46.23	9.32	17.93	21.03	48.03	69.70	67.43
Finetune	36.98	58.19	60.76	35.79	46.46	45.93	46.87	68.98	67.02
EWC	36.89	58.16	59.98	35.52	47.14	46.16	48.15	69.53	68.59
MR	37.98	59.19	61.11	36.98	49.39	48.02	53.93	74.49	74.53
RMR	38.46	59.48	61.14	38.94	51.23	47.12	54.12	74.98	75.13
Full	37.99	59.33	61.04	39.53	51.33	47.64	55.93	76.56	76.41
Table 7: Results of all the methods when testing new models on previous domains.
Quora test with Model trained with Twitter	Quora test with Model trained with Wiki_data
Models	BLEU-4	ROUGE-L	METEOR	Models	BLEU-4	ROUGE-L	METEOR
Quora-trained	36.98	58.19	60.76	Quora-trained	36.98	58.19	60.76
Finetune	20.77	30.80	41.75	Finetune	22.83	42.16	42.03
EWC	21.63	31.53	42.03	EWC	24.63	44.35	43.02
DSE	21.58	31.95	42.98	DSE	23.79	43.49	43.35
MR	25.47	35.88	45.27	MR	28.44	47.37	55.43
RMR	26.97	36.39	47.26	RMR	29.72	49.15	57.15
RMR_DSE	27.74	36.98	48.38	RMR_DSE	30.71	49.43	57.99
Twitter test with Model trained with Wiki_data
Models	BLEU-4	ROUGE-L	METEOR
Twitter-based	36.47	47.49	45.57
Finetune	19.99	37.20	41.57
EWC	18.84	38.65	43.33
DSE	20.78	40.0	42.75
MR	21.92	38.69	44.36
RMR	24.15	42.11	45.59
RMR_DSE	26.73	43.85	46.23
Table 8: Results of model evaluations with TQW setting
Twitter Test	Quora Test	Wiki Test
Models	bleu4	rougeL	meteor	bleu4	rougeL	meteor	bleu4	rougeL	meteor
Finetune	36.47	47.49	45.57	34.32	55.63	58.93	44.94	67.87	66.15
EWC	36.55	48.32	46.73	34.37	54.32	59.31	48.72	68.21	69.14
MR_DSE	36.95	48.87	47.24	36.83	57.45	60.78	53.24	72.64	72.93
RMR_DSE	37.26	49.33	48.58	36.90	58.86	61.33	54.53	73.25	73.50
Full	39.53	51.33	47.64	36.81	58.39	60.70	55.93	76.56	76.41
Table 9: Results of model evaluations with QWT setting
Quora Test	Wiki Test	Twitter Test
Models	bleu4	rougeL	meteor	bleu4	rougeL	meteor	bleu4	rougeL	meteor
Finetune	35.93	56.32	59.23	45.12	53.23	67.78	36.82	47.99	46.45
EWC	35.85	55.64	59.95	49.14	67.99	69.85	36.96	47.45	46.94
MR_DSE	35.53	58.47	60.59	54.32	73.92	72.37	37.08	48.43	47.91
RMR_DSE	35.61	59.38	61.58	53.87	74.63	73.13	37.58	50.36	47.52
Full	36.81	58.39	60.70	55.93	76.56	76.41	39.53	51.33	47.64
Table 10: setting for prime hyperparameters
hyperparameter	BART	CVAE	SCLSTM
updateFreq	2	2	2
regλ	0.9	0.1	0.01
annealw	0.1	0.01	0.05
pretraincof	5000	500	50
much we take parameter differences into consideration) and pretrain coefficient (pretraincof, the quadratic penalty derived from fisher information matrix, namely  $\Pi$ ), are the most important ones.
For all of them, it looks the update frequency for  $\Pi$  can be once every two epochs. However, the other three are remarkably different. It seems to show that more complex network structures need higher penalty coefficients. Further, the value of  $\Pi$  seems quite related to the complexity of network structure. The three models need 5000, 500 and 50 respectively since BART has more complex network structure than CVAE and CVAE more complex than SCLSTM.