SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Privacy Concerns Associated with Data Storage. Currently, some different regulations and rules limit the storage time of users' data. Usually, the service providers do not own the data forever and are obligated to erase it after a specific duration. Sometimes, the data is available only in the form of a stream, and it never gets stored. But most of the time, data is available for a short period of enough to perform a few rounds of training. In this way, if multiple service providers participate in federated learning, their data would dynamically change as they delete old data and acquire new ones.
MFCL and Batch Statistics. MFCL benefits from Batch Statistics Loss  $(\mathcal{L}_{BN})$  in training the generative model. However, some defense mechanisms suggest not sharing local Batch Statistics with the server. While training the generative model without the  $\mathcal{L}_{BN}$  is still possible, it can reduce the accuracy. Addressing this is an interesting future direction.
F Hyperparameters
Table 7 presents some of the more important parameters and settings for each experiment.
Table 7: Parameter Settings in different datasets
Dataset	CIFAR-100	TinyImageNet	SuperImageNet-L
Data Size	32 × 32	64 × 64	224 × 224
# Tasks	10	10	10
# Classes per task	10	20	5
# Samples per class	500	500	7500
LR	All task start with 0.1 and exponentially decay to 0.01
Batch Size	32	32	32
Synthetic Batch Size	32	32	32
FL round per task	100	100	100
Local epoch	10	10	1
G Hyperparameter tuning for MFCL
Hyperparameters can play an essential role in the final performance of algorithms. In our experiments, we have adapted the commonly used parameters, and here, we show how sensitive the final performance is regarding each hyperparameter. This is particularly important because hyperparameter tuning is very expensive in federated learning and can be unfeasible in continual learning. To this aim, we change one parameter at a time while fixing the rest. In Table 8, we report the final  $\tilde{\mathcal{A}}$  of each hyperparameter on CIFAR-100 datasets with 10 tasks.
$w_{div}$ : Weight of diversity loss  $(\mathcal{L}_{div})$
$w_{BN}$ : Weight of Batch Statistics loss  $(\mathcal{L}_{BN})$
$w_{pr}$ : Weight of Image Prior loss  $(\mathcal{L}_{FT})$
Z_dim: Input noise dimension for training the  $\mathcal{G}$  model.
gen_epoch: Number of iteration to train the  $\mathcal{G}$  model.
This is the setting that we used  $w_{div} = 1$ ,  $w_{BN} = 75$ ,  $w_{pr} = 0.001$ ,  $Z\_dim = 200$ ,  $gen\_epoch = 5000$  and the average accuracy equals  $45.1\%$ . (There may be a minor difference between this value and the result in the main manuscript. This discrepancy arises because we only ran the ablation for a single seed, whereas the results reported in the primary manuscript are the average of three different seeds.)
Table 8: Effect of different hyperparameters on the final  $\widetilde{A}$  (in %) for CIFAR-100 dataset.
wdiv	A	wBN	A	wpr	A	Z_dim	A	gen_epoch	A
0.1	44.35	0.1	40.12	0.0001	43.10	110	42.39	100	40.77
0.5	44.37	1	43.90	0.001	45.1	200	45.1	5000	45.1
1	45.1	10	44.77	0.01	43.56	1000	45.01	10000	43.35
2	44.08	75	45.1	0.1	44.73
5	44.57	100	45.02	1	44.37
This table shows how robust the final performance is with respect to each parameter, which is preferred both in federated and continual learning problems.
H Comparison between MFCL and FedCIL
Here, we would like to highlight some distinctions between our algorithm and FedCIL, both of which aim to alleviate catastrophic forgetting using generative models.
- In FedCIL, clients train the local generative model every round, which adds great computational overhead. On the other hand, in our approach, the generative model is trained on the server and only once per task.
- Training models in GANs usually require a large amount of data that is not commonly available, especially on edge devices. Our data-free generative models address this issue.
- Training the generative model directly from the training dataset may pose a risk of exposing sensitive training data, which contradicts the goal of FL. On the other hand, MFCL uses only the information from the global model.
- FedCIL is limited to simpler datasets and FL settings, such as MNIST and CIFAR10, with fewer clients and less complex architectures. In contrast, our approach can handle more complex datasets, such as CIFAR100, TinyImageNet, and SuperImagenet, with a much larger number of clients.
- Training GAN models usually require more careful hyperparameter tuning. To train FedCIL for TinyImageNet and SuperImageNet, we tried SGD and Adam optimizers with learning rates  $\in \{0.1, 0.05, 0.01\}$  and local epoch  $\in \{1, 2\}$ . Furthermore, we adopt a generative model architecture with a similar input dimension and a total number of parameters in MFCL. However, the model did not converge to a good performance. While a more extensive hyperparameter search might improve the results, it can indicate the difficulty of the hyperparameter tuning of this algorithm. It is worth mentioning that in order to train the CIFAR-10 dataset, we used a local epoch  $8 \times$  larger than the other baselines; otherwise, the performance on this dataset would also degrade.
In conclusion, FedCIL can be a good fit for a cross-silo federated learning setting with only a few clients, each possessing a large amount of data and computing resources. Meanwhile, while still applicable in the above setting, our method is also suitable for edge devices with limited data and power.