SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem D.7 (Composition Theorem [28]). For any  $\epsilon > 0, \delta \in [0,1]$ , and  $\hat{\delta} \in [0,1]$ , the class of  $(\epsilon, \delta)$ -differentially private mechanisms satisfies  $(\hat{\epsilon}_{\hat{\delta}}, 1 - (1 - \hat{\delta})\Pi_i(1 - \delta_i))$ -differential privacy under  $k$ -fold adaptive composition for  $\hat{\epsilon}_{\hat{\delta}} = \min \{k\epsilon, (e^{\epsilon} - 1)\epsilon k / (e^{\epsilon} + 1) + \epsilon \sqrt{2k\log(e + \sqrt{k\epsilon^2 / \hat{\delta}})}, (e^{\epsilon} - 1)\epsilon k / (e^{\epsilon} + 1) + \epsilon \sqrt{2k\log(1 / \hat{\delta})}\}$ .
Proof. Definition D.6 gets  $\mathcal{L}_{\mathrm{Pri}}^{(o)}$  on an outcome variable  $o$  over databases  $D$  and  $D'$ . This proof starts with a particular random mechanism  $\mathcal{M}^{\dagger}$  with further generalization. The mechanism  $\mathcal{M}^{\dagger}$  does not depend on the database or the query but relies on hypothesis  $\mathfrak{hp}$ . For  $\mathfrak{hp} = 0$ , the outcome  $O_{i}$  of  $\mathcal{M}_i^\dagger$  is independent and identically distributed from a discrete random distribution  $O^{\mathrm{hp}=0} \sim \mathcal{P}^{\dagger,0}$ .  $\mathcal{P}^{\dagger,0}(o)$  is defined to be:  $\delta$  for  $o = 0$ ;  $(1 - \delta)e^{\epsilon} / (1 + e^{\epsilon})$  for  $o = 1$ ;  $(1 - \delta) / (1 + e^{\epsilon})$  for  $o = 2$ ;  $0$  for  $o = 3$ . For  $\mathfrak{hp} = 1$ , the outcome  $O_i$  of  $\mathcal{M}_i^\dagger$  is  $O^{\mathrm{hp}=1} \sim \mathcal{P}^{\dagger,1}$ .  $\mathcal{P}^{\dagger,1}(o)$  is defined to be:  $0$  for  $o = 0$ ;  $(1 - \delta) / (1 + e^{\epsilon})$  for  $o = 1$ ;  $(1 - \delta)e^{\epsilon} / (1 + e^{\epsilon})$  for  $o = 2$ ;  $\delta$  for  $o = 3$ .
Let  $\mathcal{R}(\epsilon, \delta)$  be privacy region of a single access to  $\mathcal{M}^\dagger$ . The privacy region consists of two rejection regions with errors, i.e., rejecting true null-hypothesis (type-I error) and retaining false null-hypothesis (type-II error). Let  $\epsilon_k^\dagger, \delta_k^\dagger$  be  $\mathcal{M}_i^\dagger$ 's parameters for defining privacy.  $\mathcal{R}(\mathcal{M}, D, D')$  of any mechanism  $\mathcal{M}$  can be regarded as an intersection of  $\{(\epsilon_k^\dagger, \delta_k^\dagger)\}$  privacy regions. For an arbitrary mechanism  $\mathcal{M}$ , we need to compute its privacy region using the  $(\epsilon_k^\dagger, \delta_k^\dagger)$  pairs. Let  $D, D'$  be neighbouring databases and  $\mathcal{O}$  be the outputting domain. Define (symmetric)  $\mathcal{P}$ ,  $\mathcal{P}'$  to be probability density function of the outputs  $\mathcal{M}(D), \mathcal{M}(D')$ , respectively. Assume a permutation  $\pi$  over  $\mathcal{O}$  such that  $\mathcal{P}'(o) = \mathcal{P}(\pi(o))$ .
Let  $S$  denote the complement of a rejection region. Since  $\mathcal{R}(\mathcal{M},D,D^{\prime})$  is convex, we have  $1 - \mathcal{P}(S)\geq -e^{\epsilon_k^\dagger}\mathcal{P}'(S) + 1 - \delta_k^\dagger \Rightarrow \mathcal{P}(S) - e^{\epsilon_k^\dagger}\mathcal{P}'(S)\leq \delta_k^\dagger$ . Define  $\mathrm{Dt}_{\epsilon^{\dagger}}(\mathcal{P},\mathcal{P}') = \max_{S\subseteq \mathcal{O}}\{\mathcal{P}(S) - e^{\epsilon^{\dagger}}\mathcal{P}'(S)\}$ . Thus,  $\mathcal{M}$ 's privacy region is the set:  $\{(\epsilon_k^\dagger,\delta_k^\dagger):\epsilon_k^\dagger \in [0,\infty)\}$  s.t.  $\mathcal{P}(o) = e^{\epsilon_k^\dagger}\mathcal{P}'(o),\delta_k^\dagger = \mathrm{Dt}_{\epsilon_k^\dagger}(\mathcal{P},\mathcal{P}')$ . Next, we consider composition on random mechanisms  $\mathcal{M}_1,\ldots ,\mathcal{M}_i$ . By accessing  $\mathcal{M}_i^\dagger$ ,  $\mathcal{P}(O^{1,\mathrm{hp}} = o_1,\dots,O^{i,\mathrm{hp}} = o_i) = \Pi_{j = 1}^i\mathcal{P}^{\dagger,\mathrm{hp}}(o_j)$ . By algebra on two discrete distributions,  $\mathrm{Dt}_{(i - 2j)\epsilon}(\mathcal{P}^i,(\mathcal{P}')^i) = 1 - (1 - \delta)^i + (1 - \delta)^i\sum_{l = 0}^{j - 1}\binom{i}{l}(e^{\epsilon (i - l)} - e^{\epsilon (i - 2j + l)}) / (1 + e^{\epsilon})^k$ . Hence, privacy region is an interaction of  $i$  regions, parameterized by  $1 - (1 - \hat{\delta})\Pi_i(1 - \delta_i)$ .
Now, by Lemma D.2, we get Theorem 3.4 directly. In summary, FedFed protects two types of data features using two different protective manners, i.e., small noise for performance-sensitive features and extremely large noise for performance-robust features, and thus attains higher model performance and stronger security in the same time.
E More Results on Attack
E.1 More Details on Model Inversion Attack
In this section, we present additional results of the model inversion attack. When the central server assumes the role of the attacker without access to the original data, it exhibits a diminished performance in attacking the model. The raw data  $\mathbf{x}$  of Figure 3 (a) and (b) can be found in Figure 6(a). As we can see,  $\mathbf{x}_s$  still causes privacy leakage and Figure 6(b) also gives the intuitive necessity of DP protection on  $\mathbf{x}_s$ .
Inspired by generative model inversion methods [40, 62], we conduct another experiment where one of the clients acts as the attacker. Similar to GMI [40], the method involves leveraging an auxiliary dataset and a public dataset to launch an attack on the target model. In our experiments, we utilize the globally shared dataset as the auxiliary dataset, while the local private data serves as the public
(a) Raw data  $\mathbf{x}$
(b) Attack on  $\mathbf{x}_s$
(c) Attack on  $\mathbf{x}_p$
Figure 6: Model inversion attack. White-Box attack performance-sensitive features  $\mathbf{x}_s$  and globally shared data  $\mathbf{x}_p, \mathbf{x}_p = \mathbf{x}_s + \mathbf{n}, \mathbf{n} \sim \mathcal{N}(\mathbf{0}, \sigma_s^2\mathbf{I})$ .
Figure 7: Generative-based model inversion attack results.
dataset because the client has access to its own local data. Under this scenario (results in Figure 7), the attacker's performance is better than when the server acts as the attacker, while the attacker is still unable to recover data from the shared features. The model architectures of generative-based model inversion attack are listed in Table 8.
E.2 More Details on Membership Inference Attack
Figure 8: Attack process. Perform membership inference attack on global model per 10 communication rounds.
Is Theorem 3.3 true empirically?
Membership inference attack(MIA) [41, 15] attempts to infer whether a particular data point is in the target model's training set. Specifically, the global model on the server side can be regarded as the target model in FL. In our setting, we train a shadow model with globally shared data. Then the membership attack can be regarded as a binary classification task, member data or non-member data. The input of the attack model is the top-k vector of the output from the shadow model. To compare the superiority of sharing partial data rather than the complete data with DP, we conduct MIA to explore the divergence experimentally. Sharing raw data cause more information leakage (higher
Table 8: The model architectures used in generative-based model inversion attack
The encoder structure of the generator takes as input the globally shared data
Type	Kernel	Dilation	Stride	Outputs
conv	5x5	1	1x1	32
conv	3x3	1	2x2	64
conv	3x3	1	1x1	64
conv	3x3	1	2x2	128
conv	3x3	1	1x1	128
conv	3x3	1	1x1	128
conv	3x3	2	1x1	128
conv	3x3	4	1x1	128
conv	3x3	8	1x1	128
conv	3x3	16	1x1	128
The encoder structure of the generator takes as input the latent vector
linear	2048
deconv	5x5	1/2x1/2	256
deconv	5x5	1/2x1/2	128
The decoder structure of the generator
deconv	5x5	1/2x1/2	128
deconv	5x5	1/2x1/2	64
conv	3x3	1x1	32
conv	3x3	1x1	3
The global discriminator structure
conv	3x3	2x2	64
conv	3x3	2x2	128
conv	3x3	2x2	256
conv	3x3	2x2	512
conv	1x1	4x4	1
recall of MIA) than partial data with the same DP level. However, the FedFed needs a relatively small noise  $\sigma$  to achieve comparable protection. The attack process is shown in Figure 8.
F Supplementary for Experiments
F.1 Visualization of Data Heterogeneity
(a) LDA  $(\alpha = 0.1)$
(b)  $\# C = 2$
Figure 9: Data distribution in various FL heterogeneity scenarios. Different colours denote different labels and the length of each line denotes the data number. As we can see, in our FL setting, we mainly perform on two kinds of Non-IID scenarios, including label skew and quantity skew.
(c) Subset
We show the visualization of data distribution in Figure 9. The LDA partition and the  $\# C = 2$  partition have the label skew and the quantity skew simultaneously. And the Subset partition only has
the label skew.  $\# C = k$  means each client only has  $k$  different labels from dataset, and  $k$  controls the unbalanced degree. The subset method makes each client have all classes from the data, but one dominant class far away outnumberers other classes.
F.2 More Guessing Games
Figure 10: Guessing Games Question: Which one is the first image from in each row? A or B or C?
Let's play more guessing games here. We selected samples from the same category for the purpose of reducing the difficulty. The answer in Figure 1 is C. The answers in Figure 10 are (B, A, A) in order.
F.3 Sharing Protected Partial Features vs. Protected Raw Data
Figure 11: Performance on different information sharing strategies with DP protection: raw data and  $\mathbf{x}_p$  under the same protection strength.
In this section, we compare the performance divergence of applying DP to protect different sharing information, i.e., performance-sensitive features  $\mathbf{x}_p$  and raw data  $\mathbf{x}$ . The results is shown in Figure 11
F.4 More experimental results
We provide more comparison results in this section to demonstrate the superiority of FedFed. Figure 12, Figure 13, Figure 14, Figure 15 have shown our enhancement in four datasets. In this paper, we mainly have four settings in our experiments: (1)  $\alpha = 0.1$ ,  $E = 1$ ,  $K = 10$ ; (2)  $\alpha = 0.1$ ,  $E = 5$ ,  $K = 10$ ; (3)  $\alpha = 0.05$ ,  $E = 1$ ,  $K = 10$ ; (4)  $\alpha = 0.1$ ,  $E = 1$ ,  $K = 100$ .
Table 9: Different sampling rates of FedFed under  $\alpha = {0.1},E = 1,K = {100}$  over CIFAR-10
Sampling rates	5%	10%	20%	40%	60%
Accuracy	83.67	84.06	87.98	89.17	89.62
Round to reach target accuracy	182	163	90	70	61
(a)  $\alpha = 0.1, E = 1, K = 10$
Figure 12: Convergence and test accuracy comparison on CIFAR-10.
(b)  $\alpha = 0.1, E = 5, K = 10$
(c)  $\alpha = 0.1, E = 1, K = 100$
In addition, we investigate the impact of varying sampling rates on the performance of our FL system, and present the results in Table 9. We observe that increasing the sampling proportion leads to gradual improvements in the model's performance, as well as faster convergence. This is due to a larger number of clients participating in each round, resulting in more consistent update directions for the aggregated model and the global model.
We also conduct tests on FedFed using various levels of heterogeneity, such as  $\alpha = 0.5$  and  $\alpha = 1$ . The results are presented in Tables 10 and 11. As the value of  $\alpha$  increases, the original performance of FedAvg demonstrates significant improvement, while the performance enhancement of FedAvg deployed with FedFed is comparatively lower. This discrepancy can be attributed to the reduction in heterogeneity as  $\alpha$  increases. When  $\alpha$  reaches 1.0, the data distribution among clients becomes nearly homogeneous. As a result, the effectiveness of FedFed in mitigating heterogeneity through information sharing diminishes significantly.
F.5 More Results on Impact of DP noise
In this section, we present additional results on the impact of DP noise on three other datasets, as shown in Figure 16. Our findings indicate that as the level of DP noise increases, the privacy budget decreases and the ability to protect information increases, but the performance of the model decreases accordingly. Thus, it is crucial to find a balance between privacy and performance in practical applications of our FedFed.
(b)  $\alpha = 0.1, E = 5, K = 10$
Figure 13: Convergence and test accuracy comparison on CIFAR-100.
(c)  $\alpha = 0.1, E = 1, K = 100$
(b)  $\alpha = 0.1, E = 5, K = 10$
(c)  $\alpha = 0.05, E = 1, K = 10$
Figure 14: Convergence and test accuracy comparison on SVHN.
(a)  $\alpha = 0.1, E = 1, K = 100$
Figure 15: Convergence and test accuracy comparison on FMNIST.
F.6 More Explanation: Performance-Sensitive Features vs. Performance-Robust Features
For an intuitive understanding of why there is no drastic performance degradation for utilizing  $\mathbf{x}_s$  as a substitute for raw data  $\mathbf{x}$ . Formally, e.g. in the classification task, a classifier trained by  $\mathbf{x}$  gets comparable test accuracy on  $\mathbf{x}_s$  giving the verification that  $\mathbf{x}_s$  contains the primary features for generalization and vice versa. Specifically, three classifiers trained on  $\mathbf{x}$ ,  $\mathbf{x}_s$ , and  $\mathbf{x}_r$ , Then test the generalization ability on  $\mathbf{x}$ ,  $\mathbf{x}_r$ , and  $\mathbf{x}_s$ , separately. The result is shown in Figure 17.
As described above,  $\mathbf{x}_r$  extract from  $\mathbf{x}$  in an IB manner contains more visual information about private data  $\mathbf{x}$  than  $\mathbf{x}_s$ . To measure the visual similarity among performance-robust features  $\mathbf{x}_r$ , performance-sensitive features  $\mathbf{x}_s$ , and raw data  $\mathbf{x}$ . We choose PSNR [63, 64], prominent visual information comparisons of images, to calculate pixels error, the result is shown in Table 12.
Consequently, higher PSNR denotes that  $\mathbf{x}_r$  has the most visual information of  $\mathbf{x}$  than  $\mathbf{x}_s$ . While  $\mathbf{x}_r$  shares similarities with  $\mathbf{x}$  in terms of its form or structure, it exhibits limitations that hinder its ability to enhance performance. At the same time,  $\mathbf{x}_s$  has the commensurate generalization ability like  $\mathbf{x}$  to achieve our goal.
F.7 Overheads Analysis of FedFed
For the extra computation induced by FedFed, we provide two views to demonstrate the limited costs of extra computation, i.e., training time and FLOPs.
Table 13 shows that training a generator requires less than  $10\%$  of the time needed to train a classifier. Therefore, the additional training time required for the generator is limited.
Table 10: Top-1 Accuracy of  $\alpha  = {0.5},E = 1,K = {10}$  with  ${50}\%$  sampling rate.
CIFAR-10	FMNIST	SVHN	CIFAR-100
FedAvg	87.68	90.32	91.11	68.98
FedFed (Ours)	93.21	94.01	93.71	69.52
Table 11: Top-1 Accuracy of  $\alpha  = {1.0},E = 1,K = {10}$  with  ${50}\%$  sampling rate.
CIFAR-10	FMNIST	SVHN	CIFAR-100
FedAvg	89.45	92.38	92.03	69.02
FedFed (Ours)	94.01	94.31	93.89	70.31
(a) Test accuracy on CIFAR10 with different noise level  $\sigma_s^2$
(b) Test accuracy on SVHN with different noise level  $\sigma_s^2$
(c) Test accuracy on CIFAR100 with different noise level  $\sigma_s^2$
Figure 16: The impact of DP noise on different datasets.
(a) The generalization ability of classifier trained by raw data  $\mathbf{x}$ .
(b) The generalization ability of classifier trained by performance-sensitive features  $\mathbf{X}_s$
(c) The generalization ability of classifier trained by performance-robust features  $\mathbf{x}_r$ .
Figure 17: The generalization ability graphs of three classifiers trained by different data types. The generalization ability means a well-trained model tests unseen information and measures the capability to finish a certain task, i.e., classification test accuracy. In our setting, we evaluate the generalization ability of classifiers on different information forms: raw data  $\mathbf{x}$ , performance-sensitive features  $\mathbf{x}_s$ , performance-robust features  $\mathbf{x}_r$ .
Table 12: The visual similarity of various feature types. PSNR of  $\mathbf{x}$  and  ${\mathbf{x}}_{r},\mathbf{x}$  and  ${\mathbf{x}}_{s}$  .
XT	XS
X	31.47 ± 1.57	18.87 ± 0.98
Tables 15 and 14 show the model complexity results, which indicate that the computation required for the generator is approximately  $8.2\%$  of that required for the classifier. Additionally, the number of communication rounds needed for the generator is significantly lower than that required by the classifier, with only 15 rounds compared to 1000 rounds for the classifier (namely,  $T_{d} = 15$  and  $T_{r} = 1000$ ).
The use of FedFed incurs additional memory usage, as the size of the globally-shared dataset is equivalent to the combined size of all clients' data. This may have limited impacts on the bandwidth, as the generated data is only sent once. To quantitatively measure its effects, we compared the size of the dataset with that of the classifiers in Table 16. Sharing the generated dataset is equivalent to sharing classifiers in multiple rounds. For example, for CIFAR10, the data requires approximately 586MB of memory, while the classifier requires about 48MB. Thus, sharing the global dataset requires the same amount of communication as sending a classifier in approximately 14 communication rounds.
Based on the aforementioned analysis, we provide a comprehensive examination of the communication overheads associated with the FedFed. Consider  $K$  clients in the FL system. Let  $m$  be the size of a single local model. The size of local private data is  $\| \mathcal{D}_k\| = a$ . Then, the ratio of the entire dataset to a model parameter is  $\gamma$ , where  $\gamma = \frac{aK}{m} \approx 14$ . The extra communication cost for a single client is  $(m + m)*T_d + a + aK$  where  $(m + m)*T_{d}$  denotes the cost of download/upload models for  $T_{d}$  rounds in Algorithm 1. Here, the  $a$  denotes the performance-sensitive features sent by each client, and  $aK$  is the data received by each client from the globally shared dataset. In the general process of FL, the overall communication costs are  $(m + m)T_r\beta$ , where  $\beta$  is the sampling rate of a client. Therefore, the ratio of the extra communication overhead to the general FL process is:
$$
\frac {(m + m) \cdot T _ {d} + a (K + 1)}{(m + m) \cdot T _ {r} \cdot \beta} = \frac {T _ {d}}{T _ {r} \cdot \beta} + \frac {a (K + 1)}{2 m \cdot T _ {r} \cdot \beta} = \frac {T _ {d}}{T _ {r} \cdot \beta} + \frac {\gamma}{2 T _ {r} \cdot \beta} + \frac {\gamma}{2 K \cdot T _ {r} \cdot \beta} \tag {17}
$$
Here, we detail two examples in our experiments:
* For  $K = {10},{T}_{d} = {15},{T}_{r} = {1000}$  ,and  $\beta  = {50}\%$  ,the extra communication costs are approximately 4.54%.
* When  $K = 100, T_{d} = 15, T_{r} = 1000$ , and  $\beta = 10\%$ , the extra communication costs are approximately  $22.07\%$ .
To facilitate the further deployment of FedFed, we offer three strategies tailored to different storage hardware configurations:
1. One-time download: Local clients download the globally shared dataset once. A globally shared dataset costs approximately  $14 \times$  the storage of a classifier model.
2. Partial download: A small portion of the globally shared dataset is selected and downloaded. This strategy incurs approximately  $1.5 \times$  communication cost compared to the previous strategy, while the storage required by the clients is the same as that of local private data. This represents a storage-friendly choice that may involve a trade-off in terms of performance.
3. Intermittent download: A small set of globally shared dataset is downloaded after every  $Z$  rounds. This approach reduces the communication overhead to  $\frac{1}{Z}$  of that of strategy 2 while maintaining the storage overhead at the size of the local data.
Table 13: Comparison of training time between Classifier and Generator
Training time of Generator	5,251 s
Training time of Classifier	62,901 s
Table 14: Parameters of Classifier and Generator
Parameters of Generator	48M
Parameters of Classifier	42MB
G More Related Works
G.1 Federated Learning with Heterogeneous Data.
FL allows the distributed clients to train a model across multiple datasets jointly. It "protects" user privacy by controlling data accessibility among different clients, i.e., only the data owner has the right
Table 15: FLOPs of Classifier and Generator
FLOPs of training Generator	16.03M FLOPs
FLOPs of training Classifier	556.66M FLOPs
Table 16: Parameters of Local Classifier and Globally-shared Data
Parameters of Local Classifier	42MB
Parameters of Globally-shared Data	586MB
Table 17: Existing method about information sharing strategy to mitigate data heterogeneity in Federated Learning.
Shared Info	Protection on Shared Info	Attack Test
FD+FAug [46]	Model output & Synthetic info	X	—
FedMD [65]	logits	X	—
XorMixFL [48]	Statistic of data	X	—
FedDF [66]	Statistic of logits	X	—
FedProto [50]	Abstract class prototypes	X	—
FedFTG [67]	Synthetic info	X	—
CCVR [10]	Statistic of logits	X	Model inversion attack
Fed-ZDAC [12]	Batch normalization features	✓	—
FedAUX [68]	Synthetic info	✓	—
FedFed (ours)	Protected partial features	✓	Model inversion attackMembership inference attack
to access the corresponding data. FedAvg [4] is the seminal work designed to reduce communication via more local training epochs and fewer communication rounds. In the following, many works [9, 5] observe that the FedAvg's divergence is considerable compared with centralized training if clients hold heterogeneous distribution. Even worse, the gap accumulates as the weight aggregates, potentially hurting model performance.
Recently, a series of works have tried to calibrate the updated direction of local training from the global model. FedProx [6] adds an  $L_{2}$  distance as the regularization term in the objective function, providing a theoretical guarantee of convergence. Similarly, FedIR [42] operates on a mini-batch by self-normalized weights to address the non-identical class distribution. SCAFFOLD [7] restricts the model using the previous knowledge. Besides, MOON [43] introduces contrastive learning at the model level to correct the divergence between clients and the server.
Meanwhile, recent works propose designing new model aggregation schemes. FedAvgM [33] performs momentum on the server side. FedNova [22] adopts a normalized averaging method to eliminate objective inconsistency. A study [69] also indicates that biasing client selection with higher local loss can speed up the convergence rate. The coordinate-wise averaging of weights also induces noxious performance. FedMA [44] conducts a Bayesian non-parametric strategy for heterogeneous data. FedBN [45] focuses on feature shift Non-IID and performs local batch normalization before averaging models.
Another existing direction for tackling data heterogeneity is sharing data. This line of work mainly assembles the data of different clients to construct a global IID dataset, mitigating client drift by replenishing the lack of information of clients [9]. Existing methods include synthesizing data based on the raw data by GAN [46]. However, the synthetic data is generally relatively similar to the raw data, leading to privacy leakage at some degree. Adding noise to the shared data is another promising strategy [70, 71]. Some methods employ the statistics of data [48] to synthesize for sharing, which still contains some raw data content. Other methods distribute intermediate features [12], logits [49, 10], or learn the new embedding [50]. These tactics will increase the difficulty of privacy protection because some existing methods can reconstruct images based on feature inversion methods [51]. Most of the above methods share information without a privacy guarantee or with strong privacy-preserving but poor performance, posing the privacy-performance dilemma.
Concretely, in FD [46] all clients leverage a generative model collaboratively for data generation in a homogeneous distribution. For better privacy protection, G-PATE [47] performs discriminators with local aggregation in GAN. Fed-ZDAC(Fed-ZDAS) [12], depending on which side to play augmentation, introduce zero-shot data augmentation by gathering intermediate activations and batch