SlowGuess/NeurIPS_2023_OCR huggingface dataset:
emerge only in case of using the multiplicative noise drawn from  $\mathcal{N}(\mathbf{1},\beta^2\mathbf{I})$  for Theorem 2. Importantly, the multiplicative noise  $z\sim \mathcal{N}(\mathbf{1},\beta^{2}\mathbf{I})$  does not introduce new semantics into feature vector  $\pmb{a}$  by operation  $z\odot a$ , i.e., only feature semantics that are present in feature vector (features of  $\pmb{a}$  that are non-zero) that describe object are modulated. Yet, this multiplicative noise does help control the Rademacher Complexity (RC) due to Lemma 2 and Theorem 1. However, imposing a generic penalty of form  $\beta (\sum_{k}\nabla_{k}^{2}f(\pmb {a}) + \sum_{j,k}(H_{jk}^{(f)}(\pmb {a}))^{2}$  that arises when the additive noise  $z\sim \mathcal{N}(\mathbf{0},\beta^{2}\mathbf{I})$  is applied to  $\pmb{a}^{(i)}$ , i.e.,  $z + a$ .
Let  $z \sim \mathcal{N}(\mathbf{0}, \beta^2\mathbf{I})$ . The consistency loss regularization with the additive noise injection is:
$$
\mathbb {E} _ {\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}} \left[ \left(f (\boldsymbol {z} _ {1} + \boldsymbol {a}) - f (\boldsymbol {z} _ {2} + \boldsymbol {a})\right) ^ {2} \right] = 2 \beta^ {2} \sum_ {k} \nabla_ {k} ^ {2} f (\boldsymbol {a}) + \beta^ {4} \sum_ {j} \sum_ {k} (H _ {j k} ^ {(f)} (\boldsymbol {a})) ^ {2}.
$$
Such a variant means that semantics that are not present in  $\mathbf{a}$  may be "activated" by the noise, drastically altering the meaning of  $\mathbf{a}$  and damaging the information it carries about image/object.
While directly applying gradient penalization does not introduce noise, its connection to the additive noise suggests it will have a negative effect on semantics. Table 8 evaluates the multiplicative vs. additive noise modulators, and the direct penalties. Table 9 provides experimental results for different gradient regularization variants. Furthermore, Figure 7 shows that NICE obtains best accuracy than other variants, showing that the multiplicative noise modulation preserves the semantics better than other variants, justifying the rationale behind NICE.
Table 9: Results for ALGP (adaptive latent gradient penalization),  $\mathrm{NICE}_{add}$  (consistency regularization with additive noise), AWR (adaptive weight regularization) on  $10\%$  CIFAR-10/100 using OmniGAN ( $d' = 256$ ).
Method	10% CIFAR-10	10% CIFAR-100
IS ↑	tFID ↓	vFID ↓	IS ↑	tFID ↓	vFID ↓
OmniGAN	8.49	22.24	26.33	8.19	45.41	50.33
+ALGP	8.52	19.15	22.72	9.18	32.98	37.51
+AWR+ALGP	8.72	16.82	20.45	10.14	26.44	30.23
+NICEadd	8.64	17.94	21.59	9.34	28.59	33.02
+NICE	9.26	7.23	11.08	11.50	16.91	21.56
H Training overhead with and without NICE
We provide number of parameters/multiply-accumulate (MACs) (for both generator and discriminator), number of GPUs and cost of time (seconds per 1000 images, secs/kimg) in Table 10. With the efficient implementation (Figure 1), our NICE only introduces small fraction of cost of time on high-resolution datasets.  $15.08\%$  on ImageNet and  $18.02\%$  on FFHQ/low-shot datasets. Figure 8 illustrates the FID gain vs. speed, which shows that a slightly increase in training time yields substantial improvements, particularly when compared to recent state-of-the-art approaches like FakeCLR [32] and InsGen [63]. We believe the increased computational overhead is outweighed by the considerable benefits and the improvements justify the extra fraction of time.
Table 10: Number of parameters, MACs and secs/kimg for models with vs. without the NICE. Experiments were performed on NVIDIA A100 GPUs.
Dataset	Architecture	GPUs	Baseline	+NICE
#Par.	MACs	sec/kimg	#Par.	MACs	sec/kimg
CIFAR-10	OmniGAN (d'=256)	1	8.51M	2.79G	0.80	8.51M	3.87G	1.15
CIFAR-100	OmniGAN (d'=256)	1	8.81M	2.79G	0.81	8.81M	3.87G	1.17
ImageNet	BigGAN	2	115.69M	18.84G	1.79	115.69M	20.43G	2.06
Low-shot & FFHQ	StyleGAN2	2	48.77M	44.15G	5.66	48.77M	52.00G	6.68
Figure 8: FID vs. sec/kimg of different methods on 5 low-shot datasets. FID are averaged over the 5 low-shot datasets.
I Qualitative comparison among DA, ADA, AACR and NICE+ADA
Figure 9 provides qualitative comparison between augmentation based method DA, ADA, AACR and  $\mathrm{NICE + ADA}$ . We find DA, ADA and AACR leak augmentation clues to the generated images while  $\mathrm{NICE + ADA}$  prevents the leakage. Figure 10 further demonstrates augmentation-based method, although can prevent discriminator overfitting problem by expanding data space through diverse augmentations, has the potential to cause increased gradient norm due to Prop. 1 and the fact ADA involves random additive/multiplicative noise. However, when combining NICE with ADA, the gradient issues caused by ADA were alleviated.
J Generated images
Figures 11, 12, 13, 14 and 15 provide the generated images on CIFAR-10, ImageNet, CIFAR-100, low-shot datasets and FFHQ with or without NICE. We can see NICE improve the image generation quality.
(a) DA
(b) ADA
(c) AACR
(d) NICE+ADA
Figure 9: Generated images using (a) DA (differentiable augmentation [68]), (b) ADA (adaptive differentiable augmentation [22]), (c) AACR (adaptive augmentation-based consistency regularization) and (d)  $\mathrm{NICE + ADA}$  on  $10\%$  CIFAR-10 using OmniGAN ( $d' = 256$ ) [71] We use red boxes to bound the images that leak the augmentation. DA leaks the cutout augmentation. ADA and AACR leaks the rotation augmentation. Combining NICE with ADA prevents the leakage of augmentation.
Figure 10: The gradient norm  $\| \frac{\partial L_D}{\partial x} \|_2^2$  of ADA and ADA+NICE.
(b) ADA
(c) ADA+NICE
Figure 11: Generated images using (a) DA, (b) ADA and (c) NICE+ADA on  $10\%$  CIFAR-10 using OmniGAN  $(d' = 1024)$ . Note that DA leaks cutout augmentation and ADA leaks the rotation augmentation.
(a) ADA
Figure 12: Generated images using (a) ADA and (c) ADA+NICE on  $2.5\%$  ImageNet using BigGAN. ADA struggles to capture the semantics of images, whereas NICE provides better image quality.
(b) ADA+NICE
(a) DA
Figure 13: Generated images using (a) DA, (b) ADA and (c) NICE+ADA on  $10\%$  CIFAR-100 using OmniGAN  $(d' = 1024)$ . We present the last 40 classes out of the 100 classes 40. Note that DA leaks cutout augmentation clues (row 7 and row 21) and ADA generates images lacking diversity, while NICE+ADA provides better visual quality.
(c) ADA+NICE
Figure 14: Qualitative comparison between ADA [22] and ADA+NICE on 100-shot and AnimalFace datasets. Adding NICE clearly improves the image quality.
(a) StyleGAN2+ADA
Figure 15: Qualitative comparison between ADA [22] and ADA+NICE on FFHQ dataset. Images are generated without truncation [25]. Adding NICE can provide better visual quality.
(b) StyleGAN2+ADA+NICE