SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {z _ {1}, z _ {2}} \left(f (a z _ {1}) - f (a z _ {2})\right) ^ {2} = \mathbb {E} _ {z _ {1}, z _ {2}} \left(f (a + (z _ {1} - 1) a) - f (a + (z _ {2} - 1) a)\right) ^ {2} \\ = \mathbb {E} _ {z _ {1}, z _ {2}} \left(f (a) + (z _ {1} - 1) a f ^ {\prime} (a) + \frac {(z _ {1} - 1) ^ {2}}{2} a ^ {2} f ^ {\prime \prime} (a) - f (a) - (z _ {2} - 1) a f ^ {\prime} (a) - \frac {(z _ {2} - 1) ^ {2}}{2} a ^ {2} f ^ {\prime \prime} (a)\right) ^ {2} \\ = \mathbb {E} _ {z _ {1}, z _ {2}} \left(\left(z _ {1} - z _ {2}\right) a f ^ {\prime} (a) + \frac {\left(z _ {1} - 1\right) ^ {2} - \left(z _ {2} - 1\right) ^ {2}}{2} a ^ {2} f ^ {\prime \prime} (a)\right) ^ {2} \\ = \mathbb {E} _ {m _ {1}, m _ {2} \sim \mathcal {N} (0, \beta^ {2})} \left(\left(m _ {1} - m _ {2}\right) a f ^ {\prime} (a) + \frac {m _ {1} ^ {2} - m _ {2} ^ {2}}{2} a ^ {2} f ^ {\prime \prime} (a)\right) ^ {2} \\ = \mathbb {E} _ {m _ {1}, m _ {2}} \left[ (m _ {1} - m _ {2}) ^ {2} a ^ {2} f ^ {\prime 2} (a) + (m _ {1} - m _ {2}) (m _ {1} ^ {2} - m _ {2} ^ {2}) a ^ {3} f ^ {\prime} (a) f ^ {\prime \prime} (a) + \frac {(m _ {1} ^ {2} - m _ {2} ^ {2}) ^ {2}}{4} a ^ {4} f ^ {\prime \prime 2} (a) \right] \\ = 2 \beta^ {2} a ^ {2} f ^ {\prime 2} (a) + \beta^ {4} a ^ {4} f ^ {\prime \prime 2} (a). \\ \end{array}
$$
Proof: Let  $m_1, m_2 \sim \mathcal{N}(0, \beta^2 I^{d'})$ . Note  $z_1$  and  $z_2$  are independent of each other, as well as independent of  $a$  and  $f(a)$ . The same independence holds for  $z_2$ . By taking expectations over  $z_1$  and  $z_2$ , and applying Taylor expansion, we analyze the consistency loss of  $f(a)$ :
$$
\begin{array}{l} \ell^ {\mathrm {N I C E}} (\boldsymbol {a}) := \mathbb {E} _ {\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}} \left[ \left(f (\boldsymbol {a} \odot \boldsymbol {z} _ {1}) - f (\boldsymbol {a} \odot \boldsymbol {z} _ {2})\right) ^ {2} \right] \\ = \mathbb {E} _ {\boldsymbol {m} _ {1}, \boldsymbol {m} _ {2}} \left[ \left(f (\boldsymbol {a} + \boldsymbol {m} _ {1} \odot \boldsymbol {a}) - f (\boldsymbol {a} + \boldsymbol {m} _ {2} \odot \boldsymbol {a})\right) ^ {2} \right] \\ \approx \mathbb {E} _ {\boldsymbol {m} _ {1}, \boldsymbol {m} _ {2}} \left[ \left(f (\boldsymbol {a}) + (\boldsymbol {m} _ {1} \odot \boldsymbol {a}) ^ {T} \nabla f (\boldsymbol {a}) + \frac {1}{2} (\boldsymbol {m} _ {1} \odot \boldsymbol {a}) ^ {T} H ^ {(f)} (\boldsymbol {a}) (\boldsymbol {m} _ {1} \odot \boldsymbol {a}) \right. \right. \\ \left. - f (\boldsymbol {a}) - \left(\boldsymbol {m} _ {2} \odot \boldsymbol {a}\right) ^ {T} \nabla f (\boldsymbol {a}) - \frac {1}{2} \left(\boldsymbol {m} _ {2} \odot \boldsymbol {a}\right) ^ {T} H ^ {(f)} (\boldsymbol {a}) \left(\boldsymbol {m} _ {2} \odot \boldsymbol {a}\right)\right) ^ {2} \bigg ] \\ = \mathbb {E} _ {\boldsymbol {m} _ {1}, \boldsymbol {m} _ {2}} \left[ \left(\left(\boldsymbol {m} _ {1} \odot \boldsymbol {a}\right) ^ {T} \nabla f (\boldsymbol {a}) + \frac {1}{2} \left(\boldsymbol {m} _ {1} \odot \boldsymbol {a}\right) ^ {T} H ^ {(f)} (\boldsymbol {a}) \left(\boldsymbol {m} _ {1} \odot \boldsymbol {a}\right) \right. \right. \\ \left. - \left(\boldsymbol {m} _ {2} \odot \boldsymbol {a}\right) ^ {T} \nabla f (\boldsymbol {a}) - \frac {1}{2} \left(\boldsymbol {m} _ {2} \odot \boldsymbol {a}\right) ^ {T} H ^ {(f)} (\boldsymbol {a}) \left(\boldsymbol {m} _ {2} \odot \boldsymbol {a}\right)\right) ^ {2} \rbrack . \tag {24} \\ \end{array}
$$
Defining  $\pmb{v} := \pmb{m}_1 \odot \pmb{a}$  and  $\pmb{u} := \pmb{m}_2 \odot \pmb{a}$  with  $\pmb{m}_1$  and  $\pmb{m}_2$  are independently sampled from  $\mathcal{N}(\mathbf{0}, \beta^2 \mathbf{I}^{d'})$ , we get  $\pmb{v}, \pmb{u} \sim \mathcal{N}(\mathbf{0}, \beta^2 \mathrm{diag}(\pmb{a} \odot \pmb{a}))$ . Here  $\mathrm{diag}(\pmb{a} \odot \pmb{a})$  is a diagonal matrix with  $a_k^2$  on the diagonal. Letting  $\delta := \nabla f(\pmb{a})$  and  $\pmb{H} = H^{(f)}(\pmb{a})$ , we can rewrite Eq. 24 as follows:
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ \left(\boldsymbol {v} ^ {T} \boldsymbol {\delta} + \frac {1}{2} \boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v} - \boldsymbol {u} ^ {T} \boldsymbol {\delta} - \frac {1}{2} \boldsymbol {u} ^ {T} \boldsymbol {H} \boldsymbol {u}\right) ^ {2} \right] \\ = \mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ \left(\left(\boldsymbol {v} - \boldsymbol {u}\right) ^ {T} \boldsymbol {\delta} + \frac {1}{2} \boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v} - \frac {1}{2} \boldsymbol {u} ^ {T} \boldsymbol {H} \boldsymbol {u}\right) ^ {2} \right] \\ = \mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ \left(\left(\boldsymbol {v} - \boldsymbol {u}\right) ^ {T} \boldsymbol {\delta}\right) ^ {2} \right] (25) \\ + \mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ \left(\boldsymbol {v} - \boldsymbol {u}\right) ^ {T} \boldsymbol {\delta}\right) \left(\boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v} - \boldsymbol {u} ^ {T} \boldsymbol {H} \boldsymbol {u}\right) (26) \\ + \frac {1}{4} \mathbb {E} _ {\boldsymbol {v}} \left[ (\boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v}) ^ {2} \right] + \frac {1}{4} \mathbb {E} _ {\boldsymbol {u}} \left[ (\boldsymbol {u} ^ {T} \boldsymbol {H} \boldsymbol {u}) ^ {2} \right] (27) \\ - \frac {1}{2} \mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ \left(\boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v}\right) \left(\boldsymbol {u} ^ {T} \boldsymbol {H} \boldsymbol {u}\right) \right]. (28) \\ \end{array}
$$
Next, we succinctly derive the four terms: the first (Eq. 25), second (Eq. 26), third (Eq. 27), and fourth (Eq. 28) term, respectively.
The first term (Eq. 25):
Leveraging the property  $\mathbb{E}_{z_1,z_2}[(z_1 - z_2)^2] = 2\sigma^2$  for  $z_{1},z_{2}\sim \mathcal{N}(0,\sigma^{2})$ , we can simplify (Eq. 25) as follows:
$$
\mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ \left(\left(\boldsymbol {v} - \boldsymbol {u}\right) ^ {T} \boldsymbol {\delta}\right) ^ {2} \right] = \mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ \sum_ {k = 1} ^ {d ^ {\prime}} (\boldsymbol {v} - \boldsymbol {u}) _ {k} ^ {2} \delta_ {k} ^ {2} \right] = \sum_ {k = 1} ^ {d ^ {\prime}} \mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ (\boldsymbol {v} - \boldsymbol {u}) _ {k} ^ {2} \right] \delta_ {k} ^ {2} = 2 \beta^ {2} \sum_ {k} a _ {k} ^ {2} \delta_ {k} ^ {2}. \tag {29}
$$
The second term (Eq. 26):
Utilizing  $E[z^3] = \mu^3 + 3\mu \sigma^2$  for  $z \sim \mathcal{N}(\mu, \sigma^2)$ , and noting that  $E[z^3] = 0$  when  $\mu = 0$  and elements in  $v$  are independent, Eq. 26 is derived as:
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ \left(\left(\boldsymbol {v} - \boldsymbol {u}\right) ^ {T} \boldsymbol {\delta}\right) \left(\boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v} - \boldsymbol {u} ^ {T} \boldsymbol {H} \boldsymbol {u}\right) \right] \\ = \mathbb {E} _ {\boldsymbol {v}} \left[ (\boldsymbol {v} ^ {T} \boldsymbol {\delta}) (\boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v}) \right] + \mathbb {E} _ {\boldsymbol {u}} \left[ (\boldsymbol {u} ^ {T} \boldsymbol {\delta}) (\boldsymbol {u} ^ {T} \boldsymbol {H} \boldsymbol {u}) \right] - \mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ (\boldsymbol {v} ^ {T} \boldsymbol {\delta}) (\boldsymbol {u} ^ {T} \boldsymbol {H} \boldsymbol {u}) \right] - \mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ (\boldsymbol {u} ^ {T} \boldsymbol {\delta}) (\boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v}) \right] \\ = 2 \mathbb {E} _ {\boldsymbol {v}} \left[ \left(\boldsymbol {v} ^ {T} \boldsymbol {\delta}\right) \left(\boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v}\right) \right] = 2 \mathbb {E} _ {\boldsymbol {v}} \left[ \sum_ {k} v _ {k} \delta_ {k} v _ {k} H _ {k k} v _ {k} \right] = 2 \mathbb {E} _ {\boldsymbol {v}} \left[ \sum_ {j} v _ {k} ^ {3} \delta_ {k} H _ {k k} \right] = 0. \tag {30} \\ \end{array}
$$
The third term (Eq. 27):
$$
\begin{array}{l} \frac {1}{\Lambda} \mathbb {E} _ {\boldsymbol {v}} \left[ (\boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v}) ^ {2} \right] + \frac {1}{\Lambda} \mathbb {E} _ {\boldsymbol {u}} \left[ (\boldsymbol {u} ^ {T} \boldsymbol {H} \boldsymbol {u}) ^ {2} \right] \\ = \frac {\ddagger}{2} \mathbb {E} _ {\boldsymbol {v}} \left[ \left(\boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v}\right) ^ {2} \right] = \frac {\ddagger}{2} \mathbb {E} _ {\boldsymbol {v}} \left[ \sum_ {j} \sum_ {k} \sum_ {p} \sum_ {q} v _ {j} H _ {j k} v _ {k} v _ {p} H _ {p q} v _ {q} \right]. \tag {31} \\ \end{array}
$$
Given the independence of elements in  $\pmb{v}$ , only terms with an element repeated two or four times contribute non-zero results, leading to four distinct, non-overlapping cases. Using  $\mathbb{E}[z^2] = \sigma^2 + \mu^2$  and  $\mathbb{E}[z^4] = \mu^4 + 6\mu^2\sigma^2 + 3\sigma^4$  for  $z \sim \mathcal{N}(\mu, \sigma^2)$ , and simplifying to  $\mathbb{E}[z^2] = \sigma^2$  and  $\mathbb{E}[z^4] = 3\sigma^4$  when  $\mu = 0$ , we have:
Case 1:  $j = k \neq p = q$ , the independence of  $v_{j}$  and  $v_{p}$  simplifies our calculation, leading to:
$$
\mathbb {E} _ {\boldsymbol {v}} \left[ \sum_ {j} \sum_ {p \neq j} v _ {j} ^ {2} H _ {j j} v _ {p} ^ {2} H _ {p p} \right] = \sum_ {j, p \neq j} H _ {j j} H _ {p p} \mathbb {E} \left[ v _ {j} ^ {2} \right] \mathbb {E} \left[ v _ {p} ^ {2} \right] = \beta^ {4} \sum_ {j, k \neq j} H _ {j j} H _ {k k} a _ {j} ^ {2} a _ {k} ^ {2}. \tag {32}
$$
Case 2: For  $j = p \neq k = q$ , given the independence of  $v_{j}$  and  $v_{k}$ , we have:
$$
\mathbb {E} _ {\boldsymbol {v}} \left[ \sum_ {j} \sum_ {k \neq j} v _ {j} H _ {j k} v _ {k} v _ {j} H _ {j k} v _ {k} \right] = \sum_ {j, k \neq j} H _ {j k} ^ {2} \mathbb {E} \left[ v _ {j} ^ {2} \right] \mathbb {E} \left[ v _ {k} ^ {2} \right] = \beta^ {4} \sum_ {j, k \neq j} H _ {j k} ^ {2} a _ {j} ^ {2} a _ {k} ^ {2}. \tag {33}
$$
Case 3: For  $j = q \neq k = p$ , leveraging the independence of  $v_{j}$  and  $v_{k}$  as well as the symmetry  $H_{jk} = H_{kj}$ , we obtain:
$$
\mathbb {E} _ {\boldsymbol {v}} \left[ \sum_ {j} \sum_ {k \neq j} \boldsymbol {v} _ {j} H _ {j k} v _ {k} v _ {k} H _ {k j} v _ {j} \right] = \sum_ {j, k \neq j} H _ {j k} ^ {2} \mathbb {E} \left[ v _ {j} ^ {2} \right] \mathbb {E} \left[ v _ {k} ^ {2} \right] = \beta^ {4} \sum_ {j, k \neq j} H _ {j k} ^ {2} a _ {j} ^ {2} a _ {k} ^ {2}. \tag {34}
$$
Case 4: For  $j = q = k = p$ , using  $\mathbb{E}[z^4] = 3\sigma^4$  where  $z \sim \mathcal{N}(0, \sigma^2)$ , we deduce:
$$
\mathbb {E} _ {\boldsymbol {v}} \left[ \sum_ {j} v _ {j} H _ {j j} v _ {j} v _ {j} H _ {j j} v _ {j} \right] = \sum_ {j} H _ {j j} ^ {2} \mathbb {E} \left[ v _ {j} ^ {4} \right] = \beta^ {4} \sum_ {j} 3 H _ {j j} ^ {2} a _ {j} ^ {4}. \tag {35}
$$
Combining Cases 1-4 together, we arrive at for Eq. 27:
$$
\frac {\beta^ {4}}{2} \left(\sum_ {j} 3 H _ {j j} ^ {2} a _ {j} ^ {4} + \sum_ {j, k \neq j} \left(H _ {j j} H _ {k k} + 2 H _ {j k} ^ {2}\right) a _ {j} ^ {2} a _ {k} ^ {2}\right). \tag {36}
$$
The forth term (Eq. 28):
$$
\begin{array}{l} - \frac {1}{2} \mathbb {E} _ {\boldsymbol {v}, \boldsymbol {u}} \left[ (\boldsymbol {v} ^ {T} \boldsymbol {H} \boldsymbol {v}) (\boldsymbol {u} ^ {T} \boldsymbol {H} \boldsymbol {u}) \right] \\ = - \frac {1}{2} \left(\mathbb {E} _ {\boldsymbol {v}} \left[ \sum_ {j} H _ {j j} v _ {j} ^ {2} \right] \mathbb {E} _ {\boldsymbol {u}} \left[ \sum_ {k} H _ {k k} v _ {k} ^ {2} \right]\right) \\ = - \frac {1}{2} \left(\sum_ {j} H _ {j j} \mathbb {E} \left[ v _ {j} ^ {2} \right]\right) \left(\sum_ {k} H _ {k k} \mathbb {E} \left[ v _ {k} ^ {2} \right]\right) \\ = - \frac {\beta^ {4}}{2} \left(\sum_ {j} H _ {j j} ^ {2} a _ {j} ^ {4} + \sum_ {j, k \neq j} H _ {j j} H _ {k k} a _ {j} ^ {2} a _ {k} ^ {2}\right). \tag {37} \\ \end{array}
$$
Substitute Eq. 29, 30, 36, 37 back into Eq. 25, 26, 27, 28, we have the final results:
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}} \left[ \left(f (\boldsymbol {z} _ {1} \odot \boldsymbol {a}) - f (\boldsymbol {z} _ {2} \odot \boldsymbol {a})\right) ^ {2} \right] \\ \approx 2 \beta^ {2} \sum_ {k} a _ {k} ^ {2} \delta_ {k} ^ {2} + 0 \\ + \frac {\beta^ {4}}{2} \left(\sum_ {j} 3 H _ {j j} ^ {2} a _ {j} ^ {4} + \sum_ {j, k \neq j} \left(H _ {j j} H _ {k k} + 2 H _ {j k} ^ {2}\right) a _ {j} ^ {2} a _ {k} ^ {2} - \sum_ {j} H _ {j j} ^ {2} a _ {j} ^ {4} - \sum_ {j, k \neq j} H _ {j j} H _ {k k} a _ {j} ^ {2} a _ {k} ^ {2}\right) \\ = 2 \beta^ {2} \sum_ {k} a _ {k} ^ {2} \delta_ {k} ^ {2} + \beta^ {4} \left(\sum_ {j} H _ {j j} ^ {2} a _ {j} ^ {4} + \sum_ {j, k \neq j} H _ {j k} ^ {2} a _ {j} ^ {2} a _ {k} ^ {2}\right) \\ = 2 \beta^ {2} \sum_ {k} a _ {k} ^ {2} \delta_ {k} ^ {2} + \beta^ {4} \sum_ {j} \sum_ {k} H _ {j k} ^ {2} a _ {j} ^ {2} a _ {k} ^ {2} \\ = 2 \beta^ {2} \sum_ {k} a _ {k} ^ {2} \nabla_ {k} ^ {2} f (\boldsymbol {a}) + \beta^ {4} \sum_ {j} \sum_ {k} a _ {j} ^ {2} a _ {k} ^ {2} \left(H _ {j k} ^ {(f)} (\boldsymbol {a})\right) ^ {2}. \tag {38} \\ \end{array}
$$
Enforcing the consistency regularization implicitly penalizes the first- and second-order gradients of  $f$  w.r.t.  $\pmb{a}$ , with an adaptive penalty controlled  $\beta$  and modulated by  $\pmb{a}$ . This modulation recalibrates the squared norms of gradient: the squared norm of the first-order gradient,  $\| \nabla f(\pmb{a}) \|_2^2 = \sum_k \nabla_k^2 f(\pmb{a})$ , and the Frobenius norm of the Hessian,  $\| H^{(f)}(\pmb{a}) \|_F^2 = \sum_j \sum_k (H_{jk}^{(f)}(\pmb{a}))^2$ . Specifically, each term is reweighted by  $a_k^2 \geq 0$ , ensuring that the penalty applied to each gradient component is scaled proportionally to its associated magnitude of feature.
Corollary 2 Enforcing invariance of  $f$  under different additive noise injections, where  $z_{1},z_{2}\sim \mathcal{N}(\mathbf{0},\beta^{2}\mathbf{I}^{d^{\prime}})$  leads to the gradient regularization as:
$$
\mathbb {E} _ {\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}} \left[ \left(f (\boldsymbol {z} _ {1} + \boldsymbol {a}) - f (\boldsymbol {z} _ {2} + \boldsymbol {a})\right) ^ {2} \right] = 2 \beta^ {2} \sum_ {k} \nabla_ {k} ^ {2} f (\boldsymbol {a}) + \beta^ {4} \sum_ {j} \sum_ {k} \left(H _ {j k} ^ {(f)} (\boldsymbol {a})\right) ^ {2}. \tag {39}
$$
D Implementation details for Experiments
Following previous benchmarks [68, 22], we augmented all datasets with the simple  $x$ -flips augmentation.
CIFAR-10/100. We experiment with OmniGAN ( $d' = 256$ , 1024) and BigGAN ( $d' = 256$ ) with the batch size of 32. We follow [68] and train the OmniGAN and BigGAN for 1K epochs on the full data and 5K epochs on  $10\% / 20\%$  data setting. We equip the discriminator with the adaptive noise modulation after convolution weights  $c \in \{C_1, C_2, C_S\}$  at all blocks  $l \in \{1, 2, 3, 4\}$ . We set  $\Delta_\beta = 0.001$ ,  $\eta = 0.5$ ,  $\Delta_\gamma = 10$ . Features before loss function are used for NICE on BigGAN.
ImageNet. We experiment with BigGAN with the batch size of 512. We use a learning rate of 1e-4 for generator and 4e-4 for discriminator. The noise modulation is placed after convolution layers  $c \in \{C_1, C_2, C_S\}$  at blocks  $l \in \{3, 4, 5\}$ ,  $\Delta_\beta = 0.001$ ,  $\eta = 0.5$ ,  $\Delta_\gamma = 5$ . Features before loss function are used for NICE.
Low-shot images. We build our NICE upon StyleGAN2 with batch size of 64 and train the networks until the discriminator had seen 25M images. We apply noise modulation with Bernoulli noise after convolutions  $c \in \{C_1, C_2\}$  at blocks  $l \in \{3, 4, 5, 6\}$ ,  $\Delta_{\beta} = 0.0001$ ,  $\eta = 0.9$ ,  $\Delta_{\gamma} = 0.05$
FFHQ. We experiments on StyleGAN2 with batch size of 64 and train the networks until the discriminator had seen 25M images. We apply noise modulation with Bernoulli noise after convolutions  $c \in \{C_1, C_2\}$  at blocks  $l \in \{3, 4, 5, 6\}$ ,  $\Delta_\beta = 0.0001$ ,  $\eta = 0.6$ ,  $\Delta_\gamma = 0.05$
D.2 Implementation details for AWD, AN+AGP, and AACR
In Figure 5, we compare NICE with three different approaches: adaptive weight decay (AWD), adaptive noise + adaptive gradient penalty (AN+AGP) and adaptive augmentation-based consistency regularization (AACR). For AWD, we dynamically control the weight decay using  $\beta \lambda_{\mathrm{WD}}$ . We search
the  $\lambda_{\mathrm{WD}} \in \{2\mathrm{e}-4, 1\mathrm{e}-4, 5\mathrm{e}-5, 2\mathrm{e}-5, 1\mathrm{e}-5, 5\mathrm{e}-6\}$  and obtain the best result at  $5\mathrm{e}-5$ . For  $\mathrm{AN}+\mathrm{AGP}$ , we employ the adaptive noise modulation and also adaptively penalize the gradient of  $\left\| \frac{\partial L_D^{\mathrm{AN}}}{\partial x} \right\|_2^2$  or  $\left\| \frac{\partial f}{\partial x} \right\|_2^2$  with adaptive strength  $\beta \lambda_{\mathrm{GP}}$ . We test different values of  $\lambda_{\mathrm{GP}} \in \{0.2, 0.5, 1, 2, 5, 10\}$  and obtain the best results with  $\lambda_{\mathrm{GP}} = 2$ . For AACR, we incorporate the adaptive differentiable augmentation (ADA) to provide two different views for the same images, and enforce the discriminator to be consistent to the two views of the same images. We adaptively control the strength of AACR with  $\beta \lambda_{\mathrm{AACR}}$ . We search the  $\lambda_{\mathrm{AACR}} \in \{0.2, 0.5, 1, 2, 5, 10\}$  and obtain the best result with  $\lambda_{\mathrm{AACR}} = 2$ .
E Impact of various factors
Below, we analyze the importance of different factors in NICE by conducting ablation experiments. The results are presented in Figure 6. In Figure 6a, we observe that placing the noise modulation module after  $C_1$  and  $C_2$  improves performance, and the best results are obtained when the noise is placed after all convolution layers. Figure 6b indicates that placing the noise at deeper layers leads to better tFID scores, and the best performance is achieved when the noise is applied at all blocks. Figure 6c shows that  $\Delta_{\gamma} = 10$  is a good choice for NICE. Figure 6d illustrates  $\eta = 0.5$  produce overall good performance. Figure 6e plots the results for consistency regularization among  $M$  feature extractors  $f$  modulated with noise. We use  $M = 2$  (i.e.,  $f_1$  and  $f_2$ ) in the main paper. Figure 6e indicates there is no need to use more feature extractors, as  $f_1(\cdot)$  and  $f_2(\cdot)$  for consistency regularization produce the best results. Figure 6f presents the results for consistency regularization at every  $N$  discriminator steps, indicating that regularizing the discriminator during each iteration is preferable, as larger intervals result in worse performance.
(a)  $c\in \{C_1,C_2,C_S\}$
(d)  $\eta$
Figure 6: tFID  $\downarrow$  under different factors. Ablation studies on  $10\%$  CIFAR-10 with OmniGAN  $(d' = 256)$  w.r.t. different places, different blocks,  $\Delta_{\gamma}$ ,  $\eta$ ,  $M$ -time and  $N$ -step for NICE.
(b)  $l\in \{1\dots L\}$
(e)  $M$  -time
(c)  $\Delta_{\gamma}$
(f)  $N$  -step
F Rationale for choosing multiplicative noise over direct weight regularization
For the multiplicative modulation with the noise, we have:
$$
\begin{array}{l} \hat {L} _ {\text {m u l . n o i s e}} := \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {z} \sim \mathcal {N} (\mathbf {1}, \beta^ {2} \boldsymbol {I})} \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} (\boldsymbol {z} \odot \boldsymbol {a} ^ {(i)}) \| _ {2} ^ {2} \\ = \hat {\mathbb {E}} _ {i} \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)} \| _ {2} ^ {2} + \beta^ {2} \sum_ {k} \hat {a} _ {k} \| \boldsymbol {w} _ {k} \| _ {2} ^ {2}, \\ \end{array}
$$
where  $\pmb{z} \odot \pmb{a}^{(i)}$  is element-wise multiplication of  $\pmb{z}$  with  $\pmb{a}^{(i)}$  and  $\hat{\pmb{a}} = \hat{\mathbb{E}}_i[\pmb{a}^{(i)} \odot \pmb{a}^{(i)}]$ . This implies that while the direct penalty of network weights,  $\sum_{k} \| \pmb{w}_{k} \|_{2}^{2}$ , is possible, our approach enjoys a dynamic modulation  $\beta^2 \hat{a}_k^2$  where variance  $\beta^2$  is adapted based on the discriminator decisions and  $\hat{a}_k \geq 0$  depends on the magnitude of feature vectors.
The importance of such a multiplicative modulation is that semantic contents is not added by noise  $\mathbf{z}$  to the modulated feature vector  $\mathbf{z} \cdot \mathbf{a}^{(i)}$ , i.e., only "active" channels  $\mathbf{a}^{(i)} \neq 0$  are modulated: they can be suppressed or magnified according to variance  $\beta^2$ .
In contrast, the additive noise modulation results in the standard penalty  $\beta^2\sum_k\| \pmb {w}_k\| _2^2$
$$
\begin{array}{l} \hat {L} _ {\text {a d d \_ n o i s e}} := \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {z} \sim \mathcal {N} (\mathbf {0}, \beta^ {2} \boldsymbol {I})} \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} (\boldsymbol {z} + \boldsymbol {a} ^ {(i)}) \| _ {2} ^ {2} \\ = \hat {\mathbb {E}} _ {i} \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)} \| _ {2} ^ {2} + \beta^ {2} \sum_ {k} \| \boldsymbol {w} _ {k} \| _ {2} ^ {2}. \\ \end{array}
$$
While obvious,  $\beta^2\sum_k\| \pmb {w}_k\| _2^2$  does not enjoy the dynamic penalty term related to feature norms. This also means (based on the derivation) that the additive noise may change feature semantics, i.e.,  $z + a^{(i)}$  may "activate" channels which are non-active in  $a^{(i)}$  (features  $a^{(i)} = 0$ ). Therefore, our multiplicative modulator not only controls the Rademacher Complexity (RC), but controls it in a meaningful manner for discriminator. Only feature semantics that are present in a feature vector (that describe an object) are modulated while helping control RC. In contrast, the additive noise introduces semantics that are not present for a given image or object.
While directly regularizing the network using  $\beta^2\sum_k\| \pmb{w}_k\| _2^2$  does not introduce the additive noise, the explicit regularization effect is similar to introducing the additive noise. Thus, if we connect the direct weight regularization to the additive noise, it will also have the effect of changing the semantics, which will negatively impact the classification accuracy.
We provide experimental comparisons in Table 8. In addition, Figure 7 shows that AN achieves higher classification accuracy than AAN and AWR, demonstrating that multiplicative noise preserves the feature semantics better than the additive noise and the weight regularization.
Table 8: Results for different methods on  $10\%$  CIFAR-10/100 using OmniGAN ( $d' = 256$ ). AAN (adaptive additive noise), AWR (adaptive weight regularization), AN (our adaptive multiplicative noise). Please note the consistency loss and dual branch from Figure 1 are not used here as that would result in additional penalties on gradient norms.
Method	Equation	10% CIFAR-10	10% CIFAR-100
IS ↑	tFID ↓	vFID ↓	IS ↑	tFID ↓	vFID ↓
OmniGAN	8.49	22.24	26.33	8.19	45.41	50.33
+AAN	a(i) := a(i) + z; z ~ N(0,β2I)	8.52	20.12	24.65	9.64	37.68	42.01
+AWR	β2∑k||wk||2	8.44	18.42	22.56	9.80	32.05	36.53
+AN	a(i) := a(i) ⊙z; z ~ N(1,β2I)	9.16	10.14	13.80	11.22	23.76	28.34
Figure 7: Classification accuracy of different methods. We input test images into the classifier of discriminator and assess its accuracy in correctly categorizing the images into their true classes (out of 10 categories in CIFAR-10). All other methods have lower accuracy than the baseline OmniGAN, but NICE preserves the classifier accuracy.
G Rationale behind choosing NICE over the direct gradient regularization
Consider the regularization term in Theorem 2 and its Taylor expansion. Indeed, this expression penalizes the squared norms of first- and second-order gradients of  $f(\pmb{a})$ . However, notice that our expression has specific reweighting effect for penalizing the first- and second-order gradients. The specific gradient penalties
$$
2 \beta^ {2} \sum_ {k} a _ {k} ^ {2} \nabla_ {k} ^ {2} f (\boldsymbol {a}) + \beta^ {4} \sum_ {j, k} a _ {j} ^ {2} a _ {k} ^ {2} \left(H _ {j k} ^ {(f)} (\boldsymbol {a})\right) ^ {2} \tag {40}
$$