SlowGuess/NeurIPS_2023_OCR huggingface dataset:
NICE is a solution to limiting the extensive image usage in GAN training. Conventional GANs heavily rely on massive training data, which can jeopardize data privacy during large-scale data collection. NICE is an innovative method for data-limited image generation. By successfully training GANs with just 100 images, NICE not only minimizes the dependency on large-scale training datasets, but also significantly reduces the training cost. NICE decreases the risk of privacy violations (using smaller datasets equals to the lesser risk of data mismanagement). NICE also provides a cost-effective alternative for training GANs. As the data required for training is lesser compared with large-scale models, NICE is also energy-consumption friendly. Of course, with small-scale dataset comes a limitation on diversity. The model cannot generate instances for which the data is completely lacking. Thus, special care needs to be taken regarding gender, race and other biases in generated images.
NICE advances GANs with Theorems 1 and 2, along with Corollaries 1 and 2. It offers novel insights into the weight regularization. These findings, driven by the multiplicative and additive noise modulation, along with the gradient regularization due to the consistency regularization, pave the way for further future research.
B Notations
Below, we explain the notations used in this work.
**Scalers:** Represented by lowercase letters  $(e.g., m, n, \delta)$ .
Vectors: Bold lowercase letters (e.g.,  $\mathbf{v}$ ,  $\mathbf{u}$ ,  $\mathbf{m}$ ).
Matrices: Bold uppercase letters (e.g.,  $W, X, H$ ).
Vector/Matrix elements:  $v_{k}$  and  $a_{k}$  represent  $k$ -th element of  $\mathbf{v}$  and  $\mathbf{a}$ .  $H_{jk}$  denotes the element located at the  $j$ -th row and  $k$ -th column of matrix  $\mathbf{H}$ .
Functions: Letters followed by brackets (e.g.,  $\sigma(\cdot)$ ,  $h(\cdot)$ ,  $\nabla f(\cdot)$ ,  $H(\cdot)$ ).
Function sets: Calligraphic uppercase letters are used, but note that  $\mathcal{N}$  and  $\mathcal{B}$  specifically denote Gaussian and Bernoulli distributions, respectively (e.g.,  $\mathcal{H}$ ,  $\mathcal{F}$ ).
Probability measures: Denoted by letters  $\mu, \nu$  and  $p_z$ .
Dataset samples: Expressed as  $\pmb{x}^{(i)}, \pmb{y}^{(i)}, \pmb{a}^{(i)}, \tau^{(i)}$ .
Multiplication/Composition:  $\odot$  represents element-wise multiplication and  $\circ$  denotes function composition.
Expectation/empirical expectation:  $\mathbb{E}[\cdot ]$  represents the average or expected value of a random variable while  $\hat{\mathbb{E}} [\cdot ]$  denotes the empirical expectation calculated over observed data samples.
We define  $\| \pmb{v} \|^2 = \pmb{v}^T \pmb{v}$  as the squared norm of vector  $\pmb{v}$ . Note the commutative property of the dot product,  $\pmb{v}^T \pmb{u} = \pmb{u}^T \pmb{v}$ , and the distributive property over matrices,  $\pmb{v}^T \pmb{A} + \pmb{u}^T \pmb{A} = (\pmb{v} + \pmb{u})^T \pmb{A}$ . With these definitions and properties established, we now prove Theorem 1, Proposition 1 and Theorem 2. We begin with a simplified scalar-based derivation, ensuring a rapid derivation of the final results, followed by comprehensive proofs encompassing vector and matrix calculus for a holistic understanding.
Theorem 1 (Regularization by the Gaussian multiplicative noise in deep regression) Consider a regression task on a two-layer neural network. Let  $\mathcal{X} \subseteq \mathbb{R}^{d_0}$  and  $\mathcal{Y} \subseteq [-1,1]^{d_2}$  be the input and output spaces, and  $\mathcal{D} = \mathcal{X} \times \mathcal{Y}$ . Let  $n$  examples  $\{(\pmb{x}^{(i)},\pmb{y}^{(i)})\}_{i=1}^n \sim \mathcal{D}^n$ . Let  $f_w: \mathcal{X} \to \mathcal{Y}$  be parameterized by  $\{w:\pmb{W}_1 \in \mathbb{R}^{d_1 \times d_0},\pmb{W}_2 \in \mathbb{R}^{d_2 \times d_1}\}$  and  $f_w(\pmb{x},\pmb{z}) = \pmb{W}_2(\pmb{z} \odot \sigma(\pmb{W}_1\pmb{x}))$  where  $\sigma$  is 1-Lipschitz activation function,  $\odot$  denotes element-wise multiplication and  $z \sim \mathcal{N}(\mathbf{1},\beta^2\pmb{I}^{d_1})$ . Let  $\pmb{a} = \sigma(\pmb{W}_1\pmb{x})$ ,  $\hat{\pmb{a}} = \hat{\mathbb{E}}_i[\pmb{a}^{(i)} \odot \pmb{a}^{(i)}]$ ,  $\hat{a}_k \geq 0$  denotes the  $k$ -th element of  $\hat{\pmb{a}}$ . Let  $\| \pmb{W}_2 \|_{2,1} = \sum_k \| \pmb{w}_k \|_2$  where  $\pmb{w}_k$  is the  $k$ -th column vector in  $\pmb{W}_2$ . The regression task with the  $l_2$  loss leads to the weight regularization as:
$$
\hat {L} _ {\text {n o i s e}} (w) := \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {z}} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} (\boldsymbol {z} \odot \boldsymbol {a} ^ {(i)}) \| _ {2} ^ {2} \right] = \hat {\mathbb {E}} _ {i} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)} \| _ {2} ^ {2} \right] + \beta_ {2} ^ {2} \sum_ {k} \hat {a} _ {k} \| \boldsymbol {w} _ {k} \| _ {2} ^ {2}.
$$
We commence with a simplified derivation using scalars, which can quickly lead to the final results:
$$
\begin{array}{l} \hat {\mathbb {E}} _ {i} \mathbb {E} _ {z \sim \mathcal {N} (1, \beta^ {2})} \left[ \left(y ^ {(i)} - w _ {2} (z a ^ {(i)})\right) ^ {2} \right] = \hat {\mathbb {E}} _ {i} \mathbb {E} _ {z} \left[ \left(y ^ {(i)} - w _ {2} a ^ {(i)} - w _ {2} (z - 1) a ^ {(i)}\right) ^ {2} \right] \\ = \hat {\mathbb {E}} _ {i} \mathbb {E} _ {z} \left[ \left(y ^ {(i)} - w _ {2} a ^ {(i)}\right) ^ {2} - 2 \left(y ^ {(i)} - w _ {2} a ^ {(i)}\right) w _ {2} (z - 1) a ^ {(i)} + \left(w _ {2} (z - 1) a ^ {(i)}\right) ^ {2} \right] \\ = \hat {\mathbb {E}} _ {i} \left[ \left(y ^ {(i)} - w _ {2} a ^ {(i)}\right) ^ {2} \right] + w _ {2} ^ {2} \hat {\mathbb {E}} _ {i} a ^ {(i) 2} \mathbb {E} _ {z} [ (z - 1) ^ {2} ] \\ = \hat {\mathbb {E}} _ {i} \left[ \left(y ^ {(i)} - w _ {2} a ^ {(i)}\right) ^ {2} \right] + \beta^ {2} w _ {2} ^ {2} \hat {\mathbb {E}} _ {i} a ^ {(i) 2}. \\ \end{array}
$$
$$
\begin{array}{l} \hat {L} _ {\text {n o i s e}} (w) := \hat {\mathbb {E}} _ {i} \mathbb {E} _ {z} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} (\boldsymbol {z} \odot \boldsymbol {a} ^ {(i)}) \| _ {2} ^ {2} \right] = \hat {\mathbb {E}} _ {i} \mathbb {E} _ {z} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)} - \boldsymbol {W} _ {2} \big ((\boldsymbol {z} - \mathbf {1}) \odot \boldsymbol {a} ^ {(i)} \big) \| _ {2} ^ {2} \right] \\ = \hat {\mathbb {E}} _ {i} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)} \| _ {2} ^ {2} \right] - 2 \hat {\mathbb {E}} _ {i} \left[ \left(\boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)}\right) ^ {T} \boldsymbol {W} _ {2} \left(\mathbb {E} _ {\boldsymbol {z}} [ \boldsymbol {z} - \boldsymbol {1} ] \odot \boldsymbol {a} ^ {(i)}\right) \right] + \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {z}} \left[ \| \boldsymbol {W} _ {2} \left((\boldsymbol {z} - \boldsymbol {1}) \odot \boldsymbol {a} ^ {(i)}\right) \| _ {2} ^ {2} \right] \\ = \hat {\mathbb {E}} _ {i} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)} \| _ {2} ^ {2} \right] - 2 \hat {\mathbb {E}} _ {i} \left[ (\boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)}) ^ {T} \boldsymbol {W} _ {2} (\boldsymbol {0} \odot \boldsymbol {a} ^ {(i)}) \right] + \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {z}} \left[ \| \boldsymbol {W} _ {2} ((\boldsymbol {z} - \boldsymbol {1}) \odot \boldsymbol {a} ^ {(i)}) \| _ {2} ^ {2} \right] \\ = \hat {\mathbb {E}} _ {i} \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)} \| _ {2} ^ {2} + \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {z}} \| \boldsymbol {W} _ {2} ((\boldsymbol {z} - \mathbf {1}) \odot \boldsymbol {a} ^ {(i)}) \| _ {2} ^ {2}. \tag {12} \\ \end{array}
$$
Let  $\pmb{m} \sim \mathcal{N}(\pmb{0}, \beta^2 \pmb{I}^{d_1})$ , with  $m_k$  and  $a_k^{(i)}$  representing the  $k$ -th elements of vectors  $\pmb{m}$  and  $\pmb{a}^{(i)}$ , respectively. Using these definitions, we derive the second term of Eq. 12 as follows:
$$
\begin{array}{l} \hat {\mathbb {E}} _ {i} \mathbb {E} _ {z} \left[ \| \boldsymbol {W} _ {2} \left(\left(z - 1\right) \odot \boldsymbol {a} ^ {(i)}\right) \| _ {2} ^ {2} \right] = \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {m}} \left[ \| \boldsymbol {W} _ {2} \left(\boldsymbol {m} \odot \boldsymbol {a} ^ {(i)}\right) \| _ {2} ^ {2} \right] \\ = \hat {\mathbb {E}} _ {i} \mathbb {E} _ {m} \left[ \left(\boldsymbol {m} \odot \boldsymbol {a} ^ {(i)}\right) ^ {T} \boldsymbol {W} _ {2} ^ {T} \boldsymbol {W} _ {2} \left(\boldsymbol {m} \odot \boldsymbol {a} ^ {(i)}\right) \right] \\ = \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {m}} \left[ \sum_ {j = 1} ^ {d _ {1}} \sum_ {k = 1} ^ {d _ {1}} \left(m _ {j} a _ {j} ^ {(i)}\right) \left(\boldsymbol {W} _ {2} ^ {T} \boldsymbol {W} _ {2}\right) _ {j k} \left(m _ {k} a _ {k} ^ {(i)}\right) \right]. \tag {13} \\ \end{array}
$$
Given that the elements in  $\mathbf{m}$  are independent, for  $j \neq k$  we have:
$$
\mathbb {E} _ {\boldsymbol {m}} \left[ \left(m _ {j} a _ {j} ^ {(i)}\right) \left(\boldsymbol {W} _ {2} ^ {T} \boldsymbol {W} _ {2}\right) _ {j k} \left(m _ {k} a _ {k} ^ {(i)}\right) \right] = 0. \tag {14}
$$
Observe that  $(\pmb{W}_2^T\pmb{W}_2)_{kk} = \| \pmb{w}_k\|_2^2$  represents the squared norm of  $k$ -th column vector in  $\pmb{W}_2$ . Let  $\hat{\pmb{a}}$  denote the element-wise mean square of  $\pmb{a}^{(i)}$ , i.e.,  $\hat{\pmb{a}} = \hat{\mathbb{E}}_i[\pmb{a}^{(i)} \odot \pmb{a}^{(i)}]$ . We leverage the distribution property  $\mathbb{E}[z^2] = \sigma^2 + \mu^2$  for  $z \sim \mathcal{N}(\mu, \sigma^2)$ , simplifying to  $\mathbb{E}[z^2] = \sigma^2$  when  $\mu = 0$ . This lets us
reformulate Eq. 13 explicitly as:
$$
\begin{array}{l} \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {m}} \left[ \sum_ {j = 1} ^ {d _ {1}} \sum_ {k = 1} ^ {d _ {1}} \left(m _ {j} a _ {j} ^ {(i)}\right) \left(\boldsymbol {W} _ {2} ^ {T} \boldsymbol {W} _ {2}\right) _ {j k} \left(m _ {k} a _ {k} ^ {(i)}\right) \right] \\ = \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {m}} \left[ \sum_ {k = 1} ^ {d _ {1}} \left(m _ {k} a _ {k} ^ {(i)}\right) \left(\boldsymbol {W} _ {2} ^ {T} \boldsymbol {W} _ {2}\right) _ {k k} \left(m _ {k} a _ {k} ^ {(i)}\right) \right] \\ = \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {m}} \sum_ {k = 1} ^ {d _ {1}} \left(m _ {k} ^ {2} a _ {k} ^ {(i) 2}\right) \| \boldsymbol {w} _ {k} \| _ {2} ^ {2} = \beta^ {2} \hat {\mathbb {E}} _ {i} \sum_ {k = 1} ^ {d _ {1}} a _ {k} ^ {(i) 2} \| \boldsymbol {w} _ {k} \| _ {2} ^ {2} \\ = \beta^ {2} \sum_ {k} \hat {a} _ {k} \| \boldsymbol {w} _ {k} \| _ {2} ^ {2}. \tag {15} \\ \end{array}
$$
Substituting the second term in Eq. 12 with Eq. 15, we arrive at:
$$
\hat {L} _ {\text {n o i s e}} (w) := \hat {\mathbb {E}} _ {i} \mathbb {E} _ {z} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} (\boldsymbol {z} \odot \boldsymbol {a} ^ {(i)}) \| _ {2} ^ {2} \right] = \hat {\mathbb {E}} _ {i} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)} \| _ {2} ^ {2} \right] + \beta^ {2} \sum_ {k} \hat {a} _ {k} \| \boldsymbol {w} _ {k} \| _ {2} ^ {2}. \tag {16}
$$
Integrating the Gaussian multiplicative noise into the optimization of regression task leads to an adaptive regularization  $\beta^2\sum_k\hat{a}_k\| \pmb {w}_k\| _2^2$  on  $W_{2}$ , effectively reweighting the squared norm  $\| \pmb {w}_k\| _2^2$  of each column in accordance with  $\hat{a}_k\geq 0$ . This mechanism, driven by the variance  $\beta^2$  of noise enhances the robustness of model and its generalization capability. It is worth highlighting that this process relates closely to the (2,1)-norm of  $W_{2}$ , defined as  $\| W_2\|_{2,1} = \sum_k\| w_k\| _2$ .
Corollary 1 Utilizing the additive noise injection where  $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \beta^2 \mathbf{I}^{d_1})$ , the regression task with the  $l_2$  loss yields weight regularization as:
$$
\hat {L} _ {\text {a d d n o i s e}} (w) := \hat {\mathbb {E}} _ {i} \mathbb {E} _ {\boldsymbol {z}} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} (\boldsymbol {z} + \boldsymbol {a} ^ {(i)}) \| _ {2} ^ {2} \right] = \hat {\mathbb {E}} _ {i} \left[ \| \boldsymbol {y} ^ {(i)} - \boldsymbol {W} _ {2} \boldsymbol {a} ^ {(i)} \| _ {2} ^ {2} \right] + \beta^ {2} \sum_ {k} \| \boldsymbol {w} _ {k} \| _ {2} ^ {2}. \tag {17}
$$
Proposition 1 Define  $f \coloneqq f_t \circ \ldots \circ f_2$  as the feature extractor of the discriminator from the second layer onward. Let  $\tilde{\mathbf{a}} = f_1(\tilde{\mathbf{x}})$  and  $\mathbf{a} = f_1(\mathbf{x})$  be the latent features of the first layer for fake and real images, respectively. The discriminator is defined as  $h(\cdot) = \phi(\varphi(f(\cdot))$ , with  $\varphi$  as the prediction head and  $\phi$  as the loss function. Introduce a multiplicative noise  $z \sim \mathcal{N}(1, \beta^2 I^{d'})$  modulating solely the first layer. Let  $H_{kk}^{(h)}(\mathbf{a})$  be the  $k$ -th diagonal entry of the Hessian matrix of  $h$  at  $\mathbf{a}$  and  $a_k$  be the  $k$ -th element of  $\mathbf{a}$ . Applying Taylor expansion to  $h(\cdot)$ , the GAN loss can be expressed as follows:
$$
\begin{array}{l} \min  _ {\boldsymbol {\theta} _ {d}} L _ {D} ^ {A N} := \mathbb {E} _ {\tilde {\boldsymbol {a}}} \mathbb {E} _ {\boldsymbol {z}} \left[ h (\boldsymbol {z} \odot \tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \mathbb {E} _ {\boldsymbol {z}} \left[ h (\boldsymbol {z} \odot \boldsymbol {a}) \right] \\ \approx \mathbb {E} _ {\tilde {\boldsymbol {a}}} [ h (\tilde {\boldsymbol {a}}) ] - \mathbb {E} _ {\boldsymbol {a}} [ h (\boldsymbol {a}) ] + \frac {\beta^ {2}}{2} \left(\mathbb {E} _ {\tilde {\boldsymbol {a}}} \left[ \sum_ {k} \tilde {a} _ {k} ^ {2} H _ {k k} ^ {(h)} (\tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \left[ \sum_ {k} a _ {k} ^ {2} H _ {k k} ^ {(h)} (\boldsymbol {a}) \right]\right). \\ \end{array}
$$
$$
\min _ {\boldsymbol {\theta} _ {g}} L _ {G} ^ {A N} := - \mathbb {E} _ {\boldsymbol {z}} \mathbb {E} _ {\tilde {\boldsymbol {a}}} \left[ h (\boldsymbol {z} \odot \tilde {\boldsymbol {a}}) \right] \approx - \mathbb {E} _ {\tilde {\boldsymbol {a}}} \left[ h (\tilde {\boldsymbol {a}}) \right] - \frac {\beta^ {2}}{2} \mathbb {E} _ {\tilde {\boldsymbol {a}}} \left[ \sum_ {k} \tilde {a} _ {k} ^ {2} H _ {k k} ^ {(h)} (\tilde {\boldsymbol {a}}) \right].
$$
We provide a succinct proof for the discriminator using scalar values to quickly arrive at the final result, where  $h'(\cdot)$  and  $h''(\cdot)$  are the first- and second-order derivatives, respectively.
$$
\begin{array}{l} \mathbb {E} _ {z} h (z \tilde {a}) - \mathbb {E} _ {z} h (z a) = \mathbb {E} _ {z} h (\tilde {a} + (z - 1) \tilde {a}) - \mathbb {E} _ {z} h (a + (z - 1) a) \\ = \mathbb {E} _ {z} \left[ h (\tilde {a}) + (z - 1) \tilde {a} h ^ {\prime} (\tilde {a}) + \frac {(z - 1) ^ {2}}{2} \tilde {a} ^ {2} h ^ {\prime \prime} (\tilde {a}) \right] - \mathbb {E} _ {z} \left[ h (a) + (z - 1) a h ^ {\prime} (a) + \frac {(z - 1) ^ {2}}{2} a ^ {2} h ^ {\prime \prime} (a) \right] \\ = h (\tilde {a}) - h (a) + \mathbb {E} _ {z} \left[ \frac {(z - 1) ^ {2}}{2} \right] \left(\tilde {a} ^ {2} h ^ {\prime \prime} (\tilde {a}) - a ^ {2} h ^ {\prime \prime} (a)\right) \\ = h (\tilde {a}) - h (a) + \frac {\beta^ {2}}{2} \left(\tilde {a} ^ {2} h ^ {\prime \prime} (\tilde {a}) - a ^ {2} h ^ {\prime \prime} (a)\right). \\ \end{array}
$$
Proof: Consider a multivariate normal distribution  $m \sim \mathcal{N}(0, \beta^2 I^{d'})$ . Let  $\nabla h(\pmb{a})$  and  $H_{jk}^{(h)}(\pmb{a})$  represent the first-order derivative and the  $(j,k)$ -th entry of the Hessian matrix of  $h$  at  $\pmb{a}$ , respectively,
implying that  $H_{jk}^{(h)} = H_{kj}^{(h)}$  due to the symmetry of the Hessian matrix. With these notations established, we proceed as follows:
$$
\begin{array}{l} L _ {D} ^ {\mathrm {A N}} := \mathbb {E} _ {\tilde {\boldsymbol {a}}} \mathbb {E} _ {\boldsymbol {z}} \left[ h (\tilde {\boldsymbol {a}} \odot \boldsymbol {z}) \right] - \mathbb {E} _ {\boldsymbol {a}} \mathbb {E} _ {\boldsymbol {z}} \left[ h (\boldsymbol {a} \odot \boldsymbol {z}) \right] \\ = \mathbb {E} _ {\tilde {\boldsymbol {a}}} \mathbb {E} _ {\boldsymbol {z}} \left[ h (\tilde {\boldsymbol {a}} + (\boldsymbol {z} - \boldsymbol {1}) \odot \tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \mathbb {E} _ {\boldsymbol {z}} \left[ h (\boldsymbol {a} + (\boldsymbol {z} - \boldsymbol {1}) \odot \boldsymbol {a}) \right] \\ = \mathbb {E} _ {\tilde {a}} \mathbb {E} _ {\boldsymbol {m}} \left[ h (\tilde {\boldsymbol {a}} + \boldsymbol {m} \odot \tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \mathbb {E} _ {\boldsymbol {m}} \left[ h (\boldsymbol {a} + (\boldsymbol {m} \odot \boldsymbol {a}) \right] \\ \approx \mathbb {E} _ {\tilde {\boldsymbol {a}}} \mathbb {E} _ {\boldsymbol {m}} \left[ h (\tilde {\boldsymbol {a}}) + (\boldsymbol {m} \odot \tilde {\boldsymbol {a}}) ^ {T} \nabla h (\tilde {\boldsymbol {a}}) + \frac {1}{2} (\boldsymbol {m} \odot \tilde {\boldsymbol {a}}) ^ {T} H ^ {(h)} (\tilde {\boldsymbol {a}}) (\boldsymbol {m} \odot \tilde {\boldsymbol {a}}) \right] \\ - \mathbb {E} _ {\boldsymbol {a}} \mathbb {E} _ {\boldsymbol {m}} \left[ h (\boldsymbol {a}) + \left(\boldsymbol {m} \odot \boldsymbol {a}\right) ^ {T} \nabla h (\boldsymbol {a}) + \frac {1}{2} \left(\boldsymbol {m} \odot \boldsymbol {a}\right) ^ {T} H ^ {(h)} (\boldsymbol {a}) \left(\boldsymbol {m} \odot \boldsymbol {a}\right) \right] \\ = \mathbb {E} _ {\tilde {\boldsymbol {a}}} h (\tilde {\boldsymbol {a}}) - \mathbb {E} _ {\boldsymbol {a}} h (\boldsymbol {a}) \\ + \mathbb {E} _ {\tilde {\boldsymbol {a}}} \mathbb {E} _ {\boldsymbol {m}} \left[ (\boldsymbol {m} \odot \tilde {\boldsymbol {a}}) ^ {T} \nabla h (\tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \mathbb {E} _ {\boldsymbol {m}} \left[ (\boldsymbol {m} \odot \boldsymbol {a}) ^ {T} \nabla h (\boldsymbol {a}) \right] (18) \\ + \mathbb {E} _ {\tilde {\boldsymbol {a}}} \mathbb {E} _ {\boldsymbol {m}} \left[ \frac {1}{2} (\boldsymbol {m} \odot \tilde {\boldsymbol {a}}) ^ {T} H ^ {(h)} (\tilde {\boldsymbol {a}}) (\boldsymbol {m} \odot \tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \mathbb {E} _ {\boldsymbol {m}} \frac {1}{2} (\boldsymbol {m} \odot \boldsymbol {a}) ^ {T} H ^ {(h)} (\boldsymbol {a}) (\boldsymbol {m} \odot \boldsymbol {a}). (19) \\ \end{array}
$$
Addressing the first-order gradient term presented in Eq. 18, we notice the following:
$$
\begin{array}{l} \mathbb {E} _ {\tilde {a}} \mathbb {E} _ {\boldsymbol {m}} \left[ (\boldsymbol {m} \odot \tilde {\boldsymbol {a}}) ^ {T} \nabla h (\tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \mathbb {E} _ {\boldsymbol {m}} \left[ (\boldsymbol {m} \odot \boldsymbol {a}) ^ {T} \nabla h (\boldsymbol {a}) \right] \\ = \mathbb {E} _ {\tilde {\boldsymbol {a}}} \left[ (\boldsymbol {0} \odot \tilde {\boldsymbol {a}}) ^ {T} \nabla h (\tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \left[ (\boldsymbol {0} \odot \boldsymbol {a}) ^ {T} \nabla h (\boldsymbol {a}) \right] = 0. \tag {20} \\ \end{array}
$$
Examining the second-order gradient in Eq. 19, and leveraging the independence of elements in  $m$ , we apply the result from Eq. 14. With  $\tilde{a}_k$  and  $a_k$  representing the  $k$ -th elements in  $\tilde{a}$  and  $a$ , respectively, we derive:
$$
\begin{array}{l} \mathbb {E} _ {\tilde {\boldsymbol {a}}} \mathbb {E} _ {\boldsymbol {m}} \left[ \frac {1}{2} (\boldsymbol {m} \odot \tilde {\boldsymbol {a}}) ^ {T} H ^ {(h)} (\tilde {\boldsymbol {a}}) (\boldsymbol {m} \odot \tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \mathbb {E} _ {\boldsymbol {m}} \left[ \frac {1}{2} (\boldsymbol {m} \odot \boldsymbol {a}) ^ {T} H ^ {(h)} (\boldsymbol {a}) (\boldsymbol {m} \odot \boldsymbol {a}) \right] \\ = \mathbb {E} _ {\tilde {\boldsymbol {a}}} \mathbb {E} _ {\boldsymbol {m}} \left[ \frac {1}{2} \sum_ {k = 1} ^ {d ^ {\prime}} m _ {k} \tilde {a} _ {k} H _ {k k} ^ {(h)} (\tilde {\boldsymbol {a}}) m _ {k} \tilde {a} _ {k} \right] - \mathbb {E} _ {\boldsymbol {a}} \mathbb {E} _ {\boldsymbol {m}} \left[ \frac {1}{2} \sum_ {k = 1} ^ {d ^ {\prime}} m _ {k} a _ {k} H _ {k k} ^ {(h)} (\boldsymbol {a}) m _ {k} \boldsymbol {a} _ {k} \right] \\ = \frac {\beta^ {2}}{2} \left(\mathbb {E} _ {\tilde {\boldsymbol {a}}} \left[ \sum_ {k} \tilde {a} _ {k} ^ {2} H _ {k k} ^ {(h)} (\tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \left[ \sum_ {k} a _ {k} ^ {2} H _ {k k} ^ {(h)} (\boldsymbol {a}) \right]\right). \tag {21} \\ \end{array}
$$
By substituting Eq. 18 with the results from Eq. 20, and Eq. 19 with Eq. 21, we arrive at the following expression:
$$
L _ {D} ^ {\mathrm {A N}} \approx \mathbb {E} _ {\tilde {\boldsymbol {a}}} [ h (\tilde {\boldsymbol {a}}) ] - \mathbb {E} _ {\boldsymbol {a}} [ h (\boldsymbol {a}) ] + \frac {\beta^ {2}}{2} \left(\mathbb {E} _ {\tilde {\boldsymbol {a}}} \left[ \sum_ {k} \tilde {a} _ {k} ^ {2} H _ {k k} ^ {(h)} (\tilde {\boldsymbol {a}}) \right] - \mathbb {E} _ {\boldsymbol {a}} \left[ \sum_ {k} a _ {k} ^ {2} H _ {k k} ^ {(h)} (\boldsymbol {a}) \right]\right). \tag {22}
$$
Utilizing the same derivation process as before for  $L_G^{\mathrm{AN}}$ , we arrive at the following result:
$$
L _ {G} ^ {\mathrm {A N}} \approx - \mathbb {E} _ {\tilde {\boldsymbol {a}}} [ h (\tilde {\boldsymbol {a}}) ] - \frac {\beta^ {2}}{2} \mathbb {E} _ {\tilde {\boldsymbol {a}}} \left[ \sum_ {k} \tilde {a} _ {k} ^ {2} H _ {k k} ^ {(h)} (\tilde {\boldsymbol {a}}) \right]. \tag {23}
$$
Introducing the multiplicative Gaussian noise during training of GAN produces the gradient maximization effect on the real samples in the discriminator training, specially through the term  $\frac{\beta^2}{2}\sum_k a_k^2 H_{kk}^{(h)}(\pmb{a})$ . This term intertwines the second-order partial derivative (diagonal entry  $H_{kk}^{(h)}(\pmb{a})$  of the Hessian matrix) with the non-negative factor  $\frac{\beta^2}{2} a_k^2 \geq 0$ , and is related to the trace of the Hessian matrix given by  $\operatorname{trace}\big(H^{(h)}(\pmb{a}_r)\big) = \sum_k H_{kk}^{(h)}(\pmb{a})$ . Employing similar reasoning for the fake samples when training the generator reveals that the noise-modulated discriminator training of GAN tends to amplify the diagonal entry of the second-order partial derivative in the Hessian matrix at both the real and fake points, influencing the optimization stability.
Theorem 2 Using notations from Prop. 1 and with  $f: \mathbb{R}^{d'} \to \mathbb{R}$ , let  $\nabla_k^2 f(\pmb{a})$  and  $\left(H_{jk}^{(f)}(\pmb{a})\right)^2$  denote the squares of the  $k$ -th gradient element and  $(j, k)$ -th entry of Hessian matrix of  $f$  at  $\pmb{a}$ , respectively. Given  $z_1, z_2 \sim \mathcal{N}(\pmb{1}, \beta^2 \pmb{I}^{d'})$ , enforcing invariance of  $f$  under different noise modulations yields:
$$
\ell^ {N I C E} (\boldsymbol {a}) := \mathbb {E} _ {\boldsymbol {z} _ {1}, \boldsymbol {z} _ {2}} \big [ \big (f (\boldsymbol {z} _ {1} \odot \boldsymbol {a}) - f (\boldsymbol {z} _ {2} \odot \boldsymbol {a}) \big) ^ {2} \big ] \approx 2 \beta^ {2} \sum_ {k} a _ {k} ^ {2} \nabla_ {k} ^ {2} f (\boldsymbol {a}) + \beta^ {4} \sum_ {j, k} a _ {j} ^ {2} a _ {k} ^ {2} (H _ {j k} ^ {(f)} (\boldsymbol {a})) ^ {2}.
$$
A concise derivation employs scalar values for the Taylor expansion, swiftly leading to results using  $f^{\prime}(\cdot)$  and  $f^{\prime \prime}(\cdot)$  for first and second-order derivatives, respectively.