SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \frac{\partial}{\partial\theta}\mathcal{D}^{(F)}(p_{\theta},q) = \mathbb{E}_{\substack{\boldsymbol {x} = g_{\theta}(\boldsymbol {z}),\\ \boldsymbol {z}\sim p_{z}}}\left[\boldsymbol{s}_{\theta}(\boldsymbol {x}) - \boldsymbol{s}_{q}(\boldsymbol {x})\right]\left[\frac{\partial\boldsymbol{s}_{\theta}(\boldsymbol{x})}{\partial\boldsymbol{x}} -\frac{\partial\boldsymbol{s}_{q}(\boldsymbol{x})}{\partial\boldsymbol{x}}\right]\frac{\partial\boldsymbol{x}}{\partial\theta} \\ + \mathbb {E} _ {\boldsymbol {z} \sim p _ {z}} \left[ \boldsymbol {s} _ {\theta} \left(g _ {\theta} (\boldsymbol {z})\right) - \boldsymbol {s} _ {q} \left(g _ {\theta} (\boldsymbol {z})\right) \right] \frac {\partial \boldsymbol {s} _ {\theta} (\boldsymbol {x})}{\partial \theta} | _ {\boldsymbol {x} = g _ {\theta} (\boldsymbol {z})} \tag {12} \\ = \operatorname {G r a d} ^ {(F, 1)} (\theta) + \operatorname {G r a d} ^ {(F, 2)} (\theta). \\ \end{array}
$$
$$
\mathcal {L} ^ {(F, 1)} (\theta) = \mathbb {E} _ {\boldsymbol {z} \sim p _ {z}} \frac {1}{2} \| \boldsymbol {s} _ {\theta^ {\dagger}} (g _ {\theta} (\boldsymbol {z})) - \boldsymbol {s} _ {q} (g _ {\theta} (\boldsymbol {z})) \| _ {2} ^ {2}. \tag {13}
$$
$$
\begin{array}{l} \operatorname {G r a d} ^ {(F, 2)} (\theta) = \mathbb {E} _ {\boldsymbol {x} \sim p _ {\theta}} \left[ \boldsymbol {s} _ {\theta} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) \right] \frac {\partial}{\partial \theta} \frac {\partial}{\partial \boldsymbol {x}} \log p _ {\theta} (\boldsymbol {x}) \\ = \mathbb {E} _ {x \sim p _ {\theta}} \left[ \boldsymbol {s} _ {\theta} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) \right] \frac {\partial}{\partial x} \frac {\partial}{\partial \theta} \log p _ {\theta} (\boldsymbol {x}) \\ = \int \left[ \boldsymbol {s} _ {\theta} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) \right] p _ {\theta} (\boldsymbol {x}) \frac {\partial}{\partial \boldsymbol {x}} \frac {\partial}{\partial \theta} \log p _ {\theta} (\boldsymbol {x}) d \boldsymbol {x} \\ = (\frac {\partial}{\partial \theta} \log p _ {\theta} (\boldsymbol {x}) [ \boldsymbol {s} _ {\theta} (\boldsymbol {x}) - \boldsymbol {s} _ {g} (\boldsymbol {x}) ]) p _ {\theta} (\boldsymbol {x}) | _ {\boldsymbol {x} \in \Omega} - \int p _ {\theta} (\boldsymbol {x}) \left[ (\nabla_ {\boldsymbol {x}} \log p _ {\theta} (\boldsymbol {x})) [ \boldsymbol {s} _ {\theta} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) ] \right. \\ \left. + \nabla_ {\boldsymbol {x}} \left[ \boldsymbol {s} _ {\theta} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) \right] \right] \frac {\partial}{\partial \theta} \log p _ {\theta} (\boldsymbol {x}) \\ = \mathbf {0} - \mathbb {E} _ {p _ {\theta}} \left[ \boldsymbol {s} _ {\theta} ^ {T} (\boldsymbol {x}) [ \boldsymbol {s} _ {\theta} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) ] + \nabla_ {\boldsymbol {x}} [ \boldsymbol {s} _ {\theta} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) ] \right] \frac {\partial}{\partial \theta} \log p _ {\theta} (\boldsymbol {x}) \\ = - \int \left[ \boldsymbol {s} _ {\theta} ^ {T} (\boldsymbol {x}) \left[ \boldsymbol {s} _ {\theta} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) \right] + \nabla_ {\boldsymbol {x}} \left[ \boldsymbol {s} _ {\theta} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) \right] \right] \frac {\partial}{\partial \theta} p _ {\theta} (\boldsymbol {x}) d \boldsymbol {x} \\ = - \frac {\partial}{\partial \theta} \int \left[ \boldsymbol {s} _ {\theta^ {\dagger}} ^ {T} (\boldsymbol {x}) \left[ \boldsymbol {s} _ {\theta^ {\dagger}} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) \right] + \nabla_ {\boldsymbol {x}} \left[ \boldsymbol {s} _ {\theta^ {\dagger}} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) \right] \right] p _ {\theta} (\boldsymbol {x}) \\ = - \frac {\partial}{\partial \theta} \mathbb {E} _ {p _ {\theta}} \left[ \boldsymbol {s} _ {\theta^ {\dagger}} ^ {T} (\boldsymbol {x}) \left[ \boldsymbol {s} _ {\theta^ {\dagger}} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) \right] + \nabla_ {\boldsymbol {x}} \left[ \boldsymbol {s} _ {\theta^ {\dagger}} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) \right] \right]. \tag {14} \\ \end{array}
$$
The notation  $\pmb{x} \in \Omega$  means the boundary of distribution density of  $p_{\theta}(\pmb{x})$  and the value  $p_{\theta}$  vanishes on the boundary. Usually, the integral over the boundary  $\Omega$  of the support of  $\pmb{x}$  vanishes if the density  $p_{\theta}$  decays fast enough and the logarithm of density has finite parameter derivatives  $\| \frac{\partial}{\partial \theta} \log p_{\theta}(\pmb{x}) \|_2$  for all  $\theta \in \Theta$ , then the first term vanishes. In practice, we find that the first term of the above expression does not bother the performance of Fisher training, however, for complex circumstances, the first term can not be guaranteed to be exactly 0. Besides, the above derivations are for the Fisher training, so our proposed KL training method is not bothered by them.
A.3 Additional Discussions on Proposed Algorithms
The training algorithm 1 consists of two alternative phases for training implicit sampler: the score estimation phase and the generator update phase. The former phase uses score-matching-related techniques to update the score network to match the score function of the implicit distribution. The latter phase uses the score function of the target distribution and the learned score network to update the generator's parameter in order to minimize the KL divergence between the implicit distribution and the target distribution.
Concerns on blindness and ill-landscape. Our proposed methods incorporate a score estimation phase, where a score neural network is used to estimate the score function of the sampler. However, two concerns arise regarding this score estimation step: the blindness issue and the presence of an ill-landscape in score estimation.
Blindness. The blindness [60] is a pervasive practical issue of score-based methods that states that two multi-modal distributions could potentially have very small Fisher divergence even if they are highly dissimilar. For our proposed training method that minimizes the Fisher divergence, there exist concerns that blindness would cause the training to fail. However, our intensive study shows that, with the introduced annealing techniques that train samplers in a progressive way, the blindness issue does not occur.
Ill-landscape. As is pointed out in Gong and Li [61], the parameter landscape of the Fisher divergence to heavy-tail distribution can be ill-posed, having a unique minimizer but hard to be caught by gradient-based optimization algorithms. For our proposed Fisher training, we find that the annealing technique and suitable parametrization of the score network overcome the ill parameter landscape of training. For a straight demonstration, we train an implicit sampler to capture the 1D student-t distribution, a heavy-tail distribution that has been studied in Gong and Li [61] and find that the sampler is capable of learning such a heavy-tail distribution correctly.
A.4 Details on Ill-Landscape
Ill-landscape. As is pointed out in Gong and Li [61], the parameter landscape of the Fisher divergence to heavy-tail distribution can be ill-posed, having a unique minimizer but hard to be caught by gradient-based optimization algorithms. The student-t distribution, a heavy-tail distribution, is a representative distribution that has an ill landscape as is shown in Gong and Li [61]. Since our Fisher training aims to minimize the Fisher divergence, there is a concern about the ill landscape of the Fisher training. However, for both our proposed KL and Fisher training, we find that suitable parametrization of the score network overcomes the ill parameter landscape of training. For a straight demonstration, we train an implicit sampler to capture the 1D student-t distribution, a heavy-tail distribution that has been studied in Gong and Li [61] and finds that the sampler is capable of learning such a heavy-tail distribution correctly. More specifically, We plot a comparison of the student- $t$  distribution and our learned sampler with Fisher training in Figure 3.
Figure 3: Illustration of proposed Implicit Sampler on a heavy-tail student-t target distribution.
A.5 Proofs of Equivalence to Fisher-Stein Sampler
We provide proof of Proposition 1 here.
Proposition 2. Estimating the sampler's score function  $s_{\phi}(.)$  with score matching is equivalent to the maximization of test function  $\mathbf{f}$  of Fisher-Stein's Discrepancy objective. More specially, the optimal score estimation  $S^{*}$  and Fisher-Stein's optimal test function  $\mathbf{f}^{*}$  satisfy
$$
\mathbf {f} ^ {*} (\boldsymbol {x}) = \frac {1}{2 \lambda} \left[ \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}) - S ^ {*} (\boldsymbol {x}) \right].
$$
Proof. With fixed  $p$  and known target  $q$ , the optimal test function  $\mathbf{f}^*$  has representation
$$
\mathbf {f} ^ {*} = \underset {\mathbf {f}} {\arg \min } \mathcal {L} (\mathbf {f})
$$
Where functional  $\mathcal{L}(\mathbf{f})$  has integral representation
$$
\begin{array}{l} \mathcal {L} (f) = \mathbb {E} _ {x \sim p} \left\{\langle \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}), \mathbf {f} (\boldsymbol {x}) \rangle + \langle \nabla_ {\boldsymbol {x}}, \mathbf {f} (\boldsymbol {x}) \rangle - \lambda [ \mathbf {f} ^ {T} (\boldsymbol {x}) \mathbf {f} (\boldsymbol {x}) ] \right\} \\ = \int p (\boldsymbol {x}) \left\langle \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}), \mathbf {f} (\boldsymbol {x}) \right\rangle + p (\boldsymbol {x}) \left\langle \nabla_ {\boldsymbol {x}}, \mathbf {f} (\boldsymbol {x}) \right\rangle - \lambda p (\boldsymbol {x}) \left[ \mathbf {f} ^ {T} (\boldsymbol {x}) \mathbf {f} (\boldsymbol {x}) \right] d \boldsymbol {x} \\ = \int l (\boldsymbol {x}, \mathbf {f}, \nabla \mathbf {f}) d \boldsymbol {x}. \\ \end{array}
$$
Here  $l(\pmb{x},\mathbf{f},\nabla \mathbf{f}) = \int p(\pmb{x})\langle \nabla_{\pmb{x}}\log q(\pmb{x}),\mathbf{f}(\pmb{x})\rangle +p(\pmb{x})\langle \nabla_{\pmb{x}},\mathbf{f}(\pmb{x})\rangle -\lambda p(\pmb{x})[\mathbf{f}^{T}(\pmb{x})\mathbf{f}(\pmb{x})]$ . By Euler-Lagrange equation, the optimal function  $\mathbf{f}$  satisfies
$$
\frac {\partial l}{\partial \mathbf {f}} - \frac {\mathrm {d}}{\mathrm {d} x} \left(\frac {\partial l}{\partial \mathbf {f} ^ {\prime}}\right) + \frac {\partial^ {2}}{\partial x ^ {2}} \left(\frac {\partial l}{\partial \mathbf {f} ^ {\prime \prime}}\right) = 0.
$$
$$
\begin{array}{l} \frac {\partial l}{\partial \mathbf {f}} (\boldsymbol {x}) = p (\boldsymbol {x}) \nabla \log q (\boldsymbol {x}) - 2 \lambda p (\boldsymbol {x}) \mathbf {f} (\boldsymbol {x}) \\ \frac {\mathrm {d}}{\mathrm {d} \boldsymbol {x}} (\frac {\partial l}{\partial \mathbf {f} ^ {\prime}}) (\boldsymbol {x}) = \nabla_ {\boldsymbol {x}} p (\boldsymbol {x}) \\ \frac {\partial l}{\partial \mathbf {f} ^ {\prime \prime}} (\boldsymbol {x}) = 0. \\ \end{array}
$$
So the optimal  $\mathbf{f}^*$  satisfies the Euler-Lagrange equation as
$$
p (\boldsymbol {x}) \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}) - 2 \lambda p (\boldsymbol {x}) \mathbf {f} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} p (\boldsymbol {x}) = 0.
$$
Divide the both side with  $p(\pmb{x})$  and note that  $\nabla_{\pmb{x}} p(\pmb{x}) / p(\pmb{x}) = \nabla_{\pmb{x}} \log p(\pmb{x})$ , the equation turns to
$$
\mathbf {f} ^ {*} (\boldsymbol {x}) = \frac {1}{2 \lambda} \left[ \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \log p (\boldsymbol {x}) \right].
$$
Next, consider optimal  $S^{*}$ . The  $S^{*}$  is obtained by minimizing the Score Matching objective, which is equivalent to minimizing the Fisher divergence between  $p$  and  $S$  induced family, thus the optimal  $S^{*}(\boldsymbol{x}) = \nabla_{\boldsymbol{x}} \log p(\boldsymbol{x})$ . Substitute  $\nabla_{\boldsymbol{x}} \log p(\boldsymbol{x})$  with  $S^{*}$  into  $f^{*}$  formula, we have
Next, we prove Theorem 2. Recall the definition of Fisher-Stein objective
$$
\min  _ {\theta} \max  _ {\eta} L (\theta , \eta) \tag {15}
$$
$$
L (\theta , \eta) = \mathbb {E} _ {p _ {\theta}} \left\langle \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}), \mathbf {f} _ {\eta} (\boldsymbol {x}) \right\rangle + \sum_ {d = 1} ^ {D} \frac {\partial}{\partial x _ {d}} f _ {d} (\boldsymbol {x}) - \lambda \| \mathbf {f} _ {\eta} (\boldsymbol {x}) \| _ {2} ^ {2} \tag {16}
$$
Theorem. Assume  $\mathbf{f}^*$  is the optimal test function that maximizes Fisher-Stein's objective 15. Then the minimization part of objective 15 for sampler  $G_{\theta}$  with the gradient-based algorithm is equivalent to the Fisher training, i.e. the minimization objective of 15 shares the same gradient with  $\mathrm{Grad}^{(F)}(\theta)$ .
Proof. Notice that by Proposition 2, the maximization of equation 15 to get an optimal  $\mathbf{f}^*$  is equivalent to learning a score function  $S^*$  with the score-matching related techniques, i.e.
$$
\mathbf {f} ^ {*} (\boldsymbol {x}) = \frac {1}{2 \lambda} \left[ \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}) - S ^ {*} (\boldsymbol {x}) \right], \tag {17}
$$
$$
S ^ {*} (\boldsymbol {x}) = \boldsymbol {s} _ {\theta} (\boldsymbol {x}), \forall x \tag {18}
$$
Here  $s_{\theta}(\pmb{x})$  denotes the unknown score function of the implicit sampler's distribution. Here we use Stein's identity which states
$$
\mathbb {E} _ {p _ {\theta}} \sum_ {d = 1} ^ {D} \frac {\partial}{\partial x _ {d}} f _ {d} (\boldsymbol {x}) = - \mathbb {E} _ {p _ {\theta}} \left\langle \boldsymbol {s} _ {\theta} (\boldsymbol {x}), \mathbf {f} (\boldsymbol {x}) \right\rangle \tag {19}
$$
Put the equation 19 and  $\mathbf{f}^*$  in objective 15 with  $S^{*}$  according to the relation 17, we obtain
$$
\begin{array}{l} \min  _ {\theta} L (\theta) (20) \\ L (\theta) = \mathbb {E} _ {p _ {\theta}} \left\langle \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}), \mathbf {f} ^ {*} (\boldsymbol {x}) \right\rangle - \left\langle \boldsymbol {s} _ {\theta}, \mathbf {f} ^ {*} (\boldsymbol {x}) \right\rangle - \lambda \| \mathbf {f} ^ {*} (\boldsymbol {x}) \| _ {2} ^ {2} (21) \\ = \mathbb {E} _ {p _ {\theta}} \left\langle \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}) - \boldsymbol {s} _ {\theta}, \mathbf {f} ^ {*} (\boldsymbol {x}) \right\rangle - \lambda \| \mathbf {f} ^ {*} (\boldsymbol {x}) \| _ {2} ^ {2} (22) \\ = \mathbb {E} _ {p _ {\theta}} \left\langle \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}) - S ^ {*} (\boldsymbol {x}) + S ^ {*} (\boldsymbol {x}) - \boldsymbol {s} _ {\theta} (\boldsymbol {x}), \mathbf {f} ^ {*} (\boldsymbol {x}) \right\rangle - \lambda \| \mathbf {f} ^ {*} (\boldsymbol {x}) \| _ {2} ^ {2} (23) \\ = \mathbb {E} _ {p _ {\theta}} \left\langle \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}) - S ^ {*} (\boldsymbol {x}), \mathbf {f} ^ {*} (\boldsymbol {x}) \right\rangle + \left\langle S ^ {*} (\boldsymbol {x}) - s _ {\theta} (\boldsymbol {x}), \mathbf {f} ^ {*} (\boldsymbol {x}) \right\rangle - \lambda \| \mathbf {f} ^ {*} (\boldsymbol {x}) \| _ {2} ^ {2} (24) \\ = \mathbb {E} _ {p _ {\theta}} 2 \lambda \| \mathbf {f} ^ {*} \| _ {2} ^ {2} + \left\langle S ^ {*} (\boldsymbol {x}) - \boldsymbol {s} _ {\theta} (\boldsymbol {x}), \mathbf {f} ^ {*} (\boldsymbol {x}) \right\rangle - \lambda \| \mathbf {f} ^ {*} (\boldsymbol {x}) \| _ {2} ^ {2} (25) \\ = \mathbb {E} _ {p _ {\theta}} \lambda \| \mathbf {f} ^ {*} \| _ {2} ^ {2} + \left\langle S ^ {*} (\boldsymbol {x}) - s _ {\theta} (\boldsymbol {x}), \mathbf {f} ^ {*} (\boldsymbol {x}) \right\rangle (26) \\ \end{array}
$$
Taking the  $\theta$  gradient of the above objective, we have
$$
\begin{array}{l} \frac {\partial}{\partial \theta} L (\theta) = \lambda \frac {\partial}{\partial \theta} \mathbb {E} _ {p _ {\theta}} \| \mathbf {f} ^ {*} (\boldsymbol {x}) \| _ {2} ^ {2} + \mathbb {E} _ {p _ {\theta}} \mathbf {f} ^ {*} (\boldsymbol {x}) ^ {T} \cdot \frac {\partial \boldsymbol {s} _ {\theta} (\boldsymbol {x})}{\partial \theta} (27) \\ = \frac {1}{4 \lambda} \frac {\partial}{\partial \theta} \mathbb {E} _ {p _ {\theta}} \| \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}) - S ^ {*} (\boldsymbol {x}) \| _ {2} ^ {2} - \frac {1}{2 \lambda} \mathbb {E} _ {p _ {\theta}} \left(\nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}) - S ^ {*} (\boldsymbol {x})\right) ^ {T} \cdot \frac {\partial \boldsymbol {s} _ {\theta} (\boldsymbol {x})}{\partial \theta} (28) \\ = \frac {1}{4 \lambda} \frac {\partial}{\partial \theta} \mathbb {E} _ {p _ {\theta}} \| S ^ {*} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}) \| _ {2} ^ {2} + \frac {1}{2 \lambda} \mathbb {E} _ {p _ {\theta}} \left(S ^ {*} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x})\right) ^ {T} \cdot \frac {\partial s _ {\theta} (\boldsymbol {x})}{\partial \theta}. (29) \\ \end{array}
$$
In the above equation, some gradient terms vanish because  $S^{*}(\pmb{x}) = \pmb{s}_{\theta}(\pmb{x})$  for all  $x$ . Since the  $p_{\theta}$  is implemented with the implicit sampler, i.e.  $x \sim p_{\theta}$  means  $x = G_{\theta}(z), z \sim p_{z}(z)$ . So above gradient term can be further simplified as
$$
\begin{array}{l} \frac {1}{4 \lambda} \frac {\partial}{\partial \theta} \mathbb {E} _ {p _ {\theta}} \| S ^ {*} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x}) \| _ {2} ^ {2} + \frac {1}{2 \lambda} \mathbb {E} _ {p _ {\theta}} \left(S ^ {*} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x})\right) ^ {T} \cdot \frac {\partial s _ {\theta} (\boldsymbol {x})}{\partial \theta} (30) \\ = \frac {1}{4 \lambda} \frac {\partial}{\partial \theta} \mathbb {E} _ {z \sim p _ {z}} \| S ^ {*} (G _ {\theta} (z)) - \nabla_ {\boldsymbol {x}} \log q (G _ {\theta} (z)) \| _ {2} ^ {2} + \frac {1}{2 \lambda} \mathbb {E} _ {p _ {\theta}} (S ^ {*} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x})) ^ {T} \cdot \frac {\partial \boldsymbol {s} _ {\theta} (\boldsymbol {x})}{\partial \theta} (31) \\ = \frac {1}{2 \lambda} \mathbb {E} _ {\substack {x = G _ {\theta} (z) \\ z \sim p _ {z}}} \left[ S ^ {*} (\boldsymbol {x}) - \boldsymbol {s} _ {q} (\boldsymbol {x}) \right] \cdot \left[ \frac {\partial S ^ {*} (\boldsymbol {x})}{\partial x} - \frac {\partial \boldsymbol {s} _ {q} (\boldsymbol {x})}{x} \right] \cdot \frac {\partial x}{\partial \theta} (32) \\ + \frac {1}{2 \lambda} \mathbb {E} _ {p _ {\theta}} \left(S ^ {*} (\boldsymbol {x}) - \nabla_ {\boldsymbol {x}} \log q (\boldsymbol {x})\right) ^ {T} \cdot \frac {\partial \boldsymbol {s} _ {\theta} (\boldsymbol {x})}{\partial \theta}. (33) \\ \end{array}
$$
Here  $s_q(\pmb{x}) \coloneqq \nabla_{\pmb{x}} \log q(\pmb{x})$ . So the  $\theta$  gradient is the same as the Fisher training (equation (5)) in the paper.
B.1 Experiment Details on 2D Synthetic Sampling
Model architectures. For toy 2-dimensional data experiments, we use a 4-layer MLP neural network with 200 hidden units in each layer as the sampler. The activation of the sampler is chosen as LeakyReLU non-linearity with a 0.2 coefficient. The score network is a 4-layer MLP with 200 hidden units in each layer. The activation of the score network is GELU non-linearity.
Evaluation metrics. We calculate the Kernelized Stein Discrepancy with multi-scale RBM kernel as is implemented in codebase 7. We set the bandwidth of the kernel to be 0.25.
Comparison of KL and Fisher training methods. In this work, we propose the KL training method and the Fisher training method for training neural implicit samplers. Both methods require two phases which update a neural score network to match the score function of implicit distribution and update the implicit sampler's parameter in order to minimize either the KL divergence or the Fisher divergence. To this end, we give some comparisons of the proposed two training methods. Empirically, we find that the KL training methods work better than Fisher training under the KSD for 2D targets. We also observe that the Fisher training tends to learn an implicit sampler that collapses on certain modes of the target distribution. Besides, since the Fisher training requires calculating the data derivative of the score function, i.e.  $\nabla_{\boldsymbol{x}}[s_q(\boldsymbol{x}) - s_{\theta^{\dagger}}(\boldsymbol{x})]$ , thus requires the score function to be differentiable. The non-differentiable activate functions such as ReLU and LeakyReLU are not suitable for implementing Fisher training. However, the KL training does not suffer from such issues. Besides, the computational costs of Fisher training are also more expensive than KL training for each iteration.
B.2 Experiment Details on Bayesian Regression
Experiment settings. We use the same setting as [11], where the prior of the weights is  $p(w \mid \alpha) = \mathcal{N}(w; 0, \alpha^{-1})$  and  $p(\alpha) = \mathrm{Gamma}(\alpha; 1, 0.01)$ . The Covertype dataset [62] (581,012 samples with 54 features) is randomly split into the training set (80%) and testing set (20%). Our methods are compared with Stein GAN, SVGD, SGLD, doubly stochastic variational inference (DSVI) [63], KSD-NS, and Fisher-NS [11] on this data set. For SGLD, DSVI, and SVGD, the model is trained with 3 epochs of the training set (about 15k iterations), while for neural network-based methods (Fisher-NS, KSD-NS, and Stein GAN), the training is run until convergence. Since we have shown the equivalence of Fisher-NS [11], we only train the implicit sampler with the KL training method. We set the training learning rate for both the score network and the sampler to be 0.0002. The target distribution (posterior distribution) is computed with 500 random data samples. We set the number of score estimation phases as 2. We evaluate the test accuracy of the logistic regression model with 100 samples from the trained sampler.
<sup>7</sup>https://github.com/WenboGong/Sliced_Kernelized_Stein_Discrepancy.git
Figure 4: Test accuracy of KL sampler with the different number of iterations.
We use the same settings for Bayesian inference as in Hu et al. [11]. The neural samplers are implemented via a 4-layer MLP with 1024 hidden units in each layer and GELU activation functions. The output dimension of the sampler is 55 and the input dimension is set to be  $55 \times 10 = 550$ , following the same setting as in [1]. The score network has the same neural architecture as the sampler, but the input dimension is set to 55 which matches the output dimension. We use Adam optimizers with a learning rate of 0.0002 and default beta values for both sampler and score networks. We use a batch size of 100 for training the sampler and use 2 updates for score network for each update of the sampler network. We use the standard score matching for learning the score network. We train the sampler for 10k iterations for each repeat and use 30 independent repeats to compute the mean and std of the test accuracy. The learning rate of SGLD is chosen to be  $0.1 / (t + 1)^{0.55}$  as suggested in Welling and Teh [51], and the average of the last 100 points is used for evaluation. For DSVI, the learning rate is  $1e - 07$ , and 100 iterations are used for each stage. For SVGD, we use RBF kernel with bandwidth h calculated by the median trick as in Liu and Wang [50], and 100 particles are used for evaluation with step size being 0.05.
Model architectures. We use 4-layer MLP neural networks as the sampler and score network, respectively. The activation of the sampler is chosen as Gaussian Error Linear Units (GELU) [64] non-linearity for both the score network and the sampler. We set the hidden dimension to 1024. The input dimension of the sampler is set to 128.
B.3 Experiment Details on Sampling from EBMs