SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We start by proving the universality of the free energy for a Gibbs distribution defined through the objective  $\widehat{\mathcal{R}}_n(\Theta; X, y)$  for the data distribution (1) and its equivalent Gaussian mixture (8).
Theorem 1 (Universality of Free Energy). Let  $\mu_p(\Theta)$  be a sequence of Borel probability measures with compact supports  $S_p$ . Define the following free energy function:
$$
f _ {\beta , n} (\boldsymbol {X}) = - \frac {1}{\beta n} \log \int \exp \left(- \beta n \widehat {\mathcal {R}} _ {n} (\boldsymbol {\Theta}; \boldsymbol {X}, \boldsymbol {y} (\boldsymbol {X}))\right) d \mu_ {p} (\boldsymbol {\Theta}) \tag {12}
$$
Under Assumptions 1-4 on  $\mathbf{X}$  and  $S_{p}$ , for any bounded differentiable function  $\Phi$  with bounded Lipschitz derivative, we have:
$$
\lim _ {n, p \to \infty} | \mathbb {E} \left[ \Phi \left(f _ {\beta , n} (\boldsymbol {X})\right) \right] - \mathbb {E} \left[ \Phi \left(f _ {\beta , n} (\boldsymbol {G})\right) \right] | = 0.
$$
When  $\mu_p$  corresponds to discrete measures supported on an  $\epsilon$ -net in  $\mathcal{S}_p$ , using the reduction from Lemma 1 to Theorem 1 in [17], we obtain the following corollary:
Corollary 2 (Universality of Training Error). For any bounded Lipschitz function  $\Phi : \mathbb{R} \to \mathbb{R}$ :
$$
\lim  _ {n, p \rightarrow \infty} \left| \mathbb {E} \left[ \Phi \left(\widehat {\mathcal {R}} _ {n} ^ {\star} (\boldsymbol {X}, \boldsymbol {y} (\boldsymbol {X}))\right)\right] - \mathbb {E} \left[ \Phi \left(\widehat {\mathcal {R}} _ {n} ^ {\star} (\boldsymbol {G}, \boldsymbol {y} (\boldsymbol {G}))\right)\right]\right| = 0
$$
In particular, for any  $\mathcal{E} \in \mathbb{R}$ , and denoting  $\xrightarrow{\mathbb{P}}$  the convergence in probability:
$$
\widehat {\mathcal {R}} _ {n} ^ {\star} (\boldsymbol {X}, \boldsymbol {y} (\boldsymbol {X})) \xrightarrow {\mathbb {P}} \mathcal {E} \quad i f a n d o n l y i f \quad \widehat {\mathcal {R}} _ {n} ^ {\star} (\boldsymbol {G}, \boldsymbol {y} (\boldsymbol {G})) \xrightarrow {\mathbb {P}} \mathcal {E}, \tag {13}
$$
The full theorem, as well as its proof, is presented in Appendix A, along with additional remarks and an intuitive sketch. The proof combines the conditional 1d-CLT in Assumption 4 with the
interpolation of the free-energy in [17]. For strongly-convex losses, one may alternatively use the Lindeberg's method as in [16].
In a nutshell, this theorem shows that the multi-modal data generated by any generative neural network is equivalent to a finite mixture of Gaussian in high-dimensions: in other words, a finite mixture of Gaussians leads to the same loss as for data generated by (for instance) a cGAN. Since the function  $\ell : \mathbb{R}^{k+1} \to \mathbb{R}$  need not be convex, we can take
$$
\ell (\boldsymbol {x} _ {\mathrm {o u t}}, y) = \ell^ {\prime} (\boldsymbol {\Psi} (\boldsymbol {x} _ {\mathrm {o u t}}), y),
$$
where  $\Psi$  is an already pretrained neural network. In particular, if  $x$  is the output of all but the last layer of a neural net, we can view  $\Psi$  as the averaging procedure for a small committee machine.
Note that Corollary 2 depends crucially on Assumption 4 (the one-dimensional CLT), which is by no means evident. We discuss the conditions on the weights matrix for which it can be proven in Section 2.4. However, one can observe empirically that the validity of Corollary 2 goes well beyond what can be currently proven. A number of numerical illustrations of this property can be found in the work of [23, 39, 40], who already derived similar (albeit more limited) results using random matrix theory. Additionally, we observed that even with trained GANs, when we observed data through a random feature map [46], the Gaussian mixture universality is well obeyed. This scenario is illustrated in Fig. 1, with a concrete example in Fig. 2. Even though we did not prove the one-dimensional CLT for arbitrary learned matrices, and worked with finite moderate sizes, the realistic data generated by our cGAN behaves extremely closely to those generated by the corresponding Gaussian mixture.
A second remark is that the interest of Corollary 2 lies in the fact that it requires only a finite mixture to approximate the loss. Indeed, while we could use the standard approximation results (e.g. the Stone-Weierstrass theorem) to approximate the data density to arbitrary precision by Gaussian mixtures, this would require a diverging number of Gaussian in the mixture. The fact that loss is captured with finite  $\mathcal{C}$  is key to our approach.
2.2 Convergence of expectations for Joint Minimization and Sampling
Our next result establishes a general relationship between the differentiability of the limit of expected training errors or free energies for empirical risk minimization or free energies for sampling and the universality of expectations of a given function of a set of parameters arising from multiple objectives. As a motivating example, consider the uncertainty quantification in Section 1 that uses both Bayesian and ERM estimators [19, 24]. The parameters  $\hat{\Theta}_{\mathrm{erm}}$  and  $\Theta_{\mathrm{Bayes}}$  are obtained through empirical risk minimization and posterior sampling respectively on the same sequence of training data. In general, the inputs used in different objectives could be different but have some correlation structure. In the setup of ensembling (Equation 7), they are correlated through the feature mapping  $z_{i} \mapsto x_{i}^{(m)} = \varphi_{m}(z_{i})$ . In light of these considerations, we present the following general setup: Consider a sequence of  $M$  risks:
$$
\widehat {\mathcal {R}} _ {n} ^ {(m)} (\boldsymbol {\Theta}; \boldsymbol {X} ^ {(m)}, \boldsymbol {y} ^ {(m)}) := \frac {1}{n} \sum_ {i = 1} ^ {n} \ell_ {m} \left(\boldsymbol {\Theta} ^ {\top} \boldsymbol {x} _ {i} ^ {(m)}, y _ {i} ^ {(m)}\right) + r _ {m} (\boldsymbol {\Theta}), \quad m \in [ M ] \tag {14}
$$
with possibly different losses, regularizers and datasets. For simplicity, we assume that the objectives are defined on parameters having the same dimension  $\Theta \in \mathbb{R}^{p\times k}$ . We aim to minimize  $M_{1}$  of them:
$$
\hat {\boldsymbol {\Theta}} ^ {(m)} (\boldsymbol {X}) \in \underset {\boldsymbol {\Theta} \in \mathcal {S} _ {p} ^ {(m)}} {\arg \min } \widehat {\mathcal {R}} _ {n} ^ {(m)} (\boldsymbol {\Theta}; \boldsymbol {X} ^ {(m)}, \boldsymbol {y} ^ {(m)}), \quad m \in [ M _ {1} ] \tag {15}
$$
and the  $M_2$  remaining parameters are independently sampled from a family of Gibbs distributions:
$$
\boldsymbol {\Theta} ^ {(m)} \sim P _ {m} (\boldsymbol {\Theta}) \propto \exp \left(- \beta_ {m} \widehat {\mathcal {R}} _ {n} ^ {(m)} \left(\boldsymbol {\Theta}; \boldsymbol {X} ^ {(m)}, \boldsymbol {y} ^ {(m)}\right)\right) d \mu_ {m} (\boldsymbol {\Theta}), \quad m \in [ M _ {1} + 1, M ], \tag {16}
$$
where  $M = M_{1} + M_{2}$ . The joint distribution of the  $\pmb{x}_i = (\pmb{x}_i^{(1)},\dots,\pmb{x}_i^{(M)})$  is assumed to be a mixture of the form (1). However, we assume that the labels  $\pmb{y}_i^{(m)}$  only depend on the vectors  $\pmb{x}_i^{(m)}$ :
$$
y _ {i} ^ {(m)} \left(\boldsymbol {X} ^ {(m)}\right) = \eta \left(\boldsymbol {\Theta} _ {\star} ^ {(m) \top} \boldsymbol {x} _ {i} ^ {(m)}, \varepsilon_ {i} ^ {(m)}, c _ {i}\right). \tag {17}
$$
The equivalent Gaussian inputs  $\pmb{g}_i = (\pmb{g}_i^{(1)},\dots,\pmb{g}_i^{(M)})$  and their labels  $\pmb{y}(\pmb{G})$  are defined as in (8).
Statistical metric and free energy — Our goal is to study statistical metrics for some function  $h: \mathbb{R}^{M \times k \times p} \to \mathbb{R}$  of the form  $h(\Theta^{(1)}, \dots, \Theta^{(M)})$ . For instance, the metric  $h$  could be the population risk (a.k.a. generalization error), or some overlap between  $\Theta$  and  $\Theta_{\star}$ . We define the following coupling free energy function:
$$
f _ {n, s} (\boldsymbol {\Theta} [ 1: M _ {1} ], \boldsymbol {X}, \boldsymbol {y}) = - \frac {1}{n} \log \int e ^ {- s n h (\boldsymbol {\Theta} ^ {(1)}, \dots , \boldsymbol {\Theta} ^ {(M)})} d P ^ {(M _ {1} + 1): M}, \tag {18}
$$
where  $P^{(M_1 + 1):M}$  denotes the product measure of the  $P_{m}$  defined in (16). This gives rise to the following joint objective:
$$
\widehat {\mathcal {R}} _ {n, s} \left(\boldsymbol {\Theta} [ 1: M _ {1} ], \boldsymbol {X}, \boldsymbol {y}\right) = \sum_ {m = 1} ^ {M _ {1}} \widehat {\mathcal {R}} _ {n} ^ {(m)} \left(\boldsymbol {\Theta} ^ {(m)}; \boldsymbol {X} ^ {(m)}, \boldsymbol {y} ^ {(m)}\right) + f _ {n, s} \left(\boldsymbol {\Theta} [ 1: M _ {1} ], \boldsymbol {X}, \boldsymbol {y}\right). \tag {19}
$$
In particular, when  $s = 0$  we have  $f_{n,0} = 0$  and the problem reduces to the joint minimization problem in (15). Our first result concerns the universality of the minimum of the above problem:
Proposition 3 (Universality for joint minimization and sampling). Under Assumptions 1-4, for any  $s > 0$  and bounded Lipschitz function  $\Phi : \mathbb{R} \to \mathbb{R}$ , and denoting  $\widehat{\mathcal{R}}_{n,s}^{\star}(\boldsymbol{X},\boldsymbol{y}) \coloneqq \min \widehat{\mathcal{R}}_{n,s}(\Theta; \boldsymbol{X},\boldsymbol{y})$ :
$$
\lim  _ {n, p \rightarrow \infty} \left| \mathbb {E} \left[ \Phi \left(\widehat {\mathcal {R}} _ {n, s} ^ {\star} (\boldsymbol {X}, \boldsymbol {y} (\boldsymbol {X}))\right)\right] - \mathbb {E} \left[ \Phi \left(\widehat {\mathcal {R}} _ {n, s} ^ {\star} (\boldsymbol {G}, \boldsymbol {y} (\boldsymbol {G}))\right)\right]\right| = 0
$$
The proof uses a reduction to Corollary 2, and can be found in App. A.5. The next result concerns the value of  $h$  at the minimizers point  $(\hat{\Theta}^{(1)},\dots ,\hat{\Theta}^{(M)})$ . We make the following additional assumptions:
Assumption 5 (Differentiable Limit). There exists a neighborhood of 0 such that the function  $q_{n}(s) = \mathbb{E}\left[\widehat{\mathcal{R}}_{n,s}^{\star}(\pmb {G},\pmb {y}(\pmb {G}))\right]$  converges pointwise to a function  $q(s)$  that is differentiable at 0.
The above assumption stems from the convexity based argument used to prove Theorem 4.
For a fixed realization of the dataset  $X$ , we denote by  $\langle h(\Theta^{(1)}, \dots, \Theta^{(M)}) \rangle_{X}$  the expected value of  $h$  when  $(\hat{\Theta}^{(1)}, \dots, \hat{\Theta}^{(M_1)})$  are obtained through the minimization of (15) and  $(\Theta^{(M_1 + 1)}, \dots, \Theta^{(M)})$  are sampled according to the Boltzmann distributions (16).
Assumption 6. With high probability on  $X$ ,  $G$ , the value  $\left\langle h(\Theta^{(1)}, \dots, \Theta^{(M)}) \right\rangle_{X}$  (resp. the same for  $G$ ) is independent from the chosen minimizers in (15).
The above assumption is motivated by the fact that commonly non-convex problems contain minima exhibiting a specific symmetry. For example, all the global minima for a two-layer neural network are permutation invariant. Assumption 6 reflects that the quantity  $h$  respects these symmetries by taking the same value at each global minimum. This can be replaced by the stronger condition of a unique minimizer. Then the following holds:
Theorem 4. Under Assumptions 1-6, we have:
$$
\lim  _ {n, p \rightarrow \infty} \left| \mathbb {E} \left[\left\langle h \left(\boldsymbol {\Theta} ^ {(1)}, \dots , \boldsymbol {\Theta} ^ {(M)}\right)\right\rangle_ {\boldsymbol {X}} \right] - \mathbb {E} \left[\left\langle h \left(\boldsymbol {\Theta} ^ {(1)}, \dots , \boldsymbol {\Theta} ^ {(M)}\right)\right\rangle_ {\boldsymbol {G}} \right]\right| = 0, \tag {20}
$$
Proof Sketch: Our proof relies on the observation that  $q_{n}(s)$  is a concave function of  $s$ . Further:
$$
q _ {n} ^ {\prime} (0) = \mathbb {E} \left[ \left\langle h \left(\boldsymbol {\Theta} ^ {(1)}, \dots , \boldsymbol {\Theta} ^ {(M)}\right) \right\rangle_ {\boldsymbol {G}} \right]. \tag {21}
$$
This allows us to leverage a result of convex analysis relating the convergence of a sequence of convex or concave functions to the convergence of the corresponding derivatives, bypassing the more involved probabilistic arguments in [16, 17]. Our approach also generalizes in a straightforward manner to the setup of multiple objectives.
The above result shows that the expected value of  $h\left(\Theta^{(1)}, \dots, \Theta^{(M)}\right)$  for a multi-modal data satisfying the 1d CLT is equivalent to that of a mixture of Gaussians. The full theorem is presented and proven in Appendix A.
2.3 Universal Weak Convergence
Theorem 4 provides a general framework for proving the equivalence of arbitrary functions of parameters obtained by minimization/sampling on a given mixture dataset and the equivalent gaussian mixture distribution. However, it relies on the assumption of a differentiable limit of the free energy (Assumption 5). If the assumption holds for a sequence of functions belonging to dense subsets of particular classes of functions, it allows us to prove convergence of minimizers themselves, in a weak sense. We illustrate this through a simple setup considered in [12], which precisely characterized the asymptotic distribution of the minimizers of empirical risk with GMM data in the strictly convex case. Consider the following setup:
$$
\left(\hat {\boldsymbol {W}} ^ {\boldsymbol {X}}, \hat {\boldsymbol {b}} ^ {\boldsymbol {X}}\right) = \underset {\boldsymbol {W}, \boldsymbol {b}} {\arg \min } \sum_ {i = 1} ^ {n} \ell \left(\frac {\boldsymbol {W} \boldsymbol {x} _ {i}}{\sqrt {d}} + \boldsymbol {b}, \boldsymbol {y} _ {i}\right) + \lambda r (\boldsymbol {W}), \tag {22}
$$
where  $\pmb{W} \in \mathbb{R}^{|\mathcal{C}| \times d}$ ,  $\pmb{b} \in \mathbb{R}^{|\mathcal{C}|}$  and  $\pmb{y}_i \in \mathbb{R}^{|\mathcal{C}|}$  is the one-hot encoding of the class index  $c_i$ . We make the following assumptions:
Assumption 7. All of the covariance matrices  $\Sigma_{c}$  are diagonal, with strictly positive eigenvalues  $(\sigma_{c,i})_{i\in [d]}$ , and there exists a constant  $M > 0$  such that for any  $c\in \mathcal{C}$  we have  $\sigma_{c,i}\leq M$  and  $\| \pmb {\mu}_c\| _2\leq M$
Secondly, since we aim at obtaining a result on the weak convergence of the estimators, we assume the same weak convergence for the means and covariances, and that the regularization only depends on the empirical measure of  $W$ .
Assumption 8. The empirical distribution of the  $\mu_c$  and  $\Sigma_c$  converges weakly as follows:
$$
\frac {1}{d} \sum_ {i = 1} ^ {d} \prod_ {c \in \mathcal {C}} \delta \left(\mu_ {c} - \sqrt {d} \mu_ {c, i}\right) \delta \left(\sigma_ {c} - \sigma_ {c, i}\right) \quad \xrightarrow {\mathcal {L}} \quad p (\boldsymbol {\sigma}, \boldsymbol {\mu}) \tag {23}
$$
Assumption 9. The regularizer  $r(\cdot)$  is a pseudo-Lipshitz function of finite-order having the following form:  $r(\mathbf{W}) = \sum_{i=1}^{d} \psi_r(\mathbf{W}_i)$ , for some convex, differentiable function  $\psi_r : \mathbb{R} \to \mathbb{R}$ . This includes, in particular the squared regularization  $r(\mathbf{W}) = \sum_{i=1}^{d} \mathbf{W}_i^2$ .
We briefly comment on the choice of the above assumptions. The boundedness of the  $\Sigma_{c}$  and  $\mu$  in Assumption 7 guarantees that we are in a case covered both by [12] and by the assumptions of Theorem 4. The diagonal property of the  $\Sigma_{c}$  in 7, as well as the joint convergence in Assumption 8, ensure that we can view the minimization problem 22 ensures that  $W^{\star}, b^{\star}$  converge towards a well-defined limit. Finally, the separability assumption on  $r$  in Assumption 9 responds to the fact that we aim for a result on the empirical coordinate distribution of  $W^{\star}, b^{\star}$
Under these conditions, the joint empirical measure of the minimizers and of the data moments converges weakly to a fixed limit, independent of the data-distribution:
Theorem 5. Assume that conditions 1-9 hold, and further that the function  $\ell(\bullet, y) + r(\bullet)$  is convex, coercive and differentiable. Then, for any bounded-Lipschitz function:  $\Phi : \mathbb{R}^{3|\mathcal{C}|} \to \mathbb{R}$ , we have:
$$
\mathbb {E} \left[ \frac {1}{d} \sum_ {i = 1} ^ {d} \Phi \left(\left\{\left(\hat {\boldsymbol {W}} ^ {\boldsymbol {X}}\right) _ {c, i} \right\} _ {c \in \mathcal {C}}, \left\{\mu_ {c, i} \right\} _ {c \in \mathcal {C}}, \left\{\sigma_ {c, i} \right\} _ {c \in \mathcal {C}}\right)\right] \xrightarrow [ n / d = \alpha > 0 ]{n , d \rightarrow + \infty} \mathbb {E} _ {\tilde {p}} \left[ \Phi (\boldsymbol {w}, \boldsymbol {\mu}, \boldsymbol {\sigma}) \right], \tag {24}
$$
where  $\tilde{p}$  is a measure on  $\mathbb{R}^{3|C|}$ , that is determined by the so-called replica equations.
Proof Sketch The proof starts with the observation that the nonlinear system of (replica) equations in [12] describes the joint-empirical measure of the parameters, means and covariances of the mixtures in a self-consistent manner. Furthermore, for  $h(\mathbf{W})$  having bounded second derivatives, the perturbation term  $sh(\mathbf{W})$  can be absorbed into the regularizer. We then utilize topological and analytical arguments to relate the weak convergence to the differentiability Assumption 5 for functions that can be expressed as expectations w.r.t the joint empirical measure in 24. More details can be found in Appendix A.8.
In particular, the above result implies the universality of the overlaps of the minimizers with means, covariances, as well as their geometrical properties such as  $L^p$  norms.
2.4 One-dimensional CLT for Random Features
We finally show a conditional one-dimensional CLT for a random features map applied to a mixture of gaussians, in the vein of those shown in [14, 16, 17]. Concretely, we consider the following setup:
$$
\boldsymbol {x} _ {i} = \sigma (\boldsymbol {F} \boldsymbol {z} _ {i}), \quad \boldsymbol {z} _ {i} \sim \sum_ {c \in \mathcal {C}} \mathcal {N} \left(\boldsymbol {\mu} _ {c} ^ {\boldsymbol {z}}, \boldsymbol {\Sigma} _ {c} ^ {\boldsymbol {z}}\right), \tag {25}
$$
where the feature matrix  $\mathbf{F} \in \mathbb{R}^{p \times d}$  has i.i.d  $\mathcal{N}(0,1 / d)$  entries. This setup is much more permissive than the ones in [16, 17], that restrict the samples  $\mathbf{z}$  to standard normal vectors. However, we do require some technical assumptions:
Assumption 10. The activation function  $\sigma$  is thrice differentiable, with  $\| \sigma^{(i)}\| \leq M$  for some  $M > 0$  and we have  $\mathbb{E}_{g\sim \mathcal{N}(0,1)}[\sigma (g)] = 0$ . Additionally, the cluster means and covariances of  $\pmb{z}$  satisfy for all  $c\in \mathcal{C}$ $\| \pmb{\mu}_c^{\pmb{z}}\| \leq M,\| \pmb{\Sigma}_c^{\pmb{z}}\|_{\mathrm{op}}\leq M$  for some constant  $M > 0$ .
We also place ourselves in the proportional regime, i.e. a regime where  $p / d \in [\gamma^{-1}, \gamma]$  for some  $\gamma > 0$ . For simplicity, we will consider the case  $k = 1$ ; and the constraint set  $S_{p}$  as follows:
$$
\mathcal {S} _ {p} = \left\{\boldsymbol {\theta} \in \mathbb {R} ^ {d} \mid \| \boldsymbol {\theta} \| _ {2} \leq R, \| \boldsymbol {\theta} \| _ {\infty} \leq C p ^ {- \eta} \right\} \tag {26}
$$
for a given  $\eta > 0$ . We show in the appendix the following theorem:
Theorem 6. Under Assumption 10, and with high probability on the feature matrix  $\mathbf{F}$ , the data  $\mathbf{X}$  satisfy the concentration assumption 2, as well as the one-dimensional CLT of Assumption 4. Consequently, the results of Theorems 1 and 4 apply to  $\mathbf{X}$  and their Gaussian equivalent  $\mathbf{G}$ .
Proof Sketch Our proof proceeds by defining the following neuron-wise activation functions:
$$
\sigma_ {i, c} (u) = \sigma \left(u + \boldsymbol {f} _ {i} ^ {\top} \boldsymbol {\mu} _ {c}\right). \tag {27}
$$
We subsequently control the effects of the means, covariances and the dimensions of the inputs to prove a result analogous to the one-dimensional CLT for random features in [16, 17, 14]. While we prove the above result for random weights, we note, however that the non-asymptotic results in [16, 14] also hold for deterministic matrices satisfying approximate orthogonality conditions. Therefore, we expect the one-dimensional CLT to approximately hold for a much larger class of feature maps. Finally, we also note that the above extension of the one-dimensional CLT to mixture of gaussians also provides a proof for the asymptotic error for random features in [11].
Conclusions - We demonstrate the universality of the Gaussian mixture assumption in high-dimension for various machine learning tasks such as empirical risk minimization, sampling and ensembling, in a variety of settings including random features or GAN generated data. We also show that universality holds for a large class of functions, and provide a weak convergence theorem. These results, we believe, vindicate the classical theoretical line of works on the Gaussian mixture design. We hope that our results will stimulate further research in this area. We also believe it crucial to understand the limitations of our extended universality framework, for instance in the cases of data with low-dimensional structure or sparsity.
We acknowledge funding from the ERC under the European Union's Horizon 2020 Research and Innovation Program Grant Agreement 714608-SMiLe, as well as by the Swiss National Science Foundation grant SNFS OperaGOST, 200021_200390.
[64] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[65] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. arXiv: 1606.03657, 2016.
[66] Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antonio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272, 2020.