Toward Learning Distributions of Distributions

Venue: NLDL_org_2025_Conference
Published: 2024-09-08 14:02:56

Abstract: We propose a novel generative deep learning architecture based on generative moment matching networks. The objective of our model is to learn a distribution over distributions and generate new sample distributions following the distribution of the training data. We derive a custom loss function for our model based on the maximum mean discrepancy test. We evaluate our model on different data sets and investigate the influence of different hyperparameters on the performance of the model.

Main paper: 
 introduction: 
 Neural networks have been shown to be universal function approximators in the limit of inifinte layer width [1] or depth [2] . Particularly generative neural networks make use of the full power of neural networks by transforming a simple prior distribution, which allows for efficient sampling, into an arbitrarily complex data generation distribution. Within the domain of deep generational models, different concepts were developed over the years such as variational autoencoders (VAE) Kingma and Welling [3] .They belong to the family of variational Bayesian methods and can generate new instances by sampling from a distribution over the latent space of the autoencoder by making use of the reparameterisation trick. Alternatively, flow-based generative models, capture complex distributions by transforming simpler ones using bijections and leveraging the Change of Variables Theorem [4] . Especially in the context of image generation, diffusion models [5] have become prominent. Borrowing concepts from the area of nonequilibrium thermodynamics, they learn to denoise images so that, beginning with noiseonly, the image is iteratively denoised until a clear image is obtained.In this paper, however, we focus on Generative Adversarial Networks (GANs) [6] . These networks consist of a generator and a discriminator network.The generator uses the beforementioned statistical transformations to generate new data. The discriminator then acts as a classifier and aims to separate original training from generated data. The networks are trained together in a zero-sum game, in which the generator tries to "fool" the discriminator into incorrectly classifying generated data points as true ones, and the discriminator tries to correctly classify generated and true samples.A interesting model related to GANs is the gen-049 erative moment matching network (GMMN) [7, 8] . 050 While the generator remains similar or even un-051 changed to that of regular GANs, the discriminator 052 is replaced by maximum mean discrepancy (MMD) 053 [9] to measure distances between distributions p and 054 q, given by 055MMD[F, p, q] := sup f ∈F (E x∼p [f (x)] -E y∼q [f (y)]).MMD, unlike other methods, such as the Kullback-057 Leibler divergence, does not rely on density estimates 058 of the distributions, but (implicitly) embeds them 059 into a reproducing kernel Hilbert space in which 060 the distance is determined. This renders MMD 061 particularly useful in settings dealing with sets of 062 (generated) instances.When the function space H is a universal RKHS 064 [10] on a compact domain X of distributions p, q, 065 MMD fulfils the properties of a metric [9] , that is 066 MMD[F, p, q] = 0 ⇔ p = q.(1) 067However, to devise a generative model that samples 068 distributions based on a custom MMD loss while 069 preserving the convergence guarantees as portrayed 070 in Dziugaite et al. [7] and Li et al. [8] , we need 071 to devise a universal kernel that acts on the set of 072 probability measures.In this paper, we develop a new model based on 074 the GMMN architecture, capable of generating not 075 only distributions of vectors, but distributions of dis-076 tributions. To achieve this goal, we devise an MMD 077 between distributions of distributions and develop a 078 new generator architecture based on Hypernetworks 079 [11] . The proposed architecture allows for a fully im-080 plicit learning procedure, where the structures of the 081 distributions are parameterized by neural networks, 082 and the loss function is able to compare arbitrary 083 statistical moments of the set of distributions.The remainder is structured as follows. We 085 present our main contribution, the derivation of 086 our GMMN for distributional data, in Section 2 and 087 report on empirical results in Section 3. Section 4 088 concludes.  
preliminaries: 
 We are concerned with the problem of learning to generate new samples from a distribution over a set of datasets. Given a sample set of datasets, our goal is to construct a model that is able to approximate the underlying sampling distribution, by minimizing a discrepancy between the distribution parameterized by the model and the distribution underlying the data. To achieve this, we apply the concept of maximum mean discrepancy to our setting. Since the entities of interest are empirical distributions represented by different data sets, we use universal kernels [10] on the set of probability measures .Christmann and Steinwart [12] introduce universal kernels on non standard input spaces. They show that for a compact metric space X and a separable Hilbert space H with an injective map ρ : X → H, one can derive a universal Gaussian-type RBF-kernelk σ with k σ : X × X → R (x, x ′ ) → exp(-σ 2 ∥ρ(x) -ρ(x ′ )∥ 2 H ), σ > 0Based on this result, the authors further derive a kernel between probability distributions, based on the Fourier transform.EQUATIONWhere the distributions P and P ′ are Fourier transforms of elements of the set X := M 1 (Ω) of probability measures on a compact set Ω ⊂ R d , and µ is a finite Borel measure on R d with support(µ) = R d .Based on this similarity measure between distributions, we will derive an MMD between distributions of distributions, which is then used as the loss function of a generative model.To solve the problem described above, we define a training objective in form of an MMD loss L(y, ŷ), a function of the generated samples and the training data. Our approach to generate the samples is to transform a prior distribution from which we can easily sample new data points into the distribution over sample sets q by using a deep neural network as the transformation function G θ . Our generation process consists of two stages. First, we need to sample a representation of the distribution The probability density of the zero-centered mul-167 tivariate Gaussian distribution with the diagonal 168 identity covariance matrix Σ = λ1 is given byQ l forEQUATIONEQUATION) 170Choosing this distribution as the Borel measure 171 µ for our L 2 function space, we get the following 172 approximation of our kernel k σ (P, P ′ ) based on the 173 empirical distributions P and P ′ , where P ′ is defined 174 analogous to P as We can now use this result to replace the integral expression in Equation ( 6) to derive a closed form solution for the empirical Fourier transformation based kernel k σ (P, P ′ ) between sample distributions given byP ′ (z) = 1 m m k=1 δ(y ′ k -z): 175 k σ (P, P ′ ) = exp -σ 2 ∥ P -P ′ ∥ 2 L2(µ) ≈ 176 exp   - σ 2 (2πλ) d/2   1 n 2 n i,j=1 R dEQUATIONWith Q i ∼ q, P i ∼ p the MMD is now given byEQUATION) - 2 M N M,N i,j=1 k σ (P i , Q i ) + 1 N 2 -N N i̸ =j k σ (Q i , Q j ).The generative model consists of two major components. The first network, generating the weights for the second network, will be called Hypernetwork, in accordance with Ha et al. [11] . Accordingly, the second network is called the main network, since it performs the main task, generating samples from the different distributions, which are parmeterized by its weights that were sampled by the Hypernetwork, see Figure 1 .By using the MMD as the loss function for our model, we build upon the ideas of the GMMN proposed by Li et al. [8] and Dziugaite et al. [7] . Instead of comparing sets of data points represented by vectors, we directly compare sets of data sets, using the custom distribution MMD defined in Equation (9) .We assume that the distributions underlying different data sets can have a shared structure that can directly be learned by the main network using layers with weights shared between all main networks. The corresponding parameters ϕ can be directly learned using gradient descent and backpropagation. Algorithm 1 shows the general structure of the Hyper-GMMN training process. 
family of univariate gaussian distributions: 
 Starting from a simple and easily interpretable case, our goal is to learn a distribution over the family of univariate Gaussian distributions using our proposed We generate data comprising 20 sample distributions of 20 samples each using the parameters µ µ = 10, σ µ = 2.5, µ σ = 1, σ σ = .5. We use the MMD loss defined in Equation ( 10) and train the model using back propagation and stochastic gradient descent for 10000 epochs. In each epoch, we generate a dataset of the same size as the training data. In Figure 2 , the evolution of the Hypernetworks' parameters are plotted over the training process. These parameters correspond to the statistical moments of the parameters of the main network, and can be interpreted as the moments of the distribution over the sample distributions. The plots suggest a stable convergence to values close to the model parameters used in the data generation process. It can be seen that the bias w.r.t. the true parameters is smaller for lower order statistical moments.+ for k ← 1 to m do w k ∼ H θ m k ← Determine m k (k), m k ∈ N + for s k ← 1 to m k do x s k k ∼ q * , s k ∈ {1, 2, . . . , m k } ŷs k k ← M ϕ,w k (x s k k ) end for end for L ← DistributionMMD(ŷ, y) θ ← θ -∇ θ L ϕ ← ϕ -∇ ϕ LData The second experiment is based on a dataset consisting of sets of data points uniformly distributed over a rectangular region of R 2 . The training data is generated by independently sampling the coordinates of vectorsz i ∈ R 2 from one dimensional uniform distributions z 0 i ∼ U[0.2, 1], z 1 i ∼ U[0.1, 0.1], and then multiplying each vector of the respective data set by a data set specific rotation matrix A(γ) = cos(γ) -sin(γ) sin(γ) cos(γ) .The data set specific parameter γ, the angle of the rotation, is sampled from a uniform distribution over [0, 2π We presented a promising proof of concept to learn-342 ing distributions over distributions. Our approach 343 relies on a Hypernetwork that parameterizes a set 344 of main networks that finally generate the desired 345 data. All networks are trained simultaneously and 346 the overall training loss is given by a tailored MMD-347 based loss function. Preliminary empirical results 348 on small-scale toy data showed that the underlying 349 concepts can indeed be learned, although stability 350 of the training process is still an issue.Note that Conditional GANs [14] may offer an 352 alternative to our approach. Conditional GANs are 353 generative models that learn to generate distribu-354 tions over a random variable x dependent on an 355 input y by transforming a simple prior distribution 356 into the desired conditional distribution G(x|y). Our 357 network also learns to parameterise distributions by 358 generator functions but, in contrast to conditional 359 GANs, our network does not receive the dependency 360 y as an input. Instead, our approach can be thought 361 of as learning a probability distribution P (y) jointly 362 with G(x|y).We leave this investigation to future work, as well 364 as studying the compactness of the generated space 365 of distributions or conducting experiments with real-366 world data on larger scales. 
 
 Appendix:  


Decision: This paper considers estimating distributions over data objects that are, themselves, distributions. This is an important problem that e.g. lies at the heart of information geometry, and has also been studied in various applications before, e.g. covering:
- Distributions of normal distributions, which have appeared in diffusion tensors in diffusion MRI (describing estimated probability distributions over white matter fiber orientations), covariance descriptors in computer vision (describing local feature distributions) and more
- Distributions of Gaussian processes describing e.g. uncertain estimates of white matter bundle trajectories or uncertain estimates of yearly temperature curves

In the last bullet, the interesting uncertainty is aleatoric, or irreducible -- and these examples are therefore special cases of the problem of incorporating irreducible data uncertainty into the downstream analysis. Distributions over the aleatoric distributions could be a useful tool in this regard.

The reviewers largely appreciate the paper as an interesting and clear proof of concept study, which is well suited to a conference. The most important highlighted concern being the relevance of the problem. The authors are encouraged to stress the motivation for the problem in their final study, as well as incorporate any concerns, questions and useful suggestions made by the reviewers.

Reviews: review: title: title: Promising Architecture for Generating Distributions of Distributions, but Limited Experimental Validation and Contextualization
paper_summary: paper_summary: This paper proposes a novel generative model based on the generative moment matching network (GMMN) architecture, designed to generate distributions over distributions. The framework employs a Hypernetwork to generate model parameters, with each parameter set corresponding to a sampled distribution. The paper derives a loss function based on the Maximum Mean Discrepancy (MMD) test, enabling the networks to learn by comparing sets of datasets. The model’s effectiveness is illustrated through experiments on two small-scale toy datasets.
main_review: main_review: The paper appears incomplete in its current form:
* The study only includes toy examples as "proof of concept" demonstrations.
* There is limited exploration of the effects of hyperparameters on training stability and generalization to the test set, leaving the experimental results unconvincing in demonstrating the proposed method's effectiveness in learning distributions of distributions.
* The absence of baseline models limits the contextual understanding of the proposed method's performance.
* The lack of real-world applications weakens the motivation for the method and its potential impact.

Although the authors have responded to these concerns, I intend to maintain my original rating based on the paper's present state.
strength_weakness: strength_weakness: * The experiment section lacks robust evidence to convincingly demonstrate the proposed method’s performance:
    * The paper only includes experiments on two toy datasets, without any real-world datasets, which limits the generalizability and practical relevance of the results.
    * In Section 3.2, only one out of 42 tested hyperparameter pairs produces a distribution of distributions similar to the training set, calling into question the method’s robustness.
* The paper does not adequately situate the proposed method within the broader context of existing methods that model distributions over distributions. It is unclear where the method’s innovations lie, and the paper lacks a performance comparison with relevant baselines, such as the Dirichlet process.
* The paper does not sufficiently motivate the importance of modeling distributions over distributions. The transition in Lines 68-73 feels abrupt, and the lack of real-world examples or applications makes it difficult to appreciate the method’s broader relevance.
* Some minor issues:
    * Line 222, $(P_i, Q_j)$ instead of $(P_i, Q_i)$
    * Line 243: it’s unclear which network the parameter $\\phi$ belongs to, since we have $\\theta$ for the Hypernetwork, and $w_k$’s for the main networks.
    * Line 258: $\\mu_{\\sigma}$ for the mean of $\\log(\\sigma)$