Kernelised Normalising Flows
Authors: Eshant English
Matthias Kirchler
Christoph Lippert
Venue: ICLR 2024 poster (2024)
Keywords: Machine Learning
Maximum Likelihood
Density Estimation
Statistics
Kernels
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: Replacing neural network with kernelised transforms can improve normalising flow likelihood fit, especially in low-data regime

Abstract: Normalising Flows are non-parametric statistical models known for their dual capabilities of density estimation and generation. They are distinguished by their inherently invertible architecture. However, the requirement of invertibility imposes constraints on their expressiveness, necessitating a large number of parameters and innovative architectural designs to achieve satisfactory outcomes. Whilst flow-based models predominantly rely on neural-network-based transformations for expressive designs, alternative transformation methods have received limited attention. In this work, we present Ferumal flow, a novel kernelised normalising flow paradigm that integrates kernels into the framework. Our results demonstrate that a kernelised flow can yield competitive or superior results compared to neural network-based flows whilst maintaining parameter efficiency. Kernelised flows excel especially in the low-data regime, enabling flexible non-parametric density estimation in applications with sparse data availability.

Published as a conference paper at ICLR 2024
KERNELISED NORMALISING FLOWS
Eshant English" Matthias Kirchler* 1.2
Christoph Lippert
[fir (first): (last) @hpie............ .de
Hasso Plattner Institute for Digital Engincering, Germany
Unniversity Satersllauern-Landaae Germany
Hasso Plattner Institute for Digital Health the Icahn School of Medicine Mount Sinai, NYC, USA
ABSTRACT
Normalising Flows are non-parametric statistical models characterised by their dual capabilities of density estimation and generation. This duality requires an inherently invertible architecture. However, the requirement of invertibility imposes constraints on their expressiveness, necessitating a large number of parameters and innovative architectural designs achieve good results. Whilst flow-based models predominantly rely on neural-network-bassee transformations for expressive designs, alternative transformation methods have received limited attention. In this work, we present Ferumal flow, a novel kernelised normalising flow paradigm that integrates kernels into the framework. Our results demonstrate that a kernelised flow can yield competitive or superior results compared to neural network-basee flows whilst maintaining parameter efficiency. Kernelised flows excel especially in the low-data regime, enabling flexible non-parametric density estimation in applications with sparse data availability.
INTRODUCTION
Maximum likelihood fundamental approach to parameter estimation in the field of machine learning and statistics. However, its direct application to deep generative modelling is rare due to the intractability of the likelihood function. Popular probabilistic generative models such as Diffusion Models (Dai & Seljak 2020b) and Variational Autoencoders (Kingma & Welling 2022) instead resort to optimising the Evidence Lower Bound (ELBO), a lower bound on the log- likelli,,,,od, due to the challenges in evaluating likelihoods.
The change of variables theorem offers a neat and elegant solution to compute the exact likelihood for deep generative modelling. These models, known normalising flows, employ invertible architectures to transform complex probability distributions into simpler ones. Normalising flows (Papamakarios et al. 1 Kobyzev al. (021) excel in efficient density estimation and exact sampling, making them suitable for various applications.
Whilst flow-based models possess appealing properties rooted in invertibility, they also impose limitations on modelling choices, which can restrict their expressiveness. This limitation can be mitigated by employing deeper models with a higher number of parameters. For instance, the Glow model Kingma& & Dhariwal (2018) utilised approximately 45 million parameters for image generation on CIFAR-IO Krizhevskk (2009a), whereas StyleGAN3 Karras et al.] (2019), a method that doesn't use likelihood optimisation, achieved a superior FID score with only about a million parameters.
The issue of ppereearamttriiitiion in flow-based models hinders their effectiveness in domains with limited data, such as medical applications. For example, normalising flows can be used to model complex phenotypic or genotypic data in genetic association studies (Hansen al.) 2022 Kirchler et al. 2022); collection of high-quality data in these settings is expensive, with many studies only including data on a few hundred to a few thousand instances. In scenarios with low data availability, a easily memorise the entire dataset, leading to unsatisfactory performance
flow-bard based network on the test set. While existing research has focused on enhancing the expressiveness of flows through clever architectural techniques, the challenge of achieving parameter efficiency has mostly been overlooked with few exceptions Lee et al. (2020). Most normalising flows developed to date rely on
* "qqual contribution

---

Published as a conference paper at ICLR 2024
neural networks to transform complex distributions into simpler distributions. However, there is no inherent requirement for flows to use neural networks. Due to their over- weer-prramttriiattion and low inductive bias, neural networks tend to struggle with generalisation in the low-data regime, making them inapplicable to many real-world applications.
In this work, propose novel approach to flow-based distribution modelling by replacing neural networks with kemels. Kernel machines work well in low-data regimes and retain expressiveness even scale. We introduce Ferumal flows, kernelised normalising flow paradigm that outperforms or achieves competitive performance in density estimation for tabular data compared to other efficiently invertible neural network baselines like RealNVP and Glow. Ferumal flows require up 93% fewer parameters than their neural network- counterparts whilst still matching or outperforming them terms of likelihood estimation. We also investigate efficient training strategies for larger-scale datasets and show that kernelising the flows works especially well on small datasets.
2 BACKGROUND
2.1 MAXIMUM LIKELIHOOD OPTIMISATION WITH NORMALISING FLOWS
normalising flow an invertible neural network, f IR + RD that maps real data x onto noise variables The noise variable is commonly modelled by simple distribution with explicitly known density, such as normal or uniform distribution, while the distribution needs to be estimated. As normalising flows maximum likelihood estimators, for given data set of instances L1,, In. we want to maximise the log-likeld over model parameters,
is unknown and
max
log (x:))
With the change of variables formula,
px (x) = pz (f(z)) det (x))],
where (x) is the Jacobian of f in æ, we can perform this optimisation directly:
max
log (pz (f (zx))) + log (Idet Jy (z:)1).
The first part, log(pz (fx))) can be computed in closed form due to the choice of pz. The second part. log() det (z;))) (z;))), can only be computed efficiently if f is designed appropriately.
2.2 COUPLING LAYERS
One standard way to design such an invertible neural network f with tractable Jacobian is affine coupling layers (Dinh et al. 2017). A coupling layer Ce RD - RD is invertible layer that maps an input yr-1 to an output ze (E is the layer index): first, permute data dimensions with fixed permutation we, and split the output into the first d and second D d dimensions:
[ ni] (eeeve- ne(Ue- ne(Uk-1)a+1:Dl-
Commonly, the permutation either a reversal of dimensions or picked uniformly at random. Next, apply neural networks, 8t, le Rdd RD-d to the first output, ut, and use it to scale and translate the second part, uz:
D exp (st (ut) + u + le (u²)
The first part of ye remains unchanged, i.e., y up. and we get the output of the coupling layer as:
[v], vi] [ug. exp (se (ut)) @ uz te (ut)] Vf C2 (ye-1)
The Jacobian matrix of this transformation is permutation of a lower triangular matrix resulting from up undergoing an identity transformation and ui getting transformed elementwise by a function of up. The Jacobian of the permutation has determinant with absolute value of 1 by default.
2

---

Published as a conference paper at ICLR 2024
The diagonal of the remaining Jacobian consists of d elements equal to unity and the other D d elements equal the scaling vector St (u²). Thus, the determinant can be efficiently computed as the product of the elements of the scaling vector sy (uz).
The coupling layers are also efficiently invertible as only some of the dimensions are transformed and the unchanged dimensions can be used to obtain the scaling and translation factors used for the forward transformation reverse the operation.
Multiple coupling layers can be linked in a chain of L layers such that all dimensions can be transformed:
f(x) = Cr ... ......... Ci(x),
i.e. M/o = x and VL- = f(x).
2.3 KERNEL MACHINES
Kernel machines (Schölkopf et al. 2002) implicitly map data vector E RP into high-dimensional (potentially
reproducing kernel Hilbert space (RKHS), H. by means of fixed feature map RD + H. The RKHS 7. has an associated positive definite kernel k(u, e) (ó(u), (())) ,. where (5. H x - R is the inner product of H. The kernel can oftentimes be computed in closed form without requiring the explicit mapping of u, into H. making computation of many otherwise intractable problems feasible. In particular, as many linear learning algorithms, such as Ridge Regression Support Vector Machines, only require explicit computations of norms and inner products, these algorithms can be efficiently kernelised. Instead of solving the original learning problem in RP, kernel machines map data into the RKHS and solve the problem with a linear algorithm in the RKHS, offering both computational efficiency (due to linearity and kernelisation) and expressivity (due the nonlinearity of the feature map and the high dimensionality of the RKHS).
FERUMAL FLOWS: KERNELISATION OF FLOW-BASED ARCHITECTURES
In this section, we extend standard coupling layers to use kernel-based scaling and translation functions instead. Whilst neural networks are known to perform well in large-data regimes or when transfer-leariing from larger datasets can be applied, kernel machines perform well even at small sample sizes and naturally trade-off model complexity against dataset size without losing expressivity.
3.1 KERNELISED COUPLING LAYERS
We keep the definition of coupling layers in Section 2, and only replace the functions sæ and te by functions mapping to and from an RKHS H. We have to deal with two main differences to the kernelisation of many other learning algorithms: firstly, the explicit likelihood optimisation does not include a regularisation term that penalises the norm of the prediction function. And secondly, instead of single mapping from origin space RKHS to iigle-dimensional output, we aim to combine multiple layers, in each of which the scaling and translation map from origin space to RKHS to miiti--dmensionall output. As result, the optimisation problem will be convex in contrast to standard kernel learning, and we have to derive kernelised and tractable representation of the learning objective.
In particular, in layer L, we introduce RKHS elements V E D-d and define scaling and translation as
se (w)) (Vi5, o (w))
D-d j=1
DDD and
te (u+) = [V 2 (ut))
D-d
E RD-
H j=1
We summarise V2 = [V V/l and V VV...., Vz) for the full flow f(x) Ct o...o (xr). Since elements V E H2 and H's dimensionality is potentially infinite, we cannot directly optimise the objective:
max Σ Pz (Cz o.....oo (zx)) log (Jdet Jc.....o, (2+)) L(V).
However, we can state a version of the representer theorem (Schölkopf et al. 2001) that allows us to kernelise the objective:
3

---

Published as a conference paper at ICLR 2024
Proposition 3.1. Given the objective L in equation 1 for any V Vi.....VV, E H1' there also exists a V with L(V) = L (V") such that
Jorany
Ve E k (. Ae,
for some Ae,: = AA2 (R((D-)) Here, uh. 0e.. C2(i))) i.e, the first
purt of the permulated input to layer E for data point i. In particular, if there exists solution V' E arg maxy L(V), then there's 's also a solution V of the form
V = Σ (3 Ae.i-
Proof. Let V H2 and 4, span spanfo(ul,,) ( (()(,) be the space spanned by the feature maps of layer inputs, and let of denote its orthogonal complement in H. We can then represent each element V2 E EH (jE (1...... D d)) as an orthogonal sum of an element of 4 and $
ViJy' = -r.j + oi.. with drs = Σ A( and (Øt o(t = 0 vi I...., i=1
for some values 4i., E R. In the objective ① we only use V2j to compute (Vi.) o(u),)) (), as part of the computation of se(ut,). But due to orthogonality, holds that
(V2y', ((,,))) = (des + oi,5 b(uk,)) = (dess (u,,,))) 2) + o(t,)) (dr.s (ve,)))
Hence, replacing V2, by Φesj Ei= of, property of the RKHS H. now states that (Øl (,,,,,, o(-)) k(
(ut,)) keeps the objective unchanged. The reproducing
Repeating this for all é = 1, ... L, all j 1, ... D d and also for translations tę yields version of V that can be represented as a linear combination of the stated form.
In contrast to the classical representer theorem, the objective doesn't contain a regulariser of the model's 'ss norm, which would ensure that any solution can necessarily be represented as a linear combination of kernel evaluations. However, if a solution exists, Proposition Jennese that there also exists solution that can be expressed linear combination of kernel evaluations. Therefore, we can re-insert this solution 4 into the objective to get kernelised objective.
For layer and arbitrary E Rd
D-d
se(a) = [VV (a))
Σ
Aiish (ub. a)
Σ (ut,e) Ař,
=
jul
As in the objective D 88 gets only evaluated in points
uubyli l,..., n } this simplifies to (m A2.s ARK U!)
(ubm) Σ de (Mix
1=1
where K (U!.U)) N: (ub.r wem
)T Timm is the kernel matrix at layer and A
A Ai......... Aim] R(D-d)xn xm is the weight matrix. A similar derivation holds for tr.
In total, we can kernelise the objective ① and optimise over parameters A E [RXxxx(DD4) instead of over V (2)( In contrast to neural network- based learning, the number of parameters, 2D - dd. fixed except for the number of layers (since is usually set to [D/2]), but increases linearly with the dataset size, n. This makes kernelised flows especially promising for learning
4

---

Published as a conference paper at ICLR 2024
in the low-data regime, their model complexity naturally scales with dataset size and does not over-paramen parammeteeee as much as neural networks (as long as one does not employ an excessive number of layers).
Since the resulting objective function is not convex, optimisers targeted to standard kernel machines such Sequential Minimal Optimisation (Platt 1998) are not applicable. Instead, optimise (T) with variations of stochastic gradient descent (Kingma & Ba 2014).
3.2 EFFICIENT LEARNING WITH AUXILIARY POINTS
The basic kernelised formulation can be very computationally expensive even at moderate dataset sizes and can tend towards overfitting the lower-data regime. Gaussian Process (GP) regression, this problem is usually addressed via sparse GPs and the introduction of inducing variables (Quinonero- Candela & Rasmussen 2005). a similar spirit, we introduce auxiliary points. In layer t, instead of computing the kernel with respect to the full data u "'. u take N < new variables W I......... w3,N] E RdxN and compute the scaling transform as
Se(ut,m) = AÇK(U) W))m
with Ag E R(D-d)xN (analogously for &2). We make these auxiliary points learnable and initialise them to a randomly selected subset of u², : This procedure reduces the learnable parameters from 2n(D - L (for both sy and tạ) to 2NDL.
In another variation make these auxiliary points shared between layers. In particular, instead of selecting times points wi.,:..., Wh,N. we instead only select W [wi,..., wxl E Rdx once and compute at layer E
(e Á2K(U?, W
This further reduces the learnable parameters to 2N(D - d)L + 2dN.
4 RELATED WORKS
We are unaware of any prior work that attempts to replace neural networks with kernels in flow-based architectures directly. However, there is family of flow models based on Iterative Gaussianisation (IG) Chen & Gopinath| (000) that utilise kernels. Notable works using Iterative Gaussianisation include Gaussianisation Flows Meng et al. (2020), Rotation- -Baed Iterative Gaussianisation (RBIG)
Laparra al. (201T), and Sliced Iterative Normalising Flows & Seljak (2020a). These IG-based methods differ significantly from our methodology. They rely on kernel density estimation and inversion of the cumulative distribution function for each dimension individually and incorporate the dependence between input dimensions through rotation matrix, which aims to reduce inter- dependence. In contrast, our method integrates kernels into coupling layer- based architectures. Furthermore, IG-based methods typically involve large number of layers, resulting in inefficiency during training and a comparable number of parameters to neural network-based flow architectures. In contrast, the Ferumal flow approach of incorporating kernels can act as a drop-in replacement in many standard flow-based architectures, ensuring parameter efficiency without compromising effectiveness. Another generative model using kernels is the work kernel transport operators (Huang et al. (022)) demonstrated promising results in low-data scenarios and favourable empirical outcomes. However, their approach differs from ours as they employed kernel mean embeddings and transfer operators, along with a pre-trained autoencoder.
deep learning context are deep Gaussian processes Damianou & Lawrence (2013) and deep kernel learning (Wilson et al., 2016 Wenliang et al., 2019). Deep GPs concatenate multiple layers of kernelised GP operations; however, they are Bayesian, non- invertible models for prediction tasks instead of density estimation and involve high computational complexity due to operations that require inverting a kerne matrix. Some works Rudi & Ciliberto (2021) Marteau-Feree
Other works focusing on kernel machines
al. [2021); : Tsuchida et al. (2023) use kernels for nonnegative functions for modelling densities but are also not invertible, Deep kernel learning, on the other hand, designs new kernels that are parametrised by multilayer perceptrons.
5

---

Published as a conference paper at ICLR 2024
Maroñas al. (2021) integrated normalising flows within Gaussian processes. Their approach differs significantly from ours as they aimed to exploit the invertibility property of flows by applying them to the prior or the likelihood. Their combined models consist of kernels in the form of GPs but also involve neural networks in the normalising flows network, resembling more of a hybrid model.
NanoFlow Lee al. (2020) also targets parameter efficiency in normalising flows. They rely parameter sharing across different layers, whereas we utilise kernels. We also attempted to implement the naive parameter-sharing technique suggested by Lee et al. (2020), but found no improvement in performance.
EXPERIMENTS
We assess the performance of Ferumal flow kernelisation both on synthetic 2D toy datasets and on five real-world benchmark datasets sourced from Dua & Graff (0117). The benchmark datasets include Power, Gas, Hepmass, MiniBoone, and BSDS300. To ensure consistency, we adhere the preprocessing procedure outlined by Papamakarios et al. (2018).
Implementation details We kernelised the RealNVP and Glow architectures. For comparison purposes, RealNVP and Glow act as direct comparisons being the neural-net counterparts of our models. We have also included basic autoregressive methods, Masked Autoregressive Flows (Papa. makarios et al. 2018), and Masked Autoregressive Distribution Estimation (Germain et al. 2013).
FFJORD (Grathwohl et al. 2018) continuous normalising flow, and Gaussianisation Flows
(Meng et al. (020). Kotation-based Iterative Gaussianisation (Laparra et al. (011). Sliced Iterative Normalising Flows (Dai & Seljak] 2020b) iterative gaussianisation methods for evaluations.
Most autoregressive flow models outperform -on-autoregressive flow models. However, they usually
come with the trade-off of either inefficient sampling or inefficient density estimation, i.c., either the forward or the inverse computation is computationally very expensive.
Training details Our Ferumal Flow ker- nelisation has a negligible number of hyper- parameters. Apart from learning rate hyper- parameters (i.e., learning rate, 31, $2 for Adam) and the number of layers, that central to both kernelised and neural-net- based flows, we only need to choose a ker- nel with its corresponding hyperparameters (and a number of auxiliary points for large- scale experiments). This is in contrast with neural-net-based flows where choices for flow layer include a number of hidden sub- layers, the number of nodes in each sub-layer, residual connections, type of normalisation, activation function, dropout, and many more. Coupled with longer convergence times this necessitates considerably more time and re- sources in hyperparameter tuning than our proposed kernel methods. In our study, we utilised either the Squared Exponential kernel or Matern Kernels exclusively for all exper- iments. We learnt all the kernel hyperparam- eters using the GPyTorch library for Python for the main experiments. Throughout the experiments, we used the Adam (Kingma & Ba (2114) optimiser, whilst adjusting the 31 and 32 parameters of the optimiser. Additionally, steps (StepLR) or with cosine annealing. In all the experiments, we incorporated auxiliary points as observed that they provided better results. In most cases, persisted with 150 auxiliary points.
Figure 1: Histogram of 2D toy datasets. Left: True distribution. Middle: NN-based. Right: FF- kernelisation
decayed the learning rate either with predefined

---

Published as a conference paper at ICLR 2024
We coded method in PyTorch (Paszke et al. 2019) and used existing implementations for the other algorithms. We ran all experiments for Ferumal flows and other baselines on CPUs (Intel Xeon 3.7 GHz). For more comprehensive training details, please refer to the Table
5.1 2D TOY DATASETS
Initially, we conducted density estimation experiments on three synthetic datasets that were sampled from two-dimensional distri- butions exhibiting diverse shapes and num- bers of modes. Figure[] showcases the orig- inal data distribution alongside the samples generated using the Ferumal flow kernelisa- tion and the corresponding neural network counterpart. The neural-net- best- archi- tecture clearly shows the distortion of den- sity in many regions whereas the kernelised counterpart has much better modelling. Table shows the corresponding log-likelihood in nats, quantitatively showing the enhancement from our kernelisation. The results demonstrate that Ferumal flow kernelisation can outperform its neural counterpart on these toy datasets. All the toy datasets were trained with a batch size of 200 and for 10K iterations. We also investigated the effect of our kernelisation on highly discontinuous densities strengthening our argument for kernelisation. Please refer to Appendix B
Table I: Results on toy datascts. Log Likelihood in nats, higher is better
Dataset
Ours (#params)
NN-based (#params)
Line
3.75 (5K)
3.15 (44K)
Pinwheel Moons
-2.44 (4K)
-2.48 (44K)
-2.43 (5K)
-2.54 (44K)
5.2 REAL-WORLD DATASETS
We conducted density estimation experiments five real-world tabular benchmark datasets (descrip- tion can be found in Appendix[[] I employing the preprocessing method proposed by PPapamakarios kernelised two flow architectures, i.e., RealNVP and Glow, that utilise the coupling layer for efficient sampling and inference and making direct comparisons with them. Additionally, we also considered comparisons with GF (Gaussianisation Flows) (Meng et al.) 2020), RBIG (Rotation-based Iterative Gaussianisation) (Laparra et al. 2011), GIS (Gaussianied Itera- tive Slicing/Sliced Iterative Normalising Flows) (Dai 2020a)[2000b), MAF (Masked Autoregressive Flows) (Papamakarios et al. 2018), and MADE (Masked Autoregressive Distribution Estimation) (Germain et al. 2015). FIJORD (Grathwohl et al. 2018), architectures that do not use coupling methods, as they have significantly higher computational costs. In particular, Gaussianisation-baa methods, require many more layers (up to 100, in our settings), whilst autoregressive flows are slow to sample from, due to their autoregressive nature. In contrast to Gaussianisation-
et al. (018). In experiments,
2020b).
layers. These methods are not directly comparable to the
methods, our kernelisation of coupling layers does not increase the computational complexity and the training time under fixed number of epochs is similar to neural-net bes--eedd coupling layers. Run-time comparisons under a fixed number of epochs are provided in Table I in Appendix
Table presents the results of our experiments, revealing that Ferumal flow kernelisation consistently achieves better competitive outcomes across all five datasets. Despite its straightforward coupling layer architecture, our approach surpasses RBIG, GIS, and MADE on all the datasets and achieves competitive performance to the much more expensive MAF, GF, and FFJORD methods, underscoring the efficacy of integrating kernels. Please refer to Table [2 for error bars on coupling and non-coupling experiments.
5.3 INITIAL PERFORMANCE
Figure presents the learning curves the train and test loss for our Ferumal flow kernelisation and the two neural-network counterparts. These findings demonstrate that the crrumal--flo-basee architecture exhibits faster convergence compared to the neural network baselines. This may be due to the parameter efficiency provided by our kernelisation or due to the stronger inductive biases provided by kernel machines. Throughout our experiments, we maintained default settings and ensured consistent batch sizes across all models.
7

---

Published as a conference paper at ICLR 2024
Table Log-likelihood measured in nats. Larger values better. Methods prepended with FF our kernelised versions and the results with are taken from existing literature.
Datasets
Method
Power
Gas
Hepmass
Miniboone
BSDS300
Coupling
RealNVP
0.17
8.33
-18.7
-13.55
153.28
FF-RealNVP (ours)
0.24
9.55
-18.20
-11.19
154.30
Glow
0.17
8.15
-1892
-11.35
155.07
FF-Glow (ours)
0.35
10.75
-17.11
-10.76
154.71
Non-coupling
MADE
-3.08
3.56
-0.988
-1.599
148.85
MAF
0.24
10.08
-17.70
-11.75
155.69
FFJORD
0.46
8.59
-1.92
-10.43
157.40
Gaussianisation methods
GF
0.57
10.08
-17.59
-10.32
152.82
GIS
0.32
10.30
-19.00
-14.26
157.31
RBIG
-1.02
-0.05
-24.59
-25.41
115.96
5.4 LOW-DATA REGIME
In certain applications, such as medical settings, data availability often limited. Neural- network-based flows typically suffer from prraatteeetii,,,,nnnn, leading challenges in gen- cralisation within low-data regimes. To assess the generalisation capability our model un- der such conditions, we trained model using only 500 examples and evaluated its per- formance on the same benchmark datasets. To address the challenges of limited data, we opted to tie the learned auxiliary variables across layers in this setting. This approach helped mitigate parameter complexity whilst maintaining the benefits of utilising auxiliary points. As highlighted by Meng al. (2020), Glow and RealNVP struggled to gen- eralise in low-data regimes, evidenced by increasing validation and test losses
Table 3: Results on small subset of 500 examples. LL in nats, higher the better
whilst the training losses decreased. To provide a stronger benchmark, we in- cluded the FFJORD model (Grathwohl al. 2018). FFJORD is continu- ous normalising flow method with a full-form Jacobian and exhibits supe- rior performance to Glow or RealNVP in density estimation and generation tasks. For our model, we used ker- nelised version of RealNVP which is notably weaker than the Glow version. This also proves that kernelisation can make flow-based models more data-efficient.
Dataset
Ours (#params)
FFJORD (#params)
Miniboone
-27.75 (58K)
-39.92 (821K)
Hepmass
-27.90 (41K)
-28.17 (197K)
Gas
0.22 (11K)
-7.50 (279K)
Power
-2.91 (8K)
-11.33 (43K)
BSDS300
121.22 (85K)
100.32 (3,127K)
Table presents the results, demonstrating that our method achieves superior generalisation. This may be attributed to the significantly lower number of parameters required compared to the continuous FFJORD method.
5.5 PARAMETER EFFICIENCY
Table Ishows the parameter counts Ferumal flows against the baseline methods. Kernelising the models results in parameter reduction of up to 93%. The parameter efficiency consequently results in less data requirement, better generalisation, and faster convergence. This reduction can be further improved by implementing strategies such as sharing auxiliary variables between layers or potentially with low-rank approximations, particularly in scenarios where data is limited and concerns about overfitting arise (see Appendix for additional details).
8

---

Published as a conference paper at ICLR 2024
Table 4: Number of parameters. Methods prepended with FF are our kernelised versions with % reduction in brackets
Architectures
Dataset
RealNVP
FF-ReaINVP (ours)
Glow
FF-Glow (ours)
Miniboone
377K
117K (69%)
395K
141K (64%)
Hepmass
288K
76K (74%)
293K
79K (73%)
Gas
236K
22K (91%)
237K
23K (90%)
Power
228K
16K (93%)
228K
20K (91%)
BSDS300
458K
171K (63%)
497K
279K (44%)
DISCUSSION AND LIMITATIONS
We have introduced Ferumal flows, novel approach integrate kernels into flow-based generative models. Our study highlighted that Ferumal flows exhibit faster convergence rates, thanks to the inductive biases imparted by data-dependent initialisation and parameter efficiency. Moreover, we have demonstrated that kernels can significantly reduce the parameter count without compromising the expressive power of the density estimators. Especially in the low-data regime, our method shows superior generalisation capabilities, while Glow and RealNVP fail entirely, and FFJORD lags significantly in performance. We also demonstrate the application of our method in hybrid modelling. (Please refer Appendix H
In contrast to selection. In classical kernel machines, the choice of kernel usually implies type of inductive bias (e.g.. for specific data types (Vishwanathan et al. 2010)). Consequently, in this work, we mostly focus on Squared Exponential kernels and Matern kernels, but incorporating kernels with strong inductive biases may be promising avenue for future research. In particular, parameter sharing for highly structured modalities such as images another potential direction for future research.
based flows, kernelised flows require a different hyperparameter
The present work introduces kernels only for some affine coupling layer architectures such as RealNVP and Glow. However, the concepts also directly apply other coupling-layer-tyy networks, such as neural spline flows (Durkan et al.] (209)), ButterflyFlows (Meng et al.) [0022), or invertible attention (Sukthanker et 2022) for greater expressiveness and parameter efficiency. Ferumal flow kernelisation can also be directly enhanced with other building blocks such as MixLogCDDF-couling layers Ho et al. 2019).
Whilst our method can be applied to coupling- type dow-based architectures, it poses challenges when comes to ResFlow-like architectures [Behrmann et al. (2019;; Chen & Gopinath| (2000), which require explicit control of Lipschitz properties of the residual blocks. As result, extending our approach to ResFlow- architectures is left as a direction for future research.
One major drawback of existing normalising flow algorithms is their dependence of training data. The introduction of kernels into these models may allow the application of flows in low-data settings. Additionally, in the of increasingly large and complex models, energy consumption has become a significant concern. Faster convergence can contribute to energy savings. Notably, our models, owing their faster convergence and few hyperparameters needed fewer training runs than the neural-network counterparts, We anticipate that future research will continue to explore efficient methodologies and strive for reduced energy and data demands.
abundance
ACKNOWLEDGMENTS
We extend our gratitude to Noel Danz for his valuable discussions on coding. We would also like to express our appreciation Arkadiusz Kwasigroch and Alexander Rakowski for their initial feedback on the draft. This research was funded by the HPI research school on Data Science and by the European Commission in the Horizon 2020 project INTERVENE (Grant agreement ID: 101016775)

---

Published as a conference paper at ICLR 2024
REFERENCES
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jöm--enriik Jacobsen. Invertible residual networks, 2019.
Clare Bycroft, Colin Freeman, Desislava Petkova, Gavin Band, Lloyd T. Elliott, Kevin Sharp. Allan Motyer, Damjan Vukcevic, Olivier Delaneau, Jared O'Connell, Adrian Cortes, Samantha Welsh, Alan Young. Mark Effingham, Gil Veanan, Stephen Leslie, Naomi Allen, Peter Donnelly, and Jonathan Marchini. The uk biobank resource with deep phenotyping and genomic data. Nature, 562 (7726):203-22) 209, October 2018. ISSN 0028-088 0088-066..200 doi: 10. 10110048118880
118-0579-2. Funding In- formation: Acknowledgements We acknowledge Wellcome Trust Core Awards 090532/2/09/Z and 203141/7/16/Z and grants 095552/7/112 1/Z (to P.D.). 100956/7/1//Z (to G.M.) and 100308/2/12/Z (to A.C.). J.M. is supported by European Research Council grant 617306. S.L. supported by Australian NHMRC Career Development Fellowship 1053756. The sample processing and geno- typing was supported by the National Institute for Health Research, Medical Research Council, and British Heart Foundation. We thank the Research Computing Core at the Wellcome Centre for Human Genetics for assistance with the computational workload. We thank Affymetrix for discussions concerning quality control. We thank A. Young. A. Dilthey and L. Moutsianas for their assistance with aspects of the data analysis. We acknowledge UK Biobank co-ordinating centre staff for their role in extracting the DNA for this project. We thank M. KuzzaaaKu (http://mybioccience. org/) for Fig. 1. Publisher Copyright: @ 2018, Springer Nature Limited.
Scott Chen and Ramesh Gopinath. Gaussianization. Advances in neural information processing systems, 13, 2000.
Biwei Dai and Uros Seljak. Sliced iterative generator. https: http://wwww.//////////////
CoRR, abs/2007.006744 2020a. URL
Biwei Dai and Uros Seljak. Sliced iterative normalizing flows. In International Conference on Machine Learning, 2020b.
Andreas C. Damianou and Neil D. Lawrence. Deep gaussian processes, 2013.
Laurent Dinh, Jascha Sohl- Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL attp:://rcciv:
ics uci. .ed.meddmmmm
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows, 2019.
Eshant English, Matthias Kirchler, and Christoph Lippert. Mixerflow for image modelling, 2023.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation, 2015.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models, 2018.
Derek Hansen, Brian Manzo, and Jeffrey Regier. Normalizing flows for knockoff- free controlled feature selection. Advances in Neural Information Processing Systems, 35:16125-16113 2022.
Jonathan Ho. Xi Chen, Aravind Srinivas, Yan Duan, and Picter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722-2730. PMLR, 2019.
Zhichun Huang, Rudrasis Chakraborty, and Vikas Singh. Forward operator estimation in generative models with kernel transfer operators, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks, 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.66 1442.6980, 2014.
10

---

Published as a conference paper at ICLR 2024
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions, 2018.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes, 2022.
Matthias Kirchler, Christoph Lippert, and Marius Kloft. Training normalizing flows from dependent data. arXiv preprint arXiv:2209. 14933, 2022.
Ivan Kobyzev, Simon J.D. Prince, and Marcus A. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions Pattern Analysis and Machine Intelligence, 43 (11):3964-399 3979, nov 2021. doi: 10. 10/0/00mi..0000029990000 2992934. URL
110982Ftpam1 2020 2992934
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009a.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009b. URL https
7/api. www.ntttncncchhoo........
1822668744
Laparra, G Camps--Vall and Malo. Iterative gaussianization: From ICA random rotations.
IEEE Transactions on Neural Nenworks, 22(4) 549, apr 2011. doi: 20.1009/m..2111.210666
URL /ttt::::/doi.rgg100.111111 11t.10...11922tttt. 20t.ttmmn...1111155553
Sanggil Lee, Sungwon Kim, and Sungroh Yoon. Nanoflow: Scalable normalizing flows with sublinear
parameter complexity, 2020.
Juan Maroñas, Oliver Hamelijnck, Jeremias Knoblauch, and Theodoros Damoulas. Transforming gaussian processes with normalizing flows, 2021.
Ulysse Marteau-Ferey, Francis Bach, and Alessandro Rudi. Sampling from arbitrary functions via psd models, 2021.
Chenlin Meng, Yang Song, Jiaming Song, and Stefano Ermon. Gaussianization flows, 2020.
Chenlin Meng, Linqi Zhou, Kristy Choi, Tri Dao, and Stefano Ermon. ButterflyFlow: Building invertible layers with butterfly matrices. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 15360-15375. PMLR, 17-23 Jul 2022. URL htt.:htts////oooeee meng222..htr
ngs ..r....r..... wtt..mml.....ss..e 62/
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation, 2018.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, heghhperformance deep learning library. Advances in neural information processing systems, 32, 2019.
John Platt. Sequential minimal optimization: fast algorithm for training support vector machines. 1998.
Joaquin Quinonero-Candela and Carl Edward Rasmussen. A unifying view of sparse approximate gaussian process regression. Journal of Machine Learning Research, 1639))193911955 2005. URL
http:///wwggiiioooooeeroo cande html
Alessandro Rudi and Carlo Ciliberto. Psd representations for effective probability models, 2021.
Bernhard Schölkopf, Ralf Herbrich, and Alex Smola. A generalized representer theorem. In
Computatiional Learning Theory: 14th Annual Conference on Computational Learning Theory,
COLT 2001 and 5th European Conference on Computational Learning Theory, EuroCOLT 2001 Amsterdam, The Netherlands, July 16-19, 2001 Proceedings 14, pp. 416-426. Springer, 2001.

---

Published as a conference paper at ICLR 2024
Bernhard Schölkopf, Alexander Smola, Francis Bach, et al. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2002.
Sören Sonnenburg. Gunnar Rätsch, Christin Schäfer, and Bernhard Schölkopf. Large scale multiple kernel learning. The Journal of Machine Learning Research, 7:1531-1565, 2006.
Rhea Sanjay Sukthanker, Zhiwu Huang, Suryansh Kumar, Radu Timofte, and Luc Van Gool. Genera- tive flows with invertible attentions, 2022.
Russell Tsuchida, Cheng Soon Ong, and Dino Sejdinovie. Squared neural families: A new class of tractable density models, 2023.
S Vichy N Vishwanathan, Nicol N Schraudolph, Risi Kondor, and Karsten M Borgwardt. Graph kernels. Journal of Machine Learning Research, 11:1201-2242, 2010.
Li Wenliang, Danica J Sutherland, Heiko Strathmann, and Arthur Gretton. Learning deep kemelss for exponential family densities. In International Conference on Machine Learning, pp. 6737-6746. PMLR, 2019.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
12

---

Published as a conference paper at ICLR 2024
A ADDITIONAL EXPERIMENTAL DETAILS FOR OUR METHOD
We employed the Adam optimiser exclusively for all our experiments. The other hyperparameters are chosen using a random grid search, i.e. Ir E [0.01,00 0.005, 0.001], 11, 82 E [0.85, 0.9, 0.95, 0.99), kernel E |matern32. matern52, rbf]. We used Cosine decay for all experiments with the minimum learning rate equalling zero. In initial experiments, we found that 150 auxiliary points performed satisfactorily and we persisted with it for the majority of datasets and tried 200 for datasets with high-dimenssinnality. For comprehensive information, please refer to Table 5
For the experiments presented in Table [ we employed Glow-based architectures for both approaches. We trained these datasets for 1000 training steps, with the data synthesised at every training step (as done in existing implementations). For experiments in Table I T we exclusively use the RBF/Squared-Exponeee kernel and randomly sampled the kernel length scale from log-uniform distribution, i.e., ~ exp(U), where U ~ U1-22221
Table 5: Model Architectures and hyperparameters for our method.
Datasets
Method
Power
Gas
Hepmass
Miniboone
BSDS300
Dimensionality
8
21
43
63
Training Points
1,615,917
852,174
315,123
29,556
1,000,000
layers
14
12
15
12
12
kernel
matern52
matern32
matern32
matern52
rbf
auxiliary points
150
150
150
150
200
learning rate(Ir)
0.005
0.005
0.01
0.005
0.005
31
0.95
0.9
0.99
0.99
0.95
32
0.9
0.99
0.99
0.90
0.85
Ir schedular
cosine
cosine
cosine
cosine
cosine
min Ir epochs
0.95
0.95
0.95
0
200
200
500
600
400
batchsize
2000
2000
1024
2000
1024
B MODELLING DISCONTINUOUS DENSITIES
We consider two toy datasets with dis- continuous densities, CheckerBoard and Diamond, We used flow steps for all the models and piecewise poly- nomial kernel and learnt its hyperpa- Figure?Jshowcases showcases the orig- inal data samples(Left) alongside the modelled (endity(Middle) and the sam- ples generated using the floh(Right). The ceural-net-based architecture for the checkerboard dataset in the first row shows blurry boundaries and ill-defined corners. The kernelised counterpart in the second row has better-defined boundaries in some cases. However, this effect is even more pronounced in the diamond dataset with the kernelised counterpart in the fourth row modelling the discontinuity way better. Table 2ooles the corresponding log- likelikelihood in nats, quantitatively showing the enhancement from our kernelisation. The results demonstrate that Ferumal flow kernelisation can outperform its neural net counterpart on these toy datasets. All the toy datasets trained with batch size of 512 and for 100K iterations. We found that the piecewise polynomial kernel was better suited for discontinuous densities than Matern kernels. This provides another evidence of better performance due inductive biases of a kernel.
Table 6: Results on discontinuous densities. Log Likelihood in nats, higher is better
Dataset
Ours (#params)
NN-based (#params)
rameters. Figure 2
Checkerboard
-3.61 (1.8K)
-3.68 (18.4K)
Diamond
-3.24 (1.8K)
-3.39 (18.4K)
13

---

Published as a conference paper at ICLR 2024
1. Neural-Net-based flow for the Checkerboard dataset
2. Our kernelised flow for the Checkerboard dataset
3. Neural-Net-based flow for the Diamond dataset
4. Our Kernelised flow for the Diamond dataset
Figure 2: Discontinuous distributions. Shown are training data (left column), flow density (center column), and and histogram of flow samples (right column)
14

---

Published as a conference paper at ICLR 2024
Table 7: Results on small subset of 500 examples. Log-Likelihood in nats, Higher the better
Dataset
shared auxiliary (#params)
low rank (#params)
no auxiliary (#params)
Miniboone
-27.75 (58K)
-8.53 (19K)
-41.83 (345K)
Hepmass
-27.90 (41K)
227.91 (14K)
-29.01 (126K)
Gas
+0.22 (11K)
-1.85 (6K)
-10.19 (32K)
Power
-2.91 (8K)
-3.13 (6K)
-9.26 (36K)
BSDS300
+121.22 (85K)
+111.91 (17K)
+109.265 (505K)
C LOW-RANK APPROXIMATIONS
For datasets characterised by high dimensionality and complex structures, relying solely auxiliary points for the weight matrix (A E R2(D-4)xN) is inefficient. When half of the dimensions are transformed (as the case in any coupling layer), this matrix becomes excessively large.
To preserve the desirable quality of generalisation in our models, we propose alternative approach to obtaining the weight matrix for kernelised layer. We suggest using the product of two smaller matrices (with fewer parameters) instead. For weight matrix A E RPxN responsible for producing p affine parameters using N auxiliary points, we can learn two smaller matrices: A E ReeXN and A E ReXP where с < p. By employing the outer product Å = A A ~ A as proxy for a full-weight matrix,
effectively reduce the number of parameters.
In Table 7 we present the effectiveness of employing a low-rank approximation on the identical subset of 500 samples, depicted in Table 3 in the main manuscript. This technique ensures minimal number of parameters while achieving satisfactory generalisation. During the experimental setup, we endeavoured to utilise the lowest feasible value of (chosen via a hyperparameter grid of {4. 8, 12}) that would still yield reliable generalisation. Notably, our approach achieves good results while providing superior control over the parameters, contrast the shared auxiliary variable method.
Learning without auxiliary points We also present a comparison of another variation of our method, i.e., learning without the use of auxiliary variables. Notably, the sharing of auxiliary variables yields the best results, followed by the utilisation of low-rank matrices in conjunction with auxiliary variables. Whilst the results obtained with low matrices are not significantly different from those obtained with shared auxiliary variables, they offer further reduction in parameters. As seen Table 77 not using auxiliary variables causes a high number of parameters and the model overfits easily causing comparatively poor results (notably fewer parameters than FFJORD, depicted in Table in the main manuscript, while achieving somewhat similar performance).
D DENSITY ESTIMATION ON REAL WORLD MEDICAL DATASET
Following our experiments within the low-data regime, as discussed in Section 5.4 we extend our analysis real-world medical dataset-the the UK Biobank (Bycroft al. 2018 Kirchler al.) 20022). This dataset encompasses phenotype and genotype information for substantial cross-section of the UK population, encompassing total of 30 biomarkers. Notably, only 3,240 individuals within the dataset possess complete information on all biomarkers.
In line with our experiments in Section 5.4 we conducted a comparative analysis between our kemelised-Realle and FFJORD. The density estimation results presented in Table 8 exhibit better performance whilst training significantly faster, thereby reinforeing the findings in Table
Table 8: Results on the UKBiobank's biomarker data. Log Likelihood in nats, higher is better
Method
Ours
params
train time
Ours
-29.11
41K
21 min
FFJORD
-31.01
1.1M
13.1 hrs
15

---

Published as a conference paper at ICLR 2024
E DISCUSSION ON OUR USE OF KERNELS
Effect of kernel length scale As with any kernel machine, the length scale serves as a highly sensitive hyperparameter in our method. During our investigation, we discovered that in identical settings, distinct kernel length scales produce varying outcomes. Certain scales have a tendency to overfit easily on the training set (e.g.. -3.76 nats training likelihood on Miniboone), while some tend to underfit (e.g.,-40.23 nats training likelihood on Miniboone). This diverges from neural-net-base flows, where overfitting necessitates additional layers or nodes within each layer (bearing mind that this results in an increase in parameters unlike our method). Such findings vividly demonstrate the high expressiveness of kernelisation flow-based models.
Composite kernels In our experiments, we mostly employed the Squared Exponential Kerne//BBF, Matern kernels. Nevertheless, it is feasible to employ a combination of kernels, also known as multiple kernel learning (Sonnenburg al.. 2006). We defer the comprehensive analysis of kernel composition and its application to future endeavours.
F
RUNTIME COMPARISONS
Power train loss
Gas train loss
Hepmass train loss
RealNVP
RealMW
Rew
IF RCaINVP Glow
2.5
T-TF RealliV? Glow
26
=ff Rea NVP
Train Loss
Train Loss
0.0
Train Loss
Glow
24
22
-7.5
20
1509 2007 2500
Iterations
Iterations
Iterations
Power test loss
Gas test loss
Hepmass test loss
28
Real/VP RealNVV Clow
2.5
RealMV FF RealhW? FF Glow
- RVa
Test Loss
Test Loss
0.0
26
= Ff ReanNV
FF Glow
Test
24
-5.0
22
+7.5
20
Iterations
Iterations
lterations
Figure 3: Negative log- ligllikelhoodd (loss in nats) on training and test sets over the first 3,000 training iterations. Methods prepended with FF are our kernelised versions. All models were further trained until convergence.
Table 9: Training time, hours and minutes
Dataset
Glow
Ours
FFJORD
GF
Miniboone
63m
51m
5h 21m
4h 37 m
Hepmass
8h 23m
8h 49m
day
> 1 day
Gas
8h 45m
9h 07m
> 1 day
> 1 day
Power
8h 13m
8h 50m
day
> day
BSDS300
10h 57m
10 49m
> 1 day
> 1 day
We picked the best models in each category from Table and compared them with our kernelised Glow model. For fair comparisons, we ran the models for the same number of epochs and used the same number of flow steps for the neural-net counterpart, Glow model. It is worth noting that our model has faster convergence up to 3 times than the Glow model. Table Ihsoww that despite using kemelss, we have comparable times with the neural-network counterpart. We perform significantly
16

---

Published as a conference paper at ICLR 2024
better than continuous time normalising flow, FFJORD, and iterative- Srraaiie-gaussiaiiaatiino based kernel method, Gaussianisation flow, both take longer than a day on bigger datasets.
G IMPROVING MIXERFLOW'S IMAGE GENERATION
(a) NN-based MixerFlow
(b) Our kemeliied MixerFlow
Figure 4: Sampled images from MixerFlow
MixerFlow (English al. 2023) is flow architecture for image modelling. Unlike Glow-based architectures that relies on convolutional neural networks for image generation, MixerFlow offers flexible way to integrate any flow method, making it suitable for image generation with our kernelised coupling layers. In this section, we showcase the application of our kernelisation to the MixerFlow model on the CIFAR-10 1-10 dataset (Krizhevsky) 20096). We employed small models(30 layers) and trained on single gpu, For fair comparison, we kept the same model architecture with the sole change being the replacement of the neural-network- hetwork-bbsedd coupling layers with our kernelised coupling layer. Our results as shown in Table [0 demonstratee kernelisation can yield better result with faster convergence attributed to reduction of parameters. The generated samples can be seen in Figure 4a
and 4b
Table 10: Results on the CIFAR-10 dataset. Log Likelihood in nats, higher is better
Method
Ours
params
convergence step
Ours(kernelised)
-7644.39 -7665.65
4.85M
103K
NN-based MixerFlow
11.43M
195K
H IMPROVING VAE'S ELBO
We also try our kernelised flows in hybrid settings, demonstrating that we can integrate them with beural-net-based architectures. We apply our kernelised model, FF-GLow, to the variational autoencoder (Kingma Welling 2022) in the form of flexible prior and approximate posterior distributions. We apply the methods to KuzushijiMMNIST, which is a variant of MNIST containing Japanese script. We investigate the capacity of our kernelisation to improve over the baseline of standard-normal prior and diagonal-nommn
approximate posterior, and its neural network counterpart, Glow. We use 6 flow steps for each Hlw-based model and the latent hidden dimension equals 16. The quantitative results shown in Table II and generated image samples in Figure 7
Both models (Glow and ours FF-Glow) improve significantly over the standard baseline. However, there is considerable quantitative gain by using the kernelised version. We believe that this might
17

---

Published as a conference paper at ICLR 2024
be due the Glow model being sufficient to model the latent space the dataset and having little margin for kernelisation to shine. However, our kernelisation still helps in making the model parameter efficient with only small increase in parameter complexity compared to the baseline.
Table 11:
VAE test-set results (in nats) for the evidence lower bound (ELBO) on the Kuzushiji-Mo dataset. Error bars correspond to two standard deviations.
ELBO
log p(z)
Params
Bascline
195.611
-182.33= 1.30
1.18M
Glow
-189.99-1135 11.9.4811.36
-178.89-1.22
2.05M
FF-Glow (ours)
-178.54=1.25
1.23M
I DETAILS THE DATASETS
In the following paragraphs, brief description of the five datasets used in Table 2 (POWER, GAS, HEPMASS, MINIBOONE, BSDS300) and their preprocessing methods is provided.
Table 12:
Log-like log-likelihood measured in nats. larger values are better. Methods prepended with FF are our kernelised versions and the results with are taken from existing literature. Error bars correspond to standard deviations.
Datasets
Method
Power
Gas
Hepmass
Miniboone
BSDS300
Coupling
RealNVP
0.17 0.01
8.33 0.14
-18.7.1 0.02
-13,551 0.49
153.28 1.78
FF-ReaINVP (ours)
0,24 0.01
9.55=0003
-18.20+0.04
1-1.19 0.35
154.30= 2.11
Glow
0.17=
8.15= 0.40
-18.92:1.008
-1..35=0.07
155.07 + 1.03
FF-Glow (ours)
0.35 0.01
10.75 0.02
-17.1120.02
-10.76=0.44
154.71= 0.28
Non-coupling
MADE
-3.08 0.03
3.56+ 0.04
-0.988 0.02
-15.59
0.50
148.85+ 0.28
MAF
0.24
10.08 0.02
-17.701 0.02
-11.752
0.44
155.69 + 0.28
FFJORD
0.46
8.59 0.12
-14.92
-10.43# 0.04
157.40= 0.19
The POWER dataset comprises measurements of electric power consumption in household spanning 47 months. Although it is essentially a time series, each example is treated as an independent and identically distributed (i.i.d) sample from the marginal distribution. The time component was converted into an integer representing the number of minutes the addition of uniform random noise. The date information is omitted and the global reactive power parameter, as it had numerous zero values that could potentially introduce large spikes in the learned distribution. Uniform random noise was also added to each feature within the interval [0, e:]. where e, is chosen to ensure that there are likely no duplicate values for the i-th feature while maintaining the integrity of the data values.
POWER:
day, followed by
The GAS dataset records readings from an array of 16 chemical sensors exposed to gas mixtures over a 12-hour period. Like the POWER dataset, it essentially a time series but was treated as if each example followed an i.i.d. distribution. The data selected represents a mixture of ethylene and earbon monoxide. After removing strongly correlated attributes, the dataset's dimensionality was reduced to 8.
GAS:
The HEPMASS dataset characterizes particle collisions in high-energy physics. Half of the data correspond to particle-producing producing collisions (positive), while the remaining data originate from background source (negative). In this analysis, we utilized the positive examples from the "1000" dataset, where the particle mass is set to 1000. To prevent density spikes and misleading results, five features with frequently recurring values were excluded.
HEPMASS:
MINIBOONE:
The MINIBOONE dataset is derived from the MiniBooNE experiment at Fermilab. Similar to HEPMASS, it comprises positive examples (electron neutrinos) and negative examples
18

---

Published as a conference paper at ICLR 2024
(muon neutrinos). In this case, only the positive examples were employed. Some evident outliers (11) with values consistently set to -000 across all columns were identified and removed. Additionally, seven other features underwent preprocessing to enhance data quality.
The dataset was created by selecting random 8x8 monochrome patches from the BSDS300 dataset, which contains natural images. Initially, uniform noise was introduced dequan- tize the pixel values, after which they were rescaled to fall within the range [O, 1]. Furthermore, the average pixel value was subtracted from each patch, and the pixel located the bottom-right corner was omitted.
BSDS300:
an
11100000
7
18
Figure 5: Original samples after binarisation
19

---

Published as a conference paper at ICLR 2024
TRanen
HELOR
Figure 6: VAE samples from neural-net-basee Glow
20

---

Published as a conference paper at ICLR 2024
KAPP
Figure 7: VAE samples from our kernelised flow. FF-Glow
21


Reviews:
[{"review_id": "ZRxyoMsBeZ", "reviewer": "Reviewer_12iZ", "summary": "The authors introduce their research, which aims to enhance traditional normalized flow generative models. These models typically employ neural-network-based transformations to map simple prior distributions to more complex, invertible distributions for density estimation and data generation. In their study, the authors concentrate on integrating various kernels into flow-based generative models. Their goal is to accommodate smaller datasets, reduce the number of parameters (improve parameter efficiency), and lower computational costs, all while maintaining model expressiveness. The authors propose a novel method called \"Ferumal flows,\" which extends popular coupling layer architectures such as RealNVP (Real Non-Volume Preserving) and Glow (Generative Latent Optimization) by incorporating widely-used kernels, including the Squared Exponential and Matérn kernels.", "questions": "In your introduction, you alluded to the potential application of your models in the medical field. Why did you choose not to evaluate your models on a medical dataset as part of your research?\n\nMany researchers and practitioners might find value in accessing your code and models for further exploration and application. Could you please share your considerations regarding the possibility of providing links or access to your code and models in the future, and if so, where could interested parties expect to find them?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I commend the authors for their dedicated focus on tackling the challenge of low-data scenarios in generative modeling. Furthermore, I value their efforts in quantifying the improvements brought about by their proposed architectures, particularly in terms of reducing computational demands for hyperparameter tuning and training convergence compared to other approaches.", "weaknesses": "It would have been beneficial if the authors had provided links to a repository containing their code and models for improved accessibility and reproducibility. Additionally, given the authors' initial reference to the potential application of their models in the medical field, it would have been valuable to include an evaluation of their models' performance on a medical dataset to demonstrate their practical applicability and potential benefits in that specific domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v2HI92P2Rj", "reviewer": "Reviewer_ffkL", "summary": "In the paper, the authors introduce a kernelized normalizing flow paradigm that integrates kernels into the classical normalizing framework. The authors introduced theoretical fundamentals and presented results on relatively small datasets.", "questions": "1. How the method works on large image datasets.\n2. Ho the model works on a 2D dataset in comparison to FFJORD.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduce interesting concept in classical flow model\n2. The paper has good theoretical fundaments.", "weaknesses": "1. In Fig 1, authors should add results from more methods, like FFJORD \n2. How does the model work on a spiral 2D dataset?\n3. In the main paper, we do not have any image datasets. In the appendix, we have Kuzushiji-MNIST dataset. Authors should evaluate the model on MNIST, CIFAR, and CELEBA data special when we compare methods with Glow.\n4. In the paper, there is a lack of some illustration of the method. It can help to understand what exactly the kernels are in coupling layers.\n5. Section 3.1 is unclear. Authors to fast introduce formula (1). Maybe some simple example to present the main idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vQh65lmDjr", "reviewer": "Reviewer_wvm3", "summary": "This work focuses on developing normalizing flow-based approaches for cases where we have limited number of samples from the data distribution. Instead of neural network based transformations in coupling layers, the proposed method relies on kernel based transformations to reduce the number of samples required for convergence. The proposed approach slows promising results on 2D synthetic data and five real-world tabular benchmark datasets.", "questions": "The paper should include a more detailed comparison with FFJORD  especially in case of 2D synthetic datasets such as the discontinuous checkerboard dataset.\n\n·         The paper should discuss in more detail if the performance of the proposed approach can be further boosted by scaling the number of parameters.\n\n·         The paper should discuss in more detail if the proposed approach is applicable to complex datasets such as images.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed approach is novel and interesting. The paper correctly claims that normalizing flows which employ neural network based coupling layers are data and parameter hungry. The proposed kernel based approach in contrast is data and parameter efficient.\n\n·         The results in Table 3 show that the proposed approach shines in the low data regime. It clearly outperforms FFJORD and obtains impressive results even when only 500 data samples are available.\n\n·         The paper is well written and theoretically well founded. Proposition 3.1 is especially interesting as it states that the model  complexity of the proposed approach scales with naturally with the dataset size. This shows that the proposed approach is not over-parametrized like neural network based approaches and thus should be more suitable for the low-data regime.", "weaknesses": "·         Methods like FFJORD (\\cf Figure 2 in FFJORD) report better results compared to the proposed approach (as shown in Table 1). The performance advantage of FFJORD is even more apparent in case of the challenging discontinuous checkerboard dataset in Figure 3 (supplementary). It is not clear if the proposed model has the modelling capacity to capture complex distributions.\n\n·         The proposed method is outperformed significantly by FFJORD, although FFJORD uses more parameters as reported in Table 4. Can the performance of the proposed method be improved in Table 4 by increasing the number of parameters?\n\n·         For the experiments on the 2D toy datasets in Table 1, it is not clear which NN based approach is employed. Furthermore, the number of data samples used for training for all models in Table 1 should also be reported.\n\n·         The method is motivated by alluding to medical settings in Section 1 and 5.4, where data availability is often limited. However, the method is never applied to any medical data.\n\n·         Finally, it not clear if the proposed model can be applied to complex data distributions such as images. The qualitative examples of samples generated by the proposed model trained on the Kuzushiji-MNIST dataset as shown in Figure 4-6 are not promising.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RFnX80Iaab", "reviewer": "Reviewer_UUvL", "summary": "Update after rebuttal:\nI thank the authors for their response. The clarifications around the representer theorem and tabular / non image data were helpful. I encourage the authors to include the promised changes in their updated manuscript. I read the other reviews and rebuttal and do not see the negative points raised as a significant reason to reject this paper. I have raised my score to an 8.\n\n\nNormalising flows are probabilistic machine learning models that are capable of jointly solving the task of density estimation and generative modelling (sampling). Unfortunately, since they are parameterised in terms of a pushforward and their log-likelihood is determined using a change-of-variables formula involving a Jacobian, the mappings involved must be invertible, and are often even more constrained (e.g. through coupling layers). Invertibility imposes a big constraint on the model, meaning that the only lever one may pull to obtain expressiveness is often the depth and overparameterisation of the neural network. This results in data-hungriness, and makes them highly unsuitable for modelling tabular and low-dimensional data.\n\nThe authors introduce a kernelised version of normalising flows, which are suitable for modelling low-dimensional and tabular data. Efficacy is demonstrated on a set of benchmark datasets.", "questions": "Could the authors please clarify the weaknesses mentioned above? Specifically:\n- Is the discussion around the representer theorem accurate?\n- Can the model be overparameterised by increasing L, and how does this fit into the discussion around suitability of non-overparameterised/data hungry models for tabular / low dimensional data regimes?\n\nAdditional questions (neither strengths nor weaknesses)\n- I missed why the method is called \"Ferumal\". Can the authors quickly explain (apologies if this is mentioned somewhere already in the paper)?\n- Is there any method of constraining the support of the distribution (e.g. to lie on the simplex or some other set)? Perhaps this is as simple as constraining the support of the base distribution p_Z?\n- \"Our study highlighted that Ferumal flows exhibit faster convergence rates, thanks to the inductive biases imparted by data-dependent initialisation and parameter efficiency.\" I missed where the authors talked about data-dependent initialisation. Is this because the kernel matrix is seen as a parameter, which is initialised directly from the data, and then the auxiliary points are learned?\n- Future work. Is there a possibility to use deep kernel learning to learn the kernel as well? Would this be possible under MLE / MAP estimation, or is another objective required?\n- Future work. Can these be trained in a Bayesian manner? If the ELBO is used as an objective, can the representer theorem still be applied?\n- Open question: When is density estimation/sample generation of tabular data of practical concern in a machine learning context? I can see how sample generation is particularly interesting for high dimensional structured data (e.g. images). \n\n\nRelated works:\nThe authors might like to mention the more recent related literature of kernel methods for nonnegative functions as a way of building kernel-based probability density functions, which are also applicable and demonstrated in low dimensional / tabular data regimes, as well as their related neural-network based models through the NNGP. These also do not require invertibility. For example:\n- PSD representations for effective probability models, NeurIPS 2021\n- Sampling from arbitrary functions via PSD models, AISTATS 2022\n- Squared Neural Families: A New Class of Tractable Density Models, NeurIPS 2023", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper ideas presented in this paper are conceptually simple and do not require a strong leap-of-faith on the reader's end.\n- The mathematical idea (a representer theorem) appears to be mostly sound. \n- The experiments on tabular / low dimensional data are convincing and demonstrate the appeal of the method. A suitable benchmark is considered.\n- The paper is well-written. Text, equations, tables and figures are appropriately laid out and given the right amount of real-estate.", "weaknesses": "- I believe the following statement, after the proof of Proposition 3.1, is misleading (or possibly even incorrect): \"Note that, in contrast to the classical representer theorem, since the objective doesn’t contain a strictly convex regularisation term that penalises the model complexity, the solution is not necessarily unique\". I believe the authors might be referring to Theorem 1 of Scholkopf et al. 2001. In this classical representer theorem, the objective need not be convex, nor does the regularisation term need to be complex, nor does the solution need to be unique, nor does a convex regularisation term even necessarily enforce that the solution is unique (or, more weakly, that the objective is convex). The theorem simply says that any minimiser (if it exists), must be a finite linear combination of the kernel evaluations. It is true that in many typical applications of the representer theorem (e.g. kernel ridge regression, L2 penalised kernel logistic regression), the loss is strictly convex and the regularisation is strongly convex so the overall objective is strongly convex. However more generally, the theorem applies to non-convex objectives.\n- \"This makes kernelised flows especially promising for learning in the low-data regime, as their model complexity naturally scales with dataset size and does not over-parametrise as much as neural networks\", however there remains the previously mentioned caveat that the number of layers L is still a hyperparameter and increases model complexity. One can still over-parameterise with increasing L, or am I wrong?\n\nI am willing to update my scores if the authors can respond to these weaknesses, and also help me understand the answers to my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)