Injective flows for star-like manifolds
Authors: Marcello Massimo Negri
Jonathan Aellen
Volker Roth
Venue: ICLR 2025 Poster (2025)
Keywords: Normalizing Flows
Injective Flows
Bayesian Inference
Variational Inference
Objective Bayes
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We propose injective flows for star-like manifolds and show that they allow for an exact and efficient Jacobian determinant.

Abstract: Normalizing Flows (NFs) are powerful and efficient models for density estimation. When modeling densities on manifolds, NFs can be generalized to injective flows but the Jacobian determinant becomes computationally prohibitive. Current approaches either consider bounds on the log-likelihood or rely on some approximations of the Jacobian determinant. In contrast, we propose injective flows for star-like manifolds and show that for such manifolds we can compute the Jacobian determinant exactly and efficiently. This aspect is particularly relevant for variational inference settings, where no samples are available and only some unnormalized target is known. Among many, we showcase the relevance of modeling densities on star-like manifolds in two settings. Firstly, we introduce a novel Objective Bayesian approach for penalized likelihood models by interpreting level-sets of the penalty as star-like manifolds. Secondly, we consider probabilistic mixing models and introduce a general method for variational inference by defining the posterior of mixture weights on the probability simplex.

Published as a conference paper at ICLR 2025
INJECTIVE FLOWS FOR STAR-LIKE MANIFOLDS
Marcello Massimo Negri Jonathan Aellen & Volker Roth Department of Mathematics and Computer Science, University of Basel
(marcel Mharcellomassing, negri, Jonathan. volker malker.rothhere rothh]uniibss
ABSTRACT
Normalizing Flows (NFs) are powerful and efficient models for density estima- tion. When modeling densities manifolds, NFs can be generalized to injective flows but the Jacobian determinant becomes computationally prohibitive. Current approaches either consider bounds on the log- le-liikiihood or rely on some approx- imations of the Jacobian determinant. In contrast, we propose injective flows for star-like manifolds and show that for such manifolds we can compute the Jacobian determinant exactly and efficiently. This aspect is particularly relevant for varia- tional inference settings, where no samples are available and only some unnormal- ized target is known. Among many, we showcase the relevance of modeling densi- ties on star-like manifolds two settings. Firstly, we introduce a novel Objective Bayesian approach for penalized likelihood models by interpreting level-sets of the penalty as star-like manifolds. Secondly, we consider probabilistic mixing models and introduce general method for variational inference by defining the posterior of mixture weights on the probability simplex.
INTRODUCTION
Normalizing Flows (NFs) flexible and efficient models that allow to accurately estimate arbi- trary probability distributions. The key idea is to transform a simple base distribution into a com- plicated one through series of bijections. However, in many applications we know that the target density lies on a certain lower-dimensional manifold. common approach is to have the base distri- bution defined on the lover-dimensional space and use an injective transformation to embed into higher dimensions. Unfortunately, the computation of the transformed density involves an expensive Jacobian determinant term, which makes the model computationally prohibitive. Currently, exact and efficient Jacobian determinant is possible only for trivial manifolds like spheres and tori (Gemici al. 2016 Rezende et al.) 2020)) or for very restrictive transformations (Ross and Cresswell] (021]) )). In practice, most work renounce exact density estimation and approximate the Jacobian determinant term (Kumar et al. 2020 Kothari et al. 202 Sorrenson et al.) 2024), often with high variance estimators. Exact density estimation might not be critical when training with maximum likelihood, but is crucial in many applications like variational inference, where samples are not available.
In this paper, we introduce injective flows for star-like manifolds and show that efficiently compute the associated Jacobian determinant term, with the same computational cost as NFs. We consider star-like manifolds with intrinsic dimensionality d-1 and embedded in Rd which is a general class of manifolds that are particularly relevant in many statistical applications. Note that learning distributions, including the uniform distribution, arbitrary manifolds is non-trivial and often requires complicated sampling schemes (Pennec) 2006) Diacomis et al., 2012). Distributions on the hypersphere are crucial in directional statistics, astrophysics. medicine, biology, meteorology and many other fields (Chikuse 2012). Particularly relevant for geoscience applications is the fact that the Earth an oblate spheroid, which is star-like manifold. The probabilistic simplex, another star-like manifold, is very useful to model vectors that represent true probabilities by construction. Possible applications include probabilistic mixing models and other variations like Bayesian tracer mixing models, which are common in ecology, geoscience, zoology and many others (Phillips) 2012 Stock et al. 2018). and probabilistic treatment of archetype models (Seth and Eugster [0166] Keller et al. 20211. ) In some Bayesian settings, level sets of posteriors with one-parameter priors (e.g. sparsity parameter) define star-like manifolds. As we will argue, defining the posterior on such manifolds allows for objective Bayesian treatment of the problem.
exactly and

---

Published as a conference paper at ICLR 2025
We showcase the generality of our approach with two very distinct applications in Bayesian mod- eling. First, we introduce a novel Objective Bayesian approach to penalized likelihood methods. In this case the star-like manifold defines a level-set of the penalty constraint. Second, consider probabilistic mixing models and introduce a general framework for variational inference on the mix- ing weights. We constrain the posterior on the simplex, such that mixing weights always to one, and showcase an application in portfolio analysis.
We summarize the contributions of the present work as follows:
We propose injective flows for star-like manifolds.
In contrast to most existing work on injective flows, we show that we can exactly and efficiently compute the associated Jacobian determinant.
We showcase the relevance and broad applicability of the proposed method in a novel Objective Bayesian approach and for posterior inference in probabilistic mixing models.
2 BACKGROUND
Density and Jacobian determinant for bijective functions Let æ be a d-dimensional random variable with unknown distribution pa (z) and let z be a d-dimensional random variable with known base distribution p= (z(z). The key idea of NEs is to model the unknown distribution (a. through transformation T Rd - TR' such that
T(=). If T is diffeomorphism, i.e. differentiable
bijection with differentiable inverse T-1 the change of variable formula (Rudin) 1987) allows to express pz (a)) solely in terms of the base distribution (=() and T: par(2) (z(z) Idet JT())
where Jr is the Jacobian of the transformation T. The hypothesis can be relaxed to T being diffeomorphism a. e. (Munkres and Spivak| [965), i.c. except on zero-measure sets. Therefore, the trade-off consists of implementing bijections with tractable det Jr which are still flexible enough to approximate any well-behaved distribution. One key idea to exploit the property that, given a set bijections ( ) their composition T= (A) ..... o ((1) still a bijection. Since for bijections the Jacobian is square matrix, the determinant of composition of bijections factorizes as the product of the determinants of the individual bijections. Overall, NFs are built as
(x(x) p=(z) /det (z)2 with det 1T
II det J (u:-1) ==1
In general, computing the det JT has cost of (dd)) because requires some form of matrix decomposition (typically LU or QR). Instead, what most efficiently model bijection T by stack- ing simpler bijective layers (1)) with tractable (analytical) Jacobian determinant. Typically, each det J is made tractable by designing bijections 7 with triangular Jacobian by construction, such that its determinant is given by the diagonal entries, which has a (d)) complexity.
where u:-1 ((;))) and un
implementation exploit is the property in Eq. (
Density and Jacobian determinant for injective functions NFs are limited by the use of bijec- tions, which prevents modeling densities on lower dimensional manifolds. In such cases the target distribution lives on a m-dimennional manifold M embedded in a d-dimensional Euclidean space M IRd where m < In order to constrain ps (æ) to live on the manifold M, we need an injec- tive transformation that inflates the dimensionality T m- Rm + Rd. The transformed probability distribution (x) can still be computed by accounting for the volume change (Ben-Israel 1999):
(x) = Pe(2) (vol J (z)) with vol JT = V ((JT J JTm-a)
(2)
where JT., va(z) E dd is a rectangular matrix. Note that m = d, JJ is a square matrix so /det Jf det JT det JT and Eq. ( reduces Eq. ① Crucially, since Jmm_. is now rectangular, the Jacobian determinant cannot decomposed as the product of stacked transformations anymore, which is the crucial property that makes bijective flows tractable (see Eq. (l). Instead, we need to explicitly compute the Jacobian determinant, which is O(m²) in general. This makes injective flows computationally prohibitive for high dimensional manifolds.
vol JT /det
2

---

Published as a conference paper at ICLR 2025
Even if were stack bijective layers after the injective transformation, would still be unable to write the overall Jacobian determinants as the product of individual bijections and would remain O(m)). To see this, consider the transformation T T T m-id m. where T R + R
and T gd + R2 are arbitrary bijections. Then, det JT factorizes:
det JT = det J V (TTT.JTu.ad) )a)' JT
(3)
We refer to Appendix A.2 for full derivation. Note that we can factorize only det JT.,. (the bi- jection that precedes the inflating step), while the Jacobian determinant of the bijections Ta after the inflating step cannot be disentangled. Naturally, the same would hold true if the bijections T were replaced by arbitrary injective transformations. In both cases we would need to compute the Jacobian product JT and determinant, which has cubic complexity.
INJECTIVE FLOWS FOR STAR-LIKE MANIFOLDS
We now present the proposed injective flow model densities arbitrary star-like manifolds. In particular, in Section 3.1 we introduce the definition of star-like manifolds and show that they can always be parametrized with generalized spherical coordinates. Then, in Section 3.2 define injective flows on star-like manifolds and show that the Jacobian determinant can be computed exactly and efficiently. Lastly, in Section 3.3 we discuss the limitations of the proposed approach.
3.1 STAR-LIKE MANIFOLDS
Definition 1.
We call domain a star domain S if there exists one point E S such that, given any other point s S in the domain, the line segment connecting So to s lies entirely in S. Furthermore, we define star-like manifold Ms as the manifold defined by the boundary of star domain. In this work we consider d dimensional star-like manifolds embedded in Rd
In other words, a star-like manifold is such there exists a point from which the entire manifold can be "viewed".. Intuitively, this suggests that we can always parametrize it with generalized spherical coordinates, which consist of d - 1 angles O E U = [0, r x [0, 2m] and one radius r -0 Let Ms be d dimensional star-like manifold embedded in IR.. Then, we need d 1 variables identify any point E Ms. In particular, we can parametrize = [0, r())]* with d - spherical angles O E Und-1 and suitable radius function r(0). If we choose sg as the origin of the spherical coordinate system, we can define the radius the line segment connecting and so- Crucially, by def- inition of star-like manifolds, the segment intersects the manifold only once, the radius is uniquely defined. See Figure [ for graphical rep- resentation. Star-like manifolds are the most general class of manifolds that always allow such parametrization.
10
Figure 1: 1D star-like manifold parametrized in spherical coordinates.
3.2 PROPOSED INJECTIVE FLOWS FOR STAR-LIKE MANIFOLDS
We now show how to define injective flows on star-like manifolds such that the Jacobian determinant, and hence the log density, can be evaluated exactly and efficiently. Relevantly, the modeled distribu- tion is expressed in Cartesian coordinates, which is crucial for most applications. In order to do so we compose three transformations T = Tg-+e o T 7 T (see Figure 2 (i) an arbitrary diffeomor- phism that maps to d - 1 spherical angles T To. (ii) the injective transformation T, that parametrizes the radius r()) as a function of the angles and (iii) the coordinate transformation T from spheri- cal to Cartesian coordinates. Note that the change of coordinates T, is diffeomorphism almost everywhere, i.c. except on a zero-measure set. Therefore, the change of variable formula holds and the probability resulting from the transformation is still exact. We refer to Appendix A. and A.4 for the explicit expression of To-+c and a discussion of its special structure, respectively. This way the resulting densities will be defined by construction the manifold parametrized by T while the density will be conveniently expressed in Cartesian coordinates. In other words, the push-forward samples points in the ambient space.
3

---

Published as a conference paper at ICLR 2025
01
01
8/
6P2
A
w
Fd-2
6d-2
02-2
24-22
2/x
A--1
0g-1
04-1
r(0)
T2
To
T,
T
arbitrary bijections
map angles
inflate radius
((0)
spherical to Cartesian
Figure 2: Architecture the proposed injective flows for star-like manifolds, see Theorem !
Naively computing the Jacobian determinant as in Eq. 3 requires cubic complexity. Instead, in Theorem we show for star-like manifolds how to compute analytically and efficiently ((P)) We provide proof sketch of the theorem below and the full proof in the Appendix A.4
Theorem 1. Let T = Ta-o T To as in Figure where To E Rd-1
θ E Ug 10, (0) E U
is any diffeomorphism to d-spherical angles, T, 0 U
Eso with
r(-) : 0 E Ug Y r E Ry0 being differentiable, and To O, r (O)) U x R80 R the d-spherical to Cartesian transformation (see Definition 2 in Appendix
Then, the Jacobian determinant of the full transformation is equal to
det (JT) = det (JTo) det (JT...) I[ yl12,
(4)
where y == (0) 1]". Relevantly, det JT can be computed exactly and efficiently in (dP)).
Proof sketch. (i) Injectivity It is easy see that T Tx oT, bijective, T, is injective and Totc is also bijective. The injectivity of T, can be easily seen by noting that 0 # 0' (0. (0) * [0', r(0'),, independently of r(-).
To is injective because To is
(ii) Analytical Jacobian determinant Since Jr E Raxd-1 is not square, we cannot use the usual property of bijections in Eq. (]). Instead, we use the result for injective functions in Eq. (3):
det (JT) det (JTa) det (JTrse = det (JTo) det (JJT..JT..) (JT...
where Jr, E dRdxd-1 and JT, E Rdddd, Note that, we show in Appendix A.4 Remark 1 we can factor out only the Jacobian determinant of To. i.e. the bijection that precedes the dimensional inflation step with T,.. The term det JT, is the standard Jacobian determinant for bijective layers and can be computed efficiently. We are then left to compute det (Jr... To do so we consider the matrix JT.- [JT, 0xx1] 1xdd and substitute the determinant with the pseudo-determinant:
det (Jr, = vdctt (J..JJJ.)- = - dett (J..J.T.))
where j==== Jf JT, E Idd With det. we denote the pseudo-determiinnt,, which is defined the product of all non-zero eigenvalues. The rest of the proof is based the key observation that JT_ has rank d - 1 equivalently, that its null space is onc-dimensional. As result, we can use Lemma (see Appendix A.4) and re-write the pseudo-determinen as trace of the adjugate matrix
det JJ..J
Tr adj (JF.J J J..))r det (J") Tr adj (J+))( adj (JF.))
where adj(.) is defined as the transpose of the cofactor matrix. If A is invertible (as for J 14. JT, ..) (A) det(A)A Since J4. has rank d Lemma holds (see Appendix A.4
adj(Jr.)= = det (eet((J.) my²
yrx
4

---

Published as a conference paper at ICLR 2025
where x = = 10, and y [- Vor(0), Finally, by substitution we get: 1|"
dett (H.. = det (Jr) det (JT.)² Tr (xyxxy" (J) v=²)) det ( (y²')²²
analyze the time complexity required evaluate Eq. The term det JT. is the standard Jacobian determinant for bijective layers, which can be computed in O(d). The Jacobian determinant for spherical to Cartesian coordinates det Jg is known in closed form (Muleshkov and Nguyen 2016) and can be evaluated in O(d). Therefore, we only need to show
(iii) Complexity We
that also w )Jf y can computed efficiently. Solving the full linear system would require
a complexity of Odd)). However, JT is almost- trinnsstrrianuuarr (see Appendix A.4 Eq. (22) and we
can make it triangular with one step of Gaussian elimination, which requires O(d²). The resulting triangular system can be solved in (dd)) Finally, note that we can compute J very efficiently
and analytically (see Appendix A.4 Eq. 24 ) without requiring autograd computations.
3.3 FURTHER DETAILS AND LIMITATIONS
Variational inference vs maximum likelihood In this paper we focus on variational inference settings (without observations), and not on the maximum likelihood setting (with observations), which is the main focus of most injective flows. Our motivation is two-fold. First, our method is designed for star-like manifolds (with codimension 1), which are more useful in variational inference settings. In maximum likelihood applications, the underlying manifolds are often assumed to much lower dimensional. Second, while in maximum likelihood settings the approximate Jacobian work well in practice, in variational inference settings the exact Jacobian determinant is crucial to learn the correct target distribution. We show this empirically, already in very simple cases.
Implementation details The proposed approach is straightforward to implement and only requires to adapt the function r(0) to the star-like manifold under study. This means that we should have access to the parametrization r(0), which is often trivial (see e.g. Appendix A.4 Eq. (25) and Eq. (26). Our method is also versatile as the transformation Tg can be implemented with any bijective layers of choice. Lastly, we easily avoid numerical instabilities arising from singular points in Ta-e by offsetting the critical angles with a small epsilon.
I Limitations The main limitation of the proposed method is that it cannot be easily generalized to manifolds with intrinsic dimensionality lower than 1. In particular, the proof of Theorem (I) relies on the fact that the cero-padded Jacobian matrix has rank d 1. Secondly, the expressivity of the proposed method also depends on the flexibility of the bijective layers. Despite state-of-the-
art bijective layers being extremely expressive (Perugachi- Diaz al. 2021), Liao and He (2021)
showed that the number of modes that can be modeled remains limited.
4 RELATED WORK
Normalizing Flows (NFs) consist of simple base distribution that is trans- formed into more complicated one through bijective transformations. One can show that such a construction allows us to approximate any well-hehaved distribution (Papamakarios et al. [021). In practice, the bijective transformations are implemented with neural networks that show trade-off between expressiveness and computational complexity. However, recently developed bijective lay- ers provide very efficient transformations that satisfy the universality property (Huang et al., 2018 Durkan et al. 2019 Jaini et al., 2019). For a comprehensive review of different bijective layers and for a discussion about applications we refer to PPapamakarios et al. (2021) and and Kobyzev et al. 202
Normalizing Flows
Variational Inference with NFs NFs are popular two distinct applications: (i) as generative models trained data or (ii) as powerful density estimators in variational inference settings. In the first case, we are given some samples and NFs are trained by maximum likelihood (forward KL. divergence) to approximate the data generating distribution and to later generate samples (Dinh) cet al., 2017] Papamakarios et al. 2017). In the variational inference setting, no samples are available and the target distribution is typically known only up a normalization constant. NFs are thus
5

---

Published as a conference paper at ICLR 2025
trained by reverse KL divergence and, once trained, they allow for both sampling and evaluation of the (approximate) target distribution (Rezende and Mohamed, 2015 Kingma et al. 2016). There is good evidence that more faithful posterior approximation leads to improved performance for variational inference tasks (Rezende and Mohamed| 2015), which is one reason that makes normal- izing flows attractive. This setting is the focus of our work and is particularly relevant for Bayesian inference (Louizos and Welling 2017), where the goal is to learn and sample from the posterior distribution given the (unnormalized) product of the prior and likelihood. In such settings NF' have also proven to be an attractive alternative to MCMC samplers (Negri et al. 2023).
Injective flows on manifolds The computational challenge of injective flows is the evaluation of the Jacobian determinant in Eq. 2 In the literature, exact and efficient computation of the Jacobian determinant has been shown only for trivial manifolds like spheres and tori (Gemici et al. 2016 Rezende et al. 2020 or for very restrictive transformations (Ross and Cresswell, 2021), J. In partic- ular, the latter focuses only maps for which JTJ x II, making the determinant trivial. These are however the only cases where the modeled density can be computed exactly and efficiently dur- ing training. This has a significant impact on variational inference settings where the quality of the approximation influences the exploration of the distribution domain. It is, in contrast, less relevant in data driven settings trained by maximum likelihood. Some early work ignored the determinant term altogether (Brehmer and Cranmer 2020). which was shown to have detrimental effects al- ready in simple -owddimensional settings (Caterini et al., (02T). Current work is focused instead on finding some tractable approximation to the Jacobian determinant. The most common one is to employ the Hutchinson's 's trace estimator (Mathieu and Nickel 2020, Caterini et al. 2021: Flouris and Konukoglu, 2023). which is characterized by high variance and is biased if used to estimate the log-detern log-determinant of the Jacobian (Kumar et al., 2020). State-of-the-art works employ surrogate log-like oogliikelhood losses and still approximate the Jacobian determinant (Sorrenson et al. 2024).
In contrast, we are the first to propose exact and computationally efficient injective flows for a wide class of manifolds, namely star-like manifolds.
5 APPLICATIONS
In this section we showcase some interesting applications of the proposed injective flows. In par- ticular, we overcome some limitations of current work with exact density estimation, which are applicable only trivial manifolds for very restrictive transformations. In Section [. we show empirically that the proposed approach provides significant speedup compared to the explicit com- putation of the Jacobian determinant, We also show that in variational inference the exact Jacobian determinant is crucial for training while the approximation commonly employed in injective flows results in poor reconstruction. In Section 5.2 we illustrate how the proposed approach learns distri- butions on simple 3D manifolds. In Section 5.3 we use injective flows to define a novel Objective Bayes approach to penalized likelihood problems. Lastly, in Section 5.4 we introduce a general framework for variational inference in probabilistic mixing models.
5.1 EFFECTIVENESS AND EFFICIENCY OF THE PROPOSED METHOD
We showed that for star-like manifolds we efficiently compute the Jacobian determinant in Eq. (2) and we argued that exact evaluation is crucial for variational inference. We now verify both statements empirically. Firstly, we compare the runtime associated with computing the Jacobian determinant via the explicit formula in Eq. (2) with our approach in Eq. (4). In particular, we use very simple manifold, the hypersphere in d dimensions, and measure runtime over 20 repetitions of Jacobian determinant computation. The results in Figure 33 show that our method provides a significant speedup compared to the explicit computation of the Jacobian. By fitting linear function to the log-runtime and dig-diennioon we get that the explicit computation is approximately cubic, (d2.96). while our proposed approach is approximately quadratic, (1d- Secondly, we show that training with the exact Jacobian determinant is crucial for variational inference. To do so we first train the proposed injective flow with the exact (and efficient) Jacobian in Eq. (4). Then, we train second identical flow where the gradient of the Jacobian is approximated with the Hutchinson trace estimator with different number of Gaussian samples n = 1, 10, 50, 100. This approach is
common in most injective flow papers (Caterini et al. 2021 Flouris and Konukoglu 2023). We use

---

Published as a conference paper at ICLR 2025
brute-force (exact JI (exact
оpprox. J (1
spprox. 7 [n-10]
аррrox: 50n
spprox. J (00)100)
runtime
mse
O(d
100 s0ie)
23155
5000
10000
dimension
dimension
a
Figure 3: (a) Runtime comparison of Jacobian determinant computation with proposed approach in Eq and with the brute-force computation Eq. (2) (b) MSE of learned log-density for an injective flow trained to learn a uniform distribution on a (p-(pseudo) norm ball with p 0.5. We compare the same model trained with our exact Jacobian and with the Hutchinson trace estimator with n = 1, 10, 50, 100 number of Gaussian samples. Note that the number of samples is upper bounded by the dimensionality of the problem and that at evaluation the exact Jacobian is used.
the implementation of state-of-the-art work (Sorrenson al. 2024). The task is to learn the uniform distribution on lp-(pseudo) norm ball with p = 0.5. In Figure 3wee compare the ground truth (log- ) density with that obtained with the two models. Note that at test time the model trained with the approximate Jacobian is evaluated with the exact Jacobian. In the Appendix in Figure I0 [0 we measure the similarity between samples of the two models and samples from the distribution. In all cases the proposed model achieves significantly better results both in terms of density reconstruction and samples quality. Even at the cost of increased runtime (shown in Figure increasing the number of samples the approximate method fails to achieve similar performance to ours.
5.2 ILLUSTRATIVE DISTRIBUTIONS IN 3D
We first illustrate the proposed model on simple 3D setting. In particular, we train the injective flow in variational inference setting where the target distribution is known only up a constant and no samples are given. Note that this is much more challenging than the usual maximum likelihood settings where flows are trained on data. In particular, it is not trivial to learn densities with many modes. To showcase the effectiveness of the proposed method we show the modeled density for increasingly difficult targets: (i) von Mises-Fisher distribution (x 5), (ii) mixture of 50 von Miscs-Fisher distributions arranged on a spiral (k = 50) and (iii) sinusoidal density (log p(0, Φ) = sin si0( sin (4))), where is the concentration parameter and 0 and angles, respectively. The sinusoidal density (iii) defined on a deformed sphere. In Figure A we show that the modeled densities perfectly match the ground truth in all cases. Furthermore, compare the modeled density with the ground truth over 10'000 samples and obtained an accurate reconstruction: MSE 0.013 (a), 0.011 (b). As the normalized density the deformed sphere is not given, we only provide qualitative visualization, which shows an accurate reconstruction.
the polar and azimuthal
gt
oaLs
gt
ours
gt
OurS
34
03
0.2
02
(8)
(b)
(c)
Figure 4: Learned ("ours") and ground truth ("gt") density for proposed injective flow trained via reverse KL divergence (no samples). Only the (unnormalized) target is given: (a) Mises-Fisher (k = 5), (b) mixture 50 von Mises-Fishee (ı = 50) and (c) sinusoidal density on deformed sphere

---

Published as a conference paper at ICLR 2025
5.3 OBJECTIVE BAYESIAN APPROACH TO PENALIZED LIKELIHOOD
Objective and subjective Bayes Bayesian inference is a powerful statistical method that requires a likelihood term, which explains the observed data, and a prior, which quantifies our initial belief. However, in many cases we might not have enough problem-specific knowledge to specify an in- formative subjective prior. This has led to the development of objective priors, which are designed to be minimally informative. Some objective priors include Jeffrees rule (Jeffreys 1961), reference priors (Bernardo 1979). and maximum entropy priors (Jaynes) 2003). Given the vast literature on objective priors (Berger 2006), in this work we do not intend to discuss whether objective priors preferable or not. Instead, we provide a new framework to define objectíve priors in settings where only subjective ones have been explored
far. Specifically, we consider Bayesian penalized likeli- hood problems and show that level sets of the penalty define star-like manifolds. Then, we implicitly define the objective prior as the uniform distribution on such manifolds. This choice aligns with the literature on objective priors for distributions on surfaces since a uniform distribution assigns equal
mass to equal volume Kass 1989
Kass and Wasserman 1996).
Objective Bayes for penalized likelihood models Let ~ X3 + e with e ~ (0, TTT,). where X E IR Raxd is the data matrix, E IR the targets and B E R the regression coefficients. We then optimize the mean-squared error |yy X 8 subject to the (pseudo-) norm penalties || with p > O, which encourages sparsity for S 1. Note that for p penalty (Tibshirani 1996) and for p 2 the Ridge penalty. (Tibshirami (1996) noted that we can interpret such penalized fikelihood in a Bayesian way with a Gaussian likelihood and suitable prior. Park and Casella (2008) showed that with an independent I Laplace prior the Maximum a Posteriori (MAP) of the posterior coincides with the frequentist solution. The above reasoning can be extended to any lp (pseudo-) norm : Ipp by using a generalized Gaussian prior on 3:
we recover the LASSO
arg min 1 lyy x + B1 arg max maxxN(XXB o o exp( (xp111111))) B (5) BER"
n0
BERd
A(
However, the generalized Gaussian is not the only prior for which Eq. (5) holds. Any monotonic transforma- tion h of p(3 results in the same MAP solution 3 (see Figure 8 in the Appendix). Therefore, the choice of A)), and hence of the prior, is subjective and, crucially, it influences the posterior. We show this empir- ically on toy data with the Laplace prior and two mono- tonic transformations: the square ("square laplace") and square root ("root laplace"). In Figure 5 we can clearly see that the posterior is influenced by the choice of the subjective prior, which is undesirable in the absence of specific assumptions. In contrast, we circumvent the choice of a subjective prior and propose general frame- work for objective priors in penalized likelihood methods.
aquare laplace
laplace
root laplace
regression coefficients
Figure 5: 95% posterior C.I. for 3 sub- jective priors with the same MAP. The choice of prior affects the posterior.
Objective Bayesian penalized likelihood with injective flows With subjective priors the penalty ||@|| Eq. (5) is enforced as a soft constraint controlled by such that ||33|p S k(A), for some &(A). Our idea is to enforce the norm penalty as hard constraint by defining the posterior on the manifold || k by construction. This way we do not require to explicitly specify a subjective prior and we are implicitly assuming a uniform prior on the manifold [|@||p = k. In summary:
Objective Bayes
Subjective Bayes
p(y) X, 9) N(X
p(y]X, (3) = -NX3,
posterior on manifold: || = k
prior: (( x II, exp(-
The equality || 1e induces star-like manifold which can parametrize with suitable radius function; see Appendix A.5 Eq. (25) for the explicit parametrization. Therefore, with our framework can define the (approximate) posterior qo (13) to be constrained on || by construction.
8

---

Published as a conference paper at ICLR 2025
We now illustrate the differences between the subjective and objective approaches with syn- thetic data. We use a NF to approximate the posterior
Objective Bayes
Subjective Bayes
)p(pp) (3) A) with the "square laplace" subjective prior p( A). We choose À such that the MAP has specific norm || k. Furthermore, we use an injective flow de- fined on || = k to approximate the target (X o II.) (implicitly uniform prior). In both cases training is performed by minimizing the reverse KI. divergence (see Appendix [A66] shows a crucial difference: Eq. (2)). Figure [
1|,01]
samples from the objective posterior lie exactly on the manifold while the subjective posterior is scattered. The bottom panel of Figure 6 shows the distribution of the sample norms varying significantly with the choice of the subjective prior, which agrees with Figure 5 We include
objective
square laplace
laplace
root laplace
Figure 6: Above: with the objective prior posterior samples lie on the manifold. Below: the norm of posterior samples depends on the subjective prior.
more implementation details in Appendix A.6] Lastly, note that we do not claim that the proposed
prior better in general, but rather preferable cases where less informative prior desired.
5.4 VARIATIONAL. INFERENCE FOR PROBABILISTIC MIXING MODELS
Probabilistic mixing models Mixing models are used to study the relative contribution of sources to observed mixture and are particularly relevant in ecology, geoscience, metcorology and many other fields (Phillips) 2012] Stock et 20188) Jiskra et al. 2021]. Formally, the aim is to reconstruct the mixture π of sources from which the observations D were generated, with æ being defined on the probabilistic simplex Cd {I E R² n. 2 0 E n. 1}. In the most general Bayesian formulation, we require prior p(TT) and some likelihood p(D) and the challenge is then to define the posterior p(T( x p(D p((T) on the probabilistic simplex Cd. Most approaches rely on the Dirichlet distribution, which is defined on Cd by construction: Dir(Tr) x II, x. -..-1 with a > 0. Since the posterior can be obtained in closed form only with a multinomial likelihood, most approaches rely on sophisticated MCMC samplers (Stock et al., 2018). As a more flexible alternative to MCMC methods, we present general variational inference framework where pl (D) is always defined on Cd,, leaving complete freedom in the choice of prior and likelihood.
Injective flows vs MCMC in Bayesian mixing models is easy to see that the probabilistic sim- plex Cd is a star-like manifold, see Appendix A.5 Eq. (26) for the explicit parametrization. There- fore, with the our framework we can define an injective flow qe (Tr on Cd by construction and train it approximate the posterior p(T p(D) n ((T)). In the simplest case when no prior is spec- ified, we are implicitly assuming a uniform distribution on the simplex, i.e. a Dirichlet prior with a; = 1 Vi. In the more general case, we can always plug in any combination of likelihood p(D)xr) and prior p(T), and the (approximate) posterior qe (T) will always be defined on Cd by design. In contrast, with MCMC methods it is not trivial guarantee posterior samples be on Cd., already for very simple likelihoods (Altmann et al. 2014 Baker et al. 2018). We compare our method with an MCMC sampler in the conjugate case (Dirichlet prior and multinomial likelihood) such that we can compare with the true posterior, which is available in closed form. We report the results in Appendix (..nddxxA.66inn i. Figure together with a description of the MCMC sampler. Results show that (i) the proposed injective flow correctly estimates the posterior distribution across all tested dimensions and (ii) it outperforms the MCMC sampler already in low dimensions. Interestingly, this experiment shows that our approach is able to learn very sparse solutions with many modes.
Application: Bayesian portfolio optimization We select a minimal Bayesian mixing model that already shows the advantages of our proposed method in terms of ficxibly choosing likelihood and prior. One such setting is index replication in the context of portfolio optimization (Markowitz and Todd 2000). A portfolio is defined as a set of n stocks which are held proportionally to the mixture components π E R such that E. ग. = 1. Let R e I Tx" be the returns over the time-steps (1,...,7) T) of the n stocks We are interested in optimizing the portfolio weights
9

---

Published as a conference paper at ICLR 2025
Uniform Dirichlet
0.2
0.14
10
10
Enum het non--wrr entries
mumber of non-acro entries
number non-tera entries
number of non-scro entries
1
samples
samples
samples
samples
Figure 7: Trained injective flow the simplex with target (D) ()) ("uniform" prior, implicitly) and p( of non-zero entries in posterior samples for fixed likelihood values (so fixed reconstruction). Below: we compare sparsity patterns of samples π with uniform (blue) and Dirichlet (orange) prior.
(D) (D) x p(D π Dir(Tr) ("Dirichlet" prior). Above: we compare the distribution
such that replicate the reference index returns E R while also incorporating investors personal preferences. For instance, a sparse portfolio allows to reduce transaction costs arising from trading (Sokolov and Polson, 2019). We formulate the problem in Bayesian fashion by specifying
Gaussian likelihood [(p π = - T²T,) and some sparsity-inducinn prior p(ær).
With the proposed framework we approximate the posterior p( R. p) p(p) π p((r)))))))))))))))) with an injective flow (œ(T)) defined on C" by design. The flow qe (T) is trained by minimizing the reverse KL divergence with the unnormalized target p(p) R, π ))(p (T). For the sake of illustration,
select portfolio with 10 stocks over period of 200 time steps from the dataset in Tu and Li (2024). We define qo (лr on the manifold and consider two priors: the uniform prior on the simplex and the Dirichlet distribution. In Figure 7 we show the distribution of non-zero entries of the posterior samples for the uniform and Dirichlet distribution. In particular, we consider the distribution and the sparsity patterns at 4 fixed values of the likelihood (one plot). Despite the likelihood being the same, the Dirichlet prior leads to a sparser solution with fewer non-zero entries. This is also noticeable in the sparsity patterns of the posterior samples in the bottom panel. In Appendix A.6 Figure [2] we also show the cumulative return and how it is affected by sparsity. Overall, we showed how easily can specify any likelihood and priors while constraining the posterior the simplex.
CONCLUSIONS
Previous work injective flows manifolds relies approximations or lower bounds to cireum- vent the computation of the Jacobian determinant term. In this work we showed how to exactly and efficiently compute the Jacobian determinant term for the general class of star-like manifolds. We validate empirically the claimed computational advantage and we show that exact Jacobian com- putation is crucial for variational inference, already in very simple settings. We then showed that the proposed flow allows for interesting applications that were not possible before. First, with the proposed framework we introduced novel Objective Bayes approach to penalized likelihood meth- ods. The idea is to cireumvent the choice of subjective prior by constraining the posterior on manifold defined by level-sets of the prior. Second, we introduced general variational infer- ence framework for modeling the posterior in probabilistic mixing models. Overall, the proposed framework allows us to efficiently model distributions on arbitrary star-like manifolds and to flexibly specify any choice of prior and likelihood.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Yoann Altmann, McLaughlin Stephen, and Nicolas Dobigeon. Sampling from multivariate gaus- sian distribution truncated on a simplex: A review. In IEEE Workshop on Statistical Signal Pro- cessing Proceedings, pages 113-116, 2014.
Fabricio Arend Torres, Marcello Massimo Negri, Marco Inversi, Jonathan Aellen, and Volker Roth. Lagrangian flow networks for conservation laws. In The Twelfth International Conference on Learning Representations, 2024.
Jack Baker, Paul Fearnhead, Emily Fox, and Christopher Nemeth. Large-scale stochastic sampling from the probability simplex. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2018.
Adi Ben-Israel,
The -harills--variables formula using matrix volume. SIAM Journal on Matrix Analysis and Applications, (1):300-312, 1999.
James Berger. The case for objective Bayesian analysis. Bayesian Analysis, 1(3). 2006.
Jose M Bernardo. Reference posterior distributions for bayesian inference. Journal of the Royal Statistical Society Series B: Statisticul Methodology, 1979.
Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estima- tion. In Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc., 2020.
Kenier Castillo and Ion Zaballa. On formula of thompson and mcenteggert for the adjugate matrix. Linear Algebra and its Applications, 2022.
Anthony L. Caterini, Gabriel
Geoff Pleiss, and John Patrick Cunningham. Rectan- gular flows for manifold learning. Advances in Neural Information Processing Systems, 2021.
Yasuko Chikuse. Statistics on Special Manifolds. Lecture Notes in Statistics. Springer New York, 2012. ISBN 9780387215402.
Persi Diaconis, Susan Holmes, and Mehrdad Shahshahani. Sampling from a manifold, 2012.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real NVP. In International Conference Learning Representations, 2017.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Kyriakos Flouris and Ender Konukoglu. Canonical normalizing flows for manifold learning. In Thirty-sevnnth Conference on Neural Information Processing Systems, 2023.
Mevlana C. Gemici, Danilo Rezende, and Shakir Mohamed. Normalizing flows riemannian manifolds, 2016.
Chin-Wa Wei Huang. David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In Proceedings of the 35th International Conference on Machine Learning, pages 2078- 2087. PMLR, 2018.
Priyank Jaini, Kira A. Selby, and Yaoliang Yu. Sum-of-squares polynomial flow. In Proceedings of the 36th International Conference on Machine Learning, pages 3009-3018, PMLR, 2019.
Edwin T Jaynes. Probability theory: The logic of science. Cambridge university press, 2003.
Harold Jeffreys. The theory of probability. Oxford University Press. 1961.
Martin Jiskra, Lars-Eris
Heambürger-Boaviidee Marle-Maëlle Desgranges, Mariia V Petrova, Aurélie Dufour, Beatriz Ferreira- Araujo, Jérémy Masbou, Jérôme Chmeleff, Melilotus Thyssen, David Point, et al. Mercury stable isotopes constrain atmospheric sources to the ocean. Nature, 597 (7878)6678-63) -882, 2021.

---

Published as a conference paper at ICLR 2025
Robert E Kass. The geometry of asymptotic inference. Statistical Science, pages 188-219, 1989.
Robert E Kass and Larry Wasserman. The selection of prior distributions by formal rules. Journal of the American statistical Association, 14435)):333-17000 1996.
Sebastian Mathias Keller, Maxim Samarin, Fabricio Arend Torres, Mario Wieser, and Volker Roth. Learning extremal representations with deep archetypal analysis. International journal of com- puter vision, 129:805-820, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, 2016.
Oliver Knill. Cauchy-binet for pseudo-determinants. Linear Algebra and its Applications, 459: 522-547, 2014.
Ivan Kobyzev, Simon J.D. Prince, and Marcus A. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, page 3964-3979, 2021.
Konik Kothari, AmirEhsan Khorashadizadeh, Maarten de Hoop, and Ivan Dokmanié. Trumpets: In- jective flows for inference and inverse problems. In Uncertainty in Artificial Intelligence. PMLR, 2021.
Abhishek Kumar, Ben Poole, and Kevin Murphy. Regularized autoencoders via relaxed injective probability flow. In Proceedings of the Twenty Third International Conference Artificial Intel- ligence and Statistics. PMLR, 2020.
Huadong Liao and Jiawei He. Jacobian determinant normalizing flows, 2021.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational Bayesian neural networks. In Proceedings of the 34th International Conference Machine Learning, pages 2218-2227. PMLR, 2017.
Harry M Markowitz and G Peter Todd. Mean-variance analysis in portfolio choice and capital markets, volume 66. John Wiley & Sons, 2000.
Emile Mathieu and Maximilian Nickel. Riemannian continuous normalizing flows. Advances in Neural Information Processing Systems, volume 33. Curran Associates, Inc., 2020.
Angel Muleshkov and Tan Nguyen. Easy proof of the jacobian for the n-dimensional polar coordi- nates. Pi Mu Epsilon Journal, 14(4):269-273, 2016.
James R. Munkres and Michael Spivak. Calculus on manifolds. 1965. URL https://api. Semanticscho
Marcello Massimo Negri, Fabricio Arend Torres, and Volker Roth. Conditional matrix flows for gaussian graphical models. In Phirty-seventh Conference on Neural Information Processing Sys- lems, 2023.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density es- timation. Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lak- shminarayanan. Normalizing flows for probabilistic modeling and inference. Journal of Machine Learning Research, 22(57): 1-44, 2021.
Trevor Park and George Casella. The bayesian lasso. Journal the american statistical association, 103(482), 2008
Xavier Pennec. Intrinsic statistics on riemannian manifolds: Basic tools for geometric measure- ments. Journal of Mathematical Imaging and Vision, 25:127-154 2006.
12

---

Published as a conference paper at ICLR 2025
Yura Perugachi-Diaz. Jakub Tomezak, and Sandjai Bhulai. Invertible densenets with concatenated lipswish. In Advances in Neural Information Processing Systems. Curran Associates, Inc., 2021.
Donald L. Phillips. Converting isotope values to diet composition: the use of mixing models. Jour- nal of Mammalogy, pages 342-352, 2012.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. Proceedings of the 32nd International Conference on Machine Learning, pages 1530-1538. PMLR, 2015.
Danilo Jimenez Rezende, George Papamakarios, Sebastien Racaniere, Michael Albergo, Gurtej Kanwar, Phiala Shanahan, and Kyle Cranmer. Normalizing flows on tori and spheres. In Pro- ceedings of the 37th International Conference Machine Learning. PMLR, 2020.
Brendan Leigh Ross and Jesse C Cresswell. Tractable density estimation learned manifolds with conformal embedding flows. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors. Advances Neural Information Processing Systems, 2021.
W. Rudin. Real and Complex Analysis.
Mathematics series. McGraw-Hill, 1987. ISBN
9780071002769
Sohan Seth and Manuel JA Eugster. 85-113, 2016.
Probabilistic archetypal analysis. Machine learning, 102:
Vadim Sokolov and Michael Polson. Strategic bayesian asset allocation, 2019.
Peter Sorrenson, Felix Draxler, Armand Rousselot, Sander Hummerich, Lea Zimmermann, and Ullrich Koethe. Lifting architectural constraints of injective flows. In The Twelfth International Conference on Learning Representations, 2024.
Vincent Stimper, David Liu, Andrew Campbell, Vincent Berenz, Lukas Ryll, Bernhard Schölkopf, and José Miguel Hernández-Lobato, normflows: A pytorch package for normalizing flows. Jour- nal of Open Source Software, 2023.
Brian C. Stock, Andrew L. Jackson, Eric J. Ward, Andrew C. Parnell, Donald L. Phillips, and Brice X. Semmens. Analyzing mixing systems using new generation of bayesian tracer mixing models. PeerJ, 6, 2018.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B (Methodological)) 58:267-288, 1996.
Xueyong Tu and Bin Li. Robust portfolio selection with smart return prediction. Economic Mod- elling, 135(C), 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
The Appendix is organized in six parts. Subsection A.1 we define the generalized spherical coordinate system and define the transformation from spherical to Cartesian coordinates. In Sub- section A.2 and A.3 we provide some auxiliary theorems and Lemmas that are used the main proof. In Subsection A.4 we provide the full proof of Theorem TT In Subsection [A.3] we provide some details about the implementation of the proposed injective flows and we make some further comments about the associated computational cost. Finally, in Subsection A.6 include further plots and implementation details about the experiments.
A.1 GENERALIZED SPHERICAL COORDINATES
Definition We define the d-spherical coordinate system generalization of the spherical coordinate system for d-dimensional Euclidean spaces. Such coordinate system is defined with and one radius E Ryo, where 8, E [0, π| for i < d and 0--1 [0, 2T). We further define transformation T + a. that maps spherical coor- dinates v, = 10.,... O2-1, r|" Cartesian coordinates Be [.,,...,
d angles .1.....
cos 0
22 sin 01 cos 02
Fg = r sin 0 sin ..... ......sin sin 0d- cos Od-1 Fd rin 02 sin 02 ... sin 0a sin 0a-1
We denote with Up-1 x Rzo the domain definition for d-spherical coordinate system, where Ud--1 = [0, m] x [0, 2x..
A.2 JACOBIAN DETERMINANT FOR ARBITRARY INJECTIVE FLOWS
Remark 1. Let mm R + R and Ta IRd + Rd be arbitrary bijective transformation and let T m-- Ih" H be an injective transformation with < d. The transformation
T = Too T mm T is also injective and its Jacobian determinant factorizes as
det JT det JT v doct ((JT.TT..o) JT JT
Proof. The injectivity T trivial since it is by definition a composition of injective functions.
Since T is injective, its Jacobian matrix Jr e Id is not squared and we cannot use the usual property of bijections in Eq. 1. Instead, use the definition of Jacobian determinant for injective functions in Eq. 2
det JT V det (IFT)) = vact (JJraJn /
(T JT (Taaa))))))))
where Jm mm JT Rdxm and JT, E Raxd We show that we factor the Jacobian determinant of T m. i.c. the bijection that precedes the dimensional inflation step with T Tn-dd To do we use the property that for square matrices A, B det(AB) det det and that det A = det AT:
det JT Vdet [JFm. FF
Vde If. del (IF Jt Tursa Sf.
JT sa
del del
(9)
det det det (UTTaTmmma) (JTanl
det JTm det de
14

---

Published as a conference paper at ICLR 2025
A.3 AUXILIARY THEOREMS: ADJUGATE MATRIX AND PSEUDO- PSEUDO-DETERMIN DETERMINANT
Theorem 2. (Castillo and Zaballa 2022) Let A E Raxd and let X E R be an eigenvalue of A. Let w,w E Ra be a right and a left eigenvector, respectively, of A X. Then
w" w"v adj A) = P'A())ow..
(10)
where (AA is the derivative of the characteristic polynomial PA(A) det(ATd - -A). Lemma 1. Consider the special case where A &dddd and rank = or, other words, the
nullspace of A is one dimensional. Then
det a (A) w v vw
adj(A)
where det* the pseudo-determinee
Proof. Since rank A = d - 1, then there exists one zero eigenvalue. For X O Theorem reduces to w" v aaj((-A A) (')ow''"" We can now use the following property of the adju- gate matrix: adj(cA)
dl adj(() for any scalar As particular case, for c -1 we have adj(-A) = (-) (A). Therefore, we obtain that w e aA)(A) (O) Now, the pseudo-determinant is equal to the smallest non-zero coefficient of the characteristic polynomial p()) det(AL. (A)aAll))KKnnn Knnill 2014). If we expand the definition we obtain p(A) = (-) -) All)
(0)uw*
pod" + (-)p) ) dd11 +
(-) A= - + (-)"pa (see Proposition 2, 8. in Knill (0014). Since
A has rank d 1, Pa
0 and the smallest non-zero coefficient is Pd-1- Finally, note that (-)² det (A).
P'A (0)
Lemma 2. Consider the special case where A Rddd and rank A d - Then,
Tr(adj(A)) det (A).
(12)
Proof. We take the trace of the left and right-hand side of Eq. 11 We get Tr(adj(A)) (A) Tr(ow?") (A) Tr(wF'v) det (A). In the first equality we used the linearity of w"' the trace and factored out the constants det (AA) and wh v. Lastly, we used the cyclic property of the trace Tr(w" u) Tr(ww").
w'v
A.4 PROOF OF THEOREM
Theorem Let T Ts-se oT, as Figure where To E Rd-1 + Ug-1 is any diffeomorphism to d-spherical angles, T, 0 Up + (0, (0)] E UØ x Rs0 with
r(.) 0 E - - E Rzo being differentiable, and Te O, (O) E U x R + x Rd
the d-spherical to Cartesian transformation (see Definition 2 in Appendix A.7
Then, the Jucobian determinant of the full transformation T is equal to
det (JT) = det (JTo) det (JT,.e) I1 yll2.
(4)
where y (or 1 Relevantly, det JT can be computed exactly and efficiently in Odd)).
Proof. We start the proof by noting that the transformation T == To oT, To is injective because , can be easily seen by noting that 0 # 0' = [0, r()] |0', "(()), independently of Since the Jacobian matrix J7 e ddxd-1 is not squared, we cannot use the usual property of bijections in Eq. (T). Instead, we use the definition of Jacobian determinant for injective functions Eq. (
To and Te-te are bijective and T, is injective. The injectivity of T,,
det JT V /det (JFJT) Vdet (TT) " (JTTo
(13)
15

---

Published as a conference paper at ICLR 2025
where Jr, Rdxd-1 and JT, Raxd According to Remark | we factor out the Jacobian determinant of To. i.e the bijection that precedes the dimensional inflation step with T.
det Jy = det Jr. det J = det Jr, det (JJr...T.) (JT...T.))
(14)
The term det JT is the standard Jacobian determinant for bijective layers and can be computed efficiently. We are then left to compute det o We now consider the matrix JT, = [JT. Oaxi] E Igdxd and substitute the determinant with the pseudo-determinant:
Valot det (.-J..)-//oa (J....
det JT oT, =
uT,
(15)
where J == JT
RRdxd With det. we denote the pseudo-dettrmiinnt,, which is defined as the product of all non-zero eigenvalues. The second equality follows from the fact that J7, J JT. and J.F JJJ have the same spectrum up coincides with the pseudo-determinnnt of the latter (by definition). To see that they share the same spectrum up one zero eigenvalue, consider the explicit structure of the matrix product:
eigenvalues, so the determinant of the former
JTJ
Jf. 01xd-1
0d-1x1 0
(16)
The rest of the proof is based the key observation that Jr, has rank d - or, equivalently, that its null space is one-dimensional. As a consequence, we can use Lemma 2 and re-write the pseudo-determinantii in terms of the trace of the adjugate matrix:
det + (J.. (J..JrTT)) Tr adj (J+, (J..)
Tr adj (JF,) adj (J*) adj (Jr.))
(17)
det (J*) Tr adj (JT.) (J*) adj (J+)))
In the second equality we used the property that adj( (BB) adj( B) adj(A) for any A, B E Rdxd which easily generalizes to adj(ABC) = adj(C) (B) ad) Lastly, A is invertible, (A) det(A)AA In this case J J JT. has full rank and is thus invertible. Since the trace is linear operator we can take out det(J"). which is constant.
Since Jy_ has rank d - 1, its nullspace is dimensional and span the entire nullspace. The same holds for Jy,,, equivalently for the left nullspace of Jr., and can pick y Rd Jf. y O. We easily compute and y by looking at the structure of J7,-
pick Z E Rd Jr, 0 to
10
1
0
1
0
002
Jr.
:
(18)
0r
O
88
Or
Dr
Or 00
09;
00g
We now make of Lemma
for JT., which gives
adj(Jr.) det (JT.((T.) mnj
(19)
yT
We can now substitute Eq. (19 in Eq.
det+ (J. J Jr.) det (J*) det (Jr:)2
(y Tr xzy (J₂) (z²T(1²) yx")
det (JT, 2rT Tr ( (v"(
(20)
det (JZ JT,
(yTx)²
det (JT..e)) ( (1((F
v1l2
16

---

Published as a conference paper at ICLR 2025
In the first equality we used the fact that adj(AT) adj(A)) and we factored out det (Jr, ) and (y" (y"z)² x)², which are constants. In the second equality we used the cyclic property of the trace and factored out x²ax. x. Lastly, we substituted the numerical values det (JT,) = 1, y'x = 1 and x" 1.
We can now analyze the time complexity required to evaluate Eq. (4). The Jacobian determinant for spherical to Cartesian coordinates is known (Muleshkov and Nguyen 2016
d-2
det Ja+c (-)a-1
II k=1
sin d k-1 Ok
(21)
and can be computed efficiently in ((d) time. Therefore, we only need to show that also w (JT -1 y can be computed efficiently. Solving the full linear system would require complexity of (dd). However, we can exploit the almost-triangular structure of
da
dzg
ax
az
001
a01
281
301 0ra 200g
20g
80g
(22)
36a-1
884-1
Oar
d22 02
Or
Dr
Dr
0r
to solve the linear system in Odd)) One possibility is to perform one step of Gaussian elimination, which requires (d²),, and make the linear system triangular. The resulting triangular system can be
solved in (dd). Note that we can compute JT,... very efficiently and analytically (see Eq. (24).
without requiring autograd computations. Overall, the determinant of the full transformation
can
be obtained as
det JT det JT, det (I
(23)
and can be computed efficiently (dd).
A.5 IMPLEMENTATION DETAILS
Implementation of injective flows for star-like manifolds We provide some details about the implementation of the proposed injective flows and particularly for star-like manifolds in Cartesian We implement the layers in three steps:
coordinates in Figure 2
bijective layers T and Te. The first bijection T : + z' consists of arbitrary (condi- tional) bijective layers conditioned on the parameter The conditioning realized with an expressive Residual network. Then, To z' + O maps the transformed z' into spheri- cal angles 0 E Ug This last transformation is also a bijection that can be implemented with an element wise non-linear activation like Sigmoid (hence diagonal Jacobian). Oth- erwise, one could use a base distribution which is already defined on the d - 1 spherical angles and use a bijective transformation that transforms O within their domain U as T Teic 0 Uy-1 + 0' E U/-11 In short, any bijective layers followed by clement-wiss non linear function that maps to spherical angles could be used. We use the circular spline layers proposed by |ezende et al. (2020) because they allow to nicely integrate the bound- ary conditions arising from the use of spherical coordinates. These layers are based on the neural spline layers proposed in Durkan et (2019), which consist of a combination of K segments where each segment is a simple rational-quadratic function. The flexibility of the layers increase with K. This way the transformation can be designed such that is monotonically increasing (hence invertible) and such that it fulfills given boundary condi- tions. The main difference with Durkan al. (2019) that circular splines require periodic boundary conditions in order to enforce continuity of the density at the houndary. This way we can define bijections from [0, 2x] to itself. As a consequence, circular splines require the base distribution to be defined on U In practice, we use the distribution of spher- ical angles, which results in uniform points the d dimensional sphere, and can be implemented efficiently. We use the implementation circular layers provided in SSimper et al. (2023). Such construction scales well with the dimensions.
17

---

Published as a conference paper at ICLR 2025
injective layer T. The injective step T, 0 (0,r only consists in padding the spherical angles with some specified radius function r(0). The specific expression for the radius function depends the manifold considered and is detailed in Eq. 25 and Eq. (26) for the lp (pseudo-) norm ball and for the probabilistic simplex C³, respectively. In variational inference settings T, is not learnable transformation. In maximum likelihood settings trained on samples, if we assume the samples were generated from a d - star-like manifold, r(0) can be implemented with neural network and made learnable. This would allow to learn the manifold and would provide with a very practical global parametrization.
bijective layer Te-c: the bijective layer Tg-4 10. (0)) + æ simply implements the spherical to Cartesian transformation in Eq. (6). which is a bijection and can be imple- mented efficiently. Ty-re is not a trainable transformation.
For the implementation we rely on the (conditional) normalizing flow library FlowConductor which was introduced in Negri et al. (202) and Arend Torres et al. (2024).
Efficient implementation of the Jacobian of spherical to Cartesian transformation In order to compute the determinant in Eq. 4 we need compute the Jacobian determinant of the transforma- tion from spherical to Cartesian coordinates J By looking at the definition of the coordinate transformation in Eq. 6 we can easily derive the following expression:
781
TC182 ... Sd-2Cd-1 0
TC ... 8d-28d-1
0
rS182
TS102
Sd-2Cd11
TS102 $d-2$d-1
T
0
(24)
0
TS182
9d 8d-28d-1
8182
$4 20--1
$102
$182 2d-2
8182
2d
where we used the shorthand 8; = sin 0g and e; cos 0. This allows to compute J extremely efficiently without requiring to use autograd computations and results in significant speed up.
Parametrization of 1p (pseudo-) norm balls Here we show how to parametrize the 1p (pseudo-) norm balls in spherical coordinates. Let the Ip (pseudo-) norm of E Rd be defined ||>||p (|=1|P +.... ..... |xaa 2//p with p >> We consider now the manifold defined by ||>||p for some k E 18 If we write æ in spherical coordinates according to Eq. (6), we can take the radius outside of the norm and express it as a function of the d spherical angles as:
r(01...,00-11
(25)
PP +
d-1 II sin On R=1
1") 1/p
cos 0g P
Σ cos 02 II k=
sin P
II
We can use this expression to parametrize the 1p norm balls with the proposed injective flows. Sim- ilarly, we can also parametrize the probabilistic simplex Cd.. To see this consider the norm ball ||œ||| | + ... |ra]]. If restrict the domain to the positive quadrant æ Rd and set the
norm to 1, the resulting manifold defined as ||æ:|1 = 21 +... + Ed = 1 and coincides with Cd. The radius is then parametrized by
with 0, E [0, (26)
d-1
cos 01
Σ cos 0g II sin Og
II 0g
where the constraint on the angles enforces æ R2-- Note that straightforward analytically Br in Eq. (26). This makes the computation of y in derive the expression for the partial derivatives 20,
Eq. more efficient than computing the gradients with autograd and results in a speed up.
18

---

Published as a conference paper at ICLR 2025
laplace
root laplace
square laplace
13
25
Figure 8: MAP solution for Laplace prior, root laplace and square laplace, which are all monotonic transformation of the Laplace distribution. The MAP solution paths coincide.
A.6 APPLICATIONS: FURTHER DETAILS
A.6.1 ARCHITECTURE
We use two different architectures. One is for standard NFs that we use for the subjective penal- ized likelihood regression problem. The other architecture is the injective flow that is used for the objective Bayes version of the regression problem and the portfolio diversification application.
Standard NF It consists of normal distribution base distribution. Then permutation transformation, sum of Sigmoids layer (Negri et al., (023) and an activation norm. The sum of Sigmoid layer consists each of 30 individual Sigmoid functions three blocks.
blocks of
Injective flows The base distribution is either the probabilistic simplex or the complete || = depending on the application. We follow this with again 5 layers of the circular bijective layers (Rezende et al. 2020), each consisting of three blocks with 8 bins. At the end these values are mapped to Cartesian coordinates with the proposed dimensionality inflation step.
A.6.2 TRAINING
Both the standard NFs and the injective flows are trained by minimizing the reverse KL divergence with respect to the (unnormalized) target density p(æ)):
arg min E Eazggo ge (I)) p(x))
40 (x) = arg min (L(gg(=)))le ace
log
(27)
AE
We optimize the reverse KL divergence using Adam (Kingma and Ba fault parameters. Notably, all trained flows converged a matter minutes on a standard consumer- grade GPU (RTX2080Ti in specific case).
2017) optimizer with de-
A.6.3 PENALIZED LIKELIHOOD REGRESSION
In the next paragraph we provide further details on the experiment introduced in Section 5.3 which involves the penalized likelihood model defined in Eq. (5)
Synthetic dataset creation The synthetic regression dataset is created by sampling X from a 5 dimensional Wishart distribution W5(7, /). The response variable y then created by X*3 + where 3 is standard normal distributed and is normal distributed with zero mean and standard deviation 4.0.
Subjective Bayes
The subjective Bayes relies prior p(3[A). The Laplace prior is given by II, exp(
(1ap(3)
(28)
The two other test priors are P., (3 A) x Ptap Ptap 2 and p. (B x Ptop (8)2)) Any monotonic transformation may change the A-axis but leave the MAP solution path unchanged. This can be
19

---

Published as a conference paper at ICLR 2025
Subjective Dayes
Objective Bayes
[[0]]:
10
Figure 9: Solution paths for subjective prior as a function of X, and objective priors, as a function of the norm [|]| 1- Below report the solution paths in the MAP limit.
seen in Figure where show the MAP solution path for different subjective priors. For this visualization we reparametrize the axis such that the A-axis is transformed into []@]|l-axis. This makes clear that the solution paths equivalent.
The objective Bayes approach circumvents the definition of p(B))). The flow is directly defined on the manifolds coinciding with the contour lines of A() As such, samples from the posterior all share a chosen norm value ||,31| k. Figure highlights the different parametrizations of the subjective and objective approach.
Objective Bayes
A.7 APPROXIMATE VS EXACT JACOBIAN: SAMPLE QUALITY AND RUNTIME
A.7.1 BAYESIAN MIXING MODEL
Comparison with MCMC sampler We compare the proposed injective flows with an ad-hoc MCMC sampler in the conjugate case (Dirichlet prior and multinomial likelihood). In this setting we know the posterior analytically and we can compare it to both the MCMC sampler and the proposed injective flow. In this experiment the injective flow is trained to minimize the reverse KL divergence with the unnormalized target posterior (prior times likelihood). We compare the methods for increasing dimensionality (d = 15, 30, 50) and in Figure we show 95% credibility intervals of the posterior. Results show that (i) the proposed injective flow correctly estimates the posterior distribution for all dimensions and (ii) it outperforms the MCMC sampler already in low dimensions (d 50). Interestingly, this experiment also shows that our approach is able to learn very sparse solutions characterized by multiple modes.
Details about the MCMC sampler In Bayesian mixing models the posterior lives the simplex, which known to be challenging for MCMC samplers (Altmann et al., 2014 Baker et al. 2018) because of the constraint that the sampled vector must be positive and up to one. Designing proposals close to the boundary is particularly difficult (Baker et al. 2018). For sparse distributions, where mass is concentrated at the boundaries, this is especially noticeable and explains why the MCMC sampler struggles in high dimensions for sparse solutions. What makes a fair comparison difficult is that various existing samplers (Altmann al. 2014 Baker et al. 2018) suited for
20

---

Published as a conference paper at ICLR 2025
Jin 101
100m1031
(11-100))
energ 18
sinkhom
dimension
dimension (b)
(a)
CMB (enucl appros (0m0)
(000100)
runtime
dimension
(c)
Figure 10: Additional comparison between two identical models trained with the exact Jacobian, propose, according to Eq. 4 and with the Hutchinson trace estimator, commonly used in injective papers. The experimental setup is described in Section 5.1 We compare samples of the two models with samples from the true distribution of samples used in the Hutchinson trace estimator (n = 1, 10, 50, 100). Note that the number of samples is upper bounded by the dimensionality of the problem. We quantify the similarity of the samples with the samples from the true distribution with (a) the Energy distance MMD ("energy") and (b) the Sinkhorn divergence ("sinkhorn"). The Energy distance MMD computed using the kernel ||ar yll2- The Sinkhorn divergence interpolates between Wasserstein (blur=0) and kernel (blur=o)) distances and we used the default value blur=0.05. Below (c) we show the runtime of our exact model compared to the approximate one as a function of increasing number of samples.
function of the dimension and of the number
specific families of target distributions. For ideal performance we would need to change the entire sampler depending on the target, while we can use our flow without any fine-tuning.
We compare to a Metropolis Hastings sampler that can handle mixtures of Gaussians targets well in a non-sparse setting. It uses proposals that are by construction on the manifold by sampling from down-scaled Dirichlet distribution centered on the current state. Despite trying different proposals we did not observe any relevant improvement. As shown, this sampler however struggles when significant sparseness is present. Our sampler uses 1000 chains each with 100'000 samples. In contrast, the proposed injective flow did not require any fine-tuning and learnt sparse solutions. For a fair comparison, we set the runtime of the flow to match that the MCMC samplers and used roughly similar memory footprint (=20%). Lastly, checked that running the sampler for significantly longer time (5x) did not achieve the performance of the flow.
Portfolio optimization In portfolio optimization the cumulative return often of interest. Fig- ure 12 shows the effect of the different priors on the cumulative return. The sparser priors lead to a slightly wider distribution of the return. In this example, this leads the target index being a closely matched by some of the posterior samples, where the samples of the uniform prior seem to be further away from the target index in some parts of the time interval. The bottom row of the Figure further shows the sampled sparsity patterns. These show that the sparse priors can lead to significantly different mixtures with similar data fitting quality.
21

---

Published as a conference paper at ICLR 2025
15
d = 30
= 50
IEOTIC
valuee
28
15
90
confficient
coefficient
Figure 11: 95% posterior credibility intervals for Bayesian model with multinomial likelihood and Dirichlet prior. We compare the posterior obtained with samples from an ad-hoc MCMC sampler ("meme"), samples from the approximate posterior modeled by the proposed injective flow ("flow"") and with the analytical posterior ("gt"). We evaluate the samplers for increasing dimensionality of the problem 15, 30,50 50.
Uniform
Dirichlet
L, pseudo- norm norm
cesest samp
5 clowest samples average 92% C.I. index
clesest samples
average
95-8
ret Index
Index
520
150
200
time
time
time
samples
Figure 12: Cumulative return as function of time. We plot 95% posterior C.I. with the uniform prior, Dirichlet prior and norm prior. We also plot the 5 samples that the closest the ground truth cumulative return. In the bottom panel we visualize the weight samples as a heatmap to high- light sparsity patterns.
22


Reviews:
[{"review_id": "qRER2NN7Tl", "reviewer": "Reviewer_pwAr", "summary": "The authors propose a normalizing flow technique for a specific class of manifolds in which the computation of the Jacobian determinant is both exact and efficient, while related works rely on approximations or predefined manifolds. This approach is particularly relevant for variational inference problems where the manifold is known, and the goal is to approximate a density on it. The effectiveness of the method is demonstrated through variational inference on well-designed problems that meet the manifold constraints.", "questions": "Q1. I believe Fig. 2 does not significantly aid in understanding the proposed approach, and a more intuitive figure could better illustrate the concept of the injective flow graphically. Additionally, moving the proof sketch to the appendix could free up space for a more intuitive figure(s), details on the mappings considered (e.g., as in Appendix A.5), and discussions on variational inference, maximum likelihood, and additional experiments.\n\nQ2. The proposed approach appears to rely on the circular bijective layers of Rezende et al. (2020); a brief description in the appendix would be helpful. How does this approach scale in higher dimensions, and is this critical to the proposed approach?\n\nQ3. Regarding experiment 5.1, is it possible that the density effectively modeled through the spherical flows (circular bijective layers) used for the transformation $\\mathcal{T}_{\\theta}$?\n\nQ4. It would be interesting to include some (synthetic 2D or 3D) experiments where the manifold is more complicated, such as variational inference problems with a posterior concentrated on a star-shaped manifold, where $r(\\theta)$ is potentially a learnable function.\n\nQ5. Minor: What is meant by the density will be expressed in Cartesian coordinates (line 160)? Does this simply mean that the push-forward samples $x$ are just points in the ambient space?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The general idea of focusing on this class of manifolds, which has desired properties, is both simple and interesting, and the technical aspects appear solid.\n- Based on the experiments conducted, the proposed model supports the claims.\n- The paper is generally well-written and well-structured (see Questions for some suggestions).", "weaknesses": "- Even if this class of manifolds is useful for density modeling in the considered scenarios, it becomes somewhat restrictive if the $r(\\theta)$ is not trainable.\n- Further experiments and/or comparisons with related works, such as in a maximum likelihood setting, could have been beneficial for demonstrating both the effectiveness and efficiency of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U1v6Xe8Lop", "reviewer": "Reviewer_HG9C", "summary": "A method is presented for efficiently computing normalizing flows on star-shaped d-manifolds embedded in (d+1)-dimensional Euclidean space. Using a decomposition based on spherical coordinates, the Jacobian determinant can be analytically computed in O(d^2) time. Numerical examples are given to show that the method outperforms the (d+1)-dimensional default sampler.", "questions": "1. The asymptotic efficiency of the method is shown to be O(d^2), it could be interested to look at the rate in more detail. Specifically, I am curious about the constant factors in the rate as compared to the normal normalizing flow methods. This can be done either analytically or empirically with comparisons of runtimes.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "1. Paper is well written and readable.\n2. The method is clearly explained and understandable, and it improves over sampling in the ambient dimension.\n3. Claims are supported with proof and numerical evidence.", "weaknesses": "1. The method only applies to star-shaped manifolds embedded in one dimension higher. The authors acknowledge the limitation that their proof cannot be extended to manifolds with higher co-dimension. This is the main concern with the method - since this method only applies to a very specific class of co-dimension 1 submanifolds, I wonder about its applicability to problems of general interest. Since the method applies to a very specific class of manifolds instead of a more general class, the benefits for this specific class need to be greater to warrant acceptance into ICLR, in my opinion\n2. The numerical examples seem limited (as a result of 1). To justify this approach, more examples of star-shaped manifolds and real-world applications should be presented and/or the benefit of the method over normal normalizing flows should be significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1Yf8TDAIvV", "reviewer": "Reviewer_xkKJ", "summary": "This paper addresses the substantial computational cost associated with normalizing flows when applied to densities on manifolds. The proposed method is specifically tailored for star-like manifolds and efficiently computes the Jacobian determinant, outperforming standard normalizing flows. Furthermore, the paper introduces Bayesian methods for penalized likelihood models, where penalty level-sets are regarded as start-like manifolds, and variational inference methods for probabilistic mixture models.", "questions": "- In Theorem 1, it is stated that $\\mathcal T_\\theta$ can be an arbitrary diffeomorphism in $\\mathbb{R}^{d-1}$. However, to the best of my understanding of normalizing flows, the evaluation of its Jacobian could be as computationally expensive as $O(d^2)$ if not specifically designed. Do you require any constraints on $\\mathcal{T}_\\theta$?\n- How do the properties of the manifold influence the training process of the flow model?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed injective flow method for star-like manifolds significantly reduces the standard complexity of $O(d^3)$ to $O(d^2)$.\n- The injective flow approach appears to be both intriguing and novel, and the proof presented is sound.\n- The paper outlines several potential applications, including the objective Bayesian approach and posterior inference in probabilistic mixing models.", "weaknesses": "- Experiments are relatively limited, and no baselines are employed for comparison apart from the MCMC sampler in FIgure 9. I would suggest state-of-the-art normalizing flow methods and possibly other generative models for the baselines.\n- The purported efficiency enhancement from $O(d^3)$ to $O(d^2)$ is not substantiated by experimental results. The authors could include runtime comparisons between their method and standard normalizing flows for increasing dimensionality $d$. This would provide empirical evidence for the claimed computational improvement.\n- A minor issue: the caption of Figure 6 should include the correct probability distribution: $p(\\boldsymbol\\pi)|\\mathcal{D}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U41fiew4MI", "reviewer": "Reviewer_xKG1", "summary": "The authors propose an embedding of arbitrary d-1 dimensional star-like manifolds (i.e., deformations of the unit sphere) into the $\\mathbb{R}^d$ which allows the computationally efficient evaluation of its determinant. They apply this embedding for constructing normalizing flows on such manifolds, by using an injective mapping from a d-1 dimensional latent space to the manifold. The proposed model is applied to several examples.", "questions": "I outlined my questions and suggestions in the \"weaknesses\" part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is easily readable and written in a clear way. As far as I know the approach is new, implementable and computationally tractable. In contrast to prior papers it covers a whole class of manifolds.", "weaknesses": "My main critic on the paper is the missing comparison to existing injective flows. The authors emphasize at several points of the paper that it is important to have density evaluations of the model which are i) exact and ii) have improved complexity. As outlined in the literature part of the authors, there exist plenty of methods meeting one of the properties i) or ii). To validate the importance of meeting both at the same time, I would suggest two experiments:\n\ni) Run an experiments to fit a not too simple distribution on a star-like manifold (maybe something like Fig 3b) and train an injective flow once with naive O(d^3) evaluation of the change-of-variables formula and once with the formula proposed in Thm 1. Then, create a plot of execution time vs dimension.\n\nii) Run an experiment in a similar setup for variational inference. Train an injective flow with exact determinant (as proposed) and some approximate determinant (e.g., one of the models which are cited in the related-work part) and compare the quality of generated samples (e.g. in Wasserstein/MMD or a similar two-sample comparison of distributions). Following the authors claims, the injective flow with exact determinant should lead to more accurate results than the approximate.\n\nEven though the existing numerical experiments provide some proof-of-concept, I think that the paper could benefit a lot by evaluating more the proposed architecture itself in terms of speed and accuracy instead of using it for specific applications. The comparison with the completely different MCMC samplers looks a bit out of place since it is a completely different model which is known not to scale well along the dimensions (which is also observed in Fig 9).\n\nAdditionally, I would like to point the authors to the papers [1,2]. [1] considers a specific architectures which makes the evaluation of the necessary determinant cheap and easy, [2] which trains the normalizing flows with a VAE loss.\n\nFrom a technical side, I am bit confused since the change from spherical to Cartesian coordinates is not a bijection and definitely not a diffeomorphism. The vectors $(0,\\theta_2,...,\\theta_{d-1},1)$ map all to the same point in the $\\mathbb{R}^d$ regardless of the choice of $\\theta_2,...,\\theta_{d-1}$. Certainly, one can exclude sets of measure zero from domain and image of this transformation such that it becomes bijective, but then still the determinant of this transform becomes arbitrary close to zero when $\\theta_1$ approaches zero such that the inverse is no longer a $C^1$-function (and hence the embedding not a $C^1$ diffeomorphism). I am not completely sure if this can cause limitations in actual applications. But the issue should be treated more thoroughly in the paper. In fact, it is easy to see that there exists no diffeomporphism from $\\mathbb{R}^{d-1}$ to a sphere (or any other star-like manifold) since those have different topologies (e.g., $\\mathbb{R}^{d-1}$ is contractible in contrast to star-like manifolds). A possible way to overcome such issue is the use of a mixture of several embeddings (for star-like manifolds two should be enough), see [3]. As the authors do not need too many geometrical properties of their embedding an \"we exclude zero sets\"-solution might be enough, but again, this issue should be discussed in the paper.\n\nIn summary, I think some numerical comparisons to existing works are necessary such that I think that the paper is below the acceptance threshhold, but I am willing to raise my score if the critics from above is addressed.\n\n\n[1] Kothari et al. \"Trumpets: Injective Flows for Inference and Inverse Problems\", UAI 2021\n\n[2] Chen et al. Learning flat latent manifolds with VAEs, ICML 2020\n\n[3] Alberti et al. \"Manifold Learning by Mixture Models of VAEs for Inverse Problems\", JMLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)