Para-CFlows: $C^k$-universal diffeomorphism approximators as superior neural surrogates
Authors: Junlong Lyu
Zhitang Chen
Chang Feng
Wenjing Cun
Shengyu Zhu
Yanhui Geng
ZHIJIE XU
Chen Yongwei
Venue: NeurIPS 2022 Accept (2022)
Keywords: Invertible Neural Networks
Bayesian Optimization
Universality

Abstract: Invertible neural networks based on Coupling Flows (CFlows) have various applications such as image synthesis and data compression. The approximation universality for CFlows is of paramount importance to ensure the model expressiveness. In this paper, we prove that CFlows}can approximate any diffeomorphism in $C^k$-norm if its layers can approximate certain single-coordinate transforms. Specifically, we derive that a composition of affine coupling layers and invertible linear transforms achieves this universality. Furthermore, in parametric cases where the diffeomorphism depends on some extra parameters, we prove the corresponding approximation theorems for parametric coupling flows named Para-CFlows. In practice, we apply Para-CFlows as a neural surrogate model in contextual Bayesian optimization tasks, to demonstrate its superiority over other neural surrogate models in terms of optimization performance and gradient approximations.

Para-CFlows: Ck-universal Diffeomorphism Approximators as Superior Neural Surrogates
Junlong Lyu" Huawei Noah's 's Ark Lab Hong Kong SAR, China Jynjunlong@huawell com
Zhitang Chen'
Chang Feng Huawei Noah's Ark Lab China cunvenjing@huavee com
Huawei Noah's ' Ark Lab Hong Kong SAR, China
chenzhitaang28huarei.co com
Wenjing Cun Huawei Noah's Ark Lab China fengchang1 Sernchanng@@uaweii..mm wwwwwei.wa..... com
Shengyu Zhu
Yanhui Geng
Huawei Noah's Ark Lab
Huawei Noah's Ark Lab Hong Kong SAR, China geng yanguyynnui @huawei.
China zhushengyu@nuawei.com com
Zhijie Xu Huawei China xuzhi jichawiieeh
Yongwei Chen
Huawei
China
ttteryongeeihhuawei.ccm
Abstract
Invertible neural networks based on Coupling Flows (CFlows) have various ap- plications such as image synthesis and data compression. The approximation universality for CFlows is of paramount importance to ensure the model expres- siveness. this paper, we prove that CFlows can approximate any diffeomor- phism in C*-moorm if its layers can approximate certain single-coordinate trans- forms. Specifically, we derive that a composition of affine coupling layers and invertible linear transforms achieves this universality. Furthermore, in parametric cases where the diffcomorphism depends on some extra parameters, we prove the corresponding approximation theorems for parametric coupling flows named Para-CFlows. In practice, we apply Para-CFlowt as a neural surrogate model contextual Bayesian optimization tasks, to demonstrate its superiority neural surrogate models in terms optimization performance and gradient approx- imations. Code will be avaliable at https::/g/t /ttpt//gi.eeeeeeeeeee www.ooiooooonneeeeeeeeeee www.nnddspo/oooooeoo / tree/master resear ch/ bo/ httpaarccflo..
other
Introduction
Invertible neural networks (INNs) such as coupling flows are firstly introduced as a class of generative models with tractable likelihood [14, 28, 44), and have shown their usefulness and powerfulness in various machine learning tasks such as inverse problems [2], probabilistic inference [32] and feature extraction [25] in recent years. With plenty of successful applications of INNs, one would wonder such type of models have the universal expressiveness. As most generative models mainly concern about the transform between distributions, existing works such as [22, 26] focused on the expressiveness from the distribution perspective. However, the expressiveness from the distribution perspective does not result in the expressiveness from the mapping perspective, there are large (or even infinite) number of functions mapping the given source to the given target v. In many
EEqual contribution.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

applications, knowing the distributional universality is not yet enough. One may be interested in knowing if the optimal transport [45], which finds emerging applications in many fields, e.g.. machine learning [35], wireless communication [33] and economics [18], can be approximated by invertible neural networks. Therefore, beyond the distributional universality, is also important to investigate the universality from the mapping perspective. As INNs are always invertible, it is natural to consider their approximation ability to diffcomorphisms. Diffeomorphism plays an important role in mathematics, physics and engineering domains with applications in fluid dynamics [16], wave propagation [20]. robot controls [37] etc. A natural question comes to the surface: can all diffeomorphisms be approximated by INNs? Besides, whether the INNs can approximate the derivatives in the meanwhile is also interesting and important, e.g., it provides theoretical guarantee for black box optimization tasks based surrogate models when gradients are utilized.
More importantly, in many real real-world problems, e.g., 3D Euclidean groups ODE systems and invertible PDE systems with time / as its parameter, diffeomorphisms are usually described as parametric type: a parametric diffeomorphism is a function f(y, Rm+dd -> IRd such that for any fixed parameter yu IR.", f(o. is diffeomorphism between Rd and IR is important to know if they can also be approximated by INNs with additional inputs of parameters. Existing works including [42] have been proposed to investigate the universality of INNs over diffeomorphisms. however it is not able to handle C* t=-ddfffcomophhimm when k = d + with d as the dimension of the space. this paper, we address this limitation and more importantly. inspired by the contextualization way proposed in [3], we generalize this structural theorem to the parametric diffeomorphism situation, by concatenating parameters with the variable to be transformed, forming a new subgroup over a higher dimensional space, and provide a complete proof. This structural theorem shows that any compact-supported diffeomorphisms can be decomposed into finitely many compositions of single- coordinate transforms, which we can easily prove to be C* C*-approximated by, e.g., affine coupling flows together with dimension augmentation ([15), [46]).
Lastly, we suggest using Para- CFlow, an affine coupling flow model with dimension augmentation and contextualization, as superior neural surrogate for black box optimization. The superiority lies in: 1) it uses INN architecture to preserve full rank of the action; 2) it has Ch-universality when the target system satisfies certain conditions. Specifically, 2) facilitates much more efficient surrogate and gradient-descent descent based black box optimization compared to other neural surrogates.
Overall, our contribution is three-fold: 1) We improve the results in [42] higher-order derivatives with a simpler proof, and give an upper bound on number of layers needed for approximating a certain diffeomorphism; 2) We generalize coupling flows to parametric coupling flows and prove their C* -universal approximation to parametric diffeomorphisms using a novel proving technique; 3) We propose a practical neural network structure of the parametric affine coupling flows and verify the advantage of using this for contextual Bayesian Optimization (BO) tasks.
2 Preliminary
Here we introduce some prior knowledge on (parametric) diffeomorphism, INNs and universality, as well as existing works on universality of INNs. As our theoretical results are motivated by and improve upon [42], some of our notations and preliminaries are adopted from them. In what follows, we always assume k. m, n, d E N+, where & represents the derivative order, and m, n, d represent dimensionalities for some Euclidean spaces. All vectors are supposed to be row vectors.
2.1 C -ddiiffcomooppiss groups on Rd
C* diffeomorphisms (group). Consider an invertible map f from Edd Rd. f is said be Ckk diffeomorphism, if has up to k-th continuous derivatives, and det Df(x)) * 0 for all æ. One can easily verify that, given f,g O* diffcomorphisms from Rdd to Rd, fo and f is still a C*. is a C diffeomorphism over IRd), diffeomorphism from IRd to Rdd If we denote Diff' (IR)) & f is
we see that Dift* (IRd) has a natural group structure with composition as its group operator.
Compactly supported C* d'-diifeomorrphisss. In real applications, finite data cannot cover the whole space Rdd Besides, existing approximation theories only guarantee the capability over some bounded compact set K. Therefore we consider compactly supported functions or diffeomorphisms.
A function f Ed + R is said to be compactly supported, if there exists a compact set K such

---

that f(æ) O, for all a & K. Similarly, a diffeomorphism f is said to be compactly supported, if there exists a compact set K such that f(z) æ, for all æ K, resulting in that all the non- diagonal components of Df are compactly supported. We define Dint (IR*) & (s E Dift" Dif""(R")) f compactly supported). Obviously Dift. (IR") is a subgroup of Diff" (R
Ck p-r-amrtatrcc diffeomorphisms. A parametric diffeomorphism is family of diffeomorphisms with some parameter a: {fa where for any given a. fa is a IR + gd diffeomorphism. Usually the parameter is described by some vector y E R" and fy varies continuously or smoothly w.r.t. y. We denote (y, æ) = (æ). We can embed it into a higher dimensional diffeomorphism Fly x) = [v. x)) One can verify directly that F E Diff (Rm+*) given that (1) (wo: 2) E Diff" (IR4) for any fixed yo: (2) f is k-th differentiable w.r.t. x)). Obviously Diff & {F Dift" (Rm++): Fly, a) (y, I (y, m)) with y E RM, x [Rd), whose elements keep the first coordinates and change the last d ones, a subgroup of Dift! (R=+d).
2.2 INNs based on parametric coupling flows
Here introduce the classical INNs and investigate the space generated by them. Invertible linear transforms. First, let define the invertible linear transforms (ILT):
1 _ { Lx" Ax= b", A GL., R²)
and the parametric case where y are parameters, here GL. includes all d d regular matrices.
ILT m, $ (f: L
xxT)
y Am
B
y x
+
6o A (L., b Rd B Rd
Coupling flows. We now define invertible coupling flows [34], specitic type of nonlinear transforms:
dd Rd + RR, (I x3x) - (agi ((
where o(r Rd-i Rd-i is diffeomorphism for each fixed Egi. Specifically, when b(rni x Fyi @ exp exp (a(as))) + ((rtss), it is the so-called affine coupling flow [13] and denote Pd,i,ort Pd for such ø. att are some functions with i output units, typically modeled with deep neural networks. @ represents the point-wise product. Egi =
æ> (I1+1,-.. .aa) for æ
(I1,--- .. .a). Similarly, for parametric cases, we have:
Pdiimo td m=
(y. Tgi, Eii) - (y. Dgi ply, Zgis T>;)))
(1)
and specifically, o0y, Rgg, 2>1) = Ez @ exp (a a ai):)) + (y. xgg)) for affine-type coupling flows and we denote Ddii,m,,,,t dd for such é.
Single-coordinate affine coupling flows (SACFs) denotes the flows with only the last coordinate changed. Let H. be set of functions from E2d to R. We define
H-SACF, A ( a, E Ha-1), and H-SACF e (0a,d-1,moot : a, I E Hm+d-1)-
Note that any multi-coordinates affine coupling flows can represented by finite composition of SACFs and invertible linear transforms, it suffices to just consider the universality of SACFs.
Invertible neural networks. Now let combine linear invertible transform layers and some coupling flow layers to construct our INNs. Let G be a set consisting of invertible coupling flows. We define the set of INNs based G and the parametric case:
G-INN, A {ga o W o ....o o WT : A E EG, E EITT2),
G-INN m. A {g. W. ................................ 9i W1 E N,9. E W,W, E (I.Tm.d).
When G contains 2 (or H-SACF m. in parametric cases), it is equivalent to replacing ILTa (or ILT,,,d by the symmetric group S, containing all the permutation over d coordinates operating on a. This type of networks are well known Real-NVPs [13]. One can also define other types of coupling flows. Nevertheless, the theoretical guarantee of any coupling flow can verified by simply checking their universality to singlo-coordinate transforms, our main result stated in Thm. 3.5.
3

---

2.3 Different types of universality and their relations
Here we define the functional universality. See Appx. A for distributional universality and more.
Definition 2.1. (LP L |Ct JCKK-Cmiiii -uiiersaliity [9]). Let M be a set of measurable mappings from IR." to R. Lerp E [1, 00), l E N+ and let F be a set of measurable mappings f JUS - RR, where U is measurable subset of R" which may depend on f. We say that M has L' (or L C Junivvesalii for F, for any f E F any 6 >> O, and any compact subset K C U, there exists g E M such that, (ll < 6, definition of these norms Appx. A).
|l - -l] < (or If - (K))
If M has the C* -u-iiveersality, M has the 1 univerrrssl because (*) ) is dense in 1 (K) for any compact K. Also, if M has the I -nniverrsall,y M has the T/- uu-iniverslllty for any 1 A p < co. If an LP-universality is satisfied for some < p < oc, then for any S q < p, L'-universality is ensured. Finally, L L'uuniversality implies distributional universality but the reverse is not true.
2.4 Related works
Coupling (or triangular) flows. Authors of [24, 6] proved that the distributional universality of flow family can be deduced, if h is dense in the set of all monotone functions by pointwise convergence topology. In particular, [22] proved the distributional universality for neural autoregressive flows, and [26] proved that for sum-of-square flows. [42] generalized it to TP" universality with S p < oo for affine coupling flows, and L universality for autoregressive flows and sum sum-of-squane flows.
Non-coupling flows. The expressiveness of general non-coupling flows is not well studied, as most existing works restrict the form of nonlinearity to certain types, e.g., planar and radial flows [36], Sylvester flows [43], in order to easily compute the determinant of the Jacobian matirx and the inverse maps. [29] gave a first study over some specific distributions, while the universal distributional results still remains unknown. Another type, iResNet [5], was proposed based on residual network (ResNet) [21] to improve nonlinearity as well as the log determinant computation efficiency. [46] proved that iResNet, capped by linear layers or with extra dimensions, has C*) -niiversalltyy
Continuous time flows. The ODE-based method is also a major class of flow models as introduced
in [10, 19, 11, 38]. [11] gave counterexamples for the -o-universalltyy of neural ODEs, however its distributional universality is not yet addressed. An "augmented" neural ODEs was proposed by [15] and then analyzed by [46]. They showed that embedding the original d dimensional neural ODE into a a ++ 1 dimensional space can bypass the original counterexamples, achieving a CO-universality.
3 Main results
In this section, we present our main results, following the architecture similar to [42]. Section 3.1 provides a proof for non- -onaammmetri cases, showing that: 1) the C* CM-universality over the Dift" (R.) can be achieved when the -c-ivvversallty over a simple subgroup of Ditt" (IR)) is ensured; 2) affine coupling layers with one zero- vero-padding has C* -*-universality over such a simple subgroup. In addition, we estimate the number of layers needed for approximate certain diffeomorphism in Thm 3.3. In Section 3.2, we generalize these results to parametric cases with novel proof technique.
3.1 Non-parametric cases
Here outline the main steps of our proof, and the complete proof is available in Appx. B. Recall the space D (Rd): All the O* _d + & diffeomorphisms which are compactly supported. There are two advantages of choosing Dift" (Rd) instead of Dift* (IR)): firstly the compactly supported property greatly simplifies the structure of this group; secondly for any diffeomorphism F e Dift* (IEd) and any compact set K, there exists a compactly supported diffeomorphism f E Difr (I2d)
such that Fix k = flx. i.e., F(x)= fxx) for all K (See Lemma 6 in [42]).
Directly proving the universality to a general diffeomorphism is difficult. is beneficial to decompose the original hard problem into series of simpler ones as follows.
Definition 3.1. A diffeomorphism 5 E Dift" (IR") is in single- coordinate transforms st.d, if r(x) = (I1, (2,,, ,I,-1, (x),, "141,"' (Ka). i.e. only coordinate is altered, and T, Tz(
(T .a) is function from R to R which is monotonic for It.

---

The theorem as follows shows that, all diffeomorphisms in Dift" (R") can be constructed from S
Theorem 3.2. For any subgroup H C Diff s.t. 5t d C H, = Dift' (R."). That is, for any Shid s s.N.t. f = 5--1 0
E Diff: (R.), there exists T1, 72,
Proof: The following two steps are sketched: If H S Dift. (R.) and S C H. H contains all near-identity diffeomorphisms (Def. B.3). The detail is stated in Cor. B.6. 2) If H C Dift. (R") contains all near-identity diffeomorphisms, H Dift" (IRd). The detail is stated in Lem. B.4. L
We further investigate the number of single-coordinate transforms needed to represent an arbitrary diffeomorphism Thm. 3.3 as follows. The definition of distpirg (Bx1) and the proof is provided in Appx. B.2
Theorem 3.3. Given f E (Rd)) we denote dist (R-y (
T) (J, f.ror some O, the minimal length of paths between I and the identity map T lying in Diff (R.) with the Ch-norm induced metric. Then can be decomposed into d(d - T)E single-coordiatt transforms.
Thm. 3,2 decomposes the original general complex diffeomorphism into finitely-many simple diffeo- morphisms. It is natural to ask: if we can approximate all the Sk.d well, can we also approximate Dift. (I well? Thm. 3.4 gives positive answer to the question.
Theorem 3.4. (Approximation for composition.) Suppose G is group of diffeomorphisms over TRd. Given a set of diffeomorphisms F over Rd, denote F as the semigroup generated by F. If G has Ck uunieersality for F, G has C -u-iiversallity for F.
Proof. We only need prove the following statement: If G has C -h-unieersallty for f1, f2 E F. then G has CK -H-universality for f1 fa E F. By the property of semigroups, we know that G has Ct -f-uuivvessality for the whole F. Details can be found in Appx. B.3.
Combining Thm. 3.2 and 3.4, noticing that Sk.d is inverse-invariant (for any E Ski -- E Skid), we can immediately obtain following corollary.
Corollary 3.5. Suppose G a group of diffeomorphisms over Rd, if has Ck -n-uuveersality for St.d. then G has Ck -niiverrslllty for Diff" (R.4).
As stated in Cor. 3.5, any INNs that approximate Sh.d will eventually approximate Dift" (R.) in Ck-norm, which is extension from the LP and L universality stated in [42]. Furthermore, generalize this result to parametric cases in the next section.
Affine-Coupling Flows. Now only need to prove the universality of G-INN, over Sk.d. when G contains H. H-SA However, even for such a simple class of diffcomorphisms, G-INN, is not able to approximate it very well: existing works [42] shows that G-INN, has LP universality over St.d.
Whereas they gave negative conjecture for L universality, let alone the C* -nniveersallty
However, despite the lack of easily computed log-determinant for density evaluation, dimension augmentation has been proven to reach better approximation ability for Neural ODEs [15], [46]. Similarly. when the kingletrransform is embedded into higher dimension Euclidean space by padding zeros, the approximation ability becomes much stronger, achieving -u--niversaalit, far beyond L universality. We first define the canonical immersion lds and submersion ma:
la (I1 ..)) + (z1 Fd, 0), Wa (#1, Ld. Zd+1) ;
Thm. 3.6 says by raising the dimension by 1, we can achieve C -universality. Proof in Appx. B.4.
Theorem 3.6. For any compact set K C Rd, and any T E Sk.d T (x1, D2. Id-1, I.) -Id-1, (01, 22, ". ra(x)), there exists 7 in G-INNa41 with G 2 H--ACF such that || πa
Fo (K)) given that H.a has kkk -uiiversallty over functions K - R.
5

---

3.2 Parametric cases
Following Section 3.1, we generalize the results to parametric cases. We use a novel proof technique. by concatenating parameters with the primary input to be transformed, forming new subgroup over a higher dimensional space, and exploiting the structure of this subgroup.
We denote 5% S.hmmdd e Sh.m+ Dift which is the set of transforms acting single coordinate of æ. Recall the definition of compactly supported parametric diffeomorphisms group in Section 2.1, where y acts as parameters. One might raise question: why not directly the result in non-parametric case, as all the elements in Diffi. also in Diftz (R)m+)) Note that the decomposition theorem only ensures that Sh.m can generate Dift. (Rm+d) and then Din it does not guarantee that Sk.m.d can generate Dift Without such guarantee, we have to use operators that may alter coordinates of both y and æ, and the coordinates of in the output layer should be exactly the same as the coordinates y the input layer, which restricts the flexibility and increases the approximation difficulty. Thus we derive new dedicated theorems for parametric cases.
Theorem 3.7. For any subgroup H S Diff. s.I. Sc Sk.m,d C H, H Dife. m... That is, for any E Diff..m.d there exists TI, 72." To St. 5k.m.dd E N s.I. f = -
Proof. The following two steps are sketched: 1) If H S. Ditr and St Sk.nnd , C H. H contains all near-identity diffeomorphisms (Def. B.3). The detail is stated in Cor. B.6. 2) If H C Diffl.m. contains all near-identity diffeomorphisms, H Din The detail stated in Lem. B.10.
L
Given the above theorem, we easily obtain the -n-universalltyy for Dift following Section 3.1. Theorem 3.8. Suppose G is group of diffeomorphisms over Bm+ IfG has -h-uvieersality for Se Sk.m,d then G has C* -univerraality for Difft.m.dd
proof of G-INN universality over Sk.m,, when G contains H-SACF md follows Thm. 3.6 in Section 3.1, and thus the detail of the proof is skipped here.
4 Para-CFlows as superior neural surrogates for black box optimization
In real applications such as control and optimization of telccommunication networks, power grid systems, and electronic systems, aim optimize complicated system under various contexts. Specifically, we are interested in an unknown function b(a, c) : A x C + R, where A S Rdd, C S Rd are connected and positive measurable. a is a set of actions one can tune and e is a context vector representing exogenous stimuli. In most cases, we are interested in optimizing the function b, i.e. finding a' = arg maxa b(c, a) given a context c. Our task is to quickly find a" at each e using as few queries as possible. In this paper, with dimension augmentation and parameterization (context information inputted as parameters), to model the relationship between b and a, c. Specifically, Para-CFlows are constructed as follows: Step 1: We first map into higher dimensional space I& (d da): Φ + (a, aW) & a(0) where W e Rd. x(d-d,), By construction, rank( dg,
advocate to use Para-CFlowe affine coupling flow models
Step 2: We further compose @o with a series of invertible affine coupling layers o;, siS operating in Idd (Eq. (1)). For any x E Rdd let d' = max(da, [g]], we define
(x,((z) = (x E>d' O exp(ox(c,
+ ta(c,
oe ti are functions with d - d' outputs, implemented by any nonlinear functions like DNNs. S & N, where P; are any fixed random permutation, and ā (N) (Φ PN) o ... (1),1 o (a) A d(c, a).
Step 3: We write a 6, (:(a(t-1))
(2)
which is followed by a simple function à Rd - IR represented by simple neural network that shall be trained. Then the whole model is trained by minimizing the Mean Square Error (MSE).
min mi, TV EisjgM (b, b(o(e,,))))
with by b(c,, a.) are trainning samples.
Although it is trright-forward extension of coupling flows and a similar one was found in [3], our key contribution mainly lie in the theoretical part: for the first time, we prove its C* - to

---

AT: Affine Transform
ST: Straight- Through
DA: Dimension Shuffling
SimeensionAsceeddngg Ascending
Regrosix
1.
1B
(00)
Figure 1: Network structure of Para-CFlow
parametric diffeomorphisms and with this property, we advocate to apply it to address an vital action sensitivity issue in black box optimization other than computer vision tasks where flow models are most widely applied.
We now analyze the superiority of Para-CFlows as the surrogate for black box optimization.
Property A: Para-CFlows preserves full information actions. Existing methods [30, 12, 1, 17, 23] for BO do not handle well the weak action sensitivity issue of many real system optimization tasks [40], where the actions/configuration parameters' effeet is overwhelmed by the highly volatile system state and thus the surrogate model f(c, a) can easily degenerate to f(c) due to small sample size, especially when de >> da- On the contrary, Para-CFlow, by construction, preserves all the information (the rank) of a up to the last feature extraction layer before the final regression layer.
Property B: Para-CFlows has C* universality to all rank-preserving systems. Based on the universality of parametric coupling flows in Section 3, we further investigate what kind of objective function b can be approximated by Para-CFlows. We give the necessary condition to as follows. Assumption 4.1. is an observation of the inner system state b(c, a) = 1(0(c, a)) and the system is Sonk-preserving i.e. the inner state ((c, a) C A I&ds is a d da.
and preserves full information of actions, i.e., rank (34
0a
Under Assumption 4.1, we can get the following result, which ensures the expressiveness of our architecture,
Theorem 4.2. For any system & : C x A + Rd satisfying Asmp. 4.1, if d 2 2ds and 4 have up to k-th order derivatives, then for any E >> O, we can always find a é in the form (2), such that a) (ffcc.))lo<((xx I E, given that o;, tú, b are (R-universal function approximator Here
π is the canonical submersion: Rd Rd,, (I1, T22 Idg: " ..) + (x1, 22. F.a))
Readers check Appx. B.6 for the proof. Thm 4.2 shows that the inner system & can be uniformly approximated by the feature extraction layer e of Para-CFlow. The observation function ((-) can then be approximated by i as result of the universality of Multi-Layer Perceptons (MLPs).
The C* -pproximabiiiity property of Para-CFloo Let us just focus on the exploitation phase, i.e., a" arg max b( a)). When the action space is continuous and high dimensional (though in this paper, we focus on the context dominating tasks where d, > d,), one is not able find the optima easily by brute-force search. Many existing works utilize meta-heuristic such as Evolutionary Algorithm (EA) of which the performance heavily depends on the population size and the number of iterations, and thus is computation demanding and time consuming. Instead, one can apply Gradient Descent (GD) to the surrogate if is differentiable,
of vital importance for black box optimization.
i.e. a+1 + a' + v/V b(o(c, a)) a=aa. However, for the GD method to work, it requires that the surrogate approximates not only the function value but also its first-order derivative. However function value approximation does not necessarily guarantee derivative approximation. A simple illustrative example fn (x) g(x) + sin(ne)) One can easily observe that fn(x) - g(x), but (x) o (x( + cos(nz:) will never converge to g'(x) when - xc. With this argument, we advocate Para-CFlos as an adequate choice for surrogate modeling for continuous action space BO. We also find it suitable to approximate Q-function in Reinforcement Learning (RL), especially
7

---

for DPG [391, DDPG[31] and references therein where one need to estimate the gradient of the Q-function w.r.t. the action to train the actor network. Extension DRL will be left for future work.
With the affrrmmenioned properties, our proposed Para-CFlow model can serve as a superior neural surrogates and in the setting of BO, learning and deep ensembles to facilitate the uncertainty modeling with Para-CFlos
easily adopt existing SOTA methods of Bayesian deep
Experiments
In this section, we conduct experiments to verify the expressiveness of our proposed Para-CFlow and demonstrate its advantage as neural surrogate models in application to black-box optimizations.
5.1 Learning diffeomorphism "Taiji"
We demonstrate the expressiveness of our model to learn a parametric diffeomorphism with a synthetic task called "Taiji". We define our target parametric diffeomorphism:
fw cos A, p psin 0) + (p é, p sin @in where On the unit circle p 1, it is not differentiable, thus we expect the trained model to have irregular derivatives over the unit circle, while maintain accurate derivatives elsewhere.
+ arccos(min((. 1)). (3)
First, we design task with sufficient amount of data to verify the C' approximation ability of Para-CFlows. We generate samples { 1133000000000 with I; u[-1, 1]2 and yr 24 1] independently. Under the polar coordinate representation, i.e., = (p: cos 0,, P; sin 0:), we calculate the target fu, (zx; (p; cos 0,, Pt sin 0.) according to Eq. (3). We use 6-layers affine coupling flows, each layer composed with random permute. The coupling functions a, I are implemented by 1 I-hidden-lay layer Multi- Layer Perceptron (MLP) with hidden-uni number comparable to input dimension.
We pad zeros to the input æ. and use the first two output dimensions to compute the loss.
fx())
fo(x)
(x)
(x)
Para
Fiee (.0
FMLP 0o0
f
(x)
(x(x
(x)
(x))
Para y y (x)(X)
yy y 1,,5x)) (x)
Aas
y (x()
G
G
(a) Trained [x( v.s. Target fs(æ)
(b)
Figure 2: 2(a): learning "Taiji" diffeomorphism tasks in Section 5.1. Black region, 32 0n 00000pp S white region, 42 S o00 p < yellow region, p p / + and 22). fa denotes the fold composition fi o .... ft. 2(b): Comparing Para-CFlow with Resnet and MLP with small sample size and parameter ( = y, 1 S < 100. Grey color area has no point resides.
On the left panel of Fig. 2(a), we observe good consistency between the trained model fy(2)) and (x), when y E [0, 1]. Note that we only use samples with y E [0, 1] for training, and thus the model generalizability to y is not guaranteed. However, the true diffeomorphism satisfies f fuz = fur +22 we iterate the trained model recurrently and expect the final output to achieve the effect for 1. The results the right panel of Fig. 2(a) justify that the cumulative error from composition does not grow fast, and our model can approximate complex diffeomorphisms with sufficient layers. In Fig. 3(a) and Fig. 3(b) plot the derivatives of true one. The non-differentiabiliyy of trained model
compared to the the groundtruth on the unit circle results in irregular behaviors the unit circle; elsewhere it shows good consistency.
To further verify our theorem stated in Section 3, specifically the effect of padding zero(s), we test the same model under different hidden dimensions dhid and number of stacking layers in Fig. 3(c).

---

127
121
131
309
60
(a)
0 (p)
Ow
V.s.
Oz
(y,m)
(b)
0y
V.s.
(f()))))
0y
(c)
Figure 3: 3(a) and 3(b): Derivatives of the trained model fy(xr) (f() æ)), f(2) x)) with black arrows and the groundtruth fy(2) (f x), f(2) (y. x)) with red arrows, all at y 1. 3(c):
MSE under different hidden dimensions and number of layers.
Here hidden dimension equals to the input dimension (2 in this experiment) plus the number of padding zero(s). As we expected, according to our theorems, with padding zero(s) Para-CFlog enjoys universality. We also investigate the expressiveness when hidden dimensions = 2. When increase the number of stacking layers, MSE also decrease, which justifies the L universality result in [42]. However, increasing the hidden dimension is much more efficient in terms of improving expressiveness than increasing the number of stacking layers. Such results support our theory of zero-padding. More results of different architecture settings are available in Appx. D.1.
Next we design more challenging task to compare Para-CFlox against a simple MLP with hicden-layer-ssetinng as (128, 64, 32), and Residual Nets (Resnet) [21]. that concatenates y and x as the network's input and then appends æ to the output of each layer forward for prediction. Unlike the previous experiment where scalar y is used, generate 3000 samples of 100-dimensional
parameter vector y; (y) ()) .... y, (100)) where y! [N 0.16) and y; 4[0 1]. Then (æ; y:) is fed to each model to learn the target Javg(u:) (av.). Fig. 2(b) shows that Para-CFlow learn much more continuous mapping than MLP and Resnet. While Resnet learns relatively better the outer part than MLP, similar to MLP it suffers from bad mapping in the inner part, i.e. the grey area inside the unit circle is much larger than Para-Flow, which represents bad continuity of the learnt mapping.
5.2 Application to contextual BO
In this section, we ensemble Para-CFlows novel surrogate model for BO with three well-known benchmark functions: Rastrigin, Ackley and Trid [41]. Note that the original benchmark tasks contextless, here we simply the first de dimensions the context vector sampled from multivariate uniform distribution and leave the last dimension as the action to optimize. To construct uniformly
context-dominating tasks set de > 1. In all benchmarks above, the contexts sampled at random from [-3, 3]de sequentially while the action is optimized by gradient descent in [-3, 3] over surrogate models. The used ensemble-like surrogate models for contextual BO are listed as follows: (1) Para-CFlow Ensemble of Para-CFlows for prediction with uncertainty; (2) MLP: Ensemble of MLPs for uncertainty modelling by concatenating context c and a to obtain input (c, 11 (3) MLP- Ascend: Similar to MLP, the difference is that it firstly raises both and to the same hidden dimension max(5, de) and then concatenates them as inputs to a MLP; (4) Resnet: Ensemble of multiple Resnet that concatenates and the network's 's input and then appends to output each layer forward for prediction. More experimental settings are available in Appx. D.
Firstly, we investigate the action sensitivity of each model trained with 100 random samples. The dimensionality of context is set 5, 10. 20, which is significantly greater than that of action. The action sensitivity is calculated as the ratio of Permutation Feature Importance (PFI) [8] of the action w.r.t. that of the context, i.e. PFI //n)///xx//// (PF)( and reported in Tab. 1. We observe that Para-CFlow and Resnet have much higher action sensitivities thanks to the model architecture, while MLP and MLP-Ascend are dominated by contexts as expected. We further investigate whether different surrogates and their gradients manage to guide the real objective function optimization. Denote each trajectory as {bb, 8' a)))) a'){=ilc), where (a (a')T= is a T Ttep gradient trajectory over the

---

(a)
(b)
Figure 4: 4(a): quiver plot of (b') (x-axis) v.s. (b') (x-axis) along each averaged gradient trajectory over 10 random initialization x 10 random contexts. 4(b):The mean and standard deviation of the cumulative regret under 10 independent trials with the context dimension as 5, 10 and 20.
surrogate b(c,, a) and b(c, is the corresponding real objective value We visualize the quiver plot of (Bt)T= (æ-axis) v.s. (b')T=1 (y-axis) along each gradient trajectory for compared surrogates in Fig.4(a). We can easily observe that Para-CFlow outperforms other surrogate models as the red arrows move towards the bottom left corner which exactly means the gradient trajectory on PFra-Cllow indeed minimizes the real objective function, and thus the result supports our main results of C approximability, whereas the compared trajectories move towards the top left corner after few steps probably because those surrogates suffer from more severe local oscillation w.r.t. a, comparing to Para-CFlow, resulting in unexpected gradient explosion. We then compare different surrogate models' performances BO. The mean and standard deviation of the cumulative regrets for different surrogate models are reported in Fig. 4(b) where we observe improvement using Para-CFlow the neural surrogates, especially for the Rastrigin benchmark.
Table 1: Comparison of action sensitivity over 10 independent trails
Rastrigin d. = 10
Ackley d= 10
d. =
4. 21
d. =
d.
20
d.
10
d. 20
1.100051
MLP
0.89=0.16
0.70=0.14
0.71#0.13
0.91
0.79#0.15
0.51=0.16
0.71=0.20
0.64#0.19
0.47=0.20
MLP-Aseend
0.90=0.24
0.80=0.21
0.1540.16
0.95+0.20
0.8840.10
0.58=0.14
0.8120.29
0.6420.19
0.19:0.15
Resnet
1.6520.22
1.60.0.22
2.3240.34
2.0340.38
1.8840.25
3.4421.02
1.2920.32
1.7740.39
1.7420..5
6 Conclusion
In this paper, we firstly prove the equivalence between the -niiiveeraaltty over compactly-supported diffeomorphisms and that over single-coordinate transforms, resulting the C* universal approxima- tion ability of affine coupling flows. Furthermore, we generalize the main theorems to parametric cases and propose practical model called Para-CFlows which could serve as good neural surrogate for gradient-basee search and optimization. With good capabilities both in universal approximation and robust sensing of critical features in parametric diffeomorphisms, we empirically exhibit the advantages of Para-CFlow using various benchmarks. However, due the dimension augmentation (for expressiveness purpose), Para-CFlos not able to benefit from fast log-determinant estimation if it is applied to generative model tasks which will be studied in our future work.
10

---

References
[1] Robin Allesiardo, Raphaël Féraud, and Djallel Bouneffouf. neural networks committee for the contextual bandit problem. In ICONIP, 2014.
[2] Lynton Ardizzone, Jakob Kruse, Sebastian J. Wirkert, Daniel Rahner, Eric Pellegrini, Ralf S. Klessen, Lena Maier-Hein, Carsten Rother, and U. Köthe. Analyzing inverse problems with invertible neural networks. ArXiv, abs/1808 04730, 2019.
[3] Lynton Ardizzone, Carsten Lüth, Jakob Kruse, Carsten Rother, and U. Köthe. Guided image generation with conditional invertible neural networks. ArXiv, (bbs/190..0225 2019.
[4] A. Banyaga. The Structure of Classical Diffeomorphism Groups. Springer US, 1997.
[5] Jens Behrmann, David Kristjanson Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In ICMI., 2019.
[6] Vladimir Bogachev, Alexander V. Kolesnikov, and Kirill Medvedev. Triangular transformations of measures. Sbornik Mathematics, 196:309-335, 2004.
[7] W. M. Boothby. An introduction to differentiable manifolds and riemannian geometry. 1975.
[8] L. Breiman. Random forests. Machine Learning, 45:5-32, 2004.
[9] Haim Brezis. Functional analysis, sobolev spaces and partial differential equations. 2010.
[10] Changyou Chen, Chunyuan Li, Liquan Chen, Wenlin Wang, Yunchen Pu, and Lawrence Carin. Continuous-time flows for efficient inference and density estimation. ICML, 2018.
[11] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David Kristjanson Duvenaud. Neural ordinary differential equations. In NeurlPS, 2018.
[12] Youngseog Chung, Ian Char, Willie Neiswanger, Kirthevasan Kandasamy, Andrew, Oakleigh Nelson, Mark Boyer, Egemen Kolemen, and Jeff G. Schneider. Offline contextual bayesian optimization for nuclear fusion. ArXiv, abs/2001.017 2020.
[13] Laurent Dinh, Jascha Sohl-Dickstenn and Samy Bengio. Density estimation using real nyp. Abs/1605.08803. 2016.
[14] Laurent Dinh, Jascha ArXiv, abs//1055008000 2017.
and Samy Bengio. Density estimation using real nvp.
[15] Emilien Dupont, A. Doucet, and Yee Whye Teh. Augmented neural odes. In NeurlPS, 2019.
[16] David G. Ebin and Jerrold E. Marsden. Groups of diffeomorphisms and the motion of an incompressible fluid. Anmals of Mathematics, 92:102, 1970.
[17] Raphaël Féraud, Robin Allesiardo, Tanguy Urvoy, and Fabrice Clérot. Random forest for the contextual bandit problem. In AISTATS, 2016.
[18] Alfred Galichon. Optimal transport methods in economics. 2016.
[19] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Kristjanson Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. ArXiv, abs//810.01366.. 2019.
[20] David Grifliths, Andrew M. Stuart, and H. C. Yee. Numerical wave propagation in an advection equation with a nonlinear source term. SIAM Journal on Numerical Analysis, 29:1244 -2640, 1992.
[21] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[22] Chin- Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autore- gressive flows. ArXiv, abs/1804.00079, 2018

---

[23] Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model- optimization for general algorithm configuration. In LION, 2011.
[24] Aapo Hyvärinen and Petteri Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural nefworks the official journal of the International Neural Network Society, 12 3:429-439, 1999.
[25] Pavel Izmailov, P. Kirichenko, Mare Finzi, and Andrew Gordon Wilson Semi-supervised learning with normalizing flows. ArXiv, abs/1912.13025, 2020.
[26] Priyank Jaini, Kira A. Selby, and Yaoliang Yu. Sum-of-squares polynomial flow. ArXiv, (tbs/1905.02325 2019.
[27] M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-92 1938.
[28] Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolu- tions. In NeurlPS, 2018.
[29] Zhifeng Kong and Kamalika Chaudhuri. The expressive power of class of normalizing flow models. ArXiv, abs/2006.00392, 2020.
[30] Andreas Krause and Cheng Soon Ong. Contextual gaussian process bandit optimization. In NIPS, 2011.
[31] Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, 500/00999..911 2016.
[32] Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In ICML, 2017.
[33] Mohammad Mozaffari, Walid Saad, Mehdi Bennis, and Mérouane Debbah. Wireless com- munication using unmanned aerial vehicles (uavs): Optimal transport theory for hover time optimization. IEEE Transactions Wireless Communications, 16:8052-8066, 2017.
[34] George Papamakarios, Eric T. Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference. Mach. Learn. Res., 22:57:1-57:64, 2021.
[35] Gabriel Peyré and Marco Cuturi. Computational optimal transport: With applications to data science. 2019.
[36] Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In ICML, 2015.
[37] Elon D. Rimon and Daniel E. Koditschek. The construction of analytic diffeomorphisms for exact robot navigation on star worlds. Proceedings, 1989 International Conference on Robotics and Automation, pages 21-26 vol.1, 1989.
[38] Hadi Salman, Payman Yadollahpour, Tom Fletcher, and Nematollah Batmanghelich. Deep diffeomorphic normalizing flows. ArXiv, Abs//810.0322 2018.
[39] David Silver, Guy Lever, Nicolas Manfred Otto Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In ICML, 2014.
[40] Miguel Suau, Alexandros Agapitos, David Lynch, Derek Farrell, Mingqi Zhou, and Aleksandar Milenovic. Offline contextual bandits for wireless network optimization. ArXiv, abs/2111.08587 2021.
[41] Sonja Surjanovic and Derek Bingham. Optimization test functions and datasets. http://www. sfu. caaccaaaaaae saar-ssuranms
html, 2017.
[42] Takeshi Teshima, Isao Ishikawa, Koichi Tojo, Kenta Oono, Masahiro Ikeda, and Masashi Sugiyama. Coupling-based invertible neural networks are universal diffeomorphism approxima- tors. ArXiy, 1ttp20006.11469) 2020.
12

---

[43] Rianne van den Berg, Leonard Hasenclever, Jakub M. Tomezak, and Max Welling, Sylvester normalizing flows for variational inference. ArXiv, abs/1803.05649, 2018.
[44] Aäron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C. Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalch- brenner, Heiga Zen, Alex Graves, Helen King, Tom Walters, Dan Belov, and Demis Hassabis. Parallel wavenet: Fast high-fidelitg speech synthesis. In JCML, 2018.
[45] Cédric Villani. Optimal transport: Old and new. 2008.
[46] Han Zhang. Xi Gao, Jacob Unterman, and Tom Arodz. Approximation capabilities of neural ordinary differential equations. ArXiv, abs/1907, 12998, 2019.
13


Reviews:
[{"review_id": "a2FbVr6gZbQ", "reviewer": "Reviewer_UuMf", "summary": "This paper proves that coupling flows - in particular, flows with single coordinate affine coupling layers - can universally approximate any compactly supported diffeomorphism with non-degenerate derivative in $C^k$ norms. They also extend the result to parameteric setting, where additional parameters are to be thought of as context (and are passed to all the layers in the network without changing). \n\nFurther, they provide some experiments comparing the results to known models on synthetic datasets - to approximate known diffeomorphisms and contextual bayesian optimization tasks", "questions": "## Questions / Clarifications\nI am very confused with proof of theorem 3.3 - it needs some more clarification. References to lemma B.4 and corollary B.6 seem to be interchanged. I am not sure that it suffices to look at the $C^1$ norm induced on $\\text{Diff}_C^k$, since I believe we do want all the functions $g_i$ to be $(\\delta,k)$ approximations of identity, which would require a bound on $C^k$ induced norm. Even otherwise, it is not clear to me how the bound on $s_1$ follows. \n\nApart from this, I found flow of the proofs to be fairly readable, and the organization of the paper was great!\n\n## Additional Comments\n1. 52: gradeint instead of gradient\n2. 221, 224: confusing / inconsistent notation for function b(a,c)\n3. 353: Te instead of the\n4. 568: for any $f$ that is $(\\delta,1)$-near-identity\n5. 652: mark -> remark? ", "limitations": "Authors have adequately addressed the limitations and potential negative societal impact of the work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\nThe paper seems to be theoretically sound up to my understanding, and provides easily readable and understandable proofs for the results. The proof technique is pretty neat, and can potentially be used in other theoretical results in deep learning. It also provides more theoretical backing along the recent line of work that shows that padding helps in training coupling flows - this has been observed recently though multiple results.\n\n## Weaknesses\n1. The bound is extremely existential and does not give any sort of bounds on number of layers or parameters in terms of function properties.\n2. Would be good to know how the distance function defined over $\\text{Diff}_c^k(\\mathbb{R}^d)$ relates to more commonly used norms of the diffeomorphism.\n3. Experiments for diffeomorphism approximation are on low dimensional datasets - would be great to see how dimension affects these results.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KH38KRfoWJC", "reviewer": "Reviewer_hkZK", "summary": "This paper discussed the expressive power of invertible neural networks based on coupling flows (CFlows) for approximating diffeomorphisms. Specifically, suppose a class of invertible functions can approximate a small class of diffeomorphisms consisting of single coordinate affine transformations in $C^k$-norm. In that case, this class can approximate a class of compactly-supported diffeomorphisms (Corollary 3.5 for non-the parametric case, Theorem 3.8 for the parametric case). Furthermore, This paper showed that single-coordinate affine coupling flow (SACF) with one-dimensional dimension augmentation has $C^k$-universality (Theorem 3.6).\nBased on the above analysis, this paper proposed Para-CFlow, which has universality under the rank-preserving assumption. Then, this paper applied Para-CFlow to a contextual BO task to verify its usefulness empirically.", "questions": "- l.221: $\\mathcal{A}\\in \\mathbb{R}^{d_a}$ -> $\\mathcal{A}\\subset \\mathbb{R}^{d_a}$\n- l.353: Te -> The", "limitations": "The conclusion section discussed the technical limitations of the proposed Para-CFlow. Specifically, this paper pointed out that a fast log-determinant is impossible (l.362).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- This paper extended existing theories with simpler proofs.\n- The numerical experiments tasks that require function derivatives. These experiment settings align with the theory part.\n- Presentation is good.\n\nWeaknesses\n\n- Other studies also dealt with the $C^k$-universality of invertible models.\n- The proposed method, Para-CFlow, is difficult to use for applications to which we usually employ invertible models such as image generation.\n\n\nOriginality（Novelty）\n\n- Recently, [1] extended Teshima et al. (2020) by developing the theory of universality of invertible models with respect to the Sobolev-norm, which includes function derivatives. However, since this paper has been recently published, I do not think that this paper significantly reduces its novelty.\n  - [1] https://arxiv.org/abs/2204.07415\n\n\nQuality（Soundness）\n\n- As far as I checked, the proof is correct.\n- Applying the theory and models to contextual BO tasks is appropriate,  as it requires (first-order) derivatives (l.262). \n\nClarity（Presentation）\n\n- The organization of this paper is OK.\n- Mathematical descriptions are good, and proofs are easy-to-follow.\n\nSignificance\n\n- Although the architecture is different, this paper proved the $C^k$-universality of SACFs (with minimal preprocessing) that Teshima et al. (2020) did not deal with.\n- This paper gave a more straightforward proof of universality than Teshima et al. (2020). It makes the theory more accessible to many researchers and deepens the understanding of the expressive power of the invertible model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-GpgvkVvvpa", "reviewer": "Reviewer_15hV", "summary": "The paper studies the approximation power of coupling flows in terms of the general functional universality, beyond the typical notion of distribution universality. The authors first develop theorems that show that the space of invertible neural networks generated by coupling flows have C^k-universality, assuming that we perform dimensional augmentation by adding a padding dimension. The authors then develop similar results for parametric problems. These approximation theorems are used to provide guarantees for the authors' new model Para-CFlows, which works for parametric problems and incorporates both dimension augmentation. Experimental results are used to corroborate the power of Para-CFlows in learning complex diffeomorphisms. ", "questions": "Instead of asking questions, I will focus on giving suggestions to improve the clarity of the paper:\n - For section 3, instead of showcasing a list of partial theorems leading to the result (which is more suitable for the Appendix), I would suggest first stating one main theorem that represents the \"final conclusion\" that you want to get across to the reader. Afterwards, you can summarize the techniques/partial theorems that you used to prove that main theorem. This would make that section shorter (allowing for more space later on) and less dense for readers. \n - For section 4, I would suggest highlighting the model construction steps. This can either be done by indenting the three steps (e.g. using itemize), or creating a text box around these steps. I would also seriously consider moving Figure 6 in the Appendix to the main text, if you can create more space by cutting down content in section 3. ", "limitations": "N/A", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Quality and Clarity__: In general, the paper is well-written. The authors clearly introduce all the concepts that they used, and presented a detailed roadmap of how they developed their theorems (I placed some suggestions on how to organize them in the Questions part). The experiments are also comprehensive and convincing. \n\n__Originality and Significance__: Unfortunately, I work more on normalizing flows for learning distributions, so the subject of this paper is a bit beyond my expertise, and I would refer to other reviewers to comment more on the significance. That said, I believe understanding the approximation power of coupling flows on parametric models is a relevant and strong result, and having algorithms that achieve universality would be a useful contribution to the field. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept