The Star Geometry of Critic-Based Regularizer Learning
Authors: Oscar Leong
Eliza O'Reilly
Yong Sheng Soh
Venue: NeurIPS 2024 poster (2024)
Keywords: inverse problems
regularization
unsupervised learning
convex geometry
Primary area: optimization
TL;DR: We use star geometry and dual Brunn-Minkowski theory to analyze what is the ``best'' regularizer for a given data distribution.

Abstract: Variational regularization is a classical technique to solve statistical inference tasks and inverse problems, with modern data-driven approaches parameterizing regularizers via deep neural networks showcasing impressive empirical performance. Recent works along these lines learn task-dependent regularizers. This is done by integrating information about the measurements and ground-truth data in an unsupervised, critic-based loss function, where the regularizer attributes low values to likely data and high values to unlikely data. However, there is little theory about the structure of regularizers learned via this process and how it relates to the two data distributions. To make progress on this challenge, we initiate a study of optimizing critic-based loss functions to learn regularizers over a particular family of regularizers: gauges (or Minkowski functionals) of star-shaped bodies. This family contains regularizers that are commonly employed in practice and shares properties with regularizers parameterized by deep neural networks. We specifically investigate critic-based losses derived from variational representations of statistical distances between probability measures. By leveraging tools from star geometry and dual Brunn-Minkowski theory, we illustrate how these losses can be interpreted as dual mixed volumes that depend on the data distribution. This allows us to derive exact expressions for the optimal regularizer in certain cases. Finally, we identify which neural network architectures give rise to such star body gauges and when do such regularizers have favorable properties for optimization. More broadly, this work highlights how the tools of star geometry can aid in understanding the geometry of unsupervised regularizer learning.

The Star Geometry of Critic-Based Regularizer Learning
Oscar Leong
Eliza O'Reilly
Department of Statistics and Data Science University of California, Los Angeles ol oleong@stat. ucla.edu edu
Department of Applied Mathematics and Statistics Johns Hopkins University eoreil12@jh, edu
Yong Sheng Soh Department of Mathematics National University of Singapore matsys@nus edu. sg
Abstract
Variational regularization is classical technique to solve statistical inference tasks and inverse problems, with modern data-driven approaches parameterizing regu- larizers via deep neural networks showcasing impressive empirical performance. Recent works along these lines learn task-dependent regularizers. This is done by integrating information about the measurements and ground-truth data in an unsu- pervised, critic-based loss function, where the regularizer attributes low values likely data and high values to unlikely data. However, there is little theory about the structure regularizers learned via this process and how it relates to the two data distributions. To make progress on this challenge, we initiate study of optimizing critic-based loss functions to learn regularizers over particular family of regulariz- ers: gauges (or Minkowski functionals) of star-shaped bodies. This family contains regularizers that are commonly employed in practice and shares properties with regularizers parameterized by deep neural networks. We specifically investigate critic-based losses derived from variational representations of statistical distances between probability measures. By leveraging tools from star geometry and dual Brunn-Minkowski theory, we illustrate how these losses can be interpreted as dual mixed volumes that depend the data distribution. This allows us derive exact expressions for the optimal regularizer in certain cases. Finally, we identify which neural network architectures give rise to such star body gauges and when such regularizers have favorable properties for optimization. More broadly, this work highlights how the tools of star geometry can aid in understanding the geometry of unsupervised regularizer learning.
Introduction
The choice and design of regularization functionals to promote structure in data has long history in statistical inference and inverse problems, with roots dating back to classical Tikhonov regularization [89]. In such problems, one is tasked with solving the following: given measurements y E I of the form = A(xa) + v for some forward model A Rd + T and noise E I recover an estimate the ground-truth signal Zo E Rd Typically, the challenge in such problems is that they ill-posed, meaning that either there are solutions, infinitely many solutions, the problem is discontinuous in the data y 77. A pervasive and now classical technique is to identify regularization functionals R. Rdd R such that, when minimized, promote structure that is present in the ground-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

truth signal. Well-known examples of hand-crafted regularizers to promote structure include the total variation 774 the nuclear norm 29
& --normm to promote sparsity [23] 16 26
15 70
and
more generally, atomic norms 18
78
64
With the development of modern machine learning techniques, have seen a surge of data-driven methods to directly learn regularizers instead of designing regularizers in a hand-crafted fashion. Early works along these lines include the now-mature field of dictionary learning sparse coding (see
8D and the surveys [56] 27 for more). More recently, we have seen powerful nonconvex regularizers parameterized by deep neural networks. These come in a variety of
60 61 84
75
flavors, including those based on plug-and and-play 791,711 generative models [3
39 40 g 20), and
regularization functionals directly parameterized by deep neural networks [53 59
80
36 47
Despite the widespread success of such learning-based regularizers, several outstanding questions remain. In particular, it is unclear what type of structure is learned by such regularizers. For example, what is the relationship between the underlying data geometry and the type of regularizer found by data-driven approaches? Is the found regularizer "optimal" in some meaningful sense? If not, what properties of the data distribution are lost due to structural constraints placed on the regularizer?
In this work, we aim to tackle the above questions and others further understand what types of regularizers are found via learning-based methods. We focus on a class of unsupervised methods that dependennn,,,, i.e., those that learn regularizer without paired training data, but still integrate information about the measurements in the learning process. This is done by identifying loss inspired by variational representations of statistical distances, such as Integral Probability Metrics (IPMs) or divergences, where the regularizer plays the role of the "critic" or test function.
As an example, the recent adversarial regularization framework used in [53 59 80) learns regularizer that assigns high "likelihood" to clean data D, and low "likelihood" to noisy data Dn via loss derived from dual formulation of the I-Wasserstein distance. This framework has showcased impressive empirical performance in learning data-driven regularizers, but there is still a lack of an overarching understanding of the structure of regularizers learned. Moreover, while this Wasserstein distance interpretation has shown to be useful, it is natural to consider if other losses can be derived from this (critic-based perspective to learn regularizers.
1.1 Our contributions
In order to make progress on these challenges, we first fix a family of regularization functionals to analyze. We aim for this family to be (i) expressive (can describe both convex and nonconvex regularizers), (ii) exhibit properties akin to regularizers used in practice, and (iii) tractable to analyze. A family of functionals that satisfy such criteria are gauges of star bodies. In particular, will consider regularization functionals of the form
||>|| k = inf > 0: x tK}
where K is a star body, i.c., a compact subset of Rd with E int(K) such that for each æ e Rd {0), the ray {tr it 0} intersects the boundary of K exactly once. Such regularizers for general star bodies K, but note that || is convex if and only if K a convex body. Any norm is the gauge of convex body, but this class also includes nonconvex quasinorms such as the qu-quasinorm for q E (0, 1).
nonconvex
Focusing on the above class of functionals, we aim to understand the structure of regularizer || ||x found by solving an optimization problem of the form
inf H. (II -lk:; D,, Dn.) II-E
(I)
where H(: D,, P.)) : F + compares the values || llx assigns D, and D, and F class of functions on IRd. The adversarial regularization framework of (53) is recovered by setting
H(;; D,, D,) = [f()] ]- Ep, [f)] and F. = Lip(1) == { Rd + R. f is 1-Lipschitz).
Our contributions are as follows:
1. Using tools from star geometry 41
and dual Brunn-Minkowski theory 54 we prove that under certain conditions, the solution to ( under the adversarial regularization frame- work of [53] can be exactly characterized. In particular, we show that the objective is
33

---

equivalent to a dual mixed volume between our star body K and a data-dependent star body L This allows us exploit known bounds on the dual mixed volume via (dual) Brunn-Minkower theory.
2. We investigate new critic-based loss functions H(: Dr., Pa) in inspired by variational rep- resentations of divergences for probability measures. We specifically analyze a-divergences and show how they can give rise to loss functionals with dual mixed volume interpretations. We also experimentally show that such losses can be competitive for learning regularizers in a simple denoising setting.
3. We conclude with results showcasing when these star body regularizers exhibit useful optimization properties, such as weak convexity, as well as analyzing what types of neural network architectures give rise to star body gauges.
This paper is organized as follows: Section 2 analyzes optimal adversarial regularizers using star geometry. We establish general existence result in Section 2.1 and use dual mixed volumes to characterize the optimal star body regularizer under certain assumptions in Section 2.2 Visual examples are provided in Section 2.3 Section 3 introduces new critic-based losses for learning regularizers inspired by a-divergences. An empirical comparison between neural network-based
regularizers learned using these losses and the adversarial loss presented in Section [3.]] Section examines computational properties of star body regularizers, such as beneficial properties for optimization and relevant neural network architectures. We conclude with a discussion in Section
f
1.2 Related work
We discuss the broader literature on learning- based regularization in Section A of the appendix and focus on optimal regularization here. To our knowledge, there are few works that analyze the optimality of regularizer for given dataset. The authors of the original adversarial regularization framework of [53] analyzed the case when D, is supported on a manifold and D,, is related to Dr via the push-forwatd of the projection operator onto the manifold. They showed that in this case the distance function to the manifold is an optimal regularizer, but uniqueness does not hold (i.e., other regularizers could be optimal well). Our work complements such results by analyzing the structure of the optimal regularizer for a variety of 'D and DD, which may not be related in this way. In [4] the authors analyze the optimal Tikhonov regularizer in the infiniteddimension and show that the optimal regularizer is independent of the forward model and only depends on the mean and covariance of the data distribution.
Hilbert space setting,
The two most closely related papers to ours are 900 and (52] In 900, the authors aim to characterize the optimal convex regularizer for linear inverse problems. Several notions of optimality for convex regularizers are introduced (dubbed compliance measures) and the authors establish that canonical www-dimensional models, such as the fo-norm for sparsity,
optimal for sparse recovery under such compliance measures. In [52], similarly to this work, the authors analyze the optimal regularizer amongst the family of star body gauges for given dataset. They also leverage dual Brunn-Minkowski theory to show that the optimal regularizer is induced by a data-dependent star body, whose radial function depends on the density of the data distribution. Interestingly, these results also characterize data distributions for which convex regularization is optimal and they provide several examples. The present work is novel in relation to (52) for several reasons. First, our work introduces novel theory and new framework for understanding critic-based regularizer learning, addressing significant gap in existing theoretical foundations for unsupervised regularizer learning. This new setting also brings novel technical challenges that were not present in [52). For example, the losses in Section 3 exhibit more complicated dependencies on the star body of interest. leading to the need for new analysis. Such losses are also experimentally analyzed in Section C.I Finally, we present new results that are relevant to downstream applications of star body regularizers in Section
1.3 Notation
In this section, we provide some brief background on star geometry and define notation used in the main body of the paper. For further details, please see Section B. in the appendix and the 41 for more. We say that a closed set K C IRd is star-shuped (with respect to the origin) for all x E K, we have [0, xr] C where, for two points ar, y E Rdd we define the
sources (77 33
line segment [z, y] {(1 t)z + ty t E [0, 1]}. K is star body if it is a compact star- shaped
3

---

set such that for every I # O, the ray R₂ := ftz: : >>> 0} intersects the boundary of K exactly once. Equivalently, K is a star body if its radial function Pk positive and continuous over the unit sphere s6-1, where PK is defined as px(x)) == sup(t > U.t. E K). It follows that the gauge function of K satisfies || = (/) for all a E Id such that x of O. Let S to be the space of all star bodies on R For K, L E S it is easy to see that K S if and only Pk & Pr.- The kernel of a star body K is the set of all points for which K is star-shaped with respect to, i.e., ker(K) = {i E K [x. vl C K, Vy E K]. Note that ker K) is a convex subset of K and ker(K)
K if and only if K is convex. For parameter y > O, define the following subset of 8d consisting of well-conditioned star bodies with nondegenerate kernels: (*))) == {K E BB' - kcr(K)} where Bd == {or E B [|aalle, & 1). For two distributions P and Q, let & Q denote their convolution, i.e., P Q is the distribution of X + Y where X P and Y ~ Q. Finally, for a function f and a measure P. let f+ (P) denote the push-forward measure of under S. i.e. for all measurable subsets A, we have f (A) == P (A)-
2 Adversarial star body regularization
To illustrate our tools and results, we first consider regularizer learning under the adversar- ial regularization framework of [53]. Given two distributions D, and Dn, on IR and setting H(f: D,,, Dn) Ep, [f()]] Ev,,,,_,.. [f(z)] and F := Lip(1) in L aim to understand mini- mization of the functional F(K; F,,; Dr) == Ep, || Ep,
[ over all star bodies K such that + ||ж|| x is 1-Lipschitz. A result due to [73] shows that ||x is 1-Lipschitz if and only if the unit ball 8 C ker(K). Here and throughout this paper, it will be useful to think of D, as the distribution of ground-truth, clean data while D, a user-defined distribution describing noisy, undesired data. Using the notation from Section
[.3 we are interested in analyzing
inf KES4(1)
D,,:),
(2)
2.1 Existence of minimizers
2 requires minimizing a functional over structured subset of the infinite- dimensional space of star bodies, even basic questions such as the existence of minimizers are unclear. Despite this, one can exploit tools in the star geometry literature to obtain guarantees regard- ing this problem. The proof of this result exploits Lipschitz continuity of the objective functional and local compactness properties of (T))), proved in [52], akin to the celebrated Blaschke's k Selection Theorem from convex geometry [77]. We defer the proof to the appendix in Section B.2
As the problem
Theorem 2.1. For any two distributions DD and D, on R.d, we have that
F(K; > (D,,,, for all K E (1)
where Wi (-.+) :) is the 1-Wasserstein distance between two distributions. Moreover, (EED(Il=<( < oo
for each = T, n, then we always have that minimizers exist:
arg min KES*(1)
+ 0.
2.2 Minimization via dual Brunn-Minkowski theory
We now aim to understand the structure of minimizers the above problem. To do this, we first show that the objective in (2) can be interpreted as the dual mixed volume between K and a data-dependent star body. To begin, we start with a definition.
Definition 2.2 (Definition 2* in (54). Given two star bodies K, L E Sd, the i-th dual mixed volume between L and K for E IR is given by
/ (x()))
V((LK)) ga-1
One can think of dual mixed volumes as functionals that measure the size of a star body K relative to another star body L. Note that for all i, V,(K, K) S Px (u) vola(K) K) is the usual d-dimensional volume of K. Of particular interest to us will be the case -1.

---

To see how our main objective can be interpreted as dual mixed volume, we need be able to summarize our distributions D, and D, in such way that can naturally be related to the gauge -I Il Since the gauge is positively homogenous, it is characterized by its behavior on the sphere sd-1. Hence, given distribution, we require a "summary statistic" that describes the distribution in each unit direction u E gd-1, Since star bodies are precisely defined by the distance of the origin their boundary in each unit direction, we ideally would like to summarize the distribution in similar fashion. For a distribution D with density on R. define the map
denslty p
K
Pp(u)= = (f,²
( (pptt)aee) 1/(d+1)
Figure 1: (Left) Contours of the Gaus- sian mixture model density p. (Right) The star body Kp induced by the radial function B
This function measures the average mass the distribution accrues in each unit direction and how far this mass lies from the origin. More preciscly,
can show [52] that
pg+1 1 is the density of the measure pD(() := Ep | 1 (a/ll=]le, = If the map ( is positive and continuous over the unit sphere, defines a unique data-dependent star body. In Figure [ we give an illustrative example showing how the geometry of the data distribution relates to the data-dependent star body. Here, the data distribution is given by a Gaussian mixture model with 3 Gaussians.
We now aim to understand the structure of minimizers of the functional K + F(K; D,, Dn). As we will show in our main result, the map (3) aids interpreting the functional via a dual mixed volume. Extrema of such dual mixed volumes have been the object of study in convex and star geometry for some time. We cite a seminal result due to Lutwak.
Theorem 2.3 (Special case of Theorem 2 in (54) For star bodies K, I. Sd, have
(L, KYd > vola(L)d+1 vola(K))),
and equality holds if and only if K and L are dilates, i.e., there exists X >> 0 such that K = AL.
Armed with this inequality, we show that under certain conditions, we can guarantee the existence and uniqueness of minimizers of the above map. Note that optimal solutions are always defined up to scaling. To remove this degree of freedom, we impose an additional constraint on the volume of the solution. We chose unit-volume constraint for simplicity, but changing the constraint would simply scale the optimal solution.
Theorem 2.4. Suppose D, and Dn are distributions on I& that are absolutely continuous with respect to the Lebesgue measure with densities Pr and Pa. respectively. Suppose Pp. and Pp.. as defined in 3 are continuous over the unit sphere and that Pp. (u) Pp.. (u) 2 for all Sd
Then, there exists star body Lrn E Sa such that the unique solution to the problem
min [D,[[]==][cc]] IED,, (1l2, [|]]
KES*voL
is given by K. vola( (L. -//d T If Bdd C ker(K.), then K. in fact, the unique solution to (2 with the additional constraint vola =
Proof Sketch. One can prove that for each i = r, n. ED, [<]]|x]]]]]]] = Sga- Pp. (u) PK (u)dd.. Then since Pp. > Pp. > on the unit sphere, we have that the map (r. == (u) Pr. (u)d+1)1/(d+1) is positive and continuous on gd-1 We can then prove that it is the radial function new data-dependent star body Lr,n- Combining the previous two observations yields
Ep, ||| x] Ep,, [|| = dV_ (Ir,nn K). Applying Theorem 2.3 obtains the final result. We
defer the full proof to Section B.2
Remark 2.5 (Uniqueness guarantees). We highlight that in our result we are able to obtain uniqueness of the solution everywhere. This notably different than previous guarantees in the Optimal Transport literature, where the optimal transport potential for the I-Wasserstein loss is not unique, but can be
5

---

shown to be unique D, -llmmt everywhere [58]. A significant reason for this is that our optimization problem is over the family of star body gauges, which have additional structure over the general class of 1-Liipschitz, functions. In particular, star bodies are uniquely defined by their radial functions and, hence, dual mixed volumes can exactly specify them (see (54) for more details).
Remark 2.6 (Distributional assumptions). Note that the conditions of this theorem require that the density-induced map P-(-) must be a valid radial function (i.e., positive and continuous over the unit sphere). Many distributions satisfy these assumptions. For example, if the distribution's 's density is of the form p(z)= (1++]] [ where Jo t" )dt < oo, q >> O, and L E S then the map py is valid radial function. See [52] for more examples of distributions that satisfy these assumptions.
Remark 2.7 (Finite-data regime). While the conditions of this theorem would not apply to the case when D, and Dn, are empirical distributions, results are stable in the sense that as the amount of available data goes to infinity, the solutions in the finite-data regime converge (up to subsequence) to a population minimizer. The proof exploits tools from variational analysis, such as I"convergence [14].
prove in the appendix (Section D.I that these
Remark 2.8 (Implications for inverse problems). In the original framework of (53], the authors considered D = A4 (D₂) where A is a linear forward operator in an inverse problem. Since inverse problems are typically assumptions of the theorem. One can solve this, however, by convolving 2,, with a Gaussian to ensure it has full measure.
such a distribution would be singular and not satisfy the
Remark 2.9 (Scaling distributions). One can always reweight the objective so that the assumptions of the theorem are satisfied. This is due to positive homogeneity of the gauge, which guarantees Ep I|l |r| = Ex k] for any distribution D. In particular, if Pp. > Pp. is not satisfied, one can choose a scaling À that Pp. > x3/(d+1) Pp.. Inspecting the proof of Theorem 2.4. one can see that the result goes through when applied to the objective ED, [bp[[]]= [|]]]] - A ED, |||||] K..
2.3 Examples
We now provide examples of optimal regularizers via different choices of DD, and Dan Further examples can be found in the appendix in Section D.3 Throughout these examples, the star body induced by D, and D,, is denoted by L, and Ln. respectively.
Example (Gibbs densities with 4 and f--norm energies). Suppose the distributions D, and D. for some >> are given by Gibbs densities with f and by-norm energies. respectively:
(x) = -llelle (ca vol)(Bi.) and (z) = e Vavāllllee /(ca vola /(avd)).
Here ca == [(d 1) where I()) is the usual Gamma function. The star bodies induced by Pr and Ph are dilations of the bi-ball and Ez-ball, respectively. Denote these star bodies by L, and L respectively. Then, L is defined by the difference of radial functions via
PLe.. (u) = (c-1l=lle, -c.(avV[(lllee) (c
1/(d+1)
where c, := $00 tad exp(-t)/ (//t))/( (ca (Bl)( and cn == = tdd exp( (ep(--t/lee vola vola(B" (Bld(B)((Vaa)) We visualize the geometry of Li for different values of a 2 1 in Figure 2 for d = 2. We see that for directions such that the boundaries of L, and La are far apart (namety, the te, directions), the optimal regularizer assigns small values. This is because such directions are considered highly likely under the distribution Dr and less likely under Da. However, for directions in which the boundaries are close (e.g., in the [0.5, dire-direc,,on), the regularizer assigns high values, since such directions are likely under the noise distribution De This aligns with the aim of the objective function - namely, that a regularizer should assign low values to likely directions and high values to unlikely ones.
Example 2 (Toy inverse problem). Consider the following example where I ~ Dr := NN0, Σ) where Σ E Rdxd and we have measurements y = Ax where E I Bddd with rank m & d. We consider the case when D, := A (D,) @ N(1 o² Ta) where y ~ Dr, if and only if = Ax where & ~ D,,- Let d = 2, Σ UUT, and A [1,0] of E B1x2 Then AT -1. Note that y ~ Dy if and only yf =
of where u. is the first row of and N' I2). By standard properties of Gaussians, D, = (0, 1|u2 l12,). Hence D,,- = NN(0, D.) where D= == diag([|uzl + 0², o²2) E
R2X2 We visualize this example in Figure 3 We see that the regularizer induced by Lyn penalizes directions in the row span of A, but does not for directions in the kernel of A.

---

a 1.3
0 1.6
a 2.3
0.5
-0.5
-0.5
-0.5
Figure 2: We plot L from Example I for different values of (Left) 1.3, (Middle) 1.6,
and (Right) a 2.3. A full figure with L, and L% can be found in Section D.3
Plots Ins and Less
-0.5
0.5
-0.5 0.5
-0.5
Figure 3: We visualize the distributions from Example when E = [0.5477, 0.2739; 0, 0.5477] E
R2x2 d² 0.01, and we set D, := 0.01 diag([|u, 1 o² o²): (Left) the boundaries of L, and Im. (Middle) the boundaries of Ly, Lnn and by,n and (Right) Lr,n.
3 Critic-based loss functions via f-divergences
Inspired by the use of the variational representation of the Wasserstein distance to define loss functions to learn regularizers, we consider whether other divergences can give rise valid loss functions and if they can be interpreted as dual mixed volumes. A general class of divergences of interest will be f-divergences: for convex function f (0, oo) with f(1) AP dQ.
Q.
(y(P))Q) = fR+
a-Divergences: For a E (-00, 0) (0, 1). where f = fo with (a) x" -0--v-(1-- (1-a)
there is variational representation of the a-divergence
69
a(a-1)
h(x)²
Th(x)
hhx))==11)
D,. (PIIQ) D. (PIIQ)
inf
EQ
+ Ep
a(1 a) h.Rd (0,oc)
Note that the domain of functions in this variational representation consists of positive functions, which aligns with the class of gauges || |lx. Many well-known divergences are a-divergences for specific such as the xi-diveree (a = 1) and the (squared) Hellinger distance (a = 1/2) [69] We discuss in the appendix (Section B.3.I) how one can derive general loss based on a-divergences, but focus on particular case here to illustrate ideas.
The Hellinger Distance: Setting = 1/2 and performing an additional change of variables in we have the following useful representation of the (squared) Hellinger distance:
H²(P11Q)
inf Er[h(=)] + Eglh(z)~*).
420
This motivates minimizing the following functional, which carries a similar intuition to 2 that the regularizer should be small real data and large unlikely data:
K + Ep, [||=]|| xc + Ep, [1]=+1=1]
(5)
The following theorem, proved Section B.3 shows that this loss functional equal to single dual mixed volume between a data-dependent star body L, and star body that depends on D,, D,,, and
7

---

K. We can show that the equality cases of the lower bound only hold for dilations within a specific interval, with only two specific star bodies achieving equality.
Theorem 3.1. Suppose Dr and D, admit densities pr and pm, respectively, such that the following maps are positive and continuous over the unit sphere:
u ト PD.(p) I-- (tur(t and u Po. 00 ed
Let Lr and Ln denote the star body with radial function PD, and PD., respectively. Then, for any K E Stt there exists a star body K that depends on Dr. Dn, and such that the functional (5) is equal to 1 Kr We also have the inequality V-1(Lr, a) 20 with equality if and only if K n a dilate of Ly- Moreover, there exists a X, == (D, Dn,) m) > such
- vola ( Lp Id++1//d
that for every X E (0, x,), there are only two star bodies K K+,x and K that satisfy Kr, XLp
(u) ( = 2XPI, (u)
P (u)
d-1
1 1
4x2
PL.
Remark 3.2. Among the two star bodies K+AA that achieve Kr, = ALy, we argue that K induces better regularizer than K-,x- As seen in previous examples, would like our regularizer || || to assign low values to likely data and high values to unlikely data. Equivalently, we would like (-) to be large on likely data and small on unlikely data. This can be stated in terms of the geometry of T., and I.,: we would like pK to be large when pr., is large and Pī., is small. Likewise, PK should be small when Pi., is large and PL, is small. Due to the small sign difference between K+, and K_,, ,. K+,> better captures this intuition. We show visual example of this in Figure where D, and D, are the same distributions as in Example
by and
Ki vo/ 04681
0.5
0.5
1
4.5 0.5
Figure 4: (Left) The star bodies L, and T., induced by the distributions D, and Dn, from from Theorem 3.T and Example [ with a = 0.5. Then we have (Middle) K and (Right) K_,,. as defined in Theorem 3.1 Note that K better captures the geometry of a regularizer that assigns higher likelihood to likely data and lower likelihood to unlikely data, while K K_,, does not.
3.1 Empirical comparison with adversarial regularization
We now aim to understand whether the Hellinger- loss (5) can provide a practical loss function to learn regularizers. To do this, we consider denoising on the MNIST dataset [50). We take 10000 random samples from the MNIST training set (constituting our DD distribution) and add Gaussian noise with variance x² 0.05 (constituting our D,, distribution). The goal is then to reconstruct test samples from the MNIST dataset corrupted with Gaussian noise of the same variance seen during training. The regularizers were parameterized via a deep convolutional neural network with positively homogenous Leaky ReLU activation functions, giving rise to a star-shaped regularizer. They were trained using the adversarial loss and Hellinger-based loss (5). We also used the gradient penalty term from [53] for both losses. Once each regularizer RS has been trained, we then denoise a new noisy MNIST digit y by minimizing + 2| v|I2, + *R)((z) via gradient descent.
We note that in [53] was recommended to set À : 2 where A Ex(0.."7) [|= which was motivated by the fact that the regularizer will be 1-Lipschitz. We found our Hellinger-baat based regularizer performed best with x A= 5. 5.1 We thus also compare to the adversarial regularizer with further tuned parameter X. Please see Section C.T of the appendix for implementation details and further discussion of regularization parameter choices.
As baseline, we compare both methods to Total Variation (TV) regularization
8

---

We show the average Peak Comallto-Noise Ratio (PSNR) and mean squared error (MSE) over 100 test images in the table below. We see that the Hellinger-based loss gives competitive performance relative to the adversarial regularizer and TVV regularization, while the tuned adversarial regularizer slightly outperforms all methods. These promising results suggests that these new a-divergence based losses are potentially worth exploring from practical perspective as well.
PSNR
MSE
Hellinger
23.06
0.0052
Adversarial
22.32
0.0064
Adversarial (tuned)
24.14
0.0041
TV
20.52
0.0091
4 Computational considerations
We now discuss various issues relating to employing such star body regularizers. In particular, we discuss optimization- based concerns due to the (potential) nonconvexity of x +> ||œ|| x and how one can use neural networks to learn star body regularizers in high figh-dimeensios.
Weak convexity: When such regularizer is employed in inverse problems, one may require mini- mizing a cost of the form x + 1 A(z) + >>[[|=]]<). Here ((-) is continuous monotonically increasing function (e.g., (t) = t for 0). While the first term is convex for linear A(.), the second term can be highly nonconvex for general star bodies. One could enforce searching for convex body as opposed to star body, but convexity is strictly more limited in terms of expressibil- ity. There a large body of recent work [24 25
80 that has analyzed weak convexity in optimization, showing that while nonconvex, weakly convex functions can be provably optimized using simple first-order methods in some cases. Based on this, it is important understand under what conditions is dill [|x) weakly convex function. Recall that a function f is p-weakly convex if J() & I12, is convex. For certain functions (6.), there is natural way to determine cc[1 ||x)) is weakly convex.
19
36
In particular, in considering the quantity -1 + fll Il2,, there is natural way in which one can describe this functional as the gauge of a new star body.
Definition 4.1 (55)). For star bodies K, T. Sd and scalars a, > 0 (both not zero), the harmonic qommbination for e is the star body a o K K o L whose radial function is defined by
Paok i (1)
apx (u) -9 + (p)
Observe that the scaling operation a o K differs from the usual aK == {ax : x E K} since app² = -,-- so that a K o
Notice that in our context, we have similar combination given by the harmonic 2-combination of K and the unit ball Bd: M2.p := (2,,(K) = K+2 Bd. By the definition of the harmonic 2-combination, the gauge of M2.p satisfies ||ar|] = || + - We thus have the following Proposition, proven in Section B.4 showing the equivalence between the weak convexity of || 11%k and the convexity of M2,p- We also show a visual example in Section D.4 of data-dependent star body whose gauge squared is weakly convex.
Proposition 4.2. Let K Sd. Then x + || is p-weakly convex if and only if M2.p is convex.
Deep neural network-based parameterizations: Since deep neural networks allow for efficient learning in fighh-imensions, is natural to ask under what conditions does the regularizer Re(x)) == fo o f/ o... .... f (x) define a star body regularizer, i.e., when is the set K₂ == fo E Rd Ra(x) & 1} a star body? As star bodies are in one-to-one correspondence with radial functions, one answer this question by studying conditions under which the map + 1/Ro(u) defines radial function. This leads to the following Proposition, which is proven in Section B.4
Proposition 4.3. For L N, consider regularizer R Rdd + R of the form R(x) Go fl o... fi (2) where G 2d - R and each f: Rd:- + d. for i E [L] satisfy the following conditions:

---

(i) d = do < d1 < < dr,
(ii) G()) is non-negative, positively homogenous, continuous, and only vanishes at the origin,
(iii) ft(.) is injective, continuous, and positively homogenous.
Then the set K == (x E Ed:: R(x) < 1} is a star body in TRd with radial function px == 1/R(u).
Below we discuss the different types of commonly employed layers and activation functions that satisfy the conditions of the above Proposition.
Example 3 (Feed-forward layers). Any intermediate layer Ji Rd:-- - Rdd, can be parameterized by a single-layer feedforward ML. with biases: fe(x) 0z (W,r)) where W. E Rdd &R has rank d-11- For the activation function (,(.), one can ensure injectivity and positive homogeneity by choosing the Leaky ReLU activation LeakyReLUga
(tL aRReL((-) where a E (0, 1). Observe that this map is positively homogenous, continuous, and bijective. The layer fa thus satisfies condition (iii). For the final layer G()), note that any norm on Ra satisfies condition (ii).
Example 4 (Residual layers). If d, = di-1, one could use residual layers of the form f((x) == + ge(xr) where gt RR + Rd, is Lipschitz continuous, positively homogenous function with Lip(g)) < 1. This can be achieved by having 9: be a neural network with 1-Lipschitz positively homogenous activations (such as ReLU), biases, and weight matrices with norm strictly less than 1. Then each fo is positively homogenous, continuous, and invertible [II), satisfying condition (ii)).
Remark 4.4 (Star-shaped regularizers). Note that if one does not require the set K := (i E TR RR(x)) & 1} to be a star body, but simply star-shaped, then there is more flexibility in the neural network architecture used. particular, one simply needs the layers to be positively homogenous and continuous, and the output layer be non-negative, positively homogenous, and continuous. Invertibility would not be necessary, so that the dimensions d, need not be increasing. This can easily be achieved via linear convolutional layers and positively homogenous activation functions. In fact, in the original work (53). experiments were done with network precisely of this form.
Remark 4.5 (Positive homogeneity of activations). While our theory for neural networks mainly applies to positively homogenous activation functions such as ReLU and LeakyReLU, we show in Section of the appendix that focusing on such activations does not limit performance. Specif- ically, we compare networks with non-positively homogenous activation functions (such Tanh or the Gaussian Error Linear Unit (GELU)) with a network exclusively using Leaky ReLU activa- tions in denoising. We show that the Leaky ReLU-based regularizer outperforms regularizers with non non-positively homogenous activations. This potentially highlights empirical benefits of positive homogeneity, which warrants further analysis.
Discussion
In this work, we studied optimal task-dependent regularization over the class of regularizers given by star body gauges. Our analysis focused on learning approaches that optimize a critic-based loss function derived from variational representations of probability measures. Utilizing tools from dual Brunn-Minkowski theory and star geometry, we precisely characterized the optimal regularizer in specifie cases and provided visual examples illustrate our findings. Overall, our work underscores the utility of star geometry in enhancing our understanding of data-driven regularization.
Limitations and future work: There are numerous exciting directions for future research. While our analysis concentrated on formulations based on the Wasserstein distance and a-divergence, it would be valuable to extend the analysis other critic-based losses for learning regularizers. Additionally, it would be interesting to give precise characterization of global minimizers for the Hellinger-inspired loss (5) Relaxing the assumptions in our results is another significant challenge. For instance, Theorem 2.4 hinges on a "ccntainment" property of the distributions, and finding ways to relax this assumption could broaden the applicability of our theory. Moreover, many of our results assume a certain regularity in the densities of our distributions. Addressing this would involve extending dual mixed volume inequalities to operate on star-shaped sets rather than star bodies. Finally, it is crucial to assess the performance of these regularizers in solving inverse problems. This includes studying questions regarding sample complexity, algorithmic properties, robustness to noise, variations in the forward model, and scenarios with limited clean data 22 45
32
51
85
86
21 31)
10

---

References
[1] Alckh Agarwal, Animashree Anandkumar, Prateek Jain, and Praneeth Netrapalli. Learning sparsely used overcomplete dictionaries via alternating minimization. SIAM Journal Opti- mization, 26(4):2775-2799, 2016.
[2] Alckh Agarwal, Animashrce Anandkumar, and Praneeth Netrapalli. A clustering approach to Iearn sparsely-used overcomplete dictionaries. IEEE Transactions on Information Theory, 63(1):575-552, 2017.
[3] Michal Aharon, Michael Elad, and Alfred Bruckstein. K-svd: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing, (A(11):43114422 2006.
[4] Giovanni S. Alberti, Eressto De Vito, Matti Lassas, Luca Ratti, and Matteo Santacesaria. Learning the optimal tikhonov regularizer for inverse problems. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[5] Dennis Amelunxen, Martin Lotz, Michael B. McCoy, and Joel A. Tropp. Living on the edge: Phase transitions in convex programs with random data. Information and Inference: A Journal of the IMA, 3(3):224-294, 2014.
[6] Sanjeev Arora, Rong Ge, Tengyu Ma, and Ankur Moitra, Simple, efficient, and neural algorithms for sparse coding. Conference Learning Theory, 2015.
[7] Simon Arridge, Peter Maass, Ozan Öktem, and Carola-Bibiane Schönlieb. Solving inverse problems using data-driven models. Acta Numerica, 28:1 18:1-174, 2019.
[8] Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. Proceedings of the 37th International Conference on Machine Learning, 2020.
[9] Muhammad Asim, Fahad Shamshad, and Ali Ahmed. Blind image deconvolution using deep generative priors. IEEE Transactions on Computational Imaging, 6:1493 1506, 2018.
[10] Boaz Barak, Jonathan Kelner, and David Steurer. Dictionary learning and tensor decomposi- tion via the sum-of-squares method. Proceedings of the Forty-seventh Annual ACM Symposium on Theory of Computing, page 143-151, 2015.
[11] Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and förn-Henris Jacobsen. Invertible residual networks. International Conference on Machine Learning, 2018.
[12] Badri Bhaskar and Benjamin Recht. Atomic norm denoising with applications to line spectral estimation. IEEE Transactions on Signal Processing, 61(23):59875599) 2013.
[13] Ashish Bora, Ajil Jalal, Eric Price, and Alexandros Dimakis. Compressed sensing using generative models. International Conference on Machine Learning, 2017.
[14] Andrea Braides. A handbook of converggge, Handbook of Differential Equations: Stationary Partial Differential Equations, Volume 3, 2007.
[15] Emmanuel J. Candès and Benjamin Recht. Exact matrix completion via convex optimization. Foundations of Computational Mathematics, 76)777777 2009.
[16] Emmanuel J. Candès, Justin K. Romberg, and Terence Tao. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics, 19(7)) 2006.
[17] Venkat Chandrasekaran and Michael I. Jordan. Computational and statistical tradeoffs via convex relaxation. Proceedings of the National Academy of Sciences, 110(13):E1181-E11990 2013.
[18] Venkat Chandrasekaran, Benjamin Recht, Pablo A. Parrilo, and Alan S. Willsky. The convex geometry of linear inverse problems. Foundations of Computational Mathematics, 12:805-849, 2012.

---

[19] Vasileios Charisopoulos, Yudong Chen, Damek Davis, Mateo Díaz, Lijun Ding, and Dmitrry Drusvyatskiy. Low-rank matrix recovery with composite optimization: good conditioning and rapid convergence. Foundations of Computational Mathemalics, 21:1505-1593, 2021.
[20] Hyungjin Chung, Jeongsol Kim, Michael Mccann, Mare L. Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. International Conference on Learning Representations (ICLR), 2023.
[21] Giannis Daras, Kulin Shah, Yuval Dagan, Aravind Gollakota, Alexandros G. Dimakis, and Adam Klivans. Ambient diffusion: Learning clean distributions from corrupted data, Advances in Neural Information Processing Systems (NeurIPS), 2023.
[22] Mohammad Zalbagi Darestani, Akshay S, Chaudhari, and Reinhard Heckel. Measuring ro- bustness in deep learning based compressive sensing. International Conference Machine Learning (ICML), 2021.
[23] Ingrid Daubechies, Michel Defrise, and Christine de Mol. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics, 57(11))1413-144 2004.
[24] Damek Davis and Dmitriy Drusvyatskiy. Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization, 29(1):207-233 2019.
[25] Damek Davis and Dmitriy Drusvyatskiy. Proximal methods avoid active strict saddles weakly convex functions. Foundations of Computational Mathematics, 22:561-600 153561666.660 2022.
[26] David Donoho. For most large underdetermined systems of linear equations the minimal I1- norm solution is also the sparsest solution. Communications Pure and Applied Mathematics, 59(6):797-82) 829, 2006.
[27] Michael Elad. Sparse and redundant representations: From theory to applications in signal and image processing. Springer, 2010.
(28) Zhenghan Fan, Sam Buchanan, and Jeremias Sulam. What's in prior? learned proximal networks for inverse problems. International Conference on Learning Representations (ICLR), 2024.
[29] Maryam Fazel. Matrix rank minimization with applications. Ph.D. Thesis, Department of Electrical Engineering, Stanford University, 2002.
[30] Berthy T. Feng, Jamie Smith, Michael Rubinstein, Huiwen Chang, Katherine L. Bouman, and William T. Freeman. Score-based diffusion models principled priors for inverse imaging. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10520-10531 10531, 2023.
[31] Angela Gao, Jorge Castellanos, Yisong Yue, Zachary Ross, and Katherine Bouman. Deepgem: Generalized expectation expectation-maxiiiieii maximization for blind inversion. Advances Neural Information Processing Systems (NeurIPS), 14:11592-11603, 2021.
[32] Angela Gao, Oscar Leong, He Sun, and Katherine L. Bouman. Image reconstruction without explicit priors. ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[33] Richard J. Gardner. Geometric tomography. Cambridge: Cambridge University Press, 2006.
[34] Richard J. Gardner, Markus Kiderlen, and Peyman Milanfar. Convergence of algorithms for re- constructing convex bodies and directional measures. The Annals of Statistics, 34(3)133311137, 2006.
[35] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances Neural Information Processing Systems, 2014.
12

---

[36] Alexis Goujon, Sebastian Neumayer, Pakshal Bohra, Stanislas Ducotterd, and Michael Unser. neural-network-baree convex regularizer for inverse problems. IEEE Transactions on Com- pulational Imaging, 9:781 - 795, 2023.
[37] Alexis Goujon, Sebastian Neumayer, and Michael Unser. Learning weakly convex regularizers for convergent reanstrrccon algorithms. SIAM Journal on Imaging Sciences, 17(1):91-
115, 2024.
[38] Adityanand Guntuboyina. Optimal rates of convergence for convex estimation from support functions. Annals of Statistics, 40(1):385 1,1, 2012.
[39] Paul Hand, Oscar Leong, and Vlad Voroninski. Phase retrieval under a generative prior. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
[40] Paul Hand, Oscar Leong, and Vladislav Voroninski. Compressive phase retrieval: Optimal sam- ple complexity with deep generative priors. Communications Pure and Applied Mathematics, 77(2):14471122,, 2024.
[41] G. Hansen, I. Herburt, H. Martini, and M. Moszynska. Starshaped sets. Aequationes Mathemat- icae, 94:1001-1092, 2020.
[42] Reinhard Heckel, Wen Huang, Paul Hand, and Vladislav Voroninski. Rate-optimal denoising with deep neural networks. Information and Inference: A Journal of the IMA, 12(4):125 2021.
[43] Rakib Hyder, Viraj Shah, Chinmay Hedge. and M. Salman Asif. Alteratinn phase projected gradient descent with generative priors for solving compressive phase retrieval. Proc. IEEE Iut. Conf. Acoust., Speech, and Signal Processing (ICASSP), 2019.
[44] Leonid Ruddin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlineur Phenomena, 60:259-268, 1992.
[45] Ajil Jalal, Marius Arvinte, Giannis Daras, Eric Price, Alexandros G. Dimakis, and Jon Tamir. Robust compressed sensing mri with deep generative priors. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[46] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. International Confer- ence on Learning Representations (ICLR), 2014.
[47] Erich Kobler, Alexander Effland, Karl Kunisch, and Thomas Pock. Total deep variation for linear inverse problems. IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[48] Frederic Koehler, Lijia Zhou, Danica J. Sutherland, and Nathan Srebro. Uniform convergence of interpolators: Gaussian width, norm bounds, and benign overfitting. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[49] Gil Kur, Alexander Rakhlin, and Adityanand Guntuboyina. On suboptimality of least squares with application to estimation of convex bodies. Conference Learning Theory, pages 2406-2424, 2020.
[50] Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2788-2324, 1998.
[51] Oscar Leong, Angela Gao, He Sun, and Katherine Bouman. Discovering structure from corruption for unsupervised image reconstruction. IEEE Transactions on Computational Imaging, 9:992-1005, 2023.
[52] Oscar Leong, Eliza O' O'Reilly, Yong Sheng Soh, and Venkat Chandrasekaran. Optimal regular- ization for data source. arXiv preprint arXiv:2212.13557,, 2022.
[53] Sebastian Lunz, Ozan Öktem, and Carola-Bibiane Schönlieb. Adversarial regularizers in inverse problems. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
[54] Erwin Lutwak. Dual mixed volumes. Pacific Journal of Mathematics, 58(2):531-538, 1975.
13

---

[55] Erwin Lutwak. The brunn-minkowski firey theory ii: Affine and geominimal surface areas. Advances Mathematicc, 118(2):244 294, 1996.
[56] Julien Mairal, Francis Bach, and Jean Ponce. Sparse modeling for image and vision processing. Foundations and Trends in Computer Graphics and Vision, 8(2-3):85-283 2014.
[57] Morteza Mardani, Enhao Gong, Joseph Y Cheng, Shreyas S Vasanawala, Greg Zaharchuk, Lei Xing, and John M Pauly. Deep generative adversarial neural networks for compressive sensing mri. IEEE Transactions on Medical Imaging, 38(1): 167--79, 2019.
[58] Tristan Milne, EEteenee Bilocq, and Adrian Nachman. A new method for determining wasser- stein optimal transport maps from kantorovich potentials, with deep learning applications. arXiv preprini (tt.iv:22/1.002222 2022.
[59] Subhadip Mukherjee, Sören Dittmer, Zakhar Shumaylov, Sebastian Lunz, Ozan Öktem, and Carola-Bibiane Schönlieb. Learned convex regularizers for inverse problems. arXiv preprint arXiv:2008.0223 02833, 2021.
[60] Bruno A. Olshausen and David J. Field. Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381 (8166683)66076609, 1996.
[61] Bruno A. Olshausen and David J. Field. Sparse coding with an overcomplete basis set: A strategy employed by vI? Vision in Reseurch, 37(22):331 1997.
[62] Gregory Ongie, Ajil Jalal, Christopher A. Metzler, Richard G. Baraniuk, Alexandros G. Dimakis, and Rebecca Willett. Deep learning techniques for inverse problems imaging. IEEE Journal on Selected Areas in Information Theory, 1(1):39-56, 2020.
[63] Eliza O' O'Reilly and Ngoc Mai Tran. Stochastic geometry to generalize the mondrian process. SIAM Journal on the Mathematics of Data Science, 4(2):531 2022.
[64] Samet Oymak and Babak Hassibi. Sharp mse bounds for proximal denoising. Foundations of Computational Mathematics, 16:965-1029, 2016.
[65] Samet Oymak, Benjamin Recht, and Mahdi Soltanolkotabi. Sharp time-data tradeoffs for linear inverse problems. IEEE Transactions on Information Theory,
2018.
[66] Yaniv Plan and Roman Vershynin. The generalized lasso with non-linear observations. IEEE Transactions on Information Theory, 62:1528-1533 2016.
[67] Yaniv Plan, Roman Vershynin, and Elena Yudovina. High-dimense figh--dimeniionl estimation with geomet- ric constraints. Information and Inference: Journal of the IMA, 6(1):1-40, 2017.
[68] David Pollard.
Convergence of stochastic processes. Springer- Verlla. 1984.
[69] Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning. Cambridge University Press, 2023.
[70] Benjamin Recht, Maryam Fazel, and Pablo A. Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review, 52(3):471-501, 2010.
[71] Yaniv Romano, Michael Elad, and Peyman Milanfar. The little engine that could: Regularization by denoising (red). SIAM Journal on Imaging Sciences, 1044)4180411844, 2017.
[72] Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alexandros G. Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems (NeuriPS), 2023.
[73] A.M. Rubinov. Radiant sets and their gauges. In: Quasidifferentiabiiiyy and Related Topics, 43:235-261 2000.
[74] Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: nonlinear phenomena, 60(1-4):259-266 1992.
14

---

[75] Karin Schnass. On the identifiability of overcomplete dictionaries via the minimisation principle underlying k-svd. Applied and Computational Harmonic Analysis, 37(3):464 491, 2014.
[76] Karin Schnass. Convergence radius and sample complexity of itkm algorithms for dictionary learning, Applied and Computational Harmonic Analysis, 45(1):22-58, 2016.
[77] Rolf Schneider. Convex bodies: The brunn-minkowski theory. Cambridge: Cambridge University Press., 2013.
[78] Parikshit Shah, Badri Narayan Bhaskar, Gongguo Tang, and Benjamin Recht. Linear system identification via atomic norm regularization. Proceedings of the 51st Annual Conference on Decision and Control, 2012.
[79] Fahad Shamshad and Ali Ahmed. Compressed sensing- based robust phase retrieval via deep generative priors. IEEE Sensors Journal, 21(2):2286 2298, 2017.
[80] Zakhar Shumaylov, Jeremy Budd, Subhadip Mukherjee, and Parola-Bibiane Schönlieb. Weakly convex regularisers for inverse problems: Convergence of critical points and primal-dual optimisation. arXiv preprint arXiv:2402.01055, 2024.
[81] Yong Sheng Soh and Venkat Chandrasekaran, Learning semidefinite regularizers. Foundations of Computational Mathematics, 19:375-434, 2019.
[82] Yong Sheng Soh and Venkat Chandrasekaran. Fitting tractable convex sets to support function evaluations. Discrete and Computational Geometry, 66:510-555 2021.
[83] Mahdi Soltanolkotabi. Structured signal recovery from quadratic measurements: Breaking sample complexity barriers via nonconvex optimization. IEEE Transactions on Information Theory, 65(4))23774 2400, 2019.
[84] Daniel A. Spielman. Huan Wang. and John Wright, Exact recovery of sparsely useed dictionaries. Conference on Learning Theory, 23(3))11-18, 2012.
[85] Julián Tachella, Dongdong Chen, and Mike Davies, Unsupervised learning from incomplete measurements for inverse problems. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[86] Julián Tachella, Dongdong Chen, and Mike Davies. Sensing theorems for unsupervised learning linear inverse problems. Journal of Machine Learning Research, 24(39): 2023.
[87] Gongguo Tang, Badri Narayan Bhaskar, Parikshit Shah, and Benjamin Recht. Compressed sensing off the grid. IEEE Transactions on Information Theory, 59(11):7465-7490, 2013.
[88] Ryan Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B, 58:267-288, 1994.
[89] Andrey Nikolayevich Tikhonov. On stability of inverse problems. Dokl. Akad. Nauk SSSR, 176-1177-177,, 1943.
[90] Yann Traonmilin, Rémi Gribonval, and Samuel Vaiter. A theory of optimal convex regularization for www-dimensional recovery. Information and Inferenee: A Journal of the IMA, 13(2)riaae013, 2024.
[91] Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. Plug-and-plle priors for model based reconstruction. In 2013 IEEE Global Conference Signal and Information Processing, pages 945-948 IEEE, 2013.
[92] Cédric Villani. Topics in optimal transport. Providence, RI: American Mathematical Society, 2003.
15

---

Contents
Introduction
1.1
Our contributions
1.2
Related work
Notation
2
Adversarial star body regularization
2.1
Existence of minimizers
2.2
Minimization via dual Brunn- Minkowskk theory
2.3
Examples
3
Critic-based loss functions via -divergences
Empirical comparison with adversarial regularization
Computational considerations
Discussion
10
Further discussion related work
16
Proofs
17
B.1
Preliminaries on star and convex geometry
17
B. 2
Proofs for Section
18
Proofs for Section
20
B.3.1
A general result for a-divergences
B. 4 Proofs for Section|4
24
Experimental details
Hellinger and adversarial loss comparison on MNIST
24
C22 Analysis positive homogeneity
25
Additional results
25
D.I Stability results
25
D.2 Proof of Theorem D.2
27
................................
D.3 Further examples for Section
28
D.4 A star body with weakly convex squared gauge
31
A
Further discussion on related work
We survey topics relating data-driven regularization and convex geometry for inverse problems. For more in-depth discussion on learning- based regularization methods, we refer the reader to the following survey articles [7] 62]
Regularization via generative modeling:
Generative networks have demonstrated their effec- tiveness signal priors in various inverse problems, including compressed sensing [45] [3 57]
16

---

and blind deconvolution [9. Early approaches in this area leveraged "push-forward" generators such as Generative Adversarial Networks (GANs) B5 and Variational Autoencoders (VAFs) 46 In some cases, generative models can serve as traditional regularizers by providing access to likelihoods, which can incorporated as penalty functions into the objective [8]. Alternatively, they can constrain reconstructions to lie on the natural image mani- fold, ensuring that solutions are realistic and consistent with the underlying data distribution. More recently, there has been a surge of interest in techniques based on diffusion models 20 30 21 72).
denoising [42], phase retrieval B3 40 43 79,
Explicit regularization: There have also been recent works that have directly learned regularizers as penalty functions in data-driven fashion. Early works in this area include the now-mature field of dictionary learning (see the surveys [56] 27 for more). Regularization by Denoising [1] constructs an explicit regularization functional using off-the-shelf deep neural network-based denoiser. The works [53] 55 80] learn a regularizer via the critic-based adversarial loss inspired by the Wasserstein distance. The difference in these works lie in the way the neural network is parameterized to enforce certain properties, such as convexity or weak convexity. Recently, (28) developed an approach that can approximate the proximal operator of potentially nonconvex data-driven regularizer. Other works along these lines include 36 37.
Convex geometry: Our work is also a part of a line of research showcasing the tools of convex and star geometry in machine learning. In statistical inference tasks and inverse problems, the utility of convex geometry cannot be understated. In particular, complexity measures of convex bodies, such their Gaussian width [18 17 statistical dimension [5]. have proven useful in developing sample
complexity guarantees for a variety of inverse problems [64 66 48 65 83]. Additionally, convex
geometry has been fruitful in problems such as function estimation from noisy data [8
82
49].
B Proofs
B.1 Preliminaries on star and convex geometry
In this section, we provide relevant background concerning convex bodies and star bodies. For more detailed treatment of each subject, we recommend [77 for convex geometry and [3 for star geometry.
We say that closed set K C IRd is star-shaped (with respect to the origin) if for all K, the line
segment [0, x == (tz E [0, 1]} S K. In order for to be considered a star body, we require K be compact star-shaped set such that for every o the ray R2 == {ta >> 0} intersects the boundary of K exactly once. Note that every convex body (a compact, convex set with the origin its interior) is star body.
In an analogous fashion to the correspondence between convex bodies and their support functions, star bodies are in one-to-one correspondence with radial functions. For star body K, its radial function is defined by
pk (x) == sup(t > -
It follows that the gauge function of K satisfies ||=|| 1/px(<z) for all x E Rd such that a + 0. The radial function can also help capture important geometric information about K. For example, one can show that vola(K) = à fge- p% (u)du.
For K, L E Sd, it is easy to see that K C L if and only if Px & PL. We say that L is a dilate of K if there exists À > 0 such that L = XK; that is, PL = App. In addition, given a linear transformation & GL((R)), one has (x)(x) px ( 2x) for all x E Rd
An additional important aspect of star bodies is their kernels. Specifically, we define the kernel of a star body K as the set of all points for which K is star-shaped with respect to, i.e., ker K) := {a E K : [x, y] C K, Vy E K}. Note that ker (K) is a convex subset of K and ker(K) = K and only if K is convex. For a parameter 7 > well-conditioned star bodies with nondegenerate kernels:
define the following subset of S consisting of
(t) = {K E Sa 17B" S kee(K)))
17

---

Here B : E ||alle, & 1t is the unit (Euclidean) ball in R
We finally discuss metric aspects of the space of star bodies. While the space of convex bodies enjoys the natural Hausdorff metric, there is a different, yet more natural metric over the space of star bodies dubbed the radial metric. Concretely, let K, L. IR be star bodies. We define the radial sum + between K and L as K+L := {a + I E K, E L, Ay}: that is, unlike the Minkowski sum, we restrict the pair of vectors be parallel. The radial sum obeys the relationship (x)L(u) = (K( + pz. (u). We denote the radial metric between two star bodies K, L as
6(K, L) == in > 0: K C C L CSKKte -K
In similar fashion to how the support function induces the Haussdorf metric for convex bodies, the radial metric satisfies 8KK, L) == llk - PL.loo.
A functional F S RR on the space of star bodies is continuous if it is continuous with respect to the radial metric, i.e., F-) is continuous if for every K S and (K;) such that 6(K,, K) - 0 as i + oc, then F + F() as i + oo. Moreover, a subset C S closed if it is closed with respect to the topology induced by the radial metric. Finally, a subset C Sd is bounded if there exists an 0 < R < oo such that K C RB for every K E C.
B.2 Proofs for Section
We require some preliminary results for proofs. The first is simple extension of result in 52 to show continuity of our objective functional.
Lemma B.1. Let D, and D,, be distributions over Rdd such that Epp(llerlled) oo for r, n. Then for any K, L E (T)), have
|F(K; D.) F(L,D,,) Ep, [Ix=]|ee]]] Eo, [| 8(K, L).
72
Proof of Lemma B.1 We first note via the triangle inequality that
[F(K; Dr, D,) F(, < Ep, ||| k] Er, [|]=]|=]] + Ep, [[]]|]]]]]]]] - Ep, [[]]]
Then the result follows from Corollary 2 in [22
We additionally need the following local compactness result over the space of star bodies.
Theorem B.2 (Theorem 5 in (22) Fix 0 7 and let & be a bounded and closed subset of (T))). Let (K,) be a sequence of star bodies in C. Then (K:) has subsequence that converges in the radial and Hausdorff metric star body K E C.
Proof of Theorem 2.7 The condition Ep, ]e Ex, [Il=]led] < oo guarantees that the expectation of ||=||| x exists for any K E (1). To see this, note that since K E (1),,, we have that Bd C ker(K) () so that ||> |xx & || for all x E Rd. Thus
[|=]]]] & [ll < 00 for = n.
To prove the existence of minimizers, we first make the following useful connection regarding the minimization problem and the Wasserstein distance. Use the shorthand notation F(K) F(K; (T, D,). Recall the dual Kantorovich formulation of the 1-Wasserstein distance [92]
Wi(D,,Dn)
sup ED,, Ep, [f()).
/ELip(1)
Since (1I KE S*()) C Lip(1), we see that
inf F(K) (lIkELpp(1) KES4(1)
inf [o][|]]] - Ep, [x]
inf seLip(1)
[f, - Ep,]f(=))
sup
Ep, [f(x)) [p.(((a))
feLip(1)
= -D, D,).
18

---

This will prove to be useful in the existence proof.
To show the existence of minimizer, only consider the case when Wi (D,, D,) 0 since if Wi (D,, (D), O, then any K (1) is minimizer of the objective since F(K) for all K E (1). Consider a minimizing sequence (K;) C (1) such that
inf F(K) as + oc. F(K:) KES*(1)
We claim that this sequence must be bounded, i.c., there exists R such that K₁ RB4 for all i > 1. Suppose this is not true. Then, for any / = 2,2..... there exists subsequence (Ki, S (K,) such that K;, 2 LB for all E. That is, K3, grows arbitrarily large as l - cc.
By Lemma B. we know that F(.) is continuous. Thus, also must have that F(Ki)) -
(1)
F(K) as l + co. But since tBd S Kiç, we must have that | & } Elaalle, for any
Rd. This gives
?
[Dpllll d Ep, Illlll,, F(K;)) >> ED, ED,, EDD [112x
Taking the limit as l + oc shows
inf KES
F(K)= 0.
But this a contradiction since
inf KESa(1)
F(K) WD,D..D.) >
by assumption. Thus, we must have that the sequence (K;) is bounded, i.c., there exists an such that K; S RB4 for >
To complete the proof, observe that this minimizing sequence (K,) C == {K E S*(1: K₁ c RB4), which is closed and bounded subset (1). By Theorem B.2 this sequence must have a
convergent subsequence (K,,) with limit K, C (1). ). By continuity of F(.), this means
lim F(K,,,) F(K.) inf F(K)
6n +00
KES*(1)
desired.
Proof of Theorem 2.4 We first focus on proving the identity
ED, [[]lll] = -f Pp. (u) Px (u
du for i r, n.
We prove this for D,,, since the proof is identical for D.. Observe that for any Sd,, since ||u]| = pk(u) integrating in spherical coordinates gives
Ep.[|=/|a|= = - (e]lxpp((2dd:
J. ts (lulp,,
J (fo0 (tw)dt) ||ul
[ Pp. (u) px -u. -du.
Now, applying this identity to our objective yields
Ep, [lll//l Ep, [||=||]] ( (Po. - Ppn (w)dtt))pp (u) -t
Since Pp. Ppr > 0 on the unit sphere, we have that the map (r. = (pp. (u)d+1
Pp. (1((ti) is positive and continuous the unit sphere. It is also positively homogenous
19

---

of degree -1, as both Pp. and Pp.. are. Then, we claim that Pr, is the radial function of the unique star body
Lnn {o R² (r.n(x) 1}.
We first show that Legn is star-shaped with gauge || Pr.n(x) Note that the set is star- shaped since for each Lr,,, we have for any & [0,1];
Pri (nn(t (pr. (x)/t))))- = (pri(x) Et
since Pr,n. is positively homogenous of degree -. Hence [0, x] C Lr,n for any x br,n. Then, have that the gauge L satisfies
|| k.,. = inf(tt: bra) inf(t: (a/( <
= =in((l: (1) < (r,n
Hence the gauge of Lyg is precisely Pr. (xx) Moreover, by assumption, u -+ Pr.x (u) is positive and continuous over the unit sphere so Lr.n is a star body and is uniquely defined by Pr.a.
Combining the above results, we have the dual mixed volume interpretation
Ep, ||| [x] Ep, [1]]]]]]] [
J Prin(u)at px(u) = (Lnmsk)
Armed with this identity, we can apply Theorem 2.3 obtain
[p,||<<> - Ep, [|x]]]]]]]]] > d vola ( (dd1/// vola(K) -
with equality if and only if Ty,, and are dilates. Hence the objective is minimized over the collection of unit-volume star bodies when K. == vola(Lmn)) Ly
B.3 Prool's for Section 3
Proof of Theorem 3./
Let PD, (u) denote the function induced by D, via equation 3 We first
understand Ep, K
E
| K
|
K Pm (z)drr
Rd
p (ru)ddt
ga
PD (1)dd11|ulli du
ga-1
7D, (1) d-1 px (u)du
Sa
where here we have defined PD. (u) (Jo0 ed (tu)dt) 1/(a-1)
Then, can further write
/ pk(u) (1) (w)
Ep, [||=|||]] ED, |||
pk(u)ĀD, du
ga-1
gd-
Iga-1 pk(u) PD. (w)
pk(u)- "D, (1) du
PD. (D, at)
(D)(u)dt) PD.
[ (k((M) + pk
PD. (2)
PD. (u) ldu
PD. PD. (u)dtT
=: (Lr, Krn)
where the star body Kr, is defined by its radial function
u + (k(())-1+ + (x( PD. (w)d-1
PD. (D.(u)d+) (u)
20

---

Note that Kr, is the L1 harmonic radial combination of K and the star body with radial function PD. (u) + PR (u)pp, (wja-I- Let L, denote the star body with PD, as its radial function and Ln denote the
star body with radial function (D., The equality in the lower bound on V-1 ("K,,r,n L,) is achieved by a star body K such that for some 0,
pk(u) + px()) Pi. (u)d-1 1
PL, (u)otI (u)
This is equivalent to
Pi, (u)d-1
PL, PL, (u)d PK PK(U)2 - Pk(u) + (p),
For notational simplicity, for u E gd-1 let a(u) = -r., and b(u) = Pi., (u). Then for > 0,
K MIr = X Mu)
a(u)d PK Pk(u) (u)2 px(u) Aa(u) 0, Vu E gd-1
To solve for Pk (u), we must analyze the roots of the polynomial f(x) = X b())0-1 But its roots are given by
a(w)d x² x + Aa(u).
d-1
+
4X2
b(u)
a(u)
T. : (u)
2Ab(u)d-2 a(u)²
Recall the requirement that Pk must be positive and continuous over the unit sphere. For each gd-1, f has 2 positive roots if the discriminant is positive and positive root if it is Otherwise, both roots are complex. Hence, in order to have positive solutions for each u, we must have that
(b(u) aa(u)
d-1
1
[((u) b(u)
d-1
1 1 412
0, Vu W -
Yu E S
Set
a(u)) d-1
:= X. (D,, Da) min
0.
wEgd-1
b(u)
Observe that X, positive and finite since Pt, and PL. are positive and continuous over the unit sphere. Then for any À E (0, A.], there are two possible star bodies K K K_,x that satisfy the above equation:
(K) = PL. (u)a 2dpï, (u)a-1
Pi,, (u) PL, (u)
d-1
1+ 1 4X2
PK_ (1) = PL. (u)a
Pi., (w) d-1
1 1-
-22
2Ap;., (u)d-1
PL (u)
Notice that for any u, that achieves the minimum in the definition of X,, we have that PK+... (u.) = PK-,a. (u.).
B.3.1
general result for a-divergences
As discussed in the main body of the paper, we are interested in deriving critic-based loss functions inspired by variational representations of divergences between probability measures. For general convex functions f. there well-known variational representation the f-divergence Dy defined via its convex conjugate (2) == supzes zy fxx).
21

---

Proposition (Theorem 7.24 in (69]). For f convex. let f denote its convex conjugate and S2.
denote the effective domain of f*. Then any f diiiergeence has the following variational representation
Dy(PIIQ)
sup Erlg] R o g).
g:Rd-o.
When applying this result to a-divergences with f(x) = fa(z) -² -"-as-(1-a) saw that it admits the representation In particular, for (-o0, 0) (0, 1), we can consider analyzing the following loss:
a(a-1)
[1]=[]]] + -1--)) ED, -===1111
One can show that this loss can be written in terms of dual mixed volumes. When a E (0, 1). the loss can be written as a single dual mixed volume. We will define the following notation, which will be useful throughout the proof: for a distribution D with density pD and B E we
PB,D(1) = (oo jd+8 1 "po(tu)dtt
We prove the following:
Theorem 5. Fix E (-00, (0,)))0))))))) (0, 1). For two distributions Dr and Dn with densities pr and pu. suppose that Pa,D, and Pa-1.D,, and are positive and continuous over the unit sphere. Let L and L, denote the star bodies with radial functions Pa,D, and P respectively. Then there exists star bodies LØ and Lg that depend on D,, Dn, and K such that
a- -=,E1,ll=111 + (1 a)- EDD |||===1< = av K) + +1 (1 K).
a (0, 1), then there exists body K that depends D, Dn,, and K such that
a [1= + (1 (1-a) (1) ED, ||==11>
( pmm T.))
where V-a K) the -a-dual mixed volume between and K. Moreover, we have the inequality V- K > vola(L;)) (ld+o//d vola(KF,.) )aa/
with equality if and only if KG,, is dilate of L, Any star body K such that K is dilate of LE must satisfy the following: there exists a X >> 0 such that
a (()) + (1 a)- PL (u)
pk(u) (u) 1/a for all gd-1
PL; (u)d-a
Proof of Proposition Consider the loss
"E0), Ep, [||==1]]] (1 a)) ED, ||===110
We first focus Ep,
||| J. Observe that
ED. [pp.[1111 [|0|||]]
JuN (
ga-1 ydta-1 pp (tu)dt du
(u) |12|||1110 ||ul|xde du
Ja-1
Jg Pa.D. (u) (k( p
pn 2u)
Let i. be the star body with radial function
Pie (u) = (pa,D, (u)++a px -a)
Indeed, this is valid radial function since Piş is positively homogenous degree -1 1 and is positive and continuous over the unit sphere. This shows that
Ep, [|]=]]]]] = fS- Pi. Pi.(() (k(()) -td -V.,(ī;, K).
(x()
22

---

Similarly, we see that
|001 Pn (z)d2
R2
jota- (a(Lu)dd(lul12 du
sa
So
ga-1 Pa- Pa-1,D, (u) 1 2 |ull
Jg Pa- (u))t px(u)2 pxx ( du.
Let ï.g be the star body with radial function
Pig (u) = (u)
(ix))taappx(() (x(())=a) atr
This shows that
ED, [||=|] 1-11 1 f Pig (w) px((u) (x)) .V- K).
For the final result, note that if a (0, 1), then
a- 1B,_1ll011
(1 11 a)_ "DD'EE,,lllle [|=,111
{
Sa a Pig (igg(u) (w) ( (u) + (1 (1- a1l--a)'pppp Pig (ig'ii((()))) (u)
PK (u)du
-1
uoa
+ (1 - a) Pa 1, (2) pk (x-])t---x(uu---- PK (x((u) (u) dd
a pk(U) + (1 - a) Pa- (u)
J Pa,D, (w)
Pa.D, 10)40 px (x((1)= du
=. K7Tz)
where we have defined the star body K via its radial function
u + a px-)) + (1 - a) -1 Pa- 1,D, (u)dta-1
Pa.D, (u)dta PK px(u)=-0)
and we also use the du-dual mixed volume, defined by
1 / V_, K) == / gu pr(u) pk a
The general dual mixed volume inequality of Lutwak (Theorem 2 in reads
V (L. K) 1 (L,K)VZ
(L, i< (L, sejjkhijjkke R. L,
Setting a. 0, k gives
(L, > vold(L)(d+a)/ vol,(K)
with equality if and only if K is a dilate of L. Applying this in our scenario gives
V-a (LP, Ky,)) vola vol,(K?,,) -add with equality + K XL
This only holds if there exists a K and X > 0 such that the radial function of K satisfies
pxxpp(() px(u) aa + (1 a) PLo (1)
-a
PLg (ujdta PK (w)1=a)
(1) for all E gd-1
23

---

B.4 Proofs for Section
Proof of Proposition 4.2 Suppose M2 is convex. Then its gauge || . is convex function. One can show that for any non-decreasing convex function & and non non-negative convex function f. the composition Φ f convex. Setting f(x) = || and <(t) f² shows that || Ma. convex.
Recalling that || liv = || + 111= implies that || is p-weakly convex.
On the other hand, suppose ||>1 is p-weakly convex. Then || is convex. This implies that the level set 5g1 = {or E Rd: || & 1 is convex. But notice that | & 1 and only M|
so
S (x E : | 1 fxr IR: || < { M2.p
which implies Mz,p is convex.
Proof of Proposition 4.3 Observe that R(:)) is positively homogenous as the composition of posi- ) is positively homogenous of degree -1. Then, tively homogenous functions. Hence + 1x)(z) we claim that
min R(u) > 0.
Note that since each fü injective, positively homogenous, and continuous, only vanishes 0 and is non-zero otherwise. Hence we have that for any E gd-1, ft. a .... fi fu( & so that R(u) of 0. Moreover, since R(-) is continuous and @d-1 is compact, this minimum must be achieved by some MA E 8d-1 If R(uu uo) = 0, then this would contradict injectivity of each layer and positivity of g()). We conclude by noting + (/R(u) continuous over Sd-1 since R(u) is continuous the composition of continuous functions and it positive over Sd-1, This guarantees K' is a star body.
Hence R(ua)
Note that this proof also shows that R(-) is coercive, since for any sequence (Irk) C Rd: ||allle as k + 0o, we have by positive homogenity of R()..
R(xx) = | R(zw/|<xlll)) e || min WESA-1
R(u) + oo as k + co.
C Experimental details
C.1 Hellinger and adversarial loss comparison MNIST
We provide additional implementation details for the experiments presented in Section 3.1 For training data, take 10000 random samples from the MNIST training set, constituting D, distribution. We then add Gaussian noise of variance o² 0.05 to each image, constituting our D, distribution. The regularizers were parameterized via deep convolutional neural network. Specifically, the network has 8 convolutional layers with LeakyReLU activation functions and an additional 3-layer MLP with LeakyReLU activations and bias terms. The final layer is the Euclidean norm. This network implements a star-shaped regularizer, as outlined in the discussion of our Proposition 4.33 The regularizers were trained using the adversarial loss and Hellinger- bastnger--baeedddd based loss (5). We also used the gradient penalty term from [53] for both losses. We used the Adam optimizer 20000 epochs and learning rate 10 The experiments were run on single NVIDIA A.00 GPU.
After training the regularizer Ra, we then use it to denoise noisy test samples = 3o + 2 where mo is random test sample from the MNIST training and NN(0, TTT with o² 0.05. We do this by solving the following with gradient descent initialized at y:
min | vlli, + XRo(x).
sERd
We ran gradient descent for 2000 iterations with a learning rate of 10-3, For the choice of reg- ularization parameter A. we note that in (53), the authors fix this value to be A 2 where

---

:= 12( [|]= as the regularizer that achieves a small gradient penalty will be (approx- imately) 1-Lipschitz. For the Hellinger-based network, we found that À 5.1X² gave better performance, so we used this for recovery. This may potentially be due to the fact that the regu- larizer learned via the Hellinger-based loss is less Lipschitz than the one learned via adversarial regularization. We additionally hypothesize this may also result from the fact that the Hellinger loss and adversarial loss weight the distributions D, and D, differently, resulting in different regularity properties of the learned regularizer. As such, we decided to additionally tune the regularization strength for the adversarially trained regularizer and found À 0.75A performed better than the original fixed value. Further studying these hyperparameter choices and how they are influenced by the loss function is an interesting direction for future study.
C.2 Analysis on positive homogeneity
We conducted the same experiment in Section 3.1 The main difference is that changed the activation functions used in the network. Specifically, we use the same deep neural network discussed in Section C.I. but considered the following four activation functions: LeakyReLU, Exponential Linear Unit (ELU), Tanh, and the Gaussian Error Linear Unit (GELU) activation. In Table I we show the average PSNR and MSE of each regularizer the same 100 test images used Section 3.1 We see that the Leaky Rel-U-based regularizer outperforms regularizers using non-positively homogenous activation functions.
PSNR
MSE
Leaky ReLU
22.32
0.0065
ELU
20.74
0.0090
Tanh
13.66
0.0431
GELU
18.90
0.0157
Table 1: We show the average PSNR and MSE in recovering 100 test MNIST digits using regularizers
trained via the adversarial loss. Each regularizer was parameterized by convolutional neural network utilizing single activation function from the following options: Leaky ReLU, ELU, Tanh, or GELU, The Leaky ReLU-based regularizer achieves the highest PSNR and lowest MSE.
D Additional results
D.1 Stability results
We now illustrate stability results showcasing that if one learns star bodies over a finite amount of data, the sequence will exhibit a convergent subsequence whose limit will be solution the idealized population risk. We prove this result when the constraint set corresponds to star bodies of unit-volume with Bdd ker(K). Define
(7, (K E E 7B C ker(( (K), vola(K) 1}.
A result due to [52] shows that this subset of star bodies is uniformly bounded in the sense that there exists an R > 0 such that every K E Sa 1) satisfies K C RB
Lemma D.I (Lemma 1 in 122 ). For any >> 0, the collection S 1) is a bounded subset of Sd Sdl()).
In particular, for R, 7d 1aa-1
d+1
where Kd-1 == (B
S ( S K R.
Now, to state the convergence result, for distribution D. let DN denote the empirical distribution over i.i.d. draws from DD The proof of this result uses tools from T-convergence 14 and local compactness properties of (*))
25

---

Theorem D.2. Let Dr and Dr, be distributions over R such that ED, | ED. I|zlle, < co and fix O < < co. Then the sequence of minimizers (Kx) C (", 1) of F(K; D.,. D over S (7, 1) has the property that any convergent subsequence converges in the radial and Hausdorff metric minimizer of the population risk almost surely:
any convergent (Kr,) satisfies K'N,- + K. E arg F(K;2,, D.)) KeS"(1.))
Moreover, a convergent subsequence of (K3)) exists.
To prove this result, we first state the definition of T-convergence here and cite some useful results that will be needed in our proofs.
Definition D.3. Let (Ft) be sequence of functions F; X + R some topological space X.
Then we say that F; r-converges to limit F and write F; + F if the following conditions hold:
For any X and any sequence (zi) such that x, + 2, we have
F(x) lim inf F (x;).
* For any X, we can find sequence d;; + such that
F(x) > lim sup Fi (z:).
60
In fact, if the first condition holds, then the second condition could be taken to be the following: for any x E X, there exists a sequence i; - x such that lim,- (x,) = F(z).
In addition to T-convergence, mizers of a sequence functions are attained over a compact domain.
also require the notion of equi-coercivity, which states that mini-
Definition D.4. family of functions F; : X - R is equi-coercive if for all a, there exists a compact set K, C X such that {x E X (x)x) a} C Ka.
Finally, this notion combined with I'-oonvereence guarantees convergence of minimizers, which is known as the Fundamental Theorem of I'coovvrgencce [1).
Proposition D.5 (Fundamental Theorem of T-Convergence). Ff I, F and the family (F.) is equi- coercive, then the every limit point of the sequence of minimizers (x.) where 24 E arg min F((x) converges to some arg min F(x).
In order to show that our empirical risk functional I l'-converges to the population risk, we need to show that the empirical risk uniformly converges to the population risk the amount of samples N + co. We exploit the following result, which shows that uniform convergence is possible when the hypothesis class of functions admits e-coverings.
Theorem D.6 (Theorem 3 in (68). Let Q be probability measure, and let Qm be the corresponding empirical measure. Let G be a collection of Q-integrable functions. Suppose that for every s there exists a finite collection of functions G5 such that for every E G there exists g, Ge satisfying (i) g & y & D. and (ii) EQ D7 < E. Then supgeg [BQ... [g] [Q]l]] - O almast surely.
We now state and prove the uniform convergence result.
Theorem D.7. Fix 0 < < < oo and let D, and Dn be distributions on IRd such that ED, |1 < 0o for = n. Then, we have strong consistency in the sense that
[F(KDDN D F(K; D.)I a N + 3o almost aulmostsurlly....
sup KES*(;,1)
Proof of Theorem D.7 By Lemma B.I we have that the map + F(K;Dr,D,) CMp- Lipschitz over S (y, 1) where
C(r) == 1/7² and Mp == max ED, ||alle < co. Now, K (7, 1)}. We will show that we can construct
consider the set of functions G := [11- finite set of functions that approximate || for any K E S 1) via e-covering argument. This will allow to apply Theorem D.6 First, note that ( 1) is closed (with respect to both
|| -11
26

---

the radial and Hausdorff topology) and bounded as subset of (*)). Thus, by Theorem B.2
we have that the space (S* 1),6) is sequentially compact, as every sequence has a convergent subsequence with limit in (7 1). On metric spaces, sequential compactness is equivalent to compactness. Since the space is compact, it is totally bounded, thus guaranteeing the existence of finite E-net for every & > 0 as desired. For fixed > 0, construct a 1y-ovvrr S, of S" 1) such that (U)KE5((1.) minnes, 5 5(K L) & yj where 1) & c/(2MMp) Define the following sets of functions:
Gn-- = {(Il || CMPp)++ K E Sn} and G =- {ll || k CMpp) K Sn}-
Let || ||x E E be arbitrary. Let Ko E Sy be such that 6( K, Ko) < 7). Define J= I| 1l, + +CMpn E In.- and f (ll II - C.Mpn)+ 9 follows that f < < f. Moreover, by our choice of
77, we have that
Ep, f f < 20Mpn & E for each = r, n.
Thus, the conditions of Theorem D.6 are met and we get
sup (ES((.1)
[F(K; D F(K; 1n)
sup (x8((1,1)
D [||>||]] ED, [|xx]]]]]
sup (T8(
Epy [|]]||]] - Ev, |x|
0 as N 00 a.s.
D.2 Proof of Theorem D.2
We first establish the two requirements of I cenvvergeece. For the first, fix K E 8th 1) and consider a sequence KN -> K. Then we have that
F(K; Dr, Fr, D,) F(K; F(;; + D FKKN; FK
F(KN;
< DF( - F(K;D, (
+ F(K:D, ,DN) - -F(KN: D.N. DD F(KN:D, )
[F(K;D,; Dr) F(K:D,, D.
sup KES((a))
+ F(K; DNN) F F(KN: D. D. F(KN; D. DNN.
(6)
By Theorem D.7 we have that the first term goes to as goes to oo almost surely. We now show that [F(K: K: D. DN) F(KN; DN as N + almost surely. To see this, observe that
by Lemma B.I we have
IF(K:DN, DNN) - P(KN; D.,, D.N) < max; EDN [|=]lee]]
6(K, KN)
2
Since Epy [||=||c]] + ED, ||1= | < 0o for each r, and 8(K, KN) 0 as + oo almost
surely, attain [F D.N, DNN) F(KN; D.. DN O. Thus, taking the limit inferior of both sides in equation (6) yields
F(K; D.) < lim inf F(KN; DNN). m-
For the second requirement, we exhibit a realizing sequence so let K E S Sa 1) be arbitrary. By the proof of Theorem D.7 for any N > 1, there exists a finite -ent S,/N of S Sa 1) in the radial metric 5. Construct a sequence (KN) C &S'(7, 1) such that for each N, KN E SI// and satisfies
5( 6(KN. K) & 1/N. Hence this sequence satisfies K2 K in the radial metric and KN E S8(7,1
for all N > Hence we can apply Theorem D.7 to get
F(KN; D.N. DÑN
F(K; P)
< F(KN; D,N DN [(;,,," ,NDD
+ [K D.N. -
0 as N - ao
27

---

so limm-~oc F(KN: ,,,,D,D, ,D) = F(K; Dr, Dn).
Now, we show that LF(:;;; D )) is equi-coercive on (" 1). In fact, this follows directly from Theorem B.2 the variant of Blaschke's 's Selection Theorem proved for the radial metric. Thus cqui-coerciveness the family (F(:;
(F(::) D.N. D. trivially holds over (", 1). As a result, applying if we define the sequence of minimizers Ky E arg mig F )NN (E81(;
Proposition D.5 D.5 to the family F(;DD, D S0
we have that any limit point of Kj converges to some
K, arg mig F(K;D,, D,) KES4().1)
almost surely, as desired. The existence of convergent subsequence of (KK) follows from S* 1) being closed and bounded subset of (x) and Theorem B.2
D.3 Further examples for Section 2
Example (f1- and bz-ball example). We additionally plot the undertying sets Ly and L from the example in the main body in Figure
Example ball and Gaussian). We consider the following case for d = 2, when our distri- butions D, and Dn are given by a Gibbs density of the 30 -norm and a Gaussian with mean 0 and covariance E E IRdxd,
pr(x)) - r(d + 1) (2
el and P (x)
(2, Ax)
((2T) det(E)
Here, we will set Σ of the b--ball and the ellipsoid induced by E, respectively. Denote these star bodies by L, and Ly, respectively. Then, the depeddepnndent star body Tur defined by
R The star bodies induced by pr and Ph are dilations
Pr.. (u) (collzcle (d+1) c..l12 - (d--1) 1/(d+1)
where the constants
+ 1) vola(2B; )ddl and en det()) Jo tst 2xpp(-2(2)d We visualize the star bodies in Figure 6
(xx(((())
Example 8 (Densities induced by star bodies). We give general examples of Lr, when the distribu- tions D. and Dn have densities induced by star bodies. If we assume the conditions of Theorem 2.4
using the definition of Pv. and Ppn.. we have that radial function Pr. is of the form
Prsalu)at== = (p.(tt)) - Pm. (tu)))d
Suppose for some functions the the such that So (b)dd < oo for = r, n, we have (u) = 2b (l[(lx]),,) and pm (u) the (1I=I for two star bodies Kr and Kn. Then we have that
fo (p,(tt)) - (tu)) c(b)) Ix.
(d+1) xx=1)) (d-1)
where c(v) = S0 (t) Note then we must have
<( (d++1) c(vn))lal (d+1) (20)))/((++))11a (1 (///l1))|//lka....
This effectively requires the containment property
((dt),, C c( 2((4411)
Example 9 (Exponential densities). Suppose, for simplicity, that have exponential densi- ties. We need them to integrate to 1, so, for example, letting ca == T(d + 1), we know from -Ialx dx = ca Vo), Hence suppose our two distributions have densities (r(x) =--llll /(ca vo.a(Kr)) and Pn(x) -1llkk (ca vola(Kn)) Then Sge Pr (z)ddr
152V that Jga -
28

---

Plots of L,, Lg and Lin with 1.3
0.5
0.5
0.5
0
-0.5
-0.5
-0.5
-1
-1
0
Plots of Lr,, L and L with a 1.6
0.5
0.5
0.5
-1
-1
-1
Plots of Ly, Lú, and L with a 2.3
0.5
0.5
0.5
0
-0.5
-0.5
-0.5
-1
-1
Figure 5: We plot the sets L,, Lg and L for different values of a: (Top) a 1.3, (Middle)
= 1.6, and (Bottom) = 2.3. In each row, the left figure shows the boundaries of L, and L the middle figure additionally overlays the boundary of T and the right figure shows I = {a
R² |جا|ا < 1}.
29

---

Pluts of Los Los and Lea
0.5
0.5
0.5
-0.5
-05
-05
0.5
-0.5 0.5
-0.5 0.5
Figure 6: We consider the example where L, induced by Gibbs density with the & norm and Ln is induced by mean 0 Gaussian distribution with covariance Σ [0.1, 0.3; 0, 0.1]. (Left) We show the boundaries of Tup and In. (Middle) We additionally overlay the boundary of L,,. (Right) We show Ly,n = {or I2² ||>|| < 1].
-norm
Ja (r(rddx = 1. Here, 1t(t) = e- (ea vol)(K,)) and (t) e vola( (<(((c)ooo((K))) Hence, order for the distributions D, and Dn, satisfy the assumptions of Theorem 2.4 we require
PK. (u) PK, (u) vola Kx) vola(K,) for all u sd-1
In the end, our star body will have radial function of the form
(uk((u)tt Pr.((u)+1 = =d vol)(K,)
PK. (u) (ola(Kn)
for some constant that depends the dimension.
Example 10 (Scaling star bodies). Using the previous example, see how scaling star body allows one to satisfy the conditions of the theorem. In particular, suppose we have exponential densities where K, aK, where >> 1. Then note that the above requirement for containment will be satisfied since for any u @d-1
pk.(u)6+1/ ((ol(k)) = a pK. (u) //al vola(K,))
Capr, (u) vola (Kn) PK, (u) (vla(k)))
The above example shows that for certain distributions, we are always able to scale them in such way that they will always satisfy the assumptions of our Theorem. Below we give more general result along these lines.
Proposition D.8. Suppose Kr and Kn are two star bodies in Rdd Then, there exists constants < m M < oo (depending on K, and Kn) such that if we set
a. 1 vola(Kn)
d+1
>0
vola(K,)
M
we have that
PK. (u) (o,)(K,)
Pa.k, (u)d+1 (ol)(a.K,))
Yu E Sd-1
That is, for any star bodies Kr, Kn E Sd, the assumptions of Theorem 2 will be satisfied with the distributions 'D, and Dn with densities
Ca vovol((K)) and pn(x)) == -lalloo
e -1=1lk
Dr (r(x):
Ga vola(a,
respectively.
30

---

Lp, for
0.1
M2.r for p p= 10
M2 for p 50
M2 for p 100
0.5
0.2
Le
Ma
0.2
Ma.e
0.5
0.1
0.1
0
-0.1
-0.5
-0.1
-0.2
-1
-0,5
-0.2
-0.5
0.5
-0.2
0.2
-0.2
0.2
Figure 7: (First) The Gaussian mixture model in Section D.4 induces a star body Lp, Here, we set
& = 0.1. Smaller values of induce a higher degree of nonconvexity in Lr. In the next three images, we show the resulting harmonic 2-combination M2 == Lp. f2 for (Second) p = 10, (Third) p 50, and (Fourth) p 100. We see that for some value of p.(e) p. 10, M2,p. becomes will be p.-weakly convex.
convex. By Proposition 4.2
|
Proof of Proposition D.8 Because K, and K, are star bodies, they are bounded with origin in their interior. Hence, there exists constants 0 M < such that mBd C Kn, Kr MB Note that in order for us to have
PK. PK.(u)** (u) (o,) > PaK, (u)a+1/ vola(aK,,) = a.k.
(o,)(K,) V (vl(tt///oo((KK))) g
we require
vola(Kn) DK. (k.(u) vola(K,) PK, (u)
d+1
Vuesu-1
But by our assumptions on K, and Kn, note that Pk, > PMBL = mpp. = m and also Pk., < РMRн MM the sphere. This follows by the monotonicity property of the radial function for star bodies: C - PK & PL. Thus, using these bounds achieves the desired result.
D.4 A star body with weakly convex squared gauge
We will consider an example of a data-dependent, nonconvex star body such that its squared gauge
is weakly convex for sufficiently large parameter p 0. For parameter E (0, 1). consider the following 2-dimensional Gaussian mixture model P. = W0 Eg,1) + (o Σx,2) where Σe,1 = [1, 0; e] E 2R2x2 and Σ, = [e, 0, 1] E R2x2 Denote its density by (r()). One can
show that ps induces valid radial function via equation given by
PP. (u) = (% P
1/3
- -c 1/2
//// ull c2 E uulle
1/2
1/3
where C,.à = = det 1/2 (So (2--P12dt) for = 1,2. Let Lp, denote the star body with radial function Pp. For fixed E (0. 1), we investigate the geometry of Ma,p := Lp f Bd for various values of p in Figure In this example, we set s = 0.1. As discussed in (52]. Lp. can be described as the harmonic Blaschke linear combination between the star bodies induced by the distributions (0, E, and (0, E..2). These distributions induce ellipsoids that are concentrated one of the axes. and their resulting harmonic Blaschke lincar combination is the nonconvex star body shown in the left panel in Figure 7 We see that as p increases, the resulting harmonic 2-combination M2,p eventually becomes convex for large enough parameter (x(E) p. that depends s. By Proposition 4.2 this shows that the squared gauge I1ž,, is p.-weakly convex.
NeurIPS Paper Checklist
1. Claims
31

---

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, all claims made in the abstract and introduction reflect our contributions. We reiterate our main contributions in Section [.] and highlight in which sections each contribution made.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, we have a section dedicated to limitations of the work and potential future directions. Please see Section [5]
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification: Each theoretical result provides the full set of assumptions made, and no informal results are given. A proof sketch of our main theorem (Theorem (.4) provided. but the full complete and correct proof is given in Section B.2 All proofs are also complete and correct.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, a full discussion regarding experimental settings (hyperparameter choices, neural network architecture, datasets, etc.) are provided in Section C.I and in Section C.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
cases, in which
33

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [No]
Justification:
We provide full discussion of experimental settings in Sections and C.2
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.cc.
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7 //nips. htcp//blcc/gudeeeeeeeeeeeeeeeeee
ssiobiiiiisiooccccyc. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
These details can be found in Section and and in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
34

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, please see Section C.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The current work is foundational theoretical research, and there are no direct societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
35

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
36

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "7Tjy4XGf8w", "reviewer": "Reviewer_5WpE", "summary": "This submission extends the techniques of [50], i.e., tools from star geometry and dual Brunn-Minkowski theory, to characterize the optimal regularizer under the adversarial regularization framework [51] of inverse problems. \\alpha-Divergence as loss functions for learning regularizers is also discussed, with the dual mixed volume interpretations. The weak convexity and compatible neural network architectures are further discussed for computational concerns related to the proposed star body regularizers.", "questions": "Prop. 4.3 requires positive homogeneity of each layer, which limits the choice of activation functions in practice to a subset of piecewise linear functions, such as ReLU and its variants. I feel that this could be a limitation of the proposed approach. Moreover, while each layer is an injective function, due to the homogeneity of activations, the composite of such layers will not be an injective (see Sec. 3 of the paper below for example), will this break the proof of Prop. 4.3?\n\nDinh, Laurent, et al. \"Sharp minima can generalize for deep nets.\" ICML 2017.\n\nAfter the authors' rebuttal\n---------------------------------------------------------------------------------------------------\nThe question regarding the composite of injectivities does not apply to Prop. 4.3.", "limitations": "Mostly. Please refer to the Questions session for a concern I have regarding limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Extending the analysis and results of [50] to the adversarial regularization framework and showing its connections to the \\alpha-divergence is an interesting theoretical contribution. The specified neural network layers compatible with the star body regularizer can also shed light on practice.", "weaknesses": "My major concern with this work is that it is unclear if the proposed new \\alpha-divergence-based loss functions are useful for the adversarial regularization problem this submission studies. The original adversarial regularization work [51] for inverse problems, albeit published in NeurIPS 6 years ago, has reported experimental results to validate the proposed framework. On the other hand, if positioned as a pure theory work, given the existence of [50], I feel that the theoretical contribution of this submission seems a bit short for publication in NeurIPS.\n\nAs a minor thing, it would be helpful for readability if an overview of the organization and the flow of the paper could be briefly presented in the introduction.\n\nAfter the authors' adding new experiments during the discussion phase\n---------------------------------------------------------------------------------------------\nThese two concerns are partially addressed. On one hand, I can see the potential of the proposed approach from these experiments. On the other hand, the experiments are still preliminary and small scale.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review and positive comments that our insights offer \"an interesting theoretical contribution\". Below we would like to address concerns/questions that were raised.\n\n>Regarding novelty in relation to [50]\n\nWe appreciate the reviewer's concern regarding novelty in relation to [50]. Please see our global comment above regarding the novelty and significance of our work. We would additionally like to note that the new proposed $\\alpha$-divergence-based loss functions are meant to offer an alternative to the adversarial regularization framework that still falls under the umbrella of \"critic-based\" regularizer learning frameworks. While we did not provide experiments in this paper, we believe our theoretical contributions provide novel insights into unsupervised, critic-based regularizer learning, an area that has been underdeveloped theoretically, and offer convincing evidence that other critic-based losses are worth exploring both from a theoretical and practical perspective.\n\n>Adding an overview\n\nThank you for this suggestion. We agree that this would help with readability of the manuscript. We will add this in an updated version of the manuscript.\n\n>Regarding Proposition 4.3\n\nYes, we agree that requiring positive homogeneity of each layer limits the activation functions one can use. However, such activations are commonly employed in such tasks. For example, the experiments that were done in Lunz et al [51] and in subsequent applications also used a network of this form, i.e., a convolutional neural network with no biases and LeakyReLU activations. \n\nRegarding the concern of a lack of injectivity, we first note that Prop 4.3 requires that each layer $f_i(\\cdot)$ is injective and positively homogenous. Note that the composition of any two injective and positively homogenous functions is again injective and positively homogenous. To see this, consider two injective and positively homogenous functions $f$ and $g$. Then note that for any $\\lambda \\geqslant 0$ and $x$, $f(g(\\lambda x)) = f(\\lambda g(x)) = \\lambda f(g(x))$. To see injectivity, note that $g(x) = g(y)$ implies $x = y$. The same holds true for $f$. Hence if $f(g(x)) = f(g(y))$, then $g(x) = g(y)$ by injectivity of $f$. But by injectivity of $g$, $x = y$. Hence $f(g(\\cdot))$ is injective.\n\nWhen applying this to neural networks, note that our result requires each layer $f_i(\\cdot)$ to be injective and positively homogenous. This means that we require that the linear layer $W_i$ *composed with the activation function* $\\sigma_i$ must be injective and positively homogenous, i.e., $f_i(\\cdot) = \\sigma_i(W\\cdot)$ must be injective and positively homogenous. This is automatically satisfied if the linear layer is injective and the activation is LeakyReLU, since it is bijective. This can fail, however, if one uses ReLU. For example, note that $\\mathrm{ReLU}(I \\cdot)$ is not injective, even though the identity matrix $I$ is invertible. Certain matrices $W$, however, allow for injective maps $x \\mapsto \\mathrm{ReLU}(Wx)$. See, for example, the notion of a \"directed spanning set\" in Puthawala et al or the injective 1x1 convolutions of Kothari et al.\n\nPuthawala et al, Globally Injective ReLU Networks, Journal of Machine Learning Research 23 (2022) 1-55\n\nKothari et al, TRUMPETS: Injective Flows for Inference and Inverse Problems, Uncertainty in Artificial Intelligence (2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KcivRGJbWo", "reviewer": "Reviewer_3HXp", "summary": "This paper leverages the star geometry and dual Brunn-Minkowski theory to study the optimal critic-based regularizers. The authors illustrate the optimal regularizer can be interpreted using dual mixed volumes that depend on data distribution. Theorems are proved for the existence and uniqueness of the optimal regularizer. The authors also identify the neural network architectures for learning the star body gauges for the optimal regularizer.", "questions": "In what scope the submitted work extends [50]?\nOn line 68, \"assigns\" should be \"assigned\"?\nOn line 118, what does \"[x, y]\" mean if both x, y are points in R^d? Did you mean the line segment connecting x and y?", "limitations": "See the weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The paper leverages the star geometry in understanding the geometry of unsupervised regularizer learning.", "weaknesses": "As cited in the submission, this paper is closely related to [50]. Many concepts, theoretic results, even examples resemble or coincide with those in [50]. The paper failed to clearly distinguish itself from the existing work [50].", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and questions. Below we would like to address concerns/questions that were raised.\n\n>Regarding novelty in relation to [50]\n\nWe appreciate the reviewer's concern regarding novelty in relation to [50]. Please see our global comment regarding the novelty and significance of our work. The following points summarize our work's impact and novelty:\n- Our research introduces novel theory and a new framework for understanding critic-based regularizer learning, addressing a significant gap in existing theoretical foundations.\n- We tackle several novel technical challenges inherent to the critic-based setting, distinguishing our work from [50].\n- This work also demonstrates the applicability and utility of star geometry tools for regularizer learning and broader machine learning contexts.\n\nWe would also like to highlight that while our results and example look similar to those in [50], all results in the present work are novel. Moreover, all examples we visualize in the paper are different than those in [50]. The only example that is similar is the one shown in Figure 7 in the supplemental. Our experiment is different, however, since we are showing that this star body induces a weakly convex (squared) gauge, which was not explored in [50].\n\n>Typos and notation\n\nThank you for your catching those typos. We will update them in our revised submission. Yes, $[x,y]$ refers to the line segment between these two points in $\\mathbb{R}^d$, i.e., $[x,y] := \\set{(1-t)x + ty : t \\in [0,1]}.$ We will add this definition in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bo9gYpSSUx", "reviewer": "Reviewer_XeHQ", "summary": "This paper explores the learning of task-dependent regularizers using critic-based loss functions in the context of variational regularization for statistical inference and inverse problems. It particularly focuses on a specific family of regularizers, namely gauges of star-shaped bodies, which are common in practice and similar to those parameterized by deep neural networks. The study introduces a novel approach utilizing tools from star geometry and dual Brunn-Minkowski theory, which allows the derivation of exact expressions for the optimal regularizer under certain conditions and explores the properties of neural network architectures that can yield such regularizers. This work contributes to a deeper understanding of the structure of data-driven regularizers and their optimization characteristics.", "questions": "1. Could you provide additional context on the adversarial regularization framework, particularly regarding the roles and definitions of the two distributions $D_r$ and $D_n$?\n\n2. The significance of the results in Theorem 3.1 is not clear to me. Could you elaborate on why these results are important and what they contribute to the field of regularizer learning?\n\n3. The paper lacks experimental validation of its theoretical constructs through simulations or empirical data. Some experiments (even simple ones) will be helpful for better understanding.", "limitations": "Yes.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The problem setup and the theoretical framework of the paper appear rigorous and methodologically sound. The motivation behind the study is robust, addressing the theoretical gaps in understanding how regularizers are learned.", "weaknesses": "Some of the results presented in the paper are complex and difficult to interpret, which may limit their accessibility to a broader audience. Moreover, the paper does not clearly articulate the practical implications of these theoretical findings for real-world applications, which could hinder its impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review and positive comments that our theoretical framework is \"rigorous and methodologically sound\" and that our results \"address the theoretical gaps in understanding how regularizers are learned.\" Below we discuss some of the main questions that were raised.\n\n>Additional context on the adversarial regularization framework\n\nIn the adversarial regularization framework of Lunz et al [51], the distributions $\\mathcal{D}_r$ and $\\mathcal{D}_n$ are user-specified distributions that are aimed to capture “real” or “good” data and “noisy” or “bad” data, respectively. The choice of these distributions depends on the specific application one is interested in. In the context of inverse problems, $\\mathcal{D}_r$ is often chosen to be the distribution of images similar to the image we wish to reconstruct and $\\mathcal{D}_n$ is the distribution of noisy or poor reconstructions. For example, suppose one is interested in solving an inverse problem $y = \\mathcal{A}(x) + \\eta$ where $\\mathcal{A}$ is known and $\\eta$ is drawn from some noise distribution $\\mathcal{P}$. Suppose we have prior knowledge about $x$ that it is drawn from a distribution $\\mathcal{D}_r$ and we have access to many samples from this distribution (e.g., we want to solve inpainting problems on human faces, and we have access to many samples from the CelebA dataset). One choice for $\\mathcal{D}_n$ corresponds to the distribution of backprojected reconstructions, which corresponds to taking one's measurements and \"naively\" inverting them using the pseudoinverse of $\\mathcal{A}$: $\\hat{x} \\sim \\mathcal{D}_n$ if and only if $\\hat{x} = \\mathcal{A}^{\\dagger} y$ where $y = \\mathcal{A}(x) + \\eta$, $x \\sim \\mathcal{D}_r$, and $\\eta \\sim \\mathcal{P}$. These images will be corrupted by noise that depends on both the measurement matrix $\\mathcal{A}$ but also the additive noise $\\eta$. \n\nThis backprojected distribution is a good choice for the \"bad\" distribution for two reasons. One is that the distribution contains noise that is pertinent to the inverse problem (i.e., the samples depend on $\\eta$ and $\\mathcal{A}$). Another reason is that backprojected reconstructions are often used as the initial iterate for gradient-based methods to solve \n\n$$\\min_{x \\in \\mathbb{R}^d} \\frac{1}{2}|| y - \\mathcal{A}(x)||_2^2 + \\mathcal{R}(x).$$\nHence, if the regularizer $\\mathcal{R}(x)$ has been learned to assign very low likelihood to backprojected reconstructions, using them as initial iterates to a gradient-based algorithm would aid in the algorithm making good progress initially, as the regularizer would give gradients that would push the algorithm away from these poor, noisy solutions.\n\nWe will add this additional discussion as background on the adversarial regularization framework in an updated version of the manuscript.\n\n>The significance of Theorem 3.1\n\nThank you for raising this point. At a high level, Theorem 3.1 is significant and has implications for regularizer learning for three reasons:\n- This theorem introduces and analyzes a novel critic-based loss function for learning regularizers. To our knowledge, the only other critic-based loss function that has been considered in the literature is the loss from the adversarial regularization framework of Lunz et al [51]. Introducing novel loss criteria to learn regularizers is broadly useful for the field of regularizer learning, and opens new avenues for future research, both from a theoretical and practical perspective.\n- The theorem offers novel insights into the structure of regularizers one would learn from this loss. We give explicit expressions for their structure and are able to compare and contrast such regularizers with those found via the adversarial regularization framework of Lunz et al [51]. This can be seen by comparing the visual examples shown in Figures 2 and 4.\n- Analyzing this loss brings new challenges and technical contributions for both the field of star geometry and regularizer learning. Please see our global comment for more details on the new technical challenges that this theorem addresses.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CH002NCzjS", "reviewer": "Reviewer_kSE2", "summary": "The paper presents a theoretical analysis of learning regularizers for inverse problems using a critic-based loss. By focusing on a specific family of regularizers (gauges of star-shaped bodies), amenable to theoretical analysis, the authors provide a number of theoretical insights towards existence, uniqueness within existing frameworks (based on wasserstein distance) and further extensions to f-divergences. This is further connected to the existing literature on learned regularization by considering star bodies corresponding to weakly convex regularisers.", "questions": "The paper is very well-writen, and as such there are very few questions that I have: \n\n* One of the motivations, also discussed in 1.2 (and line 57), is that uniqueness of the transport potential does not hold when considering the wasserstein 1 based loss. I would like to refer the authors to arxiv.org/abs/2211.00820, as in fact (under some assumptions), this uniqueness can be shown to be unique $D_n$ almost everywhere. With this result in mind, could you explain intuitively why in Theorem 2.4, it is possible to prove uniqueness without a.e.? \n\n* Line 130 \"this map\" - which map is this referring to?\n\n* I am not entirely sure what the relevance of Remark 2.8 is. In practice, rescaling the distribution destroys information from the true distribution - the critic that is desired is the one that would be operating on $D_r$ and $D_n$, and not $D_r$ and $\\lambda D_n$. \n\n* It would be very intersting to see whether the optimal regularisers derived as minimisers of the variational objective are also optimal regulariser in the sense of Leong et al. \n\n*Line 50 \"about the measurements\". Clasically the measurements themselves live in a different space from the original data. For this reason Lunz et al. utilises backprojection to first map it to the same space.\n\n* Line 29 \"ill-posed meaning that there are an infinite number ...\" - In the inverse problem literature, ill-posedness does not correspond to non-uniqueness only. I suggest referring to Hadamards definition of well-posedness. E.g. see Shumaylov et al. or Arridge, Simon, et al. \"Solving inverse problems using data-driven models.\" Acta Numerica 28 (2019): 1-174.", "limitations": "* See weaknesses.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The paper presents a very novel idea of utilising the theoretical framework of star-bodies in order to provide theoretical interpretability of critic based regularization. Given the recent interest in learned regularization, this paper opens up a number of new research directions both theoretically and numerically.", "weaknesses": "There are a few weaknesses, which in my opinion are not limiting. To be precise, the paper focuses on a specific class of regularizers (gauges of star-shaped bodies) and a specific type of critic-based loss functions (derived from variational representations of statistical distances). It would be interesting to see if the results can be extended to other classes of regularizers and loss functions.\n\nThe paper primarily investigates this class of regularizers theoretically and with very few experiments. The paper does not include any experiments to demonstrate the practical performance of the learned regularizers, and while the theoretical results are valuable, it would be helpful to see how they translate into practice. This could be of interest as future work for practitioners working on inverse problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review and positive comments that our paper is \"very well-written\" and that our theoretical framework \"opens up a number of new research directions both theoretically and numerically.\" Below we discuss some of the main questions that were raised.\n\n>How do we obtain uniqueness\n\nThank you for raising this point. The reason why Theorem 2.4 does not require an \"almost everywhere\" stipulation is that our optimization problem is over the class of star body gauges, as opposed to star-shaped gauges, or a more general function class. At a high-level, star body gauges have additional structure that allow for us to obtain uniqueness, which can be lost when one considers more general star-shaped gauges.\n\nIn more detail, star bodies are in one-to-one correspondence with their radial functions (or the reciprocal of their gauge) and are uniquely defined by them. As a result, dual mixed volumes also exactly specify star bodies, e.g., for star bodies $K,\\tilde{K}, L,$ and $\\tilde{L}$, we have $\\tilde{V}(K,\\tilde{K}) = \\tilde{V}(L,\\tilde{L})$ if and only if $K = L$ and $\\tilde{K} = \\tilde{L}$ (see Lutwak [52]). This property aids in establishing uniqueness in our main Theorem. \n\nIf we relax the requirement of being a star body to simply being star-shaped, we can lose such a property. In particular, if we consider gauges of star-shaped sets, it could be possible for the dual mixed volumes of different sets to be equal because two star-shaped gauges can be equal \"almost everywhere\". For example, consider the following two star-shaped sets in $\\mathbb{R}^2$: $K = B_{\\ell_2}$ is the unit $\\ell_2$ ball, and $L = B_{\\ell_2} \\cup \\set{(0,t) : t \\geqslant 0}$. Both sets are star-shaped with respect to the origin, but $K$ is a star body while $L$ is not. This is because the ray $\\set{t e_2 : t \\geqslant 0}$ where $e_2 = (0,1)$ intersects the boundary of $L$ infinitely many times. The gauges of $K$ and $L$ are equal to $||x||_{\\ell_2}$ for any $x$ outside of the set of measure zero $\\set{(0,t) : t \\geqslant 0}.$ This is because for any $(0,t)$ with $t > 0$, $||(0,t)||_K = |t|$ while $||(0,t)||_L = 0$.\n\n>Line 130\n\nOur apologies, \"this map\" refers to the map $x \\mapsto ||x||_K$. We will fix this in the manuscript.\n\n>Remark 2.8\n\nWe thank the reviewer for noting this. Our goal in this remark was to show that if the distributions do not satisfy the assumptions of the theorem, there is a way to reweight the objective so that the assumptions of the theorem are satisfied. By positive homogeneity of the gauge, this essentially amounts to reweighting/scaling one of the terms $\\lambda\\mathbb{E}_{\\mathcal{D}_i}[||x||_K]$ for some $\\lambda \\geqslant 0$. We will update the discussion in the manuscript to make this more clear.\n\n>Line 50\n\nThank you for pointing this out. We intended to use the phrase \"integrate information about the measurements\" in a vague sense, meaning that the \"bad\" distribution $\\mathcal{D}_n$ depends on $y$ in some way. We will update this phrasing to avoid this confusion.\n\n>Line 29\n\nThank you for making this point. Indeed, ill-posedness additionally encompasses a lack of a solution's existence and whether it varies continuously in the data. We will update this in the revised manuscript and cite Arridge et al's survey.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their detailed feedback and positive comments that our work is "rigorous and methodologically sound" and that our theory "opens up a number of new research directions both theoretically and numerically". We will address each reviewer's specific questions and concerns individually, but we would also like to address a main concern that was shared by multiple reviewers here. In particular, we would like to discuss the novelty and significance of our results in relation to [50]. We intend to update the manuscript with additional discussion highlighting the differences between our work and [50].

## Novelty in relation to [50]

We appreciate the reviewers' concerns regarding the novelty of our work. While it is true that our current paper employs similar mathematical tools to [50], we would like to highlight the following key differences and novel contributions:

**1. Novel setting and new theory for unsupervised regularizer learning**

The current work focuses on understanding critic-based regularization, an inherently different problem than the one considered in [50]. Critic-based regularizers must learn to prefer a pre-defined "clean/good" distribution over a "noisy/bad" distribution, whereas the setting in [50] asks to find a regularizer that maximizes the likelihood of the data. Understanding how these two distributions differ from one another creates novel challenges that make this setting inherently different than [50], a point that we will discuss further in our novel technical contributions.

Moreover, the theoretical foundations of critic-based regularization remain underdeveloped. To date, aside from the seminal work of Lunz et al. [51], there has been minimal theoretical exploration of the structure of regularizers learned in the critic-based setting. Our results demonstrate that star-geometric tools can help establish new theoretical frameworks for critic-based regularizer learning, addressing this notable gap in the literature.

**2. Novel technical contributions**

While our tools are superficially similar, the novel setting of critic-based regularizer learning brings new challenges that we believe are of interest to the regularizer learning community. Additionally, this work provides new analyses of star body gauges that were not present in [50]. We highlight specifics below:

- **New challenges**: As an example, the new critic-based loss for regularizer learning in equation (5) brings novel challenges in star geometry. In particular, we show that this loss has a dual mixed volume interpretation, but its dependency on $K$ is more complicated than in [50]. As a result, its extremizers no longer directly follow from the dual mixed volume inequality (as they do in [50]). We show in Theorem 3 that we can explicitly characterize extremizers of the dual mixed volume inequality and provide visual examples in Figure 4. Finally, we show in the supplemental (Section B.3.1, Theorem 5) that these results can be generalized to a broader class of objectives. These types of challenges were not present in [50]. To our knowledge, this type of result is novel even in the star geometry/Brunn-Minkowski literature, let alone the regularizer learning literature. 

- **New applications:** We present new results on the application of star body regularizers that are relevant to both the inverse problems and machine learning communities. These include identifying beneficial properties for optimization, such as weak convexity, and providing guidelines for designing neural network architectures to learn these regularizers. Our findings offer valuable insights into the behavior of these regularizers in downstream tasks and practical methods for learning them. These types of results were not present in [50].

- **New visualizations:** All examples we visualize in the paper are different than those in [50]. The only example that is similar is the one shown in Figure 7 in the supplemental. Our experiment is different, however, since we are showing that this star body induces a weakly convex (squared) gauge, which was not explored in [50].

**3. Impact and significance**

We would like to highlight and summarize the impact and significance of our work:
- Our research introduces novel theory and a new framework for understanding critic-based regularizer learning, addressing a significant gap in existing theoretical foundations.
- We tackle several novel technical challenges inherent to the critic-based setting, distinguishing our work from [50].
- This work also demonstrates the applicability and utility of star geometry tools for regularizer learning and broader machine learning contexts.

Meta-review: This paper studies the optimization of critic-based loss functions to learn star-shaped gauges regularizers. It uses tools from star geometry and dual Brunn-Minkowski theory which allow deriving the optimal regularizer in certain settings. The paper claims to highlight how "tools from star geometry can aid in understanding the geometry of unsupervised regularizer learning."

The initial reviews for this paper were mixed, with the main concerns being around novelty, usefulness and practical implications of the proposed methodology, and some concerns on the experimental part. However, after the authors' rebuttal, all these concerns have been addressed, the contributions of the paper have been highlighted, and new experiments have been included. As a consequence, now all four reviewers support acceptance of the paper.

I believe this work makes some interesting theoretical contribution, and it additionally sheds light on the practical side through the proposed neural network layers that are compatible with the regularizer. To improve the paper, I strongly encourage the authors to incorporate the points discussed with Reviewer kSE2 regarding hyperparameter choices and updated results including TV denoising.
Decision: Accept (poster)