A Bregman Proximal Viewpoint on Neural Operators
Authors: Abdel-Rahim Mezidi
Jordan Patracone
Saverio Salzo
Amaury Habrard
Massimiliano Pontil
Rémi Emonet
Marc Sebban
Venue: ICML 2025 poster (2025)
Keywords: neural operators
proximal optimization
bregman divergence
fourier neural operator
Primary area: optimization->convex

Abstract: We present several advances on neural operators by viewing the action of operator layers as the minimizers of Bregman regularized optimization problems over Banach function spaces. The proposed framework allows interpreting the activation operators as Bregman proximity operators from dual to primal space. This novel viewpoint is general enough to recover classical neural operators as well as a new variant, coined Bregman neural operators, which includes the inverse activation operator and features the same expressivity of standard neural operators. Numerical experiments support the added benefits of the Bregman variant of Fourier neural operators for training deeper and more accurate models.

A Bregman Proximal Viewpoint on Neural Operators
Abdel-Rahim Mezidi
Jordan Patracone
Saverio Salzo Amaury Habrard 13 Massimiliano Pontil 45
Remi Emonet 13
Mare Sebban'
Abstract
tries to design equivariant Fourier layers, thereby enhancing the representation power and robustness of the architec- ture. To better scale the depth of neural operators, the F- FNO (Tran et al., 2023) proposed separable spectral layers and improved residual connections, along with a bag of training tricks. The FNO are extended to Wavelet Neural Operators (WNO) (Tripura & Chakraborty, 2023) by re- placing Fourier layers with wavelet layers to further exploit multiscale information. The U-shaped Neural Operator (U- NO) (Rahman et al., 2023) adapts the U-net architecture for neural operators, enabling mapping between function spaces through integral operators, thus broadening the applicabil- ity of neural architectures to diverse domains. Differently, the DeepONet architecture (Lu et al., 2021) comprises two intertwined components: a branch network responsible for encoding discrete input function spaces, and a trunk net- work dedicated encoding the domain of output functions. Operating as conditional model, DeepONet leverages the embedding of inputs and outputs via dot product operation, facilitating the approximation of complex functions through a structured network topology. Finally, Neural Inverse Oper- ators (NIO) (Molinaro et al., 2023) tackle inverse problems by combining DeepONet and FNO architectures to map operators to functions, thereby extending the applicability of neural operators coefficient estimation tasks.
We present several advances on neural operators by viewing the action of operator layers as the minimizers of Bregman regularized optimization problems over Banach function spaces. The pro- posed framework allows interpreting the activa- tion operators as Bregman proximity operators from dual to primal space. This novel viewpoint is general enough to recover classical neural op- erators as well new variant, coined Bregman neural operators, which includes the inverse acti- vation operator and features the same expressivity of standard neural operators. Numerical experi- ments support the added benefits of the Bregman variant of Fourier neural operators for training deeper and more accurate models.
1. Introduction
Neural operators (Kovachki et al., 2021; 2023), a recent extension of neural networks, have emerged as versatile framework for learning mappings between function spaces. These operators have shown great potential in solving par- tial differential equations (PDEs) and simulating complex dynamical systems. The exploration of neural architectures for the approximation and learning of operators has led to the development of a variety of models.
Some approaches inspired by attention mechanisms, pivotal in image and natural language processing, have also been considered in operator learning. LOCA (Learning Opera- tors with Coupled Attention) (Kissas et al., 2022) facilitates robust gradient estimation, particularly in scenarios with limited training data, by combining attention with kernel mechanisms. The General Neural Operator Transformer (GNOT) (Hao al., 2023) is scalable framework based self-attention mechanisms allowing to deal with heteroge- neous inputs useful for modeling diverse physical systems. Some physics-infommat variants integrating information from PDEs during the learning process have been proposed enhancing model interpretability and generalization: PI- DeepONet (Wang al., 2021) and its Long-Time Inte- gration variant CUTI-PI-DeepONe (Wang & Perdikaris, 2023), PINO (Physics- Informed Neural Operator) (Li et al., 2021b) hybrid extension of FNO, or other variations such as V-DeepONet (Goswami et al., 2022) and Modified Deep- ONet (Wang et al., 2022).
One influential contribution the Fourier Neural Opera- tor (FNO) (L.i et al., 2021a), sketched in Figure 1, which transforms encoded input data into frequency components in order to learn intricate relationships in the frequency domain. More recently, the Group-Fquivvriant FNO (G- FNO) (Helwig et al., 2023) additionally leverages symme-
"Equal contribution Universitéé Jean Monnet Saint-Ftienna CNRS, Institut d' O'Optique Graduate School, Inria, Laboratoire Hubert Curien UMR 5516, F-42023, SAINT-ETIENNE, France 'DIAG. Sapienza University of Rome, 00185 Rome, Italy 'Instiuu Universitaire de France (IUF) 'Computational Statisties and Ma- chine Learning, IIT. Genova, Italy Departement of Computer Science, UCL, London, United Kingdom. Correspondence to: Radel- Mezidi sabdel.rabim mezidi wtw.ssttteeeeetrr..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

A Bregman Proximal Viewpoint on Neural Operators
R
VWWv
1/
K (vx)
Win.
Wees b
Ke b
Figure Illustration of the 1-th layer of Fourier Neural Operators. The upper branch applies a linear transformation R. the Fourier modes using the Fourier transform F and its inverse F=!1 The lower branch performs an affine transformation in the latent space.
(a) Neural Operator.
(b) Bregman Neural Operator.
Figure 2: Illustration of the t-th layer of (Bregman) Neural Operators. On the left, the identity term and the linear term Ke + by have been merged into (7 + K2),t = We,, For both, KCf represents any absolutely continuous operator.
Contributions. Unlike previous works (Kovachki et al., 2021), which directly consider the compositional form of neural operators, our approach introduces distinct per- spective by formulating the action of each operator layer as the minimizer of a regularized optimization problem over functions. This optimization connects the current hidden representation to the next, with the choice of a regulariza- tion implicitly defining the activation operator through the lens of the Bregman proximity operator. Our framework not only covers existing neural operators but also introduces a novel variant, termed Bregman neural operator, which demonstrates improved predictive performance as its depth increases. Its applicability is grounded by universal ap- proximation results proven for sigmoidal-type activation operators. Beyond its unifying aspect and its ability to de- sign novel neural operators, the proposed framework allows applying the extensive body of literature on proximal numer- ical optimization, of which Bregman proximity operators belong to, in order to study neural operators. This opens the way to extend the analysis done on neural networks to (Bregman) neural operators in the same spirit of Combettes & Pesquet (2020a;b).
such as subdifferential, To space and Fenchel conjugate, whose definitions are recalled in Appendix A.
2.1. Operator Learning
Operator learning finds significant applications in the con- text of PDEs in order to efficiently approximate solutions to PDEs without the need to solve them repeatedly from scratch (Li et al., 2021b; Serrano et al., 2023; Raonic et al., 2023). Given a nonempty bounded open set D C IRd, and some time horizon T > O, we consider the generic family of PDEs over x [0, 7] of the form
Fa (()
f(z, on x 10, u[xx, 0) uz( on D, [uxx () us( 1) on 0D x J0,-],
and
(I) where F. is (possibly) nonlinear partial differential opera- tor, f source term, us boundary condition, uo an initial condition, and u: D -> IR. the PDE solution.
The main problem we will tackle in our numerical section is the initial value problem. This involves finding the oracle mapping & from any initial condition function uo to the so- lution u(-;7) of the PDE at certain time horizon 7 E}0, r]. More generally, the oracle operator G could be a mapping be- tween two different function spaces A and U. Without loss of generality, given some bounded open sets D C Rdd with d E N4. we let A = A(D, R") and 4 = U(D, R*), with n, le E N., be some separable Banach spaces of functions. For instance, A can represent the spaces of continuous func- tions from D - R.. Hereafter, A and 2 will be referred to as the spaces of input functions and output functions, respec- tively. In nutshell, operator learning consists in finding the unknown ground-truth correspondence operator G: A + U given N E N pairs of input-output functions {a,, 2, N
Outline. The paper is organized as follows: Section 2 is dedicated to the presentation of definitions and background knowledge on neural operators and Bregman proximity op- erator. In Section 3, we introduce the operator layers as the solution of a functional optimization problem. In ad- dition, we show that this new mapping allows recovering the classical neural operators and creating a more general family of so-called Bregman neural operators. In Section 4, we provide preliminary universal approximation result for Bregman neural operators. Finally, in Section 5, we conduct an experimental study comparing on benchmark datasets our Bregman variant with the different FNO im- provements. The source code of this work is available on:
2.2. Neural Operators
2. Background and Definitions
Among the existing models to approximate G, we focus on neural operators, which are parametric mappings N: A i of the form
Here, we introduce some definitions required for the under- standing of the rest of the paper as well as the necessary background on neural operators and Bregman proximity operator. We will use basic concepts from convex analysis
(Va E A), N((a) = Q Lr o
P(a),
(2)
2

---

A Bregman Proximal Viewpoint on Neural Operators
where
setting. Extending this work to neural architectures acting Banach function spaces requires addressing non-trivial mathematical challenges. These include defining operator layers rigorously, particularly the proper formulation of Leg- endre functions on function spaces, the associated Bregman divergence, and the Bregman proximity operator. In the next section, we formalize these notions, laying the groundwork for the proposed novel perspective on neural operators. The reader interested in the technical details is invited to refer to Appendix A.
P: A(D, + A(D, is local lifting operator mapping the input function to its first hidden representa- tion;
Q: ((D, RTT)- - L(D, IR*) is a local projection opera- tor mapping the last hidden representation to the output function:
For every
{L....,T). Le: Vi-i(D;, (D,,, (D,, I=*) is operator layer where each De C Rd is an open bounded set, V.- = (D,,, (R*))) is suitable Banach space of functions such that Vo- = A(D, R
2.3. Bregman Proximity Operator
and 1/ = (D, ", for consistency.
Each component of the neural operator (2) depends on a fi- nite dimensional parameter. Collectively those parameters constitute a vector 0 E C R..
At the core of our framework is the link between activation operators and Bregman proximity operators. The definition of the Bregman proximity operator hinges on a Bregman divergence, often referred to as a distance, which is derived from a Legendre function (see, e.g., Rockafellar (1970)..
Most methodological developments in neural operators have focused on tailoring the operator layers LI...., Cr to spe- cific application. Traditionally, their design mirrors stan- dard neural networks, replacing inite-dimensiona linear layers with integral linear operators in function spaces and interpreting activation functions as Nemytskii operators that apply nonlinear transformations pointwise. When the input spaces D, are the same throughout the layers and equals D, popular class of operator layers, sketched in Figure 2a, is of the form
Definition 2.1 (Legendre function). A function Φ: IR" + --x, +oo] is called Legendre if it is proper convex lower semicontinuous and satisfies the following properties: i) int(dom の) = dom dø and 0Φ is single-valued on its do- main; ii) Φ strictly convex on int(dom Φ).
In the finite dimensional setting, Legendre functions o are typically built from an elementary Legendre function ::: R +] 0o, +oo| as o: E R" + Σn= p(x;). Since here stand in infinite dimensional setting, i.e., Lebesgue function space, the counterpart of the previous finite sum structure is convex integral functional defined below (see Fact 2 in Appendix for a more rigorous treat- ment). Also, we will allow vector valued functions.
(ve = oWee, + KC (vx) bb),
(3)
where W, 1 is matrix, be RT is bias vector and a is a local nonlinear map acting pointwise from RT to R Moreover, we have a non-local linear operator k:c= T22(D, R=--)-- + (,2 plest version, KCT is an integral kernel operator of the form (v) = JD & (2. y)u(y)dy, for all x D, with k. being kernel to be specified (Kovachki et al., 2023). Specific examples include those based upon a convolution performed in the Fourier space (Li et al., 202 Kovachki et al.. 2021), a graph kernel network (Anandkumar et al., 2020) or its multipole variant (Li et al., 2020) to name few. Hereafter, we follow a different path and propose to interpret operator layers from the viewpoint of a proximal optimiza- tion by seeing the parametric form of (3) as the minimizer of Bregman regularized optimization problem. This novel perspective allows us to propose novel architecture, dis- played in Figure 2b, of the form
In its sim-
Faet 1. Let D c IRd be bounded set and set the dual spaces == LP IR") and V* L R²") appropriately paired. Given a Legendre function é, then
((v) = S. o(v(x))dr
(5)
defines a convex integral functional, with its subdifferential O consisting of functions e for which v(x) lies within the interior of d'ss domain and VΦoo E V*. The subdifferential
is singlc-valued, and V v, will be denoted by ( suggesting it will serve kind of gradient of t at
The integral functional Tb in (5) inherits certain properties of d, such as p-uniform convexity - an extension of strong convexity when p 2. This characteristic, proven in Propo- sition A.3, is key to the mathematical soundness of our analysis (see also Remarks A.2 and A.4).
Ce(r.) = (ve) + KeV, + (v:) + be),
(4)
involving an additional nonlinear term o= (vx). and where K, E R is a matrix. In this formulation, when all the weights are zero, then L, is the identity operator. In prac- tice, we observe that this property allows training deeper and more accurate models. A similar architecture was originally proposed in Frecon et al. (2022) in the finite dimensional
We are now equipped to define Bregman distances in Lebesgue spaces. First introduced by Bregman in (Bregman, 1967), Bregman divergence extends the notion of distance beyond metric spaces, capturing asymmetries and curvature induced by convex functions. Unlike Euclidean distance, it
3

---

A Bregman Proximal Viewpoint on Neural Operators
reflects the local geometry of the function defining it, mak- ing them valuable in optimization and variational analysis.
function de (RR"),,, as defined in Fact 1. Da.: : V/
V2 -> [0, + is the corresponding Bregman distance as detailed in Definition 2.2
Definition 2.2 (Bregman distance in Lebesgue spaces), Un-
der the notations of Fact 1, the Bregman distance with re-
Mr V + V4 is bounded linear operator which maps
spect to & reads, (Vu E V, Vu E V).
dom ; into dom 00,,
b. E V/ and K2; V + Vē is a bounded linear operator of the form
(u v) ( - (o())) if V dom 0 otherwise.
+00
(((v)(x)== = / Ko(xr, (o(x,dy))ol dy)o(y),
Dz-1
Finally, we can define the Bregman proximity operator (Nguyen, 2017), which extends the (Euclidean) proxim- ity operator, widely used in optimization. The Euclidean proximity operator itself generalizes projections by replac- ing the indicator function of a convex set with appropriate convex functions. For additional details, the reader can refer to Bauschke & Combettes (2017).
with Ky: Do x
R" a (transition) ker-
nel from Di-1 to Dr, meaning a function which is measur- able with respect to the first variable and a finite measure with respect to the second variable.
9: E [o(V)) and ran + 0g:)=10
Equation (6) is highly general, featuring an outer opera- tion (the prox.,' and an inner operation (the v and can formally represent various layer architectures sketched in Figure 3. A key step in establishing this connection involves relating the proximity operator to activation oper- ators. There are multiple ways to achieve this by varying the choice of the pair (Pr:ge). In the following sections, we explore two specific choices for this pair, demonstrating how (6) recovers classical neural operators (3) (where V is the identity) and introduces a novel architecture (4), in which 74, acts as the inverse activation operator.
Definition 2.3 (Bregman proximity operator). Let V LP(D, R") with p E coll Let g E (VV and let E (V()) be defined as in Fact 1, with ø E (R") be Legendre and such that ran 0(0 + g) = V Then the Bregman proximity operator of g relative to 4 is defined as
prox V" VV, v² + argmin {(- -0"> + +- + 9}-
Note that prox & is well-defined since tb + g is strictly con- vex, lower semicontinuous and ran (( db + g) V", and it holds prox (0 + =1
y
Remark 3.1 (Form of linear operator K2)). Often in applica- tions, the kernel of the linear operator K. is split into two terms: an absolutely continuous part and a single pure point part, i.e., Kit = Kit + ni, where, for every x E D,, and measurable set A C Dz-1.
3. Revisiting Neural Operators
In Section 3.1, we propose a novel Bregman proximal view- point on operator layers. Then, we establish several con- nections. First, we show in Section 3.2 that the proposed framework is general enough recover most classical oper- ator layers when the Legendre function ø is the Euclidean distance. Second, we show in Section 3.3 how it yields new variant of neural operators when & defines a general Bregman divergence. Finally, we apply our framework to Fourier neural operators in Section 3.4.
aC (r., A) = / ke( and *((A)) (e)p:(s)(A)
with ks: De x DI-1 + RR 1, Ke E RT 1, Pe: Dr + Di-1 measurable, and 6p((z) the delta Dirac at (xx) E DI-1- Thus, we have
(,(u)(z) = k((r,y)(()dd + (x)()r(x))
= =f D...
3.1. Bregman Proximal Viewpoint on Operator Layers
Departing from usual kernel-based points of view (Kovachki et al., 2021), we suggest defining operator layers as the solution of functional optimization problems. For every t 1,... ,T, L,: V1-1 + Vi,
Remark 3.2 (Special case of identical domains). The linear operator M, should be chosen so that it maps dom 00,-1 to dom db,, However, in (6), if the function d, does not depend on and all the domains D, are the same, then it is also true that the convex integral functional $ does not depend either. Then, we have dom 0&;-1 dom 0b and for the linear operator M, we allowed to choose the identity operator.
(w) Cx(v)= argmin - (w, KC, +b)) (ud) DD, (w, M,t) wEW,
=prox 2. (v (v) b.),
(6)
where
Remark 3.3 (Link with convex optimization). When V = Ve and Me is the identity, the operator layer (6) reads
Φe: Vi2 -000, +o0 is a convex integral functional on an appropriate Lebesgue space based on some Legendre
prox 9.
T. (v) B.v) = (0., + ag:) - -V - B,)(
4

---

A Bregman Proximal Viewpoint on Neural Operators
1/4 E
Vi-1
1/41
E
V/
P
4
L
C141
Cr
@
prode
Kedt
60
Pe41 = Le(ve) wrv. {
(Gr =Ki(o,))
(Gx" (ex) - Ken +62 + +a(()) + Do, (wn, Men)
Figure 3: Illustration of the Bregman proximal viewpoint on operator layers. The action of each operator layer is viewed as the minimizer of the regularized optimization problem where each term in the objective can be linked part of the architecture, evidenced by the color code.
where B, V2= - Vr. This is Bregman forward-backward operator, which is well-known in the context of operator splitting methods in optimization (Nguyen, 2017: Bui & Combettes, 2021).
Table I: Legendre function & and its related activation 30%
domb
(1)
(t)
(t)
1,
VI &
1/v1 8
ISRU
00, 1]
(logg + (1 1) log 1)
log 1-t
Sigmoid
Concluding this section, we stress that as long as the couple ($t,9) 9c) admits a closed form Bregman proximity operator, this would define additional new types of operator layers. In
1,
log(1 (²) + (rctanh(t))
arctanh
tanh
1,
v1 82 (tresin(t)
aresin
sin
Rx00
82 LLi(ee 0t)
log log(e"(-1)
SoftPlus
Nguyen (2017), the author shows number of examples (at the end of Section 2, from Example 2.9 to Example 2. 2.) of such couples that yield an explicit Bregman proximity operator. Actually, one may consider layers of type
where VV; = -+))(()) matches a variety of monotone activation operators a. In addition, when the domains are all the same, say D D. M, = I. and the linear operator Kg = K. KG is as given in Remark 3.1, then Lv(v) ( K.)u + (C)(() + be). where (I Kt) can be wrillen W.. A schematic representation is reported in Figure 2a.
U ++ (2 (v) K,(v) + be),
with being strictly monotone and d2 monotone, serving as activation operators appropriately coupled. Classical and Bregman neural operators emerge as special cases, where i) σ1 = Id and 02 is any monotone function, for the former, and ii) 01 = 02 is strictly monotone, for the latter. Note that having σ d2 implies that the numerical implementation does not require to have an explicit form of 01 as later discussed in Remark 3.7.
In essence, Proposition 3.4 shows that the parametric struc- ture of operator layers can be interpreted via the Bregman proximal operator, when the Bregman distance reduces to the Euclidean distance. The crucial aspect in establish- ing this connection is the observation that the Euclidean proximity operator of g, = v - (1/2)]l ||a simplifies to V = (v)")(()), )). aligning with broad spectrum of acti- vation operators given an appropriate selection of tb. We report in Table the corresponding và to retrieve several well-known activation operators. A proof concerning the characterization of the SoftPlus is included in the appendix. To the best of our knowledge, V can only match mono- tonic activation operators, which notably discards GeLu and swish. To be more precise, Proposition 3.4 general enough to deal with the broad class of activation functions that can be viewed as proximity operators, which essentially boils down to any increasing 1-Lipschitzian function (see Propo- sition 2.3 in Combettes & Pesquet (2020a)). While this connection has been previously noted in the neural network literature (Combettes & Pesquet, 2020a; Frecon et al., 2022), our work extends this analysis to function spaces.
3.2. Classical Neural Operators
Our first result, stated in the proposition below, unifies broad class of classical neural operator layers through the prism of the optimization viewpoint of (6) when Da, is the Euclidean distance.
Proposition 3.4 (Unifying classical neural operators). Let V, (D, R²r) be some Hilbert function space and (e(N) = So. Ene ((,(x))dx, where ub E [T( is a strongly convex Legendre function. Consider the Euclidean distance defined from the elementary Legendre function
de = (1/2) 12 E (TR*)) (see Section 2.3) and set It- =
V [1/ 112. Then 9t E (V.V.) and Le defined in (6)
acts between 1?2 spuces as follows
Li(v) = prox 11111 (Miu+ + +K(()) + bi)
111111
= <V (K(M)))((())) +b),
5

---

A Bregman Proximal Viewpoint on Neural Operators
3.3. Bregman Neural Operators
and output spaces, i.e., V. = Vi2 (2), for every E (1,....7).
We now provide the counterpart of Proposition 3.4 for gen- eral Bregman distance.
A popularly encountered scenario in practice is that where D = T is the unit torus and the kernel associated to the absolutely continuous part of Kr is translation invariant, i.e., (e( y) - ke - y), thus indicating a convolution structure. Fourier operator layers (Li et al.. 2021a) are then devised by leveraging the convolution theorem, stating that the action of K., can be written linear operator in the Fourier domain:
Proposition 3.5 (Designing Bregman neural operators), Let Vi = (D,,,, ) be some Lebesgue function space and (e(() = S.. Ei (x),, where v E [o( is a p- uniformly convex Legendre function (+1 1. 12/2). Consider the Bregman distance in function space defined from the elementary Legendre function dx(w)= = =En ((w)) (see Section 2.3) and set 9t- O. Then Lop defined in (6) acts between L spaces as follows
17 (v)(())) (((=--)()ddy==-111
= F (R, (B(-F(())
(1) = =prox ' ( (VR,(M)) + (((0)+b)) = =VW;(V + Kc(l)) + b,),
with F; LL + Clzz, R² being the Fourier trans- form, F-1 its inverse, and R. E (Z R"**). Often, Rg does not range in the entire (Z B=XD) space but is parametrized by finite parameter (Kovachki et al., 2023). It follows that the Bregman variant of Fourier operator layer reads (v) a (()) + We + F (R, F(u)) by). The classical Fourier neural operator layer is retrieved by omitting the o -w) (v) term.
(8)
where V = (+* matches a variety of monotone activation operators a. In addition, when the domains are all the same, say D, = D and the linear operator K,, is of the form given in Remark 3.1, then we can take M₁- = 1 and
In this section, we addressed FNOs because they are widely used and simplify the analysis. In this respect, we note that we just specified the action of K. by expressing it via direct and inverse Fourier series. So, the end, it is only about finding efficient parametrizations, in some ( space, of linear integral operators between Lebesgue spaces. This has been achieved by using the Fourier transform, but in principle other transformations could be considered, pro- vided we have an unconditional basis of the Lebesgue space of functions and an efficient way to compute the coefficients. For instance, the wavelet transform can be incorporated in Proposition and Proposition 2 to retrieve WNOs (Tripura & Chakraborty, 2023) and their novel Bregman variant, re- spectively. In nutshell, our framework is transparent to the parametrization of IC.0.
(v) = VV, + K.U + K. (v) + be). (9)
Concerning the operators VØ, (+) and V ('(.), we stress that any of the vo listed in Table 1 are appropriate choices. Since (+)))()) and ()) are inverse of each other, the layer of (9) boils down to
Co(e) = o + K + (x)(() + b.),
(10)
where any invertible and monotone activation operator is al- lowed. Its schematic representation is reported in Figure 2b. This novel variant, called Bregman Neural Operator sim- ply differs from classical neural operators by the additional term involving the inverse activation operator. Finally, we note that the form of (9) corresponds to a mirror descent step (Nemirovskij & Yudin, 1983; Beck & Teboulle, 2003) with mirror map V.,,.
4. Expressivity of Bregman neural operators
Remark 3.6. When K,, KC and by are zeros and M₁ is the identity, then L, reduces to the identity.
In this section, we give preliminary positive result con- cerning the universal approximation properties of Bregman neural operators.
Remark 3.7. Concerning (I0), we should ensure to feed the first layer with functions in dom Li as discussed in Remark A.5. This condition is for instance satisfied if (Pv)(u)= = (Pu(()) = (Pu(z))) Note that in such situation, the inverse activation function does not need to have an explicit form. Indeed, when composing the different layers in (10), the inner inverse activation function will be cancelled out by the outer one.
In the following, the activation function o: R + / is re- quired to be a homeomorphism between IR and an open interval of R and of sigmoidal type, meaning that lom (1) = 0 and lim; att) = 1. Moreover, we assume that A and 24 are as follows
C(D,R")
LP(D,R") ,U(D,(*)- ((DD W""(DD
A(D,R")
IP(D,
3.4. Case of Fourier Neural Operators
We study the implications of the proposed viewpoint in the peculiar case of Hilbert function spaces with equal input
where C is the space of continuous functions and Wm is the T.P-type Sobolev space with m E N4 derivatives for
6

---

A Bregman Proximal Viewpoint on Neural Operators
FNO
ResPNO
P-FNO
BPNO()
ResFNO
0.070
FNO F-FNO
8 relativee error
0.065
Density
(oUS(
0.060
0.055
0.050
-1.5
-1.0
0.5
1.0
1.5
16
32
61
Weight value
Layers
Figure 5: Models' weight density distribution.
Figure 4: e relative errors across different number of layers for 2D Navier Stokes (v 10=1).
(a,,u, u. The only exception is the 2D Darcy problem (marked with in latter results), where the goal is to predict the steady-state solution from the viscosity function over the domain. A description of the experimental settings and the learning procedure is provided in Appendix C.
E [1, +oo Here, D denotes the closure of D, and must be considered in PDE applications to evaluate functions on the domain's 's boundary.
Theorem 4.1. Let Annd U be set as above. Let G: A u be a continuous operator. Then for any compact set K C A and ş > there exists a Bregman neural operator No: + U of the type (2) such that each component depends on a finite dimensional Bregman neural network and
Models. We consider four models: the standard FNO (Li et al., 2021a), our Bregman variant (BFNO) described in Section 3.4, the Factorized FNO (F-FNO) (Tran et al., 2023), and ResNet-inspired variant (ResFNO) that isolates the impact of residual connections by adopting the update e - U + (v) + bx), which should not be confused with Chen et al. (2021). Details can found in Appendix C.6. Additional models such as WNO (Tripura & Chakraborty, 2023) and its Bregman variant are studied in the appendix. The lifting and projection layers, namely P and Q in (2), convolutional layers with kernel size 1 and width 128. Note that, for BFNO, we add an activation operator after P to ensure that the conditions of Remark 3.7 are met. Following the code of Li et al. (2021a), we use the ReLU activation for FNO while, for BFNO, we resort to an invertible ap- proximation: SoftPlus with parameter
sup [|g(u) NNo(ulll S..
WEK
Here E TRP collects all the (finite number of) parameters of the finite dimensional Bregman neural networks defining the components in (2).
Proof. The proof is reported in Appendix B and partly re- lies on also proving this same result for Bregman neural networks in finite dimensional spaces.
10 make almost indistinguishible from ReLU. Hereafter, consider models made of E (4, 8, 16, 32, 64) Fourier layers with width 64 (resp. 32) and 16 (resp. 12) maximum number of Fourier modes for ID (resp. 2D) problems. Note that two ablation studies in Appendices D.4 and D.5 reveal marginal improvements from adding batch normalization layers or replacing SoftPlus with ReLU,
5. Numerical Experiments
The primary objective of our numerical experiments is to evaluate and assess the added benefits of the Bregman vari- ant of the simplest neural operator, namely Fourier Neural Operator (FNO), and its improvements, as they often serve as the building blocks for more sophisticated models.
5.1. Experimental Setting
5.2. Results and Analysis
Datasets. We have selected a range of benchmark datasets resulting from the resolution of PDEs used both the orig- inal FNO paper (Li et al., 2021a) and in the PDEBench suite (Takamoto al., 2022), which the top leading repos- itory providing datasets commonly studied in physics-based machine learning. They represent various dynamics and complexities pertinent to physical modeling tasks. Here- after, we consider initial value problems where the goal is to learn the mapping between the initial condition a, and the solution some future time u; from 10a pairs
Impact of the number layers T. We investigate how increasing the number of operator layers T affects perfor- mance using the 2D Navier-Stokes dataset with viscosity 10-4 chosen for its complexity, that typically favor deeper models to capture fine-grained structures and long- range dependencies. Results in Figure 4 show that BFNO systematically achieves lower prediction error regardless of T. In contrast, FNO (resp. ResFNO) degrades from = (resp. T 16) onward, while F-FNO (resp. BFNO) im-

---

A Bregman Proximal Viewpoint on Neural Operators
FNO
ResENO
F-FNO
(oUO()
ing a solution map from initial conditions, the goal is to map the spatial viscosity function to the steady-state solu- tion. In this case, it seems that the additional MLP layers added at the end of the operator layers of F-FNO help to better capture the complex dependencies of the viscosity-to- solution mapping. To complement the analysis, following Takamoto et al. (2022), we include a comparison between BFNO and FNO over several metrics measuring the rela- tive errors in low, mid and high frequency bands in Ap- pendix D.3. BFNO consistently outperforms FNO in the low and mid bands, indicating improved recon- struction of dominant modes. However, its performance in the high- figh-frequency range varies: for simpler datasets, BFNO achieves substantial error reductions, whereas for more complex cases, the improvements are marginal. Pre- diction examples are reported in Appendix D.I.
f relatiie error
0.3
0.04
0.2
0.02
0.1
0.00
0.0
ID Advection Burgers
ID
2D NS (10-3)
2D NS (10-1)
2D Darey
ID NS
(10-*)
Figure 6: Comparison of e test relative errors across dif- ferent models, with valddaadd number of layers in {4, 8, 16, 32) and learning rates, PDE benchmarks.
proves with depth until plateauing at = 16 (resp. T 64). The earlier saturation of F-FNO suggests that while residual connections help with depth, BENO scales more effectively. Similar conclusions hold for other datasets and our Bregman variant of WNO, as illustrated in Appendix D.2. Neverthe- less for simpler tasks, when smaller models are sufficient, the benefits of depth become less pronounced as expected. We believe that the added term in BFNO stabilizes learning by allowing its layers to reduce to identity when all weights are zero (see Remark 3.6). However, the same argument could be used for residual architectures (ResFNO and F- FNO). To gain further insights, we next examine the weight distributions.
Additional insights. We highlight that BENO can also be viewed from the ODE point of view through a change of variables. More precisely, let us consider updates of the form 1/+1 = o (wx) + K.wt). Then, for 2 = o (v)), it follows that 2++1 = 2t + (2)) which can be seen as a discretization of da(t) residual- based architecture of the form Ut+1 = Wy (KCem))) such as ResENO and, to some extent also F-FNO, would lead to the following ODE self. The fact that, in BFNO, the linear operator is placed outside the activation function can be seen different way of mitigating vanishing gradients compared to residual architectures like ResFNO and F-FNO.
dt = (=t)) In contrast, a
dv(() o ((((t))())) on v it-
de
Comparison of weight density distribution. Figure illlus- trates the Iearned weight distributions of the operator layers for the best- pest-performin models considered above. ResFNO, F-FNO, and FNO exhibit Gaussian-like distributions, with F-FNO showing additional peak at O. In contrast, BFNO has sharply peaked distribution around O, resembling a Laplace distribution. While BFNO, ResFNO, and F-FNO layers reduce to the identity when all weights are zero, only BFNO exhibits a distinct clustering of weights around zero. This suggests implicit regularization, enhancing general- ization and stability by preventing large deviations from the identity mapping. Additionally, the weight distribution in BFNO alleviates issues such as vanishing or exploding gradients, which are common in deeper architectures.
6. Conclusion
In summary, our contributions are twofold: we have pro- vided new theoretical framework that broadens the under- standing of neural operators through the lens of a Bregman regularized optimization problem, and we have introduced Bregman neural operators that achieve enhanced perfor- mance as their depth increases. As part of our theoreti- cal advancements, we have also established universal ap- proximation results for Bregman neural architectures with sigmoidal-type activation functions. However, it must be acknowledged that a gap exists between this result and com- mon practices, which predominantly rely on ReLU-like acti- vations, as in our work, opening the door to new theoretical developments. Beyond the unifying aspect of our frame- work and its ability to design novel neural architectures, our setting also paves the way to use the rich body of literature on monotone operators to study neural operators. In the con- text of neural networks, an example of fruitful application of the latter is given in Combettes & Pesquet (2020a) where the authors provide interesting asymptotic properties on the networks (as the number of layers tends to infinity). One can also consider the work in Combettes & Pesquet (2020b)
Extensive comparison on multiple datasets. We now eval- uate the models on diverse datasets of varying complexity. For each dataset, splitting realization, and model, we cross- validate the optimal number of layers T {4, 8, 16, 32) and report the average test error over multiple realizations in Figure 6. Results show that BFNO consistently achieves superior or comparable performance, with notable gains on moderately to highly complex datasets such as ID Burgers and 2D Uovier-Stokess (v> = 10 4). The only exception is that of 2D Darcy, where the task differs: instead of learn-
8

---

A Bregman Proximal Viewpoint on Neural Operators
where the authors yield quantitative insights into the stabil- ity properties of neural networks. As for setting, we can guess that such results might be extended to Bregman neural networss/operators by leveraging the notion of so called D-firm operators studied in Bauschke et al. (2003), meaning operators that are firmly nonexpansive with respect to Bregman divergence.
mon point of convex sets and its application the solu- tion of problems in convex programming. USSR com- putational mathematics and mathematical physics, 7(3): 200-217, 1967.
Brezis, H. Functional Analysis, Sobolev Spaces, and Partial Differential Equations. Springer, New York, 2011.
Büi, M. N. and Combettes, P. Bregman barkward operator splitting. Set-Valued and Variational Analysis, 29:583-600 603, 2021.
Acknowledgments
This research conducted within the context of the Inria- DYNAMO Associate Team.
Chen, G., Li, Y., Meng, Q., Zhou, J., Hao, X., et al. Resid- ual fourier neural operator for thermochemical curing of composites. arXiv preprint arXiv:21 10266, 2021.
This work has been funded by a public grant from the French National Research Agency (ANR) under the "Franee 2030" investment plan, which has the reference EUR MANUTECH SLEIGHT- ANR-17-EURE-0006
Combettes, P.
L. and Pesquet, J.-C. Deep neural network structures solving variational inequalities. Set- Valued and Variational Analysis, 28(3):49--51,, February 2020a.
This research was funded in whole or in part by the French National Research Agency (ANR) under project number ANR-24-CE23--1400001
ISSN 1877-0541.
Combettes, P.
L. and Pesquet, J.-C. Lipschitz certificates for layered network structures driven by averaged activation operators. SIAM Journal on Mathematics of Data Science, 2(2):529-557, 2020b.
Impact Statement
This paper presents work whose goal is advance the field of Machine Learning. More precisely, our goal is to provide novel general framework for neural operators by consid- ering the action of each operator layer as the solution of a regularized optimization problem over functions. This work essentially fundamental, comes with theoretical results, and is evaluated in the context of PDE approximation for illustration and comparison purposes. There many po- tential societal consequences of our work, none of which we feel must be specifically highlighted here.
Cybenko, G. Approximation by superposition of sigmoidal function. Mathematics of Control, Signals, and Systems, 2:303-314, 1989.
Frecon, J., Gasso, G., Pontil, M., and Salzo, S. Bregman neural networks. In Chaudhuri, K., Jegelka, S., Song. L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceed- ings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learn- ing Research, pp. 677966777777 6792. PMLR, 17-23 Jul 2022.
References
Anandkumar, A., Avizzadenesheli, K., Bhattacharya, K., Kovachki, N., Li, Z., Liu, B., and Stuart, A. Neural operator: Graph kernel network for partial differential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
Goswami, S., Yin, M., Yu, Y., and Karniadakis, G. E. A ohysics-informed variational DeepONet for predicting crack path in quasi- materials. Computer Methods in Applied Mechanics and Engineering, 191577, mar 2022.
Bauschke, H. H. and Combettes, P. L. Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer Publishing Company, Incorporated, 2nd edition, 2017. ISBN 978-3-819-4811-8.
Hao, Z., Wang, Z., Su, H., Ying, C., Dong, Y., Liu, S., Cheng, Z., Song. J., and Zhu, J. GNOT: A general neural operator transformer for operator learning. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 12556- 12569, PMLR, 23-29 Jul 2023.
Bauschke, H. H., Borwein, J. M., and Combettes, P. L. Breg- man monotone optimization algorithms. SIAM Journal on control and optimization, 42(2):596-636, 2003.
Beck, A. and Teboulle, M. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-1775 2003.
Helwig, J., Zhang, X., Fu, C., Kurtin, J., Wojtowytsch, S., and Ji, S. Group equivariant fourier neural operators for partial differential equations. In International Conference on Machine Learning, (M.. JMLR.org 2023.
Bregman, L. M. The relaxation method of finding the com-

---

A Bregman Proximal Viewpoint on Neural Operators
Kissas, G., Seidman, J. H., Guilhoto, L. F., Preciado, V. M., Pappas, G. J., and Perdikaris, P. Learning operators with coupled attention. Journal of Machine Learning Research, 23(215): 63, 2022.
Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 77187-77200, Curran Associates, Inc., 2023.
Rockafellar, T. Convex Analysis. Princeton University Press, Princeton, NJ, 1970.
Kovachki, N., Lanthaler, S., and Mishra, S. On universal approximation and error bounds for Fourier neural oper- ators. Journal of Machine Learning Research, 22(290): 1-76, 2021.
Serrano, L., Le Boudec, L.,, Kassai Koupai, A., Wang, T. X., Yin, Y., Vittaut, J.-N., and Gallinari, P. Operator learning with neural fields: Tackling pdes on general geometries. In Advances in Neural Information Processing Systems, volume 36, pp. 70581-70611, 2023.
Kovachki, N., Li, Z.,, Liu, B., Azizzadenesheli, K., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Re- search, 24(89):1-97, 2023.
Takamoto, M., Praditia, T., Leiteritz, R., MacKinlay, D., Alesiani, F., Pflüger, D., and Niepert, M. Pdebench: An extensive benchmark for scientific machine learning, In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Stuart, A., Bhattacharya, K., and Anandkumar, A. Multipole graph neural operator for parametric partial differential equations. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Infor- mation Processing Systems, volume 33, pp. 6755-6766. Curran Associates, Inc., 2020.
Tran, A., Mathews, A., Xie, L., and Ong, C. S. Factorized fourier neural operators. In The Eleventh International Conference on Learning Representations, 2023.
Tripura, T. and Chakraborty, S. Wavelet neural operator for solving parametric partial differential equations in computational mechanics problems. Computer Methods in Applied Mechanics and Engineering. 404:115783 feb 2023.
Li, Z., Kovachki, N. B., Azizzadenesheli, K., liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Fourier neu- ral operator for parametric partial differential equations. In International Conference on Learning Representations 2021a.
Wang. S. and Perdikaris, P. Long-timi integration of para- metric evolution equations with physics-informed Deep- ONets. Journal of Computational Physies, 475:1 11855, feb 2023.
Li, Z., Zheng, H., Kovachki, N., Jin, D., Chen, H., Liu, B., Azizzadenesheli, K., and Anandkumar, A. Physics- informed neural operator for learning partial differential equations. arXiv preprint arXiv:2 11.07944 2021b.
Wang. S., Wang, H., and Perdikaris, P. Learning the solution operator of parametric partial differential equations with physics-informed DeepONets. Science Advances, 7(40), oct 2021.
Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. E. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):188-2)
mar 2021.
Wang. S., Wang, H.,, and Perdikaris, P. Improved architec- tures and training algorithms for deep operator networks. Journal of Scientific Computing, 92(2), jun 2022.
Molinaro, R., Yang, Y., Engquist, B., and Mishra, S. Neural inverse operators for solving pde inverse problems. In International Conference on Machine Learning, 2023.
Zalinescu, C. Convex Analysis in General Vector Spaces. World Scientific, Singapore, 2002.
Nemirovskij, A. S. and Yudin, D. B. Problem complexity and method efficiency in optimization. 1983.
Nguyen, Q. borward- splitting with bregman dis- tances. Vietnam J. Math,, 45:519-539, 2017.
Rahman, M. A., Ross, Z. E., and Azizzadenesheli, K. U- NO: U-shaped neural operators. Trunsactions on Machine Learning Research, 2023. ISSN 2835-8856.
Raonic, B., Molinaro, R., De Ryck, T., Rohner, T., Bar- tolucci, F., Alaifari, R., Mishra, S., and de Bézenac, E. Convolutional neural operators for robust and accurate learning of pdes. Oh, A., Naumann, T., Globerson, A.,
10

---

A Bregman Proximal Viewpoint on Neural Operators
A. Additional Technical Facts
We begin by introducing the necessary notations used throughout the paper.
Notations. Let V and 1 be two Banach spaces put in duality via the pairing (.-): V V RR if V + -0, +00
we denote by dom +- {u V ((v) < +oo} its effective domain. For every proper convex function cb V - -00, +00 we set its subdifferential
( fo- E V* for all V, p(u)= D(u) + (u v, 1,
if U E dom o, and @+(()) 2, otherwise. We set dom 0$ {or dom + a # Ø) and the range ran 0$ {u" V" Ee E V s.t. v² (.(e)). When a+(v) is a singleton, we denote by V its unique element. If tb - to) its Fenchel conjugate is the function Ø* V* + |-o0, +oo] such that (0"("*)
suprey(U, v²) <(v). We denote by To(V) the set of proper convex and sevr-onnmioosssuuuus functions on V. The Fenchel-Morent theorem ensures that E <To(V) = t- E (VV"). We denote by (,, .) and | the Euclidean scalar product and norm in IR... If D C IR4 is a nonempty bounded Borel set and p E [1, +o0,, we denote by LP(D, IR") the Lebesgue space of p-integrable functions (essentially bounded functions, if p = -20 from D R..
A.1. Considerations for Legendre Function and Bregman Proximal Operators
At the core of our framework, lies the connection between activation operators and Bregman proximity operators whose definition involves the Bregman divergence itself defined from a Legendre function &b E (V)).. The latter acts on Lebesgue function space = L (D I²) and can be built from an elementary legendre function e [P") through the convex integral functional described in Fact 1. We provide below several considerations.
Remark A. One can prove that @ is Legendre if and only if d² is Legendre. Moreover, if Φ is Legendre, then é and 3* are differentiable on idom の) and int (dom (²) respectively and
VΦ: int x) + int (dom )") and
VΦi int x²) + int(com ())
are bijective and inverse of each other.
Faet 2 (Convex integral functionals on Lebesgue spaces based on Legendre function). Let D C R be an open-bounded set. Let p, q E [1, +o0] be conjugate exponents, that is such that 1/p + 1/9 = 1, and set V = L,(D, RT) and V = (D, R"). The spaces V and V can put in duality via the pairing V x V + IR, (v. u) + (v, u) = (v(x), dz. Let
E To (TB²)) be a Legendre function and let क: V + -00 +00 be such that
(v)) / o(v(z))dx.
(11)
Then & E TooV), dom 04- = {v E for a.c. x E D, v(x) int(dom and (Vo) o E V*J 06 is single valued on dom 0$, and, for every u E dom 0Φ, (v(v) (Vo.o.. The unique element Vhcv of (b(()) will be denoted by (.(v), suggesting it will serve as a kind of gradient of 4 at v'.
Remark A.2. In Fact 2, suppose that p 1 and dom = R". Then ran U V2. Indeed, we note that VΦ: int(dom σb - IR is a continuous bijection with inverse VØ" which is also continuous. Therefore if we let E V- = - (D, R)" and set v = (Vo*) u, since u is essentially bounded, we have that v is essentially bounded too, and hence integrable. In the end u E (D, R") and u = (VØ) o E (v).
Definition 2.3 of Bregman proximity operators in general Banach spaces requires that ran a( + g) is the full dual space. The following result gives simple situation in which such condition is satisfied.
Proposition A.3. Let & E (K") be a Legendre function, let E [l, +o0l, and suppose that & is p-uniformly convex with constant e > O, meaning that
Vy.u' E ER",VAEJO,1 6((1- Xyy) + A(1 - 1) ~) 1 v/Pp (1- ())())) + Xo(y')-
(12)
Let == LP", Then the integral functional Q: V + --x, +o] defined as in Fact I is p-uniformly convex with
respect to the norm 1lppp Moreover, for every E To(V) such that dom + n dom g & 2, we have dom (b + g)* = V and ( + g)* is Fréchet differentiable on V*. Thus V = dom (( + g) = ran @(Φ - g).
'Note that in general the domain of the function + has empty interior, Gâteaux and/or Frechet differential cannot be properly defined.

---

A Bregman Proximal Viewpoint on Neural Operators
Proof. It follows by integrating (12). The second part follows by Zalinescu (2002, Theorem 3.5. considering that + + g is also p-uniformly continuous.
We now provide conditions ensuring that the Bregman proximity operator is well-defined by guaranteeing that the subdiffer- ential covers the entire dual space. This is important because having full range means that every possible dual variable has a corresponding primal solution. Remark A.4.
(i) V = L") with p E |1, +ool, the condition ran 0( + = V is satisfied if Φ is p-uniformly convex (see Proposition A.3 in the appendix). Moreover, by Remark A.2, if = 1 and dom Ø' = R", then ran 0$ = V*. (ii) If instead of ran 0(++g) = v one asks the stronger condition ran (0h Fan(0++0gg) +dg) = y*, then we have 0 ((b-g) g) = +0-+0g and the Bregman proximity operator writes down as prox (0 + 0g) and ran(prox)) C dom ÔD.
Finally, we apply these results to iterative process. specifically the compositional form of (Bregman) neural operators, to ensure well-posedness at each step. By confirming that the proximity operator consistently maps to the correct domain, we establish stable recursive structure. This prevents domain mismatches and ensures the validity of compositions. Additionally, we emphasize the importance of a compatibility condition on the lifting operator, which guarantees a well-defined initialization for the iterative scheme.
Remark A.5. In view of Remark A.i(ii), the condition (0n, + 0g.) V implies that prox (04, + 09:)) and hence ran (prox ) C dom , In this way dom L. = ' (dom 0.,-1) and (an( C dom 0b, and the composition (2) is well-defined provided that for the lifting operator P it holds ran(P) C dom 0dp (c.g.. if (x) = (Pv()()))
A.2. Link Between Activation Function and Proximity Operator
As demonstrated in the work of Combettes & Pesquet (2020a), many activation functions a can be expressed as proximity operators prox, = argmin,ER g(t) + ( - t)² for some appropriate convex function g. The simplest case is that of the ReLuu activation function, recalled below.
Example 1 (ReLu). The rectified linear unit function a: t E R max(t, 0) I can be expressed the proximity operator prox, of g +00 Henceforth, prox, reduces to the projection onto the positive orthant.
We also provide a novel characterization of SoftPlus.
Example 2 (SoftPlus). Given 3 > O, the SoftPlus activation function, i.e, az + SoftPlus;(() A (1/8) log(exp(31) +1), the proximity operator of
1 Liz(e (t) E R2o:
g: 1200 +
(13)
32
where Lig is the dilogarithm funciioonn defined as Lig; t - So lom(1-w) du.
"
- 1) with h()) = Lig(e B1) + (1/2)(s - 1)2 = (t) - st + (1/2)s² where we introduced (t) = (1/82) (Liz 01) + (1/2) log(c (0))) (1/B²){ ʃ (1g(r/(1-))//a
Proof. For every 8 E R, (r)x,()) = argmingeR A s(() + (1/2)(8-1)3
The latter can be written as (t) (1/3) l'log(e 1)dr up to constant. Finally,
since h is strongly convex, the minimum is attained for t such that h (t) = 0. which yields log(e"* 1) = 3B + t = o thus ending the proof.
We present an illustration of the convex function defined in Eq. 13 in Figure 7a. Intuitively, it serves as a smooth surrogate for the indicator function of the positive orthant tjo,+oo[- A larger value of 3 > leads to a closer approximation. This aligns with the representation of SoftPlus as the proximity operator of from Eq. 13, depicted in Fig. 7b where a larger makes SoftPlus closer to ReLU.
12

---

A Bregman Proximal Viewpoint on Neural Operators
0.100
B =
0.075
3
B
g(t)
0.050
$ 10
prox
P 10
0.025
0.000)
2
(a) Representation of g(t) gz Lia(e -B)
(b) Representation of prox, (1) = SoftPlusp (t)
Figure Illustration of SoftPlus as a proximity operator.
B. Approximation Results for Bregman Neural Networks and Operators
B.1. Bregman Neural Networks
We consider first shallow Bregman neural networks for finite dimensional spaces. Let a: T be a homeomorphism, where / an open interval in R. We d E N4 and
(a 14) span (a(o 3mm x)) Tx) + w x ++ m E Ad-1 R E RJ.
(14)
Remark B.I. Since m belongs to the standard simplex A dd-1, m Ta is a convex combination of elements and so it an element of I. Thus, since a -1; IR, the funetions in BN2(a; 14) are well-defined from 1d + R.
The following result follows from an adaptation of the argument in Cybenko (1989) to our different architecture (14). Theorem B.2. Suppose that a is sigmoidal, meaning that lim (t) = 0 and lim (t) = 1. Then, the space (N 1 is dense C(Id, ; R) with respect to the topology of uniform convergence on compact sets.
Proof. Let C Idd be a compact set. We prove that the trace space (N I*) is dense in C(K, R). To that purpose, we rely on the following general fact concerning dense sets in Banach space (see, e.g., Brezis (2011), Let B be Banach space, let A c B. Then the following propositions are equivalent.
spanA is dense in
A'' = {u" E B VuE A: (u, u²) = 0} = (0).
Vu" E B", (VuEA: uu, = 0) = u 0.
This implies that for our purpose we can equivalently prove that
E (+ BNa(a; / f- =0= = 1 0,
where M(K) is the space of signed finite Radon measures on K (the dual of C(K)). Thus, let be signed measure on K and suppose that
v BN2(o;1)): / fdu O.
(15)
Fix w E Rd. Ad-1, and b E R. Define, for every > 0 and c C
Txe: I R,
I x o o (mTJ)) Tx) + A(w Ta x + b) +
is clear that ax. BNg(a;1d) Moreover,
11
1
if w T + 0 if n x b 0
lim xx0(x) A++00
0
= >2x)
a (m) +
if w Tr 0.
13

---

A Bregman Proximal Viewpoint on Neural Operators
Define the sets
lobb = f w Tu xx 0}: W = {a E K w x + 0}, 1w for E K www x x +++-=0 m000=0)) 0).
They are intersections of half-spaces and hyperplanes with K. So.
7(x) = =x (x) a( - (m"l xx) + C) (z),
where Xa is the characteristic functions of the set A ld Since is bounded apply the Lebesgue's dominated convergence theorem and get
xx / ydp = ((TT
f
lim +00000
alo- c)dp(x))
K
In,
Note that the integral on the left zero by the hypothesis (15). this way we proved that
S (m x)) + ccdp(x)) = O.
mm E A Au- V.
RR Yb, R: m(II)))) +
(16)
JT..
Now observe that (16) implies
(m x) + c)du(x) $ / lolo-1(m' [x) + c))((() + 0 as c + -00;
(((Ttt
m.b =
Ilw.a
11m
since la(o- (m x) + c)l + 0 aa c -00 (pointwise), where |e| is the total variation of Therefore, 1( = 0.
Then (16) yields
In o(o (m Tx) - c)du(x) = 0.
R:
Moreover, by assumption (m (anT)) x:) + c) a c too (pointwise) and hence, again by Lebesgue's dominated convergence theorem,
a (m (mJx) x) + c)du(x)= = S TT. 1dp ((Il,,)),
lim
TT..
which yields μ(Ilw,)) 0. In the end we proved that the measure is zero all the sets of type
1e and
II w.
Now the proof continues in Cybenko (1989, Lemma 1), and we can conclude that O.
Now address the vectorial case. We
BN2(a; 1 (*)
(oo( + Wx +b)
r e N, Q e W,M E Rrxd with M right stochastic, and b R
where a and σ -1 are applied component-wise.
Corollary B.3. We have that
BN 1d, Td, (BN2(o; 1²))* = BN2(o; 14) x .... BN2(a; 1d)
(17)
times
and it dense in C(Td, ) in the topology of uniform convergence on compact sets.
14

---

A Bregman Proximal Viewpoint on Neural Operators
Proof. In view of Theorem B.2, it is clear that (oN 14) is dense in C(14, IR C( R in the topology of uniform convergence compact sets. Let's prove equality (17). The inclusion (N 1 R c (BN2(a; Ta) is immediate. Let f: ld + R* with components fj BN 14), = ,,...,k. k. Then, there exists E N4. and for each j E {I,..., k).
4 E R W,, EER" ggxd, by IR," and My R2xd right stochastic matrix (the rows are positive and sum one), such that
fz(2) - o o Wzz b,).
Then considering the block matrices
91
Mi
Wi
bi
Q2
M
E Rkrxdd
W
E Rkrxd
b
E
RR.
Q
R
M
W
by
4
we have
f(x) Qo(o Wx b),
and hence f E (N 14, R*). The statement follows.
A general deep Bregman neural network with 7' layers is defined as follows
(N ( I
where, for every T
L:: 11- + 1" x x (o (Mar) + Wx + bb),
(18)
with W, E R b, E Th and Mf E R right stochastic, for - 1.... T- with ni and W7 E R
Note that also the dimensions ,1...,,7T-11 can be chosen freely. Clearly for deep network with >>> 2, if we take, for every f = 2...., T 1, n4 n1, W, = 0, b = 0, and M, equals to the identity, then the layers L, with = 2, T act as the identity operator and hence
BN 1 R*) c (o; TR)
Therefore, 14, (+)iss is dense in c(I4, R for the topology of uniform convergence on compact sets.
Remark B.4. Often in applications it is desirable to have functions defined on the entire space RRd, In this case one can simply precompose the functions in (N Id R$) by the homeomorphism
x Rd o(x) Id
obtaining a dense set in C(Rd, (for any T 2 2). Such space is then denoted by (Nr )
Let D Rd be any nonempty bounded open set. If F(RR) is any class of real functions from RRd to RR we denote by F. D the set of restrictions to D of the functions in F(Rd). In the following according to Remark B.4 we put
BN 18 l (Wr LT L1 oo),
(19)
which is dense space in C(Rd, RR*) with respect the topology of uniform convergence on compact sets.
Lemma B.5. Suppose that is a sigmoidal activation function as Theorem B.2. Let p E [1,+ Then BNr(a; Rd. R) is dense in (D, R²) (in the norm I1-Il.).
Proof. is well known that (D,IR) dense in LP(D, R$) and hence C(R", R²) ) is dense L*(D,R*) (in the norm I1 Moreover, (NT(o; R", R*) is dense in C(Rd, jR* ) (in the norm I1-11..). On the other hand
vf E C(Rd, RR) 1lll,, (1Pda))" 1/8 < I1lfll 1D/1/
15

---

A Bregman Proximal Viewpoint on Neural Operators
Thus, if f E LP(D, R*) and
gg E C(R", RT) s.t. IIf 1l1p -
3h BNr(O; Rd RD s.t. 11g - 10 -
11g - 1 S 2
2/1/
and hence ||ʃ h e.
Remark B.6. It sometimes required that neural networks, of any depth, include constant functions. Standard feed-forward neural networks have the form
(Wr +br) oWW--1 +br-1)
oWW +b1),
so it is clear that they include constant functions (just take WT = 0). However, for Bregman neural networks as defined in (19)-(18) this is not clear. An immediate modification to achieve this goal is to explicitly add a constant by in the last layer. Another possibility is to lift the input space by one dimension, precomposing the neural network with a (free) linear embedding. In particular, if we consider the canonical embedding
J: RRd Rd+1,
x
0
and define the following matrices
W.
Mr 0
-000 (for
be
W. 1
M.
be
Wr Wr br/a(0)];
1
then, for t = I,... T - 1, according to (18), we have
Vy E 1 i, [00)
= o (0
(001(x Mi (2(000) + We [o(0]
+ +bb = - [(0)
Lyy
and hence
Wr o
W LT-10 1100
B.2. Bregman Neural Operators
Now start addressing the proof of Theorem 4.1. We will rely the work of Kovachki et al. (2023), from which, for the sake of reader's convenience, report the following facts.
Fact (Lemma 28 and 30 Kovachki et al. (2023)). Let D Rd be a bounded set and let L E (Wm'"(D))", for some m and 1 $ p < +o,, or L E (C(D))". Then, for any closed and bounded set K C A and 0, there exists function (000 (D) such that
((v) - <(x))((z)dx <s.
sup rEK
Fact 4 (Lemma 22 and 26 in Kovachki et al. (2023)). Let D c 12d be a bounded set and let A and ii be any one of the Banach spaces C(D) or (D),(D), with m 2 0 and 1 < p < to0. Let G: ll continuous operator, K C A be a compact set and & 0. Then there exist J E N and
R: A- JR'' S:R'' + uu,
with and S linear continuous and f continuous, such that
" [1 fo ( <e.
sup
In the following we set D C. dd be a bounded set and
A(D, Rm = (Dm. or
A(D,R = C(D, RT))
16

---

A Bregman Proximal Viewpoint on Neural Operators
where the integer 2 0 and p E [1, too Moreover will assume that (by possibly changing the definition slightly) Bregman neural networks include constant functions (recall Remark B.6). Because of the density result given in the previous section, we can essentially follow the same line of arguments in Kovachki et al. (2023), but we need to take special care of the different structure of Bregman neural network/operators (in particular in Lemma B.10).
Lemma B.7. Let L E A" and K C A be compact set. Then there exists (o; R such that
L(v) - -f (h(()), dx <.
sup vEK
Proof. The space is (isomorphic to) product space, meaning A = -1==1 A,, where A; is space of real valued functions on D. Set K, = (K) which is compact set of A,, so that K C I K, Then L: A + R can be written as Lu = n L with L2 A; + R. By Fact 3, for every = =1,... 1,..., no. there exists Hi E CD(D) such that
L So Kil; dæ <
sup o
2no
Let Co( with components E CD( Then
Le (x(()), dr = Σ 1=1
L / Kg di S
Σ L,U;
/ K; dxl <
6 2.
2
Since A C (D, RTT)) 7 suPtek ||0|| the space of continuous functions (Remark B.4), there exists h E (N Rdd _2")) such that ||/ Koloo S =2 and hence, for every NEA K,
+00. Moreover, since Bregman shallow neural networks are dense in
1/, (n, u)d - S (h, u)d = 1 (n h1, (1, dxl $ |m((0)) (o(l))--o)llv)))ell & l| 11low |/ll < 6 /
2
Therefore,
Lu 200-fol (1, dx S Lo 1 (1x, (1, dx + (M. dr de vie-/oo (h, 2) do <E
and the statement follows.
Lemma B.8. Let R: A- + R be a linear continuous operator, C compact set and e > 0. Then there exists a linear continuous operator RBN; A + Rd acting
v + RBN / h(y)u(v) dy,
where h E BN2(a; R.. Jxno) (T such that
Rv RBN ml < <E
sup o
Proof. Consider the components Ry: A + jR j= 1, I.....J. J. Then Rg E A", and by Lemma B.7
RSv - / (1)(0)) o(x)) dx < e
Th E BN2(0; Rd, Rno)D s.t.
sup ek
vJ
Let h: Rdd - Jxno with
hn(())
h(z))
T
Clearly h (R; R.R.RRRRRR.. IR xo I and
VuEK: Rv - / h(x)u(x)) dx
2
Σ 1=1
R₂ / (1)((())
do <E²
and the statement follows.
17

---

A Bregman Proximal Viewpoint on Neural Operators
Remark B.9. Both the linear continuous operators R and ReN in Lemma B.8 be canonically lifted to Lebesgue spaces follows.
R: LDD, Rv = (RU)LD RBN A + LD,,
RBN - (R²MM I1:
where D denotes the constant function I + on D. Moreover RBN actually an integral operator. Indeed if we define the kernel
Kh: D D Jxmo, kx(x,y)= = :hyy
we have
(RBNu)( 02) = RBN 8 h(y)u(y) dy= I (z( y)u(y) dy.
The following result is the analogue of Kovachki al. (2023, Lemma 35) and establishes that finite dimensional Bregman neural network can be canonically lifted in Lebesgue spaces. However, here we need to take care of the domain of the Bregman operator layers.
Lemma B.10. Let f RR, D Rd nonempty open and p E [1, +oo). Then there exists neural operator
AMNN TP(D, TP(D, N Kr 1 L1 00,
for every 1 = 1.....,
- olo Kev + by)
and such that the linear integral operators Me and Kr and the functions by are defined (purametrized) by finite dimensional Bregman shallow neural networks and
VWER:: A/ (WNN) f(w)LD,
where D denotes the constant function
+ on D.
Proof. By definition
f = Kr o LT-1 o.... L,00 L.(w) = o + K,w + by),
where R - and, for 1. = 1, 1,..., T, K₁ E Re and by E R and for every t 1...., 1, M, E R is right stochastic, no = J and nr = J'. Since, we are assuming that Bregman neural networks contain constant functions (recall the sentence before Lemma B.7), we have
b.ll BN2((;; Rd, R"*) C C(D, "*))
1 KIIDDD E (N Rdd x IRd, p x DDD C CCD D,
Kr
R7 xmx-1) and
D
K4: IP(D, (D R= + I9(D,R" RT))
e + (K) = / ne ne(x, g)(z)))oly) dy = - 1 Kovl)) = KTū,
where 0 is the mean value of v. So that K.v = (K,ū)1 d is constant function.
# 1
MI R RM. DXD C C(D x D.
D
My L' L'(,,,
((( S. [
u ヒナ (Mc)(
Mev(y) dy M,U.
18

---

A Bregman Proximal Viewpoint on Neural Operators
Moreover, since Mj is right stochastic, if the function has range (almost everywhere) in ["=,,, have that 8 ma11 = Mj m", Hence
M.(dom - C dom a
Indeed, recall that Φ,: L"(,,R")) +1 +1-00, +o0| and
AA E IP I (1(1)) / (c(u(x)) do, YWER" (r(w)) Σ (w)) i=1
N1
with wh: IR - -000 too Legendre, int(dom (b) I, dom 3 RR R. (1²)) and a - w², so that dom 0b;
{o E LP(D, I*)) for a.c. e D, v(x) [ and for dom D, 0b, [Voor).
It follows from the previous considerations that if u E dom C LP R" 1), we have K,(o) = (K,ō)1 and M2u = (M, p. and hence
Le(1) a(a aaooao(MMe (Mez) K&v (b) (Mai) + K2 + bb)-
Note that here Vi = (, ). Thus, we have
(())=(L))))))
meaning that the operator layer L. transforms any function in L"(D, into constant function, where the constant is the mean value of the function, transformed via the standard (finite dimensional) Bregman layer IT. particular, if w e R.y we have
[i(o(w1p)) = L((o(w) = (I (2(((((((1))))) = L=(L)(o(w))LLp) = (2(L)o(w)))
and Therefore, if set
A/BN_ = Kr o 1 L1 o0.
the statement follows.
Remark B.11. Let S: RJ' UUD, RR) be linear (and continuous) and set
Vi = 1, ,...,,',''' J': sy Se, E UI
where (e)) is the canonical basis of JR.." Define the function 8: + Rkx with s(x) = [81( ... (.85(x)),,, which has the 8j as columns. Then
j' wgej
3'
Yw YWER' Sw = S
Wy88 = (Sw)(z) = Σ wyss(z) = s(x)w.
Thus, the action S can be represented by matrix-valued function with columns in U. Moreover, the linear operator can be lifted to linear integral operator from LP(D,R'' U. Indeed if we define the kernel
:g D D R
k)(x, D s(x),
for every LP(D,R RR),, have
(Sv)(x) = -/1 (a(l, dy J, 1 ((0))( = s(x)o,
where ū is the mean value of In the end S: (D, Rd) ld and
Vo e (D, Se= ST.
and hence, for every w E RJ,, S(w1p) = Sw, meaning that S is actually an extension of S to the Lebesgue space (D( ).
19

---

A Bregman Proximal Viewpoint on Neural Operators
Lemma B.12. Let S: RR' UD, R be linear (and continuous). Let K C R be compact and Then there exists a function h E BN2(0; R Rkx D that for the corresponding linear operator SBN, R ll defined
YWER' ())N = Σ
wyhy(a) = k(x)
1=1
according Remark B.11, we have
sup |Sw - SSB wlll <. weK
Finally we are ready for the proof of Theorem 4.1.
Proof of Theorem 4.1. It follows from Fact 4 that there exist J' E N and
R: A-
RR, S: R +u
with R and 5 linear continuous and f continuous, such that
[[g(v) (S fo foR() <E. "E
sup
Now, taking advantage of the previous lemmas we want to replace the operators R and S with analogue operators depending on shallow Bregman neural networks, and the function f with Bregman neural network. follows from Lemma B.8 that for every n E N there exist
1
BNN A TR. linear continuous operator such that sup Re
Ra ul <
v
++1
where REN depends on Bregman shallow network hn as specified in Lemma B.8. Clearly this implies that limn RENU v= = Rv uniformly on K, so that the set
K₁ R(K) U U (K)
(K) c CR'
nEN
is compact (see Kovachki al. (2023, Lemma 21)). Since is continuous, it uniformly continuous on K1, hence given e > 0 there exists 5 > 0 such that
Yw, w E K1: |w - w/1 <5 = If(w) - fww')l <
e
3 |||
Moreover, there exists fBN E (N2 R,, Rd) such that
sup [f(w) - ,BN (BM(2)) < 3|
we
Let's take n E N such that 1/(n + 1) < 8. Then,
VUEK: Rv, RBNN v E K1 and Rv RBN vl <
< < 8 + |f( - F(1R8N))) ا)1
E
n + 1
S|
Finally, since (Kn is compact, by Lemma 1., there exist SBN
Rd'
+ ll such that
||Sw - 8BN w/
SBN lue
E 3
sup
<
w (e)(K(
Therefore, for every U E K we have
[((()))) - SBN (BN()BN (R w)) < ([S((( S(((((N (() o)) + [|S(((R - S(S ( v)) 0))llx
+ +1 u)) SBN (J8N (12EN u))
< ||SS] I((o)) - |BN (n b)1 + ||5|] 1/ (R BN - (12N
+ (1888
v)) SBN ( v))
_ -6
3 3
20

---

A Bregman Proximal Viewpoint on Neural Operators
In the end, for every E K, [G(e)
11) - [[[(v) - S(f( ||S(((R))) ( b)) llue 2E.
Now in order to conclude the proof, it is sufficient to lift the operators R and SBN Lebesgue spaces, as described in Remark B.9 and Remark B,11, and the function fBN to Bregman neural operator described in Lemma B.10 and recognize that
SBN N RR SBN faN R
Indeed, for every v E A, we have
(R, SBN (NNBN "N b)) = ((B =SBN (( (S
The statement follows.
C. Experimental Settings
We adopt the same experimental setting as in the PDEBench repository (Takamoto et al., 2022). For the sake of information, we recall the considered problems and PDEs and the specific settings we consider when appropriate. The learning procedure used is presented at the end of this section.
C.1. 1D Advection Equation
The advection equation is a linear Partial Differential Equation (PDE) modeling the transport of a fluid quantity u, namely its velocity field, defined by the following equation:
0,((2,)) t) 30zu(z, t) 0, x E (0,1),t e u(r, u uz(x), (0,1),
(20)
(21)
with 3 constant advection speed. Note that this system admits exact solution: u(t,x) = uo -tt.
For this dataset, we follow the setting given in Takamoto et al. (2022), Section D.I by taking 3 = 0.4. We learn the mapping between the value of the field at = 0 (u(x, 0)) and the value at time = 2 (u(x, 2)), Le we learn the mapping between the first and the last temporal value of each sample.
C.2. 1D Burgers Equation
The Burgers' equation is a PDE describing the nonlinear advection and diffusion a velocity field, defined as follows:
0eu(a, t) + O (u² (x, f)/2) = =1:2)=vv//Aa u(x t), æ E (0,1),t (0,2),
(22)
u(z, 0) uz(z), (0,1),
(23)
where v is the diffusion coefficient, which assumed be constant in this dataset.
We follow again the setup presented in Takamoto al. (2022), section D.2, with 0.001. As in the previous dataset, we leam the mapping from the field at = 0 as input to the field at t = 2 as target.
C.3. ID Compressible Navier-Stokes Equations (1D NS)
The compressible Navier-Stokes equations describe the motion of viscous fluids that can change in density due to compression or expansion. This can be described through the following partial differential equations:
0,0 + Oz (ou) = 0,
(24)
a (0,u + u.u. 0,u) -d,p + n nou (S + am..)
(25)
a, + o0² /2) + I,, [(p + + σv² /2)u u u o'| =
(26)
where σ is the mass density, u = u(x, t) is the fluid velocity, p is the gas pressure, e an internal energy described by the equation of state, a' is the viscous stress tensor, and r) and ( are shear and bulk viscosity, respectively.
21

---

A Bregman Proximal Viewpoint on Neural Operators
In our experiments, we consider the setup introduced in Takamoto et al. (2022), Section D.5, fixing 1/ 10 <- 10-
and out-going boundary conditions. We learn the mapping of the velocity from time t 10 as input time 1 = 11 as target. For this dataset, we added symmetrical padding preprocessing to replicate periodic boundary conditions (as prescribed in the original FNO code (Li et al., 2021a)).
C.4. 2D Incompressible Navier-Stokes Equations (2D NS)
We also consider dataset from the 2D Navier-Stokes equation for viscous, incompressible fluid in vorticity form on the unit torus (Li et al., 2021a) defined as follows:
0, 1) + u(x, t). Vwll 1) ()
a E (0,1)², (0, Finat] E (0, 10, E (0, Tyinat] E (0, 1)²
V w(, 1) = O,
(27)
w1z 0) w
with u is the 2D velocity field, V x u is the vorticity, wo (0, 1): - R is the initial vorticity function, E RR is the viscosity coefficient, and f: (0, 1)² + R is the forcing function.
We follow the setup introduced in Li et al. (2021a), Section A.3.3, with v = 10 and v = 10 -004
4. We learn the mapping of
the velocity field v from sample time = 10 to t = 45 for v = 10-3 33 and from t = 10 t = 15 for v/ = 10-4.
C.5. Darey Flow
We consider a dataset based on the steady state of the 2D Darcy Flow equation on the unit square, representing the flow through porous media and defined as follows:
VV((<)Vv(()) (()), u(x) = O, x 0(0, 1)2.
x (0, 1)²,
(28)
We follow the setup described in Takamoto et (2022). Section D.4, with f(z) fixed to the constant 0.1.
C.6. FNO Baselines
In this section, we further detail the FNO improvements considered as baselines.
F-FNO (Tran et al., 2023). The factorized FNO is a particularly relevant baseline for comparison, as it i) incorporates skip-like connections that share similarities with our additional a term and ii) also seeks to enable the development of deeper FNO architectures. We consider the best-performing F-FNO model (as identified by its authors), trained using optimization strategy and adapted to specific learning task. It is important to note that F-FNO was originally designed for predicting mappings between multiple consecutive time steps (e.g., from & to + 1) and it offers the option to rely on techniques such as the Markov assumption and teacher forcing. Since our task involves predicting the final state directly from the initial conditions, those techniques are not appropriate, and thus we did not include them in the implementation. Additionally, we have found that the original optimization strategy proposed by the F-FNO authors (AdamW with cosine annealing, noise injection and input normalization) did not perform well on our tasks, so we also employed the optimization strategy detailed in Appendix C.6.
ResENO. Moreover, to isolate the impact of residual connections from the broader structural modifications introduced by F-FNO, we have also implemented and compared ResNet-inspired variant of FNO, referred to as ResFNO. We did this to better understand the role of the residual connection.
C.7. Learning procedure
Models are trained using the Adam optimizer with constant learning rate, batch size of 128 for 1D problems (resp. 16 for 2D problems), maximum of 2000 epochs and an early stopping strategy with patience of 250 epochs and 8 = 10-3, The learning rate is validated on a grid of multiple values equally spaced in logarithmic scale. If not mentioned otherwise, we use 8000 (resp. 1000) training samples for ID (resp. 2D) problems, and 1000 samples each for validation and testing. All results are averaged over four random splittings.
22

---

A Bregman Proximal Viewpoint on Neural Operators
Experiments have been made internal clusters of GPUs with memory from 10Go 45Go. All the experiments be achieved with GPUs with memory of 10Go, except for models with 32 64 layers which require at least memory of 24Go.
A summary of the experimental setting along with some learning hyperparameter is detailed in Table 2.
Table 2: Experimental settings.
Dataset
Number of modes
Batch size
Width
Tinit
Tfinal
Train samples
ID Advection
[16]
128
200
8000
ID Burgers
[16]
128
64
200
8000
ID NS
[16]
128
64
10
11
8000
2D NS (10-1)
[12,12]
16
32
10
15
1000
2D NS (10-a)
[12,12]
16
32
10
45
1000
2D Darcy
[12,12]
16
32
1000
D. A Aditionna Results
D.I. Comparison of Predictions
In this section, we visually inspect to what extent the prediction made by FNO, ResFNO and BFNO is close the ground truth. We provide three examples on the Navier Stokes dataset with viscosity 10 where we have selected the best performing models.
Input
Ontgut
Topert
Outgest
PNO woidhal ResFNO noidual BFNO moidual
(a) Predictions
(b) Residuals
Figure 8: Predictions and residuals for Navier Stokes 10-4.
D.2. Extension Wavelet Neural Operators
We also extended our experiments Wavelet Neural Operators (WNO). In Table 3 is reported the comparison between standard WNO and the Bregman version BWNO. We observe similar results Fourier models, where our models outperform the standard models and are able to gain performance when increasing the number of layers. Furthermore, even with gradient clipping, 32 and 64-layer standard models could not converge during training, leading to 100% relative error rate. Further analysis shows that this divergence can be linked with the high error rates on low frequencies and boundary conditions.
D.3. Detailed Analysis of the Prediction Performance
In the same spirit of Takamoto et al. (2022), we include several metrics providing a deeper understanding the models' behavior, including relative mean squared enror on the boundary (rMSE) as well as in the low, mid, and high frequency bands (fRMSE low, fRMSE mid, FRMSE high). Results are provided Table
23

---

A Bregman Proximal Viewpoint on Neural Operators
Table 3: Relative error of WNO and BWNO models on benchmark PDEs.
ID Advection WNO
ID Burgers
ID NS
BWNO
WNO
BWNO
WNO
BWNO
4 layers
3.0 = 0.0%
2.8 = 0.2%
21.5 + 0.5%
21.3 + 0.4%
59.2 0.6%
58.3 = 0.6%
8 layers
2.5 = 0.1%
2.1 + 0.1%
19.1 + 0.6%
17.9 + 0.6%
59.0 + 0.6%
58,0 = 0.6%
16 layers
3.9 = 0.8%
-.2%%
19.7 + 0.5%
1.33%
61.1 0.6%
57.6 0.7%
32 layers
100 0%
0.1%
100 + 0%
16.4 0.5%
100 0%
57.2 = 0.6%
64 layers
100 = 0%
1.8 = 0.2%
100 - 0%
0..4%
100 = 0%
57.5 = 0.6%
Table 4: Additional comparison of the performance in terms of relative e error (rMSE), relative mean squared enror on the boundary (rMSE) as well as in the low, mid and high frequency bands (IRMSE low, fRMSE mid, fRMSE high).
T
T
T
16
Metric
BENO
ID Advectio
IMSE
1.55 10
2.43 - 10
1.45
10 3
3.22 10
1.43 10
4.38 10
ARMSE
9.04 10
1.21 10
8.28
10
1.51 10
7.82 10
2.43 10
FRMSE low
4.56
9.38
4.45
10
1.14 10
4.21
1.93
3.89
-6
3.39
-6
10
1.27
IRMSB high
3.15
4.87 10
2.89
10 -7
5.96 10-
2.76
8,08
IMSE
8.24
8.28 10
5.83 10
8.16 10
4.67 10
7.92 10
ID Burgers
ARMSE
3.83 10
3.69
2.41 10
3.65
1.85
3.72
FRMSH low
5.18 10
4.88-10
3.01 -5
4.52
2.37 10
IRMSB mid
3.41
3.44 10
2.52 10
3.61 10
2.01 10
3.29 100
IRMSB high
1.17
1.32 10
1.01 10
1.33 10
8.71 10-7
1.29 10-
ID C10 8)
IMSE
4.91
5.05
4.90 10
5.22 10
4.86 10
5.35 10
BRMSE
2.16
10
2.06 100
10
10
3.18
IRMSE low
2.
2.78 10
2.65 10
2.86 10
2.57 10
2.91
IRMSE mid
2.19
2.26 10
2.18 10
2.34 10-4
2.13 10
2.43 10-
IRMSE high
1.12 10
1.11 10
1.11 10
1.13 10-6
1.21 10-5
1.13 10-1
IMSE
9.99
1.00 10
10
10
100
Darey"
bRMSE
1.45 10
1.47 10
1.43 -2
1.52 10-
1.41-10- 10
1.40 10
FRMSH low
6.41 10
6.44 10
6.35 10
6.45 10
6.37 10
6.52 6.
10-
IRMSE mid
3,29 10-5
3.21 10
3.23 10
3.17 10
3.15 10
3.23 10-5
IRMSE high
1.68
1.99 10
-8
-6
10
10500 3)
IMSE
5.49
5.65 10
5.23 10
5.48 10-1
5.41 10
1
BRMSE 2.88
2.97 - 10
2.75 10 10-2
2.92 10-
2.83 10
2,84 10-2
IRMSE low
5.82-10-*
5.98 10
5.59 10
5.76 10-1
5.75 10-1
5.70 10
SNC C
1.46-10
10
1.11
IRMSE high
1.31
8.69
9.87
1.01
1.14
:MSB
1.31
1.34 10"
1.25 10"
1.38 10
1.23 100
1.43: 10"
ARMSE
7.16 10
6.74 10
7.45 10
6.68 10
7.73
CIDSSCA CE
IRMSE low
9.54
1.00
9.94
1011100
IRMSE mid
7.34
7.51
6.93 10
7.89
6.83 6.83
7.98
IRMSB high
-129-10"
1.33. 10
1.22 10
1.43-10
1.21-10
1.48-
D.4. Impact of the Activation Function
limitation of our framework is the fact that requires Bregman variants (such as BFNO) to have a strictly monotonic activation function, which excludes few functions such ReLU. This justifies why in our experiments used Softplus as surrogate of ReLU, On the contrary, for classical neural operators within our framework, the activation function only needs to be monotonic, not strictly monotonic. Therefore, ReLU is still valid and can be used.
As a thought experiment, we also implemented BFNO with ReLU and evaluated it on the 2D Navier-Stokes dataset (p=10-4] 4). Table 5 shows that BFNO with ReLU achieves comparable or better performance than Softplus for the same number of layers. However, the best results are the same (i.e., 12.2% for 16 layers).
24

---

A Bregman Proximal Viewpoint on Neural Operators
Architecture
4 layers
8 layers
16 layers
FNO (ReLU)
13.5 = 0.1
13.0 + 0.1
12.6 4 0.1
BFNO (Softplus)
13.7 + 0.1
12.6 + 0.1
12.2 = 0.1
BFNO (ReLU)
13.4 + 0.2
12.2 + 0.2
12.2 + 0.1
Table 5: Comparison BFNO with Softplus or ReLU.
D.5. Impact of Batch Normalization
For all the experiments presented in the previous sections, we relied on the latest available version of the FNO implementation, which does not include Batch Normalization (BN), while was used in the original FNO paper (Li et al., 2021a). We note that the original FNO code was removed from the GitHub repository by its author (i.e., the 'master' branch was deleted). While we retrieved an earlier version of the code, we observed that BN was implemented in the initial commit but was subsequently removed in later commit titled "remove unnecessary batchnorm", suggesting that adding BN layers does lead to better prediction performance.
To complement results, we have conducted an experiment with BN for both FNO and BFNO architectures the 2D Navier-Stoker dataset (v=10* 1). Results, reported in Table 6, show marginal improvements for 8-layer models (FNO: 13.0% + 12.8%, BFNO: 12.6% - 12.4%) but no consistent benefits for other configurations. This aligns with the conclusion of the recent FNO implementationn that removed BN.
Architecture
4 layers
8 layers
16 layers
FNO FNO + BN BFNO
13.5 = 0.1
13.0
12.6 + 0.1
13.5 +: 0.1
12.8
12.6 + 0.1
0.1
12.6
12.2 + 0.1
BENO + BN
1.1
12.4 0.0
12.3 + 0.1
Table 6: Impact of BatchNormalization (BN) with FNO and BFNO
25


Reviews:
[{"review_id": "Vm2Lx8VymR", "reviewer": "Reviewer_YcZa", "summary": "In this paper, the authors proposed a new type of neural operator for solving PDE problems. The idea is to set the neural operator to be the solution of a functional optimization problem, and in particular, they choose the operator to be a Bregman Proximal operator with respect to some Legendre functions. This formulation retrieves a lot of classical operators and extends to coined Bregman neural operators. The authors conducted experiments showcasing the power of the new operator formulation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for his positive feedback. We provide below our answers to the two questions mentioned in the review.\n\n1. **Normally, in the context of optimization, the Bregman divergence needs to be a strongly convex function so that it represents a notion of distance. However, here it seems it only requires it to be strictly convex. I am wondering why strictly convex functions suffices?**\n\nWe thank the reviewer for the thoughtful remark. Indeed, strict convexity of the Legendre function $\\psi$ is sufficient to define a Bregman divergence $D_\\psi(x, y)$, as it ensures that $D_\\psi(x, y) \\geq 0$ and that $D_\\psi(x, y) = 0$ if and only if $x = y$. These properties hold as long as $\\psi$ is strictly convex and differentiable on the interior of its domain — which is the standard requirement in the general theory of Bregman divergences.\n\nThat being said, we would like to emphasize that in our setting — as shown in Table 1 — **all Legendre functions considered are actually 1-strongly convex** on their respective domains. This ensures that the associated Bregman divergences are lower bounded by a quadratic form, i.e.,\n$$\nD_\\psi(x, y) \\geq \\frac{1}{2} \\|x - y\\|^2,\n$$\nwhich provides metric-like behavior. We have clarified this point in the revised version.\n\n\n2. **I am wondering what is the intuition of choosing the neural operator to be a solution of an optimization problem? In the context of optimization, proximal operators are often used for non-smooth problems since it has an explicit smoothing phenomenon. Is this one of the considerations for choosing a Bregman proximal operator?**\n\nWe appreciate the reviewer's insightful question. \n\nWhile it is true that proximal operators are classically used to handle non-smooth terms, especially in composite minimization settings, they are not restricted to non-smooth functions. In fact, proximal maps of smooth (even strongly convex) functions are well-defined and have been studied in the context of regularization and smoothing. \n\nIn our setting, the motivation for casting the neural operator as the solution to an optimization problem is not solely driven by the smoothing property of proximal operators-although this can be beneficial-but more importantly by their connection to activation functions.\n\nIt follows that, from our perspective, each layer performs a structured update—balancing proximity to the previous state with the minimization of an energy functional. This interpretation introduces a variational inductive bias into the architecture, which can encode prior knowledge and improve interpretability and performance. **Using Bregman divergences further allows for this trade-off to be geometry-aware**.\n\nWe hope this novel viewpoint will be leveraged by practitioners to encode structured prior knowledge into the architecture of neural operators through principled, optimization-based layers.", "claims_and_evidence": "yes.", "methods_and_evaluation_criteria": "yes.", "theoretical_claims": "I checked the proofs in the maim body.", "experimental_designs_or_analyses": "The authors conducted fair experiments on the benchmark datasets and compared them with many prior neural operators.", "relation_to_broader_scientific_literature": "This work extends the extensive research on designing neural operators and using neural nets to solve PDE equations. It formulated a new type of neural operator based on prior arts.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "I think this paper is clearly written with many useful intuition. Unfortunately I do not have an extensive background knowledge of using neural nets to solve PDEs and designing neural operators so I cannot provide a precise evaluation of the novelty and significance of this work compared to prior works. However, using Bregman divergence and Bregman operator seems to be an interesting idea, and the experiments showcased the superiority of this method.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Normally, in the context of optimization, the Bregman divergence needs to be a strongly convex function so that it represents a notion of distance. However, here it seems it only requires it to be strictly convex. I am wondering why strictly convex functions suffices?\n2. I am wondering what is the intuition of choosing the neural operator to be a solution of an optimization problem? In the context of optimization, proximal operators are often used for non-smooth problem since it has an explicit smoothing phenomenon. Is this one of the considerations for choosing a Bregman proximal operator?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mzysneQm0H", "reviewer": "Reviewer_Cumi", "summary": "This paper proposes a novel perspective on neural operators based on Bregman proximity operators, where the action of operator layers is interpreted as the minimizer of a Bregman-regularized optimization problem. By defining the Bregman distance through Legendre functions, activation operators are characterized as Bregman proximity operators mapping from the dual space to the primal space. To this end, the authors designed a novel operator termed B-FNO, and numerical experiments show better performance compared to classical FNO and F-FNO.\n\n## update after rebuttal\n\nAfter rebuttal I opt to maintain the score to positive.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for his time and thoughtful comments and appreciate the global positive feedback. First, we would like to make it clear that the main objective of our contribution is to provide a novel theoretical framework that allows the development of founded and effective models, which we will clarify in the abstract and introduction.\n\n# Questions\n\n**Q1:** Our theoretical framework formulates BFNO layers as solutions to Bregman regularized optimization problems. When all weights are zero, they naturally reduce exactly to the identity mapping (Remark 3.6), creating an implicit regularization effect distinct from standard residual connections.\n\nWe created a novel architecture, ResFNO, precisely to perform an ablation study with ResNet-style connection, confirming this distinction. While ResFNO still degrades with increased depth, BFNO maintains or improves performance. This difference manifests in Figure 5, where BFNO exhibits a distinctive Laplace-like weight distribution sharply peaked at zero, contrasting with the Gaussian distributions in other models.\n\nFrom an ODE perspective (see *Add. insights*), the difference becomes clearer: BFNO applies its linear operator outside the activation function $\\frac{dz(t)}{dt} = K(t)\\sigma(z(t))$, while residual networks place it inside $\\frac{dv(t)}{dt} = \\sigma(K(t)v(t))$, creating different gradient flow dynamics. \n\nOverall, BNFO's improvement can be analyzed as combination of these two elements: an implicit regularization and a different gradient flow. While a rigorous layer-wise analysis establishing a causal link between weight distribution and improved generalization would be valuable, such analysis remains an open challenge in the neural operator literature.\n\n**Q2:** The primary goal of our experiments was to validate our theoretical framework and demonstrate that the properties derived from our Bregman proximal viewpoint translate into measurable performance advantages. While focusing on the theoretical foundations, we ensured rigorous experimental comparisons to establish practical relevance (even by considering WNO models in Appendix D.2).\n\nTo ensure a fair comparison, we conducted thorough validation through multiple runs with four fixed seeds and cross-validation of learning rates for all models. To assess statistical significance, we performed non-parametric one-tailed Wilcoxon signed-rank tests. Compared to FNO, BFNO is superior on 5 datasets of Figure 6 with a p-value of 0.0625, with p=0.125 for the last one 2D NS $(10^{-3})$, which shows that improvements over FNO are consistent across diverse PDE types. Similarly, BFNO achieves the same level of superiority over ResFNO on 4 datasets (Burgers, Darcy, NS $10^{-4}$ and $10^{-8}$), and on 3 datasets over F-FNO (Burgers, NS $10^{-4}$ and $10^{-8}$).\n\n**Q3:** From a theoretical perspective, expressing activation functions as proximity operators imposes constraints: **Non-monotonic functions** like GeLU are incompatible, as they cannot be represented as proxs (page 5). For **Classical Neural Operators** (Sec 3.2), activation functions must be monotonic, making ReLU suitable despite its non-smoothness, as it can be expressed as a proximity operator $prox_g$ of $g=\\imath_{[0,+\\infty[}$. In contrast, **Bregman Neural Operators** (Sec 3.3) impose strict monotonicity, theoretically excluding ReLU.\n\nHowever, our implementation provides significant practical flexibility. As noted in Remark 3.7, our architecture doesn't require computing inverse activations when composing layers since they cancel out with previous activations (enabled by adding an activation before the first operator layer). Therefore, any activation function can be used in practice.\n\nResults below, completing Appendix D.4, confirm this flexibility, showing BFNO works effectively with both ReLU and GeLU, performing comparably to SoftPlus, which serves as a common smooth approximation to ReLU in the literature when differentiability is needed.\n\n| Architecture | 4 layers | 8 layers | 16 layers |\n|-|-|-|-|\n| FNO (GeLu)   | 13.7 ± 0.1 | 12.9% ± 0.2 | 12.9 ± 0.1 |\n| BFNO (GeLu)  | **13.3 ± 0.1** | **12.4 ± 0.2** | **12.1 ± 0.1** |\n\nAs suggested by the results, extending our theoretical framework to non-invertible and non-monotonic activation functions is an interesting research direction.\n\n# Remarks\n**On computational costs:** BFNO shares the same parameter count and memory footprint as FNO by design. Therefore, the training time per epoch is in the same order across all tasks. This will be clarified in the final version. \n\n**On universal approximation:** We indeed acknowledge this limitation in our conclusion. The theorem serves as a first theoretical result for our framework and represents, to the best of our knowledge, one of the first universal approximation results for neural operators with **Kovachki et al. (2023)**. \n\nAs for the other remarks, we tried to address them in the answers above while staying within the allowed limit.", "claims_and_evidence": "The claim that Bregman Neural Operators (BFNOs) enable deeper architectures with improved performance (Abstract, Section 3.3) is partially supported by Figure 4 (Page 7), where BFNO errors decrease as depth increases up to $T=64$, unlike FNO/ResFNO. However, the paper does not rigorously explain *why* the Bregman formulation stabilizes deep networks. While Remark 3.6 suggests the identity-preserving property helps, no ablation studies isolate this effect from residual connections. Additionally, Figure 6 (Page 8) shows marginal improvements (e.g., BFNO vs. FNO on 2D Darcy: ∼∼1% relative error reduction), but statistical significance is not tested.", "methods_and_evaluation_criteria": "The evaluation uses standard PDE benchmarks (Navier-Stokes, Burgers, Darcy) from PDEBench (Section 5.1, Page 7), which are appropriate for operator learning. But there remains rooms for improvement:\n\n- Computational costs (training time, memory) of BFNO are not compared to baselines.\n\n- The choice of SoftPlus as an invertible ReLU proxy (Section 5.1) is pragmatic but underdiscussed; no experiments validate whether this approximation introduces biases or artifacts.", "theoretical_claims": "Theorem 4.1 (Page 6) asserts universal approximation for BFNOs with sigmoidal activations. The proof (Appendix B) adapts Cybenko’s approach but assumes $\\sigma$ is a homeomorphism (e.g., sigmoid). However, the experiments use SoftPlus ($\\approx$ReLU), which is not sigmoidal.", "experimental_designs_or_analyses": "The experimental framework is broad and systematic, covering multiple PDE benchmarks (1D/2D Navier-Stokes, Burgers, Darcy) from PDEBench (Section 5.1, Page 7), which are widely accepted in operator learning literature. The comparison to strong baselines (FNO, ResFNO, F-FNO) and ablation studies (e.g., activation functions in Appendix D.4) demonstrates rigor. \n\n**Areas for Improvement**\n\nLayer-wise analysis: Figure 5 (Page 8) shows BFNO weights concentrate near zero, suggesting implicit regularization. However, no causal link is established between this distribution and improved generalization.\n\nActivation choice: Table 5 (Appendix D.4) claims BFNO with ReLU matches SoftPlus performance, but this is only tested on 2D Navier-Stokes. ReLU violates Theorem 4.1’s assumptions, yet no theoretical justification is given for its empirical viability.", "relation_to_broader_scientific_literature": "Numerical PDE by deep learning, such as Fourier Neural Operators and it's variations.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "The writing of the abstract and contribution section is quite opaque. It seems difficult for the general conference audience to catch up with the major contributions of this work. To improve readability, the authors should clarify whether the core contribution of this paper is a novel theoretical framework or the introduction of BFNO as a new state-of-the-art (SOTA) neural operator.", "questions_for_authors": "1.In Section 2.2, the statement \"we observe that this property allows training deeper and more accurate models\" suggests that the architectural design resembles skip connections, which enable deeper neural networks. Could you clarify the specific source of the BFNO improvement? \n\n2.The test error in Fig. 6 appears marginal. \n\n3.While BFNO demonstrates improved stability in deep architectures, does requiring invertible activation functions (e.g., SoftPlus) limit its applicability in scenarios where non-monotonic or non-smooth activations (e.g., ReLU, GELU) are preferred?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GOmn9VPzPT", "reviewer": "Reviewer_H357", "summary": "This paper considers the problem of efficient PDE solutions and operator learning. This paper shows that the neural operator architecture can be interpreted as the minimizer of a Bregman regularization problem, and further designs a novel architecture that includes an inverse activation function. This general framework is then applied to the Fourier neural operator and gives better empirical results. This paper also proves a universal approximation result for Bregman neural operators.\n\n## update after rebuttal\n\nThank you for your response! I will keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and the positive feedback. Should the reviewer require any further clarification, we would be delighted to provide it.", "claims_and_evidence": "The claims look well-supported; however, I have no background in this area and cannot verify the claims and proofs.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I didn't check the proofs.", "experimental_designs_or_analyses": "This paper compares different operators on various PDE benchmarks and shows that the Bregman Fourier operator gives the best results on most benchmarks.", "relation_to_broader_scientific_literature": "This paper gives a general framework to characterize neural operators as the minimizers of certain Bregman regularization problems. Classical neural operators can be interpreted as special cases of this framework.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is a paper whose overall appreciation was already positive before the rebuttal process, and stayed so during it.

I read the paper and the rebuttals / comments. The paper’s approach of learning layers using Bregman divergences is sound, and I appreciated the good balance between theory and experiments. The loss in (6) makes sense. The authors could have made a better job of explaining a few things asked by reviewers: take for example the question on generalization (Cumi). Why is solving such a problem eventually good for generalizations? One likely possibility is the fact that the Bregman formulation allows to control the Lipschitz constant of the model, which, for deep models, can be a guarantee for training behavior => (translates to) testing behavior as well.

I would also recommend the authors have a look at “one level” higher. Deep architectures have links with exponential families. The Bregman formulation equivalently is a maximum likelihood formulation for a classical exponential family, fitting the parameters of a statistical model through (6). But one can also get “above”, using generalized / deformed exponential families, in which case I am confident the same max likelihood formulation should uncover some interesting dependencies with the activation function. And finally integrating non differentiable activations like ReLU is not a hurdle.

Overall, a very interesting paper, which can both stand as a nice mark on the statistical optimization of deep architectures, but also offers some serious potential for further investigations on the topic.

A clear accept.
Decision: Accept (poster)