A Bregman Proximal Viewpoint on Neural Operators

Authors: Abdel-Rahim Mezidi, Jordan Patracone, Saverio Salzo, Amaury Habrard, Massimiliano Pontil, Rémi Emonet, Marc Sebban
Primary area: other topics in machine learning (i.e., none of the above)
Keywords: neural operators, proximal optimization, bregman divergence, fourier neural operator

Abstract:
We present several advances on neural operators by viewing the action of operator layers as the minimizers of Bregman regularized optimization problems over Banach function spaces. The proposed framework allows interpreting the activation operators as Bregman proximity operators from dual to primal space. This novel viewpoint is general enough to recover classical neural operators as well as a new variant, coined Bregman neural operators, which includes the inverse activatio and features the same expressivity of standard neural operators. Numerical experiments support the added benefits of the Bregman variant of Fourier neural operators for training deeper and more accurate models.

Decision: reject
Reviewer scores: 8, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730039862759

thread_type:
official_review

<thread_object>
creator: reviewer_n7dc

<content>
summary:
This is a primarily theoretical paper which proposes to address a lack of formalism for characterizing architecture in previous methods.  The authors propose to replace the basic layer of a Fourier Neural Operator, equation (3), with a more general layer, equation first written down in equation (4).  This architecture was actually originally proposed in Frecon et al. (2022) "Bregman neural networks".  A large part of the papers introduces technical theory from convex analysis.  University approximation results are presented for the architecture.  Numerical experiments are presented comparing FNO with BFNO.  Seven equations are solved. It is demonstrated Fig 2a, that on the Burgers equation, the accuracy of FNO degrades as the number of layers are increased while BFNO is more accurate.

strengths:
The authors bring the formalism of Bregman operators, which is normally used in convex analysis and optimization to bear on the analysis of neural network architectures.

weaknesses:
The paper introduced a heavy mathematical formalism without justification.  The ideas of the paper are not clearly presented and the theoretical contribution is not substantial. 

Overly technical:

- General conference audience will not understand the technical papers
- Neural operator specialists: will not understand the paper.
- Only experts in convex analysis will be able to follow much of the paper.
- Section 2 begins with a dense paragraph of convex analysis, unreadable to anyone who doesn't already know the area.  It's also unconnected to later sections, so not immediatly clear what is needed from this paragraph.
- Section 2.3 is an overview of bregman operators, bregman distance, textbook material, again not clear how much is needed.  "For additional details, the reader canrefer to Bauschke & Combettes (2017)"

The architecture is not clearly defined in the paper. 
- The architecture was originally proposed in  Frecon et al. (2022) "Bregman neural networks" which was not an influential paper.  In that paper, it was made clear that the architecture involves bilevel optimization.  This is not explained or made clear in the current paper. 
- In section 2.2. equation (4), which is a modification of (3), names the architecture, but it is not defined. 
Later, it is defined as a special case of (6), which takes half a page to write down. "In (6), the proximity operator plays the role of an activation function operator, which in general will have the form of a nonlinear Nemytskii operator. ... Moreover, differently from usual architectures (Kovachki et al., 2021), ..."
- Nemytskii operator is never defined. However, "This relationship is crucial for further establishing connections with neural operator layers."

Experiments: 
The numerical experiments are quite limited and show minimal improvement, or improvement in very particular situations, compared to FNO. 
"To this end, we conducted an experiment using the Burgers’ dataset with viscosity ν = 10−3, with results presented in Figure 2a. First, we observe that BFNO systematically yields lower prediction error, irrespectively of T. Second, the performanceof FNO degrades starting from T = 16, while BFNO demonstrates better performance as T increases until it reaches a plateau at T = 64. "

- Was this also the case for the other 6 equations solved, or just for this one?  Why did you just present the one equation

questions:
Questions: 

1. Explain the architecture in more direct way.  Is there a bilevel optimization? This was much easier to understand in Frecon et al. (2022) "Bregman neural networks"
2. "Previous methods often lack of a general formalism for characterizing their architecture."  But there is a well known paper that does:  "Neural Operator: Learning Maps Between Function Spaces.  How does the contribution of this paper related to that one?
3. "In this work, we propose a novel expressive framework for neural operators by conceptualizing the action of operator layers as the minimizers of Bregman regularized optimization problems over Banach function spaces."  This sentence does not make sense, please clarify.

4. "We prove universal approximation results". This is true of almost any reasonable neural network, including MLP.  How does this argument show anything special about this particular architecture?

- "the proposed framework allows applying the extensive body of literature on proximal numerical optimization, of which Bregman proximity operators belong to, in order to study neural operators."
  - convince me why this is useful.  
- "This opens the way to extend the analysis done on neural networks to (Bregman) neural operators in the same spirit of Combettes & Pesquet (2020a;b)."
  - Explain what would be achieved by this. 
5. In Fig 2a, did the same results hold the other 6 equations solved, or just for this one?  Why did you just present the results for Burgers equation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730289327699

thread_type:
official_review

<thread_object>
creator: reviewer_hy9q

<content>
summary:
This paper studies neural operators from the perspective of Bregman proximal optimization. The nonlinear activation layers such as (sigmoid, tanh, SoftPlus) are interpreted as the Bregman proximity operators. Based on the above optimization viewpoint, a new neural operator named BFNO is proposed as an extension of FNO, where an additional term sigma^{-1}(.) introduced in the expression of FNO before the function sigma(). A few experiments show that BFNO performs better than FNO especially for a large number of neural layers.

strengths:
(1) The main contribution of the paper is that the authors interpret the neural operators as Bregman proximal optimization. This opens up the possibility of bringing knowledge or theory of Bregman proximal optimization into the field of neural operators for many possible future work.  

(2) As I mentioned earlier,  BFNO is proposed as an extension of FNO by introducing an additional term sigma^{-1}(.) in the expression of FNO before the function sigma(), which I think has a similar effect as the skip connection in F-FNO or ResNet.

weaknesses:
(1) I think the authors should compare the performance of BFNO to that of F-FNO equipped with skip connection. This is because from a high level point of view, the introduction of an additional term sigma^{-1}(.) in the expression of FNO before the function sigma() is very similar to the skip connection in F-FNO. It is very interesting to find out which one performs better.  Personally I think F-FNO might perform better because the skip connection also has a strong motivation from the ODE point of view.  BatchNormalization can also be included in  F-FNO smoothly. If the authors are able to show that BFNO performs better instead with a good explanation, I would be happy to change my score of the paper. 

(2) The paper conducted theoretical analysis but were not able to show in theory why BFNO performs better than FNO, which I think is very critical. Instead, they conduct experimental results to argue the superiority of BFNO. This is also partly the reason for me to suggest the comparison between the performance of BFNO and that of F-FNO.

(3) It is not clear to me if BatchNormalization or Layer normalization can also be covered by the framework of Bregman proximal optimization. The reason I have this concern is that the FNO paper used BatchNormalization in their experiment. I would think doing so improve the training stability. If BatchNormalization cannot be covered by  the framework of Bregman proximal optimization, it suggests the limitations of the framework. 

(4) Another weakness is that the activation function needs to be monotonic for it to be invertible. This excludes a few functions such as ReLU and Swish.  I understand that SoftPlus is similar to ReLU but still it suggests the framework has some limitations.

questions:
(1) I wonder if the authors use BatchNormalization when implementing FNO in their experiments because the original FNO paper used it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655884019

thread_type:
official_review

<thread_object>
creator: reviewer_hezd

<content>
summary:
This paper proposes a novel expressive framework called BFNO to improve FNO by understanding the action of operator layers via the minimization of Bregman regularized optimization problems.

strengths:
The Bregman-based perspective on neural operators is intriguing, and the paper presents a variety of strong theoretical results.

weaknesses:
1. The writing is poor, and I recommend that the authors carefully revise the paper from start to finish, especially regarding newly defined matrices or functions. For example, in Eq. 4, the definitions of $M_t$ and $K_t$ are not clearly stated when they first appear in the paper.
2. The experiments are too simplistic. The authors only compared BFNO with FNO. I suggest including other FNO improvements as baselines.

typo: the operators in Line 370

questions:
Q: Which $\psi$ in Table 1 do you use in the experiments? Do you compare the BFNOs obtained from different $\psi$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730794747647

thread_type:
official_review

<thread_object>
creator: reviewer_jysk

<content>
summary:
This article constructs neural operators from the point 
of view of Bregman optimization problems. The proposed idea uses the dual space of Banach functional theory, 
and it allows to recover classical neural operators 
and define new ones. Numerical results 
on the newly constructed operators improve the accuracy of 
state-of-the-art results by using deeper networks.

strengths:
Although technical, the article is well-written and easy to follow. 

The contribution raises an important question on the choice 
of a metric/divergence in the functional space of the solution u.

weaknesses:
It seems that there is still a gap between the universal approximation result
and the numerical results in the article as 
the theoretical assumption about the non-linearity sigma (sigmoid type)
does not hold in the numerical models (sigma=softplus is not sigmoid type). 
Therefore it would be good to mention this gap in the conclusion.

questions:
- There is some notation inconsistency in the definition of the kernel K_t^ac in eq. 3
and the K_t in Section 3.1. Are you talking about the same type of kernel in these 2 places? 
Why do you use k^(t) as the kernel density, rather than the k_t as before (below eq .3)? 

- Section 3.1, it is unclear what the sigma_1 and sigma_2 after Remark 6 comes from, 
do they depend on g_t?

- Is bar{D} the closure of the set D in the definition of A in Section 4? 
Why do you consider the space C with \bar{D} rather than with D?

- some type in remark 9: no in?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732654689627

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Answer (Part 1: Revisions)

comment:
Dear Reviewers,


We sincerely thank you for your thorough evaluations and insightful comments, which have been instrumental in refining our work. **We kindly invite the reviewers to read our revised version**, where all the changes are indicated in blue. Below, we address key points raised across the reviews.

On the theoretical side, Reviewer JYsk highlighted a gap between our universal approximation results and their connection to the experimental setup, regarding the activation functions. Reviewer HY9Q also raised valid comments about the monotonicity assumption for the activation operators, suggesting that we further clarify its implications. In response, we have expanded our theoretical discussion to better connect these aspects, explicitly acknowledging the assumptions made and their influence on the scope of the results. We have also provided more detailed introduction and conclusion paragraphs to contextualize the universal approximation results and suggest avenues for future work.

Regarding clarity and presentation, several reviewers emphasized the importance of improving the accessibility of the paper. To address these concerns, we have moved the more technical parts to the appendix, and we have included two additional figures (Figure 1 and Figure 2). In particular, we believe that Figure 2 could motivate neural operator specialists to explore the connections with convex analysis. In addition, many parts have been reworked to provide more intuitions. **We are confident that these changes will enhance the readability of the manuscript and help make the contributions more accessible to a wider audience** despite its mathematical nature. 

In terms of numerical experiments, both Reviewer HY9Q and Reviewer HeZd suggested additional comparisons, such as incorporating baselines like F-FNO and exploring configurations with BatchNormalization. We have initiated further experiments to address these suggestions and we have integrated the results into the revised manuscript in the appendix. **Preliminary results are provided in our second general answer.** Finally, we want to stress a point that Reviewer n7Dc may have missed: our novel architecture has already been tested on multiple datasets (Table 2), with detailed results provided in the appendix (Section D). While space constraints prevent the inclusion of all findings, we could move some results in the main body of the paper if deemed useful.

We hope these improvements and the subsequent answers will address your concerns and encourage a positive reassessment of your evaluation.


Respectfully,

The authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732655168301

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Answer (Part 2: Additional Experiments) [1/2]

comment:
Dear Reviewers,

We once again sincerely thank you for your valuable feedback regarding the experimental aspects. In response, we have conducted additional experiments to address your concerns. Due to time constraints, the additional experiments were carried out on the 2D Navier-Stokes dataset ($\nu = 10^{-4}$) with the rigorous experimental setup described in Appendix C.6. In the final version, we will extend this analysis to all the PDEs considered in the paper.

**1. Comparison with F-FNO and Skip Connections.**

Reviewers HeZd and HY9Q expressed concerns about the lack of comparisons with other FNO improvements. We agree that F-FNO is a particularly relevant baseline for comparison, as it i) incorporates skip-like connections that share similarities with our additional $\sigma^{-1}$ term and ii) also seeks to enable the development of deeper FNO architectures.

We have included a comparison with the best-performing F-FNO model (as identified by its authors), trained using our optimization strategy and adapted to our specific learning task. It is important to note that F-FNO was originally designed for predicting mappings between multiple consecutive time steps (e.g., from $t$ to $t+1$) and it offers the option to rely on techniques such as the Markov assumption and teacher forcing. Since our task involves predicting the final state directly from the initial conditions, those techniques are not appropriate, and thus we did not include them in the implementation. *Hereafter, we refer to **F-FNObase** as the original architecture proposed by the authors, trained using their optimization strategy. Conversely, **F-FNO** denotes a model aligned with our experimental setup, featuring a similar architectural design (identical lifting/projection layers and operator widths) and trained using our optimization approach (see Appendix C.6)*. Results reported in table below show that the "out-of-the-box" model F-FNObase does not transfer well to our learning task, as evidenced by the poor performance across all layers, compared to the other models. On the contrary, with our training strategy and adapted hyperparameters, F-FNO shows strong performance, especially with few layers.

Moreover, to isolate the impact of residual connections from the broader structural modifications introduced by F-FNO, we have also implemented and compared a ResNet-inspired variant of FNO, referred to as ResFNO. We did this to better understand the role of the residual connection. As shown in following table, ResFNO consistently achieves lower training error than FNO, which aligns with the behavior typically observed in ResNet-like architectures. However, it falls short in terms of generalization to unseen data, as reflected in the test error.

In conclusion, BFNO demonstrates superior scalability than its competitors (see for 16 layers), evidencing that the additional $\sigma^{-1}$ term, inherent to our Bregman model, better facilitates the training and generalization of deeper architectures.

| Metric | Model | 4 layers | 8 layers | 16 layers | 32 layers |
|--------|-------|----------|----------|-----------|-----------|
| **Test $\ell^2$ (\%)** | FNO | **13.5 ± 0.1** | 13.0 ± 0.2 | 12.6 ± 0.1 | 12.7 ± 0.1 |
| | ResFNO | **13.5 ± 0.1** | 12.8 ± 0.1 | 13.0 ± 0.1|13.4 ± 0.1|
| | F-FNO$_{\rm base}$ | 15.4 ± 0.3 | 14.9 ± 0.4 |15.1 ± 0.2|15.5 ± 0.1 |
| | F-FNO | 13.6 ± 0.4 | 12.7 ± 0.2 | 13.0 ± 0.5 |13.4 ± 0.6 |
| | BFNO | 13.7 ± 0.1 | **12.6 ± 0.1** | **12.2 ± 0.1** |**12.0 ± 0.1** |

| Metric |Model|4 layers|8 layers|16 layers|32 layers|
|--------|-------|----------|----------|-----------|-----------|
| **Train $\ell^2$ (\%)** | FNO | 4.6 ± 0.2 | 3.6 ± 0.1 | 4.0 ± 0.1| 4.7 ± 0.0|
| | ResFNO | **3.7 ± 0.2** | **2.9 ± 0.1** | **2.8 ± 0.1** | **3.6 ± 0.4**|
| | F-FNO$_{\rm base}$ | 10.4 ± 0.3 | 8.7 ± 0.3 | 7.3 ± 0.1 | 8.0 ± 0.0|
| | F-FNO | 8.4 ± 0.4 | 6.7 ± 0.4 | 6.5 ± 0.8 |6.4 ± 1.2|
| | BFNO | 4.4 ± 0.3 | 3.7 ± 0.3 | 3.4 ± 0.2 | 3.7 ± 0.1|

**2. BatchNormalization Investigation.**

Reviewer HY9Q rightfully mentioned that the original FNO code used BatchNormalization (BN). However, as pointed in our specific answer to the reviewer, the authors of FNO have later removed such layer in their code. Since we have used the latest version, we have also not considered BN.

To complement our results, we have conducted new experiments with BN for both FNO and BFNO architectures. Results, reported in the table below, show marginal improvements for 8-layer models (FNO: 13.0% → 12.8%, BFNO: 12.6% → 12.4%) but no consistent benefits for other configurations. This aligns with the conclusion of the recent FNO implementations that removed BN.

| Architecture | 4 layers | 8 layers | 16 layers |
|--------------|----------|----------|-----------|
| FNO | **13.5 ± 0.1** | 13.0 ± 0.1 | 12.6 ± 0.1 |
| FNO + BN | **13.5 ± 0.1** | 12.8 ± 0.2 | 12.6 ± 0.1 |
| BFNO | 13.7 ± 0.1 | 12.6 ± 0.1 | **12.2 ± 0.1** |
| BFNO + BN | **13.5 ± 0.1** | **12.4 ± 0.0** | 12.3 ± 0.1 |
</content>
</thread_object>

</thread>


Metareview: