Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper analyzes Deep operator Networks (DeepONets) regarding the optimization convergence guarantees. The author provides optimization analysis on two different networks: smooth activations and ReLU activations. Their analysis shows that the overparameterization on branch and trunk networks allows faster convergence to the global solution. 
strength_and_weaknesses: Strength

1. The paper is well-written and easy to read.
2. The author's convergence analysis includes both smooth activations and ReLU activations.
3. Theoretical results also match the empirical observations.

Weakness
1. Deng et al. (https://arxiv.org/abs/2102.10621) have studied the convergence analysis on DeepONets. The author should cite this work in the modified draft.
clarity,_quality,_novelty_and_reproducibility: Clarity: Well written. Please add the Deng et al. to the related work.
Quality: Good
Novelty: There exists a previous literature on convergence analysis of DeepONet. However, the analysis technique is different. 
Reproducibility: N/A
summary_of_the_review: Overall, I believe it is a good paper to have the better understanding on DeepONet.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper provides convergence analyses of gradient descent for learning deep, over-parameterized operator networks. The authors show the guarantees two types of networks: (1) with smooth activations and (2) with ReLU activation using two different analysis techniques.

strength_and_weaknesses: Regarding the strengths, this paper
+ establishes first guarantees of GD for deep operator;
+ shows the non-trivial convergence analyses, albeit using standard analysis techniques (restricted convexity and NTK).

However, there are some drawbacks: 
- The authors do not explicitly state convergence results in dedicated theorems, so it is difficulty to understand the full picture (e.g., convergence rate, role of learning rate…) In particular, while I understand how the analysis goes, I cannot see how the width (or the over-parameterization) attributes to the overall convergence.
- The paper presents two different analyses due to the smoothness without discussing limitations or why one can't apply one to the other.
- The experiments are based on Adam, an adaptive SGD, instead of GD.

clarity,_quality,_novelty_and_reproducibility: Some questions and comments about clarity:

1. Why do the learning curves (in Figure 1 and similarly 2 and 3) fluctuate more with larger width?

2. How does m affect the local convexity in the smooth activation case?

3. Any intuition why the NTK is already positive definite at initialization? Does that hold true with u being an identity function? 

4. After Definition 2, L is α-RSC w.r.t. (Q, θ) not (S, θ)? And, in Theorem 4.2, the sum should be gradient of \bar(G)_t not \bar(G)_t itself?

summary_of_the_review: The analyses presented in this paper are non-trivial, but I think it needs some more work to be ready for acceptance (e.g., more explicit convergence results, discussions about limitations of each of the analysis techniques....)
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the optimization convergence guarantee of GD for training DeepONets with wide layers. Technically, the authors prove the convergence for both smooth activations and ReLU activation and show that a linear convergence rate can be achieved. The authors further present empirical results to back up the theoretical findings.
strength_and_weaknesses: Strength:
1. This paper proves the convergence for DeepONets with smooth activations based on the RSC condition.
2. This paper proves the convergence for DeepONets with ReLU activation based on the NTK analysis.
3. This paper conducts numerical experiments to justify the theory.

weakness:
1. The initialization of weights in Assumption 3 seems unreasonable to me. In fact, the authors consider using a small scaling for only the last layer, then the optimization on the last layer weights will dominate the training of the remaining weights. By (5), the neural network model will be very close to the linear model. 
2. It is also not clear to me why the authors need to use two sections to study the convergence of DeepONets with two different types of activations. My understanding is that they can be both performed via standard NTK analysis (in the lazy training regime, there is nearly no difference between NTK and RSC based analysis).
3. In the ReLU case, it would be better to give a quantitative bound on $\lambda_{0,g}$  or $c$ in Theorem 5.1.

clarity,_quality,_novelty_and_reproducibility: This paper is a bit incremental given prior works on the NTK based convergence analysis.

summary_of_the_review: Overall, I am not satisfied with the initialization in Assumption 3, the authors may need to make the training of all layers be in the same level rather than using different initialization scaling for different layers. Besides, the contributions of this paper are not strong, especially given a large number of prior works studying the convergence of GD in the NTK regime.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper establishes theoretical guarantees about overparameterized DeepONets. In particular, they show that
strength_and_weaknesses: Strengths
-----------
* The paper's results are presented reasonably cleanly. This is especially nice given the heavy analytic nature of the work and greatly enhances the communication of the results.
* The paper's results all seem to be correct.
* The experimental results show that increase network width does improve the results.

Weaknesses
--------------
* The paper does not mention any of the more recent operator networks (e.g. fourier neural operators) which tend to show improved performance over DeepONets. This tends to make the analysis feel out of date.
* The overall results feel rather weak. In particular, it's not that surprising nor insightful that overparameterization would help DeepONets, especially with the wealth of literature on overparameterization.
* The proof techniques used feel like just reapplication of standard overparameterization tools. In particular, it treats the DeepONet as just a different architectural design (akin to MLPs or Resnets). However, the DeepONet primarily acts on functionals, which should necessitate something different. Is there any operator perspective to the analysis?
* The experimental results are really not necessary, don't add much, and don't really feel like a worthwhile inclusion. This is true even in comparison with general theory papers (which tend to have a weaker experimental setup), and so I would move them to the appendix and expand upon the theoretical contributions.

clarity,_quality,_novelty_and_reproducibility: Clarity
-------
As mentioned above, the paper is quite clear despite the analytic nature of the work. This makes it quite easy to read.

Quality
--------
Overall, the paper is reasonably well executed. Nothing seems to be wrong in the theoretical justification.

Novelty
---------
The paper does not feel very novel, as the techniques seem to be heavily borrowed from previous analyses and just applied out-of-the-box to DeepONets.

Reproducibility
-----------------
No code/architectural design is provided. This technically makes it hard to reproduce the experiments, but this shouldn't really matter as they are toy/sanity checking purposes only.
summary_of_the_review: Overall, I lean reject. In particular, my concerns are primarily that 1) DeepONets are old and better, more novel architectures have been introduced, 2) the insights given by the paper seem rather weak/intuitive, 3) the proof structure feels like a reapplication of standard techniques from the field to a different architecture.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studied the convergence of gradient descent algorithm in training the DeepONet in the overparameterized regime. Specifically,  the authors proved a spectral norm bound for the Hessian of DeepONets with smooth activation functions; this bound implies convergence of GD using the RSC property. For ReLU activated DeepONets the authors showed that the corresponding neural tangent kernel at initialization is positive definite which again implies convergence of GD. 
strength_and_weaknesses: Strengths: The analytical results on the training of operator networks seem to be new and extend the prior analysis for learning finite dimensional functions. The paper is easy to read and follow.

Weaknesses: I personally feel that this paper is merely a trivial extension of the early results of Du et al 2019 and Liu et al 2021 for finite dimensional functions to operator networks. The proof ideas and techniques based on the analysis of kernels and PL-condition are identical to those shown in the early literature. I do not see the essential difficulties in the extension except the technicalities arising from the new network architecture.

clarity,_quality,_novelty_and_reproducibility: The paper is easy to read and the main results are clearly stated. However, the results are incremental and not novel enough to be published in ICLR. 
summary_of_the_review: I am inclined to reject the paper due to the lack of sufficient novelty and significance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper establishes convergence guarantees on using gradient descent. for training Deep Operator Networks (DeepONets) in the over-parameterized regime. Two settings are studied: smooth activations (for which the convergence proofs are based on a restricted strong convexity/PL-type analysis), and ReLU activations (for which the proofs are based on NTK). 

Opinions were split about this paper. In terms of strengths, most reviewers (including this meta-reviewer) thought that the paper was generally well-written and that the results were sound. The fact that DeepONets are becoming popular in the numerical analysis/PDE literature is also indicative of possible impact.

However, several weaknesses were also highlighted. The literature on provable training of over-parameterized networks is quite mature by now. A majority of reviewers pointed out the new proofs were more of an application of a well-worn set of analysis techniques (RSC, NTK) for a slightly more complex neural network architecture, rather than providing any special new insights. This meta-reviewer concurs with this view. Moreover, one reviewer pointed out that the effect of over-parameterization was not particularly well-illuminated. The authors' response pointed out the dependence between the RSC constant and the width, but this meta-reviewer believes that more can/should be said here about the "benefits of over-parameterization", as the paper title suggests. Finally, some concerns were raised about the assumption of non-uniform scaling of different layers at initialization, which is non-standard in practice.

Overall, this is a good paper but possibly falls short of the (very) high bar for acceptance set at this year's ICLR. 

Recommendation: (weak) reject.

justification_for_why_not_higher_score: As expressed in the summary above, the paper has several positive aspects, but on the balance this meta-reviewer feels that they are outweighed by the negatives. 
justification_for_why_not_lower_score: Despite the fact that the paper follows a well-trodden path in analyzing over-parameterized networks, the contributions are clearly expressed and (to the best of our knowledge) new to the literature.
Decision: Reject