Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper aims to improve the well-established attention mechanism, and challenges the necessity of its softmax normalization through an integral perspective.
main_review: This paper aims to improve the well-established attention mechanism, and challenges the necessity of its softmax normalization through an integral perspective. The outcome are two “softmax-free” attention variants, connected respectively to Fourier feature transform (and thereby to several recent work sharing this insight, as noted by the paper) and Petroc-Galerkin projection. For the latter, non-trivial theoretical results on approximation error are presented. The proposed attention variants have practical benefits too: in several operator learning experiments, they outperform baselines in terms of accuracy and efficiency.

Overall I appreciate the paper’s technical significance, and really like it that the paper connects to previous works every now and then. Although the presentation is reasonably clear, I do have some confusion/suggestions: 
- Replacing the softmax with a linear-ish interaction hints significant efficiency gain. The proposed method achieves speedup and memory saving over baselines, and the paper would appeal to a broader audience if it promises so upfront. 
- I would appreciate it if the narrative does a better job connecting to what the implantation looks like. E.g., walking the readers through a quick algorithm would do. 
- Thm. 3.3 proceeds with a discrete space with length n, but isn’t bilinear form is over the Hilbert spaces?  I am not an expert in this direction and did not have the capacity to checkout out the proof, so I might be missing something pretty obvious. It would be great if the authors can clarify.
- Adding onto the above, Thm. 3.3 definitely stills has its value, since in practice the spaces are discretized anyway. But it is intriguing to see that the bound actually does not depend on n. I’m curious whether or not there is a more general version of 3.3, that could hold for the infinite-dimensional Hilbert-space case?

Overall the paper is solid and timely. I vote for acceptance.

Strengths:
- Interesting and timely contribution to “linearizing” attention in transformers.
- Well-designed experiments and solid results.
- The theoretical treatment of attention could inspire future research.

Weaknesses:
- The paper could do a better job in connecting to implementation and benefits in practice
- The key theoretical results (Thm. 3.3) needs further clarification.

Missing reference:
https://arxiv.org/abs/2103.02143
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, they propose a general operator learner based on a simple attention mechanism to remove the softmax operation in standard scaled dot product attention. which is regarded as a PDE-related operator. They also introduce a new scale-preserving layer normalization module to better support their operator learning. Combing with  Fourier Neural Operator, the proposed model obtains great performance on some benchmarks with superior memory and time efficiency.
main_review: This paper is written clearly and they regard the standard attention mechanism in Transformer as an operator learning problem with removing the softmax operations. Extensive experiments are conducted to confirm the effectiveness and efficiency of their proposed method.   It would be interesting if this proposed method could be integrated into pre-train language models for speeding up.  Just have a few questions:
1. Could the proposed method be applied to tasks like machine translation or GLUE?
2. Is the proposed method able to model the autoregressive attention such as in NMT or language modelling?
limitations_and_societal_impact: Yes, they adequately address the limitations.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper claims to apply two Transformer variants to a data-driven operator learning problem, where the mappings between infinite-dimensional spaces of functions are learned. However, this paper only discusses the uniform grid problem setting, which can be regarded as 0D/1D/2D time series forecasting problem. After applying an interpolation-based CNN network (CiNN) to get $n_c \times n_c \times d$ features, the authors use their proposed Fourier or Galerkin Transformer variants to encode the features and use another CiNN as a decoding regressor. Compared to the previous work FNO, the novelty of the paper comes from using Transformers to encode features instead of applying convolutions by FFT/iFFT. However, the use of Transformers makes the proposed method hard to conduct a zero-shot super-resolution prediction, which can be regarded as the merit of the neural PDE solvers.

main_review: 
Advantages:
1. The interpretations of non-softmax attention as Fourier expansion and Petrov–Galerkin-type projection seem to be interesting and insightful, while the proposed methods are not novel compared to other linear attention variants.
2. The proposed Fourier or Galerkin Transformer variants show some improvements over the standard softmax normalized scaled dot-product attention and the linear variant on operator learning problems.

Weaknesses:
1. While the application of random Fourier features on Transformers is discussed in the paper, neither Performer nor RFA is compared in the experiments.
2. Unlike the previous work FNO, the use of Transformers makes the proposed method hard to conduct zero-shot super-resolution prediction. In this case, whether the proposed method can be regarded as a neural PDE solver is questionable. And the paper doesn't present any experiments w.r.t this point (i.e., zero-shot super-resolution prediction).

Possible Typo:

line 105: $y \rightarrow y + \sigma(y + \mathrm{Attn}(y))$ are not consistnet with the original definition of Transformer. It could be $y \rightarrow y + \mathrm{Attn}(y) + \sigma(y + \mathrm{Attn}(y))$ if possible.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes two Transformer variants for data-driven operator learning problems of parametric partial differential equations: the Fourier-type and the Galerkin-type. This paper includes both theoretical evidence and empirical evidence to show that these two Transformer variants are both accurate and efficient for solving various parametric PDEs.
main_review: This paper first generalizes the Transformer architecture and then shows that vanilla Transformer, Fourier-type Transformer, Galerkin-type Transformer are special cases of this generalized Transformer architecture. This generalization is natural but creative. The author uses informative diagrams and equations to illustrate different kinds of Transformers talked about in this paper clearly.

The evaluation is on three well-known PDE benchmarks. This paper compares the error of all different combinations of Layer Normalization, so the experiment section is comprehensive and solid.

Although the variants seem to be inspired by the mathematical representations of some integrations, the proposed variants have only a few differences from the standard Transformer in the source code. I suggest that the author can list all these differences explicitly to make them clear. To my understanding, the differences only lie in three places:

1. Use mesh-weighted normalization without softmax
2. Where to add layer normalizations are different
3. (For Galerkin-type Transformer only) the ordering of matrix multiplication is different

The author does not explain why the speed/memory of ST, FT, LT, and GT are different in Table 1. To my understanding, LT and GT are faster than ST and FT because they are linear, but why is FT faster than ST, GT faster than LT? It would be better if there are more detailed explanations (maybe manually calculated FLOPs/memory).

To summarize, the motivation of this paper is interesting, but the real difference of the proposed methods with standard Transformer is limited and is not shown clearly. Shown by solid experimental results, the proposed variants are both accurate and efficient in solving PDEs, but the efficiency difference requires more explanations.
limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper studies Transformer variants for data-driven operator learning problems of parametric partial differential equations: the Fourier-type and the Galerkin-type. This paper includes both theoretical and empirical evidence to show that these two Transformer variants are accurate and efficient for solving various parametric PDEs.

The authors have nice discussions with all the reviewers during the rebuttal period and put great efforts into improving the paper quality. All the reviewers agree that the mapping between Transformer and PDE solvers is interesting and important. Therefore, I recommend acceptance. I hope the authors can revise the paper in the camera-ready version.
Decision: Accept (Poster)