Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper develops an analogy between Transformer networks and dynamical systems of multiple interacting particles. They follow the work of Lu et al. 2019 using Lie-Trotter splitting scheme to derive differential equations approximating terms corresponding to operations that compose the Transformer network internals, namely multi-head self-attention and feed-forward transformation. 

This paper develops the described technique beyond previous work by noting their analogous dynamical system motivates architecture choices that would require many fewer parameters than the associated Transformer. The authors investigate performance and memory gains by exploiting time-dependence in the dynamical transformation, achieving similar results to analogous weight-sharing between layers. 
main_review: This work builds on existing literature that relates familiar deep-learning structures to solutions of corresponding dynamical systems. The Lie-Trotter splitting scheme in particular establishes the connection to Transformer networks in literature. These previously published results are well-cited and presented in this paper. As well, this work develops these techniques beyond previously published methods by investigating various parameter sharing formulations that are naturally motivated by their dynamical system. They show improvements in complexity and demonstrate practically through experimental results. 
limitations_and_societal_impact: The authors clearly reiterate their technical contributions relative to existing Transformer networks. Primarily, they address the limitations of their complexity reductions, e.g. not affecting the quadratic cost in initial conditions. The authors do not discuss potential negative societal impact. As basic research it is difficult to suggest improvement here.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces TransEvolve, a new efficient transformer architecture that approximates the multi-headed self-attention and point-wise feed-forward transformations in the standard transformers.  Drawing from recent work by Lu et al., 2019 which established the connection between transformers and modeling the dynamics of a multi-particle dynamic systems, TransEvolve approximate multiple layers of the transformer with a temporal evolution scheme that bypasses costly dot product operations.  Experiments on encoder-decoder tasks show TransEvolve to be competitive with vanilla transformers while being slightly more efficient.  On encoder-only tasks, TransEvolve yields better predictive performance with half as many parameters.
main_review: ## Originality 
The connection between transformers and multi-particle dynamical systems was established in Lu et al., 2019.  This paper leverages this connection to inform the design of more efficient transformer architectures.  The main points of novelty in this work is the design temporal approximation schemes for the multi-headed attention and point-wise feedforward components of transformers architectures.

## Quality
The overall quality of the paper is sufficient.  I do think the experimental results can benefit from additional discussion to provide insights into why TransEvolve performs much better than existing methods for encoder-only tasks. I found L341-354 interesting and believe the paper would benefit greatly from additional discussion to help readers gain more intuition for how TransEvolve behaves.   

Questions:
- Did you try other values of L besides the two settings?  How does TransEvolve perform with even higher L > 6 or lower cumulative L < 6?
- Lu et al., 2019 introduced an architecture called Macaron Net motivated by a different splitting scheme.  Did you look into approximations for that architecture? It seems like  Macaron Net does outperform vanilla transformers on WMT14 by a healthy margin.


## Clarity
The paper is well written overall but I would have benefited from more intuitive explanation of the approximations presented in Sections 4 and 5.  I did appreciate the complexity summaries provided for the two approaches in the end of each section.  

_Clarification needed:_
- L343-346: "In the case of machine translation though, it is straightforward that having a lesser number of feed-forward parameters and/or fewer encoder blocks with deeper evolution deteriorate the performance" this seems to indicate that performance increases with parameter size.  But the next sentence states "In general, the performance of TransEvolve variations changes inversely with the total parameter size on translation tasks." Am I misinterpreting something here?

_Typos:_
- L349-350: "learn more useful representations of the input with a much fewer number of trainable parameters" -> learn more useful representations of the input with much fewer trainable parameters


## Significance
The contributions of the paper are important owing to the new perspective provided through the temporal approximations derived from taking a multi-partical dynamical systems view of multi-layer attention.  That said, I am unsure how useful TransEvolve will be in practice given the apparent need for careful tuning (i.e. there isn't a setting from randomFF-1, randomFF-2, fullFF-1, and fullFF-2 that worked well across all experiments) and the mixed experimental results (i.e. TransEvolve seems to only improve upon existing approaches in the encoder-only setting).  Additionally, as the authors noted, TransEvolve is still less efficient than linear attention models since it incurs quadratic cost to compute the initial states of the system.  


===========Post Author Response===========
After reading the author's response, I have decided to change my score to 7.  I strongly encourage the authors to include the intuitive explanation of why TransEvolve performs better on encoder only tasks and discussion of potential directions for future work in the paper.
limitations_and_societal_impact: The authors sufficiently discuss the computational limitations of TransEvolve compare to existing approaches.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce the TransEvolve approach based on the reformulation of transformer architectures in terms of the temporal evolution of many particle systems. Doing so, they can significantly reduce model parameters and computational cost, while achieving competitive or better performance than existing methods on a series of benchmark tasks.  
main_review: The use of ODE formalism to cast the transformer into a dynamic framework is an interesting concept, at it allows to formulate multi-head attention and feed forward operations as the evolution of time series, eliminating the need for several costly computations and reducing the total number of parameters.

The experiments are performed on a series of benchmarks covering a range of potential applications. In almost all tasks, at least one TransEvolve variant improves upon the performance of other models demonstrating the utility of the approach. However, here the text could benefit for a more in-depth discussion on the reasons for the performance difference observed between the random matrix and full feed forward variants.

Overall the paper is written in a clear manner, the architecture and development of the model is easy to follow and results are presented in a concise fashion.

TransEvolve offers a different view on transformer architectures. The theoretical connections to dynamical systems could be exploited to better analyze and potentially improve these kinds of models.
limitations_and_societal_impact: Potential limitations and societal impact have been sufficiently addressed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper avoid computing pairwise dot-product between n input vectors at each layer by computing a functional form at the  beginning and evolving it in a temporal (depth-wise) manner.
main_review: Major questions
1.Why this point of view relates to the ODE viewpoint, can you formally written in an ODE setting to show the two models have the similar 

2. I believe [1] is highly related to your paper, I think it would be essential to discuss and compare with it to show the dynamically improving the attention weights is ensential.

Minor points
The notation in the paper is somehow confusing, I would suggest the authors to organize the presentation in next version, there are too many notations doesn't defined in the section 4.

I'm not a NLP guy, thus I would like to let other reviewers to evaluate the experiments, in my limited experience, the experiments looks good

From my perspective, this is a very good direction to reduce the quadratic computation power for transformer , the drawback is the unclear motivation from the dynamic system viewpoint to the proposed structure.

[1] Ying C, Ke G, He D, et al. LazyFormer: Self Attention with Lazy Update[J]. arXiv preprint arXiv:2102.12702, 2021.
limitations_and_societal_impact: See main review
review_title: null

Meta-review: This paper was universally well-liked by the reviewers. The reviewers found the theoretical developments to be elegant, with significant practical applications yielding a convincing new model in the transformer family. Several reviewers commented that the exposition might be improved by including more intuitive explanations.
Decision: Accept (Spotlight)