Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes the energy transformer, a transformer architecture that uses a recurrent energy transformer block. The energy transformer block updates its input in accordance to minimizing two energy functions. Experiments on image reconstruction and graph anomaly detection demonstrate the effectiveness of the proposed method. 
strength_and_weaknesses: Strength: The proposed method is interesting in the sense that it reformulates transformer architecture in a way to combine Hopefield networks and attention networks, which are recurrently optimized using two carefully designed energy functions. The empirical results look promising. 

Weakness: I am concerned about the clarity of the paper. For example, in the explanation of equation (3), "The log-sum energy function (3) is minimal when for every patch in the image its queries are aligned with the keys of a small number of other patches connected by the attention map," It is not clear what does this mean in a mathematical sense. There is also no detailed algorithmic description about how the model is trained. For example, for the image reconstruction task, how does minimizing the energy functions correspond to reconstruction? It would be helpful if the authors include a formal algorithmic description of the proposed method. 

I am also concerned about some of the experiment settings. For example, for image reconstruction task, Figure 3 only shows some of the examples. However, this does not represent its general performance. One way to make this experiment more complete may be to include the MSE of the test datasets, the variance of the MSE, and perhaps reconstructed images corresponding to the best MSE and the worst MSE. 

clarity,_quality,_novelty_and_reproducibility: Although the proposed method seems novel, the methodology of the proposed transformer architecture is not clear enough to me. The experimental results are also not very satisfying, as detailed in the previous section. 
summary_of_the_review: Although the paper is interesting to read, I have two major concerns as detailed in the weakness section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new transformer architecture that replaces the sequence of transformer blocks with a single associative memory model. Specifically, the model is designed to minimize a specific energy function that represents the relations between tokens. The model is evaluated on image completion and graph anomaly detection, and experiments show encouraging results.
strength_and_weaknesses: Strength
- Despite the great performance achieved by the Transformer model, its architecture is designed empirically. This work proposes a Transformer architecture from a purely theoretical perspective.
- This work provides a deep insight into the relationship between Transforms and associative memory models. Based on the relationship, it designs a new energy function and a corresponding Transformer architecture that minimizes the energy function.

Weakness
- To show the effectiveness of ET, it is better to evaluate it on more mainstream tasks.
clarity,_quality,_novelty_and_reproducibility: good

summary_of_the_review: It is an interesting work toward understanding the Transformer models. To show its effectiveness, more experiments have to be conducted on more challenging tasks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 
The paper puts forward a novel sequence-to-sequence neural architecture named Energy Transformer (ET) that builds
on recently introduced generalized Hopfield networks and their connection to transformer layers. 
The architecture is composed by a single block that combines two subsystems,
defined by energy functions corresponding to two different types of generalized hopfield networks, 
preceded by a layer normalization operator. 
One subsystem is in charge of pooling to each position the information coming from the other
positions, while the other enforces the consistency of learned representations.
The system dynamics is given by a differential equation driven by the gradient
of the energy function with respect to the renormalized features. The total
energy is proven to decrease along the trajectory. 
For practical purposes, the dynamics is discretized and the block applied recurrently.
The architecture is validated qualitatively on an image occlusion task and quantitatively
on a node anomaly detection task.
strength_and_weaknesses: 
The manuscript is an enjoyable read and contains a good degree of novelty. 
The output of the network is given by the final outcome of an energy minimization process.
Although this perspective is not new (see Yang et al '22), it has not been quite explored yet
and I find the particular instantiation given in the paper quite inspiring. 
I think this could trigger additional theoretical and applicative results.
The resulting architecture is also quite simple and has a good degree of interpretability.
The paper is well organized, although some details given in the appendix could be moved to the main text.
The results on the node outlier detection tasks are very good, outperforming concurrent methods in
most cases.

Main comments:

- Why the authors didn't attempt to benchmark the performance on a standard image classification task? 
  This should be doable by pooling and putting an head on top of the final outputs, as commonly done in ViT.

- Related to previous questions, is not clear if the ET can be considered as a drop-in replacement to self-attention
  transformer blocks for arbitrary task, or if its usage should be confined to some specific settings.
  I would like the authors to comment on the limitations of the model.

- It is not clear how important the HN component is and how would perform the ET without it.

- Could the author motivate why the attention module excludes the current the considere position from the input. 
What happens if you remove the restriction from the sum?

- Did the authors explor stacking multiple ET modules in a feeforward fashion?

Minor comments:
- While it is clear in the appendix, I think also the main text should mention that the DE is discretized.

- eq. 2, ignored the \bar{x} dependence on x_i when computing the derivative. Should be fixed by inserting a factor 1 / (1 - 1/D) in the definition of L

- "is then minimized to train the whole network. Above, σ is the ratio of anomalous labels (lA = 1) to
the regular label." Maybe the other way around, that is regular labels / anomalous labels?

- The main text should mention that for the graph task a neighborhood softmax (similar to GAT) is used instead of standard attention

- The authors could consider numbering more equations for readibility.
clarity,_quality,_novelty_and_reproducibility: The manuscript is overall quite clear and contains novel and interesting results.
The experimental settings are fully detailed and the code will be publicly released.
summary_of_the_review: I think this is a good paper that deserves acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose a new transformer architecture called energy transformer (ET). ET is designed to minimize a special handcrafted energy function. This function contains two terms. The attention term aligns the keys and queries between neighbor token vectors and the hopefield network term matches the token vectors to some memory vectors. The ET layer is applied recurrently to imitate a continuous time differential equation that minized the designed energy terms. The authors try to demonstrate the ability of their model through image completion task and graph anomaly detection task.
strength_and_weaknesses: Strength:
In general, I like the idea of this paper:
1. This paper propose an specially designed energy-term that may be helpful for understanding the transformer structure.
2. It derives a new transformer structure based on this handcrafted energy-term
3. The authors show that this structure outperforms baselines in the graph anomaly detection task.

Possible concerns:
1. While the authors showcase experiments on image completion and graph anomaly detection task. I think these are not enough for me to draw the conclusion that this newly designed arctecture does have advantage over the previous one. The transformer structure is famous because it performs well in various task in computer vision(CV) and netural language processing(NLP). To really demonstrate the ability of a new structure, the authors need to show competitive results on those widely accepted tasks like transfer learning to ImageNet in CV or results on GLUE [1] benchmark in NLP. Currently, the authors only showcase two tasks that are well-suited for their model (and they only show quantative results in one of them), thus the general ability of this model is still unclear to me.

2. Some designs of the model are not fully justified experimentally. For example, unlike traditional transformer architecture which uses different weights for different layers, ET apply a single block recurrently and the MLPs in their hopfiled network also share weights . The authors may find theoretical explanation for this design according to their energy design. However, whether this helps or hurts the performance is unclear. Empirically speaking, recurrent model or model with shared weight may be hard to optimize due to the vanishing gradient problem. Thus, ablation study may be needed to justify the these design choices.

3. Questions for the graph anomaly detection task: How are the edges used by the model? Does the attention layer only compute attention over the nodes that have edges between them? Also, can the model be also used on directed graph? Or it is only limited to undirected graph?

4. For the image completion task, the authors assumes the image patches can be represented as fully masked and fully open area. But what if the masked area is not divisible by the patches? Then do you need to re-design the patch size (which may influence the full structure of the model)? This might not be a problem if one just treat image completion as a self-supervised task to learn representations and then applied these learned features to downstream tasks. But if the image completion task itself is the target and its performance is used to showcase the ability of the model. Then possible limitation may need to be considered.  

[1] GLUE: A MULTI-TASK BENCHMARK AND ANALYSIS PLATFORM FOR NATURAL LANGUAGE UNDERSTANDING

=====================================================================================================

**Post rebuttal:**

For me, although in the new responses, the authors say that they are not trying to replace the original transformer architecture, what the paper mainly about is proposing a new transformer architecture based on modification of the original one. Thus, a natural question to ask is how this new design performs comparing with the original one. Although the authors show that their design principle is based on the Hopfield Networks, whether this principle does benefit the transformer architecture design needs further justification. There can be many interesting principles in modifying the designs, but whether they do make senses need to be supported by comprehensive and convincing experimental results. A valid modification to the transformer architecture needs to demonstrate its advantage over the original one. The current results, however, are far from that. The current results are mainly based in the graph domain. Solid experimental results in areas that transformers are mainly used in, like CV or NLP, are missing. I can not agree with the authors that since they expertise in graph domain, they only pick experiments in this area and miss others. As I said, if the paper narrows its objective into proposing a new technique to achieve STOA results in graph domain, then the current results are fair enough. However, if they want to make a valid and general design principle for transformer architecture, they should compare their new architecture in the areas that transformer architecture are widely used. As a reviewer, I'm not convinced by the current results and thus I still lean to reject. I would recommend the authors to polish their work with more convincing results in different areas or limit their objective to a technique that specialized in graph domain.
clarity,_quality,_novelty_and_reproducibility: This paper is in general well written and introduce its idea clearly. 
summary_of_the_review: I think this paper proposes some interesting ideas like designing a energy function and deriving a transformer structure based on this energy function. However, I think the experiments of this paper may not be strong enough to demonstrate that the new designed structure does have advantage over the original one.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a transformer architecture called Energy Transformer that replaces the sequence of feedforward transformer blocks with a single large Associative Memory model. The sequence of the energy transformer layers is designed to minimize an energy function in charge of representing the relationships between the tokens. Experiments show that Energy Transformer achieves good performances on image completion and graph anomaly detection.  
strength_and_weaknesses: [Strengths]
+ The transformer is a prominent attention mechanism in machine learning, and it is interesting to design the Transformer mechanism from energy attention and the Hopfield network.

[Weaknesses]
1) The connection between Hopfield Networks, energy function, and Transformer is mentioned in Ramsauer et al. (2020). It is not clear the new insight of the proposed Energy Transformer.
2) The effect of designing the proposed ET block is not apparent; it is better to conduct an ablation study experiment to assess each component within such a designed block.
3) The detailed structure of the Energy Transformer is not explicitly depicted. A network architecture helps the reader to understand the ET implementation.
4) Since the Energy Transformer requires recurrently updating, it is better to show the cost of training and testing compared with the vanilla Transformer.
5) The experiments are weak. The quantitative result has merely the graph anomaly detection. However, the results of BWGNN (Hetero) and [A], which shows good performance, are not included. The on-par performance of the Energy Transformer does not make it impressive to the reader.

Related paper:
[A] Derek Lim, Xiuyu Li, Felix Hohne, and Ser-Nam Lim: New Benchmarks for Learning on Non-Homophilous Graphs. Workshop on Graph Learning Benchmarks, WWW 2021.

clarity,_quality,_novelty_and_reproducibility: Unclear descriptions reduce clarity and reproducibility.

Technical novelty should be clarified.
summary_of_the_review: The primary concern of this paper is its weak experiments, which do not sufficiently demonstrate the strength of the proposed Energy Transformer.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Inspired by the Hopfield Networks and Associative Memory model, the paper proposes a new transformer architecture called energy transformer (ET) .The sequence of transformer layers in ET is purposely designed to minimize a handcrafted energy function, which captures the relationships between the tokens. Empirical results are shown to demonstrate the effectiveness of the proposed framework. After the rebuttal, most of the reviewers found the presented energy transformer new, interesting and promising, and leant to accept the paper. However, Reviewer v3Fm and VTZR think that the current experiments are insufficient to demonstrate the effectiveness of the model because the paper is not a theoretical work and extensive experimental results should be necessary to justify the idea.   Reviewer v3Fm points out that the paper only tested the ET method on a graph-based problem and lacks empirical evidence on computer vision and NLP, which are more commonly used and important domains for transformers. After an internal discussion with reviewers, the AC thinks that the research will be of interest to the community, but the current paper is not yet ready for publication because of the insufficient and incomplete empirical evidence for verifying the method. The AC thinks that Reviewer v3Fm's concern is reasonable, and the proposed architecture needs to be tested thoroughly to demonstrate its performance before drawing safe and reliable conclusion. The AC recommends rejecting the paper at the current stage, and urges the authors to improve their paper by taking into account all the suggestions from the reviewers, and re-submit the paper in the next venue.  

justification_for_why_not_higher_score: The paper only tests the ET method on a graph-based problem and lacks empirical evidence on computer vision and NLP, which are more commonly used and important domains for transformers.
Decision: Reject