Energy Transformer
Authors: Benjamin Hoover
Yuchen Liang
Bao Pham
Rameswar Panda
Hendrik Strobelt
Duen Horng Chau
Mohammed J Zaki
Dmitry Krotov
Venue: Submitted to ICLR 2023 (2023)
Keywords: Transformers
Hopfield Networks
Graph Anomaly Detection

Abstract: Transformers have become the de facto  models of choice in machine learning, typically leading to impressive performance on many applications. At the same time, the architectural development in the transformer world is mostly driven by empirical findings, and  the theoretical understanding of their architectural building blocks is rather limited. In contrast, Dense Associative Memory models or Modern Hopfield Networks have a well-established theoretical foundation, but have not yet demonstrated truly impressive practical results. We propose a transformer architecture that replaces the sequence of feedforward transformer blocks with a single large Associative Memory model. Our novel architecture, called Energy Transformer (or ET for short), has many of the familiar architectural primitives that are often used in the current generation of transformers. However, it is not identical to the existing architectures. The sequence of transformer layers in ET is purposely designed to minimize a specifically engineered energy function, which is responsible for representing the relationships between the tokens. As a consequence of this computational principle, the attention in ET is different from the conventional attention mechanism.  In this work, we introduce the theoretical foundations of ET, explore it's empirical capabilities using the image completion task, and obtain strong quantitative results on the graph anomaly detection task.

Under review as a conference paper at ICLR 2023
ENERGY TRANSFORMER
Anonymous authors
Paper under double-blind review
ABSTRACT
Transformers have become the de facto models of choice in machine learning, typ- ically leading to impressive performance on many applications. At the same time, the architectural development in the transformer world is mostly driven by em- pirical findings, and the theoretical understanding of their architectural building blocks is rather limited. In contrast, Dense Associative Memory models Mod- ern Hopfield Networks have a well-established theoretical foundation, but have not yet demonstrated truly impressive practical results. We propose transformer architecture that replaces the sequence of feedforward transformer blocks with a single large Associative Memory model. Our novel architecture, called Energy Transformer (or ET for short), has many of the familiar architectural primitives that are often used in the current generation of transformers. However, not identical to the existing architectures. The sequence of transformer layers in ET is purposely designed to minimize a specifically engineered energy function, which is responsible for representing the relationships between the tokens. As con- sequence of this computational principle, the attention in ET is different from the conventional attention mechanism. this work, we introduce the theoretical foundations of ET, explore it's empirical capabilities using the image completion task, and obtain strong quantitative results on the graph anomaly detection task.
INTRODUCTION
Transformers have become pervasive models in various domains of machine learning, including language, vision, and audio processing. Every transformer block uses four fundamental operations: attention, feed-forward multi-layer perceptron (MLP), residual connection, and layer normalization. Different variations of transformers result from combining these four operations in various ways. For instance, Press et al.) (219)) propose to frontload additional attention operations and backload additional MLP layers in a sandwich-like instead interleaved way, Lu et al. (2019) prepend an MLP layer before the attention in each transformer block, So et al. (2019) use neural architecture search methods to evolve even more sophisticated transformer blocks, and so on. Various methods exist approximate the attention operation, multiple modifications of the norm operation, and con- nectivity of the block; see, for example, (Lin et al., 2021) for a taxonomy of different models. At present, however, the search for new transformer architectures is driven mostly by empirical evalu- ations, and the thcoretical principles behind this growing list of architectural variations is missing.
Additionally, the computational role of the four elements remains the subject of discussions. Origi- nally, Vaswani et al. (2017) emphasized attention as the most important part of the transformer block, arguing that the learnable long-range dependencies are more powerful than the local inductive biases of convolutional networks. On the other hand more recent investigations (Yu et al. 2021) argue that the entire transformer block is important. The "correct" way to combine the four basic operations inside the block remains unclear, as does an understanding of the core computational function of the entire block and each of its four elements.
In a seemingly unrelated line of work, Associative Memory models, also known as Hopfield Net- works (Hopfield) 1982 1984), have been gaining popularity the machine learning community thanks to theoretical advancements pertaining to their memory storage capacity and novel architec- tural modifications. Specifically, it has been shown that increasing the sharpness of the activation functions can lead to super-lineat (Krotov & Hopfield 2016) and even exponential (Demircigil et al.) 2017) memory storage capacity for these models, which is important for machine learning appli- cations. This new class of Hopfield Networks is called Dense Associative Memories or Modern

---

Under review as a conference paper at ICLR 2023
ET for graphs
g(x)
x
a
"g
ET
Energy
Transformer Block
ET
time
E
pos
ET for images
Figure 1: Overview of the Energy Transformer (ET). Instead sequence of conventional trans- former blocks, single recurrent ET block is used. The operation of this block is dictated by the global energy function. The token representations are updated according to continuous time differ- ential equation with the time-discretized update step = dt/T. On the image domain, images split into non-overlapping patches that are linearly encoded into tokens with added learnable posi- tional embeddings (POS). Some patches are randomly masked. These tokens are recurrently passed through ET, and each iteration reduces the energy of the set tokens. The token representations at near the fixed point are then decoded using the decoder network to obtain the reconstructed image. The network is trained by minimizing the mean squared error loss between the reconstructed image and the original image. On the graph domain, the same general pipeline is used. Each token represents a node, and each node has its own positional encoding. The token representations at or near the fixed point are used for the prediction of the anomaly status of each node.
Hopfield Networks, Ramssuer et al. (2000) additionally describe how the attention mechanism in transformers is closely related special model of this family with the softmax activation function.
There are high-level conceptual similarities between transformers and Dense Associative Memories, since both architectures designed for some form of denoising of the input. Transformers typically pre-trained on a masked-token task, e.g., in the domain of Natural Language Processing (NLP) certain tokens in the sentence masked and the model predicts the masked tokens. Dense Associative Memory models are designed for completing the incomplete patterns. For instance, pattern can be the concatenation of image and its label, and the model can be trained predict part of the input (the label), which is masked, given the query (the image). They can also be trained self- self-supervised way by predicting the occluded parts of the image, denoising the image.
There are also high-level differences between the two classes of models. Associative Memories are recurrent networks with a global energy function so that the network dynamics converges fixed point attractor state corresponding are typically not described as dynamical systems all. Rather, they are thought of as feed-forward networks built of the four computational elements discussed above. Even if one thinks about them as dynamical systems with tied weights, e.g., (Bai et al. 2019), there is no reason to expect that their dynamics converge to a fixed point attractor (see the discussion in (Lan et al. 20201).
local minimum of the energy function. Transformers
Additionally, a recent study Yangg et al. 2022) uses a form of Majorization Misionn--iiiiiiiiion algorithms optimization process.
Sun al. 2016) to interpret the forward path in the transformer block This interpretation requires imposing certain constraints on the operations inside the block, and at- tempting to find energy function that describes the constrained block. We take complementary
2

---

Under review as a conference paper at ICLR 2023
approach by using intuition developed in Associative Memory models to start with energy fune- tion that is perfectly suited for the problem of interest. The optimization process and the resulting transformer block in our approach is a consequence of this specifically chosen energy function.
Concretely, we use the recent theoretical advancements and architectural developments in Dense Associative Memories to design an energy function tailored to route the information between the tokens. The goal of this energy function is to represent the relationships between the semantic con- tents of tokens describing given data point (e.g., the relationships between the contents of the image patches in the vision domain, or relationships between the nodes' attributes in the graph do- main). The core mathematical idea of our approach that the sequence of these unusual transformer blocks, which we call the Energy Transformer (ET), minimizes this global energy function. Thus, the sequence of conventional transformer blocks is replaced with a single ET block, which iterates the token representations until they converge to a fixed point attractor state. In the image domain, this fixed point corresponds to the completed image with masked tokens replaced by plausible auto- completions of the occluded image patches. In the graph domain, the fixed point reveals the anomaly status of given node given that node's s neighbors; seelFigure I] The energy function in our ET block is designed with the goal to describe the relationships between the tokens. Examples of relationships in the image domain are: straight lines tend to continue through multiple patches, given a face with one eye being masked the network should impaint the missing eye, etc. In the graph domain, these are the relationships between the attributes and the anomaly status of the connected nodes. The optimization procedure during the forward path of ET uses continuous time differential equations, and describes gradient decent on the specifically chosen energy function.
The core mathematical principle of the ET block the existence of the global energy function dictates strong constraints on the possible operations, the order in which these operations are ex- ecuted in the forward path, and the symmetries of the weights in the network. As a corollary of this theoretical principle, the attention mechanism of ET is different from the attention mechanism commonly used in feed-forward transformers (Vaswani et al. 2017).
In the following section we introduce the global energy function for the ET block and explain the block's architecture. We then explore the inner workings of the ET network for image comple- tion and qualitatively assess the learned representations (the model is trained on ImageNet- Ik in general pipeline similar to Dosovitskiy et al. (202T)). Finally, we turn to the graph anomaly detec- tion task, which is conceptually similar to the image completion setting but has record of strong published benchmarks against which our approach can be quantitatively compared. We show that the ET network stands in line with or outperforms the latest benchmarks. Although we focus the computer vision and anomaly detection domains in this paper, we believe that the computational principles developed can be applied to other exciting domains (e.g., NLP, audio, and video) in which conventional transformers have shown promising results.
ENERGY TRANSFORMER BLOCK
We now introduce the theoretical framework of the ET network. For clarity of presentation, we use language associated with the image domain. For the graph domain, one should think about "image patches" as nodes on the graph.
The overall pipeline is similar to the Vision Transformer networks (ViTs) and is shown in Figure I An input image is split into non-overlapping patches. After passing these patches through the en- coder and adding the positional information, the semantie content of each patch and its position is encoded in the token TAA- In the following the indices i,j, k = 1...D are used to denote the token vector's 's elements, indices A, B,C L...N are used to enumerate the patches and their cor- responding tokens. It is helpful to think about each image patch as a physical particle, which has a complicated internal state described by a D-dimensional vector XA. This internal state describes the identity of the particle (representing the pixels of each patch), and the particle's 's positional em- bedding (the patch's 's location within the image). The ET block is described by a continuous time differential equation, which describes interactions between these particles. Initially, at = I the net- work is given a set containing two groups of particles corresponding to open and masked patches. The "open" particles know their identity and location in the image. The "masked" particles only know where in the image they are located, but are not provided the information about what image patch they represent. The goal of ET's non-linear dynamics is to allow the masked particles to find
3

---

Under review as a conference paper at ICLR 2023
x
A2
Position Embedding Correlations
Weights in HN
Addition
0.5
Patch Row
Multi-Head Energy Attention
Hopfield
Network
0.0
10
-0.5
Norm
x
Patch Column
Figure 2: Left: Inside the ET block. The input token x passes through sequence of operations and gets updated produce the output token x'. The operations inside the ET block are carefully engineered so that the entire network has global energy function, which decreases with time and bounded from below. In contrast conventional transformers, the ET-based analogs of the atten- tion module and the feed- wordffr MLP module are applied in parallel as opposed to consecutively. Center: The cosine similarity between the learned position embedding of each patch and every other patch. In each cell, the brightest patch indicates the cell of consideration. Right: 100 se- lected memories stored the HN memory matrix, visualized by the decoder as 16x16 RGB image patches. This visualization is unique to our model, as traditional Transformers cannot guarantee image representations in the learned weights.
an identity consistent with their locations and the identities of open particles. This dynamical evolu- tion is designed so that it minimizes global energy function, and is guaranteed to arrive at fixed point attractor state. The identities of the masked particles are considered to be revealed when the dynamical trajectory reaches the fixed point. Thus, the central question is: how can design the energy function that accurately captures the task that the Energy Transformer needs to solve?
The masked particles' search for identity is guided by pieces of information: identities of the open particles, and the general knowledge about what patches are in principle possible in the space of all possible images. These
pieces of information ET's energy function: the energy based attention and the Hopfield Network, respectively. for reasons that will become clear in the next sections. Below define each element of the ET block in the order they appear in Figure 2
described by two contributions to the
LAYER NORM
Each token is represented by vector RP, At the same time, most of the operations inside the ET block are defined using layer-normalized token representation
D
I
1
91
6, where
T
D
Tk
(1)
D
Σ
(2j
+e
km
The scalar and the vector are learnable parameters, is a small regularization constant. Impor- tantly, this operation can be viewed as activation function for the neurons and can be defined as partial derivative of the Lagrangian function
L Dr D
- + Σ
(z
x)
that 9:
0L Oz,
See Krotov & Hopfield (021): Tang & Kopp (2021) Krotov 2021 for a detailed discussion of this property.
4

---

Under review as a conference paper at ICLR 2023
MULTI-HEAN ENERGY ATTENTION
The first contribution to the ET's energy function is responsible for exchanging information between the particles (patches). Similarly the conventional attention mechanism, each token generates pair of queries and keys (ET does not have a separate value matrix; instead the value matrix is function of keys and queries). The goal of the energy based attention is to evolve the tokens in such a way that the keys of the open patches are aligned with the queries of the masked patches in the internal space of the attention operation. Below we use index a 1...Y to denote elements of this internal space, and index h = I...H to denote different heads of this operation. With these notations the energy-based attention operation is described by the following energy function:
Σ KohB Qanc) ))
ATT
1 B
ΣΣ
log Σ exp
$
(3)
B&C
where the queries and keys tensors are defined as
KahB
Σ Wik 9JB.
K RYxHxN
(4)
Qanc Σ W akj 9jc,
Q QER
and the tensors W E RYXHxD and WO R xX are learnable parameters.
From the computational perspective each patch generates two representations: query (given the position of the patch and its current content, where in the image should look for the prompts on how to evolve in time?), and key (given the current content of the patch and its position, what should be the contents of the patches that attend it?). The log-sum energy function (3) is minimal when for every patch in the image its queries are aligned with the keys of a small number of other patches connected by the attention map. Different heads (index h) contribute to the energy additively.
HOPFIELD NETWORK MODULE
The next step of the ET block, which we call the Hopfield Network (HN), is responsible for ensuring that the token representations are consistent with what one expects to see realistic images. The energy of this sub-block is defined as:
HHN Σ
Σ Enj 9jB
E R
(5)
where Euz is set of learnable weights (memories in the Hopfield Network), and r(-) is activation function. Depending on the choice the activation function this step can be viewed either as a classical continuous Hopfield Network (Hopfield 1984) if the activation function grows slowly (e.g., ReLU), or as a modern continuous Hopfield Network (Krotov & Hopfield [016] Ramsauer et al.) 2020 Krotov & Hopfield, 202T) if the activation function is sharply peaked around the memories (e.g. power or softmax). The HN sub-block is analogous to the feed-forward MLP step in the conventional transformer block but requires that the weights of the projection from the token space to the hidden neuron's 's space to be the same (transposed matrix) as the weights of the subsequent projection from the hidden space to the token space. Thus, the HN module here is an MLP with shared weights that is applied recurrently. The energy contribution of this block is low when the tokens representations are aligned with some rows the matrix ç, which represent memories.
DYNAMICS OF TOKEN UPDATES
The forward path of the ET network is described by the continuous time differential equation, which minimizes the sum of the two energies described above
dria
OE
where
E = ATT HN
(6)
di
OgiA"
Here Z;A is the token representation (input and output from the ET block), and 9:n is its layer- normalized version. The first energy is low when each patch's queries are aligned with the keys
5

---

Under review as a conference paper at ICLR 2023
of its neighbors. The second energy is low when each patch has content consistent with the gen- eral expectations about what an image patch should look like (memory slots of the matrix x). The dynamical system (6) finds trade-off between these two desirable properties of each token't 's repre- sentation. For numerical evaluations equation (6) is discretized time.
To demonstrate that the dynamical system
(6 minimizes the energy, consider the temporal derivative
dE
OE OgiA drjA OgiA
OE
OE
= Σ
M
S 0
dt
θzza dt
OgiA
Agj
J.A
The last inequality sign holds if the symmetric part of the matrix
OgLA グzzz
LL
M
(8)
dmzadRjA
positive semi-definite (for each value of index A). The Lagrangian 2 satisfies this condition.
RELATIONSHIP TO MODERN HOPFIELD NETWORKS AND CONVENTIONAL ATTENTION
One of the theoretical contributions of our work is the design of the energy attention mechanism and the corresponding energy function 3 Although heavily inspired by prior work on Modern Hopfield Networks, our approach is fundamentally different from it. Our energy function (3) may look somewhat similar to the energy function of a continuous Hopfield Network with the soltmax activation function. The main difference, however, is that in order to Modern Hopfield Networks recurrently (as opposed to applying their update rule only once) the keys must be constant parameters (called memories in the Hopfield language). In contrast, our energy attention network the keys are dynamical variables that evolve in time with the queries.
To emphasize this further, it is instructive to write explicitly the ET attention contribution to the It is given by (for clarity, assume only one head of attention):
update dynamics 66
DEATT
ΣΣ
W. Koc softmax C
Σ KgC Qia) + Qao softmax (3 E K,A (r)
OgiA
C#A
In both terms the softmax normalization is done over the token index of the keys, which is indicated by the subscript in the equation. The first term in this formula is the conventional attention mecha- nism VVaswani et al. 2017) with the value matrix equal to = (W9)TK- = E., W W& K.c. The second term is the brand new contribution that missing in the original attention mechanism. The presence of this second term is crucial to make sure that the dynamical system (6) minimizes the energy function if applied recurrently. This second term the main difference of our approach com- pared to the Modern Hopfield Networks. The same difference applies to the other recent proposals Yang et 2022).
Lastly, we want to emphasize that our ET block contains two different kinds of Hopfield Networks acting parallel, Figure 2] The first one is the energy attention module, which is inspired by, but not identical to, Modern Hopfield Networks. The second one is the "Hopfield Network" module, which can be either a classical or modern Hopfield Network. These two should not confused.
QUALITATIVE INSPECTION OF THE ET FRAMEWORK ON IMAGENET
We have trained the ET network on the masked image completion task using ImageNet 1k dataset (Deng et al. 2009). Each image was broken into non-overlapping patches of 16x 16 RGB pixels, which were projected with a single affine encoder into the token space. Half these tokens were "masked", e.g., by replacing them with a learnable MASK token. A distinct learnable position encoding vector was added to each token. Our ET block then processes all tokens recurrently for T steps. The token representations after T steps are passed simple linear decoder (consisting of a layer norm and an affine transformation). The loss function the standard MSE loss on the occluded patches. See more details on the implementation and the hyperparameters in Appendix A
Examples of occluded/recoossrrctte images (unseen during training) are shown in Figure 3] In general, model learns perform the task very well, capturing the texture in dog fur (col and

---

Under review as a conference paper at ICLR 2023
Figure 3: Reconstruction examples of our Energy Transformer using images from the ImageNet- Ik validation set. Top row: input images where 50% of the patches masked with the learned MASK token. Middle row: output reconstructions after 12 time steps. Bottom row: original images.
understanding meaningful boundaries of objects. However, we observe that our single ET block struggles to understand some global structure, c.g., failing to capture both eyes of the white dog (col 4) and completing irregular brick patterns in the name of extending the un-occluded borders (last col). We additionally inspect the positional encoding vectors associated with every token, Figure 2] where the model learns locality structure in the image plane that is very similar to the original VIT (Dosovitskiy et al., 202T). The position embedding of each image patch has learned high similarity values to other patches in the same row and column, with similarity values higher for neighboring tokens than distant tokens.
Our network is unique compared to standard ViTs in that the iterative dynamics only move tokens around in the same space from which the final fixed point representation can be decoded back into the image plane. This functionality makes it possible to visualize essentially any token representation, weight, or gradient of the energy directly in the image plane. This feature highly desirable from the perspective of interpretability, since it makes it possible to track the updates performed by the network directly in the image plane as the computation unfolds time. In Figure 2hi functionality is used for inspecting the learned weights of the HN module directly in the image plane. According to our theory, these weights should represent basis vectors in the space of all possible image patches. These learned representations look qualitatively similar the representations typically found in networks trained on image datasets, e.g., Zeiler & Fergus (2014).
We additionally visualize the gradients of the energy function (which equal to the token updates, see Eeeuatino 6) of both ATTN block and the HN block, See[Figure 4 Early in time, almost all signal the masked tokens comes from the ATTN block, which routes information from the open patches to the masked ones; no meaningful signal comes from the HN block to the masked patch dynamics. Later in time we observe a different phenomenon: almost all signal to masked tokens comes from the HN module while ATTN contributes blurry and uninformative signal. Thus, the attention layer crucial early in the network dynamics, feeding signal masked patches from the visible patches, whereas the HN is crucial later in the dynamics as the model approaches the final reconstruction, sharpening the masked patches. All the qualitative findings presented this section are accord with the core computational strategy of the ET block it was designed theoretically in secetion 2
GRAPH ANOMALY DETECTION
Having built the theoretical foundation of the ET network and gained an intuition about its inner workings through visualizations, we turn to quantitatively evaluating its performance on the graph anomaly detection problem, task with plenty of strong and recently published baselines. Anomalies are outliers that significantly deviate in their properties from the majority of the samples. Detecting anomalies on graphs has broad applications in cybersecurity (Hong et al. [204 Pan et al., 2019). fraud detection (Huang et al., 2018 Pourhabibi et al., 2020). and social networks (Chaker et al. three types of graph anomalies: node anomaly, edge anomaly, and
2017). Generally, there subgraph anomaly. In this work, we focus on node anomaly detection in attributed graphs. This task is perfectly suited for the ET network, since each node's attributes can be encoded in the latent
7

---

Under review as a conference paper at ICLR 2023
Early time
Late time
HN Update
Update
Tokens [time s)
Tokens time s+1)
Tokens (time 11
Tokens Itime (+1)
Time
ATTN Update
ATTN Update
Figure 4: Token representations and gradients are visualized using the decoder at different times during the dynamics. The Energy Attention (ATTN) block contributes general structure informa- tion to the masked patches at earlier time steps, whereas the Hopfield Network (HN) significantly sharpens the quality of the masked patches later time steps.
space and treated as a token (with added learnable positional embeddings). The network iterates these representations in time, and the outputs can be used for the node anomaly classification task.
Graph Convolutional Networks (GCN) (Kipf Welling 2016) have been widely used for this task due to their capability of learning high level representations of graph structures and node attributes (Ding et al. 2019 Peng et al., 2020). However, vanilla GCNs suffer from the over-smoothing problem (Wu et al. 2019). In each layer of the forward pass, the outlier node aggregates information from its neighbors. This averaging makes the features of anomalies less distinguishable from the features of benign nodes. Our approach does not suffer from this problem, since the routing of the information between the nodes is done through the energy based attention, which uses different aggregation procedure depending on whether or not the node is anomalous.
In order to turn the anomaly detection task on graphs into the ET framework, consider an undirected graph with N nodes. Every node has vector of attributes y R where is the number of node's ' features. Additionallyy every node has a binary label la, indicating whether the node is benign or not. We focus on node anomaly and assume that all edges are trusted. The task is predict the label of the node given the graph structure and the node's 's features. Since there are far more benign nodes in the graph than anomalous ones, anomaly detection can be regarded as an imbalanced node classification task.
First, the feature vectors for every node are converted to a token representation using linear em- bedding E and adding learnalee positional embedding Aa
where the superscript t indicates the time of the update of the ET dynamics. This token rep- resentation is iterated through the ET block for T iterations. When the retrieval dynamics becomes stable, we have the final representation for each node x (or more precisely g gt=T, since the out- puts are additionally passed through a layer norm operation after the final ET update). This output concatenated with the initial (layer normalized) token form the final output of the network
final g
g.1
(10)
Following Tang al. 2022)) the node representation finall is fed into MLP with the sigmoid activation function to compute the anomaly probabilities PA. The weighted cross entropy
Loss Σ [o loggAA))
LA) log(1 pa)
(11)
used to train the whole network. Above, a is the ratio of the regular labels (lA = 0) to anomalous labels (IA 1).
4.1 EXPERIMENTAL EVALUATION
Four datasets are used for the graph anomaly detection experiments. YelpChi dataset (Rayana & Akoglu 2015) aims opinion spam detection in Yelp reviews. Amazon dataset is used to detect
8

---

Under review as a conference paper at ICLR 2023
anomalous users under the Musical Instrument Category amazon.com (McAuley & Leskovee 2013). T-Finance and T-Social datasets (Tang et al, 2022) are used for anomalous account detection the transactions and social networks, respectively. For these four datasets, the graph is treated as homogeneous graph (i.e. all the edges are of the same type), and feature vector is associated with each node. The task is predict the label (anomaly status) of the nodes. For each dataset, either 1% or 40% of the nodes are used for training, and the remaining 99% or 60% are split validation and testing, see Appendix B for details.
into
We compare with state-of-the-art approaches for graph anomaly detection, which include Graph- Consis (Liu al. 2020), CAREGNN (Dou et al., 2020). PC-GNN (Liu al., 202T) and BWGNN Tang et al. (002)). Additionally. multi-layer perceptrons (MLP) and Graph Transformer (GT) Dwwieedi & Bresson 2020) are included in the baselines for completeness. Following previous work, macro-F1 score (unweighted mean of F1 score) and the Area Under the Curve (AUC) are used as the evaluation metrics on the test datasets Davis & Goadrich (2006). See Bee/Apeendix B for more details training protocols and the hyperparameters choices. The results are reported in Table
Our ET network demonstrates very strong results across all the datasets.
Table 1: Performance of all the methods on Yelp, Amazon, T-Finance, and T-Social datasets with different training ratios. Following Tang et al. (2022), mean and standard deviation 5 runs with different train/dev/test split are reported for our method and the baselines (standard deviations are only included if they are available in the prior work). Best results are bold. Our model is state of the art or near state of the art on every category.
Datasets
Split
GraphConsis CAREGNN
PC-GNN
BWGNN
MLP
GT
ET (Ours)
1%
56.8
61.1
59.8
61.1
53.9
61.740.4
63.0.a.
Yelp
40%
58.7 58.7
63.3
63.0+23
71.02a.9
57.5+0.8
68.7
71.5
1%
68.5
68.7+1.6
79.845.6
90.9 +0.7
74.6+1.2
88.640.5
89.3
Amazon
Macro-F1
40%
75.1
86.3
89.510.7
92.210.4
79.1
91.710.
92.8103
1%
71.7
73.3
62.0
84.8
61.0
81.5
85.1
T-Finance
40%
73.4
77.3
63.1
86.8
70.5
83.6
88 88.2+1.0
1%
52.4
55.8
51.1
75.9
50.0
64.3
79.1
T-Social
40%
56.5
56.2
52.1
83.9
50.3
68.2
83.5
1%
66.4+3.4
75.0+3.8
75.4=0.9
72.0
59.8+0.4
72.5+a.6
73.2+0.8
Yelp
40%
69.8 89.8
12.9
79.8+0.1
84.0ga.9
66.5+1.
81.940.5
84.940.
1%
74.1
88.0 88.643.5
90.4+2..0
89.4...3
83.6
89.0
91.911.
Amazon
40%
87.4
90.5+1.6
95.8+u.1
98.0+a.4
89.8
95.4+o.6
9
AUC
1%
90.2
90.5
90.7
91.1
82.9
90.0
92.8
T-Finance
40%
91.4
91.2
94.3
87.1
88.2
95.013.0
1%
65.2
71.2
59.8
88.0
56.3
81.4
91.910.6
T-Social
40%
71.2
71.8
68.4
95.2
56.9
82.5
93.9
DISCUSSION AND CONCLUSIONS
A lot of recent research has been dedicated to understanding the striking analogy between Hopfield Networks and the attention mechanism in transformers. At high level, the main message of our work is that the entire transformer block (including feed-forward MLP, layer normalization, and residual connections) can viewed as a single large Hopfield Network, not just attention alone. At recent advances in the field of Hopfield Networks design a novel energy
deeper level, function that is tailored for dynamical information routing between the tokens and representation of large number of relationships between those tokens. When used in the encoder-decoder setting. an appealing feature of our network is that any state, weight, or state update can be mapped di- rectly into the data domain. This provides the possibility inspect the inner workings of the whole network, contributing to its interpretability. The attention mechanism in our network contains an important extra term compared conventional attention. We have tested the network on the image completion task (qualitatively) and node anomaly detection on graphs (quantitatively). The qualitative investigation reveals the perfect alignment between the theoretical design principles of network and its empirical computation. The quantitative evaluation demonstrates strong results, which stand in line or exceed the methods recently developed specifically for this task. Although we have only tested ET on two tasks, intentionally picked two entirely different data domains (im- ages and graphs). We believe that the proposed network will be useful for other tasks and domains and deserves comprehensive investigation in line with other popular variants of transformers.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
Section does not count towards page limit
In the experiments presented in this paper we have taken several steps to ensure reproducibility of our results. Namely, all the training protocols and implementation details including the hyperparameter selection are described in Appendix A Appendix B, and Appendix F The model and training code is released anonymouslyl | The code for image reconstruction is written in JAX Bradbury et al. (2018) with a single entry script launch the training process, with defaults set the configuration that produced the models used in this paper. The training script sets a seed that can recreate the exact same training setup augmentation provided default arguments, No additional training data was used beyond the training set (ImageNet-IK 2012), which is publicly available. The code for Graph Anomaly Detection written in PyTorch. Given the nature of the anomaly detection problem (random splits into training, validation, testing sets) all our results on graphs are reported with mean and standard deviations, describing typical variability in the performance.
had, ensuring the exact same weight initialization and random data
REFERENCES
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
Muhammet Balcilar, Guillaume Renton, Pierre Héroux, Benoit Gaüzère, Sébastien Adam, and Paul Honeine. Bridging the gap between spectral and spatial domains in graph neural networks. CoRR, abs/2003.11702, 2020. URL bttpttss////a org abs 220.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Minn Minne,, and Qiao Zhang. JAX: Composable transformations of Python+NumPPy programs, 2018.
Rima Chaker, Zaher Al Aghbari, and Imran N Junejo. Social network model for crowd anomaly detection and localization. Pattern Recognition, 61:266-281, 2017.
Jesse Davis and Mark Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on Machine learning, pp. 233 233-240, 2006.
Mete Demircigil, Judith Heusel, Matthias Löwe, Sven Upgang, and Franck Vermet. On a model of associative memory with huge storage capacity. Journal of Statistical Physics, 168(2)288-299,, May 2017, ISSN 1572-9613, doi: 10.10077/ 10. 10.1000/$5 1111000/11995---0 017 187-6
URL /xxx doi. org/
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Kaize Ding, Jundong Li, Rohit Bhanushali, and Huan Liu. Deep anomaly detection on attributed networks. In Proceedings of the 2019 SIAM International Conference Data Mining, pp. 594- 602. SIAM, 2019.
Kaize Ding, Jundong Li, Nitin Agarwal, and Huan Liu. Inductive anomaly detection attributed networks. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 1288- 1294, 2021.
Miguel Domingue, Rohan Dhamdhere, Naga Durga Harish Kanamarlapudi, Sunand Raghupathi, and Raymond Ptucha. Evolution of graph classifiers. In 2019 JEEE Western New York Image and Signal Processing Workshop (WNYISPW), pp. 1-5, 2019. doi: 1191/NNYIPW.2001 9,8933 10.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https:
https:://ancnynouss 4open http:///ww.pppoooo.../eee/////nn
10

---

Under review as a conference paper at ICLR 2023
Yingtong Dou, Zhiwei Liu, Li Sun, Yutong Deng, Hao Peng, and Philip S Yu. Enhancing graph neural network- based fraud detectors against camouflaged fraudsters. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 315-324, 2020.
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks graphs. arXiv preprint arXiv:2012.09699 2020.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Jonathan Godwin Thomas Keck*, Peter Battaglia, Victor Bapst, Thomas Kipf, Yujia Li, Kimberly Stachenfeld, Petar Veličković, and Alvaro Sanchez-Gonzalez. Jraph: A library for graph neural networks in jax., 2020. URL
wttpc//wo/cccieeeeee deepmind/ jraph
com/
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL
/ttt:://githbt com www.gooooooooocooooooooooooooo.o e fax
Junho Hong. Chen-Ching Liu, and Manimaran Govindarasu. Integrated anomaly detection for cyber security of the substations. IEEE Transactions on Smart Grid, 5(4):1643-1653, 2014.
John Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 7955)22554-2558 1982.
John J Hopfield. Neurons with graded response have collective computational properties like those of two-state neurons. Proceedings of the national academy of sciences, 81(10):3088-33 -392,, 1984.
Dongxu Huang, Dejun Mu, Libin Yang, and Xiaoyan Cai. Codetect: Financial fraud detection with anomaly feature detection. IEEE Access, 6:19161-19174, 2018.
Thomas N Kipf and Max Welling. Sumi-super classification with graph convolutional net- 2016.
works. arXiv preprint
Dmitry Krotov. Hierarchical associative memory. arXiv preprint arXiv: 20X.v:210..06446, 2021.
Dmitry Krotov and John J. Hopfield. Dense associative memory for pattern recogni- tion. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. Baae339c4d8995 62ee1 Pappeee
URL httpps://proceeiiing.s. neuripss cc/paperr2016//1ee
Dmitry Krotov and John J. Hopfield. Large associative memory problem in neurobiology and machine learning. In International Conference on Learning Representations, 2021. URL
https::/opperevieee nnt. wnt.t/ooouumll=-ly__DD00
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations, In International Conference on Learning Representations, 2020.
/forum?
iid=HleA7AEtVS
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu. A survey of transformers. arXiv preprint arXiv:2 106.::2206..45.... 2021.
Yang Liu, Xiang Ao, Zidi Qin, Jianfeng Chi, Jinghua Feng. Hao Yang, and Qing He. Pick and choose: gan-based imbalanced learning approach for fraud detection. In Proceedings of the Web Conference 2021, pp. 3168-3177, 2021.
Zhiwei Liu, Yingtong Dou, Philip S Yu, Yutong Deng, and Hao Peng. Alleviating the inconsis- tency problem of applying graph neural network to fraud detection. In Proceedings of the 43rd international ACM SIGIR conference research and development in information retrieval, pp. 1569-1572, 2020.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, abs/1711.05100 2017. URL http:::p/aww...o/////aaaaa

---

Under review as a conference paper at ICLR 2023
Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie- Yan Liu. Understanding and improving transformer from a multi-partion dynamic system point of view. arXiv preprint arXiv: 027602, 2019.
Julian John McAuley and Jure Leskovec. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. In Proceedings of the 22nd international conference World Wide Web, pp. 897-908, 2013.
Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. CoRR, abs/2007.0866. 2020. hRtps:ts:://aar
.rgg abs 2007 08663
Dai Quoe Nguyen, Tu Dinh Nguyen, and Dinh Phung. Universal graph transformer self-attention networks. In Companion Proceedings of the Web Conference 2022, WWW '22, pp. 193-196, New York, NY, USA, 2022. Association for Computing Machinery, ISBN 978 1450391300 doi:
00.1145234855333522258. |RL pettps://coi.rrgget 1140/ 348 3524258
Kaikai Pan, Peter Palensky, and Peyman Mohajerin Esfahani. From static to dynamic anomaly detection with application to power system cyber security. IEEE Transactions on Power Systems, 1596), 2019.
Zhen Peng, Minnan Luo, Jundong Li, Luguo Xue, and Qinghua Zheng. A deep multi-view frame- work for anomaly detection on attributed networks. IEEE Transactions on Knowledge and Data Engineering, 2020.
Tahereh Pourhabibi, Kok-Leong Ong, Booi H Kam, and Yee Ling Boo. Fraud detection: A system- atic literature review of graph-base based anomaly detection approaches. Decision Support Systems, 133: 13303, 2020.
Ofir Press, Noah A Smith, and Omer Levy. Improving transformer models by reordering their sublayers. arXiv preprint arXiv:1911.0388 2019.
Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlović, Geir Kjetil Sandve, et al. Hopfield networks all you need. arXiv preprint urXiv:2008.02211 02217, 2020.
Shebuti Rayana and Leman Akoglu. Collective opinion spam detection: Bridging review networks and metadata. Proceedings of the 21th acm sigkdd international conference on knowledge discovery and data mining, pp. 985-988-9999 994-994,220 2015.
David So, Quoe Le, and Chen Liang. The evolved transformer. In International Conference on Machine Learming, pp. 5877-5886. PMLR, 2019.
Ying Sun, Prabhu Babu, and Daniel Palomar. Majorization-mininizzaooo algorithms in signal processing, communications, and machine learning. IEEE Transactions on Signal Processing, 65 (3):794-816, 2016.
Fei Tang and Michael Kopp. A remark paper of krotov and hopfield [arxiv: 2008.06996.. arXiv preprint arXiv:2105.1 1002100..10,,,,, 2021.
Jianheng Tang, Jiajin Li, Ziqi Gao, and Jia Li. Rethinking graph neural networks for anomaly detection. arXiv preprint arXiv:2205. 15508, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 1710 10903 2017.
Ross Wightman. Pytorch image models.
https:// com/ wtt.:hhhaa.ccmmm//ghhtaaaaa////
pytorch- mage models 2019.
12

---

Under review as a conference paper at ICLR 2023
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International conference on machine learning. pp- 6861-6871. PMLR, 2019.
Yongyi Yang, Zengfeng Huang, and David Wipf. Transformers from an optimization perspective. arXiv preprint arXiv:2205. 10.0005,38991 13891, 2022.
Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng. Guolin Ke, Di He. Yanming Shen, and Tie-Yar Liu. Do transformers really perform bad for graph representation? CoRR, Abss2100.052533 2021. [RRhtts::://axx
abss/216. 05234
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. arXiv preprint arXiv:2111. 11418, 2021.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. $18-833. Springer, 2014.
Zhen Zhang, Jiajun Bu, Martin Ester, Jianfeng Zhang, Zhao Li, Chengwei Yao, Dai Huifen, Zhi Yu, and Can Wang. Hierarchical multi-view graph pooling with structure learning. JEEE Transactions on Knowledge and Data Engineering, pp- 1-1, 2021. doi: 20.1109/TKDE:2011 2399609
Qi Zhao and Yusu Wang. Learning metrics for persistence-based summaries and applications for graph classification. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bus E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran As- sociates, Inc., 2019. URL http::ttps//s....... http://tttttt/////////ttae htttttpps///rrooeeeeiniie www.nng....eiiiiee.eeeeee... iss httppps...pppeee http:/papr/2219//// 9//1//11e///
12780ea686a (ri.e638877722220004444444-----3
Tong Zhao, Chuchen Deng, Kaifeng Yu, Tianwen Jiang, Daheng Wang, and Meng Jiang. Error- bounded graph anomaly loss for gnns. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 1873-1882, 2020.
A DETAILS OF TRAINING ON IMAGENET
We trained the ET network on masked-image completion task on the ImageNet-Ik (INIK) dataset. We treat all images in INIK as images of shape 224 x 224 that are normalized according to standard INIK practices (mean variance 1 on the channel dimension) and use data augmentations provided by the popular timm library (Wightman) 2019) (See Table 2}. Following the conventional ViT pipeline (Dosovitskiy et al., 2021), we split these images into non-overlapping patches of 16x16 RGB pixels which are then projected with single affine encoder into the token dimension D) for total of 196 encoded tokens per image. We proceed to randomly and uniformly assign 100 of these tokens as "occluded" which are the only tokens considered by the loss function. "Oecluded" tokens are designated as follows: of the 100 tokens, 90 tokens are replaced with a learnable MASK token of dimension D and 10 we leave untouched (which we find important for the HN to learn meaningful patch representations). To all tokens we then add a distinct learnable position bias.
These tokens are then passed to Energy Transformer block which recur for T steps (the "depth" of the model in conventional Transformers). At each step, the feedback signal (the sum of the energy gradients from our attention block and HN block) is subtracted from original token representation with a scalar step size in our experiments. The token representations after T' steps are passed (consisting of a layer norm and an affine transformation) to project our representations back into the image plane. We then use the standard MSE Loss between the original pixels and reconstructed pixels for only the 100 occluded patches. We allow self attention as in the following formula for the energy of multiheaded attention.
dt which we treat as non-learnable hyperparameter
simple linear decoder
KaNB Qurc) ))
ATT
Σ
log
Σ exp 8 Σ
(12)
3
We give details of our architectural choices in Table 2 In the main paper we present our Energy Transformer with configuration similar the standard base Transformer configuration (e.g., token
13

---

Under review as a conference paper at ICLR 2023
1 dimension 768, 12 heads each with Y 64, softmax's 's B v8* vg+...), ...), with several considerations learned from the qualitative image evaluations:
The T (step size) of implicitly used in the traditional transformer noticeably degrades our ability smoothly descend the energy function. We find that a step size of 0.1 provides smoother descent down the energy function and benefits the image reconstruction quality.
We observe that our MSE loss must include some subset of un-occluded patches in order for the HN to learn meaningful filters.
Values of 3 in the energy attention that are too high prevent our model from training. This is possibly due to vanishing gradients in our attention operation from softmax operation that is too spiky.
Without gradient clipping, our model fails to train at the learning rates we tried higher than le-4. We observe that gradient clipping not only helps our model train faster at the trainable learning rates, it also allows to train at higher learning rates.
Our architecture and experiments for the image reconstruction task were written in JAX (Bradbury et al. (0188) using Flax (Heek et al. (020). This engineering choice means that our architecture definitions are quite lightweight, as we can define the desired energy function of the ET and use JAX's autograd to automatically calculate the desired update. All training code and software will be released upon the paper's 's acceptance.
Table 2: Hyperparameter, architecture, and data augmentation choices for ET-base during ImageNet- Ik masked training experiments. Data augmentations are listed as parameters passed to the equiva- lent timm dataloader functionality.
Training
Architecture
Data Augmentation
batch size
768
token dim
768
random erase
None
epochs
100
num mua_heads
12
horizontal_fi _fii
0.5
Ir
5e-4
head_dim
64
vertical-flig
0
warmup-epochs
2
B
1/8
color_jitter
0.4
start & end Ir
5e-7
trainbetaa
No
scale
(0.08,
b1, 62 (ADAM)
0.9, 0.99
step size a
0.1
ratio
(3/4, 4/3)
weight decay grad clipping
0.05
depth
12
auto autgent
None
1.
hidden dim HN
3072
bias HN
None
bias in ATT
None
bias in LNORM
Yes
A.1 EXPLORING THE HOPFIELD MEMORIES
A distinctive aspect of our network is that any variable that has a vector index of tokens can be mapped into the data domain by applying the decoder network this variable. This makes it possible to inspect all the weights in the model. For instance, the concept of "memories" is crucial to understanding how Hopfield networks function. The memories within the HN module represent the building blocks of all possible image patches in our data domain, where an encoded image patch a superposition of subset of memories. The complete set of memory vectors from the HN module is shown in Figure 5) The same analysis can be applied to the weights of the ET-attention module. In Figure 6 we show all the weights from this module mapped into the image plane.
A.2 BIAS CORRELATIONS
The relationships between our position embeddings exhibit similar behavior to the position correla- tions of the original ViT in that they are highly susceptible to choices of the hyperparameters (Figure 10 of Dosoviiskiy al. 2021)). In particular, rameter that serves as the inverse temperature of the attention operation (see Equuaiio 3). The lower the temperature (i.c.. the higher the value of (8), the spikier the softmax distribution. By using lower 3, we encourage the attention energy to distribute its positional embeddings across a wider range of patches in the model.
consider the effect of weight decay and the 3 pa-
14

---

Under review as a conference paper at ICLR 2023
Figure 5: Visualizing a randomly selected 3025 patch memories of the 3072 learned by weight matrix in the Hopfield Network module (HN) of our model. These memories are vectors of the same dimensions D as the patch tokens, stored as rows in the weight matrix &. Each image patch is visualized using the model's ''s trained decoder.
A.3 OBSERVING ENERGY DYNAMICS
We include as part of our supplemental submission a video showing the dynamics of one of our trained models through time together with the corresponding energy at every step. From the video, clear that the image progressively improves quality as the energy decreases up to the point when the token representations are passed to the decoder. At the same time, we found it challenging to find the time constants and number of training steps so that the energy of each image reaches the energy minimum (fixed point) when the loss is computed. For instance, for the image shown in the video, its quality actually starts to degrade when the dynamics is allowed to run for longer than what was used training (while the energy is still decreasing). Additionally, when training models at greater depth the gradients can vanish since many recurrent applications of the ET block necessary. We hope comprehensively investigate these questions/imitationss in future work. We include screenshot of the video in Figure 8 and encourage readers to watch the full video for the dynamics.
15

---

Under review as a conference paper at ICLR 2023
W keys
W queries
Figure 6: Visualizing the token dimension of the "key" and "query" matrices of the attention as image patches. Each head is represented cell on the 4 x 3 grid above. We use the trained decoder of our model to visualize each weight.
WD=0.001: =0.02
WD=0.001: 8=0.12
WD=0.05; 8=0.125
1.0
0.5
-055
-1.0
The cosine similarity between position biases of patches when the ET-bbae model is trained under different hyperparameter choices for 3 (inverse temperature of attention energy) and weight decay. Our ET sees trend where smoother correlations are observed with smaller ( and weight decay.
Figure 7:
16

---

Under review as a conference paper at ICLR 2023
deweniicc fur
Catond Le
Figure 8: Screenshot of the accompanying video showcasing the energy dynamics of our model. Top left: the energy of our model over time (with units 7) for the dog image highlighted at the top right. Each cell of the dog represents the (masked input, reconstructed image at time 6, original image). We step through time using time step of dt 0.1 and record the total energy of our system on the image tokens as the black dot descending the blue energy curve. The dashed vertical black line shows the point in the energy curve where the representations were passed the loss function when training the model, whereas the horizontal red dashed line shows the "fixed point" at the end of the simulated dynamics (in reality, the energy still descends slightly after that). Bottom: We display 11 other images as (masked image, reconstructed image at time t, original image) aligned with the time step. Each image's 's energy trajectory will be slightly different (not shown).
17

---

Under review as a conference paper at ICLR 2023
B DETAILS OF ET TRAINING ON ANOMALY DETECTION TASK
Graph anomaly detection refers to the process of detecting outliers that deviate significantly from the majority of the samples. Neural network based methods are very popular due to their capa- bility of learning sophisticated data representations. DOMINANT (Ding et al. 2019) utilizes an auto-encoder framework, using a GCN as an encoder and two decoders for structural reconstruc- tion and attribute reconstruction. ALARM (Peng et al., 2020) aggregates the encoder information from multiple view of the node attributes. Another study Zhao et al. (2020), propose novel loss function to train graph neural networks for anomaly-detectable node representations. In Ding et al. (202T) generative adversarial learning is used detect anomaly nodes where a novel layer is de- signed to learn the anomaly-aware node representation. Recently, Tang al. (2022) pointed out that anomalies can lead to the "rightshift" the spectral energy distribution the spectral energy concentrates more the high frequencies. They designed a filter that better handle this phe- nomenon. We propose a new anomaly detection model from the perspective of Associative Memory (pattern matching), which does not have the over-smoothing problem often faced by GCNs, and has better model interpretability (outliers should be far from the common pattern).
B. DETAILED MODEL STRUCTURE FOR THE GRAPII ANOMALY DETECTION
First, we compute the features that are passed our energy- transformer. Each node's features Yл E n are mapped into the token space X E R2 using linear projection E. Learnable positional embeddings d are added to this token at 1,
Ey A Aa
(13)
At each time step the input to the ET-block is layer normalized:
g = LayerNorm(x(x)
(14)
Let WΩ RXHHD and w E RR x be the query and key weight matrices, respectively. Here Y is the projection dimension in the attention operation, H is the number of heads. We define
KahB = Σ W 9jB
(15)
Qanc = Σ
w ahj 9&C
If we let & indicate the index of the head, we have
Azt Σ Σ
Wº ahi KatC WCA + W.K. Qatc wsc + + r (E Ewgsss)
(16)
CENA h,a
where
WCA = softmax C
8 Σ KinC Quna)
(17)
Here 3 controls the temperature of the softmax, NA stands for the neighbors of node A A-a a set of all nodes connected to node A, r the ReLU function. Restriction of the attention operation to the neighborhood of given node is similar to that used in the Graph Attention Networks (GAT), see (Veličković et al. 2017). Finally, we have residual connection
x
Ax
(18)
Intuitively, the first term considers the influence (attention score) of the neighbor nodes with respect to the target node, the second term considers the influence of the target node with respect to each of its neighbor, and the third term is the contribution of the Hopfield Network module. It can be shown that the forward pass of our energy-based transformer layer minimizes the following energy function:
E
log Σ exp BeNo
Σ KahB Qunc))
I' C,u
Xus 2)c)² (19)
C
18

---

Under review as a conference paper at ICLR 2023
This energy function will decrease as the forward pass progresses until it reaches a local minimum.
After T' iterations when the retrieval stable, we have the final representation for each node final as
SA
SA TAT
(20)
where || is the concatenation sign. Following Tang et al. (2022), we treat anomaly detection as semi- supervised learning task in this work. The node representation g is fed to another MLP with the
sigmoid function to compute the abnormal probability Pa. weighted log-likelihood then used to train the network. The loss function is follow:
Loss = Σ
I
La log(pA) + (1 - (A) log(1 PA)
(21)
where is the ratio of normal labels (l, 0) anomaly labels (IA 1).
B.2 EXPERIMENTAL DETAILS
We train all models for 100 epochs using the Adam optimizer with a learning rate of 0.001, and use the model with the best Macro-FI on the validation for reporting the final results on the test set. Following Tang et al. (2022). we use training ratios 1% and 40% respectively (randomly select I 1 and 40% nodes of the dataset to train the model, and use the remaining nodes for the validation and testing). These remaining nodes are split 1:2 for validation::estingg The statistics of the datasets are listed Table 3 For the four datasets used the experiments, Amazon and Yelp datasets can be obtained from the DGL library, T-Finance and T-Social can be obtained from Tang et al. (2022).
Datasct V
E
Anomaly(5))
Features
Amazon
11944
4398392
6.87%
25
Yelp
45954
3846979
14.53%
T-Finance
39357
21222543
4.58%
10
T-Social
5781065
73105508
3.01 %
10
Table 3: Summary of all the datasets.
We report the average performance of runs on the datasets. The hyperparameters of our model are tuned based on the validation set, selecting the best parameters within 100 epochs. To speedup the training process, for the large graph datasets T-Finance and T-Social, different subgraph to train for each epoch (subgraphs have 5% of the nodes with respect to the whole training data). The hyperparameterss include the number of hidden dimensions in ET-attention Y, the number of neurons K in the hidden layer within the Hopfield Network Module, the number of time iterations T, and the number of heads H. The weights are learned via backpropagation, which includes embedding projection E, positional embedding AA, softmax inverse temperature parameter 3, ET-attention weight tensors Wº and W The optimal hyperparameters used in Table are reported in Table The last row in that table summarizes the range of the hyperparameter search that was performed our experiments. In general, we have observed that for small datasets (Yelp, Amazon, T- (TFinance) 1 applications of network is sufficient for achieving strong results, for larger datasets (T-Social) more iterations (3) are necessary. For even bigger dataset (ImageNet) network needs about 12 iterations.
sample
Dataset
K
Amazon (40%)
128
640
Amazon (1%)
64
128
1
Yelp (40%)
128
256
1
Yelp (1
128
256
T-Finance (40%)
128
256
T-Finance (1%)
128
256
1
T-Social (40%)
128
256
3
T-Social (1%)
128
256
Range of hyperparameters
64.
128, 256
(2Y, 3Y, 4Y, 5Y
[12.3)
[12,3
Table 4: Hyperparameters choice of our method on all the datasets.
19

---

Under review as a conference paper at ICLR 2023
C
NOTATIONS
Table 5
lists all the notations used in this paper.
Table 5: Notations used the paper.
Notation
Description
dimension of node's feature space
D
dimension of token space number of tokens number of hidden dimensions in the attention
Y
M
number of hidden dimensions in the Hopfield Network
T
number of recurrent time steps
H
number of heads
k
height of each image patch
kar
width of each image patch number of pixels per image patch k4 x kw)
A
input feature vector of node A vector representation each element of vector representation of token A vector representation of token after layernorm
XA
token
ZiA
BA
each element of vector representation of token A after layernorm key tensor query tensor KahB
91A
K
Q
each element of the key tensor K each element of the query tensor Q label of node A on graph
QatC
LA
D ABLATION STUDY FOR ATTENTION AND HOPFIELD NETWORK MODULES
As we described in the main text the the ET network consists of two modules processing the tokens in parallel: the attention module (ATT) and the Hopfield Network module (HN). The ATT module is responsible for routing the information between the tokens, while the HN module is responsible for reinforcing the token representation to consistent with general expectation about particular data domain. It is interesting to explore the contribution that these two subnetworks produce on the task performed by the network. this section we ablate the ET architecture by dropping each of the two subnetworks and measuring the impact of the ablation the performance.
D.1 ON GRAPHS
The results on graphs are reported in Table 6 From this table it is clear that most of the computation performed by the ATT block this task, which pools the information about other tokens the token of interest. When ATT block is kept, but HN block is removed the network looses 1% or less relative to the full ET (occasional improvements of the ablated model compared to the full ET are within the statistical error bars). In contrast, removing ATT module and keeping only the HN, the ET network effectively turns into an MLP with shared weights that recurrently applied. In this regime the network can only use the features of given node for that node's 's anomalous status prediction. This results in a more significant drop in performance, which is about 5% on average.
D.2 ON IMAGES
The ablation results for image reconstruction shown in Table Each experiment was trained using the same hyperparameter settings as shown in Table 2 After training the model on INIK, we calculate the average MSE the reconstructed masked tokens for the validation set (using the same 50% masking ratio used for training) across 10 different random seeds for the masking.
We make several conclusions from these ablation studies.
20

---

Under review as a conference paper at ICLR 2023
Table 6: Ablation study with respect to ATT block and HN Block. Best results are in bold.
Datasets
Split
HTT/
HN
full model (Ours)
1%
62.5+0.3
57.4
(-5.6)
63.0+0.6
Yelp
40%
70.640.5
71.240.7
71.5
1%
89.5
87.4+1.0
89.3+0.7
Amazon
Macro-FI
40%
91.7
(-1.1)
88.7
(-4.1)
92.8
1%
84.7
80.340.6
V-4.8)
85.111.0
T-Finance
40%
87.4
82.310.8
(-5.9)
88.211.0
1%
79.8
72.7+1.0
V(-..4)
79.1
T-Social
40%
82.9+1.0
78.6+1.2
(-4.9)
83.5+0.4
1%
42.9
67.440.7
(-5.8)
73.2
Yelp
40%
83.5
(-1.4)
83.1
(-1.8)
84.9
1%
90.740.8
89.8+1.2
91.9+1.0
Amazon
40%
96.8
95.7
(-1.6)
97.3 97.3+0.4
AUC
1%
91.7
90.2
(-2.6)
92.8
T-Finance
40%
94.3
90.2+2.1
95.0+3.0
1%
92.210.8
86.4
(-5.5)
91.9
T-Social
40%
93.1
88.341.3
(-5.6)
93.9+0.2
We gain several insights regarding the use of "self-attention" in our ET (when token patch query is allowed to consider itself as a key in the attention weights). When both self- attention and HN are present (ET-Full+Self), there is no noticeable benefit over ET-Full for token attend to itself. In fact, preventing the ATTN energy module from attending to itself slightly improves the performance. However, when the HN is removed (ET-NoHN**), we notice that allowing self-attention (ET-NoHN+Self) outperforms the version that pre- vents self-attention (ET-NoHN).
On its own, allowing self-attention (ET-NoHN+SelI) in the ATTN module performs nearly as well the full ET fraction of the total parameters. However, MSE is a forgiving metric for blurry reconstructions. While ATTN can capture the global structure of the image quite well, it does so at the expense image sharpness (Figure 4).
As expected, removal of the ATTN energy module performs the worst, because the HN operates tokenwise and has no way to aggregate token information across the global image without ATTN.
Figure 9 shows best performing model (ET-Full) the qualitative image reconstructions responding to the largest errors across INIK validation images, averaged across all masking seeds. Likewise, Figure 10 shows the lowest errors across IN1K validation images and masking seeds. In general, image reconstructions that require ET to produce sharp, high frequency, and high contrast lines negatively impact MSE performance.
Module ablation tests for image reconstruction task, reporting average IN1K validation MSE on masked tokens after 100 epochs. Reported number of parameters excludes the constant number of parameters in the affine encoder and decoder.
Table 7:
Allow self-attn?
NParams (in ET block)
Model
Has ATTN?
Has HN?
MSE
ET-Full (Ours)
3.7M
0.306+1..0
ST-Full+Self
3.7M
0.312+0.10
ET-NoHN+Self
X
1.3M
0.343 10.10
ET-NoHN
X
X
1.3M
0.403
ET-NoATT
X
2.5M
0.825+0.20
21

---

Under review as a conference paper at ICLR 2023
Avg MSE: 0.854
Avg MSE: 0,764
Avg MSE: 0.762
Avg MSE:
Avg MSE:
Avg MSE: 0.735
Avg MSE:
Avg MSE:
Avg MSE:
Aug MSE: 0.731
0.76
0.744
0.735
0.734
0.732
Figure 9: Reconstruction examples images with the worst MSE from the INIk validation Top row: input images where 50% of the patches are masked with the learned MASK token. Middle row: all tokens reconstructed after 12 time steps. Bottom row: original images.
Avg MSE: Avg MSE: 0.0099
Avg MSE:
Avg MSE: 0.032
Avg MSE: 0.0324
Avg MSE: 0.0341
Avg MSE: 0.0389
Avg MSE: 0.0396
Avg MSE: 0.0421
Avg MSE: 0.0423
0.0246
0.0269
Figure 10: Reconstruction examples of images with the best (lowest) MSE from the INIK validation set. Top row: input images where 50% the patches are masked with the learned MASK token. Middle row: all tokens reconstructed after 12 time steps. Bottom row: original images.
E ET FOR HETEROGENEOUS GRAPH
In this section, we show the performance of our ET model in the heterogeneous graph case. For the heterogeneous graph case (where more than one type of edges exist in the graph), we first run our ET model for different subgraphs (corresponding to different types of edges), and then aggregate the final representations using two methods. We have tried max pooling and concatenation for the aggre- gation step. Max pooling means to pick the largest value across all the subgraph representations, and concatenation means to concatenate the representations obtained from different subgraphs. Table shows the comparison between these two variants of our model and BWGNN (heterogeous case). ET performs better than heterogeneous BWGNN on Amazon, but loses to heterogeneous BWGNN on Yelp. Interestingly, BWGNN in the heterogeneous setting performs worse than BWGNN in the homogencous setting on Amazon.
F GRAPH CLASSIFICATION WITH ET
As mentioned prior, GNNs have emerged popular approach to handle graph-related tasks due to their effective and automatic extraction of graph structural information. There have been attempts of leveraging Transformer into the graph domain, but only certain key modules, such as feature aggregation, are replaced in GNN variants by the softmax attention Ying et al.) (202T). However, the Transformer model has yet achieved competitive performance popular leader boards of graph-level prediction compared to mainstream GNN variants Ying et al. (202I) In this section we explain how ET can be used for graph classification.
22

---

Under review as a conference paper at ICLR 2023
Table 8: ET for anomaly detection in heterogeneous graph setting. Best results are in bold.
Datasets
Split
MaxPool
Concatenation
BWGNN (Heterogenous)
Macro-F F1
1%
61.540.4
61.7+0.2
67.02
Yclp
40%
70.7+0.6
71.1
76.96
1%
88.341.6
87.4+1.4
83.83
Amazon
40%
92.110.2
91.8+0.2
91.72
1%
72.240.5
72.8+0.2
76.95
Yelp
AUC
40%
84.340.4
85.1+0.1
90.54
1%
90.8+1.4
90.641.0
86.59
Amazon
40%
97.5+0.1
97.2+0.6
97.42
F.1 DETAILS OF GRAPH CLASSIFICATION ET MODEL
Given a graph G (V, where V = (ui, v......... uN) and A E {0, 10, is the adjacency matrix of the graph, each feature vector XB E R corresponding to node UB first projected to the token space Xg IR via a linear embedding. Then, the CLS token XCLS is concatenated to the set of tokens resulting in X E R[N+1)xD and the positional embedding À R(++1)x is added afterwards.
To obtain the positional embedding Å, the adjaceney matrix A is first padded in the upper left corner with ones resulting in Á E {0, 1) This particular step provides the CLS token full connectivity with all of the nodes in the graph. The top & smallest eigen-vectors À E R(N+1)x then obtained from the eigen-wel cigen-value decomposition of the unnormalized Laplacian matrix
L = D AĀD
(22)
and projected to the token space via a linear embedding to form the positional embedding Ā E [[N+))x
Meanwhile, the attention in ET is modified to take in Å E (=x(++I)((++1) the parameterized adjacency tensor, which acts as the weighted 'attention mask" that enables the model to consider the graph structural information. To obtain A, a 2D-convolutional layer with filters equals to the number of heads in the attention block, 'SAME' padding, and stride of is performed on the outer product of X itself. The result is then multiplied with Á element-wis (denoted by @) via broadcasting.
= Conv2D & & Ā
(23)
Altogether, the resulting energy equation is
EAIT_ 1 B
ΣΣ log Σ exp B B
KanB Qaic Aanc)
(24)
B&C
Additionally, in this implementation, the overall model is consisted of S vertically stacked ET blocks, where each block shares the same number of T depth and has different LayerNorm. Similarly, the token representation X each dynamic step corresponding to block is first layer-normalized. Keep in mind, X xt=1, t=1 is the initial token representation.
gx f LayerNorm X
(25)
Following the dynamic equations 6 we inject small amount of noise et E (0, 1). generated from normal distribution with standard deviation of 0.02 and zero mean, into the gradient of energy function to produce X,+ the new token representation of block t. The premise of this noise injection is to 'robustify" the model and help push towards a local minimum of the energy function.
23

---

Under review as a conference paper at ICLR 2023
8 -
(26)
Once stability is reached in the retrieval dynamics of block E, the final representation X c is then passed on to the next block + 1 and the whole process is repeated again. When the final token representation Y = Xx=T, t=8 is computed by the last block S, the resultant CLS token Yo E R D extracted from Y is utilized as the predictor of the current graph G.
F.2 EXPERIMENTAL EVALUATION
Eight datasets of the TUDataset Morris et al. 2020) collection used for experimentation. NCTI, NCT109, MUTAG, MUTAGENICITY,
FRANKENS TEIN are common class of graph datasets consists of small molecules with class labels representing toxicity or biological activity determined in drug discovery projects Morris et al. (2020). Meanwhile, DD, ENZYMES. and PROTEINS represent macromolecules. The task for both DD and PROTHINS is to classify whether protein is an enzyme. Lastly, for ENZYMES, the task is to assign enzymes one of the 6 EC-top-level classes, which reflect the catalyzed chemical reaction Morris et al. (202
We compare ET with the current state-of-the approaches for the mentioned datasets, which include WKPI- kmeans Zhao & Wang (2019). WKPP-kcenters Zhao & Wang (201)), DSGCN Balcilar et al. (2020), HGP-SL Zhang et al. (2021), ,. U2GNN Nguyen et al. (2022). and Evolution Graph Classifier (EvoG) Domingue al.] (2019). Additionally, approaches that are close the baselines are included further contrast the performance of our model. Following the 10-fold cross validation process delineated in Morris et al.) (2020), accuracy score is used as the evaluation metric and reported in Table 9
Table 9: Performance of all the methods the graph classification datasets, where additional fea- tures are used if exist. Following Morris et al. (2020), mean and standard deviation obtained from 10 runs of 10-fold cross validation are reported and the baselines (standard deviations are only included if they are available in the prior work). If the entry is unavailable in prior literature it is denoted by :.'; best results are in bold. The performance difference between non-baseline approaches (includ- ing ours) and the baseline (specified by the gray cell in each column) is indicated by (decrease) and (increase) along with the value.
Dataset
Method
PRCITEINS
NCT
DD
ENZYMES
MUTANG
MUTAGENICITY FRANKENSTRIN
(kmeam)) 78.52m4 v(8.4)
WKPI
87.5.0.
85.9.94 T(L5)
82.0ans ((3.7)
85.8225 v(142)
(koenters)
WKPI
75.2 v(07))
84.5.aa w(3.0)
80.3as.4 V(15.4)
88.312 ((117))
Spec-GN
84.8 v(2.7)
83 $3.6 T(3.8)
72.5 v(5.))
Norm-GN
84.9g1 ((2.6)
83.9gia ((39))
79.3g. (3.1)
GWL-WL
75.8 (90.)
71.3a. v(7.1)
78.9.m.1
HGP-SI. 84.9.1
78.5an v(U.)
807 v(67))
$1.0gia ((147)
68.8gax v(9.6
82.2
DSGCN
77.3guc v(7.6)
78.4au
U2GNN
50.0gam (4.9)
68.5ge v(11.5)
NDP
73.4aa
T(1.5)
742 ((13.))
725 ((2.0)
46.5a74 ((SL0)
87.0gar ((12.1) 77.9.s v(4.3)
ASAP
74.2ps 7(077) 71.5gs T(160) 70 10 1/17.3) 76.9 ((18.8)
06.3gns T(12.6)
Evo
55.7 ((2.7)
100.0
ET (Durs)
78.9 (6.0) 83.6a0a ((a.))
82 82-4 ((6.0)
84.6ga ((111) 98.8 415.4 98.7 ((0.3)
88.5 18.5 A62 99.3 9. 421.0
In general, we have observed that the modified ET demonstrates consistent performance across all datascts that is near the current state-of-the-art approaches. Based on the statistics of the experimental datasets table To ET performs extremely well when trained on large graph datasets (c.g., MUTAGENICITY and FRANKEN. STEIN). However, with respect to NCII, NCI109, and DD datasets, there remains open question on which graph characteristics (e.g., assortativity and density) impair the performance of the model.
F.3 EXPERIMENTAL DETAILS
In the graph domain, it is common to concatenate all of the feature vectors of all graphs in batch together. However, in order for ET work, form the batch dimension by separating the feature vectors of all graphs
24

---

Under review as a conference paper at ICLR 2023
given batch and utilize the largest node count to pad all graphs such that they all share the number of nodes. Additionally, we limit on the number of nodes (set as 400) to prevent out-of-me out-of-memory error. Specifically, graph has node count exceeding the limit, the number of utilized nodes is equal to the limit. Hence, a portion of the graph structural information is ignored as result. However, it is worth mentioning such graph is rare in the experimental datasets.
Table 10: Graph classification dataset statistics and properties (additional node attributes are indi-
cated by '+' if exist).
Dataset
Graphs
Avg. Nodes
Avg. Edges
Node Attr
Classes
MUTAG
188
17.93
19.79
ENZYMES
32.63
62.14
18 +
PROTEINS
1113
39.06
72.82
0
DD
1178
284.32
715.66
89
2
NCII
4110
29.87
32.30
37
NC1109
4127
29.68
32.13
38
MUTAGENICITY
4337
30.32
30.77
FRANKENSTEIN
4337
16.90
17.88
780
We train all models for 200 epochs using AdamW [Lohccilov & Hutter (2017). The best model is selected based on its performance obtained from the 10-fold cross validation process delineated in Morris et al. (2020). Since the task is classification, all models Additionally, the cosine-annealing with warm-up learning learning rates are set as 8 and 0.001, respectively. The number warm-up steps is set to 30 epochs while the batch size is 64 for all datasets with the exception of the DD dataset, which requires a batch size of 256 for faster training time. The whole experiment is implemented using JAX JAXBradbury et al. (2088), Flax (Heek et al. 2020
trained with the cross-entropy loss function with temperature. scheduler utilized, where the initial and peak
Jraph G al. 2020). and Py Torch Geometric Fey & Lenssen (2019) packages.
Lastly, report the average performance of runs the 10-fold cross validation process with random seed- ing. The hyperparameters of our model are tuned based on the performance of the cross validation, selecting within 100 epochs. The optimal hyper-parameten are reported in table [1 and the statisties of the used datasets are reported in TT
Table 11: Hyperparameter and architecture choices for ET during graph classification training ex- periments.
Training
Architecture
batch_size
64
token_dim
128
batch size
256
num heads
12
epochs
200
head dim
64
Ir
le-3
B
/64
warmup-epochs
30
tetas
No
start & end Ir
5e-7
step size a
0.01
b1, b2 (ADAM)
0.9, 0.99
eigenvalues
15
weight_decay
0.05
depth
grad clipping
None
block size
kernel size
[3, 3]
dilation_sis
[1,
hidden_die _im HN
512
bias HN
None
bias in ATT
None
bias LNORM
Yes
of params
530,084
no. of params per block
262,929
G FORMAL ALGORITHM
We describe the algorithm for the training and inference of ET in algorithm assuming backpropagation through time using SGD. Symbols not defined in the algorithm itself reported in Table 5 Infer function operate independently over each item in batch.
We define the
25

---

Under review as a conference paper at ICLR 2023
H PARAMETER COMPARISON
model, which means ET has fewer parameters than its ViT attention mechanism, and the HN module has only one of the two matrices in the standard MLP of the traditional Transformer block. We report these differences in Table 12 We take the ET configuration used in this paper, which has architecture fully comparable to the original VīT- base (Dosovitskiy al. 2021) with pat ch_s 122=66 6, and report the number of parameters against ViT-base and "ALBERT" version of ViT (Lan al. 2020) where single ViT block shared across layers. We saw no benefit when including biases in ET, so we also exclude the biases from the total parameter count in the configuration of ViT and ALBERT-ViT. We report both the total number of parameters and the number of parameters per Transformer block.
The energy function enforces symmetries counterparts. In particular, ET has no "Value Matrix" W
Comparison between the number of parameters in a standard VIT, an ALBERT version of VIT where standard Transformer blocks are shared across layers, and our ET. Comparison is done assuming no biases in any operation.
Table 12:
NParams (per block)
Model
NParams
ViT-Base
86.28M
70.00%
7.08M
+0.00%
ALBERT_ViT-BaL
8.41M
90.25%
7.08M
70.0%%
ET
4.87M
94.36% %
3.54M
50.02% %
26

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Training and inference pseudocode ET for image reconstruction task
HyperParameters
a:: Energy descent stepsize
Learning rate
Token mask probability
b: batch size
Parameters W E ER w E ERXMXP; Key, Query kernels of the Energy Attention
& E CERMXD, Kernel of Hopfield Network
"norm E IR, bunna E PR Scale, bias of LayerNorm
MASK P: Mask token
10
11 8pos RNDD; Position bias, added to each token
Wese PxD dene I Kernel, bias of affine Encoder
Wdec R DXP Oáee E P: Kernel, bias of affine Decoder
13
14
Infer
15
Inputs
Corrupted image tokens X NXD
16
Add position biases: X <- X Opus:
18
for timesteps 1 ,...., do
19
Normalize each token:
20
9 f LayerNorm(A) "horm, Onom); Calculate Energy of tokens:
g E RND
21
22
E <- Energy Transfomme(s W W .):;
EER
23
+ X VV
24
return
25
Train
Inputs
Dataset Smain with elements X R gehannels height width width
27
Initialize
29
Randomly initialize from (0, 0.02):
W W &, MASK, W ene, Wdse doe; Opos (N, 0.02) Set other biases zero: Sene, ddee, Euorm f-
31
32
Set LayerNorm scale to one: "Ynor f-
for epoch ,, do
Sepoch Swrii
for batch B C Seyech
B x
36
37
Convert image into non overllapan patches:
Bpacc +- Patchify (B):
Bpuch E RRxNNxPP
39
Embed image patches into tokens:
49
X + Encode( Bpann' Wese. eae, 6ese):
X E R
41
Replace image tokens randomly by MASK:
42
X, Tmaa +- Mask( X; MASK, p)
X E RIXNDD Track {0, 12
Reconstruct tokens with ET:
X + Infer(XX)
Decode tokens:
Bpanc Decode X [maak]: Ware: dee, ddec);
Bpuch xxP
Calculate MSE loss on corrupted tokens:
L Meaa[[Bunnce [maa]] Bpaee [maxx] )1²)
R
49
params <- params <V parameL
50
Sepoch + Sepoct B
51
return params
27


Reviews:
[{"review_id": "pjD3Ze3n0i", "reviewer": "Reviewer_VTZR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a transformer architecture called Energy Transformer that replaces the sequence of feedforward transformer blocks with a single large Associative Memory model. The sequence of the energy transformer layers is designed to minimize an energy function in charge of representing the relationships between the tokens. Experiments show that Energy Transformer achieves good performances on image completion and graph anomaly detection.  ", "summary_of_the_review": "The primary concern of this paper is its weak experiments, which do not sufficiently demonstrate the strength of the proposed Energy Transformer.", "strength_and_weaknesses": "[Strengths]\n+ The transformer is a prominent attention mechanism in machine learning, and it is interesting to design the Transformer mechanism from energy attention and the Hopfield network.\n\n[Weaknesses]\n1) The connection between Hopfield Networks, energy function, and Transformer is mentioned in Ramsauer et al. (2020). It is not clear the new insight of the proposed Energy Transformer.\n2) The effect of designing the proposed ET block is not apparent; it is better to conduct an ablation study experiment to assess each component within such a designed block.\n3) The detailed structure of the Energy Transformer is not explicitly depicted. A network architecture helps the reader to understand the ET implementation.\n4) Since the Energy Transformer requires recurrently updating, it is better to show the cost of training and testing compared with the vanilla Transformer.\n5) The experiments are weak. The quantitative result has merely the graph anomaly detection. However, the results of BWGNN (Hetero) and [A], which shows good performance, are not included. The on-par performance of the Energy Transformer does not make it impressive to the reader.\n\nRelated paper:\n[A] Derek Lim, Xiuyu Li, Felix Hohne, and Ser-Nam Lim: New Benchmarks for Learning on Non-Homophilous Graphs. Workshop on Graph Learning Benchmarks, WWW 2021.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "fJIrZsXf66p", "reviewer": "Reviewer_v3Fm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a new transformer architecture called energy transformer (ET). ET is designed to minimize a special handcrafted energy function. This function contains two terms. The attention term aligns the keys and queries between neighbor token vectors and the hopefield network term matches the token vectors to some memory vectors. The ET layer is applied recurrently to imitate a continuous time differential equation that minized the designed energy terms. The authors try to demonstrate the ability of their model through image completion task and graph anomaly detection task.", "summary_of_the_review": "I think this paper proposes some interesting ideas like designing a energy function and deriving a transformer structure based on this energy function. However, I think the experiments of this paper may not be strong enough to demonstrate that the new designed structure does have advantage over the original one. ", "strength_and_weaknesses": "Strength:\nIn general, I like the idea of this paper:\n1. This paper propose an specially designed energy-term that may be helpful for understanding the transformer structure.\n2. It derives a new transformer structure based on this handcrafted energy-term\n3. The authors show that this structure outperforms baselines in the graph anomaly detection task.\n\nPossible concerns:\n1. While the authors showcase experiments on image completion and graph anomaly detection task. I think these are not enough for me to draw the conclusion that this newly designed arctecture does have advantage over the previous one. The transformer structure is famous because it performs well in various task in computer vision(CV) and netural language processing(NLP). To really demonstrate the ability of a new structure, the authors need to show competitive results on those widely accepted tasks like transfer learning to ImageNet in CV or results on GLUE [1] benchmark in NLP. Currently, the authors only showcase two tasks that are well-suited for their model (and they only show quantative results in one of them), thus the general ability of this model is still unclear to me.\n\n2. Some designs of the model are not fully justified experimentally. For example, unlike traditional transformer architecture which uses different weights for different layers, ET apply a single block recurrently and the MLPs in their hopfiled network also share weights . The authors may find theoretical explanation for this design according to their energy design. However, whether this helps or hurts the performance is unclear. Empirically speaking, recurrent model or model with shared weight may be hard to optimize due to the vanishing gradient problem. Thus, ablation study may be needed to justify the these design choices.\n\n3. Questions for the graph anomaly detection task: How are the edges used by the model? Does the attention layer only compute attention over the nodes that have edges between them? Also, can the model be also used on directed graph? Or it is only limited to undirected graph?\n\n4. For the image completion task, the authors assumes the image patches can be represented as fully masked and fully open area. But what if the masked area is not divisible by the patches? Then do you need to re-design the patch size (which may influence the full structure of the model)? This might not be a problem if one just treat image completion as a self-supervised task to learn representations and then applied these learned features to downstream tasks. But if the image completion task itself is the target and its performance is used to showcase the ability of the model. Then possible limitation may need to be considered.  \n\n[1] GLUE: A MULTI-TASK BENCHMARK AND ANALYSIS PLATFORM FOR NATURAL LANGUAGE UNDERSTANDING\n\n=====================================================================================================\n\n**Post rebuttal:**\n\nFor me, although in the new responses, the authors say that they are not trying to replace the original transformer architecture, what the paper mainly about is proposing a new transformer architecture based on modification of the original one. Thus, a natural question to ask is how this new design performs comparing with the original one. Although the authors show that their design principle is based on the Hopfield Networks, whether this principle does benefit the transformer architecture design needs further justification. There can be many interesting principles in modifying the designs, but whether they do make senses need to be supported by comprehensive and convincing experimental results. A valid modification to the transformer architecture needs to demonstrate its advantage over the original one. The current results, however, are far from that. The current results are mainly based in the graph domain. Solid experimental results in areas that transformers are mainly used in, like CV or NLP, are missing. I can not agree with the authors that since they expertise in graph domain, they only pick experiments in this area and miss others. As I said, if the paper narrows its objective into proposing a new technique to achieve STOA results in graph domain, then the current results are fair enough. However, if they want to make a valid and general design principle for transformer architecture, they should compare their new architecture in the areas that transformer architecture are widely used. As a reviewer, I'm not convinced by the current results and thus I still lean to reject. I would recommend the authors to polish their work with more convincing results in different areas or limit their objective to a technique that specialized in graph domain.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TghgcpJcrYw", "reviewer": "Reviewer_iuLg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the energy transformer, a transformer architecture that uses a recurrent energy transformer block. The energy transformer block updates its input in accordance to minimizing two energy functions. Experiments on image reconstruction and graph anomaly detection demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "Although the paper is interesting to read, I have two major concerns as detailed in the weakness section. ", "strength_and_weaknesses": "Strength: The proposed method is interesting in the sense that it reformulates transformer architecture in a way to combine Hopefield networks and attention networks, which are recurrently optimized using two carefully designed energy functions. The empirical results look promising. \n\nWeakness: I am concerned about the clarity of the paper. For example, in the explanation of equation (3), \"The log-sum energy function (3) is minimal when for every patch in the image its queries are aligned with the keys of a small number of other patches connected by the attention map,\" It is not clear what does this mean in a mathematical sense. There is also no detailed algorithmic description about how the model is trained. For example, for the image reconstruction task, how does minimizing the energy functions correspond to reconstruction? It would be helpful if the authors include a formal algorithmic description of the proposed method. \n\nI am also concerned about some of the experiment settings. For example, for image reconstruction task, Figure 3 only shows some of the examples. However, this does not represent its general performance. One way to make this experiment more complete may be to include the MSE of the test datasets, the variance of the MSE, and perhaps reconstructed images corresponding to the best MSE and the worst MSE. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PueTw2UFKAr", "reviewer": "Reviewer_UVr7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThe paper puts forward a novel sequence-to-sequence neural architecture named Energy Transformer (ET) that builds\non recently introduced generalized Hopfield networks and their connection to transformer layers. \nThe architecture is composed by a single block that combines two subsystems,\ndefined by energy functions corresponding to two different types of generalized hopfield networks, \npreceded by a layer normalization operator. \nOne subsystem is in charge of pooling to each position the information coming from the other\npositions, while the other enforces the consistency of learned representations.\nThe system dynamics is given by a differential equation driven by the gradient\nof the energy function with respect to the renormalized features. The total\nenergy is proven to decrease along the trajectory. \nFor practical purposes, the dynamics is discretized and the block applied recurrently.\nThe architecture is validated qualitatively on an image occlusion task and quantitatively\non a node anomaly detection task.", "summary_of_the_review": "I think this is a good paper that deserves acceptance.\n", "strength_and_weaknesses": "\nThe manuscript is an enjoyable read and contains a good degree of novelty. \nThe output of the network is given by the final outcome of an energy minimization process.\nAlthough this perspective is not new (see Yang et al '22), it has not been quite explored yet\nand I find the particular instantiation given in the paper quite inspiring. \nI think this could trigger additional theoretical and applicative results.\nThe resulting architecture is also quite simple and has a good degree of interpretability.\nThe paper is well organized, although some details given in the appendix could be moved to the main text.\nThe results on the node outlier detection tasks are very good, outperforming concurrent methods in\nmost cases.\n\nMain comments:\n\n- Why the authors didn't attempt to benchmark the performance on a standard image classification task? \n  This should be doable by pooling and putting an head on top of the final outputs, as commonly done in ViT.\n\n- Related to previous questions, is not clear if the ET can be considered as a drop-in replacement to self-attention\n  transformer blocks for arbitrary task, or if its usage should be confined to some specific settings.\n  I would like the authors to comment on the limitations of the model.\n\n- It is not clear how important the HN component is and how would perform the ET without it.\n\n- Could the author motivate why the attention module excludes the current the considere position from the input. \nWhat happens if you remove the restriction from the sum?\n\n- Did the authors explor stacking multiple ET modules in a feeforward fashion?\n\nMinor comments:\n- While it is clear in the appendix, I think also the main text should mention that the DE is discretized.\n\n- eq. 2, ignored the \\bar{x} dependence on x_i when computing the derivative. Should be fixed by inserting a factor 1 / (1 - 1/D) in the definition of L\n\n- \"is then minimized to train the whole network. Above, σ is the ratio of anomalous labels (lA = 1) to\nthe regular label.\" Maybe the other way around, that is regular labels / anomalous labels?\n\n- The main text should mention that for the graph task a neighborhood softmax (similar to GAT) is used instead of standard attention\n\n- The authors could consider numbering more equations for readibility.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DxFrdU9TsGw", "reviewer": "Reviewer_zHgn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new transformer architecture that replaces the sequence of transformer blocks with a single associative memory model. Specifically, the model is designed to minimize a specific energy function that represents the relations between tokens. The model is evaluated on image completion and graph anomaly detection, and experiments show encouraging results.", "summary_of_the_review": "It is an interesting work toward understanding the Transformer models. To show its effectiveness, more experiments have to be conducted on more challenging tasks.", "strength_and_weaknesses": "Strength\n- Despite the great performance achieved by the Transformer model, its architecture is designed empirically. This work proposes a Transformer architecture from a purely theoretical perspective.\n- This work provides a deep insight into the relationship between Transforms and associative memory models. Based on the relationship, it designs a new energy function and a corresponding Transformer architecture that minimizes the energy function.\n\nWeakness\n- To show the effectiveness of ET, it is better to evaluate it on more mainstream tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject