SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A distinctive aspect of our network is that any variable that has a vector index  $i$  of tokens can be mapped into the data domain by applying the decoder network to this variable. This makes it possible to inspect all the weights in the model. For instance, the concept of "memories" is crucial to understanding how Hopfield networks function. The memories within the HN module represent the building blocks of all possible image patches in our data domain, where an encoded image patch is a superposition of a subset of memories. The complete set of memory vectors from the HN module is shown in Figure 5. The same analysis can be applied to the weights of the ET-attention module. In Figure 6, we show all the weights from this module mapped into the image plane.
B.2 Positional Bias Correlations
The relationships between our position embeddings exhibit similar behavior to the position correlations of the original ViT in that they are highly susceptible to choices of the hyperparameters (Figure 10 of [18]). In particular, we consider the effect of weight decay and the  $\beta$  parameter that serves as the inverse temperature of the attention operation (see Equation 3). The lower the temperature (i.e., the higher the value of  $\beta$ ), the spikier the softmax distribution. By using a lower  $\beta$ , we encourage the attention energy to distribute its positional embeddings across a wider range of patches in the model.
Figure 5: Visualizing a randomly selected 3025 patch memories of the 3072 learned by weight matrix in the Hopfield Network module (HN) of our model. These memories are vectors of the same dimensions  $D$  as the patch tokens, stored as rows in the weight matrix  $\xi$ . Each image patch is visualized using the model's trained decoder.
C Details of ET training on Anomaly Detection Task
Graph anomaly detection refers to the process of detecting outliers that deviate significantly from the majority of the samples. Neural network based methods are very popular due to their capability of learning sophisticated data representations. DOMINANT [26] utilizes an auto-encoder framework, using a GCN as an encoder and two decoders for structural reconstruction and attribute reconstruction. ALARM [27] aggregates the encoder information from multiple view of the node attributes. Another study [51], propose a novel loss function to train graph neural networks for anomaly-detectable node representations. In [52] generative adversarial learning is used to detect anomaly nodes where a novel layer is designed to learn the anomaly-aware node representation. Recently, [24] pointed out that anomalies can lead to the "rightshift" of the spectral energy distribution - the spectral energy concentrates more on the high frequencies. They designed a filter that can better handle this phenomenon. We propose a new anomaly detection model from the perspective of Associative Memory (pattern matching), which does not have the over-smoothing problem often faced by GCNs, and has better model interpretability (outliers should be far from the common pattern). We also notice
Figure 6: Visualizing the token dimension of the "key" and "query" matrices of the attention as image patches. Each head is represented as a cell on the  $4 \times 3$  grid above. We use the trained decoder of our model to visualize each weight.
Figure 7: The cosine similarity between position biases of patches when the ET model is trained under different hyperparameter choices for  $\beta$  (inverse temperature of the attention energy) and weight decay. Our ET sees a trend where smoother correlations are observed with smaller  $\beta$  and weight decay.
that Modern Hopfield Networks have been used before for node classification, link prediction, and graph coarsening tasks [53].
C.1 Detailed Model Structure for the Graph Anomaly Detection
First, we compute the features that are passed to our energy-based transformer. Each node's features  $\mathbf{y}_A\in R^F$  are mapped into the token space  $\mathbf{x}_A\in R^D$ , using a linear projection  $\mathbf{E}$ . Learnable positional embeddings  $\lambda_{A}$  are added to this token at  $t = 1$ ,
$$
\mathbf {x} _ {A} ^ {t = 1} = \mathbf {E y} _ {A} + \lambda_ {A} \tag {14}
$$
At each time step the input to the ET-block is layer normalized:
$$
\mathbf {g} _ {A} ^ {t} = \operatorname {L a y e r N o r m} \left(\mathbf {x} _ {A} ^ {t}\right) \tag {15}
$$
Let  $\mathbf{W}^Q\in R^{Y\times H\times D}$  and  $\mathbf{W}^K\in R^{Y\times H\times D}$  be the query and key weight matrices, respectively. Here  $Y$  is the projection dimension in the attention operation,  $H$  is the number of heads. We define
$$
K _ {\alpha h B} = \sum_ {j} W _ {\alpha h j} ^ {K} g _ {j B}
$$
$$
Q _ {\alpha h C} = \sum_ {j} ^ {j} W _ {\alpha h j} ^ {Q} g _ {j C} \tag {16}
$$
If we let  $h$  indicate the index of the head, we have
$$
\Delta x _ {i A} ^ {t} = \sum_ {C \in \mathcal {N} _ {A}} \sum_ {h, \alpha} \left[ W _ {\alpha h i} ^ {Q} K _ {\alpha h C} \omega_ {C A} + W _ {\alpha h i} ^ {K} Q _ {\alpha h C} \omega_ {A C} \right] + \sum_ {\mu} \xi_ {\mu i} r \left(\sum_ {j} \xi_ {\mu j} g _ {j A}\right) \tag {17}
$$
$$
\omega_ {C A} = \operatorname {s o f t m a x} _ {C} \left(\beta \sum_ {\gamma} K _ {\gamma h C} Q _ {\gamma h A}\right) \tag {18}
$$
Here  $\beta$  controls the temperature of the softmax,  $\mathcal{N}_A$  stands for the neighbors of node  $A$  —a set of all the nodes connected to node  $A$ ,  $r$  is the ReLU function. Restriction of the attention operation to the neighborhood of a given node is similar to that used in the Graph Attention Networks (GAT), see [54]. Finally, we have residual connection (which is a natural consequence of the discretized differential equation dynamics)
$$
\mathbf {x} _ {A} ^ {t + 1} = \mathbf {x} _ {A} ^ {t} + \boldsymbol {\Delta} \mathbf {x} _ {A} ^ {t} \tag {19}
$$
Intuitively, the first term in Equation 17 describes the influence (attention score) of the neighbor nodes with respect to the target node, the second term describes the influence of the target node with respect to each of its neighbor, and the third term is the contribution of the Hopfield Network module. It can be shown that the forward pass of our energy-based transformer layer minimizes the following energy function:
$$
E = - \frac {1}{\beta} \sum_ {C} \sum_ {h} \log \left(\sum_ {B \in \mathcal {N} _ {C}} \exp \left(\beta \sum_ {\alpha} K _ {\alpha h B} Q _ {\alpha h C}\right)\right) - \frac {1}{2} \sum_ {C, \mu} G \left(\sum_ {j} \xi_ {\mu j} g _ {j C}\right) \tag {20}
$$
This energy function will decrease as the forward pass progresses until it reaches a local minimum.
After  $T$  iterations when the retrieval is stable, we have the final representation for each node  $\mathbf{g}_A^{\mathrm{final}}$  as
$$
\mathbf {g} _ {A} ^ {\text {f i n a l}} = \mathbf {g} _ {A} ^ {t = 1} \mid \mid \mathbf {g} _ {A} ^ {t = T} \tag {21}
$$
where  $||$  is the concatenation sign. Following [24], we treat anomaly detection as semi-supervised learning task in this work. The node representation  $\mathbf{g}_A^{\mathrm{final}}$  is fed to another MLP with the sigmoid function to compute the abnormal probability  $p_A$ , weighted log-likelihood is then used to train the network. The loss function is as follow:
$$
\operatorname {L o s s} = \sum_ {A} \left[ \omega l _ {A} \log \left(p _ {A}\right) + \left(1 - l _ {A}\right) \log \left(1 - p _ {A}\right) \right] \tag {22}
$$
where  $\omega$  is the ratio of normal labels  $(l_{A} = 0)$  to anomaly labels  $(l_{A} = 1)$ .
C.2 Experimental Details
We train all models for 100 epochs using the Adam optimizer with a learning rate of 0.001, and use the model with the best Macro-F1 on the validation set for reporting the final results on the test set. Following [24], we use training ratios  $1\%$  and  $40\%$  respectively (randomly select  $1\%$  and  $40\%$  nodes of the dataset to train the model, and use the remaining nodes for the validation and testing). These remaining nodes are split 1:2 for validation:testing. The statistics of the datasets are listed in Table 5. For the four datasets used in the experiments, Amazon and Yelp datasets can be obtained from the DGL library, T-Finance and T-Social can be obtained from [24]. We report the average performance of 5 runs on the test datasets. The hyperparameters of our model are tuned based on the validation set, selecting the best parameters within 100 epochs. To speedup the training process, for the large graph datasets T-Finance and T-Social, we sample a different subgraph to train for each epoch (subgraphs have  $5\%$  of the nodes with respect to the whole training data). The hyperparameters
Dataset	|V|	|E|	Anomaly(%)	Features
Amazon	11944	4398392	6.87%	25
Yelp	45954	3846979	14.53%	32
T-Finance	39357	21222543	4.58%	10
T-Social	5781065	73105508	3.01%	10
include the number of hidden dimensions in ET-attention  $Y$ , the number of neurons  $K$  in the hidden layer within the Hopfield Network Module, the number of time iterations  $T$ , and the number of heads  $H$ . The weights are learned via backpropagation, which includes embedding projection  $\mathbf{E}$ , positional embedding  $\lambda_{A}$ , softmax inverse temperature parameter  $\beta$ , ET-attention weight tensors  $\mathbf{W}^{Q}$  and  $\mathbf{W}^{K}$ . The optimal hyperparameters used in Table 1 are reported in Table 6. The last row in that table summarizes the range of the hyperparameter search that was performed in our experiments. In general, we have observed that for small datasets (Yelp, Amazon, T-Finance) a 1 or 2 applications of our network is sufficient for achieving strong results, for larger datasets (T-Social) more iterations (3) are necessary. For even bigger dataset (ImageNet) our network needs about 12 iterations.
Table 5: Summary of all the datasets.
Dataset	Y	K	T	H
Amazon (40%)	128	640	1	2
Amazon (1%)	64	128	1	1
Yelp (40%)	128	256	1	1
Yelp (1%)	128	256	1	1
T-Finance (40%)	128	256	1	3
T-Finance (1%)	128	256	1	1
T-Social (40%)	128	256	3	3
T-Social (1%)	128	256	3	3
Range of hyperparameters	{64, 128, 256}	{2Y, 3Y, 4Y, 5Y}	{1,2,3}	{1,2,3}
Table 6: Hyperparameters choice of our method on all the datasets.
D Graph Classification with ET
Recently, there have been attempts of leveraging Transformer into the graph domain, but only certain key modules, such as feature aggregation, are replaced in GNN variants by the softmax attention [55]. However, there remains an interesting and open question about suitability of the Transformer architecture to model graphs and how to apply it in the graph classification domain. In this section we explain how ET can be used for graph classification.
D.1 Details of Graph Classification ET Model
Given a graph  $G = (V, \mathbf{A}, \mathcal{E})$ , we have the set of nodes  $V = \{v_{1}, v_{2}, \dots, v_{N}\}$ , the adjacency matrix  $\mathbf{A} \in \{0, 1\}^{N \times N}$ , and if it exists, the edge feature matrix  $\mathcal{E} \in \mathbb{R}^{N \times N \times P'}$  where each edge has  $P'$  raw features. Each feature vector  $\mathbf{y}_{A} \in \mathbb{R}^{F}$  corresponding to a node  $v_{A}$  is first projected to the token space  $\mathbf{x}_{\mathbf{A}} \in \mathbb{R}^{D}$  via a linear embedding. Then, the CLS token  $\mathbf{x}_{\mathrm{CLS}}$  is concatenated to the set of tokens resulting in  $\mathbf{X} \in \mathbb{R}^{(N + 1) \times D}$  and the positional embedding  $\lambda \in \mathbb{R}^{(N + 1) \times D}$  is added to it afterwards.
To obtain the positional embedding  $\lambda \in \mathbb{R}^{(N + 1)\times D}$ , the adjacency matrix  $\mathbf{A}$  is first padded in the upper left corner with ones resulting in  $\tilde{\mathbf{A}}\in \{0,1\}^{(N + 1)\times (N + 1)}$  as a means to provide the CLS token full connectivity with all of the nodes in a given graph. Following [37], the top  $k$  smallest eigen-vectors  $\tilde{\lambda}\in \mathbb{R}^{(N + 1)\times k}$  are extracted from the normalized Laplacian matrix  $\tilde{\mathbf{L}}$  obtained from  $\tilde{\mathbf{A}}$
$$
\tilde {\mathbf {L}} = \mathbf {I} - \tilde {\mathbf {D}} ^ {- \frac {1}{2}} \tilde {\mathbf {A}} \tilde {\mathbf {D}} ^ {- \frac {1}{2}}, \tag {23}
$$
and projected to the token space via a linear embedding to form  $\lambda$ . Note,  $\mathbf{I}$  and  $\tilde{\mathbf{D}}$  are the identity matrix and the degree matrix of  $\tilde{\mathbf{A}}$ .
Meanwhile, the attention in ET is modified to take in  $\hat{\mathbf{A}}\in \mathbb{R}^{(N + 1)\times (N + 1)\times H}$  the parameterized adjacency tensor, which acts as the weighted 'attention mask' that enables the model to consider the graph structural information. To obtain  $\hat{\mathbf{A}}$ , a 2D-convolutional layer with  $H$  filters equals to the number of heads in the attention block, 'SAME' padding, and a stride of one is performed on the outer product of  $\mathbf{X}$  to itself. The result is then multiplied with the tensor  $\mathbf{A}'\in \mathbb{R}^{(N + 1)\times (N + 1)\times H}$  element-wise (denoted by  $\odot$ ) via broadcasting
$$
\hat {\mathbf {A}} = \operatorname {C o n v 2 D} (\mathbf {X} \otimes \mathbf {X}) \odot \mathbf {A} ^ {\prime}. \tag {24}
$$
The tensor  $\mathbf{A}'$  is a linear projection, via a linear embedding layer, of the edge feature matrix  $\mathcal{E}$  if it exists, where  $P'$  is projected to  $H$  dimension; or the padded adjacency matrix  $\tilde{\mathbf{A}}$  where  $P' = 1$ . Altogether, the resulting energy equation is
$$
E ^ {\mathrm {A T T}} = - \frac {1}{\beta} \sum_ {h} \sum_ {C} \log \left(\sum_ {B \neq C} \exp \left(\beta \sum_ {\alpha} K _ {\alpha h B} Q _ {\alpha h C} \odot \hat {A} _ {h C}\right)\right). \tag {25}
$$
Moreover, to only consider the energy of edges in  $G$ , we set non-connected entries in  $\hat{\mathbf{A}}$  to be  $-\infty$  such that the gradient of such entries would be zero following the dynamics. Note, if non-connected entries are not ignored, the gradient of such entries would be non-zero and hence, the resultant attention energy might violate some of the graph structural information.
Meanwhile, in this implementation, the overall model consists of  $S$  vertically stacked ET blocks, where each block shares the same number of  $T$  depth and has its own LayerNorm. Similarly, the token representation  $\mathbf{X}^{\ell ,t}$  at dynamic step  $t$  corresponding to a block  $\ell$  is layer-normalized,
$$
\mathbf {g} ^ {\ell , t} = \operatorname {L a y e r N o r m} \left(\mathbf {X} ^ {\ell , t}\right). \tag {26}
$$
Keep in mind,  $\mathbf{X} = \mathbf{X}^{\ell = 1}$ ,  $t = 1$  is the initial token representation.
Following the dynamic equations 6, we inject a small amount of noise  $\epsilon^{\ell}, t \in (0,1)$ , generated from a normal distribution using a standard deviation  $\sigma_{\epsilon}$  and zero mean, into the gradient of energy function to produce  $\mathbf{X}^{\ell, t+1}$ , the new token representation of block  $\ell$ . The premise of this noise injection is to 'robustify' the model and help it escape saddle points of the energy function.
$$
\mathbf {X} ^ {\ell , t + 1} = \mathbf {X} ^ {\ell , t} - \alpha \nabla_ {\mathbf {g}} E ^ {\ell , t} + \sqrt {\alpha} \epsilon^ {\ell , t}, \quad \epsilon^ {\ell , t} \sim \mathcal {N} \left(0, \sigma_ {\epsilon} ^ {2}\right). \tag {27}
$$
Once stability is reached in the retrieval dynamics of a block  $\ell$ , the final representation  $\mathbf{X}^{\ell, t = T}$  is then passed on to the next block  $\ell + 1$  and the whole process is repeated again. When the final token representation  $\mathbf{X}^{\ell = S, t = T}$  is computed by the last block  $S$ , the resultant CLS token  $\mathbf{x}_{\mathrm{CLS}}^{\ell = S, t = T} \in \mathbb{R}^{D'}$ , extracted from  $\mathbf{X}^{\ell = S, t = T}$ , is utilized as the predictor of the current graph  $G$ .
D.2 Experimental Evaluation
As mentioned prior, eight datasets from TUDataset [36] are used for experimentation. Based on the collected results in Table 2, we observed that the modified ET demonstrates the best performance across all datasets with the exception of MUTAG. Additionally, as a means to contrast ET with other graph transformers, ET is trained and evaluated with five datasets from GNNBenchmark [37]. Specifically, PATTERN and CLUSTER for node classification; CIFAR10 and MNIST for graph classification; and lastly, ZINC for graph-regression. We compare ET to the recently introduced graph transformers – namely, EGT [56], Graphormer [57], Exphormer [58], GT [59], and GAT [54]. The results are depicted in table 7.
Based on the collected results, ET outperforms its baselines on PATTERN and ZINC datasets, and has competitive performance on MNIST and CLUSTER datasets. Meanwhile, ET performs the worst on the CIFAR10 dataset, in which its performance is relatively close to GAT [54] and Graphformer [57]. The reason for such a bad performance can be attributed to the overfitting problem seen during the training of the model, which is also experienced by [56, 59]. Hence, while ET performs very well on TUDataset, further explorations are required to enable ET to work better on CIFAR10, MNIST, and CLUSTER datasets. However, ET has overall demonstrated great performance on node binary-classification and graph-regression tasks.
Table 7: Graph classification performance on five datasets of GNNBenchmark [37]. The reported mean and standard deviation are obtained from 100 runs of evaluation on the test set of each dataset. For baselines standard deviations are only included if they are available in prior work. If the entry is unavailable in prior literature it is denoted by  $\cdot$ ; best results are in bold. The performance difference between non-baseline approaches (including ours) and the baseline (specified by the gray cell in each column) is indicated by  $\nabla$  (drop in performance with respect to the baseline),  $\triangle$  (increase in performance with respect to the baseline), and  $\nabla$  (no change). Additionally,  $\downarrow$  indicates lower is better and  $\uparrow$  indicates higher is better.
Method	Dataset
CIFAR10 (Accuracy) ↑	MNIST (Accuracy) ↑	PATTERN (Accuracy) ↑	CLUSTER (Accuracy) ↑	ZINC (MAE) ↓
GT	-	-	84.808±0.068▼(2.013)	73.169±0.622▼(6.063)	0.226±0.014▼(0.112)
GAT	64.223±0.455▼(10.531)	95.535±0.205▼(2.879)	78.271±0.186▼(8.550)	70.587±0.447▼(8.645)	0.384±0.007▼(0.276)
EGT	68.702±0.409▼(6.052)	98.173±0.087▼(0.241)	86.821±0.020	79.232±0.348	0.108±0.009
Graphomer	65.978±0.579▼(8.776)	97.905±0.176▼(0.509)	86.650±0.033▼(0.171)	74.660±0.236▼(4.572)	0.122±0.006▼(0.014)
Exphomer	74.754±0.194	98.414±0.038	86.734±0.008▼(0.087)	78.220±0.045▼(1.012)	-
ET (Ours)	63.920±0.173▼(10.834)	97.011±0.045▼(1.403)	90.690±0.013▲(3.869)	77.169±0.044▼(2.063)	0.096±0.000▲(0.012)
D.3 Experimental Details
In the graph domain, it is common to concatenate all of the feature vectors of all graphs in a batch together. However, in order for ET to work, we form the batch dimension by separating the feature vectors of all graphs in a given batch and utilize the largest node count to pad all graphs such that they all share the same number of nodes. Additionally, we set a limit on the number of nodes equal to 500, to prevent out-of-memory error. Specifically, if a graph has a node count exceeding the limit, the number of utilized nodes is equal to the limit. Hence, a portion of the graph structural information is ignored as a result. However, it is worth mentioning such a graph is rare in the experimental datasets. Additionally, instead of ignoring the padded entries, they are altered to be sink nodes in a graph, such that they are connected to the actual nodes of the graph but not vice versa. We found this approach to be helpful when it comes to computing the positional embedding and training the model. Additionally, following [37, 56], the sign of the top  $k$  eigen-vectors is flipped randomly during training and fixed during evaluation. The whole experiment is implemented using JAX[49], Flax [50], Optax [60], and PyTorch Geometric [61] frameworks.
Table 8: The statistics and properties of the eight datasets of TUDataset (additional node attributes are indicated by  $+$  ’ if exist).
Dataset	Graphs	Avg. Nodes	Avg. Edges	Node Attr	Classes
MUTAG	188	17.93	19.79	7	2
ENZYMES	600	32.63	62.14	18 + 3	6
PROTEINS	1113	39.06	72.82	0 + 4	2
DD	1178	284.32	715.66	89	2
NCI1	4110	29.87	32.30	37	2
NCI109	4127	29.68	32.13	38	2
MUTAGENICITY	4337	30.32	30.77	14	2
FRANKENSTEIN	4337	16.90	17.88	780	2
For the eight datasets of TUDataset, we train all models for 300 epochs using AdamW [62] and Gradient Centralization [63]. The best model is selected based on its performance obtained from the 10-fold cross validation process delineated in [36]. Since the task is classification, all models are trained with the cross-entropy loss function and label smoothing [64], where the smoothing parameter is set to 0.05. Additionally, the cosine-annealing with warm-up learning rate scheduler [65] is utilized, where the initial and end learning rates are both set as  $5e - 6$  while the peak learning rate is 0.001. The number of warm-up steps is set to 50 epochs while the batch size is 32 for all
datasets. We report the average performance of 100 runs on the 10-fold cross validation process with random seeding. The hyperparameters of our model are tuned based on the performance of the cross validation, selecting within 100 epochs. The optimal hyper-parameters are reported in Table 9 and the statistics of the used datasets are reported in Table 8. We also include the number ofgpu-devices used for training ET.
Table 9: Hyperparameter and architecture choices for ET during TUDataset experiments.
Training	Architecture
batch_size	32	token_dim	128
epochs	300	num_heads	12
peak_lr	1e-3	head_dim	64
warmup_epochs	50	β	1/√64
initial and ending_lr	5e-6	train_betas	Yes
b1, b2 (ADAM)	0.9, 0.99	step size α	0.01
weight Decay	0.05	k eigenvalues	15
grad_clipping	None	noise σε	0.02
num. ofgpu devices	2	depth	1
block_size	4
kernel_size	[3, 3]
dilation_size	[1, 1]
hidden_dim HN	512
bias in HN	None
bias in ATT	None
bias in LNORM	Yes
num. of params per ET block	262,929
avg. total num. of params	1071294
Similarly, for the five datasets of GNNBenchmark, we train all models using using AdamW [62] and Gradient Centralization [63]. The best model is selected based on its performance obtained from the validation set across 10 runs. For the CIFAR10 and MNIST datasets, the additional node positional features are concatenated to the node features (i.e., pixel values coordinates). All models, with the exception of ZINC, CIFAR10, and MNIST models, are trained using the cross-entropy loss function with label smoothing, where the smoothing parameter is set to 0.05. The ZINC model is trained with the mean absolute error function. Meanwhile, as a means to reduce overfitting, for the CIFAR10 and MNIST models, the mean squared error loss function is utilized to reconstruct the edge feature matrix  $\mathcal{E}$  in addition to the cross entropy loss function. Specifically, the parameterized adjacency tensor  $\hat{\mathbf{A}}$  is projected to match  $P^{\prime}$  raw feature dimension of  $\mathcal{E}$ , where both tensors are as part of the mean squared error function. In addition, all models are trained with the cosine-annealing with warm-up learning rate scheduler, where the number of warm-up steps is 50. We report the average performance of 100 evaluation runs on the testing set with random seeding. The hyperparameters of each model are tuned based on the performance of cross validation, selecting within 50 epochs. The optimal hyper-parameters are reported in Tables 10 and 11 and the statistics of the used datasets are reported in [37].
E ET for heterogeneous Graph