SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we show the performance of our ET model in the heterogeneous graph case. For the heterogeneous graph case (where more than one type of edges exist in the graph), we first run our ET model for different subgraphs (corresponding to different types of edges), and then aggregate the final representations using two methods. We have tried max pooling and concatenation for the aggregation step. Max pooling means to pick the largest value across all the subgraph representations, and concatenation means to concatenate the representations obtained from different subgraphs. Table 12 shows the comparison between these two variants of our model and BWGNN (heterogeneous case). ET performs better than heterogeneous BWGNN on Amazon, but loses to heterogeneous BWGNN on Yelp. Interestingly, BWGNN in the heterogeneous setting performs worse than BWGNN in the homogeneous setting on Amazon.
Table 10: Hyperparameter choice for ET training on GNNBenchmark datasets.
Parameter	Dataset
CIFAR10	MNIST	PATTERN	CLUSTER	ZINC
batch_size	128	128	128	128	128
epochs	150	150	100	300	500
initial lr	1e-3	1e-3	1e-3	1e-3	5e-7
ending lr	5e-5	5e-5	5e-7	5e-7	5e-7
peak lr	1e-3	1e-3	1e-3	1e-3	1e-3
warmup_epochs	50	50	50	50	50
b1, b2 (ADAM)	(0.9, 0.99)	(0.9, 0.99)	(0.9, 0.99)	(0.9, 0.99)	(0.9, 0.99)
weight Decay	0.05	0.05	0.05	0.05	0.05
grad_clipping	None	None	None	None	None
num. ofgpu devices	4	4	1	1	8
Table 11: Architecture choice for ET on GNNBenchmark datasets.
Parameter	Dataset
CIFAR10	MNIST	PATTERN	CLUSTER	ZINC
token_dim	128	128	128	128	128
num_heads	12	12	12	12	12
head_dim	64	64	64	64	64
β	1/√64	1/√64	1/√64	1/√64	1/√64
train_betas	Yes	Yes	Yes	Yes	Yes
step size α	0.25	0.25	0.1	0.1	0.1
k eigenvalues	15	15	15	15	15
noise σε	0.02	0.02	0.005	0.005	0.005
depth	4	4	4	4	1
block_size	2	2	2	2	4
kernel_size	[3, 3]	[3, 3]	[3, 3]	[3, 3]	[3, 3]
dilation_size	[1, 1]	[1, 1]	[1, 1]	[1, 1]	[1, 1]
hidden_dim HN	512	512	512	512	512
bias in HN	None	None	None	None	None
bias in ATT	None	None	None	None	None
bias in LNORM	Yes	Yes	Yes	Yes	Yes
total num. of params	529224	528968	527936	528964	1052409
F Ablation Study for Attention and Hopfield Network Modules
As we described in the main text the the ET network consists of two modules processing the tokens in parallel: the attention module (ATT) and the Hopfield Network module (HN). The ATT module is responsible for routing the information between the tokens, while the HN module is responsible for reinforcing the token representation to be consistent with the general expectation about the particular data domain. It is interesting to explore the contribution that these two subnetworks produce on the task performed by the network. In this section we ablate the ET architecture by dropping each of the two subnetworks and measuring the impact of the ablation on the performance.
Table 12: ET for anomaly detection in heterogeneous graph setting. Best results are in bold.
Datasets	Split	MaxPool	Concatenation	BWGNN (Heterogenous)
Macro-FI	Yelp	1%	61.5±0.4	61.7±0.2	67.02
40%	70.7±0.6	71.1±0.1	76.96
Amazon	1%	88.3±1.6	87.4±1.4	83.83
40%	92.1±0.2	91.8±0.2	91.72
AUC	Yelp	1%	72.2±0.5	72.8±0.2	76.95
40%	84.3±0.4	85.1±0.1	90.54
Amazon	1%	90.8±1.4	90.6±1.0	86.59
40%	97.5±0.1	97.2±0.6	97.42
F.1 On Graphs
The results on graphs are reported in Table 13. From this table it is clear that most of the computation is performed by the ATT block on this task, which pools the information about other tokens to the token of interest. When ATT block is kept, but HN block is removed the network looses  $1\%$  or less relative to the full ET (occasional improvements of the ablated model compared to the full ET are within the statistical error bars). In contrast, removing ATT module and keeping only the HN, the ET network effectively turns into an MLP with shared weights that is recurrently applied. In this regime the network can only use the features of a given node for that node's anomalous status prediction. This results in a more significant drop in performance, which is about  $5\%$  on average.
Table 13: Ablation study with respect to ATT block and HN Block. Best results are in bold.
Datasets	Split	ATT✓|HNX	ATTX|HN✓	full model (Ours)
Macro-F1	Yelp	1%	62.5±0.3▼	57.4±0.5▼(-5.6)	63.0±0.6
40%	70.6±0.5▼	71.2±0.7▼	71.5±0.1
Amazon	1%	89.5±0.9▲	87.4±1.0▼	89.3±0.7
40%	91.7±0.5▼(-1.1)	88.7±0.3▼(-4.1)	92.8±0.3
T-Finance	1%	84.7±1.0▼	80.3±0.6▼(-4.8)	85.1±1.0
40%	87.4±0.7▼	82.3±0.8▼(-5.9)	88.2±1.0
T-Social	1%	79.8±0.6▲	72.7±1.0▼(-6.4)	79.1±0.7
40%	82.9±1.0▼	78.6±1.2▼(-4.9)	83.5±0.4
AUC	Yelp	1%	72.9±0.3▼	67.4±0.7▼(-5.8)	73.2±0.8
40%	83.5±0.4▼(-1.4)	83.1±0.6▼(-1.8)	84.9±0.3
Amazon	1%	90.7±0.8▼	89.8±1.2▼	91.9±1.0
40%	96.8±0.6▼	95.7±0.5▼(-1.6)	97.3±0.4
T-Finance	1%	91.7±1.2▼	90.2±0.8▼(-2.6)	92.8±1.1
40%	94.3±2.6▼	90.2±2.1▼	95.0±3.0
T-Social	1%	92.2±0.8▲	86.4±0.7▼(-5.5)	91.9±0.6
40%	93.1±0.8▼	88.3±1.3▼(-5.6)	93.9±0.2
F.2 On Images
The ablation results for image reconstruction are shown in Table 14. Each experiment was trained using the same hyperparameter settings as shown in Table 4. After training the model on IN1K, we calculate the average MSE on the reconstructed masked tokens for the validation set (using the same  $50\%$  masking ratio used for training) across 10 different random seeds for the masking.
We make several conclusions from these ablation studies.
- We gain several insights regarding the use of "self-attention" in our ET (when a token patch query is allowed to consider itself as a key in the attention weights). When both self-attention and HN are present (ET-Full+Self), there is no noticeable benefit over ET-Full for a token to attend to itself. In fact, preventing the ATTN energy module from attending to itself slightly improves the performance. However, when the HN is removed (ET-NoHN*), we notice that allowing self-attention (ET-NoHN+Self) outperforms the version that prevents self-attention (ET-NoHN).
- On its own, allowing self-attention (ET-NoHN+Self) in the ATTN module performs nearly as well as the full ET at a fraction of the total parameters. However, MSE is a forgiving metric for blurry reconstructions. While ATTN can capture the global structure of the image quite well, it does so at the expense of image sharpness (Figure 4).
- As expected, removal of the ATTN energy module performs the worst, because the HN operates tokenwise and has no way to aggregate token information across the global image without ATTN.
Table 14: Module ablation tests for image reconstruction task, reporting average IN1K validation MSE on masked tokens after 100 epochs. Reported number of parameters excludes the constant number of parameters in the affine encoder and decoder.
Model	Has ATTN?	Allow self-attn?	Has HN?	NParams (in ET block)	MSE
ET-Full (Ours)	✓	X	✓	3.7M	0.306±0.10
ET-Full+Self	✓	✓	✓	3.7M	0.312±0.10
ET-NoHN+Self	✓	✓	X	1.3M	0.343±0.10
ET-NoHN	✓	X	X	1.3M	0.403±0.11
ET-NoATT	X	X	✓	2.5M	0.825±0.20
Figure 8 shows our best performing model (ET-Full) on the qualitative image reconstructions corresponding to the largest errors across IN1K validation images, averaged across all masking seeds. Likewise, Figure 9 shows the lowest errors across IN1K validation images and masking seeds. In general, image reconstructions that require ET to produce sharp, high frequency, and high contrast lines negatively impact MSE performance.
Figure 8: Reconstruction examples of images with the worst MSE from the IN1k validation set. Top row: input images where  $50\%$  of the patches are masked with the learned MASK token. Middle row: all tokens reconstructed after 12 time steps. Bottom row: original images.
G Parameter comparison
The energy function enforces symmetries in our model, which means ET has fewer parameters than its ViT counterparts. In particular, ET has no "Value Matrix"  $\mathbf{W}^V$  in the attention mechanism, and the HN module has only one of the two matrices in the standard MLP of the traditional Transformer
Figure 9: Reconstruction examples of images with the best (lowest) MSE from the IN1k validation set. Top row: input images where  $50\%$  of the patches are masked with the learned MASK token. Middle row: all tokens reconstructed after 12 time steps. Bottom row: original images.
block. We report these differences in Table 15. We take the ET configuration used in this paper, which has an architecture fully comparable to the original ViT-base [18] with patch_size=16, and report the number of parameters against ViT-base and an "ALBERT" version of ViT [14] where a single ViT block is shared across layers. We saw no benefit when including biases in ET, so we also exclude the biases from the total parameter count in the configuration of ViT and ALBERT-ViT. We report both the total number of parameters and the number of parameters per Transformer block. Furthermore, as a means for a fairer comparison, we reduce the parameters of ViT-Base such that the parameter count is similar to that of ET. We contrast ET and the reduced ViT on the image reconstruction task using PSNR (Peak-Signal-to-Noise-Ratio) and SSIM (Structural-SIMilarity) [66] as the evaluation metrics of the reconstructions. The results are demonstrated in Table 16, where the performance of ET is very close to that of the reduced ViT, which has a slight advantage in parameter count.
Table 15: Comparison between the number of parameters in a standard ViT, an ALBERT version of ViT where standard Transformer blocks are shared across layers, and our ET. Comparison is done assuming no biases in any operation.
Model	NParams	NParams
(per block)
ViT-Base	86.28M	▼0.00%	7.08M	▼0.00%
ALBERT ViT-Base	8.41M	▼90.25%	7.08M	▼0.00%
ET	4.87M	▼94.36%	3.54M	▼50.02%
Table 16: Comparison between ET and 'comparable-size' ViT on image reconstruction task. Given ViT-Base, we reduce its parameter count down to a number similar to that of ET for the image domain. The metrics, PSNR (Peak-Signal-to-Noise-Ratio) and SSIM (Structural-SIMilarity), are recorded for the image reconstruction evaluations.
Model	NParams	PSNR	SSIM
ViT	5.52M	▼0.00%	22.11	▼0.00	0.715	▼0.00
ET	4.28M	▼22.46%	21.51	▼0.59	0.681	▼0.03
H Formal Algorithm for Training and Inference of ET