SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
E \left(s _ {e}\right) = \frac {1}{2} \left\| s _ {e} \right\| ^ {2} - B _ {e} \cdot s _ {e} + C _ {e} \tag {102}
$$
for some constants  $B_{e}$  and  $C_e$  that do not depend on  $s_e$ . Specifically
$$
B _ {k} := \mathcal {P} \left(w _ {k} \star s _ {k - 1} ^ {(t + 1)}\right) + w _ {k + 1} ^ {\top} \star \mathcal {P} ^ {- 1} \left(s _ {k + 1} ^ {(t + 1)}\right) + b _ {k} \tag {103}
$$
for every layer  $k$ . It is easily seen that the minimum of such a function  $E(s_{e})$  in  $\mathbb{R}^{\dim (s_2)} \times \mathbb{R}^{\dim (s_4)}$  is obtained at the point  $s_e = B_e$ . It is also easily seen that the minimum of  $E(s_{e})$  in  $[0,1]^{\dim (s_2)} \times [0,1]^{\dim (s_4)}$  is obtained at the point  $s_e = \max (0,\min (B_e,1))$ . Similar expressions hold for the configuration of odd layers  $s_o$  that minimizes the energy function given the state of even layers fixed. Given that  $\Phi$  is a linear function of  $s_o$ , we have that  $B_{o} = \frac{\partial\Phi}{\partial s_{o}} (\theta ,s)$ , and similarly  $B_{e} = \frac{\partial\Phi}{\partial s_{e}} (\theta ,s)$ . Thus, we conclude that the energy function decreases at each step of the asynchronous update scheme (100)-(101).
For the max pooling operation used in this work, the primitive function  $\Phi$  is not a linear function of  $s_e$  or  $s_o$ , and the above argument no longer holds.
C.1 13.5x simulation speedup compared to Laborieux et al. [2021]
We compare in simulations two settings:
1. In the first setting, we use the asynchronous update scheme with 60 iterations at inference, 20 iterations in the perturbed phases, and 16 bit precision. This is the setting of the simulations performed for the comparative study in the present work.
2. In the second setting, we use the synchronous update scheme with 250 iterations at inference, 30 iterations in the perturbed phase, and 32 bits precision. This is (up to minor differences) the setting of the simulations of Laborieux et al. [2021].
We perform the simulations using the C-EP algorithm on CIFAR-10, and we use the hyperparameters reported as 'SOTA results on CIFAR10' in Table 5. We perform three runs in the first setting, and one run in the second setting, and we report the results in Table 3. We see that the first setting is 13.5 times faster than the second setting, requiring only 3 hours and 18 minutes to complete on a single A-100 GPU (compared to the 44 hours 29 minutes required in the second setting). Furthermore, the first setting also performs marginally better in terms of training and test error rates than the second one.
Table 3: Comparison of the performance and the wall-clock-time between the simulation setting of Laborieux et al. [2021] and ours. The simulations of Laborieux et al. [2021] use the synchronous update scheme, in conjunction with 32 bits precision, 250 iterations at inference (free phase), and 30 iterations in the perturbed phase. Our asynchronous update scheme is used in conjunction with 16 bits precision, 60 iterations at inference and 15 iterations in the perturbed phase. Simulations are performed on CIFAR-10 using the C-EP algorithm for training. We perform 100 epochs and use the hyperparameters of the 'SOTA CIFAR-10 simulations' provided in Table 5. We report the test error rate (Test Error), the training error rate (Train Error) and the wall-clock time (WCT). WCT is reported as HH:MM.
Test Error	Train Error	WCT
Synchronous - 32 bits - 250 iterations (1 run)	10.85%	3.58%	44:29
Asynchronous - 16 bits - 60 iterations (3 runs)	10.40 ± 0.10%	3.48 ± 0.14%	03:18
D Simulation details
Datasets. We perform simulations on the MNIST, Fashion-MNIST, SVHN, CIFAR-10 and CIFAR-100 datasets.
The MNIST dataset is composed of images of handwritten digits [LeCun et al., 1998]. Each image  $x$  in the dataset is a  $28 \times 28$  gray-scaled image and comes with a label  $y \in \{0, 1, \dots, 9\}$  indicating the digit that the image represents. The dataset contains 60,000 training images and 10,000 test images.
The Fashion-MNIST dataset [Xiao et al., 2017] shares the same image size, data format and the same structure of training and testing splits as MNIST. It comprises a training set of 60,000 images and a test set of 10,000 images. Each example is a  $28 \times 28$  grayscale image from ten categories of fashion products.
The SVHN dataset [Netzer et al., 2011] contains color images of  $32 \times 32$  pixels, derived from house numbers in Google Street View images. Like in MNIST, each image comes with a label corresponding to a digit from 0 to 9. The number of images per class varies due to the natural distribution of digits in the real world. The dataset contains a training set with 73, 257 images, and a test set containing 26, 032 images. The dataset also contains an additional set of 531, 131 other (somewhat less difficult) images, which we do not use in our simulations.
The CIFAR-10 dataset [Krizhevsky et al., 2009] consists of 60,000 colour images of  $32 \times 32$  pixels. These images are split in 10 classes (each corresponding to an object or animal), with 6,000 images per class. The training set consists of 50,000 images and the test set of 10,000 images.
Similarly, the CIFAR-100 dataset [Krizhevsky et al., 2009] also comprises 60,000 color images with a resolution of  $32 \times 32$  pixels, featuring a diverse set of objects and animals. These images are categorized into 100 distinct classes, each containing 600 images. Like CIFAR-10, the dataset is divided into a training set with 50,000 images and a test set containing the remaining 10,000 images.
Data pre-processing and data augmentation. The data is normalized as in Table 4.
Since the deep convolutional Hopfield network used in our simulations takes as input a 32x32-pixel image, we augment each image of MNIST and Fashion-MNIST to a 32x32-pixel image by adding two pixels at the top, two pixels at the bottom, two pixels to the left and two pixels to the right.
On the training sets of Fashion-MNIST, CIFAR-10 and CIFAR-100, we also use random horizontal flipping. (Random flipping is not used on the training sets of MNIST and SVHN, and it is never used at test time either.)
Table 4: Data normalization. We normalize the input images using the recommended mean  $(\mu)$  and std  $(\sigma)$  values for each dataset. The MNIST and Fashion-MNIST images are gray-scale, i.e. they have a unique channel. The SVHN, CIFAR-10 and CIFAR-100 images are color images, i.e. they have three channels.
mean (μ)	std (σ)
MNIST	0.1307	0.3081
Fashion-MNIST	0.2860	0.3530
SVHN	(0.4377, 0.4438, 0.4728)	(0.1980, 0.2010, 0.1970)
CIFAR-10	(0.4914, 0.4822, 0.4465)	(3*0.2023, 3*0.1994, 3*0.2010)
CIFAR-100	(0.5071, 0.4867, 0.4408)	(0.2675, 0.2565, 0.2761)
Network architecture and hyperparameters. Table 5 contains the architectural details of the network, as well as the hyperparameters used to obtain the results presented in Table 1, Table 2 and Table 6. The hard-sigmoid activation function is defined as  $\sigma(h) := \max(0, \min(h, 1))$ .
Weight initialization. We initialize the weights of dense interactions and convolutional interactions according to
$$
w _ {i j} \sim \mathcal {U} (- c, + c), \quad c = \alpha \sqrt {\frac {1}{\mathrm {f a n} _ {\mathrm {i n}}}}, \tag {104}
$$
Table 5: The top table indicates the architecture of the deep convolutional Hopfield network (DCHN). We denote  $n_{\mathrm{in}}$  the number of input filters (either 1 or 3 depending on the dataset) and  $n_{\mathrm{out}}$  the number of output units (either 10 or 100 depending on the dataset). The bottom table indicates the hyper-parameters used for initializing and training the network. Ir means learning rate. The first two columns (Comparative study) provide the values used for the extensive comparison of learning algorithms (Table 1 and Table 6). The other columns (SOTA results) provide the values used to obtain the best performances on MNIST, CIFAR-10 and CIFAR-100, after 100 epochs and 300 epochs (Table 2)
Layer shapes	Weight shapes	Bias shapes	Activation functions
Layer 0 (inputs)	\(n_{\text{in}}\times32\times32\)
Layer 1	128×16×16	128×\(n_{\text{in}}\)\(×3×3\)	128	Hard-sigmoid
Layer 2	256×8×8	256×128×3×3	256	Hard-sigmoid
Layer 3	512×4×4	512×256×3×3	512	Hard-sigmoid
Layer 4	512×2×2	512×512×3×3	512	Hard-sigmoid
Layer 5 (outputs)	\(n_{\text{out}}\)	512×2×2×\(n_{\text{out}}\)	10	Identity
Comparative study	SOTA results (Table 2)
Table 1	Table 6	MNIST	CIFAR-10
nudging (\(\beta\))	0.25	0.25	0.1	0.25
num. iterations at inference (\(T\))	60	60	60	60
num. iterations at training (\(K\))	15	15	20	15
gain conv-weight 1 (\(α_1\))	0.5	0.7	0.5	0.4	0.5
gain conv-weight 2 (\(α_2\))	0.5	0.7	0.5	0.7	0.4
gain conv-weight 3 (\(α_3\))	0.5	0.7	0.5	0.6	0.5
gain conv-weight 4 (\(α_4\))	0.5	0.7	0.5	0.3	0.8
gain dense-weight 5 (\(α_5\))	0.5	0.7	0.5	0.4	0.5
lr conv-weight 1 & bias 1 (\(η_1\))	0.0625	0.0625	0.03	0.03
lr conv-weight 2 & bias 2 (\(η_2\))	0.0375	0.0375	0.03	0.04
lr conv-weight 3 & bias 3 (\(η_3\))	0.025	0.025	0.03	0.04
lr conv-weight 4 & bias 4 (\(η_4\))	0.02	0.02	0.03	0.04
lr dense-weight 5 & bias 5 (\(η_5\))	0.0125	0.0125	0.03	0.025
momentum	0.9	0.9	0.9	0.9
weight decay (1e-4)	3.0	3.0	2.5	3.5
mini-batch size	128	128	128	128
number of epochs	100	100	300	300
\(T_{\text{max}}\)	100	100	300	300
\(\eta_{\text{min}}\) (1e-6)	2.0	2.0	2.0	2.0
which is the 'Kaiming uniform' scheme rescaled by a factor  $\alpha$ , that we call the 'gain' here (i.e. a scaling number). For dense weights of shape ( $size_{pre}$ ,  $size_{post}$ ), we have  $fan_{in} = size_{pre}$ ; for convolutional weights of shape ( $channel_{in}$ ,  $channel_{out}$ , height, width), we have  $fan_{in} = channel_{in} \times height \times width$ . See Table 5 for the choice of the gains.
Energy minimization via asynchronous updates. To compute the steady state of the network, we use the 'asynchronous update' scheme described in appendix C: at every iteration, we first update the layers of even indices (the first half of the layers) and then we update the layers of odd indices (the other half of the layers). Relaxing all the layers once (first the even layers, then the odd layers) constitutes one 'iteration'. We repeat as many iterations as is necessary until convergence to the steady state. In Table 5, we denote  $T$  the number of iterations performed at inference (free phase), and we denote  $K$  the number of iterations performed in the perturbed phase.
Training procedure. We train our networks with seven energy-based learning (EBL) algorithms, namely: contrastive learning (CL), positively-perturbed equilibrium propagation (P-EP), negatively-perturbed EP (N-EP), centered EP (C-EP), positively-perturbed coupled learning (P-CpL), negatively-perturbed CpL (N-CpL) and centered CpL (C-CpL). Given an EBL algorithm, at each training step of SGD, we proceed as follows. First we pick a mini-batch of samples in the training set,  $x$ ,
and their corresponding labels,  $y$ . Then we set the nudging parameter to 0 and we perform a free phase of  $T$  asynchronous iterations. This phase allows us in particular to measure the training loss and training error rate for the current mini-batch, to monitor training. We also store the free state  $(h_{\star}, o_{\star})$ . Next, let's call  $\beta_{1}$  and  $\beta_{2}$  the two nudging values used for training (which depend on the EBL algorithm). First, we set the nudging parameter to  $\beta_{1}$  and we perform a new relaxation phase of  $K$  asynchronous iterations. Next, we reset the state of the network to the free state  $(h_{\star}, o_{\star})$ , then we set the nudging parameter to the second nudging value  $\beta_{2}$ , and we perform the last relaxation phase for  $K$  asynchronous iterations. Finally, we update all the parameters simultaneously in a single 'parameter update' phase.
Optimizer and scheduler. We use mini-batch gradient descent (SGD) with momentum and weight decay. We also use a cosine-annealing scheduler for the learning rates with hyperparameters  $T_{\mathrm{max}}$  and  $\eta_{\mathrm{min}}$ .
Computational resources. The code for the simulations uses PyTorch 1.13.1 and TorchVision 0.14.1. Paszke et al. [2017]. The simulations were carried on Nvidia A100 GPUs. For the comparative study, we used five GPUs (one GPU per dataset) and let them run for six days to complete all the simulations.
E Additional simulations on SVHN for the comparative study
The results of Table 1 show that most algorithms (P-EP, N-EP, P-CpL and N-CpL) perform very poorly on SVHN: the training process often gets stuck at  $81.08\%$  training error and  $80.41\%$  test error rate. These results seem to go against our conclusion that N-EP and N-CpL generally perform (much) better than C-EP and C-CpL and CL. We hypothesize that this poor performance on SVHN is due to the network being poorly initialized for this classification task, and we perform additional simulations on SVHN where we change the initialisation scheme of the weights of the network - see Table 5 for the new hyperparameters used for these additional experiments. As for the comparative study of Section 4.1, with this new set of hyperparameters, we compare the seven EBL algorithms of section 2 by training the DCHN of section 3 on SVHN. We also compare the performance of these algorithms to recurrent backpropagation (RBP) and truncated backpropagation (TBP). The results of these additional simulations are reported in Table 6.
Table 6 shows that the N-EP and P-CpL simulations have large variance, so we also report the results for each run. The three runs of N-EP gave the following results: 1)  $3.73\%$  test error, 2)  $3.87\%$  test error, and 3)  $80.41\%$  test error. The third N-EP run was indeed also blocked at  $81.08\%$  training error and  $80.41\%$  test error (like many other runs). The three runs of P-CpL gave the following results: 1)  $71.05\%$  test error, 2)  $15.35\%$  test error, and 3)  $15.57\%$  test error.
The results of Table 6 are consistent with our conclusions of Section 4.1: 1) strong positive perturbations (CL) yield better results than weak positive perturbations (P-CpL and P-EP), 2) negative perturbations (N-EP, N-CpL) yield better results than positive perturbations (CL, C-CpL, C-EP), 3) two-sided perturbations (C-EP, C-CpL) perform better than one-sided perturbations (N-EP, N-CpL, CL, P-CpL, P-EP), and 4) the EP perturbation technique yields better results than the CpL perturbation technique.
Table 6: Additional experiments on SVHN using a different set of hyperparameters. We compare the seven EBL algorithms and benchmark them against truncated backpropagation (TBP) and recurrent backpropagation (RBP). We report the training and test error rates, averaged over three runs. The hyperparameters used for this study are detailed in Table 5.
SVHN
Test Error (%)	Train Error (%)
TBP	3.66 ± 0.05	2.36 ± 0.04
RBP	3.91 ± 0.05	3.29 ± 0.04
CL	9.64 ± 1.59	27.08 ± 5.68
P-EP	74.88 ± 10.60	77.32 ± 5.84
N-EP	29.34 ± 36.12	29.01 ± 36.82
C-EP	3.58 ± 0.07	3.00 ± 0.05
P-CpL	33.99 ± 26.21	47.28 ± 26.52
N-CpL	4.53 ± 0.05	5.27 ± 0.05
C-CpL	4.10 ± 0.04	4.68 ± 0.10
In Table 7, we report the results of Table 1 together with the test std values.
Table 7: Results of Table 1 reported with the test std values.
MNIST	FashionMNIST	SVHN	CIFAR-10	CIFAR-100
Test	Train	Test	Train	Test	Train	Test	Train	Test	Train
TBP	0.42±0.02	0.23	6.12±0.10	3.09	3.76±0.06	2.37	10.1±0.2	3.1	33.4±0.5	17.2
RBP	0.44±0.04	0.33	6.28±0.12	3.70	3.87±0.09	3.43	10.7±0.1	5.2	34.4±0.2	18.2
CL	0.61±0.02	2.39	10.10±0.17	15.49	6.10±0.13	15.8	31.4±2.2	45.2	71.4±3.6	88.6
P-EP	1.66±0.80	2.29	90.00±0.00	89.98	83.9±4.9	81.9	72.6±0.7	79.5	89.4±9.5	95.5
N-EP	0.42±0.01	0.19	6.22±0.10	3.87	80.4±0.0	81.1	11.9±0.3	6.2	44.7±0.3	40.1
C-EP	0.44±0.03	0.20	6.47±0.13	4.01	3.51±0.07	3.01	11.1±0.1	5.6	37.0±0.1	26.0
P-CpL	0.66±0.13	0.59	64.70±35.77	65.31	40.1±28.6	50.8	46.9±3.8	57.7	77.9±0.5	91.0
N-CpL	0.50±0.04	0.88	6.86±0.05	6.27	80.4±0.0	81.1	13.5±0.3	10.2	51.9±1.3	50.6
C-CpL	0.44±0.01	0.38	6.91±0.11	5.29	4.23±0.09	5.05	14.9±0.1	14.6	46.5±0.1	37.9