SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\frac {\partial F}{\partial \beta} (\beta , \theta) = \frac {\partial \bar {F}}{\partial \beta} \left(\beta , \theta , h _ {\beta} ^ {\mathrm {E P}}, o _ {\beta} ^ {\mathrm {E P}}\right) + \underbrace {\frac {\partial \bar {F}}{\partial h} \left(\beta , \theta , h _ {\beta} ^ {\mathrm {E P}} , o _ {\beta} ^ {\mathrm {E P}}\right)} _ {= 0} \cdot \frac {\partial h _ {\beta} ^ {\mathrm {E P}}}{\partial \beta} + \underbrace {\frac {\partial \bar {F}}{\partial o} \left(\beta , \theta , h _ {\beta} ^ {\mathrm {E P}} , o _ {\beta} ^ {\mathrm {E P}}\right)} _ {= 0} \cdot \frac {\partial o _ {\beta} ^ {\mathrm {E P}}}{\partial \beta}. \tag {33}
$$
$$
\frac {\partial F}{\partial \beta} (\beta , \theta) = \frac {\partial \bar {F}}{\partial \beta} (\beta , \theta , h _ {\beta} ^ {\mathrm {E P}}, o _ {\beta} ^ {\mathrm {E P}}) = C (o _ {\beta} ^ {\mathrm {E P}}). \tag {34}
$$
Evaluating this expression at  $\beta = 0$ , and using  $o_0^{\mathrm{EP}} = o(\theta)$  by definition, we get
$$
\frac {\partial F}{\partial \beta} (0, \theta) = C (o (\theta)), \tag {35}
$$
which is the first statement of Theorem 4. Using a Taylor expansion of  $F(\beta, \theta)$  around  $\beta = 0$ , we have
$$
F (\beta , \theta) = F (0, \theta) + \beta C (o (\theta)) + O \left(\beta^ {2}\right). \tag {36}
$$
Subtracting  $F(\theta, 0)$  on both sides and dividing by  $\beta$ , we get
$$
\mathcal {L} _ {\beta} ^ {\mathrm {E P}} (\theta) = \frac {F (\beta , \theta) - F (0 , \theta)}{\beta} = C (o (\theta)) + O (\beta), \tag {37}
$$
which is the first part of equation (27). Similarly, we write Taylor expansions for  $F(\beta, \theta)$  and  $F(-\beta, \theta)$ ,
$$
F (\beta , \theta) = F (0, \theta) + \beta C (o (\theta)) + \frac {1}{2} \beta^ {2} \frac {\partial^ {2} F}{\partial \beta^ {2}} (0, \theta) + O \left(\beta^ {3}\right), \tag {38}
$$
$$
F (- \beta , \theta) = F (0, \theta) - \beta C (o (\theta)) + \frac {1}{2} \beta^ {2} \frac {\partial^ {2} F}{\partial \beta^ {2}} (0, \theta) + O \left(\beta^ {3}\right). \tag {39}
$$
Subtracting the second equation from the first one, and dividing by  $2\beta$ , we get
$$
\mathcal {L} _ {- \beta ; + \beta} ^ {\mathrm {E P}} (\theta) = \frac {F (\beta , \theta) - F (- \beta , \theta)}{2 \beta} = C (o (\theta)) + O \left(\beta^ {2}\right), \tag {40}
$$
which is the second part of equation (27).
Next, we prove the upper bound and lower bound properties. To this end, we write for any  $\beta \in \mathbb{R}$
$$
F (\beta , \theta) = \inf  _ {h, o} \left\{E (\theta , h, o) + \beta C (o) \right\} \tag {41}
$$
$$
\leq \inf  _ {h} \left\{E (\theta , h, o (\theta)) + \beta C (o (\theta)) \right\} = F (0, \theta) + \beta C (o (\theta)), \tag {42}
$$
where  $o(\theta)$  is the free state value of output variables. Subtracting  $F(\theta, 0)$  on both sides we get
$$
F (\beta , \theta) - F (\theta , 0) \leq \beta C (o (\theta)). \tag {43}
$$
Next we divide by  $\beta$ . For  $\beta > 0$ , we get
$$
\mathcal {L} _ {\beta} ^ {\mathrm {E P}} (\theta) = \frac {F (\beta , \theta) - F (\theta , 0)}{\beta} \leq C (o (\theta)) \tag {44}
$$
and for  $\beta < 0$  we get
$$
\mathcal {L} _ {\beta} ^ {\mathrm {E P}} (\theta) = \frac {F (\beta , \theta) - F (\theta , 0)}{\beta} \geq C (o (\theta)). \tag {45}
$$
Hence the result:
$$
\mathcal {L} _ {\beta} ^ {\mathrm {E P}} (\theta) \leq C (o (\theta)) \leq \mathcal {L} _ {- \beta} ^ {\mathrm {E P}} (\theta), \quad \forall \beta > 0. \tag {46}
$$
Similarly, it can be shown more generally that the function  $\beta \to \mathcal{L}_{\beta}^{\mathrm{EP}}(\theta)$  is non-increasing, or equivalently that the function  $\beta \mapsto F(\beta, \theta)$  is concave.
Proof of Theorem 5. First, using (21), we have
$$
\frac {\partial \mathcal {L} _ {\beta} ^ {\mathrm {E P}}}{\partial \theta} = \frac {1}{\beta} \left(\frac {\partial F}{\partial \theta} (\beta , \theta) - \frac {\partial F}{\partial \theta} (0, \theta)\right). \tag {47}
$$
Similar to the proof of Theorem 4, using the chain rule of differentiation, we have for every  $\beta \in \mathbb{R}$
$$
\frac {\partial F}{\partial \theta} (\beta , \theta) = \frac {\partial \bar {F}}{\partial \theta} \left(\beta , \theta , h _ {\beta} ^ {\mathrm {E P}}, o _ {\beta} ^ {\mathrm {E P}}\right) + \underbrace {\frac {\partial \bar {F}}{\partial h} \left(\beta , \theta , h _ {\beta} ^ {\mathrm {E P}} , o _ {\beta} ^ {\mathrm {E P}}\right)} _ {= 0} \cdot \frac {\partial h _ {\beta} ^ {\mathrm {E P}}}{\partial \theta} + \underbrace {\frac {\partial \bar {F}}{\partial o} \left(\beta , \theta , h _ {\beta} ^ {\mathrm {E P}} , o _ {\beta} ^ {\mathrm {E P}}\right)} _ {= 0} \cdot \frac {\partial o _ {\beta} ^ {\mathrm {E P}}}{\partial \theta}. \tag {48}
$$
Here again, we have used the first-order steady state conditions for  $h_{\beta}^{\mathrm{EP}}$  and  $o_{\beta}^{\mathrm{EP}}$ . Therefore
$$
\frac {\partial F}{\partial \theta} (\beta , \theta) = \frac {\partial \bar {F}}{\partial \theta} (\beta , \theta , h _ {\beta} ^ {\mathrm {E P}}, o _ {\beta} ^ {\mathrm {E P}}) = \frac {\partial E}{\partial \theta} (\theta , h _ {\beta} ^ {\mathrm {E P}}, o _ {\beta} ^ {\mathrm {E P}}). \tag {49}
$$
Injecting (49) in (47), we get
$$
\frac {\partial \mathcal {L} _ {\beta} ^ {\mathrm {E P}}}{\partial \theta} (\theta) = \frac {1}{\beta} \left(\frac {\partial E}{\partial \theta} \left(\theta , h _ {\beta} ^ {\mathrm {E P}}, o _ {\beta} ^ {\mathrm {E P}}\right) - \frac {\partial E}{\partial \theta} \left(\theta , h _ {0} ^ {\mathrm {E P}}, o _ {0} ^ {\mathrm {E P}}\right)\right). \tag {50}
$$
We conclude using the definitions of the equilibrium propagation learning rule (5). We prove similarly the case of centered EP with  $\frac{\partial\mathcal{L}_{-\beta; + \beta}^{\mathrm{EP}}}{\partial\theta} (\theta)$
B On the coupled learning rules
In this appendix, we prove some theoretical results related to the coupled learning rules.
Recall the definition of the free state
$$
\left(h _ {\star}, o _ {\star}\right) := \underset {(h, o)} {\arg \min } E (\theta , x, h, o), \tag {51}
$$
the definition of the perturbed state of coupled learning (CpL)
$$
h _ {\beta} ^ {\mathrm {C p L}} := \underset {h} {\arg \min } E (\theta , x, h, o _ {\beta} ^ {\mathrm {C p L}}), \quad o _ {\beta} ^ {\mathrm {C p L}} := (1 - \beta) o _ {\star} + \beta y, \tag {52}
$$
and the definition of the (positively-perturbed) CpL rule
$$
\Delta^ {\mathrm {C p L}} \theta := \frac {\eta}{\beta} \left(\frac {\partial E}{\partial \theta} (\theta , x, h _ {\star}, o _ {\star}) - \frac {\partial E}{\partial \theta} (\theta , x, h _ {\beta} ^ {\mathrm {C p L}}, o _ {\beta} ^ {\mathrm {C p L}})\right). \tag {53}
$$
Similar to the contrastive function  $\mathcal{L}_{\mathrm{CL}}$  of contrastive learning (10) and the surrogate loss  $\mathcal{L}_{\beta}^{\mathrm{EP}}$  of equilibrium propagation (21), Stern et al. [2022] define the contrast
$$
\mathcal {L} _ {\mathrm {C p L}} ^ {(2)} := \frac {1}{\beta} \left(E (\theta , x, h _ {\beta} ^ {\mathrm {C p L}}, o _ {\beta} ^ {\mathrm {C p L}}) - E (\theta , x, h _ {\star}, o _ {\star})\right) \tag {54}
$$
and argue that the CpL rule performs gradient descent on  $\mathcal{L}_{\mathrm{CpL}}^{(2)}$ . Besides, Stern et al. [2021] argue that the CpL rule optimizes both the function
$$
\mathcal {L} _ {\mathrm {C p L}} ^ {(1)} (\theta , x, y) := (y - o (\theta , x)) ^ {\top} \cdot \frac {\partial^ {2} G}{\partial o ^ {2}} (\theta , x, o (\theta , x)) \cdot (y - o (\theta , x)), \quad G (\theta , x, o) := \min  _ {h} E (\theta , x, h, o) \tag {55}
$$
and the squared error loss
$$
\mathcal {L} _ {\mathrm {M S E}} (\theta , x, y) := (o (\theta , x) - y) ^ {2}, \tag {56}
$$
where  $o(\theta ,x) = o_{\star}$  is the free output value.
In this appendix, we show that the CpL algorithm does not perform gradient descent on  $\mathcal{L}_{\mathrm{CpL}}^{(2)}$  (Appendix B.1). We also show that, in general, the CpL rule optimizes neither the squared error  $\mathcal{L}_{\mathrm{MSE}}$ , nor the function  $\mathcal{L}_{\mathrm{CpL}}^{(1)}$ . To this end, we present two examples: in the first one, the function  $\mathcal{L}_{\mathrm{MSE}}$  increases after a single step of the coupled learning rule (Appendix B.2); in the second one, the function  $\mathcal{L}_{\mathrm{CpL}}^{(1)}$  increases (Appendix B.3).
B.1 Coupled learning does not perform gradient descent on the contrast  $\mathcal{L}_{\mathrm{CpL}}^{(2)}$
Stern et al. [2022] note that  $\mathcal{L}_{\mathrm{CpL}}^{(2)}$  is non-negative and achieves its minimum possible value ( $\mathcal{L}_{\mathrm{CpL}}^{(2)} = 0$ ) when  $o_{\star} = y$ , which motivates them to minimize  $\mathcal{L}_{\mathrm{CpL}}^{(2)}$  by gradient descent. However, as we show here, the coupled learning rule does not perform gradient descent on  $\mathcal{L}_{\mathrm{CpL}}^{(2)}$ . Indeed, while the free state is characterized by
$$
\frac {\partial E}{\partial h} (\theta , x, h _ {\star}, o _ {\star}) = 0, \quad \frac {\partial E}{\partial o} (\theta , x, h _ {\star}, o _ {\star}) = 0, \tag {57}
$$
the perturbed state of CpL only satisfies
$$
\frac {\partial E}{\partial h} \left(\theta , x, h _ {\beta} ^ {\mathrm {C p L}}, o _ {\beta} ^ {\mathrm {C p L}}\right) = 0. \tag {58}
$$
The partial derivative  $\frac{\partial E}{\partial o}\left(\theta, x, h_{\beta}^{\mathrm{CpL}}, o_{\beta}^{\mathrm{CpL}}\right)$  does not vanish in general, in contrast with the perturbed state of equilibrium propagation (EP), which satisfies (31). Thus, we have
$$
\begin{array}{l} \frac {d}{d \theta} E (\theta , x, h _ {\star}, o _ {\star}) = \frac {\partial E}{\partial \theta} (\theta , x, h _ {\star}, o _ {\star}) (59) \\ + \underbrace {\frac {\partial E}{\partial h} (\theta , x , h _ {\star} , o _ {\star})} _ {= 0} \cdot \frac {\partial h _ {\star}}{\partial \theta} + \underbrace {\frac {\partial E}{\partial o} (\theta , x , h _ {\star} , o _ {\star})} _ {= 0} \cdot \frac {\partial o _ {\star}}{\partial \theta} (60) \\ = \frac {\partial E}{\partial \theta} (\theta , x, h _ {\star}, o _ {\star}), (61) \\ \end{array}
$$
$$
\begin{array}{l} \frac {d}{d \theta} E \left(\theta , x, h _ {\beta} ^ {\mathrm {C p L}}, o _ {\beta} ^ {\mathrm {C p L}}\right) = \frac {\partial E}{\partial \theta} \left(\theta , x, h _ {\beta} ^ {\mathrm {C p L}}, o _ {\beta} ^ {\mathrm {C p L}}\right) (62) \\ + \underbrace {\frac {\partial E}{\partial h} \left(\theta , x , h _ {\beta} ^ {\mathrm {C p L}} , o _ {\beta} ^ {\mathrm {C p L}}\right)} _ {= 0} \cdot \frac {\partial h _ {\beta} ^ {\mathrm {C p L}}}{\partial \theta} + \underbrace {\frac {\partial E}{\partial o} \left(\theta , x , h _ {\beta} ^ {\mathrm {C p L}} , o _ {\beta} ^ {\mathrm {C p L}}\right)} _ {\neq 0 \text {i n g e n e r a l}} \cdot \frac {\partial o _ {\beta} ^ {\mathrm {C p L}}}{\partial \theta} (63) \\ = \frac {\partial E}{\partial \theta} \left(\theta , x, h _ {\beta} ^ {\mathrm {C p L}}, o _ {\beta} ^ {\mathrm {C p L}}\right) + (1 - \beta) \underbrace {\frac {\partial E}{\partial o} \left(\theta , x , h _ {\beta} ^ {\mathrm {C p L}} , o _ {\beta} ^ {\mathrm {C p L}}\right)} _ {\neq 0 \text {i n g e n e r a l}} \cdot \frac {\partial o _ {\star}}{\partial \theta}. (64) \\ \end{array}
$$
As a result, the gradient of  $\mathcal{L}_{\mathrm{CpL}}^{(2)}$  is
$$
\begin{array}{l} \frac {\partial \mathcal {L} _ {\mathrm {C p L}} ^ {(2)}}{\partial \theta} = \frac {1}{\beta} \left(\frac {\partial E}{\partial \theta} \left(\theta , x, h _ {\beta} ^ {\mathrm {C p L}}, o _ {\beta} ^ {\mathrm {C p L}}\right) - \frac {\partial E}{\partial \theta} \left(\theta , x, h _ {\star}, o _ {\star}\right)\right) (65) \\ + \frac {1 - \beta}{\beta} \underbrace {\frac {\partial E}{\partial o} \left(\theta , x , h _ {\beta} ^ {\mathrm {C p L}} , o _ {\beta} ^ {\mathrm {C p L}}\right) \cdot \frac {\partial o _ {\star}}{\partial \theta}} _ {\neq 0 \text {i n g e n e r a l}}. (66) \\ \end{array}
$$
The last term does not vanish in general, unless  $\beta = 1$ . We note that the case  $\beta = 1$  corresponds to contrastive learning (CL, section 2.1); hence we have recovered Theorem 1, known since Movellan [1991].
Therefore, contrary to CL that performs gradient descent on  $\mathcal{L}_{\mathrm{CL}}$  and EP that performs gradient descent on  $\mathcal{L}_{\beta}^{\mathrm{EP}}$  for any value of  $\beta$ , the CpL rule does not perform gradient descent on  $\mathcal{L}_{\mathrm{CpL}}^{(2)}$  for  $\beta \neq 1$ .
B.2 One step of coupled learning may increase the squared error  $\mathcal{L}_{\mathrm{MSE}}$
Next, we show in an example that CpL does not optimize  $\mathcal{L}_{\mathrm{MSE}}$ . Let  $A$  be the  $2\times 2$  square matrix
$$
A := \left( \begin{array}{c c} 1 & - 1 \\ - 1 & 2 \end{array} \right) \tag {67}
$$
and  $b:\mathbb{R}^2\to \mathbb{R}^2$  the function defined by
$$
b \left(\theta_ {1}, \theta_ {2}\right) := \left( \begin{array}{c} 1 + \theta_ {2} \\ \theta_ {1} + 2 \theta_ {2} \end{array} \right), \quad \theta_ {1}, \theta_ {2} \in \mathbb {R}. \tag {68}
$$
We define the energy function  $E:\mathbb{R}^2\times \mathbb{R}^2\to \mathbb{R}$  by
$$
\forall \theta \in \mathbb {R} ^ {2}, \forall o \in \mathbb {R} ^ {2}, E (\theta , o) := \frac {1}{2} (o - b (\theta)) ^ {\top} A (o - b (\theta)). \tag {69}
$$
For simplicity, there is no input  $(x)$  and no hidden variable  $(h)$  in our example. Since the matrix  $A$  is symmetric positive definite, the minimum of  $o\mapsto E(\theta ,o)$  is achieved at the point  $o_{\star} = b(\theta)$ , where  $E(\theta ,o_{\star}) = 0$ . Next, let's assume that the 'desired output' is  $y\coloneqq (0,0)\in \mathbb{R}^2$ . The perturbed state of the CpL algorithm is
$$
o _ {\beta} ^ {\mathrm {C p L}} := (1 - \beta) o _ {\star} + \beta y = (1 - \beta) b (\theta), \tag {70}
$$
where  $\beta \in \mathbb{R}$  is the nudging parameter, and the P-CpL rule is
$$
\Delta^ {\mathrm {C p L}} \theta := \frac {\eta}{\beta} \left(\frac {\partial E}{\partial \theta} \left(\theta , o _ {\star}\right) - \frac {\partial E}{\partial \theta} \left(\theta , o _ {\beta} ^ {\mathrm {C p L}}\right)\right), \tag {71}
$$
where  $\eta$  is the learning rate. Using (69) we calculate
$$
\frac {\partial E}{\partial \theta} (\theta , o) = b ^ {\prime} (\theta) ^ {\top} A (o - b (\theta)). \tag {72}
$$
$$
\frac {\partial E}{\partial \theta} (\theta , o _ {\star}) = 0 \tag {73}
$$
$$
\frac {\partial E}{\partial \theta} (\theta , o _ {\beta} ^ {\mathrm {C p L}}) = b ^ {\prime} (\theta) ^ {\top} A \left(o _ {\beta} ^ {\mathrm {C p L}} - b (\theta)\right) = - \beta b ^ {\prime} (\theta) ^ {\top} A b (\theta), \tag {74}
$$
where we have used (70). So the P-CpL rule (71) rewrites
$$
\Delta^ {\mathrm {C p L}} \theta = - \eta b ^ {\prime} (\theta) ^ {\top} A b (\theta). \tag {75}
$$
Next, we define the squared error between the prediction  $o_{\star}$  and the 'desired output'  $y = (0,0)$ ,
$$
\mathcal {L} _ {\mathrm {M S E}} (\theta) := \| o _ {\star} - y \| ^ {2} = \| b (\theta) \| ^ {2}. \tag {76}
$$
To conclude, we show that for  $\eta$  small enough,  $\mathcal{L}_{\mathrm{MSE}}(\theta_0 + \Delta^{\mathrm{CpL}}\theta) > \mathcal{L}_{\mathrm{MSE}}(\theta_0)$  at the point  $\theta_0 = (0,0)$  independently of the value of the nudging parameter  $\beta \in \mathbb{R}$ . To show this, first we calculate  $\frac{\partial\mathcal{L}_{\mathrm{MSE}}}{\partial\theta}\cdot \Delta^{\mathrm{CpL}}\theta$ . We have
$$
\frac {\partial \mathcal {L} _ {\mathrm {M S E}}}{\partial \theta} = b (\theta) ^ {\top} b ^ {\prime} (\theta), \tag {77}
$$
$$
\frac {\partial \mathcal {L} _ {\mathrm {M S E}}}{\partial \theta} \cdot \Delta^ {\mathrm {C p L}} \theta = - \eta b (\theta) ^ {\top} b ^ {\prime} (\theta) b ^ {\prime} (\theta) ^ {\top} A b (\theta). \tag {78}
$$
Let us calculate:
$$
b ^ {\prime} (\theta) = \left( \begin{array}{c c} 0 & 1 \\ 1 & 2 \end{array} \right), \tag {79}
$$
$$
\begin{array}{l} b ^ {\prime} (\theta) b ^ {\prime} (\theta) ^ {\top} A = \left( \begin{array}{l l} 0 & 1 \\ 1 & 2 \end{array} \right) \left( \begin{array}{l l} 0 & 1 \\ 1 & 2 \end{array} \right) \left( \begin{array}{c c} 1 & - 1 \\ - 1 & 2 \end{array} \right) (80) \\ = \left( \begin{array}{l l} 1 & 2 \\ 2 & 5 \end{array} \right) \cdot \left( \begin{array}{c c} 1 & - 1 \\ - 1 & 2 \end{array} \right) (81) \\ = \left( \begin{array}{l l} - 1 & 3 \\ - 3 & 8 \end{array} \right). (82) \\ \end{array}
$$
Since at the point  $\theta_0 = (0,0)$  we have
$$
b \left(\theta_ {0}\right) = \left( \begin{array}{c} 1 \\ 0 \end{array} \right), \tag {83}
$$
it follows that
$$
b \left(\theta_ {0}\right) ^ {\top} b ^ {\prime} \left(\theta_ {0}\right) b ^ {\prime} \left(\theta_ {0}\right) ^ {\top} A b \left(\theta_ {0}\right) = - 1. \tag {84}
$$
Therefore, at the point  $\theta_0 = (0,0)$
$$
\frac {\partial \mathcal {L} _ {\mathrm {M S E}}}{\partial \theta} \cdot \Delta^ {\mathrm {C p L}} \theta = \eta . \tag {85}
$$
$$
\mathcal {L} _ {\mathrm {M S E}} \left(\theta_ {0} + \Delta^ {\mathrm {C p L}} \theta\right) - \mathcal {L} _ {\mathrm {M S E}} \left(\theta_ {0}\right) = \frac {\partial \mathcal {L} _ {\mathrm {M S E}}}{\partial \theta} \cdot \Delta^ {\mathrm {C p L}} \theta + O \left(\left\| \Delta^ {\mathrm {C p L}} \theta \right\| ^ {2}\right) = \eta + O \left(\eta^ {2}\right). \tag {86}
$$
Thus, provided that the learning rate  $\eta > 0$  is small enough, the loss value  $\mathcal{L}_{\mathrm{MSE}}(\theta_0 + \Delta \theta)$  after one step of coupled learning is larger than  $\mathcal{L}_{\mathrm{MSE}}(\theta_0)$ . This holds for any nudging value  $\beta \in \mathbb{R}$ .
<sup>8</sup>We note that the learning rules of N-CpL and C-CpL are identical in this example.
B.3 One step of coupled learning may increase  $\mathcal{L}_{\mathrm{CpL}}^{(1)}$
Finally, we show that  $\mathrm{CpL}$  does not optimize  $\mathcal{L}_{\mathrm{CpL}}^{(1)}$ . To this end, we consider a second example, where we define the energy function  $E:(0,5 / 4)\times \mathbb{R}\to \mathbb{R}$  by
$$
\forall \theta \in (0, 5 / 4), \quad \forall o \in \mathbb {R}, \quad E (\theta , o) := \frac {1}{2} (5 - 4 \theta) (o - \theta) ^ {2}. \tag {87}
$$
The minimum of  $o \mapsto E(\theta, o)$  is obtained at the point  $o_{\star} = \theta$ , where  $E(\theta, o_{\star}) = 0$ . As in the previous example, we assume that the 'desired output' is  $y \coloneqq 0$ , so the perturbed state of CpL is
$$
o _ {\beta} ^ {\mathrm {C p L}} = (1 - \beta) o _ {\star} + \beta y = (1 - \beta) \theta \tag {88}
$$
and the CpL rule is
$$
\Delta^ {\mathrm {C p L}} \theta = \frac {\eta}{\beta} \left(\frac {\partial E}{\partial \theta} \left(\theta , o _ {\star}\right) - \frac {\partial E}{\partial \theta} \left(\theta , o _ {\mathrm {C L}} ^ {\beta}\right)\right). \tag {89}
$$
We calculate
$$
\frac {\partial E}{\partial \theta} (\theta , o) = (5 - 2 \theta - 2 o) (o - \theta). \tag {90}
$$
$$
\frac {\partial E}{\partial \theta} (\theta , o _ {\star}) = 0 \tag {91}
$$
$$
\frac {\partial E}{\partial \theta} \left(\theta , o _ {\beta} ^ {\mathrm {C p L}}\right) = \left(5 - 2 \theta - 2 o _ {\beta} ^ {\mathrm {C p L}}\right) \left(o _ {\beta} ^ {\mathrm {C p L}} - \theta\right) = \left(5 - 4 \theta + 2 \beta \theta\right) \beta \theta , \tag {92}
$$
so the CpL rule reads
$$
\Delta^ {\mathrm {C p L}} \theta = - \eta (5 - 4 \theta + 2 \beta \theta) \theta . \tag {93}
$$
Now, recall the definition of the function  $\mathcal{L}_{\mathrm{CpL}}^{(1)}$  of (16). In this example, it is equal to
$$
\mathcal {L} _ {\mathrm {C p L}} ^ {(1)} (\theta) := o _ {\star} \cdot \frac {\partial^ {2} E}{\partial o ^ {2}} (\theta , o _ {\star}) \cdot o _ {\star} = (5 - 4 \theta) \theta^ {2}, \tag {94}
$$
and its derivative (or 'gradient') is
$$
\left(\mathcal {L} _ {\mathrm {C p L}} ^ {(1)}\right) ^ {\prime} (\theta) = 1 0 \theta - 1 2 \theta^ {2}. \tag {95}
$$
At the point  $\theta_0 = 1$  we have  $(\mathcal{L}_{\mathrm{CpL}}^{(1)})'(\theta_0) = -2$ , and  $\Delta^{\mathrm{CpL}}\theta = -\eta (1 + 2\beta)$ . This means that  $(\mathcal{L}_{\mathrm{CpL}}^{(1)})'(\theta_0) < 0$ , and  $\Delta^{\mathrm{CpL}}\theta < 0$  for any nudging value  $\beta > -1/2$ . Thus, for a small  $\eta$  and any  $\beta > -1/2$ , the function  $\mathcal{L}_{\mathrm{CpL}}^{(1)}$  increases after a single step of the CpL rule.
C Energy minimization procedure for deep convolutional Hopfield networks via asynchronous updates
Prior works on deep convolutional Hopfield networks [Ernoult et al., 2019, Laborieux et al., 2021, Laydevant et al., 2021, Laborieux and Zenke, 2022] minimized the energy function (18) using a 'synchronous update' scheme, where at every iteration, all the layers are updated synchronously. In this work, we use an 'asynchronous update' scheme.
Recall the form of the energy function of a deep convolutional Hopfield network (DCHN), given by (18). Ernoult et al. [2019] write the energy function in the form
$$
E (\theta , s) = \frac {1}{2} \| s \| ^ {2} - \Phi (\theta , s), \tag {96}
$$
where  $\| s\| ^2 \coloneqq \sum_{k = 1}^{5}\| s_k\| ^2$  and  $\Phi$  is the so-called 'primitive function', defined as
$$
\Phi (\theta , s) := \sum_ {k = 1} ^ {4} s _ {k} \bullet \mathcal {P} \left(w _ {k} \star s _ {k - 1}\right) + s _ {5} ^ {\top} w _ {5} s _ {4} + \sum_ {k = 1} ^ {5} b _ {k} ^ {\top} s _ {k}. \tag {97}
$$
In this expression,  $s = (s_0, s_1, s_2, s_3, s_4, s_5)$  is the state of the network,  $x = s_0$  is the input layer,  $h = (s_1, s_2, s_3, s_4)$  is the hidden variable,  $o = s_5$  is the output variable,  $w_k$  and  $b_k$  are the kernel (the weights) and bias of layer  $k$ ,  $\star$  is the convolution operation,  $\mathcal{P}$  is the max pooling operation,  $\bullet$  is the scalar product for pairs of tensors, and  $\theta = \{w_k, b_k \mid 1 \leq k \leq 5\}$  is the set of model parameters. Arnoult et al. [2019] define the dynamics
$$
s ^ {(t + 1)} = \sigma \left(\frac {\partial \Phi}{\partial s} (\theta , s ^ {(t)})\right), \tag {98}
$$
where  $\sigma (\cdot) = \max (0,\min (\cdot ,1))$  is the 'hard sigmoid function', which they assume to converge to a fixed point  $s_\star$  characterized by
$$
s _ {\star} = \sigma \left(\frac {\partial \Phi}{\partial s} (\theta , s _ {\star})\right). \tag {99}
$$
Importantly, the fixed point  $s_\star$  is also a critical point of the energy function,  $s \mapsto E(\theta, s)$ , since  $\frac{\partial E}{\partial s}(\theta, s_\star) = s_\star - \frac{\partial \Phi}{\partial s}(\theta, s_\star) = 0$ . This dynamics is what we call here 'synchronous update' scheme, because at every iteration (or 'time step')  $t$ , all the layers  $s_k$  ( $1 \leq k \leq 5$ ) are updated synchronously.
In this work, we use an 'asynchronous update scheme', where at each iteration, we first update the layers of even indices, and then we update the layers of odd indices. Denoting  $s_{o} = (s_{1}, s_{3}, s_{5})$  the configuration of odd layers, and  $s_{e} = (s_{2}, s_{4})$  the configuration of even layers, the asynchronous update scheme reads
$$
s _ {o} ^ {(t + 1)} = \sigma_ {o} \left(\frac {\partial \Phi}{\partial s _ {o}} \left(\theta , s _ {o} ^ {(t)}, s _ {e} ^ {(t)}\right)\right), \tag {100}
$$
where  $\sigma_{o}$  is the activation function of odd layers, and
$$
s _ {e} ^ {(t + 1)} = \sigma_ {e} \left(\frac {\partial \Phi}{\partial s _ {e}} \left(\theta , s _ {o} ^ {(t + 1)}, s _ {e} ^ {(t)}\right)\right), \tag {101}
$$
where  $\sigma_{e}$  is the activation function of even layers. For the layers of index  $k$  such that  $1\leq k\leq 4$  the activation function is  $\sigma_k(\cdot) = \max (0,\min (\cdot ,1))$ , i.e. the 'hard sigmoid function', and  $\sigma_5$  is the identity (the 'linear activation function').
In this work, we use the max pooling operation  $\mathcal{P}_{\mathrm{max}}$ . Although for this choice of pooling operation we do not have a proof of convergence of either of the two schemes ('synchronous' and 'asynchronous'), we find experimentally that the asynchronous scheme converges faster. This allows us to reduce the number of iterations in the free and perturbed phases and thus get a significant speedup, as explained in the next subsection.
In the case where  $\mathcal{P}$  is the average (mean) pooling operation, however, we can prove that the energy decreases at each step of the asynchronous update scheme. Indeed, for the average pooling operation  $\mathcal{P}_{\mathrm{avg}}$ , the primitive  $\Phi$  is a linear function of  $s_e$ , i.e. the energy  $E$  is of the form