SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We first introduce the classic Hopfield Network [23]. Let  $N$  be the number of neurons in the network and  $\mathbf{S}(t) \in \{-1, +1\}^N$  be the state of the network at time  $t$ . The task is to store  $P$  patterns,  $\{\xi^1, \dots, \xi^\mu\}$ , where  $\xi_j^\mu \in \{\pm 1\}$  is the  $j^{th}$  neuron of the  $\mu^{th}$  pattern. The goal is to design a network with dynamics such that when the network is initialized with a pattern, it will converge to one of the stored memories.
The Hopfield Network [23] attempts this by following the discrete-time synchronous update rule:
$$
\mathbf {S} (t + 1) = \mathbf {T} _ {H N} (\mathbf {S} (t)), \tag {A.1}
$$
where the transition operator  $T_{HN}(\cdot)_i$  for neuron  $i$  is defined in terms of symmetric Hebbian weights:
$$
T _ {H N} (\mathbf {S}) _ {i} = \operatorname {s g n} \left[ \sum_ {j \neq i} J _ {i j} S _ {j} \right], \quad J _ {i j} = \frac {1}{N} \sum_ {\mu = 1} ^ {P} \xi_ {i} ^ {\mu} \xi_ {j} ^ {\mu}. \tag {A.2}
$$
Note that we are excluding self-interaction terms  $(J_{ii})$  in Equation A.2. To interpret this dynamics from another useful point of view, we define the overlap, or Mattis magnetization,  $m_i^\mu$  of the network state  $\mathbf{S}$  with pattern  $\xi^\mu$ . We can then rewrite the update rule for the Hopfield Network as
$$
T _ {H N} (\mathbf {S}) _ {i} := \operatorname {s g n} \left[ \sum_ {\mu = 1} ^ {P} \xi_ {i} ^ {\mu} m _ {i} ^ {\mu} \right], \quad m _ {i} ^ {\mu} := \frac {1}{(N - 1)} \sum_ {j \neq i} \xi_ {j} ^ {\mu} S _ {j} \tag {A.3}
$$
We interpret this as at every time  $t$ , the network tries to identify the pattern  $\xi^{\mu}$  it is closest to and updates neuron  $i$  to the value for that pattern. A natural question to ask about the associative memory networks is their capacity: how many patterns can be stored and recalled with minimal error? This question has been the subject of many studies [23, 27-29, 33-36]. Intuitively, in recalling a pattern  $\xi^{\nu}$ , what limits the network's capacity is the overlap between the pattern  $\xi^{\nu}$  and other patterns, referred to as the crosstalk [26, 36].
A precise answer to the storage capacity question can be given under the assumption that the patterns  $\{\xi^{\mu}\}$  are sampled from some probability distribution. While different notions of capacity have been considered in the literature [23, 27-29, 33-36], we focus on the fixed-point capacity, which characterizes the probability that, when initialized at a given pattern, the network dynamics do not move the state away from that point. To render the problem analytically tractable, it is usually assumed that the pattern components are i.i.d. Rademacher random variables, i.e.,  $\mathbb{P}(\xi_j^\mu = \pm 1) = 1 / 2$  for all  $j$  and  $\mu$ . Then, at finite network size one can define the capacity as
$$
P _ {H N} (N, c) = \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: \mathbb {P} \left[ \cap_ {\mu = 1} ^ {P} \left\{\mathbf {T} _ {H N} \left(\boldsymbol {\xi} ^ {\mu}\right) = \boldsymbol {\xi} ^ {\mu} \right\} \right] \geq 1 - c \right\}, \tag {A.4}
$$
where  $c \in [0,1)$  is a fixed error tolerance. As we review in detail in Appendix B, one finds an asymptotic capacity estimate  $P_{HN} \sim \frac{N}{4\log(N)}$  for  $c = 0$ , which can be shown to be a sharp threshold [33-35].
A.2 Modern Hopfield Networks
Recent work from Krotov and Hopfield [30, 66] reinvigorated a line of research into generalized Hopfield Networks with larger capacity [67-72], resulting in what are now called Dense Associative Memories or Modern Hopfield Networks:
$$
T _ {M H N} (\mathbf {S}) _ {i} := \operatorname {s g n} \left[ \sum_ {\mu = 1} ^ {P} \xi_ {i} ^ {\mu} f \left(m _ {i} ^ {\mu}\right) \right] \tag {A.5}
$$
For the Hopfield network, one can also consider an asynchronous update rule in which only one neuron is updated at each timestep [23, 26].
where  $f$ , referred to as the interaction function, is a nonlinear monotonically increasing function whose purpose is to separate the pattern overlaps for better signal to noise ratio. Since  $m_i^\mu (t)$  has a maximum value of 1, this means contributions from patterns with partial overlaps will be reduced by the interaction function. This diminishes the crosstalk and thereby increases the probability of transitioning to the correct pattern. If the interaction function is chosen to be  $f(x) = x^{d}$ , then the MHN's capacity has been shown to scale polynomially with network size as  $P\sim \beta_d\frac{N^d}{\log(N)}$ , where  $\beta_{d}$  is a numerical constant depending on the degree  $d$  [30, 73-75]. Using a different definition of capacity, Demircigil et al. [31] have also shown that an exponential nonlinearity can lead to exponential scaling of the capacity. See [32] for a recent review of these results.
B Review of Hopfield network fixed-point capacity
In this Appendix, we review the computation of the classical Hopfield network fixed-point capacity. Our approach will follow—but not exactly match—that of Petritis [33]. Though these results are standard, we review them in detail both because this approach will inspire in part our approach to the DenseNet, and because several important steps of the analysis are significantly simpler than the corresponding steps for the DenseNet.
We begin by recalling that the Hopfield network update can be written as
$$
T _ {H N} (\mathbf {S}) _ {i} := \operatorname {s g n} \left[ \sum_ {\mu = 1} ^ {P} \xi_ {i} ^ {\mu} \left(\frac {1}{N - 1} \sum_ {j \neq i} \xi_ {j} ^ {\mu} S _ {j}\right) \right], \tag {B.1}
$$
and that our goal is to determine
$$
P _ {H N} (N, c) = \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: \mathbb {P} \left[ \cap_ {\mu = 1} ^ {P} \left\{\mathbf {T} _ {H N} \left(\boldsymbol {\xi} ^ {\mu}\right) = \boldsymbol {\xi} ^ {\mu} \right\} \right] \geq 1 - c \right\} \tag {B.2}
$$
for some absolute constant  $0 \leq c < 1$ , at least in the regime where  $N, P \gg 1$  [33-36]. As is standard in theoretical studies of Hopfield model capacity [26-29, 33-36], we take in these probabilities the pattern components  $\xi_k^\mu$  to be independent and identically distributed Rademacher random variables. We can expand the memorization probability as a union of single-bitflip events:
$$
\mathbb {P} \left[ \bigcap_ {\mu = 1} ^ {P} \left\{\mathbf {T} _ {H N} \left(\boldsymbol {\xi} ^ {\mu}\right) = \boldsymbol {\xi} ^ {\mu} \right\} \right] = 1 - \mathbb {P} \left[ \bigcup_ {\mu = 1} ^ {P} \bigcup_ {i = 1} ^ {N} \left\{T _ {H N} \left(\boldsymbol {\xi} ^ {\mu}\right) _ {i} \neq \xi_ {i} ^ {\mu} \right\} \right]. \tag {B.3}
$$
This illustrates why analyzing the memorization probability is complicated: the single-pattern events  $\mathbf{T}_{HN}(\pmb{\xi}^{\mu}) = \pmb{\xi}^{\mu}$  are not independent across patterns  $\mu$ , and each single-pattern event is itself the intersection of non-independent single-neuron events  $T_{HN}(\pmb{\xi}^{\mu})_i = \xi_i^{\mu}$ . However, as the single-bitflip probabilities  $\mathbb{P}[T_{HN}(\pmb{\xi}^{\mu})_j \neq \xi_j^{\mu}]$  are identical for all  $\mu$  and  $j$ , we can obtain a straightforward union bound
$$
\begin{array}{l} \mathbb {P} \left[ \bigcap_ {\mu = 1} ^ {P} \left\{\mathbf {T} _ {H N} \left(\boldsymbol {\xi} ^ {\mu}\right) = \boldsymbol {\xi} ^ {\mu} \right\} \right] = 1 - \mathbb {P} \left[ \bigcup_ {\mu = 1} ^ {P} \bigcup_ {i = 1} ^ {N} \left\{T _ {H N} \left(\boldsymbol {\xi} ^ {\mu}\right) _ {i} \neq \xi_ {i} ^ {\mu} \right\} \right] (B.4) \\ \geq 1 - \sum_ {\mu = 1} ^ {P} \sum_ {i = 1} ^ {N} \mathbb {P} \left[ T _ {H N} \left(\boldsymbol {\xi} ^ {\mu}\right) _ {i} \neq \xi_ {i} ^ {\mu} \right] (B.5) \\ = 1 - N P \mathbb {P} \left[ T _ {H N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {1} \right], (B.6) \\ \end{array}
$$
where we focus without loss of generality on the first element of the first pattern. Therefore, if we can control the single-bitflip probability  $\mathbb{P}[T_{HN}(\pmb{\xi}^1)_1 \neq \xi_1^1]$ , we can obtain a lower bound on the true capacity. In particular,
$$
P _ {H N} (N, c) \geq \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: N P \mathbb {P} \left[ T _ {H N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {1} \right] \leq c \right\} \tag {B.7}
$$
From the definition of the Hopfield network update rule, we have
$$
\begin{array}{l} \mathbb {P} \left[ T _ {H N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {1} \right] = \mathbb {P} \left\{\operatorname {s g n} \left[ \frac {1}{N - 1} \sum_ {\mu = 1} ^ {P} \sum_ {j \neq i} \xi_ {1} ^ {\mu} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1} \right] \neq \xi_ {1} ^ {1} \right\} (B.8) \\ = \mathbb {P} \left[ \frac {1}{N - 1} \sum_ {\mu = 1} ^ {P} \sum_ {j \neq i} \xi_ {1} ^ {1} \xi_ {1} ^ {\mu} \xi_ {j} ^ {1} \xi_ {j} ^ {\mu} <   0 \right] (B.9) \\ = \mathbb {P} [ C > 1 ], (B.10) \\ \end{array}
$$
$$
C = \frac {1}{N - 1} \sum_ {\mu = 2} ^ {P} \sum_ {j \neq i} \xi_ {1} ^ {1} \xi_ {1} ^ {\mu} \xi_ {j} ^ {1} \xi_ {j} ^ {\mu} \tag {B.11}
$$
and used the fact that the distribution of  $C$  is symmetric.  $C$  is referred to as the crosstalk, because it represents the effect of interference between the first pattern and the other  $P - 1$  patterns on recall of the first pattern. We can simplify the crosstalk using the fact that, since we have assumed i.i.d. Rademacher patterns, we have the equality in distribution
$$
\xi_ {j} ^ {1} \xi_ {j} ^ {\mu} \stackrel {d} {=} \xi_ {j} ^ {\mu} \tag {B.12}
$$
for all  $\mu = 2,\ldots ,P$  and  $j = 1,\dots ,N$  , yielding
$$
C \stackrel {d} {=} \frac {1}{N - 1} \sum_ {\mu = 2} ^ {P} \sum_ {j \neq i} \xi_ {1} ^ {\mu} \xi_ {j} ^ {\mu}. \tag {B.13}
$$
$$
\xi_ {1} ^ {\mu} \xi_ {j} ^ {\mu} \stackrel {d} {=} \xi_ {j} ^ {\mu} \tag {B.14}
$$
for all  $\mu = 2,\ldots ,P$  and  $j = 2,\dots ,N$ , which finally yields
$$
C \stackrel {{d}} {{=}} \frac {1}{N - 1} \sum_ {\mu = 2} ^ {P} \sum_ {j \neq i} \xi_ {j} ^ {\mu}. \tag {B.15}
$$
Therefore, for the classic Hopfield network the crosstalk is equal in distribution to the sum of  $(P - 1)(N - 1)$  i.i.d. Rademacher random variables.
B.1 Approach 1:Hoeffding's inequality
Now, we can immediately apply Hoeffding's inequality [76], which implies that for any  $t > 0$
$$
\mathbb {P} \left[ \sum_ {\mu = 2} ^ {P} \sum_ {k = 2} ^ {N} \xi_ {k} ^ {\mu} > t \right] \leq \exp \left(- \frac {1}{2} \frac {t ^ {2}}{(P - 1) (N - 1)}\right). \tag {B.16}
$$
We then have that
$$
\mathbb {P} \left[ \sum_ {\mu = 2} ^ {P} \sum_ {k = 2} ^ {N} \xi_ {k} ^ {\mu} > N - 1 \right] \leq \exp \left(- \frac {1}{2} \frac {N - 1}{P - 1}\right). \tag {B.17}
$$
We then have the bound
$$
P _ {H N} (N, c) \geq \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: N P \exp \left(- \frac {1}{2} \frac {N - 1}{P - 1}\right) \leq c \right\}. \tag {B.18}
$$
We now want to consider the regime  $N \gg 1$ , and demand that the error probability should tend to zero as we increase  $N$ . If we substitute in the Ansatz
$$
P \sim \frac {N}{\alpha \log N}, \tag {B.19}
$$
the bound is easily seen to tend to zero for all  $\alpha \geq 4$ , yielding an estimated capacity of
$$
P _ {H N} \sim \frac {N}{4 \log N}. \tag {B.20}
$$
As this estimates follows from a sequence of lower bounds on the memorization probability, it is a lower bound on the true capacity of the model [33]. However, via a more involved argument that accounts for the associations between the events  $\mathbf{T}_{HN}(\pmb{\xi}^{\mu}) = \pmb{\xi}^{\mu}$ , it was shown by Bovier [34] to be tight.
For the classical Hopfield network, the single bitflip probability  $\mathbb{P}[C > 1]$  is easy to control using elementary concentration inequalities because the crosstalk can be expressed as a sum of  $(P - 1)(N - 1)$  i.i.d. random variables. Therefore, we expect the crosstalk to concentrate whenever  $N$  or  $P$  or both together are large. However, for the DenseNet, we will find in Appendix C that the crosstalk is given as the sum of  $P - 1$  i.i.d. random variables, each of which is a nonlinear function applied to the sum of  $N - 1$  i.i.d. Rademacher random variables. Naive application of Hoeffding's inequality is then not particularly useful. We will therefore take a simpler, though less rigorously controlled approach, which can also be applied to the classical Hopfield network: we approximate the distribution of the crosstalk as Gaussian [26].
B.2 Approach 2: Gaussian approximation
For the classical Hopfield network, the fact that the crosstalk can be expressed as a sum of  $(P - 1)(N - 1)$  i.i.d. Rademacher random variables means that the classical central limit theorem implies that it tends in distribution to a Gaussian whenever  $(P - 1)(N - 1)$  tends to infinity. By symmetry, the mean of the crosstalk is zero, while its variance is easily seen to be
$$
\operatorname {v a r} (C) = \frac {P - 1}{N - 1}. \tag {B.21}
$$
If we approximate the distribution of the crosstalk for  $N$  and  $P$  large but finite by a Gaussian, we therefore have
$$
\mathbb {P} [ C > 1 ] \approx H \left(\sqrt {\frac {N - 1}{P - 1}}\right) \tag {B.22}
$$
where  $H(x) = \operatorname{erfc}(x / \sqrt{2}) / 2$  is the Gaussian tail distribution function. We want to have  $\mathbb{P}[C > 1] \to 0$ , so we must have  $(P - 1) / (N - 1) \to 0$ . Then, we can use the asymptotic expansion [26]
$$
H (\sqrt {z}) = \frac {1}{\sqrt {2 \pi z}} \exp \left(- \frac {z}{2}\right)\left[ 1 + \mathcal {O} \left(\frac {1}{z}\right)\right] \quad \text {a s} \quad z \rightarrow \infty \tag {B.23}
$$
to obtain the heuristic Gaussian approximation
$$
\mathbb {P} [ C > 1 ] \approx \sqrt {\frac {(P - 1)}{2 \pi (N - 1)}} \exp \left(- \frac {(N - 1)}{2 (P - 1)}\right). \tag {B.24}
$$
If we use this Gaussian approximation instead of the Hoeffding bound applied above, we can easily see that we will obtain identical estimates for the capacity with an error tolerance tending to zero. However, we have given up the rigor of the bound from Hoeffding's inequality, since we have not controlled the rate of convergence to the Gaussian tail probability. In particular, the Berry-Esseen theorem would give in this case a uniform additive error bound of  $1 / \sqrt{(P - 1)(N - 1)}$ , which in the regime  $P \sim N / [\alpha \log N]$  cannot compete with the factors of  $N$  or  $NP$  which we want  $\mathbb{P}[C > 1]$  to overwhelm. We will not worry about this issue, as we are concerned more with whether we can get accurate capacity estimates that match numerical experiment than whether we can prove those estimates completely rigorously.
We can also use the Gaussian approximation to estimate the capacity for a non-zero error threshold  $c$  at finite  $N$ . Concretely, if we demand that the union bound is saturated, i.e.,
$$
N P \mathbb {P} \left[ T _ {H N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {1} \right] = c, \tag {B.25}
$$
under the Gaussian approximation for the bitflip probability we have the self-consistent equation
$$
N P H \left(\sqrt {\frac {N - 1}{P - 1}}\right) = c \tag {B.26}
$$
for  $P$ , which we can re-write as
$$
P - 1 = \frac {N - 1}{\left[ H ^ {- 1} (c / N P) \right] ^ {2}}. \tag {B.27}
$$
This is a transcendental self-consistent equation, which is not easy to solve analytically. However, we can make some progress at small  $c / (NP)$ . Using the asymptotic expansion of the inverse of the complementary error function [77], we have
$$
[ H ^ {- 1} (x) ] ^ {2} = 2 \operatorname {i n v e r f c} (2 x) ^ {2} \tag {B.28}
$$
$$
\sim - \log \left[ 4 \pi x ^ {2} \log \left(\frac {1}{2 x}\right) \right] \tag {B.29}
$$
$$
= - 2 \log (x) - \log (4 \pi) - \log \log \left(\frac {1}{2 x}\right) \tag {B.30}
$$
$$
\sim - 2 \log (x) \tag {B.31}
$$
as  $x \to 0$ . Then, assuming  $c$  is such that  $-\log(c)$  is negligible relative to  $\log(NP)$ , we have
$$
P \sim \frac {N}{2 \log (N P)}, \tag {B.32}
$$
which we can solve for  $P$  as
$$
P \sim \frac {N}{2 W _ {0} \left(N ^ {2} / 2\right)}, \tag {B.33}
$$
where  $W_0$  is the principal branch of the Lambert-  $W$  function [77]. But, at large  $N$ , we can use the asymptotic  $W_0(N) \sim \log (N)$  to obtain the approximate scaling
$$
P \sim \frac {N}{4 \log (N)}, \tag {B.34}
$$
which agrees with our earlier result. Conceptually, this intuition is consistent with there being a sharp transition in the thermodynamic limit, as proved rigorously by Bovier [34].
C DenseNet Capacity
In this Appendix, we analyze the capacity of the DenseNet. As introduced in Section 2.1 of the main text, there are two notions of robustness to consider: the robustness of a single transition and the robustness of the full sequence. For a fixed  $N \in \{2,3,\ldots\}$  and an error tolerance  $c \in [0,1)$ , we define these two notions of capacity as
$$
P _ {T} (N, c) = \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: \mathbb {P} \left[ \mathbf {T} _ {D N} (\boldsymbol {\xi} ^ {1}) = \boldsymbol {\xi} ^ {2} \right] \geq 1 - c \right\} \tag {C.1}
$$
$$
P _ {S} (N, c) = \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: \mathbb {P} \left[ \cap_ {\mu = 1} ^ {P} \left\{\mathbf {T} _ {D N} \left(\boldsymbol {\xi} ^ {\mu}\right) = \boldsymbol {\xi} ^ {\mu + 1} \right\} \right] \geq 1 - c \right\}, \tag {C.2}
$$
Our goal is to approximately compute the capacity in the regime in which  $N$  and  $P$  are large. Following Petritis [33]'s approach to the HN, to make analytical progress, we can use a union bound to control the single-step error probability in terms of the probability of a single bitflip:
$$
\begin{array}{l} \mathbb {P} \left[ \mathbf {T} _ {D N} (\pmb {\xi} ^ {\mu}) = \pmb {\xi} ^ {\mu + 1} \right] \\ = 1 - \mathbb {P} \left[ \bigcup_ {i = 1} ^ {N} \left\{T _ {D N} \left(\boldsymbol {\xi} ^ {\mu}\right) _ {i} \neq \xi_ {i} ^ {\mu + 1} \right\} \right] (C.3) \\ \geq 1 - \sum_ {i = 1} ^ {N} \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {\mu}\right) _ {i} \neq \xi_ {i} ^ {\mu + 1} \right] (C.4) \\ = 1 - N \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right]. (C.5) \\ \end{array}
$$
where we use the fact that all elements of all patterns are i.i.d. by assumption. We use a similar approach to control the sequence error probability in terms of the probability of a single bitflip:
$$
\begin{array}{l} \mathbb {P} \left[ \bigcap_ {\mu = 1} ^ {P} \left\{\mathbf {T} _ {D N} \left(\boldsymbol {\xi} ^ {\mu}\right) = \boldsymbol {\xi} ^ {\mu + 1} \right\} \right] \\ = 1 - \mathbb {P} \left[ \bigcup_ {\mu = 1} ^ {P} \bigcup_ {i = 1} ^ {N} \left\{T _ {D N} \left(\boldsymbol {\xi} ^ {\mu}\right) _ {i} \neq \xi_ {i} ^ {\mu + 1} \right\} \right] (C.6) \\ \geq 1 - \sum_ {\mu = 1} ^ {P} \sum_ {i = 1} ^ {N} \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {\mu}\right) _ {i} \neq \xi_ {i} ^ {\mu + 1} \right] (C.7) \\ = 1 - N P \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right]. (C.8) \\ \end{array}
$$
Thus, as claimed in the main text, we have the lower bounds
$$
P _ {T} (N, c) \geq \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: N \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right] \leq c \right\} \tag {C.9}
$$
$$
P _ {S} (N, c) \geq \max  \left\{P \in \{2, \dots , 2 ^ {N} \}: N P \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right] \leq c \right\}. \tag {C.10}
$$
As introduced in the main text, for perfect recall, we want to take the threshold  $c$  to be zero, or at least to tend to zero as  $N$  and  $P$  tend to infinity. The capacities estimated through this argument are lower bounds on the true capacities, as they are obtained from lower bounds on the true recall probability. However, we expect for these bounds to in fact be tight in the thermodynamic limit [33, 34].
By the definition of the DenseNet update rule with interaction function  $f$  given in Equation (2), we have
$$
T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} = \operatorname {s g n} \left[ \sum_ {\mu = 1} ^ {P} \xi_ {1} ^ {\mu + 1} f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) \right] \tag {C.11}
$$
and therefore the single-bitflip probability is
$$
\begin{array}{l} \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] = \mathbb {P} \left[ \operatorname {s g n} \left[ \sum_ {\mu = 1} ^ {P} \xi_ {1} ^ {\mu + 1} f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) \right] \neq \xi_ {1} ^ {2} \right] (C.12) \\ = \mathbb {P} \left[ \xi_ {1} ^ {2} \sum_ {\mu = 1} ^ {P} \xi_ {1} ^ {\mu + 1} f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) <   0 \right] (C.13) \\ = \mathbb {P} \left[ f (1) + \xi_ {1} ^ {2} \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {\mu + 1} f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) <   0 \right] (C.14) \\ \end{array}
$$
For both the polynomial  $(f(x) = x^d)$  and exponential  $(f(x) = e^{(N - 1)(x - 1)})$  interaction functions,  $f(1) = 1$ , and so
$$
\mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] = \mathbb {P} \left[ \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {2} \xi_ {1} ^ {\mu + 1} f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) <   - 1 \right]. \tag {C.15}
$$
We refer to the random variable
$$
C = \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {2} \xi_ {1} ^ {\mu + 1} f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) \tag {C.16}
$$
on the left-hand-side of this inequality as the crosstalk, because it represents the effect of interference between the first pattern and all other patterns [26, 36].
We now observe that, as we have excluded self-interactions (i.e., the sum over neurons inside the interaction function does not include  $j = 1$ ), we can use the periodic boundary conditions to shift indices as  $\xi_1^\mu \gets \xi_1^{\mu +1}$  for all  $\mu$ , yielding
$$
C \stackrel {d} {=} \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {1} \xi_ {1} ^ {\mu} f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) \tag {C.17}
$$
Thus, the single-bitflip probability for this DenseNet is identical to that for the corresponding MHN with symmetric interactions. Then, we can use the fact that  $\xi_j^\mu \xi_j^1 \stackrel{d}{=} \xi_j^\mu$  for all  $\mu = 2, \ldots, P$  to obtain
$$
C \stackrel {d} {=} \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {\mu} f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right). \tag {C.18}
$$
Now, define the  $P - 1$  random variables
$$
\chi^ {\mu} = \xi_ {1} ^ {\mu} f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) \tag {C.19}
$$
for  $\mu = 2,\ldots ,P$  , such that the crosstalk is their sum,
$$
C = \sum_ {\mu = 2} ^ {P} \chi^ {\mu}. \tag {C.20}
$$
As the patterns  $\xi_j^\mu$  are i.i.d.,  $\chi^\mu$  are i.i.d. random variables of mean
$$
\mathbb {E} \left[ \chi^ {\mu} \right] = \mathbb {E} \left[ \xi_ {1} ^ {\mu} \right] \mathbb {E} \left[ f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) \right] = 0 \tag {C.21}
$$
$$
\operatorname {v a r} \left(\chi^ {\mu}\right) = \mathbb {E} \left[ f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {2} \right], \tag {C.22}
$$
which is bounded from above for any sensible interaction function. We observe also that the distribution of each  $\chi^{\mu}$  is symmetric because of the symmetry of the distribution of  $\xi_1^\mu$ . We will therefore simply write  $\chi$  for any given  $\chi^{\mu}$ .
Then, the classical central limit theorem implies that the crosstalk tends in distribution to a Gaussian of mean zero and variance  $(P - 1)\mathrm{var}(\chi)$  as  $P\to \infty$  , at lease for any fixed  $N$  . However, we are interested in the joint limit in which  $N,P\rightarrow \infty$  together. We will proceed by approximating the distribution of  $C$  as Gaussian, and will not attempt to rigorously control its behavior in the joint limit.
Approximating the distribution of the crosstalk for  $N, P \gg 1$  by a Gaussian, we then have
$$
\mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \approx H \left(\frac {1}{\sqrt {(P - 1) \operatorname {v a r} (\chi)}}\right) \tag {C.23}
$$