SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $H(x) = \operatorname{erfc}(x / \sqrt{2}) / 2$  is the Gaussian tail distribution function. We want to have  $\mathbb{P}[T_{DN}(\pmb{\xi}^1)_1 \neq \xi_1^2] \to 0$ , so we must have  $(P - 1)\mathrm{var}(\chi) \to 0$ . Then, we can use the asymptotic expansion [26]
$$
H (\sqrt {z}) = \frac {1}{\sqrt {2 \pi z}} \exp \left(- \frac {z}{2}\right)\left[ 1 + \mathcal {O} \left(\frac {1}{z}\right)\right] \quad \text {a s} \quad z \rightarrow \infty \tag {C.24}
$$
to obtain
$$
\mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \approx \sqrt {\frac {(P - 1) \operatorname {v a r} (\chi)}{2 \pi}} \exp \left(- \frac {1}{2 (P - 1) \operatorname {v a r} (\chi)}\right). \tag {C.25}
$$
For each model, we can evaluate  $\mathrm{var}(\chi)$  and then determine the resulting predicted capacity.
As we did for the classic Hopfield network in Appendix B, we can estimate the capacity at finite  $c$  within the Gaussian approximation by inverting the Gaussian tail distribution function. Concretely, under the union bound, we can estimate the transition capacity by solving
$$
c = N H \left(\frac {1}{\sqrt {\left(P _ {T} - 1\right) \operatorname {v a r} (\chi)}}\right), \tag {C.26}
$$
$$
P _ {T} - 1 = \frac {1}{\operatorname {v a r} (\chi) [ H ^ {- 1} (c / N) ] ^ {2}}, \tag {C.27}
$$
and the sequence capacity by solving the transcendental self-consistent equation
$$
c = N P _ {S} H \left(\frac {1}{\sqrt {\left(P _ {S} - 1\right) \operatorname {v a r} (\chi)}}\right), \tag {C.28}
$$
which we can re-write as
$$
P _ {S} - 1 = \frac {1}{\operatorname {v a r} (\chi) \left[ H ^ {- 1} \left(c / N P _ {S}\right) \right] ^ {2}}. \tag {C.29}
$$
As in the classic Hopfield case, we can simplify these complicated equations somewhat by assuming that  $c / N$  and  $c / (NP_S)$  are small. Concretely, using the asymptotic
$$
[ H ^ {- 1} (x) ] ^ {2} \sim - 2 \log (x) \tag {C.30}
$$
for  $x\to 0$  , the transition capacity simplifies to
$$
P _ {T} - 1 \sim \frac {1}{2 \operatorname {v a r} (\chi) \log (N)} \tag {C.31}
$$
under the assumption that  $-\log (c)$  is negligible relative to  $\log (N)$ . For the sequence capacity, we can follow an identical argument to that used for the classic Hopfield network to simplify the self-consistent equation to
$$
P _ {S} \sim \frac {1}{2 \operatorname {v a r} (\chi) \log \left(N P _ {S}\right)} \tag {C.32}
$$
under the assumption that  $-\log (c)$  is negligible relative to  $\log (NP_S)$ , which we can solve to obtain
$$
P _ {S} \sim \frac {1}{2 \operatorname {v a r} (\chi) W _ {0} [ N / 2 \operatorname {v a r} (\chi) ]}. \tag {C.33}
$$
Assuming that  $N / \operatorname{var}(\chi) \to \infty$  as  $N \to \infty$ , we can use the asymptotic  $W_0(N) \sim \log (N)$  to obtain the asymptotic
$$
P _ {S} \sim \frac {1}{2 \operatorname {v a r} (\chi) \log [ N / \operatorname {v a r} (\chi) ]}. \tag {C.34}
$$
Our first check on the accuracy of the Gaussian approximation will be comparison of the resulting predictions for capacity with numerical experiment. As another diagnostic, we will consider the excess kurtosis  $\varkappa = \kappa_4(C) / \kappa_2(C)$  for  $\kappa_n(C)$  the  $n$ -th cumulant of  $C$ . If the distribution is indeed Gaussian, the excess kurtosis vanishes, while large values of the excess kurtosis indicate deviations from Gaussianity. By the additivity of cumulants, we have
$$
\kappa_ {n} (C) = (P - 1) \kappa_ {n} (\chi). \tag {C.35}
$$
By symmetry, all odd cumulants of  $\chi$  and therefore all odd cumulants of  $C$  are identically zero. As noted above, we have
$$
\operatorname {v a r} (\chi) = \kappa_ {2} (\chi) = \mathbb {E} \left[ f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {2} \right]. \tag {C.36}
$$
If  $C$  is indeed Gaussian, then all cumulants above the second should vanish. As the third cumulant vanishes by symmetry, the leading possible correction to Gaussianity is the fourth cumulant, which as  $\chi$  has zero mean is given by
$$
\begin{array}{l} \kappa_ {4} (\chi) = \mathbb {E} \left[ \left(\chi\right) ^ {4} \right] - 3 \mathbb {E} \left[ \left(\chi\right) ^ {2} \right] ^ {2} (C.37) \\ = \mathbb {E} \left[ f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {4} \right] - 3 \mathbb {E} \left[ f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {2} \right] ^ {2}. (C.38) \\ \end{array}
$$
Rather than considering the fourth cumulant directly, we will consider the excess kurtosis
$$
\varkappa = \frac {\kappa_ {4} (C)}{\kappa_ {2} (C) ^ {2}} = \frac {1}{P - 1} \frac {\kappa_ {4} (\chi)}{\kappa_ {2} (\chi) ^ {2}}, \tag {C.39}
$$
which is a more useful metric because it is normalized.
C.1 Polynomial DenseNet Capacity
We first consider the Polynomial DenseNet, with interaction function  $f(x) = x^{d}$  for  $d \in \mathbb{N}_{>0}$ . To compute the capacity, our goal is then to evaluate
$$
\operatorname {v a r} (\chi) = \mathbb {E} \left[ \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {1}\right) ^ {2 d} \right] \tag {C.40}
$$
at large  $N$ . From the central limit theorem, we expect
$$
\mathbb {E} \left[ \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {1}\right) ^ {2 d} \right] \sim \frac {(2 d - 1) ! !}{(N - 1) ^ {d}}. \tag {C.41}
$$
We can make this quantitatively precise through the following straightforward argument. Let
$$
\Xi = \frac {1}{\sqrt {N - 1}} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {2}. \tag {C.42}
$$
We then have immediately that the moment generating function of  $\Xi$  is
$$
M (t) = \mathbb {E} \left[ e ^ {t \Xi} \right] = \cosh \left(\frac {t}{\sqrt {N - 1}}\right) ^ {N - 1}, \tag {C.43}
$$
hence the cumulant generating function is
$$
K (t) = \log M (t) = (N - 1) \log \cosh \left(\frac {t}{\sqrt {N - 1}}\right). \tag {C.44}
$$
The function  $x \mapsto \log \cosh(x)$  is an even function of  $x$ , and is analytic near the origin, with the first few orders of its MacLaurin series being
$$
\log \cosh (x) = \frac {x ^ {2}}{2} - \frac {x ^ {4}}{1 2} + \mathcal {O} \left(x ^ {6}\right). \tag {C.45}
$$
Then, the odd cumulants of  $\Xi$  vanish—as we expect from symmetry—while the even cumulants obey
$$
\kappa_ {2 k} = \frac {C _ {2 k}}{(N - 1) ^ {k - 1}} \tag {C.46}
$$
for combinatorial factors  $C_{2k}$  that do not scale with  $N$ . We have, in particular,  $C_2 = 1$  and  $C_4 = -2$ . By the moments-cumulants formula, we have
$$
\mathbb {E} \left[ \Xi^ {2 k} \right] = B _ {2 k} \left(0, \kappa_ {2}, 0, \kappa_ {4}, \dots , \kappa_ {2 k}\right) \tag {C.47}
$$
for  $B_{2k}$  the  $2k$ -th complete exponential Bell polynomial. From this, it follows that
$$
\mathbb {E} \left[ \Xi^ {2 k} \right] = (2 k - 1)!! + \mathcal {O} \left(N ^ {- 1}\right), \tag {C.48}
$$
as all cumulants other than  $\kappa_{2} = 1$  are  $\mathcal{O}(N^{-1})$ . Therefore, neglecting subleading terms, we have
$$
\operatorname {v a r} (\chi) = \mathbb {E} \left[ \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {1}\right) ^ {2 d} \right] = \frac {(2 d - 1) ! !}{N ^ {d}} \left[ 1 + \mathcal {O} \left(\frac {1}{N}\right) \right]. \tag {C.49}
$$
Following the general arguments above, we then approximate
$$
\mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \sqrt {\frac {P (2 d - 1) ! !}{2 \pi N ^ {d}}} \exp \left(- \frac {N ^ {d}}{2 P (2 d - 1) ! !}\right). \tag {C.50}
$$
To determine the single-transition capacity following the argument in Section 2.1, we must determine how large we can take  $P = P(N)$  such that  $N\mathbb{P}[T_{DN}(\pmb{\xi}^1)_1 \neq \xi_1^2] \to 0$ . Following the requirement that  $P\mathrm{var}(\chi) \to 0$ , we make the Ansatz
$$
P \sim \frac {N ^ {d}}{\alpha (2 d - 1) ! ! \log N} \tag {C.51}
$$
for some  $\alpha$ . We then have
$$
N \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \sqrt {\frac {1}{2 \pi \alpha \log N}} N ^ {1 - \alpha / 2}. \tag {C.52}
$$
This tends to zero if  $\alpha \geq 2$ , meaning that the predicted capacity in this case is
$$
P _ {T} \sim \frac {N ^ {d}}{2 (2 d - 1) ! ! \log N}. \tag {C.53}
$$
We now want to determine the sequence capacity, which requires the stronger condition  $N P \mathbb{P}[T_{D N}(\pmb{\xi}^{1})_{1} \neq \pmb{\xi}_{1}^{2}] \to 0$ . Again making the Ansatz
$$
P \sim \frac {N ^ {d}}{\alpha (2 d - 1) ! ! \log N} \tag {C.54}
$$
for some  $\alpha$ , we then have
$$
N P \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \frac {1}{\sqrt {2 \pi} (2 d - 1) ! ! (\alpha \log N) ^ {3 / 2}} N ^ {d + 1 - \alpha / 2}, \tag {C.55}
$$
which tends to zero if  $\alpha \geq 2d + 2$ . Then, the predicted sequence capacity is
$$
P _ {S} \sim \frac {N ^ {d}}{2 (d + 1) (2 d - 1) ! ! \log N}. \tag {C.56}
$$
If we consider the alternative asymptotic formulas obtained above from the finite-  $c$  argument, we have
$$
P _ {T} \sim \frac {1}{2 \operatorname {v a r} (\chi) \log (N)} \sim \frac {N ^ {d}}{2 (2 d - 1) ! ! \log (N)} \tag {C.57}
$$
$$
P _ {S} \sim \frac {1}{2 \operatorname {v a r} (\chi) \log [ N / \operatorname {v a r} (\chi) ]} \sim \frac {N ^ {d}}{2 (2 d - 1) ! ! \log [ N ^ {d + 1} / (2 d - 1) ! ! ]} \sim \frac {N ^ {d}}{2 (d + 1) (2 d - 1) ! ! \log (N)}, \tag {C.58}
$$
which agree with these results. For evidence of the finite- $c$  argument for the polynomial DenseNet, observe Figure C.1.
Figure C.1: The transition capacity of the polynomial DenseNet is demonstrated for different values of error tolerance  $c$ . We see that even for  $c \neq 0$ , we get similar scaling curves although the capacities slightly increase consistently as we increase  $c$ , indicated by a transition from dark to light. We plot from  $c = 0.0$  to  $c = 0.5$  for each degree  $d$ , with the legend labeling curves up to  $c = 0.3$  to demonstrate the general trend.
Using the Gaussian approximation for moments of  $\chi$  given above, we can easily work out that
$$
\begin{array}{l} \kappa_ {4} (\chi) = \mathbb {E} [ (\chi) ^ {4} ] - 3 \mathbb {E} [ (\chi) ^ {2} ] (C.59) \\ = \mathbb {E} \left[ f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {4} \right] - 3 \mathbb {E} \left[ f \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {2} \right] ^ {2} (C.60) \\ = \frac {1}{N ^ {2 d}} \left\{\left(4 d - 1\right)!! - 3 \left[ \left(2 d - 1\right)!! \right] ^ {2} \right\} \left[ 1 + \mathcal {O} \left(\frac {1}{N}\right) \right]. (C.61) \\ \end{array}
$$
Then, the excess kurtosis of the Polynomial DenseNet's crosstalk is
$$
\varkappa = \frac {1}{P - 1} \left[ \frac {(4 d - 1) ! !}{[ (2 d - 1) ! ! ] ^ {2}} - 3 \right] \left[ 1 + \mathcal {O} \left(\frac {1}{N}\right) \right]. \tag {C.62}
$$
Thus, for the Polynomial DenseNet, we expect the excess kurtosis to be small for any fixed  $d$  so long as  $P$  and  $N$  are both fairly large, without any particular requirement on their relationship. In particular, under the Gaussian approximation we predicted above that the transition and sequence capacities should both scale as
$$
P \sim \frac {N ^ {d}}{\alpha_ {d} \log N}, \tag {C.63}
$$
where  $\alpha_{d}$  depends on  $d$  but not on  $N$ . This gives an excess kurtosis of
$$
\varkappa = \frac {\alpha_ {d} \log N}{N ^ {d}} \left[ \frac {(4 d - 1) ! !}{[ (2 d - 1) ! ! ] ^ {2}} - 3 \right] \left[ 1 + \mathcal {O} \left(\frac {1}{N}\right) \right] \tag {C.64}
$$
which for any fixed  $d$  rapidly tends to zero with increasing  $N$ . This suggests that the Gaussian approximation should be reasonably accurate even at modest  $N$ , but of course does not constitute a proof of its accuracy because we have not considered higher cumulants. However, this matches the results of numerical simulations shown in Figure 2.
C.2 Exponential DenseNet capacity
We now turn our attention to the Exponential DenseNet, with separation function  $f(x) = e^{(N - 1)(x - 1)}$ . In this case, we have
$$
\begin{array}{l} \operatorname {v a r} (\chi) = \exp [ - 2 (N - 1) ] \mathbb {E} \left[ \exp \left(2 \sum_ {j = 2} ^ {N} \xi_ {j} ^ {2}\right) \right] (C.65) \\ = \exp [ - 2 (N - 1) ] \prod_ {j = 2} ^ {N} \mathbb {E} \left[ \exp \left(2 \xi_ {j} ^ {2}\right) \right] (C.66) \\ = \exp [ - 2 (N - 1) ] \cosh (2) ^ {N - 1} (C.67) \\ = \frac {1}{\beta^ {N - 1}}, (C.68) \\ \end{array}
$$
where we have defined the constant
$$
\beta = \frac {\exp (2)}{\cosh (2)} \simeq 1. 9 6 4 0 3. \tag {C.69}
$$
Then, we have the Gaussian approximation
$$
\mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \sqrt {\frac {P}{2 \pi \beta^ {N - 1}}} \exp \left(- \frac {\beta^ {N - 1}}{2 P}\right). \tag {C.70}
$$
As in the polynomial case, we first determine the single-transition capacity by demanding that  $N\mathbb{P}[T_{DN}(\pmb{\xi}^1)_1 \neq \xi_1^2] \to 0$ . We plug in the Ansatz
$$
P \sim \frac {\beta^ {N - 1}}{\alpha \log N} \tag {C.71}
$$
for some  $\alpha$ , which yields
$$
N \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \sqrt {\frac {1}{2 \pi \alpha \log N}} N ^ {1 - \alpha / 2}. \tag {C.72}
$$
This tends to zero if  $\alpha \geq 2$ , which gives a predicted capacity of
$$
P _ {T} \sim \frac {\beta^ {N - 1}}{2 \log N}. \tag {C.73}
$$
Considering the sequence capacity, which again requires that  $N P \mathbb{P}[T_{D N}(\pmb{\xi}^{1})_{1} \neq \xi_{1}^{2}] \to 0$ , we plug in the Ansatz
$$
P \sim \frac {\beta^ {N - 1}}{\alpha N}, \tag {C.74}
$$
$$
N P \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \frac {1}{\alpha \beta} \sqrt {\frac {1}{2 \pi \alpha N}} \exp \left[ \left(\log \beta - \frac {\alpha}{2}\right) N \right]. \tag {C.75}
$$
This tends to zero for  $\alpha \geq 2\log \beta$ , meaning that the predicted capacity is in this case
$$
P _ {S} \sim \frac {\beta^ {N - 1}}{2 \log (\beta) N}. \tag {C.76}
$$
Therefore, while the ratio of the predicted single-transition to sequence capacities is finite for the Polynomial DenseNet—it is simply  $P_{S} / P_{T} \sim d + 1$  —for the Exponential DenseNet it tends to zero as  $P_{S} / P_{T} \sim \log N / [\log (\beta)N]$ .
Using the asymptotic formulas obtained above from the finite-  $c$  argument, we have
$$
P _ {T} \sim \frac {1}{2 \operatorname {v a r} (\chi) \log (N)} = \frac {\beta^ {N - 1}}{2 \log (N)} \tag {C.77}
$$
Figure C.2: The transition capacity of the exponential DenseNet is demonstrated for different values of error tolerance  $c$ . We see that even for  $c \neq 0$ , we get similar scaling curves although the capacities slightly increase consistently as we increase  $c$ .
$$
P _ {S} \sim \frac {1}{2 \operatorname {v a r} (\chi) \log [ N / \operatorname {v a r} (\chi) ]} = \frac {\beta^ {N - 1}}{2 \log [ N \beta^ {N - 1} ]} \sim \frac {\beta^ {N - 1}}{2 \log (\beta) N}, \tag {C.78}
$$
which agree with these results. For evidence of the finite- $c$  argument for the exponential DenseNet, observe Figure C.2.
Now considering the fourth cumulant, we can easily compute
$$
\kappa_ {4} (\chi) = \left(\frac {\cosh (4)}{\exp (4)}\right) ^ {N - 1} - 3 \left(\frac {\cosh (2) ^ {2}}{\exp (4)}\right) ^ {N - 1}, \tag {C.79}
$$
which yields an excess kurtosis of
$$
\varkappa = \frac {1}{P - 1} \left[ \left(\frac {\cosh (4)}{\cosh (2) ^ {2}}\right) ^ {N - 1} - 3 \right]. \tag {C.80}
$$
For this to be small,  $P$  must be exponentially large in  $N$ , which contrasts with the situation for the Polynomial DenseNet, in which the excess kurtosis is small for any reasonably large  $P$ . If we consider taking
$$
P \sim \frac {\beta^ {N - 1}}{\alpha \log N}, \tag {C.81}
$$
for a constant  $\alpha$ , as the Gaussian theory predicts for the Exponential DenseNet transition capacity, we have
$$
\varkappa \sim \frac {\alpha \log N}{\beta^ {N - 1}} \left[ \left(\frac {\cosh (4)}{\cosh (2) ^ {2}}\right) ^ {N - 1} - 3 \right] \tag {C.82}
$$
$$
\sim \alpha \log N \left(\frac {\cosh (4)}{\exp (2) \cosh (2)}\right) ^ {N - 1} \tag {C.83}
$$
$$
\simeq \alpha \log (N) (0. 9 8 2 3) ^ {N - 1}. \tag {C.84}
$$
This tends to zero as  $N$  increases, but only very slowly. In particular,  $\log (N)(0.9823)^{N - 1}$  increases with  $N$  up to around  $N\simeq 19$ , where it attains a maximum value around 2, before decreasing towards zero. The situation is even worse for the sequence capacity, for which the Gaussian theory predicts
$$
P \sim \frac {\beta^ {N - 1}}{\alpha N}, \tag {C.85}
$$
$$
\begin{array}{l} \varkappa \sim \frac {\alpha N}{\beta^ {N - 1}} \left[ \left(\frac {\cosh (4)}{\cosh (2) ^ {2}}\right) ^ {N - 1} - 3 \right] (C.86) \\ \sim \alpha N \left(\frac {\cosh (4)}{\exp (2) \cosh (2)}\right) ^ {N - 1} (C.87) \\ \simeq \alpha N (0. 9 8 2 3) ^ {N - 1}. (C.88) \\ \end{array}
$$
$N(0.9823)^{N - 1}$  increases with  $N$  up to around  $N\simeq 56$ , where it attains a value of approximately 21.
Taken together, these results suggest that we might expect substantial finite-size corrections to the Gaussian theory's prediction for the capacity. In particular, as the excess kurtosis of the crosstalk is positive, the tails of the crosstalk distribution should be heavier-than-Gaussian, suggesting that the Gaussian theory should overestimate the true capacity. This holds provided that the lower bound on the memorization probability resulting from the union bound is reasonably tight.
D Bounding the polynomial DenseNet capacity
Here, we adapt Demircigil et al. [31]'s proof of a rigorous asymptotic lower bound on the polynomial MHN's capacity to obtain a rigorous asymptotic lower bound on the DenseNet capacity. This proof is a step-by-step adaptation of the proof of Theorem 1.2 of Demircigil et al. [31], which we spell out in detail for clarity.
Our objective is to obtain an upper bound on the single-bitflip probability
$$
\mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right] \tag {D.1}
$$
which we have argued can be expressed in terms of the crosstalk  $C$  as
$$
\mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right] = \mathbb {P} [ C <   - 1 ] \tag {D.2}
$$
$$
C \stackrel {d} {=} \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {\mu} \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {d}. \tag {D.3}
$$
Our goal is to prove the following: First, letting  $\alpha > 2(2d - 1)!!$  and  $P = N^d / (\alpha \log N)$ , we have
$$
N \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right]\rightarrow 0 \tag {D.4}
$$
as  $N\to \infty$  . Second, letting  $\alpha >2(d + 1)(2d - 1)!!$  and  $P = N^{d} / (\alpha \log N)$  , we have
$$
N P \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right]\rightarrow 0 \tag {D.5}
$$
as  $N\to \infty$
By Chernoff's inequality (also known as the exponential Chebyshev inequality) [76], we then have
$$
\begin{array}{l} \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right] = \mathbb {P} \left[ \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {\mu} \left(\sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {d} <   - (N - 1) ^ {d} \right] (D.6) \\ \leq e ^ {- t (N - 1) ^ {d}} \mathbb {E} \exp \left[ - t \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {\mu} \left(\sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {d} \right] (D.7) \\ \end{array}
$$
for any  $t > 0$ . Using the fact that the pattern elements are i.i.d., we have
$$
\begin{array}{l} \mathbb {E} \exp \left[ - t \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {\mu} \left(\sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {d} \right] = \left\{\mathbb {E} \exp \left[ - t \xi_ {1} ^ {\mu} \left(\sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {d} \right] \right\} ^ {P - 1} (D.8) \\ = \left\{\mathbb {E} \cosh \left[ t \left(\sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {d} \right] \right\} ^ {P - 1}. (D.9) \\ \end{array}
$$
$$
M = \frac {1}{\sqrt {N - 1}} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}, \tag {D.10}
$$
and expand the expectation as a sum over the possible values  $m \in \{0, \pm (N - 1)^{-1/2}, \ldots, \pm (N - 1)^{1/2}\}$  of  $M$ :
$$
\mathbb {E} \cosh \left[ t \left(\sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {d} \right] = \sum_ {m} \cosh \left[ t (N - 1) ^ {d / 2} m ^ {d} \right] \mathbb {P} [ M = m ]. \tag {D.11}
$$
For  $N \gg 1$ , the distribution of  $M$  is nearly Gaussian. We thus split the sum over  $m$  to allow us to treat tail events separately. We fix  $\beta > 0$ , and split the sum at  $\log (N)^{\beta}$ :
$$
\begin{array}{l} \sum_ {m} \cosh [ t (N - 1) ^ {d / 2} m ^ {d} ] \mathbb {P} [ M = m ] = \sum_ {| m | \leq \log (N) ^ {\beta}} \cosh [ t (N - 1) ^ {d / 2} m ^ {d} ] \mathbb {P} [ M = m ] \\ + \sum_ {\log (N) ^ {\beta} <   | m | \leq \sqrt {N}} \cosh [ t (N - 1) ^ {d / 2} m ^ {d} ] \mathbb {P} [ M = m ], \tag {D.12} \\ \end{array}
$$
where we have used the fact that  $M \leq \sqrt{N - 1}$ .
We first consider the tail sum over  $|m| > \log (N)^{\beta}$ . As  $\cosh$  is even and non-decreasing in the modulus of its argument, we have
$$
\sum_ {\log (N) ^ {\beta} <   | m | \leq \sqrt {N}} \cosh [ t (N - 1) ^ {d / 2} m ^ {d} ] \mathbb {P} [ M = m ] \tag {D.13}
$$
$$
\leq 2 \cosh [ t (N - 1) ^ {d} ] \mathbb {P} [ M > \log (N) ^ {\beta} ] \tag {D.14}
$$
$$
\leq 2 \cosh [ t (N - 1) ^ {d} ] \exp \left(- \frac {1}{2} \log (N) ^ {2 \beta}\right) \tag {D.15}
$$
$$
\leq 2 \exp \left[ t (N - 1) ^ {d} - \frac {1}{2} \log (N) ^ {2 \beta} \right], \tag {D.16}
$$
where in the second line we have applied Hoeffding's inequality to bound  $\mathbb{P}[M > \log (N)^{\beta}]$  from above, and in the third line we have used the bound  $\cosh (z)\leq \exp (z)$  for any  $z > 0$
We now consider the sum over  $|m| \leq \log (N)^{\beta}$ . Using the bound  $\cosh (z) \leq \exp (z^2 / 2)$ , we have
$$
\sum_ {| m | \leq \log (N) ^ {\beta}} \cosh [ t (N - 1) ^ {d / 2} m ^ {d} ] \mathbb {P} [ M = m ] \leq \sum_ {| m | \leq \log (N) ^ {\beta}} \exp \left[ \frac {1}{2} t ^ {2} (N - 1) ^ {d} m ^ {2 d} \right] \mathbb {P} [ M = m ]. \tag {D.17}
$$
Using the series expansion of the exponential, we have
$$
\begin{array}{l} \sum_ {| m | \leq \log (N) ^ {\beta}} \exp \left[ \frac {1}{2} t ^ {2} (N - 1) ^ {d} m ^ {2 d} \right] \mathbb {P} [ M = m ] (D.18) \\ = \sum_ {| m | \leq \log (N) ^ {\beta}} \left\{1 + \frac {1}{2} t ^ {2} (N - 1) ^ {d} m ^ {2 d} + \sum_ {k = 2} ^ {\infty} \frac {\left(t ^ {2} (N - 1) ^ {d} m ^ {2 d}\right) ^ {k}}{2 ^ {k} k !} \right\} \mathbb {P} [ M = m ] (D.19) \\ = \mathbb {P} [ | M | \leq \log (N) ^ {\beta} ] + \frac {1}{2} t ^ {2} (N - 1) ^ {d} \sum_ {| m | \leq \log (N) ^ {\beta}} m ^ {2 d} \mathbb {P} [ M = m ] \\ + \sum_ {| m | \leq \log (N) ^ {\beta}} \left\{\sum_ {k = 2} ^ {\infty} \frac {\left(t ^ {2} (N - 1) ^ {d} m ^ {2 d}\right) ^ {k}}{2 ^ {k} k !} \right\} \mathbb {P} [ M = m ] (D.20) \\ \end{array}
$$
where on the third line we have used the linearity of summation. We will now bound each of the three contributions. The first term is trivially bounded from above by 1:
$$
\mathbb {P} [ | M | \leq \log (N) ^ {\beta} ] \leq 1. \tag {D.21}
$$
To handle the second, we first observe that
$$
\sum_ {| m | \leq \log (N) ^ {\beta}} m ^ {2 d} \mathbb {P} [ M = m ] \leq \mathbb {E} [ m ^ {2 d} ]. \tag {D.22}
$$
Then, we observe that as  $m$  is the normalized sum of  $N - 1$  Rademacher random variables, its moments tend to those of a standard normal from below as  $N \to \infty$ , and are for any  $N$  strictly bounded from above by those of the standard normal. Therefore, we have
$$
\mathbb {E} \left[ m ^ {2 d} \right] \leq (2 d - 1)!!. \tag {D.23}
$$
To handle the third term, we first use the fact that for any  $|m| \leq \log (N)^{\beta}$  we have  $m^{2d} \leq \log (N)^{2\beta d}$ , which gives
$$
\begin{array}{l} \sum_ {| m | \leq \log (N) ^ {\beta}} \left\{\sum_ {k = 2} ^ {\infty} \frac {(t ^ {2} (N - 1) ^ {d} m ^ {2 d}) ^ {k}}{2 ^ {k} k !} \right\} \mathbb {P} [ M = m ] \\ \leq \sum_ {| m | \leq \log (N) ^ {\beta}} \left\{\sum_ {k = 2} ^ {\infty} \frac {\left(t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d}\right) ^ {k}}{2 ^ {k} k !} \right\} \mathbb {P} [ M = m ] (D.24) \\ \leq \mathbb {P} [ | M | \leq \log (N) ^ {\beta} ] \sum_ {k = 2} ^ {\infty} \frac {\left(t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d}\right) ^ {k}}{2 ^ {k} k !} (D.25) \\ \leq \sum_ {k = 2} ^ {\infty} \frac {\left(t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d}\right) ^ {k}}{2 ^ {k} k !} (D.26) \\ \end{array}
$$