SlowGuess/NeurIPS_2023_OCR huggingface dataset:
At this point, [31] uses the bound
$$
\sum_ {k = 2} ^ {\infty} \frac {\left(t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d}\right) ^ {k}}{2 ^ {k} k !} \leq \frac {1}{4} (e - 2) \left(t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d}\right) ^ {2} \tag {D.27}
$$
which holds provided that we choose the arbitrary parameter  $t$  such that
$$
t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d} \leq 2. \tag {D.28}
$$
Assuming that condition is satisfied, we can then combine these results to obtain
$$
\begin{array}{l} \sum_ {| m | \leq \log (N) ^ {\beta}} \cosh [ t (N - 1) ^ {d / 2} m ^ {d} ] \mathbb {P} [ M = m ] (D.29) \\ \leq 1 + \frac {1}{2} t ^ {2} (N - 1) ^ {d} (2 d - 1)!! + \frac {1}{4} (e - 2) \left(t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d}\right) ^ {2} (D.30) \\ \leq 1 + \frac {1}{2} t ^ {2} (N - 1) ^ {d} (2 d - 1)!! + \frac {1}{4} \left(t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d}\right) ^ {2} (D.31) \\ \leq \exp \left[ \frac {1}{2} t ^ {2} (N - 1) ^ {d} (2 d - 1)!! + \frac {1}{4} (t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d}) ^ {2} \right], (D.32) \\ \end{array}
$$
where on the second line we have used the fact that  $e - 2 \simeq 0.718 \ldots < 1$  and on the third line we have used the bound  $1 + x \leq \exp(x)$  for  $x \geq 0$ .
Combining this result with the bound on the tail sum obtained previously, we have that
$$
\begin{array}{l} \mathbb {E} \cosh \left[ t \left(\sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}\right) ^ {d} \right] \leq \exp \left[ \frac {1}{2} t ^ {2} (N - 1) ^ {d} (2 d - 1)!! + \frac {1}{4} (t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d}) ^ {2} \right] \\ + 2 \exp \left[ t (N - 1) ^ {d} - \frac {1}{2} \log (N) ^ {2 \beta} \right] \tag {D.33} \\ \end{array}
$$
for any  $\beta > 1/2$  and
$$
0 <   t \leq \sqrt {\frac {2}{(N - 1) ^ {d} \log (N) ^ {2 \beta d}}}. \tag {D.34}
$$
$$
\begin{array}{l} \mathbb {P} [ T _ {D N} (\pmb {\xi} ^ {1}) _ {1} \neq \xi_ {2} ^ {1} ] \leq e ^ {- t (N - 1) ^ {d}} \Bigg \{\exp \left[ \frac {1}{2} t ^ {2} (N - 1) ^ {d} (2 d - 1)!! + \frac {1}{4} (t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d}) ^ {2} \right] \\ \left. + 2 \exp \left[ t (N - 1) ^ {d} - \frac {1}{2} \log (N) ^ {2 \beta} \right] \right\} ^ {P - 1} \tag {D.35} \\ \end{array}
$$
subject to these conditions on  $\beta$  and  $t$ .
We now want to determine the single-transition and full-sequence capacities. To do so, we fix  $\alpha > 0$  and let  $P = N^{d} / (\alpha \log N)$ . As  $t$  is arbitrary, fix  $\gamma > 0$ , and let  $t = \gamma / P$ . For our choice of  $P$ , this gives
$$
t ^ {2} (N - 1) ^ {d} \log (N) ^ {2 \beta d} = \gamma^ {2} \alpha^ {2} \frac {(N - 1) ^ {d}}{N ^ {2 d}} \log (N) ^ {2 (\beta d + 1)} \tag {D.36}
$$
which is clearly less than 2 for  $N$  sufficiently large. Therefore, we can apply the bound obtained above, which for this choice of  $t$  simplifies to
$$
\begin{array}{l} \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right] \\ \leq e ^ {- t (N - 1) ^ {d}} \left\{\exp \left[ \frac {1}{2} \gamma^ {2} \alpha^ {2} (2 d - 1)!! \frac {\log (N) ^ {2}}{N ^ {d}} + \frac {1}{4} \gamma^ {4} \alpha^ {4} \frac {\log (N) ^ {4 (\beta d - 1)}}{N ^ {2 d}} \right] [ 1 + o (1) ] \right. \\ \left. + 2 \exp \left[ \left(\gamma \alpha - \frac {1}{2} \log (N) ^ {2 \beta - 1}\right) \log (N) \right] [ 1 + o (1) ] \right\} ^ {P - 1}. \tag {D.37} \\ \end{array}
$$
We can see that the first term in the curly braces tends to 1 with increasing  $N$  — as its exponent tends to zero—while the second term tends to zero as the term in the round brackets within the exponent is negative for sufficiently large  $N$  provided that  $\beta > 1/2$ . We may therefore neglect the second term, which gives the simplification
$$
\mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right] \leq \exp \left[ - \alpha \gamma \left(1 - \frac {1}{2} \gamma (2 d - 1)!!\right) \log (N) \right] [ 1 + o (1) ]. \tag {D.38}
$$
To determine the single-transition capacity under the union bound, we want  $N\mathbb{P}[T_{DN}(\pmb{\xi}^1)_1 \neq \pmb{\xi}_2^1]$  to tend to zero. We have
$$
N \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right] \leq \exp \left\{\left[ 1 - \alpha \gamma \left(1 - \frac {1}{2} \gamma (2 d - 1)!!\right) \right] \log (N) \right\} [ 1 + o (1) ]. \tag {D.39}
$$
For this bound to tend to zero, we should have
$$
1 - \alpha \gamma \left(1 - \frac {1}{2} \gamma (2 d - 1)!!\right) <   0. \tag {D.40}
$$
As  $\gamma$  is arbitrary, we may let  $\gamma = 1 / (2d - 1)!!$ , hence the required condition is clearly satisfied if
$$
\alpha > 2 (2 d - 1)!!, \tag {D.41}
$$
as predicted by the Gaussian approximation. Next, to determine the sequence capacity, we want  $NPP\mathbb{P}[T_{DN}(\pmb{\xi}^1)_1 \neq \pmb{\xi}_2^1]$  to tend to zero. We have
$$
N P \mathbb {P} \left[ T _ {D N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {2} ^ {1} \right] \leq \frac {1}{\alpha \log N} \exp \left\{\left[ d + 1 - \alpha \gamma \left(1 - \frac {1}{2} \gamma (2 d - 1)!!\right) \right] \log (N) \right\} [ 1 + o (1) ], \tag {D.42}
$$
hence an identical line of reasoning to that used for the single-transition capacity shows that we must have
$$
\alpha \geq 2 (d + 1) (2 d - 1)!!. \tag {D.43}
$$
Again, this agrees with the Gaussian theory.
E Generalized pseudoinverse rule capacity
Here, we show that the generalized pseudoinverse rule can perfectly recall any sequence of linearly-independent patterns. We recall from (11) that the GPI update rule is
$$
T _ {G P I} (\mathbf {S}) _ {i} = \operatorname {s g n} \left[ \sum_ {\mu = 1} ^ {P} \xi_ {i} ^ {\mu + 1} f \left(\sum_ {\nu = 1} ^ {P} (O ^ {+}) ^ {\mu \nu} m ^ {\nu} (\mathbf {S})\right) \right] \tag {E.1}
$$
$$
O ^ {\mu \nu} = \frac {1}{N} \sum_ {j = 1} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {\nu} \tag {E.2}
$$
the Gram matrix of the patterns. If the patterns are linearly independent, then  $\mathbf{O}$  is full rank, and the pseudoinverse reduces to the ordinary inverse:  $\mathbf{O}^{+} = \mathbf{O}^{-1}$ . Under this assumption, we have
$$
\begin{array}{l} T _ {G P I} \left(\boldsymbol {\xi} ^ {\mu}\right) _ {i} = \operatorname {s g n} \left[ \sum_ {\nu = 1} ^ {P} \xi_ {i} ^ {\nu + 1} f \left(\delta^ {\mu \nu}\right) \right] (E.3) \\ = \operatorname {s g n} \left[ f (1) \xi_ {i} ^ {\mu + 1} + f (0) \sum_ {\nu \neq \mu} \xi_ {i} ^ {\nu + 1} \right], (E.4) \\ \end{array}
$$
for all  $\mu$  and  $i$ , hence for separation functions satisfying  $f(1) > 0$  and  $|f(0)| < f(1) / (P - 1)$  we are guaranteed to have  $T_{GPI}(\pmb{\xi}^{\mu})_i = \xi_i^{\mu + 1}$  as desired. For  $f(x) = x^{d}$ , this condition is always satisfied as  $f(0) = 0$  and  $f(1) = 1$ . For  $f(x) = e^{(N - 1)(x - 1)}$ , we have  $f(0) = e^{-(N - 1)}$  and  $f(1) = 1$ ; the condition  $P - 1 < e^{N - 1}$  must therefore be satisfied. However, as  $P \leq N$  is required for linear independence, this condition is satisfied so long as  $N > 3$ .
F MixedNet Capacity
In this Appendix, we compute the capacity of the mixed network, which from the update rule defined in (14) has
$$
T _ {M N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} = \operatorname {s g n} \left\{\sum_ {\mu = 1} ^ {P} \left[ \xi_ {1} ^ {\mu} f _ {S} \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) + \lambda \xi_ {1} ^ {\mu + 1} f _ {A} \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) \right] \right\}. \tag {F.1}
$$
Then, assuming that  $f_{S}(1) = f_{A}(1) = 1$  as is true for the interaction functions considered here, we have
$$
\begin{array}{l} \mathbb {P} \left[ T _ {M N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] (F.2) \\ = \mathbb {P} \left\{\xi_ {1} ^ {2} \left[ \sum_ {\mu = 1} ^ {P} \xi_ {1} ^ {\mu} f _ {S} \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) + \lambda \sum_ {\mu = 1} ^ {P} \xi_ {1} ^ {\mu + 1} f _ {A} \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) \right] <   0 \right\} (F.3) \\ = \mathbb {P} \left\{C <   - \lambda \right\}, (F.4) \\ \end{array}
$$
where we have defined the crosstalk
$$
C = \xi_ {1} ^ {2} \xi_ {1} ^ {1} + \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {2} \xi_ {1} ^ {\mu} f _ {S} \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right) + \lambda \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {2} \xi_ {1} ^ {\mu + 1} f _ {A} \left(\frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu} \xi_ {j} ^ {1}\right). \tag {F.5}
$$
For  $j = 2, \ldots, N$  and  $\mu = 2, \ldots, P$ , we have the equality in distribution  $\xi_j^\mu \xi_j^{1} \stackrel{d}{=} \xi_j^\mu$ , hence
$$
C \stackrel {d} {=} \xi_ {1} ^ {2} \xi_ {1} ^ {1} + \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {2} \xi_ {1} ^ {\mu} f _ {S} (\Xi^ {\mu}) + \lambda \sum_ {\mu = 2} ^ {P} \xi_ {1} ^ {2} \xi_ {1} ^ {\mu + 1} f _ {A} (\Xi^ {\mu}). \tag {F.6}
$$
where to lighten our notation we define
$$
\Xi^ {\mu} = \frac {1}{N - 1} \sum_ {j = 2} ^ {N} \xi_ {j} ^ {\mu}. \tag {F.7}
$$
However, unlike in the DenseNet, we cannot similarly simplify the terms outside the separation functions. Recalling that we have assumed periodic boundary conditions, we have
$$
C = \xi_ {1} ^ {2} \xi_ {1} ^ {1} + \lambda \xi_ {1} ^ {2} \xi_ {1} ^ {1} f _ {A} (\Xi^ {P}) + f _ {S} (\Xi^ {2}) + \sum_ {\mu = 3} ^ {P} \xi_ {1} ^ {2} \xi_ {1} ^ {\mu} f _ {S} (\Xi^ {\mu}) + \lambda \sum_ {\mu = 2} ^ {P - 1} \xi_ {1} ^ {2} \xi_ {1} ^ {\mu + 1} f _ {A} (\Xi^ {\mu}) \tag {F.8}
$$
$$
\stackrel {d} {=} \xi_ {1} ^ {1} + C _ {1} + C _ {2} + C _ {3} + C _ {4}, \tag {F.9}
$$
$$
C _ {1} = f _ {S} \left(\Xi^ {2}\right) + \lambda \xi_ {1} ^ {3} f _ {A} \left(\Xi^ {2}\right), \tag {F.10}
$$
$$
C _ {2} = \xi_ {1} ^ {P} f _ {S} (\Xi^ {P}) + \lambda \xi_ {1} ^ {1} f _ {A} (\Xi^ {P}), \tag {F.11}
$$
$$
C _ {3} = \sum_ {\mu = 3} ^ {P - 1} \xi_ {1} ^ {\mu} f _ {S} (\Xi^ {\mu}), \quad \text {a n d} \tag {F.12}
$$
$$
C _ {4} = \lambda \sum_ {\mu = 3} ^ {P - 1} \xi_ {1} ^ {\mu + 1} f _ {A} (\Xi^ {\mu}). \tag {F.13}
$$
Importantly, in this case the influence of  $\xi_1^1$  on the crosstalk is  $\mathcal{O}(1)$ , and the distribution is not well-approximated by a single Gaussian. Instead, as shown in Figure F.1, it is bimodal. We will therefore approximate it by a mixture of two Gaussians, one for each value of  $\xi_1^1$ . This approximation can be justified by noting that the boundary terms in  $C_1$  and  $C_2$  should be negligible at large  $N$  and  $P$ , while  $C_3$  and  $C_4$  should give a Gaussian contribution at sufficiently large  $P$ . We now observe that, for any  $f_S$  and  $f_A$ , the conditional means of each term are
$$
\mathbb {E} \left[ C _ {1} \mid \xi_ {1} ^ {1} \right] = \mathbb {E} \left[ f _ {S} (\Xi) \right] \tag {F.14}
$$
$$
\mathbb {E} \left[ C _ {2} \mid \xi_ {1} ^ {1} \right] = \lambda \xi_ {1} ^ {1} \mathbb {E} \left[ f _ {A} (\Xi) \right] \tag {F.15}
$$
$$
\mathbb {E} \left[ C _ {3} \mid \xi_ {1} ^ {1} \right] = 0 \tag {F.16}
$$
$$
\mathbb {E} \left[ C _ {4} \mid \xi_ {1} ^ {1} \right] = 0, \tag {F.17}
$$
where we note that all  $\Xi^{\mu}\mathrm{s}$  are identically distributed, so we can simply write  $\Xi$  for any one of them. Then, the conditional mean of the crosstalk is
$$
\mathbb {E} [ C \mid \xi_ {1} ^ {1} ] = \xi_ {1} ^ {1} + \sum_ {j = 1} ^ {4} \mathbb {E} [ C _ {j} \mid \xi_ {1} ^ {1} ] \tag {F.18}
$$
$$
= \xi_ {1} ^ {1} \{1 + \lambda \mathbb {E} [ f _ {A} (\Xi) ] \} + \mathbb {E} [ f _ {S} (\Xi) ]. \tag {F.19}
$$
Considering the variance of  $C$ , the variances of the different contributions are
$$
\operatorname {v a r} \left[ C _ {1} \mid \xi_ {1} ^ {1} \right] = \operatorname {v a r} \left[ f _ {S} (\Xi) \right] + \lambda^ {2} \mathbb {E} \left[ f _ {A} (\Xi) ^ {2} \right] \tag {F.20}
$$
$$
\operatorname {v a r} \left[ C _ {2} \mid \xi_ {1} ^ {1} \right] = \mathbb {E} \left[ f _ {S} (\Xi) ^ {2} \right] + \lambda^ {2} \operatorname {v a r} \left[ f _ {A} (\Xi) \right] \tag {F.21}
$$
$$
\operatorname {v a r} \left[ C _ {3} \mid \xi_ {1} ^ {1} \right] = (P - 3) \mathbb {E} \left[ f _ {S} (\Xi) ^ {2} \right] \tag {F.22}
$$
$$
\operatorname {v a r} \left[ C _ {4} \mid \xi_ {1} ^ {1} \right] = \lambda^ {2} (P - 3) \mathbb {E} \left[ f _ {A} (\Xi) ^ {2} \right], \tag {F.23}
$$
while the covariances are
$$
\operatorname {c o v} \left[ C _ {1}, C _ {2} \mid \xi_ {1} ^ {1} \right] = 0 \tag {F.24}
$$
$$
\operatorname {c o v} \left[ C _ {1}, C _ {3} \mid \xi_ {1} ^ {1} \right] = \lambda \mathbb {E} \left[ f _ {A} (\Xi) \right] \mathbb {E} \left[ f _ {S} (\Xi) \right] \tag {F.25}
$$
$$
\operatorname {c o v} \left[ C _ {1}, C _ {4} \mid \xi_ {1} ^ {1} \right] = 0, \tag {F.26}
$$
MixedNet Crosstalk
Figure F.1: Crosstalk of Polynomial MixedNet where  $N = 100$ ,  $\lambda = 2.5$ ,  $d_{S} = d_{A} = 3$  and  $P = 1000$  patterns are stored. Histograms are generated for patterns drawn from 5000 randomly sequences and theoretical curves are plotted. Green represents the full crosstalk for the MixedNet. Blue and red represent the asymmetric and symmetric terms of the crosstalk, respectively. Observe that the bimodality in the full model comes from bimodality in the symmetric term.
$$
\operatorname {c o v} \left[ C _ {2}, C _ {3} \mid \xi_ {1} ^ {1} \right] = 0 \tag {F.27}
$$
$$
\operatorname {c o v} \left[ C _ {2}, C _ {4} \mid \xi_ {1} ^ {1} \right] = \lambda \mathbb {E} \left[ f _ {S} (\Xi) \right] \mathbb {E} \left[ f _ {A} (\Xi) \right], \tag {F.28}
$$
$$
\begin{array}{l} \operatorname {c o v} \left[ C _ {3}, C _ {4} \mid \xi_ {1} ^ {1} \right] = \lambda \sum_ {\mu , \nu = 3} ^ {P - 1} \mathbb {E} \left[ \xi_ {1} ^ {\mu} \Xi_ {1} ^ {\nu + 1} \right] \mathbb {E} \left[ f _ {S} \left(\Xi^ {\mu}\right) f _ {A} \left(\Xi^ {\nu}\right) \right] (F.29) \\ = \lambda \sum_ {\mu = 3} ^ {P} \mathbb {E} \left[ f _ {S} \left(\Xi^ {\mu}\right) \right] \mathbb {E} \left[ f _ {A} \left(\Xi^ {\mu - 1}\right) \right] (F.30) \\ = \lambda (P - 3) \mathbb {E} [ f _ {S} (\Xi) ] \mathbb {E} [ f _ {A} (\Xi) ]. (F.31) \\ \end{array}
$$
Therefore, the conditional variance of the crosstalk is
$$
\begin{array}{l} \operatorname {v a r} \left[ C \mid \xi_ {1} ^ {1} \right] = \sum_ {j = 1} ^ {4} \operatorname {v a r} \left[ C _ {j} \mid \xi_ {1} ^ {1} \right] + 2 \sum_ {j = 1} ^ {4} \sum_ {k > j} \operatorname {c o v} \left[ C _ {j}, C _ {k} \mid \xi_ {1} \right] (F.32) \\ = (P - 3) \{\mathbb {E} [ f _ {S} (\Xi) ^ {2} ] + 2 \lambda \mathbb {E} [ f _ {S} (\Xi) ] \mathbb {E} [ f _ {A} (\Xi) ] + \lambda^ {2} \mathbb {E} [ f _ {A} (\Xi) ^ {2} ] \} \\ + \operatorname {v a r} \left[ f _ {S} (\Xi) \right] + \lambda^ {2} \mathbb {E} \left[ f _ {A} (\Xi) ^ {2} \right] + \mathbb {E} \left[ f _ {S} (\Xi) ^ {2} \right] + \lambda^ {2} \operatorname {v a r} \left[ f _ {A} (\Xi) \right] + 4 \lambda \mathbb {E} \left[ f _ {A} (\Xi) \right] \mathbb {E} \left[ f _ {S} (\Xi) \right] (F.33) \\ = (P - 1) \{\mathbb {E} [ f _ {S} (\Xi) ^ {2} ] + 2 \lambda \mathbb {E} [ f _ {S} (\Xi) ] \mathbb {E} [ f _ {A} (\Xi) ] + \lambda^ {2} \mathbb {E} [ f _ {A} (\Xi) ^ {2} ] \} \\ - \mathbb {E} [ f _ {S} (\Xi) ] ^ {2} - \lambda^ {2} \mathbb {E} [ f _ {A} (\Xi) ] ^ {2}. (F.34) \\ \end{array}
$$
For large  $P$  and  $N$ , the two terms on the second line of this result will be subleading, as they do not scale with  $P$  and have identical or subleading scaling with  $N$  to the terms that do scale with  $P$ . That is, we have
$$
\operatorname {v a r} \left[ C \mid \xi_ {1} ^ {1} \right] \sim P \left\{\mathbb {E} \left[ f _ {S} (\Xi) ^ {2} \right] + 2 \lambda \mathbb {E} \left[ f _ {S} (\Xi) \right] \mathbb {E} \left[ f _ {A} (\Xi) \right] + \lambda^ {2} \mathbb {E} \left[ f _ {A} (\Xi) ^ {2} \right] \right\}. \tag {F.35}
$$
Collecting these results, we have
$$
\mathbb {E} [ C \mid \xi_ {1} ^ {1} ] = \xi_ {1} ^ {1} \{1 + \lambda \mathbb {E} [ f _ {A} (\Xi) ] \} + \mathbb {E} [ f _ {S} (\Xi) ] \tag {F.36}
$$
$$
\operatorname {v a r} \left[ C \mid \xi_ {1} ^ {1} \right] \sim P \left\{\mathbb {E} \left[ f _ {S} (\Xi) ^ {2} \right] + 2 \lambda \mathbb {E} \left[ f _ {S} (\Xi) \right] \mathbb {E} \left[ f _ {A} (\Xi) \right] + \lambda^ {2} \mathbb {E} \left[ f _ {A} (\Xi) ^ {2} \right] \right\}. \tag {F.37}
$$
By the law of total probability, we have
$$
\mathbb {P} \left[ T _ {M N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] = \mathbb {P} [ C <   - \lambda ] \tag {F.38}
$$
$$
= \frac {1}{2} \mathbb {P} [ C <   - \lambda | \xi_ {1} ^ {1} = - 1 ] + \frac {1}{2} \mathbb {P} [ C <   - \lambda | \xi_ {1} ^ {1} = + 1 ] \tag {F.39}
$$
$$
\sim \frac {1}{2} H \left(\frac {\lambda + \mathbb {E} [ C \mid \xi_ {1} ^ {1} = - 1 ]}{\sqrt {\operatorname {v a r} [ C \mid \xi_ {1} ^ {1} = - 1 ]}}\right) + \frac {1}{2} H \left(\frac {\lambda + \mathbb {E} [ C \mid \xi_ {1} ^ {1} = + 1 ]}{\sqrt {\operatorname {v a r} [ C \mid \xi_ {1} ^ {1} = + 1 ]}}\right) \tag {F.40}
$$
under the bimodal Gaussian approximation to the crosstalk distribution. To have  $\mathbb{P}[T_{MN}(\pmb{\xi}^1)_1 \neq \xi_1^2]$ , both of these conditional probabilities must tend to zero. By basic concentration arguments, we expect to have
$$
\mathbb {E} \left[ C \mid \xi_ {1} ^ {1} \right] \sim \xi_ {1} ^ {1} \tag {F.41}
$$
up to corrections that are small in an absolute sense. Moreover, we have
$$
\mathbb {E} [ C \mid \xi_ {1} ^ {1} = + 1 ] - \mathbb {E} [ C \mid \xi_ {1} ^ {1} = - 1 ] = 2 \{1 + \lambda \mathbb {E} [ f _ {A} (\Xi) ] \} \tag {F.42}
$$
which for the separation functions considered here is strictly positive. As we keep  $\lambda$  constant with  $N$  and  $P$ , we must have
$$
\mathbb {E} [ C \mid \xi_ {1} ^ {1} = - 1 ] > - \lambda \tag {F.43}
$$
and  $\operatorname{var}[C|\xi_1^1 = -1] \to 0$  in order to have  $\mathbb{P}[T_{MN}(\pmb{\xi}^1)_1 \neq \xi_1^2] \to 0$ . But, given the formula above,  $\operatorname{var}[C|\xi_1^1 = -1] = \operatorname{var}[C|\xi_1^1 = +1]$ , so this implies that the  $\xi_1^1 = +1$  contribution to the probability will be exponentially suppressed. Then, we can apply an identical argument to that which we used for the DenseNet in Appendix C to obtain the asymptotic behavior of  $\mathbb{P}[C < -\lambda|\xi_1^1 = -1]$ , yielding
$$
\mathbb {P} \left[ T _ {M N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \frac {1}{2} H \left(\frac {\lambda + \mathbb {E} [ C \mid \xi_ {1} ^ {1} = - 1 ]}{\sqrt {\operatorname {v a r} [ C \mid \xi_ {1} ^ {1} = - 1 ]}}\right) \tag {F.44}
$$
$$
\sim \frac {1}{2 \sqrt {2 \pi}} \frac {\sqrt {\operatorname {v a r} [ C \mid \xi_ {1} ^ {1} = - 1 ]}}{\lambda + \mathbb {E} [ C \mid \xi_ {1} ^ {1} = - 1 ]} \exp \left(- \frac {1}{2} \frac {(\lambda + \mathbb {E} [ C \mid \xi_ {1} ^ {1} = - 1 ]) ^ {2}}{\operatorname {v a r} [ C \mid \xi_ {1} ^ {1} = - 1 ]}\right). \tag {F.45}
$$
For this to work, we must clearly have  $\lambda > 1$ .
We could in principle compute the excess kurtosis of the crosstalk for the MixedNet as we did for the DenseNet, but we will not do so here as the computation would be tedious and would not yield substantial new insight beyond that for the DenseNet.
F.1 Polynomial MixedNet
We first consider the polynomial mixed network, with  $f_{S}(x) = x^{d_{S}}$  and  $f_{A}(x) = x^{d_{A}}$  for two possibly differing degrees  $d_{S}, d_{A} \in \mathbb{N}_{>0}$ . We can apply the same reasoning as in Appendix C.1 to obtain the required moments at large  $N$ , which yields the first moments
$$
\mathbb {E} \left[ f _ {S} (\Xi) \right] = \mathbb {E} \left[ \Xi^ {d _ {S}} \right] = \left\{ \begin{array}{l l} 0 & d _ {S} \text {o d d}, \\ \frac {\left(d _ {S} - 1\right) ! !}{N ^ {d _ {S} / 2}} \left[ 1 + \mathcal {O} \left(\frac {1}{N}\right) \right] & d _ {S} \text {e v e n} \end{array} \right. \tag {F.46}
$$
$$
\mathbb {E} \left[ f _ {A} (\Xi) \right] = \mathbb {E} \left[ \Xi^ {d _ {A}} \right] = \left\{ \begin{array}{l l} 0 & d _ {A} \text {o d d}, \\ \frac {\left(d _ {A} - 1\right) ! !}{N ^ {d _ {A} / 2}} \left[ 1 + \mathcal {O} \left(\frac {1}{N}\right) \right] & d _ {A} \text {e v e n}, \end{array} \right. \tag {F.47}
$$
S21
and the second moments
$$
\mathbb {E} \left[ f _ {S} (\Xi) ^ {2} \right] = \mathbb {E} \left[ \Xi^ {2 d _ {S}} \right] = \frac {\left(2 d _ {S} - 1\right) ! !}{N ^ {d _ {S}}} \left[ 1 + \mathcal {O} \left(\frac {1}{N}\right) \right] \tag {F.48}
$$
$$
\mathbb {E} \left[ f _ {A} (\Xi) ^ {2} \right] = \mathbb {E} \left[ \Xi^ {2 d _ {A}} \right] = \frac {\left(2 d _ {A} - 1\right) ! !}{N ^ {d _ {A}}} \left[ 1 + \mathcal {O} \left(\frac {1}{N}\right) \right]. \tag {F.49}
$$
Then, the conditional mean of the crosstalk is given by
$$
\mathbb {E} [ C \mid \xi_ {1} ^ {1} ] \sim \xi_ {1} ^ {1} \tag {F.50}
$$
up to corrections which vanish in an absolute, not a relative, sense, while the conditional variance is asymptotic to
$$
\operatorname {v a r} [ C \mid \xi_ {1} ^ {1} ] \sim P \left\{\frac {(2 d _ {S} - 1) ! !}{N ^ {d _ {S}}} + 2 \lambda \frac {(d _ {S} - 1) ! ! (d _ {A} - 1) ! !}{N ^ {(d _ {S} + d _ {A}) / 2}} \mathbf {1} \{d _ {S}, d _ {A} \text {e v e n} \} + \lambda^ {2} \frac {(2 d _ {A} - 1) ! !}{N ^ {d _ {A}}} \right\}. \tag {F.51}
$$
We must now determine the storage capacity. We recall that, in all case, we want  $P$  to tend to infinity slowly enough that  $\mathrm{var}[C|\xi_1^1]$  tends to zero. Then, we can see that what matters is which of the terms inside the curly brackets in the expression for the conditional variance above tends to zero with  $N$  the slowest. This is of course determined by  $\min\{d_S, d_A\}$ , but the constant factor multiplying the leading term will depend on which is smaller, or if they are equal. First, consider the case in which  $d_S = d_A = d$ . Then, we have
$$
\operatorname {v a r} [ C \mid \xi_ {1} ^ {1} ] \sim \frac {P}{N ^ {d}} \left\{\left(2 d - 1\right)!! + 2 \lambda (d - 1)!! \left(d - 1\right)!! \mathbf {1} \{d \text {e v e n} \} + \lambda^ {2} (2 d - 1)!! \right\}. \tag {F.52}
$$
Now, consider the case in which  $d_S < d_A$ . Then,  $(d_S + d_A) / 2 > d_S$ , hence the  $N^{-d_S}$  term dominates and we have
$$
\operatorname {v a r} \left[ C \mid \xi_ {1} ^ {1} \right] \sim \frac {P}{N ^ {d _ {S}}} (2 d _ {S} - 1)!!. \tag {F.53}
$$
Similarly, if  $d_A > d_S$ , the  $N^{-d_A}$  term dominates, and we have
$$
\operatorname {v a r} \left[ C \mid \xi_ {1} ^ {1} \right] \sim \frac {P}{N ^ {d _ {A}}} \lambda^ {2} (2 d _ {A} - 1)!!. \tag {F.54}
$$
We can summarize these results as
$$
\operatorname {v a r} \left[ C \mid \xi_ {1} ^ {1} \right] \sim \gamma_ {d _ {S}, d _ {A}} \frac {P}{N ^ {\min  \left\{d _ {S} , d _ {A} \right\}}}, \tag {F.55}
$$
$$
\gamma_ {d _ {S}, d _ {A}} = \left\{ \begin{array}{l l} (2 d _ {S} - 1)!! & \text {i f} d _ {S} <   d _ {A}, \\ (\lambda^ {2} + 1) (2 d _ {S} - 1)!! + 2 \lambda [ (d _ {S} - 1)!! ] ^ {2} \mathbf {1} \left\{d _ {S} \text {e v e n} \right\} & \text {i f} d _ {S} = d _ {A}, \\ \lambda^ {2} (2 d _ {A} - 1)!! & \text {i f} d _ {S} > d _ {A}. \end{array} \right. \tag {F.56}
$$
Using the general arguments presented above, we then have
$$
\mathbb {P} [ T _ {M N} (\pmb {\xi} ^ {1}) _ {1} \neq \xi_ {1} ^ {2} ] \sim \frac {1}{2 \sqrt {2 \pi}} \sqrt {\frac {\gamma_ {d _ {S} , d _ {A}} P}{(\lambda - 1) ^ {2} N ^ {\min  \{d _ {S} , d _ {A} \}}}} \exp \left(- \frac {(\lambda - 1) ^ {2}}{2} \frac {N ^ {\min  \{d _ {S} , d _ {A} \}}}{\gamma_ {d _ {S} , d _ {A}} P}\right).
$$
for any  $\lambda > 1$ . We must first determine the single-transition capacity, which requires that  $N\mathbb{P}[T_{MN}(\pmb{\xi}^1)_1 \neq \xi_1^2] \to 0$ . Recalling that our argument requires us to take  $P \to \infty$  slowly enough that  $\operatorname{var}[C | \xi_1^1] \to 0$ , we make the Ansatz that
$$
P \sim \frac {(\lambda - 1) ^ {2}}{\alpha \gamma_ {d _ {S} , d _ {A}}} \frac {N ^ {\min  \left\{d _ {S} , d _ {A} \right\}}}{\log N} \tag {F.58}
$$
S22
for some  $\alpha$ . This yields
$$
N \mathbb {P} \left[ T _ {M N} \left(\boldsymbol {\xi} ^ {1}\right) _ {1} \neq \xi_ {1} ^ {2} \right] \sim \frac {1}{2 \sqrt {2 \pi \alpha \log N}} N ^ {1 - \alpha / 2}, \tag {F.59}
$$
which tends to zero if  $\alpha \geq 2$ , yielding a predicted capacity of
$$
P _ {T} \sim \frac {(\lambda - 1) ^ {2}}{2 \gamma_ {d _ {S} , d _ {A}}} \frac {N ^ {\min  \left\{d _ {S} , d _ {A} \right\}}}{\log N}. \tag {F.60}
$$