SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Milad Sefidgaran†, Abdellatif Zaidi††, Piotr Krasnowski†
† Paris Research Center, Huawei Technologies France † Université Gustave Eiffel, France {milad.sefidgaran2,piotr.g.krasnowski}@huawei.com, abdellatif.zaidi@univ-eiffel.fr
A major challenge in designing efficient statistical supervised learning algorithms is finding representations that perform well not only on available training samples but also on unseen data. While the study of representation learning has spurred much interest, most existing such approaches are heuristic; and very little is known about theoretical generalization guarantees. For example, the information bottleneck method seeks a good generalization by finding a minimal description of the input that is maximally informative about the label variable, where minimality and informativeness are both measured by Shannon's mutual information.
In this paper, we establish a compressibility framework that allows us to derive upper bounds on the generalization error of a representation learning algorithm in terms of the "Minimum Description Length" (MDL) of the labels or the latent variables (representations). Rather than the mutual information between the encoder's input and the representation, which is often believed to reflect the algorithm's generalization capability in the related literature but in fact, falls short of doing so, our new bounds involve the "multi-letter" relative entropy between the distribution of the representations (or labels) of the training and test sets and a fixed prior. In particular, these new bounds reflect the structure of the encoder and are not vacuous for deterministic algorithms. Our compressibility approach, which is information-theoretic in nature, builds upon that of Blum-Langford for PAC-MDL bounds and introduces two essential ingredients: block-coding and lossy-compression. The latter allows our approach to subsume the so-called geometrical compressibility as a special case. To the best knowledge of the authors, the established generalization bounds are the first of their kind for Information Bottleneck type encoders and representation learning. Finally, we partly exploit the theoretical results by introducing a new data-dependent prior. Numerical simulations illustrate the advantages of well-chosen such priors over classical priors used in IB.
A key performance indicator of stochastic learning algorithms is their capability to generalize, i.e., perform equally well on training and unseen data. However, designing learning algorithms with good generalization guarantees remains a major challenge. A popular approach involves learning an encoder part and a decoder part. The encoder aims at generating a suitable representation of the input (referred to as "latent variable") by extracting relevant features from the input data. The decoder aims at optimizing the performance of the learning task for the given training dataset, known as empirical risk minimization (ERM), based only on the learned latent variables. This approach is grounded on the idea that performing ERM on the latent variable (instead of the input itself) prevents overfitting.
Information Bottleneck. Several approaches have attempted to formalize the concept of a "good representation" [SST10, AFDM17, VDOV $^{+}$ 17, DKSV20]. Perhaps, the most prominent is the information bottleneck (IB) method which was first introduced in [TPB00] and then extended in several directions [SST10, AFDM17, AZ19, KTW19, Fis20, RGTS20, KASK22]. The IB approach was deemed useful to analyze deep neural networks [SZT17, ZEAS20, GP20, Gei21] and guide their training process. For
instance, in supervised learning tasks the IB seeks representations that capture "minimum" information about the input data (shoots for good generalization power) while providing "maximum" information about the label (shoots for small empirical risk), where the amounts of captured and provided information are measured using Shannon mutual information. More precisely, denoting by  $Y$  the label variable and by  $X$  the input data, the IB latent variable  $U$  is chosen so as to maximize the mutual information  $I(U;Y)$  while minimizing the mutual information  $I(U;X)$ . Equivalently, this can be formulated as maximizing the Lagrange cost
$$
I (U; Y) - \beta I (U; X),
$$
where  $\beta \geqslant 0$  designates a Lagrange multiplier. Let  $(X^m, U^m)$  denote a vector, or block, of  $m$  independent and identically distributed (i.i.d.) realizations  $(X_i, U_i), i = 1, \ldots, m$ , of discrete variables  $(X, U) \sim P_{X,U}$ . Fix a distribution  $P_{\hat{U}}$  such that  $P_{X,\hat{U}} = P_{X,U}$ . Essentially by means of the rate-distortion theoretic covering lemma [CT06], it is easy to see that one can get a suitable description  $\hat{U}$  of  $X$  using only  $I(U; X)$  bits per symbol. Precisely, generating a codebook of roughly  $l_m \approx 2^{mI(U; X)}$  vectors  $\hat{u}^m[j] \in \mathcal{U}^m$ ,  $j = 1, \ldots, m$ , all drawn i.i.d. according to  $P_U$ , the covering lemma states that for large  $m$  there exists at least one index  $j$  for which the empirical distribution of  $(X^m, \hat{u}^m[j])$  is arbitrary close to  $P_{X,U}$  in some suitable sense. That is, the empirical distributions of  $(X^m, U^m)$  and  $(X^m, \hat{u}^m[j])$  are close. Hence, an "equivalent" version  $\hat{U}^m$  of  $U^m$  can be described with roughly  $mI(U; X)$  bits. Intuitively, this makes a connection between the mutual information  $I(U; X)$  and the concept of Minimal Description Length (MDL) [Ris78, GMP05] when one considers MDL of the latent variables, not that of model parameters [VPV18, ZEAS20]. Moreover, it is now relatively well known that there exists a connection between the generalization error of a learning model and the MDL of the parameters of that model, see, e.g., [BEHW87, BO18, GR19]. A notable work [BL03] has considered MDL of the predicted labels of a super-sample of training and test data.
Critics to IB. The aforementioned connections perhaps formed a belief that  $I(U;X)$  is closely related to the generalization performance of representation learning algorithms. This belief, however, is controversial and conflicting evidence has been reported [GK19]. For instance, using the term  $I(U;X)$  as a regularizer has been criticized for four main reasons [KTVK18, RG19, AG19, DKSV20]: i. The few existing upper bounds on the generalization error which involve (among other terms) the mutual information  $I(U;X)$  reported in [VPV18] and the very recent and concurrent work [KDJH23] are not convincing. For example, the bound of [VPV18] holds only when the alphabets of the input and latent spaces are finite; and, in that case, it states that for any  $\delta > 0$ , with probability  $1 - \delta$  it holds that:  $\mathrm{gen}(S,W) \leqslant \mathcal{O}\left(\frac{\log(n)}{n}\right)\sqrt{I(U;X)} + C_{\delta}$ , where  $\mathrm{gen}(S,W)$  is the generalization error of model  $W$ ,  $C_{\delta} := \mathcal{O}(|\mathcal{U}| / \sqrt{n})$ ,  $|\mathcal{U}|$  is the size of the latent space and  $n$  is the size of the training dataset. For reasonable setups, however, the term  $C_{\delta}$  dominates and their bound becomes vacuous [RG19, LLS+23]. The bound reported in [KDJH23] suffers similar shortcomings – For further details on this, see Appendix B.2. ii. Experimental evidence shows dependence of the generalization error on the so-called geometrical compression rather than on  $I(U;X)$  [GK19]. Geometrical compression occurs when the latent variables are concentrated around a limited number of clusters. See [GK19, Fig. 2] for a visual representation. iii.  $I(U;X)$  is invariant to bijection; and, as such, it does not favor learning algorithms/representations with simple decision boundaries and does not reflect the “structure” or “simplicity” of the encoder/decoder. Please refer to [AG19, Section 4.3] for a detailed discussion and examples. iv. Finally, for deterministic algorithms  $I(U;X)$  can take large or even infinite values, especially for continuous variables or high-dimensional data, hence limiting the usefulness of IB [KTVK18, AG19].
MDL/Compressibility. Several works have studied MDL/compressibility to establish generalization bounds. In this context, key is the "Occam's razor" principle [LW86, BEHW87, BL03] which, e.g., for binary classification tasks, states that if the labels of the training data  $S = \{Z_1,\dots,Z_n\}$ ,  $Z_i = (X_i,Y_i)$  predicted by a model  $W$  can be described by  $k \ll n$  number of bits, then the learned model  $W$  has a good generalization performance. There exist three closely related lines of work that use different means to describe the dataset labels: i. The first describes the predicted labels via the hypothesis (parameter models). It includes the works of [BEHW87, AGNZ18, SAM+20, HJTW21, BSE+21, SGRS22]. Recently, it was shown in [SGRS22, SZ23] that information-theoretic bounds [RZ16, XR17, SZ20], PAC-Bayes bounds [McA98] and intrinsic dimension-based approaches [§SDE20] also fall into this category. The proof uses so-called "fixed-size" [SGRS22] and "variable-size" [SZ23] compressibility frameworks introduced therein. ii. The second formulates the minimal description of labels as follows: if a learning algorithm can predict all labels of the training data  $S$  by using only samples from a small subset of  $S$ , then the learning algorithm is guaranteed to generalize well. This second approach was initiated by [LW86] and followed by others including [HK19, HKS19, BHMZ20, HK21, HKSW20, CK22]. iii. The third, initiated by [BL03], deals directly with the compression of the predicted labels. This approach is shown to be related to the previous two lines, and also to PAC-Bayes and VC-dimension-based results. Moreover, it is the closest to the problem
(a) Single-step prediction model.
Figure 1: Considered learning frameworks.
(b) Two-step prediction model.
of establishing bounds on the generalization error of representation learning algorithms in terms of the compressibility of the latent variables. This approach is detailed in Appendix B.1.
In this work we aim at understanding the theory of representation learning through a rigorous investigation of the connection between the MDL (or compressibility) of the latent variable and the generalization performance of representation learning algorithms. In doing so, we first study the single-step prediction model of Fig. 1a; and, then, we leverage the developed tools to study the two-step (encoder-decoder) prediction model of Fig. 1b. We also examine the claimed relationship between MDL and  $I(U;X)$ .
Contributions. Specifically, the main contributions of this work are as follows.
- For the prediction model of Fig. 1a, inspired by [BL03] we establish an information-theoretic framework that allows us to measure the compressibility (MDL) of the predicted labels in terms of a new object which is the KL-divergence of a vector of labels and any arbitrary symmetric prior. For a proper choice of prior, this new measure reduces to the sample-wise mutual information and hence inherits the associated properties. However, unlike the sample-wise mutual information, it can also reflect the structure/simplicity of the learning algorithm. Furthermore, by extending the framework to "lossy compressibility", this new measure does not become vacuous when one considers continuous variables instead of discrete labels. Moreover, it subsumes geometrical compressibility as a special case.
- We also establish both in-expectation and tail generalization bounds in terms of this compressibility measure of the predicted labels. In a simple case, the in-expectation bound can be cast into the form
$$
\sqrt {\frac {2 \times \operatorname {M D L} (\text {P r e d i c t e d L a b e l s})}{n}}.
$$
The tail bound involves a similar expression. This generalization bound easily recovers the VC-dimension bound as a special case; and, hence, it also shows how the introduced compressibility measure depends on the complexity of the hypothesis class.
- Our results make a connection between the compressibility framework of [BL03] and the functional conditional mutual information (f-CMI) of [HRVSG21, HD22], which itself is an extension of the CMI-framework developed by [SZ20]. In particular, this connection shows how the f-CMI framework can be leveraged to study the compressibility of the predicted labels.
- For the encoder-decoder prediction model of Fig. 1b, the framework is extended non-trivially to the case in which instead of the compressibility of the predicted labels it is the compressibility of the latent variable which is considered. The results for this prediction model, which are our main results in this paper as given in Section 3, are generalization bounds for the representation learning algorithms in terms of the complexity of the latent variable. The established in-expectation and tail bounds hold for any decoder; and, for the  $K$ -class classification task for example, take the form
$$
2 \sqrt {\frac {2 \times \operatorname {M D L} (\text {L a t e n t V a r i a b l e s}) + K + 2}{n}}.
$$
These bounds appear to be the first of their kind for the studied representation learning setup. In part, their utility lies in that: i) they reflect the structure of the encoder class, ii) they do not become vacuous for deterministic encoders with continuous latent space and iii) they can explain the geometrical compressibility phenomenon. Thus, our framework reveals that the "joint" MDL of the latent space is more related to the encoder structure, rather than the mutual information  $I(U;X)$ .
- Finally, our results suggest that data-dependent priors can be used in place of the data-independent prior of the popular variational IB (VIB) method. We conduct experiments that illustrate the advantage brought up by proper choices of data-dependent priors over VIB.
Our results also open up several future directions as discussed in Appendix C.4.
Notation. Random variables, their realizations, and their alphabets are denoted respectively by upper-case letters, lower-case letters, and Calligraphy fonts, e.g.,  $X$ ,  $x$ , and  $\mathcal{X}$ . Their distributions and expectations are denoted by  $P_{X}$  and  $\mathbb{E}[X]$ . For ease of presentation, when  $\mathcal{X}$  is a discrete set,  $P_{X}$  is a probability mass function. Otherwise,  $P_{X}$  is a probability density function. A collection of  $n$  random variables  $(X_{1},\ldots ,X_{n})$  is denoted by  $X^n$  or  $\mathbf{X}$ . We use the notation  $\{x_i\}_{i = 1}^m$  to denote a sequence of  $m$  real or natural numbers. The set  $\{1,\dots ,K\}$ ,  $K\in \mathbb{N}$ , is denoted by  $[K]$ . Finally,  $\mathbb{R}^{+}$  denotes the set of non-negative real numbers. Our results are expressed in terms of information-theoretic functions. For two distributions  $P$  and  $Q$ , the Kullback-Leibler (KL) divergence is defined as  $D_{KL}(P\| Q)\coloneqq \mathbb{E}_P[\log (P / Q)]$  if  $P\ll Q$ , and  $\infty$  otherwise. The mutual information between two random variables  $X,Y\sim P_{X,Y}$  with marginals  $P_{X}$  and  $P_{Y}$  is defined as  $I(X;Y)\coloneqq D_{KL}(P_{X,Y}\| P_{X}P_{Y})$ . The reader is referred to [CT06, CK11] for further information.
Our results in this paper will be expressed in terms of symmetric conditional priors. The following definition formalizes three types of symmetry.
Definition 1 (Symmetric Priors). For  $U, V \sim P_{U,V}$ , let  $(U^{2n}, V^{2n})$  be vectors composed of  $2n$  i.i.d. instances  $(U_i, V_i) \sim P_{U,V}$ ,  $i \in [2n]$ . For any permutation  $\pi \colon [2n] \to [2n]$ , denote  $U_{\pi}^{2n} \coloneqq (U_{\pi(1)}, \ldots, U_{\pi(2n)})$  and  $V_{\pi}^{2n} \coloneqq (V_{\pi(1)}, \ldots, V_{\pi(2n)})$ .
- Type-I symmetry: Define type-I permutations as the set of permutations  $\pi \colon [2n] \to [2n]$  having the property that for any  $i \in [n]$ , the sets  $\{\pi(i), \pi(i + n)\}$  and  $\{i, i + n\}$  are equal. The conditional prior  $\mathbf{Q}(U^{2n}|V^{2n})$  has type-I symmetry if  $\mathbf{Q}(U_{\pi}^{2n}|V_{\pi}^{2n})$  is invariant to arbitrary type-I permutations. This definition first appeared in [Aud04] (called "almost exchangeable prior" therein) and was used for the CMI framework in [GSZ21].
- Type-II symmetry: The conditional prior  $\mathbf{Q}(U^{2n}|V^{2n})$  has type-II symmetry if  $\mathbf{Q}(U_{\pi}^{2n}|V_{\pi}^{2n})$  is invariant to any arbitrary permutations  $\pi \colon [2n] \to [2n]$ .
- Type-III symmetry: For a random variable  $Z \sim P_{Z|U^{2n},V^{2n}}$ , the conditional prior  $\mathbf{Q}(U^{2n}|V^{2n},Z)$  has type-III symmetry if  $\mathbf{Q}(U_{\pi}^{2n}|V^{2n},Z)$  is invariant to any permutation  $\pi \colon [2n] \to [2n]$  having the property that  $V_{i} = V_{\pi (i)}$  for every  $i \in [2n]$ .
Throughout, if the underlying  $(U^{2n},V^{2n})$  is clear from the context, for ease of the notation the corresponding sets of Type-I and Type-II priors will be denoted simply as  $\mathcal{Q}_i$  and  $\mathcal{Q}_{ii}$  respectively.
Problem setup. Unless indicated otherwise, we consider the  $K$ -class classification setup. Let  $Z = (X,Y)$  be some input data taking value over the input space  $\mathcal{Z} = \mathcal{X}\times \mathcal{Y}$  according to an unknown distribution  $\mu$ . We call  $X\in \mathcal{X}$  the features of the data, and  $Y\in \mathcal{Y}$  its label, where  $\mathcal{Y} = [K]$ . We assume a training dataset  $S = \{Z_1,\ldots ,Z_n\} \sim \mu^{\otimes n} \eqqcolon P_S$ , composed of  $n$  i.i.d. samples  $Z_{i} = (X_{i},Y_{i})$  of the input data, is available. We denote the features and labels of  $S$  by  $\mathbf{X} \coloneqq X^n \sim \mu_X^{\otimes n}$  and  $\mathbf{Y} \coloneqq Y^n \sim \mu_Y^{\otimes n}$ , respectively. We often use also a ghost or test dataset  $S' = \{Z_1',\dots,Z_n'\} \sim \mu^{\otimes n} \eqqcolon P_{S'}$ , where  $Z_i' = (X_i',Y_i')$ . Similarly, we denote the features and labels of  $S'$  by  $\mathbf{X}' \coloneqq X'^n \sim \mu_X^{\otimes n}$  and  $\mathbf{Y}' \coloneqq Y'^n \sim \mu_Y^{\otimes n}$ , respectively.
2 Generalization bounds in terms of predicted label complexity
In this section, we formulate a compressibility framework that allows us to derive upper bounds on the generalization error of a representation learning algorithm. Our proposed framework can be seen as a suitable generalization of the framework of Blum and Langford [BL03], in which the generalization encompasses lossy compression of the predicted labels and which exploits block-coding. However, unlike in the original framework based on a (compression) game between two agents Alice and Bob (see Appendix B.1 for details), this work adopts a rate-distortion theoretic perspective.
For ease of exposition, the results are presented for classification problems and the 0-1 loss function, but they can be extended trivially to any continuous  $\mathcal{V}$  and bounded loss function. Consider the setup in Fig. 1a. Let  $\mathcal{A}\colon \mathcal{Z}^n\to \mathcal{W}$  be a possibly stochastic learning algorithm. That is, for a given  $S = (Z_{1},\ldots ,Z_{n})\in \mathcal{Z}^{n}$  the algorithm picks a hypothesis or model  $W = \mathcal{A}(S)\in \mathcal{W}$ . Also, let the induced joint distribution over  $S\times \mathcal{W}$  be denoted as  $P_{S,W}$ ; and the induced conditional distribution over  $\mathcal{W}$  given  $S$  be denoted as  $P_{W|S}$ . For every input data  $z = (x,y)\in \mathcal{Z}$ , every choice of hypothesis  $w\in \mathcal{W}$  induces a conditional distribution  $P_{\hat{Y} |X,W}(\hat{Y} |x,w)$  on  $\hat{\mathcal{V}} = \mathcal{V}$ . For convenience, we use the following handy notations:
$$
\begin{array}{l} P _ {\hat {Y} | X, W} ^ {\otimes n} (\hat {\mathbf {y}} | \mathbf {x}, w) =: \prod_ {i \in [ n ]} P _ {\hat {Y} | X, W} (\hat {y} _ {i} | x _ {i}, w), \\ P _ {\hat {Y} | X, W} ^ {\otimes n} \left(\hat {\mathbf {y}} ^ {\prime} \mid \mathbf {x} ^ {\prime}, w\right) =: \prod_ {i \in [ n ]} P _ {\hat {Y} | X, W} \left(\hat {y} _ {i} ^ {\prime} \mid x _ {i} ^ {\prime}, w\right), \\ P _ {\hat {Y} | X, W} ^ {\otimes 2 n} (\hat {\mathbf {y}}, \hat {\mathbf {y}} ^ {\prime} | \mathbf {x}, \mathbf {x} ^ {\prime}, w) =: \prod_ {i \in [ n ]} \left(P _ {\hat {Y} | X, W} (\hat {y} _ {i} | x _ {i}, w) P _ {\hat {Y} | X, W} (\hat {y} _ {i} ^ {\prime} | x _ {i} ^ {\prime}, w)\right). \\ \end{array}
$$
The quality of the prediction is measured by the loss function  $\ell \colon \mathcal{Z} \times \mathcal{W} \to [0,1]$  given by
$$
\ell (z, w) := \mathbb {E} _ {\hat {Y} \sim P _ {\hat {Y} | X, W} (\hat {Y} | x, w)} [ \mathbb {1} _ {\{y \neq \hat {Y} \}} ], \tag {1}
$$
where  $\mathbb{1}$  stands for the indicator function. The associated empirical and population risks for this loss are defined as  $\hat{\mathcal{L}}(s, w) \coloneqq \frac{1}{n} \sum_{i \in [n]} \ell(z_i, w)$  and  $\mathcal{L}(w) \coloneqq \mathbb{E}_{Z \sim \mu}[\ell(Z, w)]$ , respectively. Finally, the generalization error is defined as  $\mathrm{gen}(s, w) \coloneqq \mathcal{L}(w) - \hat{\mathcal{L}}(s, w)$ .
2.1 Compressibility framework
Now, we introduce briefly the joint compression of a block of the predicted labels. Further details can be found in Appendix C.1. Consider  $m$  i.i.d. pairs of train and test datasets  $S_{j} \coloneqq (Z_{j,1},\ldots ,Z_{j,n})$  and  $S_{j}^{\prime} \coloneqq (Z_{j,1}^{\prime},\dots,Z_{j,n}^{\prime})$ , where  $Z_{j,i} = (X_{j,i},Y_{j,i})$  and  $Z_{j,i}^{\prime} = (X_{j,i}^{\prime},Y_{j,i}^{\prime})$ . Let  $S^{m} = (S_{1},\dots,S_{m})$ ,  $S'^{m} = (S_1',\dots,S_m')$  and  $W^{m} \coloneqq (W_{1},\dots,W_{m})$ , where  $W_{j} \sim P_{W_{j}|S_{j}}$ . Denote the predicted labels using model  $W_{j}$  for inputs  $X_{j,i}$  and  $X_{j,i}'$  as  $\hat{Y}_{j,i}$  and  $\hat{Y}_{j,i}'$ , for  $j \in [m]$  and  $i \in [n]$ . Moreover, let  $\mathfrak{Z}_j^{2n} \in \mathcal{Z}^{2n}$  denote a rearrangement of the elements of  $(S_j,S_j')$  in a way that makes it indistinguishable whether a given sample  $z$  is from  $S_{j}$  or  $S_{j}'$ . In the following two subsections, we investigate two such rearrangements of a given  $(S,S')$  as  $\mathfrak{Z}^{2n}$ . Then, for the collection  $(S^m,S'^m)$ , each pair  $(S_j,S_j')$  for  $j \in [m]$  is rearranged independently and the matrix of all rearrangements is denoted by  $\mathfrak{Z}^{2mn}$ . Finally, given some  $\mathfrak{Z}^{2mn}$ , let denote the rearranged versions of  $(Y^{mn},Y'^{mn})$  and  $(\hat{Y}^{mn},\hat{Y}^{\prime mn})$  respectively by  $\mathfrak{Q}^{2mn}$  and  $\mathfrak{V}^{2mn}$ .
Our approach is based on studying the compressibility of the rearranged model-predicted labels  $\hat{\mathfrak{Y}}^{2mn}$ , from an information-theoretic point of view. The rationale is as follows: since the (rearranged) predicted-labels vector  $\hat{\mathfrak{Y}}^{2mn}$  agrees mostly with the true labels  $\mathfrak{Y}^{2mn}$  on the dataset  $S^m$ , then in accordance with "Occam's Razor" theorem [LW86, BEHW87] the model  $W$  is guaranteed to generalize well if  $\hat{\mathfrak{Y}}^{2mn}$  can be described using only a few bits (or nats). We leverage source coding arguments to measure the compressibility of  $\hat{\mathfrak{Y}}^{2mn}$ . In our block-coding rate-distortion theoretic framework, a compression rate  $R \in \mathbb{R}^+$  is said to be achievable if there exists a compression codebook of size  $\approx e^{mR}$  (fixed a priori) which covers the space spanned by the model-predicted labels  $\hat{\mathfrak{Y}}^{2mn}$  with high probability. That is, if  $R$  is achievable then  $\hat{\mathfrak{Y}}^{2mn}$  can be described using  $R \in \mathbb{R}^+$  nats. Formally,  $R$  is achievable if there exists a sequence of label books  $\{\hat{y}_m\}_{m \in \mathbb{N}}$ , with  $\hat{y}_m := \{\hat{y}[r], r \in [l_m]\} \subseteq \mathcal{Y}^{2mn}$ ,  $l_m \in \mathbb{N}$ ,  $\hat{y}[r] = (\hat{y}_1[r], \dots, \hat{y}_m[r])$  and  $\hat{y}_j[r] = (\hat{y}_{j,1}[r], \dots, \hat{y}_{j,2n}[r]) \in \mathcal{Y}^{2n}$ , such that: i.  $l_m \leqslant e^{mR}$  and ii. there exist a sequence  $\{\delta_m\}_{m \in \mathbb{N}}$  for which  $\lim_{m \to \infty} \delta_m = 0$  such that with probability at least  $(1 - \delta_m)$  over the choices of  $S^m$ ,  $S'^m$  one can find at least one index  $r \in [l_m]$  whose associated  $\hat{y}[r]$  equals  $\hat{\mathfrak{Y}}^{2mn}$ .