SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {P} \{\underline {{\varepsilon}} (| \mathrm {c} (\boldsymbol {D}) |, \delta) \leq \phi (\boldsymbol {D}) \leq \bar {\varepsilon} (| \mathrm {c} (\boldsymbol {D}) |, \delta) \} \geq 1 - \delta , \tag {3}
$$
where  $\overline{\varepsilon}(k, \delta)$  for  $k = 0, 1, \ldots, N - 1$  is the unique solution to  $\Psi_{k,\delta}(\varepsilon) = 1$  in the interval  $[k/N, 1]$ , while  $\overline{\varepsilon}(N, \delta) = 1$ , and  $\underline{\varepsilon}(k, \delta)$  for  $k = 0, 1, \ldots, N$  is the maximum between 0 and the unique solution to  $\Psi_{k,\delta}(\varepsilon) = 1$  in the interval  $(- \infty, k/N]$  (function  $\Psi_{k,\delta}(\varepsilon)$  is defined in Section 4).
The previous statement asserts that, with probability  $1 - \delta$  over the draws of  $\pmb{D}$ , the probability of change of compression is contained in an interval with upper and lower extremes given by  $\overline{\varepsilon}(k, \delta)$
Note that this definition, and the properties that we shall introduce later in the section, apply not only to the multiset of actually observed examples, they apply to arbitrary multisets of any cardinality. Although we have used  $D$  to avoid proliferation of the symbols,  $D$  is used here, and elsewhere when needed, to denote generic multisets.
Theorem A.4 continues to hold if the non-associativity property is required to apply  $\mathbb{P}$ -almost surely (as opposed to realization-wise). We use a realization-wise version for the sake of simplicity and also because this formulation suffices for the goals of this paper.
and  $\underline{\varepsilon}(k, \delta)$ , where  $k$  is evaluated at the cardinality of the compression of the draw. Prompted by one of the reviewers, we gladly provide a concise outline of the proof structure for Theorem A.4 in the following Remark A.5.
Remark A.5 (Gist of the proof of Theorem A.4). The proof of Theorem A.4 in (Campi & Garatti, 2023) starts by associating any compression scheme that satisfies Properties A.1 to A.3 to a probability measure from which one can compute  $\mathbb{P}\{\underline{\varepsilon} (|\mathsf{c}(\pmb {D})|,\delta)\leq \phi (\pmb {D})\leq \overline{\varepsilon} (|\mathsf{c}(\pmb {D})|,\delta)\}$  and then by characterizing the class of all such probability measures in terms of certain conditions. The next step consists in obtaining an upper bound to  $\mathbb{P}\{\underline{\varepsilon} (|\mathsf{c}(\pmb {D})|,\delta)\leq \phi (\pmb {D})\leq \overline{\varepsilon} (|\mathsf{c}(\pmb {D})|,\delta)\}$  by maximizing this quantity over the class of probability measures that has been previously characterized. The ensuing maximization problem is infinite dimensional and its solution is obtained by duality. Interestingly, no conservatism is introduced at this stage because strong duality holds. See (Campi & Garatti, 2023) for details.
The interest of Theorem A.4 in the context of statistical learning lies in the fact that, whenever the compression function  $\mathsf{c}$  ties in with the learning algorithm so that inappropriateness (e.g., misclassification or misprediction) implies change of compression, then the probability of inappropriate examples is dominated by the probability of change of compression and, therefore, it can be upper bounded with high confidence based on (3). See also Section 4 in (Campi & Garatti, 2023). This means that we can prove Theorem 4.2 by exhibiting a compression function that enjoys the result of Theorem A.4 and for which whenever  $h(D)$  is inappropriate for  $z$ , i.e.,  $\operatorname{Stop} \leq_{h(D)} z$ , we also have  $\mathsf{c}(\mathsf{c}(D), z) \neq \mathsf{c}(D)$ . This is indeed the demonstration path pursued in the next Appendix A.2.
A.2 Proof of main result
Note that Theorem 4.2 does not assume the non-concentrated mass Property A.3. To ease the presentation, however, Theorem 4.2 will be first proven by assuming Property A.3, which allows for a more direct application of Theorem A.4; later, we will show how this extra assumption can be removed.
Suppose thus for the time being that Property A.3 holds true. Throughout this section, we denote with  $\mathsf{c}_{\mathcal{A}}$  the compression function obtained by  $\mathcal{A}$  when it compresses multiset  $D$  into  $T \subseteq D$  (recall that  $(h, T) = \mathcal{A}(D)$ ). We show that  $\mathsf{c}_{\mathcal{A}}$  enjoys the preference and non-associativity Properties A.1 and A.2. This licenses the use of Theorem A.4 to obtain upper and lower bounds on the probability of change of compression. We shall then relate these bounds on the probability of change of compression to the statistical risk as in Definition 4.1, so closing the proof when Property A.3 is assumed. We shall use  $\mathcal{A}(D, z)$  and  $\mathsf{c}_{\mathcal{A}}(D, z)$  as shorthand for  $\mathcal{A}(D \cup \{z\})$  and  $\mathsf{c}_{\mathcal{A}}(D \cup \{z\})$ .
We begin by showing that  $\mathsf{c}_{\mathcal{A}}(\cdot)$  is preferent.
Lemma A.6. The compression function  $\mathsf{c}_{\mathcal{A}}$  satisfies the preference Property A.1.
Proof. Consider a multiset  $D$  of elements of  $\mathcal{Z}$  and further let  $z\in \mathcal{Z}$ . Suppose that  $\mathsf{c}_{\mathcal{A}}(D,z)\neq \mathsf{c}_{\mathcal{A}}(D)$ . Our goal is to show that in this case  $\mathsf{c}_{\mathcal{A}}(D,z)\neq V$  for any  $V\subseteq D$ , from which Property A.1 immediately follows.
Denote by  $T_{k}, h_{k}$  and  $T_{k}^{\prime}, h_{k}^{\prime}$  the multisets  $T$  and hypotheses  $h$  constructed in  $\mathcal{A}(D)$  and  $\mathcal{A}(D, z)$ , respectively, at iteration  $k$  of the "while" loop.  $T_{0} = \emptyset = T_{0}^{\prime}$  and  $h_{0} = h_{0}^{\prime}$  instead denote the empty multisets  $T$  and initial hypotheses in the initialization step. Finally, let  $D_{\mathbb{S}}^{\prime} = D_{\mathbb{S}} \cup \{z\}$  (where we recall that  $D_{\mathbb{S}} = D \cup \{\mathsf{Stop}\}$ ).
Since  $\mathsf{c}_{\mathcal{A}}(D,z) \neq \mathsf{c}_{\mathcal{A}}(D)$ , there must exist a  $\bar{k} \geq 0$  such that the executions of  $\mathcal{A}(D)$  and  $\mathcal{A}(D,z)$  match for  $k < \bar{k}$  (which implies that  $T_{k}^{\prime} = T_{k}$  and  $h_{k}^{\prime} = h_{k}$  for  $k \leq \bar{k}$ ), but they differ at iteration  $\bar{k}$  because, for  $k = \bar{k}$ , the maximal element computed in  $\mathcal{A}(D)$  (line 5 if  $\bar{k} \geq 1$  or line 1 if  $\bar{k} = 0$ ) is different from the one computed in  $\mathcal{A}(D,z)$ . That is,  $\max_{h_{\bar{k}}} (D_{\mathbb{S}} \setminus T_{\bar{k}}) \neq \max_{h_{\bar{k}}^{\prime}} (D_{\mathbb{S}}^{\prime} \setminus T_{\bar{k}}^{\prime}) = \max_{h_{\bar{k}}} (D_{\mathbb{S}}^{\prime} \setminus T_{\bar{k}})$ , where the last equality holds because, as previously noticed,  $T_{\bar{k}}^{\prime} = T_{\bar{k}}$  and  $h_{\bar{k}}^{\prime} = h_{\bar{k}}$ . We claim that it must be that  $\max_{h_{\bar{k}}} (D_{\mathbb{S}}^{\prime} \setminus T_{\bar{k}}) = z$  and  $z \notin D_{\mathbb{S}} \setminus T_{\bar{k}}$ . Indeed, if this was not the case, it would be that  $\max_{h_{\bar{k}}} (D_{\mathbb{S}}^{\prime} \setminus T_{\bar{k}}) \in D_{\mathbb{S}} \setminus T_{\bar{k}}$ , which would imply that  $\max_{h_{\bar{k}}} (D_{\mathbb{S}}^{\prime} \setminus T_{\bar{k}}) = \max_{h_{\bar{k}}} (D_{\mathbb{S}} \setminus T_{\bar{k}})$  since  $D_{\mathbb{S}} \setminus T_{\bar{k}} \subseteq D_{\mathbb{S}}^{\prime} \setminus T_{\bar{k}}$ . The latter would contradict  $\max_{h_{\bar{k}}} (D_{\mathbb{S}} \setminus T_{\bar{k}}) \neq \max_{h_{\bar{k}}} (D_{\mathbb{S}}^{\prime} \setminus T_{\bar{k}})$ .
Since  $\max_{h_{\bar{k}}} (D_{\mathsf{S}}' \setminus T_{\bar{k}}) = z \neq \operatorname{Stop}, \mathcal{A}(D, z)$  executes at least one more iteration after the  $\bar{k}$ -th, and
$z \notin D_{\mathbb{S}} \setminus T_{\bar{k}}$  yields that  $T_{\bar{k} + 1}' = T_{\bar{k}} \cup \{z\}$  contains the element  $z$  as many times as it appears in  $D \cup \{z\}$ , i.e., as many times as it appears in  $D$  plus one. Therefore  $T_{\bar{k} + 1}' \not\subseteq D$  and, since the multiset  $T_{k}'$  is increasing with  $k$ , it must be that  $T_{k}' \not\subseteq D$  for all  $k \geq \bar{k} + 1$ . This implies  $\mathsf{c}_{\mathcal{A}}(D, z) \not\subseteq D$ ; that is,  $\mathsf{c}_{\mathcal{A}}(D, z) \neq V$  for any  $V \subseteq D$ .
We now show that  $c_{\mathcal{A}}$  is non-associative.
Lemma A.7. The compression function  $c_{\mathcal{A}}$  satisfies the non-associativity Property A.2.
Proof. Fix any multiset  $D$  of elements of  $\mathcal{Z}$  and further let  $z_{1},\ldots ,z_{p}\in \mathcal{Z}$ . We will prove the claim by contrapositive. Towards this goal, assume that  $\mathsf{c}_{\mathcal{A}}(D,z_1,\dots,z_p)\neq \mathsf{c}_{\mathcal{A}}(D)$ . We will then show that there exists an  $i\in \{1,\ldots ,p\}$  such that  $\mathsf{c}_{\mathcal{A}}(D,z_i)\neq \mathsf{c}_{\mathcal{A}}(D)$ .
We first focus on the executions of  $\mathcal{A}(D)$  and  $\mathcal{A}(D, z_1, \ldots, z_p)$ . Similarly to the proof of Lemma A.6, denote by  $T_k, h_k$  and  $T_k^{(p)}, h_k^{(p)}$  the multisets  $T$  and hypotheses  $h$  constructed in  $\mathcal{A}(D)$  and  $\mathcal{A}(D, z_1, \ldots, z_p)$ , respectively, at iteration  $k$  ( $k = 0$  corresponds to the initialization). Finally, let  $D_{\mathbb{S}}^{(p)} = D_{\mathbb{S}} \cup \{z_1, \ldots, z_p\}$ . Since  $\mathsf{c}_{\mathcal{A}}(D, z_1, \ldots, z_p) \neq \mathsf{c}_{\mathcal{A}}(D)$ , also in the present context there must exist a  $\bar{k} \geq 0$  such that the executions of  $\mathcal{A}(D)$  and  $\mathcal{A}(D, z_1, \ldots, z_p)$  match for  $k < \bar{k}$  (so that  $T_k = T_k^{(p)}$  and  $h_k = h_k^{(p)}$  for  $k \leq \bar{k}$ ), but they differ at iteration  $\bar{k}$  because, for  $k = \bar{k}$ , the maximal element computed in  $\mathcal{A}(D)$  (line 5 if  $\bar{k} \geq 1$  or line 1 if  $\bar{k} = 0$ ) is different from the one computed in  $\mathcal{A}(D, z_1, \ldots, z_p)$ . That is,  $\max_{h_{\bar{k}}} (D_{\mathbb{S}} \setminus T_{\bar{k}}) \neq \max_{h_{\bar{k}}^{(p)}} (D_{\mathbb{S}}^{(p)} \setminus T_{\bar{k}}^{(p)})$ . Then, following an argument identical to that in the proof of Lemma A.6, one observes that it must be that  $\max_{h_{\bar{k}}^{(p)}} (D_{\mathbb{S}}^{(p)} \setminus T_{\bar{k}}^{(p)}) = z_i$  for some  $i \in \{1, \ldots, p\}$  and  $z_i \notin D_{\mathbb{S}} \setminus T_{\bar{k}}$ .
Now compare the executions of  $\mathcal{A}(D)$  and  $\mathcal{A}(D, z_i)$ , and denote by  $T_k'$ ,  $h_k'$  the multiset  $T$  and the hypothesis  $h$  constructed in the execution of  $\mathcal{A}(D, z_i)$  at iteration  $k$  ( $k = 0$  still corresponds to the initialization). Further, let  $D_{\mathbb{S}}' = D_{\mathbb{S}} \cup \{z_i\}$ . One can first observe that also the executions of  $\mathcal{A}(D)$  and  $\mathcal{A}(D, z_i)$  for  $k < \bar{k}$  must be identical. Notice indeed that: i.  $T_0' = T_0 = T_0^{(p)}$  and  $h_0' = h_0 = h_0^{(p)}$ ; ii. when  $T_k' = T_k = T_k^{(p)}$  and  $h_k' = h_k = h_k^{(p)}$  for a  $k < \bar{k}$ , the maximal element selected at iteration  $k$  of  $\mathcal{A}(D)$  and  $\mathcal{A}(D, z_i)$  is the same since this is so for the maximal element of  $\mathcal{A}(D)$  and  $\mathcal{A}(D, z_1, \ldots, z_p)$  and  $D_{\mathbb{S}} \setminus T_k \subseteq D_{\mathbb{S}}' \setminus T_k' \subseteq D_{\mathbb{S}}^{(p)} \setminus T_k^{(p)}$ ; iii. this in turn gives  $T_{k+1}' = T_{k+1} = T_{k+1}^{(p)}$  and  $h_{k+1}' = h_{k+1} = h_{k+1}^{(p)}$ . At iteration  $\bar{k}$ , instead, the identity of executions ends, and it must be  $\max_{h_{\bar{k}}'} (D_{\mathbb{S}}' \setminus T_{\bar{k}}') = z_i$  because  $z_i$  is maximal over  $D_{\mathbb{S}}^{(p)} \setminus T_{\bar{k}}^{(p)}$  and  $z_i \in D_{\mathbb{S}}' \setminus T_{\bar{k}}' \subseteq D_{\mathbb{S}}^{(p)} \setminus T_{\bar{k}}^{(p)}$  (surely  $z_i$  still belongs to  $D_{\mathbb{S}}' \setminus T_{\bar{k}}'$  because  $T_{\bar{k}}' = T_{\bar{k}} \subseteq D$  and  $D_{\mathbb{S}}' = D \cup \{z_i\} \cup \{\text{Stop}\}$ ). Further, recall that  $z_i \notin D_{\mathbb{S}} \setminus T_{\bar{k}}$ . Since  $\mathcal{A}(D, z_i)$  executes at least one more iteration after the  $\bar{k}$ -th (because  $z_i \neq \text{Stop}$ ) and since the  $T_k'$ 's are increasing, we have that any  $T_k'$  for  $k \geq \bar{k} + 1$  is different from any of the  $T_k$ 's. This implies  $\mathsf{c}_{\mathcal{A}}(D, z_i) \neq \mathsf{c}_{\mathcal{A}}(D)$ , thus concluding the proof.
Given that Properties A.1 and A.2 hold true, and since Property A.3 is assumed, Theorem A.4 applies, yielding  $\mathbb{P}\{\underline{\varepsilon} (|\mathsf{c}_{\mathcal{A}}(D)|,\delta)\leq \phi_{\mathcal{A}}(D)\leq \overline{\varepsilon} (|\mathsf{c}_{\mathcal{A}}(D)|,\delta)\} \geq 1 - \delta$ , where  $\phi_{\mathcal{A}}(D) = \mathbb{P}\{\mathsf{c}_{\mathcal{A}}(\mathsf{c}_{\mathcal{A}}(D),z)\neq \mathsf{c}_{\mathcal{A}}(D)\mid D\}$ . As anticipated, to obtain from this the sought result for the statistical risk, it is enough to show that, given  $D$  and the hypothesis  $h$  returned by  $\mathcal{A}(D)$ , every realization  $z$  such that  $\operatorname {Stop}\leq h$ $z$  also changes the compression, i.e.,  $\mathsf{c}_{\mathcal{A}}(\mathsf{c}_{\mathcal{A}}(D),z)\neq \mathsf{c}_{\mathcal{A}}(D)$ . As a matter of fact, this implies that for every  $D$  the event  $\{\operatorname {Stop}\leq_h z\}$  is a sub-event of  $\{\mathsf{c}_{\mathcal{A}}(\mathsf{c}_{\mathcal{A}}(D),z)\neq \mathsf{c}_{\mathcal{A}}(D)\}$ , and hence that  $R(h)\leq \phi_{\mathcal{A}}(D)$  almost surely. Then, Theorem 4.2's claim readily follows since
$$
\begin{array}{l} \mathbb {P} \{R (\boldsymbol {h}) \leq \bar {\varepsilon} (| \mathrm {c} _ {\mathcal {A}} (\boldsymbol {D}) |, \delta) \} \\ \geq \mathbb {P} \left\{\phi_ {\mathcal {A}} (\boldsymbol {D}) \leq \bar {\varepsilon} (| \mathsf {c} _ {\mathcal {A}} (\boldsymbol {D}) |, \delta) \right\} \\ \geq \mathbb {P} \{\underline {{\varepsilon}} (\left| \mathbf {c} _ {\mathcal {A}} (\boldsymbol {D}) \right|, \delta) \leq \phi_ {\mathcal {A}} (\boldsymbol {D}) \leq \bar {\varepsilon} (\left| \mathbf {c} _ {\mathcal {A}} (\boldsymbol {D}) \right|, \delta) \} \\ \geq \quad 1 - \delta . \\ \end{array}
$$
Thus, we are left to prove the following lemma.
Lemma A.8. Let  $D$  be any multiset of elements of  $\mathcal{Z}$ ,  $z \in \mathcal{Z}$  be any example, and  $(h,T) = \mathcal{A}(D)$ . If  $z$  is such that  $\operatorname{Stop} \leq_h z$ , then  $\mathsf{c}_{\mathcal{A}}(\mathsf{c}_{\mathcal{A}}(D),z) \neq \mathsf{c}_{\mathcal{A}}(D)$ .
Proof. We prove the contrapositive and show that, if  $z$  is such that  $\mathsf{c}_{\mathcal{A}}(\mathsf{c}_{\mathcal{A}}(D),z) = \mathsf{c}_{\mathcal{A}}(D)$ , then  $z \leq_h \operatorname{Stop}$ .
Note that  $(h,T) = \mathcal{A}(D)$  means that  $T = \mathsf{c}_{\mathcal{A}}(D)$  and  $h = L([T]_{\mathcal{A}})$  thanks to line 4 in Algorithm 1. Consider the execution of  $\mathcal{A}(\mathsf{c}_{\mathcal{A}}(D)\cup \{z\})$ . By definition of  $\mathsf{c}_{\mathcal{A}}$ , the condition  $\mathsf{c}_{\mathcal{A}}(\mathsf{c}_{\mathcal{A}}(D),z) = \mathsf{c}_{\mathcal{A}}(D)$  means that  $\mathcal{A}(\mathsf{c}_{\mathcal{A}}(D)\cup \{z\})$  terminates with  $\mathsf{c}_{\mathcal{A}}(D) = T$ . Hence, the maximal elements selected by  $\mathcal{A}(\mathsf{c}_{\mathcal{A}}(D)\cup \{z\})$  are the same as those selected by  $\mathcal{A}(D)$ . Further, by the properties of max operators, the elements selected by  $\mathcal{A}(\mathsf{c}_{\mathcal{A}}(D)\cup \{z\})$  and  $\mathcal{A}(D)$  match at each iteration. This implies that  $\mathcal{A}(\mathsf{c}_{\mathcal{A}}(D)\cup \{z\})$  terminates returning  $h$ , besides  $T$ . For this to occur, it must be that  $\max_h((\mathsf{c}_{\mathcal{A}}(D)\cup \{z\}\cup \{\mathrm{Stop}\})\backslash T) = \mathrm{Stop}$  (line 2). This is equivalent to  $\max_h(\{z,\mathrm{Stop}\}) = \mathrm{Stop}$ , i.e.,  $z\leq h$  Stop.
Having proven Lemma A.8, we have completed the derivation of the main result under the additional assumption that Property A.3 holds true, an assumption that, however, does not appear in the statement of Theorem 4.2. To remove this assumption, and thus complete the proof of Theorem 4.2, we can proceed as follows.
Augment each random element  $\mathbf{z}_i$  with a random variable  $\mathbf{u}_i$  that is uniformly distributed over  $[0,1]$  and independent of  $\mathbf{z}_i$ . Moreover, define the  $\mathbf{u}_i$ 's so that the random elements  $\tilde{\mathbf{z}}_1, \tilde{\mathbf{z}}_2, \ldots, \tilde{\mathbf{z}}_N$ , where  $\tilde{\mathbf{z}}_i = (\mathbf{z}_i, \mathbf{u}_i)$  for all  $i$ , are independent and identically distributed (i.i.d.). Also, let  $\tilde{\mathbf{z}} = (\mathbf{z}, \mathbf{u})$  be independent of and identically distributed as each  $\tilde{\mathbf{z}}_i$ . Let  $\tilde{\mathcal{Z}} = \mathcal{Z} \times [0,1]$ , which is the space where the  $\tilde{\mathbf{z}}_i$ 's take value. For any  $\tilde{\mathbf{z}} = (z,u) \in \tilde{\mathcal{Z}}$  and  $\tilde{\mathbf{z}}' = (z',u') \in \tilde{\mathcal{Z}}$  define  $\tilde{\mathbf{z}} \leq_h \tilde{\mathbf{z}}'$  if  $z \leq_h z'$  and  $u \leq u'$ , while define  $\tilde{\mathbf{z}} \leq_h \text{Stop}$  [resp. Stop  $\leq_h \tilde{\mathbf{z}}$ ] if  $z \leq_h \text{Stop}$  [resp. Stop  $\leq_h z$ ]. Also, for any  $n$ , define  $L(\tilde{\mathbf{z}}_1, \ldots, \tilde{\mathbf{z}}_n) = L(z_1, \ldots, z_n)$ , where  $\tilde{\mathbf{z}}_i = (z_i, u_i)$ ,  $i = 1, \ldots, n$ , are elements of  $\tilde{\mathcal{Z}}$ .
Consider now the executions of  $\mathcal{A}$  over multisets of elements of  $\tilde{\mathcal{Z}}$  according to the redefinition of  $\leq_h$  and  $L$ , and let  $(\tilde{h},\tilde{T}) = \mathcal{A}(\tilde{D})$ , where  $\tilde{D} = \{\tilde{z}_1,\dots ,\tilde{z}_N\}$ . As is clear, the non-concentrated mass Property A.3 applies for the  $\tilde{z}_i$ 's and an application of the results obtained so far gives  $\mathbb{P}\{\tilde{R} (\tilde{h})\leq \overline{\varepsilon} (|\tilde{T} |,\delta)\} \geq 1 - \delta$ , where  $\tilde{R} (h) = \mathbb{P}\{\mathrm{Stop}\leq h\tilde{z}\}$ . On the other hand, by virtue of the redefinition of  $\leq_h$  and  $L$ , one can notice that, for every realization of  $\tilde{D}$ , the execution of  $\mathcal{A}(\tilde{D})$  matches step by step that of  $\mathcal{A}(D)$ , with the sole difference that elements in  $T$  that are otherwise indistinguishable are incorporated in  $\tilde{T}$  with their "u" counterparts (and the highest values are selected first). Thus, the multiset  $T$  is equal to the multiset of the "z" components of the elements of  $\tilde{T}$ , yielding  $|\tilde{T}| = |T|$ . Moreover, since the addition of elements in  $T$  and  $\tilde{T}$  occurs in the same order, we also have that  $\tilde{h} = h$ . Finally, noticing that  $\tilde{R} (h) = R(h)$  because  $\tilde{z}\leq_h$  Stop  $\iff z\leq_h$  Stop, one has that
$$
\mathbb {P} \{R (\boldsymbol {h}) \leq \bar {\varepsilon} (| \boldsymbol {T} |, \delta) \} = \mathbb {P} \{\tilde {R} (\tilde {\boldsymbol {h}}) \leq \bar {\varepsilon} (| \tilde {\boldsymbol {T}} |, \delta) \} \geq 1 - \delta ,
$$
and this concludes the proof of Theorem 4.2.
Remark A.9 (On the tightness of Theorem 4.2). Although both upper and lower bounds that hold with confidence  $1 - \delta$  are available for the probability of change of compression, only the upper bound ports over to the statistical risk, because any realization for which  $\mathsf{Stop} \leq_h z$  (i.e., a realization that is inappropriate) is also a realization that changes the compression, but the opposite does not hold. Given that the lower and the upper bounds are provably close to each other even for relatively small values of  $N$ , this means that  $\bar{\varepsilon}(|T|, \delta)$  is an accurate evaluation of the probability of change of compression, but it can be loose for the statistical risk whenever this deviates from the probability of change of compression. Interestingly, this is the only source of looseness in the evaluation of the risk provided by Theorem 4.2.
A.3 An extension of P2L: adding multiple examples in  $T$  simultaneously
A simple extension of the meta-algorithm P2L described in Algorithm 1 consists in adding at every iteration multiple, say  $R > 1$ , examples in  $T$  at a time. A possible implementation is then formalized in Algorithm 2 below, where  $\max_h^R (U)$  returns the  $R$  maximal points (the maximum, the 2nd maximum, ..., the  $R$ -th maximum) of  $U$  according to the total order  $\leq h$  (if  $U$  has less than  $R$  elements, then the whole  $U$  is returned).
Algorithm 2  $\mathcal{A}^R (D)$  -Meta-algorithm P2L with  $R$  examples added at a time
1: Initialize:  $T = \emptyset, h = h_0, \bar{z}_1, \dots, \bar{z}_R \leftarrow \max_h^R(D_{\mathbf{S}})$   
2: while  $\bar{z}_i \neq \text{Stop}$  for all  $i = 1, \dots, R$  do  
3:  $T \gets T \cup \{\bar{z}_1, \dots, \bar{z}_R\}$   
4:  $h \gets L([T]_{\mathcal{A}})$   
5:  $\bar{z}_1, \dots, \bar{z}_R \gets \max_h^R(D_{\mathbf{S}} \setminus T)$   
6: end while  
7:  $T \gets T \cup \{\bar{z}_i : \bar{z}_i \geq_h \text{Stop}\}$   
8: return  $h, T$   
9: Hypothesis  $h$  and multiset  $T$
Interestingly, Theorem 4.2 continues to hold true (with no modification in the statement) also when the pair  $h, T$  is the output of Algorithm 2 instead of Algorithm 1. To prove this, it is enough to observe that Algorithm 2 is completely equivalent to the following sequential Algorithm 3, which is identical to Algorithm 1, so that only one point is added to  $T$  at each iteration, except for the fact that  $h$  is updated every  $R$  iterations (mod(iter,  $R$ ) is the remainder of the division of iter by  $R$ ). This equivalence secures the result because the proof of Theorem 4.2 can be repeated for Algorithm 3 word for word, without any modification.
Algorithm 3  
1: Initialize:  $T = \emptyset, h = h_0, \bar{z} = \max_{h_0}(D_S)$ , iter = 0  
2: while  $\bar{z} \neq \text{Stop do}$   
3: iter ← iter + 1  
4:  $T \gets T \cup \{\bar{z}\}$  ▷ Augment T  
5: if mod(iter, R) = 0 then  
6:  $h \gets L([T]_A)$  ▷ Learn hypothesis  
7: end if  
8:  $\bar{z} \gets \max_h(D_S \setminus T)$  ▷ Compute max  
9: end while  
10: return h, T ▷ Hypothesis h and multiset T
As is clear, Algorithm 1 is a particular case of Algorithm 2 since the former is re-obtained from the latter when  $R = 1$ . We notice that adding more than one example to  $T$  at every iteration might or might not have a beneficial effect on P2L. Indeed, on one hand, adding more examples simultaneously might result in building a better  $h$  earlier on (and save computational time). However, by introducing examples in groups (and thus not allowing for the fine-grained choice of adding them one-by-one) might also result in a larger  $T$ , which could worsen the final bound on the risk. In this regard, a modulation of  $R$  across iterations seems to be a promising line of investigation, which however is beyond the scope of the present paper and is therefore left for future work.
A.4 A discussion on the P2L framework in retrospect
A close inspection of the proofs of the key Lemmas A.6 and A.7 reveals that these two lemmas strongly rely on the fact that the  $\max_h$  operator is a compression function from many elements to just one with the following properties: i) it is preferent (as a matter of fact, for any multiset  $A$  and additional element  $b$ , either  $\max_h(A \cup \{b\}) = b$  or  $\max_h(A \cup \{b\}) = \max_h(A)$ ); and ii) it is non-associative ( $\max_h(A \cup \{b_i\}) = \max_h(A)$  for all  $i \in \{1, \ldots, p\}$  implies that there is an element  $\bar{a}$  of  $A$  such that  $c \leq h$ $\bar{a}$  for all  $c \in A \cup \{b_1, \ldots, b_p\}$  and, therefore,  $\max_h(A \cup \{b_1, \ldots, b_p\}) = \max_h(A)$ ).