SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Given:  $S\in (\mathcal{X}\times \mathcal{Y})^m$  , parameters  $m_0,p > 0$
Output: A function  $\mu : \mathcal{X} \mapsto \mathcal{Y}^p$ .
1: Set  $S_{1} \coloneqq S$ .
2: for  $j = 1, \dots, p$  do
3: Let  $\mathcal{U}_j$  denote the uniform distribution over  $S_{j}$
4: Draw  $m_0$  examples from  $\mathcal{U}_j$  and pass to the weak learner, with  $\mu_0\equiv \mathcal{V}$
5: Get weak hypothesis  $h_j : \mathcal{X} \mapsto \mathcal{Y}$ .
6: Set  $S_{i+1}$  to be all the points in  $S_i$  which  $h_j$  predicts incorrectly.
7: end for
8: Output  $\mu$  defined by:
$$
\mu (x) = \left\{h _ {1} (x), \dots , h _ {p} (x) \right\}.
$$
3We also allow lists that return an infinite subset of labels. In that case we simply have that the weak hypothesis satisfies  $\operatorname*{Pr}_{(x,y)\sim p}[h(x) = y]\geq \gamma$
Algorithm 3 Recursive Boosting
Given: Training data  $S\in (\mathcal{X}\times \mathcal{Y})^m$  , edge  $\gamma >0$  parameters  $T,\eta ,p > 0$
Output: A predictor  $H: \mathcal{X} \mapsto \mathcal{Y}$ .
1: Initialize: get  $\mu_{1}$  by applying Algorithm 2 over  $S$ .
2: for  $j = 1, \dots, p - 1$  do
3: Call Hedge (Algorithm 1) with  $S$  and  $\mu_j$ , and parameters  $\eta, T$  to get  $H_j: \mathcal{X} \times \mathcal{Y} \mapsto \mathbb{R}$ .
\Modify Algorithm 1 to receive  $\mu_{j}$  as input,
and in line 4 pass it to the weak learner.
4: Construct  $\mu_{j + 1}:\mathcal{X}|_S\mapsto \mathcal{Y}^{p - j}$  such that for all  $x$
$$
\mu_ {j + 1} (x) = \left\{y: y \in \mu_ {j} (x) \wedge H _ {j} (x, y) > \frac {T}{p - j + 1} \right\},
$$
6: Output the final hypothesis  $\bar{H} \coloneqq \mu_p$ .
The following theorem is formally stating the main result given in Theorem 1.
Theorem 2 (Boosting). Let  $\mathcal{W}$  denote a learning rule that when given any set of labeled examples and a list function, returns some hypothesis  $h: \mathcal{X} \mapsto \mathcal{Y}$ . Let  $\epsilon, \delta, \gamma, m_0 > 0$ , and let  $\mathcal{D}$  a distribution over  $\mathcal{X} \times \mathcal{Y}$ . Then, when given a sample  $S \sim \mathcal{D}^m$  for  $m \geq \frac{10^2 m_0 \left( \ln^2(m) \ln \left( \frac{m}{\delta} \right) \right)}{\gamma^3 \epsilon}$ , oracle access to  $\mathcal{W}$  and  $T \geq \frac{8 \ln(m)}{\gamma^2}$ ,  $p \geq \frac{2 \ln(m)}{\gamma}$ , and  $\eta = \sqrt{\frac{\ln(m)}{2T}}$ , Algorithm 3 outputs a predictor  $\bar{H}$  such that the following holds. Denote by  $M$  the sets of list functions on which  $\mathcal{W}$  was trained throughout Algorithm 3. Then, with probability at least  $1 - \delta$ , we get that if  $\mathcal{W}$  satisfies the  $\gamma$ -BRG condition (as given in Definition 4) with respect to  $(S, \gamma, m_0, M)$  then,
Observe that Theorem 2 implicitly assumes that the sample complexity of the weak learner  $m_0$  is not strongly dependent on the overall sample size  $m$  and scales at most poly-logarithmically with  $m$ . In other words, although the statement holds for any  $m_0$ , the result becomes vacuous otherwise.
In addition, notice that Theorem 2 is quite agnostic in the sense that we have made no prior assumptions about the data distribution. Concretely, Theorem 2 tells us that the generalization error will be small if the given oracle learner  $\mathcal{W}$  happens to satisfy the  $\gamma$ -BRG condition with respect to the particular inputs it receives throughout our boosting procedure.
Adaptive boosting Boosting algorithms typically do not assume knowing the value of  $\gamma$  and are adapted to it on the fly, as in the well-known Adaboost algorithm [23]. However, the boosting algorithm given in Algorithm 1, as well as our boosting method as a whole, requires feeding the algorithm with a value estimating  $\gamma$ . If the estimation of  $\gamma$  provided to the algorithm is too large, the algorithm may fail. This can be resolved by a simple preliminary binary-search-type procedure, in which we guess gamma, and possible halve it based on the observed outcome. This procedure only increases the overall runtime by a logarithmic factor of  $O(\ln(1/\gamma))$ , and has no affect on the sample complexity bounds.
4 Applications to List PAC learning
The applications given in this section are based on the framework of List PAC learning [5, 8], and demonstrate that it is in fact closely related to the multiclass boosting theory. First, we establish an equivalence between list learnability and weak learnability in the context of the PAC model. Furthermore, we present a new result on boosting for list PAC learners. Lastly, we give a novel and alternative proof for characterization of PAC learnability and List PAC learnability. In particular, these imply a simplified algorithmic approach compared to previous works [5, 8].
We can now present our main boosting method in Algorithm 3, and state its guarantees in Theorem 2.
We start with introducing list learning in Definition 5, followed by the definition of weak PAC learning, similarly to the weak-BRG learning definition we give in this work.
Definition 5 (k-List PAC Learning). We say that a hypothesis class  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  is  $k$ -list PAC learnable, if there is an algorithm such that for every  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ , and every  $\epsilon, \delta > 0$ , when given  $S \sim \mathcal{D}^m$  for  $m \geq m(\epsilon, \delta)$ , it returns  $\mu_S: \mathcal{X} \to \mathcal{Y}^k$  such that with probability  $1 - \delta$ ,
$$
\operatorname * {P r} _ {(x, y) \sim \mathcal {D}} \bigl [ y \in \mu_ {S} (x) \bigr ] \geq 1 - \epsilon .
$$
Definition 6 ( $\gamma$ -weak PAC Learning). We say that a hypothesis class  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  is  $\gamma$ -weak PAC learnable, if there is an algorithm such that for every  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ , and every  $\delta > 0$ , when given  $S \sim \mathcal{D}^m$  for  $m \geq m(\delta)$ , it returns  $h_S: \mathcal{X} \to \mathcal{Y}$  such that with probability  $1 - \delta$ ,
$$
\operatorname * {P r} _ {(x, y) \sim \mathcal {D}} \left[ y = h _ {S} (x) \right] \geq \gamma .
$$
Next, in the following lemmas we show the strong connection between these two notions. Specifically, we give an explicit construction of a list learner given oracle access to a weak learner, and vice versa.
Lemma 2 (Weak  $\Rightarrow$  List Learning). Let  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  be a hypothesis class. Assume  $\mathcal{W}$  is a  $\gamma$ -weak PAC learner for  $\mathcal{H}$  with sample complexity  $m_w: (0,1) \mapsto \mathbb{N}$ . Let  $k$  be the smallest integer such that  $\frac{1}{k} < \gamma$ , and denote  $\sigma = \gamma - \frac{1}{k}$ . Then, there is an  $(k-1)$ -List PAC learner with sample complexity  $m(\epsilon, \delta) = \tilde{O}\left(\frac{m_w(\delta / T)}{\sigma^2\epsilon}\right)$  where  $T = \tilde{O}\left(\frac{1}{\sigma^2}\right)$  is the number of its oracle calls to  $\mathcal{W}$ .
Lemma 3 (List  $\Rightarrow$  Weak Learning). Let  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  be a hypothesis class. Assume  $\mathcal{L}$  is a  $k$ -List PAC learner for  $\mathcal{H}$  with sample complexity  $m_{\ell}:(0,1)\mapsto \mathbb{N}$ . Then, for any  $\epsilon > 0$  there is an  $\gamma$ -Weak PAC learner where  $\gamma = \frac{1 - 2\epsilon}{k}$ , with sample complexity  $m(\delta) = \tilde{O}\left(m_{\ell}(\epsilon ,1 / 2)\cdot k + (k / \epsilon)^{2}\right)$  where  $q = 2k\log (2 / \delta)$  is the number of its oracle calls to  $\mathcal{L}$ .
Lastly, Theorem 3 concludes this section demonstrating the strong ties between weak and list learnability. Concretely, it combines the results of both Lemma 2 and Lemma 3 above to show that when the appropriate parameters  $\gamma$  and  $k$  are optimal, then  $\gamma$ -PAC learnability and  $k$ -list PAC learnability are in fact equivalent.
Theorem 3 (Optimal accuracy  $\Longleftrightarrow$  Optimal list size). Let  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$ . Denote by  $k(\mathcal{H})$  the smallest integer  $k$  for which  $\mathcal{H}$  is  $k$ -list PAC learnable, assuming that  $k(\mathcal{H}) < \infty$ . Denote by  $\gamma(\mathcal{H})$  the supremum over  $\gamma \in [0,1]$  for which  $\mathcal{H}$  is  $\gamma$ -weak PAC learnable. Then, it holds that  $k(\mathcal{H}) \cdot \gamma(\mathcal{H}) = 1$ .
4.1 List boosting and conformal learning
List prediction rules naturally arise in the setting of conformal learning. In this model, algorithms make their predictions while also offering some indication of the level of reliable confidence in those predictions. For example in multiclass classification, given an unlabeled test point  $x$ , the conformal learner might output a list of all possible classes along with scores which reflect the probability that  $x$  belongs to each class. This list can then be truncated to a shorter one which contains only the classes with the highest score. See the book by [28] and surveys by [25, 2] for more details.
We now consider a closely related notion of List PAC learnability, that similarly to conformal learning allows the list size to depend on the desired confidence. This was also defined in [8], termed weak List PAC Learning, due to the dependence of the list size on the input parameter.
Indeed, it is natural to expect that the list size will increase when we require a more refined accuracy, and perhaps that this is a weaker notion of learnability than that of List PAC learning, which corresponds to a fixed list size.
Interestingly, it turns out that weak List PAC Learning is in fact equivalent to strong List PAC Learning. In other words, a list learner with a list size that varies with the desired accuracy parameter can be boosted to a list learner with a fixed list size, and arbitrarily good accuracy. The proof is by way of a generalization of our boosting technique to lists, as stated in Theorem 4.
Theorem 4 (List boosting). Let  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$ . Let  $\epsilon_0, \delta_0 > 0$ , and assume that there exists an algorithm such that for every  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ , and for some integer  $k_0 \coloneqq k_0(\epsilon_0)$ , when given  $S \sim \mathcal{D}^m$  for  $m \geq m(\epsilon_0, \delta_0)$ , it returns  $\mu_S: \mathcal{X} \to \mathcal{Y}^{k_0}$  such that with probability  $1 - \delta_0$ ,
$$
\operatorname * {P r} _ {(x, y) \sim \mathcal {D}} \left[ y \in \mu_ {S} (x) \right] \geq 1 - \epsilon_ {0}.
$$
Then, there is a  $k$ -List PAC learning algorithm for  $\mathcal{H}$  for a fixed list size  $k = \left\lfloor \frac{k_0}{1 - 2\epsilon_0} \right\rfloor$ .
Observe that Theorem 4 indeed generalizes classic boosting. Specifically, consider the binary setting and notice that when  $k_{0} = 1$ , and  $\epsilon_0$  is slightly smaller than  $1 / 2$ , Theorem 4 implies that weak learning with edge  $\approx \frac{1}{2} -\epsilon_0$ , is equivalent to strong learning with arbitrarily small error. The following corollary shows that weak List PAC Learning implies strong List PAC Learning.
Corollary 1. If a class  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  is weakly-List PAC learnable then it is also List PAC learnable.
4.2 Characterization of List PAC learnability
We now focus on the characterization of List PAC learnability, which also implies the characterization of PAC learnability. Towards that end, we define the Daniely-Shwartz (DS) dimension [9]. Specifically, we give the natural generalization of it to  $k$ -sized lists, called the  $k$ -DS dimension.
Definition 7 ( $k$ -DS dimension [8]). Let  $\mathcal{H} \subseteq \mathcal{V}^{\mathcal{X}}$  be a hypothesis class and let  $S \in \mathcal{X}^d$  be a sequence. We say that  $\mathcal{H}$ $k$ -DS shatters  $S$  if there exists  $\mathcal{F} \subseteq \mathcal{H}$ ,  $|\mathcal{F}| < \infty$  such that  $\forall f \in \mathcal{F}|_S$ ,  $\forall i \in [d]$ ,  $f$  has at least  $k$ $i$ -neighbors. The  $k$ -DS dimension of  $\mathcal{H}$ , denoted as  $d_{DS}^k = d_{DS}^k(\mathcal{H})$ , is the largest integer  $d$  such that  $\mathcal{H}$ $k$ -DS shatters some sequence  $S \in \mathcal{X}^d$ .
We note that when  $k = 1$ , this captures the standard DS dimension [9]. We show that when the  $k$ -DS dimension is bounded, one can construct a simple weak learner which satisfies our BRG condition. Thus, it is also amenable to our boosting method, leading to a qualitatively similar results for characterization of learnability as in [5, 8]. The result is given in the next theorem.
Theorem 5 (PAC and List-PAC learnability). Let  $\mathcal{H} \subseteq \mathcal{Y}^{\mathcal{X}}$  be an hypothesis class with  $k$ -DS dimension  $d < \infty$ . Then,  $\mathcal{H}$  is List PAC learnable. Furthermore, there is a learning algorithm  $A$  for  $\mathcal{H}$  with the following guarantees. For every  $\mathcal{H}$ -realizable distribution  $\mathcal{D}$ , every  $\delta > 0$  and every integer  $m$ , given an input sample  $S \sim \mathcal{D}^{m}$ , the algorithm  $A$  outputs  $\mu = A(S)$  such that<sup>4</sup>
$$
\operatorname * {P r} _ {(x, y) \sim \mathcal {D}} [ \mu (x) \not \ni y ] \leq \tilde {O} \left(\frac {d ^ {5} k ^ {4} + \log (1 / \delta)}{m}\right),
$$
with probability at least  $1 - \delta$  over  $S$ . In particular, if  $k = 1$ , then  $\mathcal{H}$  is PAC learnable.
We remark that for cases where  $d \ll k$  we have an improved result over the bound given by [8]. For comparison, the error bound given by [8], Theorem 2 is  $\tilde{O}\left(\frac{d^{1.5}k^6 + \log(1 / \delta)}{m}\right)$ .
Thus, Theorem 5 demonstrates that our boosting-based approach gives rise to an alternative proof for the characterization of PAC learnability and List PAC learnability. Moreover, our approach offers a simpler algorithm and analysis technique, that can perhaps be of use in future applications as well.
5 Generalization via compression
This section is concerned with the analysis of our boosting method given in Algorithm 3, and the proof of our main result given in Theorem 1 (and formally in Theorem 2).
The boosting algorithm given in this work is best thought of as a sample compression scheme [17]. A sample compression scheme (Definition 8) is an abstraction of a common property to many learning algorithms. It can be viewed as a two-party protocol between a compressor and a reconstructor. The compressor gets as input a sample  $S$ . The compressor picks a small subsample  $S'$  of  $S$  and sends it to the reconstructor. The reconstructor outputs an hypothesis  $h$ . The correctness criteria is that  $h$  needs to correctly classify all examples in the input sample  $S$ . We formally define it next.
Definition 8 (Sample Compression Scheme [17]). Let  $r \leq m$  be integers. An  $m \to r$  sample compression scheme consists of a reconstruction function
$$
\rho : (\mathcal {X} \times \mathcal {Y}) ^ {r} \rightarrow \mathcal {Y} ^ {\mathcal {X}}
$$
such that for every  $S \in (\mathcal{X} \times \mathcal{Y})^m$ , there exists  $S' \subseteq S$  of size  $r$ , such that for all  $(x, y) \in S$  it holds that  $h(x) = y$ , where  $h = \rho(S')$ .
4The  $\tilde{O}$  notation conceals polylog  $(m,1 / \gamma)$  factors.
We are now ready to prove the main result, given in Theorem 2. The next paragraph highlights the assumptions that were made, followed by the proof of the theorem.
Specifically, we assume for simplicity that the learning algorithm does not employ internal randomization. Thus, it can be regarded as a fixed, deterministic mapping from a sequence of  $m_0$  unweighted examples, and a function  $\mu : \mathcal{X} \mapsto \mathcal{Y}^k$ , to a hypothesis  $h : \mathcal{X} \mapsto \mathcal{Y}$ . We note that our results remain valid for a randomized learner as well, yet we assume the above for ease of exposition.
Proof of Theorem 2. The proof is given via a sample compression scheme, demonstrating that if weak learnability holds, then the final predictor  $\bar{H}$  can be represented using a small number of training examples, and that it is consistent with the entire training set.
First, we fix the sample  $S$  and assume that the  $\gamma$ -BRG condition holds for  $S$  as in the theorem statement. We will then show for each  $j = 1\dots p$  that  $\mu_{j + 1}$  satisfies the following 3 properties: (a) for each  $x\in \mathcal{X}$  it returns at most  $p - j$  labels, (b) for all  $(x,y)\in S$ , it holds that  $y\in \mu_{j + 1}(x)$ , and (c) it can be represented using only a small number of training examples.
First, note that  $\mu_{1}$  is indeed a mapping to at most  $p$  labels by its construction in Algorithm 2. Moreover, recall that by the above assumption, the weak learner is a deterministic mapping from its input to a hypothesis. Therefore, any hypothesis produced by the weak learner within Algorithm 2 can be represented simply by the sequence of  $m_0$  examples on which it was trained. Lemma 4 implies that there is a subset  $S' \subsetneq S$  of size at most  $m_0 \cdot p$ , where  $p = \lceil \log(m) / \gamma \rceil$  such that the following holds: There are  $p$  hypotheses  $h_i': \mathcal{X} \mapsto \mathcal{Y}$ , that comprise the list  $\mu_{1}$ , where each  $h_i'$  can be represented by  $m_0$  examples in  $S'$ . It is also guaranteed by Lemma 4 that for all  $(x, y) \in S$ , it holds that  $y \in \mu_1(x)$ .
Next, we will show that the 3 properties (a)-(c) above holds for  $\mu_{2}$  (and similarly for all  $j\geq 2$ ). Consider the first  $T$  weak hypotheses  $h_1^{(1)},\ldots ,h_T^{(1)}$  generated by Algorithm 3, within its first call to Algorithm 1. Notice that each  $h_t^{(1)}$  can now be represented by the sequence of  $m_0$  examples on which it was trained, as well as the same  $m_0\cdot p$  examples from above that correspond to  $\mu_{1}$ . Therefore, we can represent the mapping  $\mu_{2}$  by a total of  $T\cdot m_0 + m_0\cdot p$  examples. Next, we will show that (a) and (b) hold, by applying Lemma 1. Specifically, we use it to prove that for each  $(x,y)\in S,\mu_2(x)$  returns  $p - 1$  labels, and also that  $y\in \mu_2(x)$ .
We first show that the conditions of Lemma 1 are met, by considering a simple conversion of all the labels according to  $\mu_{1}$ . Specifically, since both Lemma 1 and Algorithm 1 assume the labels are in  $[k]$ , yet both Algorithm 3 and our weak learner assume the labels in  $S$  are in  $\mathcal{V}$ , we can think of mapping each  $y \in \mathcal{V}$  to  $[p + 1]$  according to  $\mu_{1}$ , getting its corresponding label  $\ell \in [p + 1]$  in the mapped space, and then remapping back to the  $\mathcal{V}$  space when returning to Algorithm 3.
Concretely, for each pair  $(x,y)\in S$ , convert it to  $(x,\ell)\in \mathcal{X}\times [p]$ , such that the  $\ell$ -th entry of  $\mu_1(x)$  is  $y$ , denoted  $\mu_{1}(x)_{\ell} = y$ . By Definition 4 we obtain a hypothesis  $h:\mathcal{X}\mapsto \mathcal{Y}$ . For its internal use in Algorithm 1, we convert it into a hypothesis  $h^{\prime}:\mathcal{X}\mapsto [p + 1]$  such that if  $h(x)\in \mu (x)$ , set  $h^\prime (x) = \ell$  for  $\ell$  that satisfies  $\mu_1(x)_\ell = h(x)$ , or  $h^\prime (x) = p + 1$  if there is no such  $\ell \in [p]$ . Finally, we set the output  $H_{1}$  of Algorithm 1 to be defined with respect to the original, remapped, weak hypotheses  $h$ .
Now applying Lemma 1 with  $k \coloneqq p$ , we get that for all  $(x, y) \in S$ , we have  $H_{1}(x, y) > T / p$ . Therefore, it must hold that  $y \in \mu_{2}(x)$ . Moreover, since  $\sum_{y' \in \mu_{1}(x)} H_{1}(x, y') \leq \sum_{y' \in \mathcal{Y}} H_{1}(x, y') \leq T$ ,
Lemma 1 implies that there must be a label  $y^\prime \neq y$  such that  $y^\prime \in \mu_1(x)$  for which  $H_{1}(x,y^{\prime}) < T / p$ . Therefore, by construction of  $\mu_{2}$  we get that  $y^\prime \notin \mu_2(x)$ , and  $|\mu_2(x)|\leq |\mu_1(x)\setminus \{y'\} | = p - 1$ .
Next, we continue in a similar fashion for all rounds  $j = 3,\dots ,p - 1$ . Namely, the same arguments as above show that by applying Lemma 1 with  $k\coloneqq p - j + 2$ , we get that  $\mu_{j + 1}$  satisfies the above conditions over  $S$ . Moreover, to represent each weak hypotheses  $h_t^{(j)}$  generated by Algorithm 3 within its  $j$ -th call to Algorithm 1, we use the sequence of  $m_0$  examples on which it was trained, as well as the same  $(j - 1)\cdot T\cdot m_0 + m_0\cdot p$  examples from above that correspond to  $\mu_j$ .
Overall, we have shown that if  $\mathcal{W}$  satisfies the  $\gamma$ -BRG condition (as given in Definition 4) with respect to  $(S,\gamma,m_0,M)$  then the final predictor  $\bar{H} \coloneqq \mu_p$  is both consistent with the sample  $S$ , and can be represented using only  $r$  examples, where,
$$
r = (p - 1) \cdot T \cdot m _ {0} + m _ {0} \cdot p = O \left(\frac {p \cdot m _ {0} \ln (m)}{\gamma^ {2}}\right) = O \left(\frac {m _ {0} \ln^ {2} (m)}{\gamma^ {3}}\right). \tag {5}
$$
We can now apply a sample compression scheme bound to obtain the final result. Specifically, we apply Theorem 6 (for  $k = 1$ ), for a  $m \to r$  sample compression scheme algorithm  $\mathcal{A}$  equipped with a reconstruction function  $\rho$  (see Definition 8). We denote  $\mathrm{err}_{\mathcal{D}}(\bar{H}) = \operatorname*{Pr}_{(x,y) \sim \mathcal{D}}[\bar{H}(x) \neq y]$ . Then, by Theorem 6 we get that,
$$
\operatorname * {P r} _ {S \sim \mathcal {D} ^ {m}, \mathcal {A}} \left[ \bar {H} \text {c o n s i s t e n t w i t h} S \Rightarrow \operatorname {e r r} _ {\mathcal {D}} (\bar {H}) > \frac {r \ln (m) + \ln (1 / \delta)}{m - r} \right] \leq \delta ,
$$
where the overall randomness of our algorithm is denoted by  $\mathcal{A}$ . Plugging in  $r$  from Equation (5), and  $m$  given in the theorem statement, yields the desired bound.
AD received funding from the European Research Council (ERC) under the European Union's Horizon 2022 research and innovation program (grant agreement No. 101041711), and the Israel Science Foundation (grant number 2258/19). This research was done as part of the NSF-Simons Sponsored Collaboration on the Theoretical Foundations of Deep Learning.
YM received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (grant agreement No. 882396), by the Israel Science Foundation (grant number 993/17), the Yandex Initiative for Machine Learning at Tel Aviv University and a grant from the Tel Aviv University Center for AI and Data Science (TAD).
SM is a Robert J. Shillman Fellow; he acknowledges support by ISF grant 1225/20, by BSF grant 2018385, by an Azrieli Faculty Fellowship, by Israel PBC-VATAT, by the Technion Center for Machine Learning and Intelligent Systems (MLIS), and by the European Union (ERC, GENERALIZATION, 101039692). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency. Neither the European Union nor the granting authority can be held responsible for them.