SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Example 4. Let us return back to Example 1. Instead of considering all three combinations  $(2,0)$ ,  $(1,1)$  and  $(0,2)$  assuming a target  $s = 2$ , we can consider only the first and the last combination, if the probabilities (predicted by  $f$ ) of the two images being 1 are way lower than those of being 0 or 2. Then, the top-2 label vectors  $(2,0)$ ,  $(0,2)$  can be seen as the formula  $(A_{1,2} \wedge A_{2,0}) \vee (A_{1,0} \wedge A_{2,2})$  where  $A_{i,j}$  is a Boolean variable that is true iff the  $i$ -th input digit is assigned label  $j$ .
We use  $\bigvee_{i=1}^{k} \mathbf{y}_{i}$  to denote the Boolean formula computed out of the  $k$  label vectors  $\mathbf{y}_{1}, \ldots, \mathbf{y}_{k}$ , where each  $\mathbf{y}_{i}$  is the conjunction of Boolean variables of the form  $A_{i,y}$ , as in Example 4. Given a softmax score  $f(\mathbf{x})$ , variable  $A_{i,y}$  is assigned probability  $f^{y}(x_{i})$ . We now define the top- $k$  partial loss:
Definition 3 (Top- $k$  partial loss). The top- $k$  partial loss for an integer  $k \geq 1$  subject to a classifier  $f$ , transition  $\sigma$  and sample  $(x, s)$  is defined as
$$
\ell_ {\sigma} ^ {k} (f (\boldsymbol {x}), s) := \operatorname {S L} \left(\bigvee_ {i = 1} ^ {k} \boldsymbol {y} ^ {(i)}, f (\boldsymbol {x})\right) \tag {5}
$$
where  $\pmb{y}^{(1)},\dots,\pmb{y}^{(k)}\in \mathcal{V}^M$  are the top-  $k$  maximizers of  $P_{f(\pmb{x})}$  in the preimage of  $s$ , i.e.,  $\{\pmb {y}\in \mathcal{V}^M:\sigma (\pmb {y}) = s\}$ . The top- $k$  partial classification risk of  $f$  subject to  $\sigma$  is then given by
$$
\mathcal {R} _ {\mathsf {P}} ^ {k} (f; \sigma) := \mathbb {E} _ {(X _ {1}, \dots , X _ {M}, S) \sim \mathcal {D} _ {\mathsf {P}}} \left[ \ell_ {\sigma} ^ {k} \left(f \left(X _ {1}, \dots , X _ {M}\right), S\right) \right] \tag {6}
$$
Remark. The WMC is upper bounded by the sum of probabilities, so we can approximate the SL as  $\ell_{\sigma}^{k}(f(\pmb{x}), s) \geq -\log \left( \sum_{i=1}^{k} P_{f(\pmb{x})} (\pmb{y}^{(i)}) \right)$ . Furthermore, the special case where  $k = 1$  reduces to the infimum loss [5] and minimal loss [29] in the PLL literature as we show in Appendix B.2.
Let  $\Re_m(\mathcal{F})$  denote the Rademacher complexity [4, 10] of  $\mathcal{F}$  with  $m$  samples as defined in Appendix A. We are now ready to bound the zero-one classification risk with the empirical top- $k$  partial risk.
Theorem 2 (Error bound under unambiguity). Let an integer  $k \geq 1$  and  $\delta \in (0,1)$ . If  $\sigma$  is both 1- and  $M$ -unambiguous, then with probability at least  $1 - \delta$ , we have:
$$
\mathcal {R} ^ {0 1} (f) \leq \Phi \left((k + 1) \left(\widehat {\mathcal {R}} _ {\mathsf {P}} ^ {k} (f; \sigma ; \mathcal {T} _ {\mathsf {P}}) + 2 \sqrt {k} M ^ {3 / 2} \Re_ {M m _ {\mathsf {P}}} (\mathcal {F}) + \sqrt {\frac {\log (1 / \delta)}{2 m _ {\mathsf {P}}}}\right)\right) \tag {7}
$$
where  $\widehat{\mathcal{R}}_{\mathsf{P}}^{k}(f;\sigma ;\mathcal{T}_{\mathsf{P}}) = \sum_{(\boldsymbol {x},s)\in \mathcal{T}_{\mathsf{P}}}\ell_{\sigma}^{k}(f(\boldsymbol {x}),s) / m_{\mathsf{P}}$  is the empirical counterpart of (6) and  $\Phi$  is an increasing function that satisfies  $\lim_{t\to 0}\Phi (t) / t = 1$
Proof sketch. Theorem 2 builds upon several results. Firstly, we derived an inequality that bounds the top- $k$  loss with the zero-one loss (Lemma 5), which requires the construction of an intermediate  $\ell^1$  loss (Definition 11). Secondly, we show the Lipschtness of the semantic loss (Lemma 7). This result is further combined with a contraction lemma that is proposed in [10] (Lemma 6) to bound the Rademacher complexity of the model.
Remark. Similarly to Theorem 1 and Proposition 1, Theorem 2 suggests that the learning difficulty increases as  $M$  increases. This is intuitive, since it gets harder to disambiguate the gold labels when  $M$  increases, e.g., for 2-SUM, there are  $10^{2}$  possible label vectors need to be considered, while for 4-SUM, there are  $10^{4}$  ones. Our bound also suggests a tradeoff with the choice of  $k$ : a larger  $k$  decreases the risk, but tends to increase the complexity term.
4 Learning Multiple Classifiers Under a Known Transition
In this section, we extend the learning problem from Section 3 to jointly learn  $n \geq 1$  different classifiers. Similarly to Section 3, we aim to propose minimal assumptions for the data distribution to show both results on ERM-learnability (Theorem 3) and a Rademacher-style error bound with the top- $k$  loss (Theorem 4). We formally define the learning setting below.
Problem setting. We aim to learn  $n \geq 1$  classifiers, each of which maps instances from a space  $\mathcal{X}_i$  to the corresponding label space  $\mathcal{Y}_i$  with  $|\mathcal{Y}_i| = c_i$ . For each  $i \in [n]$ , we define a scoring space  $\mathcal{F}_i$  which contains mappings of the form  $f_i: \mathcal{X}_i \to \Delta_{c_i}$ . Each  $f_i$  is used to classify  $M_i \geq 1$  instances  $(x_{i1}, \ldots, x_{iM_i}) = \boldsymbol{x}_i \in \mathcal{X}_i^{M_i}$  with (hidden) gold labels  $\boldsymbol{y}_i = (y_{i1}, \ldots, y_{iM_i})$ . We denote the vector of scoring functions as  $\boldsymbol{f} = (f_1, \ldots, f_n)$ . Let  $\mathcal{D}_i$  be the joint distribution over elements from  $\mathcal{X}_i \times \mathcal{Y}_i$ . Each training sample given to the learner  $(\boldsymbol{x}_1, \ldots, \boldsymbol{x}_n, s)$  is formed by (i) drawing  $M_i$  i.i.d. samples  $(\boldsymbol{x}_i, \boldsymbol{y}_i)$  from  $\mathcal{D}_i$ , for each  $i \in [n]$  and then (ii) obtaining its partial label as  $s = \sigma(\boldsymbol{y}_1, \ldots, \boldsymbol{y}_n)$ . We override the notation  $\mathcal{D}_{\mathsf{P}}$  to denote the distribution of the training samples in this learning setting. The partially labeled dataset  $\mathcal{T}_{\mathsf{P}}$  then contains  $m_{\mathsf{P}}$  i.i.d. samples drawn from  $\mathcal{D}_{\mathsf{P}}$ .
Similarly to Section 3, the learner aims to find the classifiers  $f_{1},\ldots ,f_{n}$  with the minimal zero-one risk defined as  $\mathcal{R}^{01}(\pmb {f})\coloneqq \sum_{i = 1}^{n}\mathcal{R}^{01}(f_{i})$ . As the gold labels are hidden, the learner uses the dataset  $\mathcal{T}_{\mathbb{P}}$  to estimate and minimize the zero-one partial risk of  $f_{1},\dots ,f_{n}$ , which is defined as  $\mathcal{R}_{\mathbb{P}}^{01}(f_1,\ldots ,f_n;\sigma)\coloneqq \mathbb{E}_{(\mathbf{X}_1,\ldots ,\mathbf{X}_n,S)}\sim \mathcal{D}_{\mathbb{P}}[\ell_{\sigma}^{01}(([f_1](\mathbf{X}_1),\ldots ,[f_n](\mathbf{X}_n)),S)]$ , where  $[f_i](\mathbf{X}_i)$  is a short for  $([f_i](X_{i1}),\ldots ,[f_i](X_{iM_i}))$ , for  $1\leq i\leq M_{i}$ .
Example 5 (Learning binary operators). Consider a setting where we aim to train a classifier  $f_{1}$  for recognizing MNIST digits and a classifier for recognizing images of addition and multiplication taken from some space  $\mathcal{X}_{2}$ . The training samples are of the form  $(x_{11}, x_{12}, x_{21}, s)$ , where  $x_{11}$  and  $x_{12}$  are non-zero MNIST digits with  $\mathcal{Y}_{1} = \{1, \ldots, 9\}$ ,  $x_{21}$  is an image of an operator with  $\mathcal{Y}_{2} = \{+, \times\}$ , and  $s$  is the result of applying the operator in  $x_{2}$  on the digits in  $x_{11}$  and  $x_{12}$ .
4.1 ERM Learnability
Similarly to Section 3, to propose a necessary condition for learnability, we consider the most challenging distribution where each  $\mathcal{D}_i$  is concentrated on a single instance  $x_i^* \in \mathcal{X}_i$ , for  $1 \leq i \leq n$ . Then, the only sample that will be observed during learning is  $(\boldsymbol{x}_1^*, \ldots, \boldsymbol{x}_n^*)$  where  $\boldsymbol{x}_i = (x_i^*, \ldots, x_i^*)$ , for  $1 \leq i \leq n$ . Assuming that  $l_i^*$  is the gold label of  $x_i^*$ , detecting a classification error is possible only if a misclassification of any  $x_i^*$ , i.e.,  $[f_i](x_i^*) \neq l_i^*$ , will lead to a prediction vector  $\boldsymbol{y}'$  with a different image under  $\sigma$  from the gold label  $\boldsymbol{y}$ , i.e.,  $\sigma(\boldsymbol{y}') \neq \sigma(\boldsymbol{y})$ . The above intuition is captured via the notion of multi-unambiguity, which generalizes the notion of  $M$ -unambiguity. Below, we use  $\boldsymbol{y}_i^{M_i}$ , for  $i \in [n]$ , to denote the  $M_i$ -ary diagonal vector including the  $y_i$  element only.
Definition 4 (Multi-unambiguity). Transition  $\sigma$  is multi-unambiguous if for any vector  $\mathbf{y} = (\mathbf{y}_1^{M_1},\dots,\mathbf{y}_n^{M_n})$ , and any position  $i\in [n]$ , such that the vector  $\mathbf{y}'$  that results after flipping the labels in  $\mathbf{y}_i^{M_i}$  to some diagonal vector  $(\mathbf{y}_i')^{M_i}\neq \mathbf{y}_i^{M_i}$ , has a different image under  $\sigma$ , i.e.,  $\sigma(\mathbf{y})\neq \sigma(\mathbf{y}')$ .
Multi-unambiguity reduces to  $M$ -unambiguity for  $n = 1$ . An example of multi-unambiguity is below.
Example 6 (Learning binary operator, cont'd). In Example 5, the multi-unambiguity condition is violated since  $2 + 2 = 2 \times 2$ . Namely, if a distribution assigns all its weight to the digit 2, then it is difficult for the model to distinguish the two operators. On the other hand, if instead, the label space is  $\mathcal{Y}_1 = \{3, \ldots, 9\}$ , then the multi-unambiguity condition is satisfied.
Bounded risk assumption. Differently from Section 3, multi-unambiguity alone is not sufficient to ensure learnability. To see this, consider a variant of SUM2 where the digits are classified by two independent classifiers,  $f_{1}$  and  $f_{2}$ . If the distributions of the first and the second image are concentrated on 1 and 7 respectively, but  $f_{1}(x_{1}) = 7$  and  $f_{2}(x_{2}) = 1$ , then these errors cannot be detected. Therefore, we assume there is a constant  $R < 1$ , such that  $\mathcal{R}^{01}(f_i) \leq R$  holds for any  $i \in [n]$  and any  $f \in \mathcal{F}_i$ . We refer to such  $f_{i}$ 's as zero-one risk  $R$ -bounded. This assumption is mild since it only requires the classifiers to be slightly better than being totally wrong. Also, when a small directly labeled dataset is available, one can bound the classification risks away from 1 with high probability in a uniform manner using standard learning theory. See Appendix C for more details. Now, we are ready to state the ERM-learnability result assuming  $\prod_{i=1}^{n} \mathcal{F}_{i}$  is realizable under  $\ell_{\mathrm{P}}^{01}$ .
Theorem 3 (ERM learnability under multi-unambiguity). Assume that there is a constant  $R < 1$  such that for each  $i \in [n]$ , each  $f \in \mathcal{F}_i$  is zero-one risk  $R$ -bounded. Assume also that there exist positive integers  $M^*$  and  $c^*$ , such that  $M_i \leq M^*$  and  $c_i \leq c_0$  hold for any  $i \in [n]$ . Then, if  $\sigma$  is multi-unambiguous, we have:
$$
\mathcal {R} ^ {0 1} (\boldsymbol {f}) \leq \mathcal {O} \left(\left(\mathcal {R} _ {\mathrm {P}} ^ {0 1} (\boldsymbol {f}; \sigma)\right) ^ {1 / M ^ {*}}\right) \quad a s \quad \mathcal {R} _ {\mathrm {P}} ^ {0 1} (\boldsymbol {f}; \sigma) \rightarrow 0 \tag {8}
$$
Furthermore, for any  $\epsilon, \delta \in (0,1)$ , there is a universal constant  $C_3$ , such that with probability at least  $1 - \delta$ , the empirical partial risk minimizer with  $\widehat{\mathcal{R}}_{\mathsf{P}}^{01}(f;\sigma) = 0$  has a classification risk  $\mathcal{R}^{01}(f) < \epsilon$  if
$$
m _ {\mathrm {P}} \geq C _ {3} \frac {n c _ {0} ^ {2 M ^ {*} - 2}}{\epsilon^ {M ^ {*}} (1 - R) ^ {M}} \left(\sum_ {i = 1} ^ {n} d _ {[ \mathcal {F} _ {i} ]} \log \left(n c _ {i} M _ {i} d _ {[ \mathcal {F} _ {i} ]}\right) \log \left(\frac {n c _ {0} ^ {2 M ^ {*} - 2}}{\epsilon^ {M ^ {*}} (1 - R) ^ {M}}\right) + \log \left(\frac {1}{\delta}\right)\right) \tag {9}
$$
Remark. Theorem 3 suggests that the rate of convergence is controlled by  $M^{*}$ . A smaller  $M^{*}$  leads to a more strict unambiguity condition. For example, the case  $M^{*} = 1$  requires  $\sigma$  to be unstable to any 1-position perturbations, while the case  $M^{*} = M$  reduces to  $M$ -unambiguity. This observation confirms the intuition that the supervision power of partial labels depends on the "instability" of  $\sigma$ .
The top- $k$  partial loss  $\ell_{\sigma}^{k}(\pmb{f}(\pmb{x}), s)$  for multiple classifiers subject to  $\pmb{f} \in \mathcal{F}^n$ , transition  $\sigma$  and  $s \in S$  straightforwardly extends the top- $k$  partial loss for the single classifier case, see Definition 3. Similarly, the (empirical) top- $k$  partial classification risk of  $\pmb{f}$  subject to  $\sigma$  straightforwardly extends the one from Section 3.2. Below, we provide a Rademacher-style error bound with the top- $k$  loss.
Theorem 4 (Error bound under multi-unambiguousity with multiple classifiers). Suppose  $\sigma$  is multi-unambiguous and each  $f_{i}$  is zero-one risk  $R$ -bounded, for  $R \in (0,1)$ . Then, for any integer  $k \geq 1$  and any  $\delta \in (0,1)$ , with probability at least  $1 - \delta$ , we have:
$$
\mathcal {R} ^ {0 1} (\boldsymbol {f}) \leq \left(\frac {n c _ {0} ^ {2 M ^ {*} - 2} (k + 1)}{(1 - R) ^ {M}} \left(\widehat {\mathcal {R}} _ {\mathsf {P}} ^ {k} (\boldsymbol {f}; \sigma ; \mathcal {T} _ {\mathsf {P}}) + \sqrt {k M} \sum_ {i = 1} ^ {n} M _ {i} \Re_ {m _ {\mathsf {P}} M _ {i}} (\mathcal {F} _ {i}) + \sqrt {\frac {\log (1 / \delta)}{2 m _ {\mathsf {P}}}}\right)\right) ^ {1 / M ^ {*}}
$$
5 Learning Under an Unknown Transition
We now explore another direction by dropping the assumption that  $\sigma$  is known. Instead, we assume the learner has access to a transition space  $\mathcal{G}$  that contains mappings of the form  $\mathcal{V}^M\to S$  including the "true" transition  $\sigma$ . Notice that learning now becomes particularly challenging, since  $\mathcal{G}$  can be expressive enough to lead to correct predictions for  $s$  from many wrong classifications.
Problem setting. To illustrate the core idea, we consider a simple setting where we only learn a single scoring function  $f \in \mathcal{F}$  as in Section 3. However,  $\sigma$  is an unknown mapping in  $\mathcal{G}$ . Given partially labeled samples, the learner aims to minimize the zero-one partial risk of  $f$  subject to  $\mathcal{G}$ ,  $\mathcal{R}_{\mathsf{P}}^{01}(f; \mathcal{G})$  that is defined as the minimal partial risk to predict  $s$  with a candidate transition in  $\mathcal{G}$ , i.e.,  $\mathcal{R}_{\mathsf{P}}^{01}(f; \mathcal{G}) := \inf_{\sigma^* \in \mathcal{G}} \mathcal{R}_{\mathsf{P}}^{01}(f; \sigma^*)$ . The risk is empirically estimated with a partially labeled dataset  $\mathcal{T}_{\mathsf{P}}$  as  $\inf_{\sigma^* \in \mathcal{G}} \widehat{\mathcal{R}}_{\mathsf{P}}^{01}(f; \sigma^*; \mathcal{T}_{\mathsf{P}})$ . We demonstrate this learning setting with the following example:
Example 7. Let  $\mathcal{G} = \{(y_1, y_2) \mapsto \alpha y_1 + \beta y_2 | (\alpha, \beta) \in \mathbb{R}^2 - \{(0,0)\}\}$ , namely all the weighted sums with at least one non-zero weight. We aim to learn an MNIST classifier  $f$ , given training samples of the form  $(x_1, x_2, s)$ , where the  $x_i$ 's are MNIST digits and  $s$  is the result of applying some  $\sigma$  from  $\mathcal{G}$  on  $(y_1, y_2)$ , where  $y_i$  is the prediction of  $f$  on  $x_i$ . The gold  $\sigma$ , i.e., the exact  $\alpha$  and  $\beta$ , is unknown.
5.1 ERM Learnability
Similarly to the known transition case, learnability requires us to learn under any data distribution  $\mathcal{D}$ . Slightly differently from Sections 3 and 4, we start with the adversarial distribution where the mass in  $\mathcal{D}$  is concentrated to a single label, i.e., all instances  $x_{i}$  have the same gold label  $l_{i}$ , and hence, the training samples are all associated with the gold label vector  $\pmb{y} = (l_{i},\dots ,l_{i})$ . Suppose there is a certain probability that  $f$  misclassifies  $l_{i}$  as  $l_{j}$ . Then, the predicted label vectors will be in  $\{l_i,l_j\} ^M$ . In this case, it will be impossible to detect the classification errors if there is a candidate transition  $\sigma^{\prime}\in \mathcal{G}$  which maps all the vectors in  $\{l_i,l_j\} ^M$  to the gold observation of  $s$ . Since the actual  $\sigma$  is unknown, we require this does not happen for any pair of  $\sigma ,\sigma^{\prime}\in \mathcal{G}$  to ensure learnability. Below, we formalize our learnability condition:
Definition 5 (Unambiguous transition space). Transition space  $\mathcal{G}$  is unambiguous, if for each  $\sigma' \in \mathcal{G}$ , each diagonal label vector  $\pmb{y} = (l_i, \dots, l_i)$ , where  $l_i \in \mathcal{V}$ , and each  $l_j \neq l_i$ , where  $l_j \in \mathcal{V}$ , there exists a vector  $\pmb{y}' \in \{l_i, l_j\}^M$ , such that  $\sigma'(\pmb{y}') \neq \sigma(\pmb{y})$ .
In practice, one can examine whether a transition space is unambiguous by checking if for each  $\sigma' \in \mathcal{G}$  and any two different labels  $l_i \neq l_j$ , set  $\{\sigma'(\pmb{y}) | \pmb{y} \in \{l_i, l_j\}^M\}$  is not a singleton. The above ensures that when given a fixed diagonal label vector, and when the classifier makes mistakes, the predicted partial labels are not unique, and hence cannot all agree with the ground truth label.
Class  $\mathcal{G}$  in Example 7 is unambiguous: for each transition  $(y_{1},y_{2})\mapsto \alpha^{\prime}y_{1} + \beta^{\prime}y_{2}$  and each two labels  $l^{\prime}\neq l$  , the set  $\{\alpha^{\prime}y_{1} + \beta^{\prime}y_{2}|(y_{1},y_{2})\in \{l,l^{\prime}\}^{2}\}$  is not a singleton, so there must exist a label vector with a different weighted sum that is different than the true sum. A counterexample is below:
Example 8. Let  $\mathcal{G} = \{(y_1, y_2) \mapsto w_1y_1 + w_2y_1^2 + w_3y_2^2 + w_4y_2 | w_i \neq 0 \forall i \in [4]\}$ . Consider the true transition  $\sigma : (y_1, y_2) \mapsto y_1 - y_1^2 + y_2 - y_2^2$  from  $\mathcal{G}$  and a candidate transition  $\sigma' : (y_1, y_2) \mapsto y_1 - y_1^2 - y_2 + y_2^2$ . Then,  $\mathcal{G}$  is not unambiguous, since  $\sigma'(\pmb{y}) = \sigma(\pmb{y}) = 0$  for any  $\pmb{y} \in \{0, 1\}^2$ . Namely, the partially labeled data are not informative enough to distinguish the label 0 from 1.
Bounded risk assumption. Similarly to Section 4, this unambiguity condition is necessary but not sufficient to show learnability. To show learnability, we additionally assume there is a constant  $r > 0$  such that  $\mathbb{P}([f](X) = y | Y = y) \geq r$  for each  $y \in \mathcal{V}$  and  $f \in \mathcal{F}$ . We refer to such an  $f$  as  $r$ -bounded. This assumption is slightly stronger than that of Section 4 as it additionally requires the classifiers to be not fully wrong for every possible label. The main learnability result of this section is below:
Theorem 5. If  $\mathcal{G}$  is unambiguous and any  $f\in \mathcal{F}$  is  $r$  -bounded, then we have:
$$
\mathcal {R} ^ {0 1} (f) \leq O \left(\mathcal {R} _ {\mathsf {P}} ^ {0 1} (f; \mathcal {G}) ^ {1 / M}\right) \quad a s \quad \mathcal {R} _ {\mathsf {P}} ^ {0 1} (f; \mathcal {G}) \rightarrow 0 \tag {10}
$$
Furthermore, suppose  $[\mathcal{F}]$  has a finite Natarajan dimension  $d_{[\mathcal{F}]}$  and the function class  $\{(y,s)\mapsto \mathbb{1}\{\sigma^{\prime}(\pmb {y})\neq s\} |\sigma^{\prime}\in \mathcal{G}\}$  has a finite VC-dimension  $d_{\mathcal{G}}$ . Then, for any  $\epsilon ,\delta \in (0,1)$ , there is a universal constant  $C_4$  such that with probability at least  $1 - \delta$ , the empirical partial risk minimizer with
$\widehat{\mathcal{R}}_{\mathbb{P}}^{01}(f;\sigma) = 0$  has a classification risk  $\mathcal{R}^{01}(f) < \epsilon$ , if
$$
m _ {\mathsf {P}} \geq C _ {4} \frac {c ^ {2 M - 2}}{r ^ {M} \epsilon^ {M}} \left(\left((d _ {[ \mathcal {F} ]} + d _ {\mathcal {G}}) \log (6 M (d _ {[ \mathcal {F} ]} + d _ {\mathcal {G}})) + d _ {[ \mathcal {F} ]} \log c\right) \log \left(\frac {c ^ {2 M - 2}}{r ^ {M} \epsilon^ {M}}\right) + \log \left(\frac {1}{\delta}\right)\right)
$$
We further explore the learning scenario in Section 5 with empirical evaluations. We aim to learn an MNIST classifier using the weighted sum of 2, 3 and 4 MNIST digits and assuming that the weights are unknown, as in Example 7. The weighted sum function is very expressive as it can model Boolean formulas including conjunction and disjunction [36]. Notice that we do not aim to exhaustively assess the weak supervision literature ([24, 43, 59, 13]), but to validate the results of our theoretical analysis.
Baselines. We considered state-of-the-art neuro-symbolic frameworks, namely DeepProbLog (DLog) [30], DeepProbLog with approximations (DLog-A) [31], NeuroLog (NLog) [43], NeurASP (NASP) [59], ABL [13], ENT [27] and Scallop [24]. Only the target sum is used during training under those frameworks. DLog and NLog employ the semantic loss without any approximations (see Section 2).  $\mathrm{NLog}(k)$  denotes an NLog variant where only the top-  $k$  predictions are considered during training, while Scallop  $(k)$  denotes Scallop using the top-  $k$  semantic loss, see Section 3.2. The approximations in DLog-A are different from the ones in NLog and Scallop; however, both DLog-A and NLog employ the semantic loss on the chosen subset of proofs. As the above frameworks learn classifiers under fixed theories only, we considered weights in  $\{1,2,3,4,5\}$ , and used an additional neural classifier to learn the unknown weights. Finally, we considered standard supervised learning (SL). To contrast learning under unknown to learning under known transitions, we also ran experiments assuming that the weights are known. Our results show that the classification accuracy exceeds the  $98\%$  even for  $M = 10$ . More details on this experiment are in the arXiv version of this paper [51].
Results. Tables 1 and 2 report the accuracy of the learned MNIST classifiers over ten runs. #0, #8 and #16 denote the number of directly labeled samples used for pretraining the MNIST classifiers: #0 means no pretraining; for #8 and #16, the accuracy of the pre-trained classifiers is  $37\%$  and  $58\%$ . Table 1 presents results for  $M = 2$ , while Table 2 presents results for  $M \in \{2,3,4\}$  for Scallop and the two best performing frameworks in Table 1, DLog and NLog- NASP could not scale for  $M \geq 3$ . We used DLog-A for  $M \in \{3,4\}$ , due to scalability reasons. Similarly, NLog without approximations does not scale for  $M = 4$  (N/A indication). In Table 1, the number of weak samples is in parentheses, e.g., DLog(#4K). For ABL and ENT, we used 10K samples, as they rely on sampling, and used the hyperparameters suggested by the authors.
Discussion. The results confirm our theory: (i) we can learn classifiers under unknown transitions as per Theorem 5 (see the accuracy for DLog and NLog in Table 1); and (ii) learning gets harder when  $M$  increases as per Eq. (5) (see the accuracy for different  $M$ 's in Table 2). For  $M = 2$ , weak supervision leads to better accuracy than SL even without pre-training and with fewer training samples: DLog(#4K) and NLog(#4K) lead to mean accuracy  $96\%$  and  $97\%$ , while SL(#4K) leads to mean accuracy  $93\%$ , see Table 1. This result suggests that the  $r$ -bounded assumption in Theorem 5 could be potentially relaxed. Another observation is that the approximations in DLog-A are less effective than the top-  $k$  one of NLog. We leave the theoretical analysis of the latter approximation as part of future research. Table 2 also suggests that the top-  $k$  semantic loss constitutes the most scalable and effective approach to training: in contrast to DLog-A and NLog, the accuracy of the digit classifier reaches  $78\%$  for  $M = 4$ , when  $k = 4$  for Scallop.