SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kaifu Wang
kaifu@sas.upenn.edu
Efhymia Tsamoura
Samsung AI
efi.tsamoura@samsung.com
Dan Roth
danroth@seas.upenn.edu
We consider a weakly supervised learning scenario where the supervision signal is generated by a transition function  $\sigma$  of labels associated with multiple input instances. We formulate this problem as multi-instance Partial Label Learning (multi-instance PLL). Our problem is an extension to the standard PLL problem and is met in different fields, including latent structural learning and neuro-symbolic integration. Despite the existence of many learning techniques, limited theoretical analysis has been dedicated to this problem. In this paper, we provide the first theoretical study of multi-instance PLL with possibly an unknown transition  $\sigma$ . Our main contributions are as follows. First, we propose a necessary and sufficient condition for the learnability of the problem. This condition nontrivially generalizes and relaxes the existing small ambiguity degree in PLL literature since we allow the transition to be deterministic. Second, we derive Rademacher-style error bounds based on a top- $k$  surrogate loss that is widely used in the neuro-symbolic literature. Furthermore, we conclude with empirical experiments for learning under unknown transitions. The empirical results align with our theoretical findings, exposing also the issue of scalability in the weak supervision literature.
Consider the scenario in Figure 1, where a learner aims to learn one or more classifiers  $f_{1},\ldots ,f_{n}$  each of which maps an instance  $x$  to its corresponding label  $y$ . The learner is given training examples, each of which consists of a vector of  $M > 1$  instances  $\pmb {x} = (x_{1},\dots,x_{M})$ , and each  $x_{i}$  is processed by one of the  $f_{j}$ 's. Differently from supervised learning, the gold labels  $\pmb {y} = (y_{1},\dots,y_{M})$  of  $\pmb{x}$  are hidden; instead, the learner is provided with a weak label  $s$  which is produced by applying a transition function  $\sigma$  to the gold labels  $\pmb{y}$ . The transition  $\sigma$  itself may be unknown to the learner.
The above weak supervision setting has been a topic of active research in NLP [42, 35, 34, 32, 45, 52, 22]. Recently, it has received renewed attention as it is met in neuro-symbolic frameworks [14], i.e., frameworks that integrate inference and learning over neural models with inference and learning over symbolic models, such as logical theories [30, 53, 13, 59, 43, 31, 24, 27]. The benefits of this setting over architectures that approximate the  $f_{i}$ 's and  $\sigma$  via an end-to-end neural model [48] are (i) the ability to reuse the latent models, something particularly useful in NLP [34, 32], (ii) higher accuracy in multiple tasks, e.g., NLP [55] and visual question answering [24], (iii) greater interpretability and (iv) the ability to encode prior knowledge via  $\sigma$ . Below, we present an example of our learning setting in neuro-symbolic learning.
Example 1 (SUM2). We aim to learn an MNIST classifier  $f$ . Differently from supervised learning, we use training examples of the form  $(x_{1}, x_{2}, s)$ , where  $x_{1}$  and  $x_{2}$  are MNIST digits and  $s$  is
<sup>1</sup>We use the term transition function (or transition in short) instead of simply function, due to the relationship between our setting and learning with partial labels literature [8, 9].
their sum. The above implies a transition function  $\sigma : \{0, \ldots, 9\} \times \{0, \ldots, 9\} \to \{0, \ldots, 18\}$ , s.t.  $\sigma(y, y') = y + y'$ . The target sum  $s$  restricts the space of labels that can be assigned to the input pair of digits  $(x_1, x_2)$ , e.g., if  $s = 2$ , then the only combinations of labels that abide by the constraint that the sum is 2 are  $(2,0)$ ,  $(1,1)$  and  $(0,2)$ . This problem is referred to as SUM2 in literature [24, 30].
The transition function  $\sigma$  is not necessarily one-to-one. For instance, multiple combinations of digits can lead to the same target sum. The above leaves us with two questions to answer: can we provide any learning guarantees for the classifiers  $f_{1},\ldots ,f_{n}$ ? and what if  $\sigma$  is unknown? Despite that multiple techniques for neuro-symbolic and latent structural learning have been recently proposed, those theoretical questions remain open. Practically, for the single input case  $(M = 1)$ , our setting can reduce to that of partial label learning (PLL), where each input instance is accompanied by a set of labels, including the gold one [25, 11, 5, 29, 39, 54, 57, 60]. However, a key PLL learnability assumption - that the probability a
Figure 1: Multi-Instance PLL. We aim to learn the  $f_{i}$ 's given the  $x_{i}$ 's and  $s$ .  $M$  may be different from  $n$  and  $\sigma$  may be unknown.
wrong label co-occurs with the gold one is always less than one [28]– is violated in our setting, rendering existing learnability results inapplicable. This is because  $\sigma$  is a deterministic but not necessarily bijective function, as opposed to  $\sigma$  in PLL which is randomized.
Due to its relevance to PLL, we refer to our learning setting as multi-instance PLL. Differently from prior art, e.g., [42, 35], we do not restrict  $\sigma$  to specific types of functions, e.g., linear functions. Hence, we can express constraints in several formal languages, including systems of Boolean equations and Datalog [1], a language widely used in neuro-symbolic techniques [24]. Notice that many neuro-symbolic learning techniques are oblivious to the implementation of the symbolic component: abstracting the symbolic component as a transition  $\sigma$  has been actually proposed as the means to compositionally integrate neural with symbolic models [43]. Furthermore, although we primarily consider deterministic transitions motivated by the neuro-symbolic and NLP literature, our results can be extended to support randomized transitions when  $\sigma$  is known.
We aim to make minimal assumptions on the data distributions by showing learnability even under the "toughest" distributions. In addition, we provide learning guarantees under the semantic loss [56], a widely-used surrogate loss for training classifiers subject to logical theories [30, 43, 24]. To our knowledge, we are the first to provide this theoretical analysis, closing a gap in the neuro-symbolic and latent structural learning literature.
Contributions. Our contributions can be summarized into the following:
- We propose necessary and sufficient learnability conditions of the multi-instance PLL problem assuming all the instances are classified by a single classifier  $f$  and the transition is known. We further provide a Rademacher-style error bound using a top- $k$  approximation to the semantic loss [56], see Section 3. Our analysis confirms the intuition that a larger  $k$  decreases the empirical risk, but increases the number of samples required for learning.
- We extend the above results for the more general case where one learns multiple classifiers  $\mathbf{f} = (f_{1},\dots ,f_{n})$  that classify instances from different domains, see Section 4.
- We prove learnability with a single classifier and an unknown transition function, see Section 5, and assess the validity of our theoretical results using SOTA neuro-symbolic frameworks, see Section 6.
Proofs, additional backgrounds and details on our empirical analysis are in the appendix.
We use  $\mathbb{P}(\cdot)$  to denote probability mass,  $\mathbb{E}[\cdot]$  to denote expectation and  $\mathbb{1}\{\cdot\}$  to denote an indicator function. For a positive integer  $M$ , we use  $[M]$  as a short for  $\{1, \ldots, M\}$ . For a vector of  $M$  elements  $(x_1, \ldots, x_M)$ , a position  $j$  is a number in  $[M]$  used to identify the element  $x_j$ . A vector is said to be diagonal if its elements in all positions are equal. Suppose  $f$  is a function on a domain  $\mathcal{X}$  and  $\pmb{x}$  is a vector of  $M$  elements in  $\mathcal{X}$ , we use the symbol  $f(\pmb{x})$  to denote the vector  $(f(x_1), \ldots, f(x_M))$ .
Classifiers. Let  $\mathcal{X}$  denote an instance space and  $\mathcal{Y}$  denote an output space with  $|\mathcal{Y}| = c$ . Let also  $\mathcal{D}$  be the joint distribution of two random variables  $(X,Y)\in \mathcal{X}\times \mathcal{Y}$  and  $\mathcal{D}_X$  be the marginal of  $X$ . Throughout the text, we use  $\pmb{x}$  for  $(x_{1},\ldots ,x_{M})$  and  $\pmb{y}$  for the corresponding vector of gold labels. We consider scoring functions of the form  $f:\mathcal{X}\to \Delta_c$ , where  $\Delta_c$  is the space of probability distributions on  $\mathcal{V}$  (e.g.,  $f$  outputs the softmax probabilities of a neural network). We use  $f^{j}(x)$  to denote the  $j$ -th output of  $f(x)$ . A scoring function  $f$  induces a classifier whose prediction on  $x[f](x)$  is defined by  $[f](x)\coloneqq \operatorname{argmax}_{j\in [c]}f^{j}(x)$ . We use  $\mathcal{F}$  and  $[\mathcal{F}]$  to denote the space of scoring functions and the space of classifiers induced by  $\mathcal{F}$ , respectively. We use the VC-dimension, the Natarajan dimension and Rademacher complexities to characterize the complexity of  $\mathcal{F}$  [40].
Loss functions. We define the problem of learning a scoring function using a fixed set of training samples as the one of choosing the scoring function from  $\mathcal{F}$  that better fits the training samples. We give semantics to the term fits by using the risk  $\mathcal{R}(f;\ell)$  of  $f$  subject to a sampling distribution and a given loss function  $\ell$ : the lower  $\mathcal{R}(f;\ell)$  becomes, the better  $f$  fits the data. When there exists an  $f^{*} \in \mathcal{F}$ , such that  $\mathcal{R}(f^{*};\ell) = 0$ , we say that the space  $\mathcal{F}$  is realizable under  $\ell$ . In supervised learning, we are provided with samples of the form  $(x,y)$  drawn from  $\mathcal{D}$ . We use  $\widehat{\mathcal{R}}$  to denote the empirical risk, i.e., the average risk computed over the training samples only. The risk $^2$  of  $f$  subject to a loss function  $\ell: \mathcal{Y} \times \mathcal{Y} \to \mathbb{R}^+$  is given by  $\mathcal{R}(f;\ell) := \mathbb{E}_{(X,Y) \sim \mathcal{D}}[\ell([f](X),Y)]$ .
A commonly used loss function is the zero-one loss defined by  $\ell^{01}(y, y') := \mathbb{1}\{y' \neq y\}$  for any pair  $y, y' \in \mathcal{V}$ . We use  $\mathcal{R}^{01}(f)$  as a short for  $\mathcal{R}(f; \ell^{01})$  and refer to it as the zero-one risk of  $f$ . We will also consider Semantic Loss (SL) [56], which has been adopted to train classifiers subject to Boolean formulas [30, 43, 24]. Let  $\varphi$  be a Boolean formula where each variable in  $\varphi$  is associated with a class from  $\mathcal{V}$ . Then, the SL of  $\varphi$  subject to  $f(x)$  is the negative logarithm of the weighted model counting (WMC) [7] of  $\varphi$  under  $f(x)$ , namely  $\mathrm{SL}(\varphi, f(x)) := -\log(\mathrm{WMC}(\varphi, f(x)))$ , where  $\mathrm{WMC}(\varphi, f(x))$  takes values in  $(0, 1)$  and denotes the probability that  $\varphi$  is logically satisfied when its variables become true with probabilities specified by  $f(x)$ .
3 Learning a Single Classifier Under a Known Transition
We begin with the setting where the goal is to learn a single classifier  $f \in \mathcal{F}$  under a known transition  $\sigma$ . Firstly, we show ERM-learnability (Theorem 1). Then, inspired by neuro-symbolic learning, we provide a Rademacher-style error bound with a top- $k$  surrogate loss based on WMC (Theorem 2). Below, we formally introduce the learning setting.
Problem setting. Let  $\sigma : \mathcal{V}^M \to S$  be a transition function, where  $S = \{1, \dots, d\}$  with  $|\mathcal{S}| = d \geq 1$ . Let also  $\mathcal{T}_{\mathbb{P}}$  be a set of  $m_{\mathbb{P}}$  partially labeled samples of the form  $(x, s) = (x_1, \dots, x_M, s)$ . Each training sample is formed by, firstly drawing  $M$  i.i.d. samples  $(x_i, y_i)$  from  $\mathcal{D}$  and then setting  $s = \sigma(y_1, \dots, y_M)$ . We use  $\mathcal{D}_{\mathbb{P}}$  to denote the distribution followed by the training samples  $(x, s)$ . In analogy to the zero-one classification loss  $\ell^{01}$ , we define the zero-one partial loss subject to  $\sigma$  as  $\ell_{\sigma}^{01}(\boldsymbol{y}, s) := \mathbb{1}\{\sigma(\boldsymbol{y}) \neq s\}$ , for any  $\boldsymbol{y} \in \mathcal{V}^M$  and  $s \in S$ . The learner aims to find the classifier  $f$  with the minimal zero-one risk  $\mathcal{R}^{01}(f)$ . As the gold labels are hidden, the learner uses the dataset  $\mathcal{T}_{\mathbb{P}}$  to estimate and minimize the zero-one partial risk of  $f$  subject to  $\sigma$  defined as  $\mathcal{R}_{\mathbb{P}}^{01}(f; \sigma) := \mathbb{E}_{(X_1, \dots, X_M, S) \sim \mathcal{D}_{\mathbb{P}}}[\ell_{\sigma}^{01}([f](X_1), \dots, [f](X_M)), S]$ .
We demonstrate these notions via Example 1. There,  $f$  is an MNIST classifier,  $\mathcal{X}$  is the space of MNIST images and  $\mathcal{Y} = \{0,\dots ,9\}$ . The training samples are of the form  $(x_{1},x_{2},s)$  where  $s = \sigma (y_1,y_2) = y_1 + y_2$ . The partial risk of  $f$  is then the probability of predicting the wrong sum.
Learnability. A partial learning algorithm  $\mathcal{A}$  takes a partially labeled dataset  $\mathcal{T}_{\mathsf{P}}$  with  $|\mathcal{T}_{\mathsf{P}}| = m_{\mathsf{P}}$  as input and outputs a function  $\mathcal{A}(\mathcal{T}_{\mathsf{P}})\in \mathcal{F}$ . Similar to standard PAC learning (see, for example, [40]), we say a multi-instance problem is learnable, if there exists a partial learning algorithm  $\mathcal{A}$ , such that for any data distribution  $\mathcal{D}$  over  $\mathcal{X}\times \mathcal{Y}$  and any  $\delta ,\epsilon \in (0,1)$ , there is an integer  $m_{\epsilon ,\delta}$ , such that  $m_{\mathsf{P}}\geq m_{\epsilon ,\delta}$  implies  $\mathcal{R}^{01}(\mathcal{A}(\mathcal{T}_{\mathsf{P}}))\leq \epsilon$  with probability at least  $1 - \delta$  with respect to  $\mathcal{D}_{\mathsf{P}}$ . We consider  $\mathcal{A}$  to be an Empirical Risk Minimizer (ERM) algorithm that finds the classifier  $f\in \mathcal{F}$  which minimizes the empirical zero-one partial risk  $\widehat{\mathcal{R}}_{\mathsf{P}}^{01}(f;\sigma ;\mathcal{T}_{\mathsf{P}}):= \sum_{(x,s)\in \mathcal{T}_{\mathsf{P}}}\ell_{\sigma}^{01}([[f](x_1),\ldots ,[f](x_M)),s) / m_{\mathsf{P}}$
As stated above, the risk is defined subject to a sampling distribution. For brevity, we will omit that distribution when defining the risk and fix the sampling distribution in each case.
3.1 ERM Learnability
To prove learnability under the partial zero-one loss, we aim to bound  $\mathcal{R}^{01}(f)$  with the partial risk  $\mathcal{R}_{\mathsf{P}}^{01}(f;\sigma)$ . Firstly, we propose a sufficient and necessary condition called  $M$ -unambiguity.
Recall that learnability requires handling all possible distributions of  $(\pmb{x},\pmb{y})$ . Therefore, to propose a necessary condition to guarantee learnability, we consider a special case: the one in which  $\mathcal{D}$  concentrates its mass on a single item  $x_0$ , i.e.,  $\mathbb{P}_{\mathcal{D}_X}(x_0) = 1$ . In that case, the only sample observed during learning includes only the instance  $x_0$ . Now, let  $l_0 \in \mathcal{V}$  be the gold label of  $x_0$ . If  $f$  mispredicts the class of  $x_0$ , i.e.,  $f(x_0) \neq l_0$ , and this misprediction leads to the same observed partial label, i.e.,  $\sigma(f(x_0),\ldots,f(x_0)) = \sigma(l_0,\ldots,l_0)$ , then the problem will be not learnable, as we will never be able to correct this error. Motivated by this special case, we propose the following condition:
Definition 1 (M-unambiguousity). Transition  $\sigma$  is  $M$ -unambiguous if for any two diagonal label vectors  $\mathbf{y}$  and  $\mathbf{y}' \in \mathcal{Y}^M$  such that  $\mathbf{y} \neq \mathbf{y}'$ , we have that  $\sigma(\mathbf{y}') \neq \sigma(\mathbf{y})$ .
Recall that a vector is diagonal if all its elements are equal. Below, we provide examples of transitions that satisfy (or not) the  $M$ -unambiguity condition.
Example 2. Consider the following learning problems.
- (SUM-M): Consider a variant of Example 1 where  $s$  is the sum of  $M$  digits, i.e.,  $\sigma(y_1, \ldots, y_M) = \sum_{i=1}^{M} y_i$ . The transition is  $M$ -unambiguous, since  $\sigma(y, \ldots, y) \neq \sigma(y', \ldots, y')$  holds for any  $y \neq y'$ .
- (PRODUCT-M): Consider a variant of Example 1 where  $s$  is given by  $\sigma^{*}(y_{1},\ldots ,y_{M}) = \prod_{i = 1}^{M}y_{i}$ . Then,  $\sigma^{*}$  is  $M$ -unambiguous, since  $\sigma^{*}(y,\dots ,y) = y^{M}\neq (y^{\prime})^{M}$  holds for any  $y\neq y^{\prime}$ .
- (XOR): Consider Boolean labels. We take the XOR of the two digits as the partial label, i.e.,  $\sigma^{XOR}(y_1, y_2) = \mathbb{1}\{y_1 \neq y_2\}$ . Then  $\sigma^{XOR}$  is not  $M$ -unambiguous, since  $\sigma^{XOR}(1, 1) = \sigma^{XOR}(0, 0)$ .
Although  $M$ -unambiguity is proposed as a necessary condition under special types of data distributions, we now show that it is also sufficient for proving ERM-learnability. Firstly, we prove that the classification risk can be bounded by the partial risk.
Lemma 1. If  $\sigma$  is  $M$ -unambiguous, then we have:
$$
\mathcal {R} ^ {0 1} (f) \leq \mathcal {O} \left(\mathcal {R} _ {\mathrm {P}} ^ {0 1} (f; \sigma) ^ {1 / M}\right) \quad a s \quad \mathcal {R} _ {\mathrm {P}} ^ {0 1} (f; \sigma) \rightarrow 0 \tag {1}
$$
Moreover, if  $\sigma$  is not  $M$ -unambiguous, then learning from partial labels is arbitrarily difficult, in the sense that a classifier  $f$  with partial risk  $\mathcal{R}_{\mathsf{P}}^{01}(f;\sigma) = 0$  can have a risk of  $\mathcal{R}^{01}(f) = 1$ .
To show learnability, we bound the partial risk with its empirical counterpart in the realizable case.
Theorem 1 (ERM learnability under  $M$ -unambiguity). Suppose  $\mathcal{F}$  is realizable under  $\ell_{\mathsf{P}}^{01}$  and  $[\mathcal{F}]$  has a finite Natarajan dimension  $d_{[\mathcal{F}]}$ . Then for any  $\epsilon, \delta \in (0,1)$ , there exists a universal constant  $C_0 > 0$ , such that with probability at least  $1 - \delta$ , the empirical partial risk minimizer with  $\widehat{\mathcal{R}}_{\mathsf{P}}^{01}(f; \sigma; \mathcal{T}_{\mathsf{P}}) = 0$  has a classification risk  $\mathcal{R}^{01}(f) < \epsilon$ , if
$$
m _ {\mathsf {P}} \geq C _ {0} \frac {c ^ {2 M - 2}}{\epsilon^ {M}} \left(d _ {[ \mathcal {F} ]} \log (6 c M d _ {[ \mathcal {F} ]}) \log \left(\frac {c ^ {2 M - 2}}{\epsilon^ {M}}\right) + \log \left(\frac {1}{\delta}\right)\right) \tag {2}
$$
Beyond Lemma 1, Theorem 1 builds upon several non-trivial intermediate results: (i) A bound of the VC dimension for the partial label predictor (Lemma 3 in the Appendix) and (ii) Construction of counter-examples for arguing the necessity of  $M$ -unambiguity (see the proof of Theorem 1).
Toward a faster convergence rate. Theorem 1 presents a rather slow convergence rate of  $(c^2 /\epsilon)^M$ . In the following, we show that a better convergence rate is achievable by forcing stricter ambiguity conditions. In the following, we introduce the concept of 1-unambiguity, which requires the transition to be sensitive to 1-position perturbations:
Definition 2 (1-unambiguity). Transition  $\sigma$  is 1-unambiguous, if there exists an index  $1\leq i\leq M$  such that flipping the  $i$ -th label of any label vector  $\pmb {y}\in \mathcal{V}^{M}$ , results in a vector  $\pmb {y}'$  with  $\sigma (\pmb {y}')\neq \sigma (\pmb {y})$
In the following, we provide examples of transitions that satisfy (or not) the 1-unambiguity condition.
${}^{3}$  A constant is universal if it that does not depend on the parameters of the learning problem (e.g.,  $M$  or  $c$  ).
Example 3. Let us continue with Example 2. The transition  $\sigma$  from SUM-M is 1-unambiguous since  $\sigma(\pmb{y}) = \sum_{i=1}^{M} y_i$  always change when replacing any of the labels  $y_i$ . However, the transition  $\sigma^*$  from PRODUCT-M is not 1-unambiguous, since for the label vector  $\pmb{y} = (0,1,1)$  (with product zero), the flipped vector  $\pmb{y}' = (0,1,2)$  also leads to product zero.
We show that a better convergence rate can be achieved if  $\sigma$  is both 1- and  $M$ -unambiguous.
Proposition 1 (ERM learnability under 1- and  $M$ -unambiguity). If  $\sigma$  is both 1- and  $M$ -unambiguous, then we have:
$$
\mathcal {R} ^ {0 1} (f) \leq \mathcal {O} \left(\mathcal {R} _ {\mathrm {P}} ^ {0 1} (f; \sigma)\right) \quad a s \quad \mathcal {R} _ {\mathrm {P}} ^ {0 1} (f; \sigma) \rightarrow 0 \tag {3}
$$
Furthermore, if  $\mathcal{F}$  is realizable under  $\ell_{\mathsf{P}}^{01}$  and  $[\mathcal{F}]$  has a finite Natarajan dimension  $d_{[\mathcal{F}]}$ , then for any  $\delta \in (0,1)$  and  $\epsilon \in (0,1)$  that is sufficiently close to 0, there exists a universal constant  $C_1$ , such that with probability at least  $1 - \delta$ , the empirical partial risk minimizer with  $\widehat{\mathcal{R}}_{\mathsf{P}}^{01}(f;\sigma) = 0$  has a classification risk  $\mathcal{R}^{01}(f) < \epsilon$ , if
$$
m _ {\mathbb {P}} \geq C _ {1} \frac {1}{\epsilon} \left(d _ {[ \mathcal {F} ]} \log (6 c M d _ {[ \mathcal {F} ]}) \log \left(\frac {2}{\epsilon}\right) + \log \left(\frac {1}{\delta}\right)\right) \tag {4}
$$
Remark 1. The supervision power of a partial label  $s$  depends on the "instability" of  $\sigma$ : if  $\sigma$  returns different results under certain perturbations in  $I$  positions for an  $I \in [M]$ , then the classification risk is bounded by  $\mathcal{R}^{01}(f) \leq \mathcal{O}(\mathcal{R}_{\mathsf{P}}^{01}(f; \sigma)^{1 / I})$ . To formalize this intuition, in the appendix, we provide a generalized definition of both 1- and  $M$ -unambiguity, called  $I$ -unambiguity, and show learnability with intermediate convergence rates. Notice that  $M$ -unambiguity is closely related to the small ambiguity degree condition in the PLL literature [28]. Appendix E.3 provides a relevant discussion and an extension of our results for randomized transitions.
Remark 2. We use the concentrated distribution described at the beginning of Section 3.1 as a pivot to design a necessary and sufficient learnability condition. However, the results in Lemma 1, Theorem 1 and Proposition 1 do not apply only to this distribution, but to any distribution  $\mathcal{D}_{\mathrm{P}}$  of partial data. The same applies to all learnability results that follow.
3.2 Error Bounds with the Top- $k$  Semantic Loss
We now study the error bounds with the semantic loss (see Section 2 and Appendix A) under top- $k$  approximations (Theorem 2). We are interested in this loss for two reasons. Firstly, risk minimization under the zero-one loss is intractable even for linear classifiers [17] and hence in practice, learning is performed by minimizing differentiable surrogate losses such as the semantic loss. Secondly, computing a surrogate loss typically introduces another source of intractability: that of enumerating the entries of  $\sigma$  [24, 43, 31]. For example, in SUM- $M$ , we have to enumerate  $10^{M}$  digit combinations to populate  $\sigma$ , which is not scalable.
A popular way to reduce the second source of inefficiency is to consider only the  $k$  most probable label combinations during training, where the probability of a label vector  $\pmb{y}$  given  $\pmb{x}$  and classifier  $f$  is defined by  $P_{f(\pmb{x})}(\pmb{y}) \coloneqq \prod_{i=1}^{M} f^{y_i}(x_i)$ . Then, training can proceed by taking the semantic loss over the top- $k$  label vectors [24]. The above is possible, as the top- $k$  label vectors can be equivalently viewed as a formula that is true iff one or more of the top- $k$  label vectors is the gold one.