SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Vinay Shukla
Department of Computer Science  
University of California, Los Angeles  
vshukla@g.ucla.edu
Kareem Ahmed*
Zhe Zeng*
Department of Computer Science  
University of California, Los Angeles  
zhezeng@cs.ucla.edu
High-quality labels are often very scarce, whereas unlabeled data with inferred weak labels occurs more naturally. In many cases, these weak labels dictate the frequency of each respective class over a set of instances. In this paper, we develop a unified approach to learning from such weakly-labeled data, which we call count-based weakly-supervised learning. At the heart of our approach is the ability to compute the probability of exactly  $k$  out of  $n$  outputs being set to true. This computation is differentiable, exact, and efficient. Building upon the previous computation, we derive a count loss penalizing the model for deviations in its distribution from an arithmetic constraint defined over label counts. We evaluate our approach on three common weakly-supervised learning paradigms and observe that our proposed approach achieves state-of-the-art or highly competitive results across all three of the paradigms.
Weakly supervised learning [56] enables a model to learn from data with restricted, partial or inaccurate labels, often known as weakly-labeled data. Weakly supervised learning fulfills a need arising in many real-world settings that are subject to privacy or budget constraints, such as privacy sensitive data [45], medical image analysis [12], clinical practice [39], personalized advertisement [9] and knowledge base completion [21, 59], to name a few. In some settings, instance-level labels are unavailable. Instead, instances are grouped into bags with corresponding bag-level labels that are a function of the instance labels, e.g., the proportion of positive labels in a bag. A key insight that we bring forth is that such weak supervision can very often be construed as enforcing constraints on label counts of data.
More concretely, we consider three prominent weakly supervised learning paradigms. The first paradigm is known as learning from label proportions [38]. Here the weak supervision consists in the proportion of positive labels in a given bag, which can be interpreted as the count of positive instances in such a bag. The second paradigm, whose supervision is strictly weaker than the former, is multiple instance learning [35, 17]. Here the bag labels only indicate the existence of at least one positive instance in a bag, which can be recast as to whether the count of positive instances is greater than zero. The third paradigm, learning from positive and unlabeled data [16, 31], grants access to
Table 1: A comparison of the tasks considered in the three weakly supervised settings, LLP (cf. Section 2.1), MIL (cf. Section 2.2) and PU learning (cf. Section 2.3), against the classical fully supervised setting for binary classification, using digits from the MNIST dataset.
the ground truth labels for a subset of only the positive instances, providing only a class prior for what remains. We can recast the class prior as a distribution of the count of positive labels.
Leveraging the view of weak supervision as a constraint on label counts, we utilize a simple, efficient and probabilistically sound approach to weakly-supervised learning. More precisely, we train a neural network to make instance-level predictions that conform to the desired label counts. To this end, we propose a differentiable count loss that characterizes how close the network's distribution comes to the label counts; a loss which is surprisingly tractable. Compared to prior methods, this approach does not approximate probabilities but computes them exactly. Our empirical evaluation demonstrates that our proposed count loss significantly boosts the classification performance on all three aforementioned settings.
2 Problem Formulations
In this section, we formally introduce the aforementioned weakly supervised learning paradigms. For notation, let  $\mathcal{X} \in \mathbb{R}^d$  be the input feature space over  $d$  features and  $\mathcal{Y} = \{0,1\}$  be a binary label space. We write  $\pmb{x} \in \mathcal{X}$  and  $y \in \mathcal{Y}$  for the input and output random variables respectively. Recall that in fully-supervised binary classification, it is assumed that each feature and label pair  $(x,y) \in \mathcal{X} \times \mathcal{Y}$  is sampled independently from a joint distribution  $p(x,y)$ . A classifier  $f$  is learned to minimize the risk  $R(f) = \mathbb{E}_{(x,y) \sim p}[\ell(f(x),y)]$  where  $\ell : [0,1] \times \mathcal{Y} \to \mathbb{R}_{\geq 0}$  is the cross entropy loss function. Typically, the true distribution  $p(x,y)$  is implicit and cannot be observed. Therefore, a set of  $n$  training samples,  $\mathcal{D} = \{(x_i,y_i)\}_{i=1}^n$ , is used and the empirical risk,  $\hat{R}(f) = \frac{1}{n} \sum_{i=1}^{n} \ell(f(x_i),y_i)$ , is minimized in practice. In the count-based weakly supervised learning settings, the supervision is given at a bag level instead of an instance level. We formally introduce these settings as below.
2.1 Learning from Label Proportions
Learning from label proportions (LLP) [38] assumes that each instance in the training set is assigned to bags and only the proportion of positive instances in each bag is known. One example is in light of the coronavirus pandemic, where infection rates were typically reported based on geographical boundaries such as states and counties. Each boundary can be treated as a bag with the infection rate as the proportion annotation.
The goal of LLP is to learn an instance-level classifier  $f: \mathcal{X} \to [0,1]$  even though it is trained on bag-level labeled data. Formally, the training dataset consists of  $m$  bags, denoted by  $\mathcal{D} = \{(B_i, \tilde{y}_i)\}_{i=1}^m$  where each bag  $B_i = \{x_j\}_{j=1}^k$  consists of  $k$  instances and this  $k$  could vary among different bags. The bag proportions are defined as  $\tilde{y}_i = \sum_{j=1}^k y_j / k$  with  $y_j$  being the instance label that cannot be accessed and only  $\tilde{y}_i$  is available during training. An example is shown in Figure 1b. We do not assume that the bags are non-overlapping while some existing work suffers from this limitation including Scott and Zhang [40].
Table 2: A summary of the labels and objective functions for all the settings considered in the paper.
TASK	LABEL	LABEL LEVEL	OBJECTIVE
Classical Fully Supervised Learning from Label Proportion	Binary y	Instance Level	- y log p(y) - (1 - y) log(1 - p(y))
Continuous ~y = ∑i yi/k	Bag Level	- log p(∑~yi=k~y)
Multiple Instance Learning Learning from Positive and Unlabeled Data	Binary ~y = max{yi}	Bag Level	- ~y log p(∑~yi ≥ 1) - (1 - ~y) log p(∑i~yi = 0)
Binary ~y	Instance Level	1) DKL(Bin(k,β) || p(∑i~yi))
2) - log p(∑~yi=k~β)
2.2 Multiple Instance Learning
Multiple instance learning (MIL) [35, 17] refers to the scenario where the training dataset consists of bags of instances, and labels are provided at bag level. However, in MIL, the bag label is a single binary label indicating whether there is a positive instance in the bag or not as opposed to a bag proportion defined in LLP. A real-world application of MIL lies in the field of drug activity [17]. We can observe the effects of a group of conformations but not for any specific molecule, motivating a MIL setting. Formally, in MIL, the training dataset consists of  $m$  bags, denoted by  $\mathcal{D} = \{(B_i,\tilde{y}_i)\}_{i = 1}^m$ , with a bag consisting of  $k$  instances, i.e.,  $B_{i} = \{\pmb{x}_{j}\}_{j = 1}^{k}$ . The size  $k$  can vary among different bags. For each instance  $x_{j}$ , there exists an instance-level label  $y_{j}$  which is not accessible. The bag-level label is defined as  $\tilde{y}_i = \max_j\{y_j\}$ . An example is shown in Figure 1c.
The main goal of MIL is to learn a model that predicts a bag label while a more challenging goal is to learn an instance-level predictor that is able to discover positive instances in a bag. In this work, we aim to tackle both by training an instance-level classifier whose predictions can be combined into a bag-level prediction as the last step.
2.3 Learning from Positive and Unlabeled Data
Learning from positive and unlabeled data or PU learning [16, 31] refers to the setting where the training dataset consists of only positive instances and unlabeled data, and the unlabeled data can contain both positive and negative instances. A motivation of PU learning is persistence in the case of shifts to the negative-class distribution [37], for example, a spam filter. An attacker may alter the properties of a spam email, making a traditional classifier require a new negative dataset [37]. We note that taking a new unlabeled sample would be more efficient, motivating PU learning. Formally, in PU learning, the training dataset  $\mathcal{D} = \mathcal{D}_p \cup \mathcal{D}_u$  where  $\mathcal{D}_p = \{(x_i, \tilde{y}_i = 1)\}_{i=1}^{n_p}$  is the set of positive instances with  $x_i$  from  $p(x \mid y = 1)$  and  $\tilde{y}$  denoting whether the instance is labeled, and  $\mathcal{D}_u = \{(x_i, \tilde{y}_i = 0)\}_{i=1}^{n_u}$  the unlabeled set with  $x_i$  from
$$
p _ {u} (\boldsymbol {x}) = \beta p (\boldsymbol {x} \mid y = 1) + (1 - \beta) p (\boldsymbol {x} \mid y = 0), \tag {1}
$$
where the mixture proportion  $\beta \coloneqq p(y = 1\mid \tilde{y} = 0)$  is the fraction of positive instances among the unlabeled population. Although the instance label  $y$  is not accessible, its information can be inferred from the binary selection label  $\tilde{y}$ : if the selection label  $\tilde{y} = 1$ , it belongs to the positively labeled set, i.e.,  $p(y = 1\mid \tilde{y} = 1) = 1$ ; otherwise, the instance  $\mathbf{x}$  can be either positive or negative. An example of such a dataset is shown in Figure 1d.
The goal of PU learning is to train an instance-level classifier. However, it is not straightforward to learn from PU data and it is necessary to make assumptions to enable learning with positive and unlabeled data [9]. In this work, we make a commonly-used assumption for PU learning, selected completely at random (SCAR), which lies at the basis of many PU learning methods.
Definition 2.1 (SCAR). Labeled instances are selected completely at random, independent from input features, from the positive distribution  $p(\pmb{x} \mid y = 1)$ , that is,  $p(\tilde{y} = 1 \mid \pmb{x}, y = 1) = p(\tilde{y} = 1 \mid y = 1)$ .
3 A Unified Approach: Count Loss
In this section, we derive objectives for the three weakly supervised settings, LLP, MIL, and PU learning, from first principles. Our proposed objectives bridge between neural outputs, which can be observed as counts, and arithmetic constraints derived from the weakly supervised labels. The idea is to capture how close the classifier is to satisfying the arithmetic constraints on its outputs.
Algorithm 1 Count Probability  $p\left( {\mathop{\sum }\limits_{{i = 1}}^{k}{\widehat{y}}_{i} = s}\right)$
Input: A set of  $k$  log probabilities  $\{t_i\}_{i=1}^k$  with  $t_i := \log p(\hat{y}_i = 1)$ , the number of instances  $k$ , and a label sum  $s$
Output: log probabilities  $\log p\left(\sum_{i=1}^{k} \hat{y}_i = s\right)$  or a set of log probability  $\{\log p\left(\sum_{i=1}^{k} \hat{y}_i = s\right)\}_{s=0}^{k}$
$$
/ / A [ i, m ] = \log p \left(\sum_ {j = 1} ^ {i} y _ {j} = m\right) \forall i, m
$$
Initialize an array  $A$  to be -Inf everywhere
$$
A [ 0, 0 ] = 0 \quad / / p (\sum_ {j = 1} ^ {0} y _ {j} = 0) = 1
$$
Compute  $t_i^\prime \gets \log 1\mathrm{mexp}(t_i) / \log p(y_i = 0)$
for  $i = 1$  to  $k$  do
for  $m = 0$  to  $s$  do
$$
a _ {+} = A [ i - 1, m - 1 ] + t _ {i}
$$
$$
a _ {-} = A [ i - 1, m ] + t _ {i} ^ {\prime}
$$
$$
A [ i, m ] = \operatorname {l o g s u m e x p} \left(a _ {+}, a _ {-}\right)
$$
return  $A[k, s]$  or  $A[\vec{k}, :]$
They can be easily integrated with deep learning models, and allow them to be trained end-to-end. For the three objectives, we show that they share the same computational building block: given  $k$  instances  $\{\pmb{x}_i\}_{i=1}^k$  and an instance-level classifier  $f$  that predicts  $p(\hat{y}_i \mid \pmb{x}_i)$  with  $\hat{y}$  denoting the prediction variable, the problem of inferring the probability of the constraint on counts  $\sum_{i=1}^k \hat{y}_i = s$  is to compute the count probability defined below:
$$
p \left(\sum_ {i = 1} ^ {k} \hat {y} _ {i} = s \mid \left\{\boldsymbol {x} _ {i} \right\} _ {i = 1} ^ {k}\right) := \sum_ {\hat {\boldsymbol {y}} \in \mathcal {Y} ^ {k}} \llbracket \sum_ {i = 1} ^ {k} \hat {y} _ {i} = s \rrbracket \prod_ {i = 1} ^ {k} p \left(\hat {y} _ {i} \mid \boldsymbol {x} _ {i}\right)
$$
where  $\llbracket \cdot \rrbracket$  denotes the indicator function and  $\hat{\pmb{y}}$  denotes the vector  $(\hat{y}_1,\dots ,\hat{y}_k)$ . For succinctness, we omit the dependency on the input and simply write the count probability as  $p(\sum_{i = 1}^{k}\hat{y}_i = s)$ . Next, we show how the objectives derived from first principles can be solved by using the count probability as an oracle. We summarize all proposed objectives in Table 2. Later, we will show how this seemingly intractable count probability can be efficiently computed by our proposed algorithm.
LLP setting. Given a bag  $B = \{x_{i}\}_{i=1}^{k}$  of size  $k$  and its weakly supervised label  $\tilde{y}$ , by definition, it can be inferred that the number of positive instances (count) in the bag is  $k\tilde{y}$ . Our objective is to minimize the negative log probability  $-\log p(\sum_{i} \hat{y}_{i} = k\tilde{y})$ . Notice that when each bag consists of only one instance, that is, when the bag-level supervisions are reduced to instance-level ones, this objective is exactly cross-entropy loss. We further show that our method is risk-consistent, that is, the optimal classifier under our proposed loss provides predictions consistent with the underlying risk as in the supervised learning setting. Details of the risk analysis can be found in Appendix A.
MIL setting. Given a bag  $B = \{x_{i}\}_{i=1}^{k}$  of size  $k$  and a single binary label  $\tilde{y}$  as its weakly supervised label, we propose a cross-entropy loss as below
$$
\ell (B, \tilde {y}) = - \tilde {y} \log p (\sum \hat {y} _ {i} \geq 1) - (1 - \tilde {y}) \log p (\sum \hat {y} _ {i} = 0).
$$
Notice that in the above loss, the probability term  $p(\sum \hat{y}_i = 0)$  is accessible to the oracle for computing count probability, and the other probability term  $p(\sum \hat{y}_i \geq 1)$  can simply be obtained from  $1 - p(\sum \hat{y}_i = 0)$ , i.e., the same call to the oracle since all prediction variables  $\hat{y}_i$  are binary.
PU Learning setting. Recall that for the unlabeled data  $\mathcal{D}_u$  in the training dataset, an unlabeled instance  $x_{i}$  is drawn from a mixture distribution as shown in Equation 1 parameterized by a mixture proportion  $\beta = p(y = 1\mid \tilde{y} = 0)$ . Under the SCAR assumption, even though only a class prior is given, we show that the mixture proportion can be estimated from the dataset.
Proposition 3.1. With SCAR assumption and a class prior  $\alpha \coloneqq p(y = 1)$ , the mixture proportion  $\beta \coloneqq p(y = 1 \mid \tilde{y} = 0)$  can be estimated from dataset  $\mathcal{D}$ .
Proof. First, the label frequency  $p(\tilde{y} = 1 \mid y = 1)$  denoted by  $c$  can be obtained by
$$
c = \frac {p (\tilde {y} = 1 , y = 1)}{p (y = 1)} = \frac {p (\tilde {y} = 1)}{p (y = 1)} (b y t h e d e f i n i t i o n o f P U l e a r n i n g).
$$
Figure 1: An example of how to compute the count probability in a dynamic programming manner. Assume that an instance-level classifier predicts three instances to have  $p(y_{1} = 1) = 0.1$ ,  $p(y_{2} = 1) = 0.2$ , and  $p(y_{3} = 1) = 0.3$  respectively. The algorithm starts from the top-left cell and propagates the results down right. A cell has its probability  $p(\sum_{j=0}^{i} y_{j} = s)$  computed by inputs from  $p(\sum_{j=0}^{i-1} y_{j} = s)$  weighted by  $p(y_{i} = 0)$ , and  $p(\sum_{j=0}^{i-1} y_{j} = s-1)$  weighted by  $p(y_{i} = 1)$  respectively, as indicated by the arrows.
that is,  $c = p(\tilde{y} = 1) / \alpha$ . Notice that  $p(\tilde{y} = 1)$  can be estimated from the dataset  $\mathcal{D}$  by counting the proportion of the labeled instances. Thus, we can estimate the mixture proportion as below,
$$
\beta = \frac {p (\tilde {y} = 0 \mid y = 1) p (y = 1)}{p (\tilde {y} = 0)} = \frac {(1 - p (\tilde {y} = 1 \mid y = 1)) p (y = 1)}{1 - p (\tilde {y} = 1)} = \frac {(1 - c) \alpha}{1 - \alpha c}.
$$
The probabilistic semantic of the mixture proportion is that if we randomly draw an instance  $\pmb{x}_i$  from the unlabeled population, the probability that the true label  $y_i$  is positive would be  $\beta$ . Further, if we randomly draw  $k$  instances, the distribution of the summation of the true labels  $\sum_{i=1}^{k} y_i$  conforms to a binomial distribution  $\mathrm{Bin}(k, \beta)$  parameterized by the mixture proportion  $\beta$ , i.e.,
$$
p \left(\sum_ {i = 1} ^ {k} y _ {i} = s\right) = \binom {k} {s} \beta^ {s} (1 - \beta) ^ {k - s}. \tag {2}
$$
Based on this observation, we propose an objective to minimize the KL divergence between the distribution of predicted label sum and the binomial distribution parameterized by the mixture proportion for a random subset drawn from the unlabeled population, that is,
$$
\mathbb {D} _ {K L} \left(\operatorname {B i n} (k, \beta) \| p \left(\sum_ {i = 1} ^ {k} \hat {y} _ {i}\right)\right) = \sum_ {s = 0} ^ {k} \operatorname {B i n} (s; k, \beta) \log \frac {\operatorname {B i n} (s ; k , \beta)}{p \left(\sum_ {i = 1} ^ {k} \hat {y} _ {i} = s\right)}
$$
where  $\operatorname{Bin}(s; k, \beta)$  denotes the probability mass function of the binomial distribution  $\operatorname{Bin}(k, \beta)$ . Again, the KL divergence can be obtained by  $k + 1$  calls to the oracle for computing count probability  $p\left(\sum_{i=1}^{k} \hat{y}_i = s\right)$ . The KL divergence is further combined with a cross entropy defined over labeled data  $\mathcal{D}_p$  as in the classical binary classification training as the overall objective.
As an alternative, we propose an objective for the unlabeled data that requires fewer calls to the oracle: instead of matching the distribution of the predicted label sum with the binomial distribution, this objective matches only the expectations of the two distributions, that is, to maximize  $p\left(\sum_{i=1}^{k} \hat{y}_i = k\beta\right)$  where  $k\beta$  is the expectation of the binomial distribution  $\mathrm{Bin}(k,\beta)$ . We present empirical evaluations of both proposed objectives in the experimental section.
4 Tractable Computation of Count Probability
In the previous section, we show how the count probability  $p\left(\sum_{i=1}^{k} \hat{y}_i = s\right)$  serves as a computational building block for the objectives derived from first principles for the three weakly supervised learning settings. With a closer look at the count probability, we can see that given a set of instances, the classifier predicts an instance-level probability for each and it requires further manipulation to obtain
count information; actually, the number of joint labelings for the set can be exponential in the number of instances. Intractable as it seems, we show that it is indeed possible to derive a tractable computation for the count probability based on a result from Ahmed et al. [6].
Proposition 4.1. The count probability  $p\left( {\mathop{\sum }\limits_{{i = 1}}^{k}{\widehat{y}}_{i} = s}\right)$  of sampling  $k$  prediction variables that sums to  $s$  from an unconstrained distribution  $p\left( \mathbf{y}\right)  = {\prod }_{i = 1}^{k}p\left( {\widehat{y}}_{i}\right)$  can be computed exactly in time  $\mathcal{O}\left( {ks}\right)$  . Moreover,the set  $\left\{  {p\left( {\mathop{\sum }\limits_{{i = 1}}^{k}{\widehat{y}}_{i} = s}\right) }\right\}  {}_{s = 0}^{k}$  can also be computed in time  $\mathcal{O}\left( {k}^{2}\right)$  .
The above proposition can be proved in a constructive way where we show that the count probability  $p(\sum_{i=1}^{k} \hat{y}_i = s)$  can be computed in a dynamic programming manner. We provide an illustrative example of this computation in Figure 1. In practice, we implement this computation in log space for numeric stability which we summarized as Algorithm 1, where function  $\log_1 \mathsf{exp}$  provides a numerically stable way to compute  $\log_1 \mathsf{exp}(x) = \log(1 - \exp(x))$  and function  $\log_2 \mathsf{exp}$  a numerically stable way to compute  $\log_2 \mathsf{exp}(x, y) = \log(\exp(x) + \exp(y))$ . Notice that since we show it is tractable to compute the set  $\{p(\sum_{i=1}^{k} \hat{y}_i = s)\}_{s=0}^k$ , for any two given label sum  $s_1$  and  $s_2$ , a count probability  $p(s_1 \leq \sum_i \hat{y}_i \leq s_2)$  where the count lies in an interval, can also be exactly and tractably computed. This implies that our tractable computation of count probabilities can potentially be leveraged by other count-based applications besides the three weakly supervised learning settings in the last section.
Weakly Supervised Learning. Besides settings explored in our work there are many other weakly-supervised settings. One of which is semi-supervised learning, a close relative to PU Learning with the difference being that labeled samples can be both positive and negative [57, 58]. Another is label noise learning, which occurs when our instances are mislabeled. Two common variations involve whether noise is independent or dependent on the instance [20, 42]. A third setting is partial label learning, where each instance is provided a set of labels of which exactly one is true [14]. An extension of this is partial multi-label learning, where among a set of labels, a subset is true [46].
Unified Approaches. There exists some literature in regards to "general" approaches for weakly supervised learning. One example being the method proposed in Hüllermeyer [23], which provides a procedure that minimizes the empirical risk on "fuzzy" sets of data. The paper also establishes guarantees for model identification and instance-level recognition. Co-Training and Self-Training are also examples of similar techniques that are applicable to a wide variety of weakly supervised settings [11, 49]. Self-training involves progressively incorporating more unlabeled data via our model's prediction (with pseudo-label) and then training a model on more data as an iterative algorithm [25]. Co-Training leverages two models that have different "views" of the data and iteratively augment each other's training set with samples they deem as "well-classified". They are traditionally applied to semi-supervised learning but can extend to multiple instance learning settings [33, 47, 32].
LLP. Quadrianto et al. [38] first introduced an exponential family based approach that used an estimation of mean for each class. Others seek to minimize "empirical proportion risk" or EPR as in Yu et al. [50], which is centered around creating an instance-level classifier that is able to reproduce the label proportions of each bag. As mentioned previously, more recent methods use bag posterior approximation and neural-based approaches [8, 43]. One such method is Proportion Loss (PL) [43], which we contrast to our approach. This is computed by binary cross entropy between the averaged instance-level probabilities and ground-truth bag proportion.
MIL. MIL finds its earlier approaches with SVMs, which have been used quite prolifically and still remain one of the most common baselines. We start with MI-SVM/mi-SVM [7] which are examples of transductive SVMs [13] that seek a stable instance classification through repeated retraining iterations. MI-SVM is an example of an instance space method [13], which identifies methods that classify instances as a preliminary step in the problem. This is in contrast to bag-space or embedded-space methods that omit the instance classification step. Furthermore, Wang et al. [44] remains one of the hallmarks of the use of neural networks for Multi-Instance Learning. Ilse et al. [24], utilize a similar approach but with attention-based mechanisms.