SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Robert Busa-Fekete
Google Research sarobi@gmail.com
Heejin Choi*
Coupang Inc
hechoi53@coupang.com
Travis Dick
Google Research  
tdick@google.com
Google Research  
cangentilek@google.com
Andres Munoz Medina
Google Research ammmedina@google.com
We consider the problem of Learning from Label Proportions (LLP), a weakly supervised classification setup where instances are grouped into i.i.d. "bags", and only the frequency of class labels at each bag is available. Albeit, the objective of the learner is to achieve low task loss at an individual instance level. Here we propose EASYLLP, a flexible and simple-to-implement debiasing approach based on aggregate labels, which operates on arbitrary loss functions. Our technique allows us to accurately estimate the expected loss of an arbitrary model at an individual level. We elucidate the differences between our method and standard methods based on label proportion matching, in terms of applicability and optimality conditions. We showcase the flexibility of our approach compared to alternatives by applying our method to popular learning frameworks, like Empirical Risk Minimization (ERM) and Stochastic Gradient Descent (SGD) with provable guarantees on instance level performance. Finally, we validate our theoretical results on multiple datasets, empirically illustrating the conditions under which our algorithm is expected to perform better or worse than previous LLP approaches.
In traditional supervised learning problems, a learner has access to a sample of labeled examples. This collection of labeled examples is used to fit a model – to name a few, decision trees, neural networks, random forests – by minimizing a loss over the observed sample. By contrast, in the problem of Learning from Label Proportions (LLP), the learner only observes collections of unlabeled feature vectors called bags, together with the proportion of positive examples in each bag. The LLP problem is motivated by a number of applications where access to individual examples is too expensive or impossible to achieve, or available at aggregate level for privacy-preserving reasons. Examples include e-commerce, fraud detection, medical databases [20], high energy physics [6], election prediction [30], medical image analysis [4], remote sensing [7].
As a weakly supervised learning paradigm, LLP traces back to at least [5, 19, 22, 23, 32], and was motivated there by learning scenarios where access to individual examples is often not available. A paradigmatic example is perhaps a political campaign trying to predict the preference of the electorate. Since voting is anonymous, a political analyst may not be able to observe individual votes, yet, they have access to aggregate voting preferences at the district level.
The problem has received renewed interest more recently (e.g., [8, 17, 26, 24, 25, 31, 16, 34]), driven by the desire to provide more privacy to user information. For instance the ad conversion reporting
*Work done while at Google.
system proposed by Apple, SKAN, allows an ad tech provider to receive information about the conversions (e.g., purchases) from customers only aggregated across multiple impressions. This aggregation is intended to obfuscate the individual customer's activity. A similar API has also been proposed by Google Chrome and Android to report aggregate conversion information (e.g., [1]). Given the importance of conversion modeling for online advertising, learning how to train a model using only these aggregates has become a crucial task for many data-intensive online businesses.
Research in LLP can be coarsely divided into two types of goals: Learning a bag classifier that correctly predicts label proportions, and learning an individual classifier that can correctly predict instance labels. The former has been the focus of most of the literature in this area. Representative papers include [5, 19, 32, 33]. While training classifiers to match label proportions is an obvious heuristic, little work has been done in trying to understand under which conditions these classifiers would be accurate when generating event level predictions. Yu et al. [33] provide guarantees that suggest that, under the absolute loss, predicting label predictions to very high accuracy can result in good predictions at event level. On the other side of this research area, finding a good instance level predictor using only label proportions has so far remained elusive and under-explored in its generality. The solutions introduced so far require either making some assumptions on the data generation process [26, 34] or on the model class [22]. Other solutions involve solving complex combinatorial problems [8] or require that an example belongs to multiple bags [24, 25].
In this work we provide a two pronged approach towards making LLP on randomly generated bags seamless. First, we contribute to a better theoretical understanding of a well-known label proportion prediction algorithm. This algorithm simply trains a model whose average prediction on a bag is as close as possible to the label proportion in that bag. This straightforward algorithm is referenced (either explicitly or implicitly) in multiple past works (e.g., [19, 23, 2]), yet is hardly analyzed theoretically, or even considered as a baseline for experimental comparisons. Here, we show under what conditions this folklore algorithm can produce good event level predictions. Moreover, we show through extensive experimentation that when those conditions are met this algorithm seems to outperform previous baselines tailored to generating event level predictions. On the flip side, when such conditions are not met, we also show that the quality of this algorithm quickly deteriorates as the bag size increases.
A more robust and flexible approach to LLP that operates at the instance level is EASYLLP, a reduction method virtually applicable to any machine learning task. Unlike many algorithmic proposals in this space, the implementation of EASYLLP requires trivial modifications to current machine learning training pipelines. We elucidate the flexibility of our approach by applying it to two widely interesting algorithmic techniques, Empirical Risk Minimization (ERM) and Stochastic Gradient Descent (SGD). Our findings are complemented by an extensive experimental investigation on a diverse suite of benchmark datasets.
Main contributions. The contribution of our paper can be summarized as follows.
1. We provide a theoretical analysis of a popular label proportion matching algorithm, that is suggestive of the conditions under which this algorithm is expected to work in practical scenarios. Despite its relative simplicity, the scope of this analysis is broad and, to the best of our knowledge, original.
2. We provide a general debiasing technique for estimating the expected instance loss (or loss gradient) of an arbitrary model using only label proportions, and thoroughly quantify the variance of this estimator.
3. We provide a reduction of ERM with label proportions to ERM with individual labels, and show that when the learner observes bags of size  $k$ , the sample complexity of ERM with label proportions increases only by a factor of  $k$ . Likewise, we provide an analysis of SGD using only label proportions and show that for bags of size  $k$ , its regret increases by only a factor of  $k$ .
4. We carry out an extensive set of experiments comparing the proportion matching method and EASYLLP to known methods available in the LLP literature. The experiments are designed to encompass diverse combinations of datasets and learning models. We identify general trends in the relative performance of the tested methods when evaluated at the instance level. To the best of our knowledge, in spite of its folklore status, this is the first thorough investigation that involves the proportion matching algorithm and its comparison to event
level classifiers. We believe the latter is by itself an important contribution on its own, as proportion matching turns out to be a strong baseline in a number of cases.
Related work. Interest in LLP traces back to at least [5, 19, 22, 23, 29, 32, 20]. The literature in recent years has become quite voluminous, so we can hardly do justice of it here. In what follows, we comment on and contrast to the references from which we learned about LLP problems.
In [5] the authors consider a hierarchical model that generates labels according to the given label proportions and proposed an MCMC-based inference scheme which, however, is not scalable to large training sets. [19] shows how standard supervised learning algorithms (like SVM and  $k$ -Nearest Neighbors) can be adapted to LLP by a reformulation of their objective function via label proportion matching. Yet, no experiments are reported on classification tasks. In [22], the authors propose a theoretically-grounded way of estimating the mean of each class through the sample average of each bag, along with the associated label proportion. The authors make similar assumptions to ours, in that the class-conditional distribution of data is independent of the bags. However, their estimators rely on very strong assumptions, like conditional exponential models, which are not a good fit to nowadays Deep Neural Network (DNN) models. Similar limitations are contained in [20]. [23] proposes an adaptation of SVM to the LLP setting through a scheme which can be seen as calibration on top of proportion matching. Yet, this turns out to be restricted to linear models in some feature space. Similar limitations are in the  $\alpha$ -SVM method proposed in [32], the non-parallel SVM formulation of [21], and the pinball loss SVM in [28]. The original  $\alpha$ -SVM formulation was extended to other classifiers; e.g., [15] extends the formulation to CNNs with a generative model whose associated maximum likelihood estimator is computed via Expectation Maximization, which turns out not to be scalable to sizeable DNN architectures. [29] proposes a method based on  $k$ -means to identify a clustering of the data which is compatible with the label proportions, but their method suffers from an extremely high computational complexity.
Many of these papers are in fact purely experimental in nature, and their main goal is to adapt the standard supervised learning formulation to an LLP formulation so as to obtain a bag level predictor.
On the learning theory side, besides the already mentioned [22, 20], are the efforts contained in [24, 25], the task of learning from multiple unlabeled datasets considered in [17, 16], and the statistical learning agenda pursued in [26, 34] (and references therein from the same authors). In [24, 25] the author is essentially restricting to linear-threshold functions and heavily relies on the fact that an example can be part of multiple bags, while we are working with non-overlapping i.i.d. bags and general model classes. In [17, 16] the authors consider a problem akin to LLP. Similar to our paper, the authors propose a debiasing procedure via linear transformations. Yet, the way they solve the debiasing problem forces them to impose further restrictions on the bags, like the separation of the class prior distributions across different bags. It is this diversity that allows the authors to construct unbiased estimates and then derive consistency results. On the contrary, the bags proposed in our setup are drawn i.i.d. from the same prior distribution, a scenario where many of these algorithms would fail. Hence, we work under the assumption that we cannot handcraft diverse bags out of our samples, as the aggregation into bags is done without having access to the class conditional distributions (which [17, 16] and related papers heavily rely upon). Moreover, the convergence results to the event level performance are only proven in those papers with specific families of loss function (e.g., proper loss functions). See Section 5 for further in-context discussion.
The work [26] introduced a principled approach to LLP based on a reduction to learning with label noise. As in [17], their basic strategy is to pair bags, and view each pair as a task of learning with label noise, where label proportions are related to label flipping probabilities. The authors also established generalization error guarantees at the event level, as we do here. [34] extend their results to LLP for multiclass classification. From a technical standpoint, these two papers have similar limitations as [17, 16]. Besides, the risk measure they focus on is balanced risk rather than classification risk, as we do here.
In our experiments (Section 7), we empirically compare EASYLLP to a folklore label proportion matching method, to the MeanMap method from [22], and to a label generation approach from [8], the latter viewed as representative of recent applications of DNNs to LLP.
2 Setup and Notation
Let  $\mathcal{X}$  denote a feature (or instance) space and  $\mathcal{Y} = \{0,1\}$  be a binary $^2$  label space. We assume the existence of a joint distribution  $\mathcal{D}$  on  $\mathcal{X}\times \mathcal{Y}$ , and let  $p = \mathbb{P}_{(x,y)\sim \mathcal{D}}(y = 1)$  denote the probability of drawing a sample  $(x,y)\in \mathcal{X}\times \mathcal{Y}$  from  $\mathcal{D}$  with label  $y = 1$ . For a natural number  $n$ , let  $[n] = \{i\in \mathbb{N}: i\leq n\}$ .
A labeled bag of size  $k$  is a sample  $\mathcal{B} = \{x_1, \ldots, x_k\}$ , together with the associated label proportion  $\alpha(\mathcal{B}) = \frac{1}{k} \sum_{j=1}^{k} y_j$ , where  $(x_1, y_1), \ldots, (x_k, y_k)$  are drawn i.i.d. according to  $\mathcal{D}$ . We assume the learner has access to a collection  $S = \{(\mathcal{B}_i, \alpha_i), i \in [n]\}$  of  $n$  labeled bags of size  $k$ , where  $\mathcal{B}_i = \{x_{ij} : j \in [k]\}$ ,  $\alpha_i = \alpha(\mathcal{B}_i) = \frac{1}{k} \sum_{j=1}^{k} y_{ij}$  is the label proportion of the  $i$ -th bag, and all the involved samples  $(x_{ij}, y_{ij})$  are drawn i.i.d. from  $\mathcal{D}$ . In words, the learner receives information about the  $nk$  labels  $y_{ij}$  of the  $nk$  instances  $x_{ij}$  only in the aggregate form determined by the  $n$  label proportions  $\alpha_i$  associated with the  $n$  labeled bags  $(\mathcal{B}_i, \alpha_i)$  in collection  $S$ . Notice, however, that the instances  $x_{ij}$  are individually observed.
Given a hypothesis set  $\mathcal{H}$  of functions  $h$  mapping  $\mathcal{X}$  to a prediction space  $\widehat{\mathcal{V}}$ , and a loss function  $\ell: \widehat{\mathcal{V}} \times \mathcal{V} \to \mathbb{R}^+$ , the learner receives a collection  $\mathcal{S}$ , and tries to find a hypothesis  $h \in \mathcal{H}$  with the smallest population loss (or risk)  $\mathcal{L}(h) = \mathbb{E}_{(x,y)} \sim_{\mathcal{D}} [\ell(h(x),y)]$  with high probability over the random draw of  $\mathcal{S}$ . When clear from the surrounding context, we will omit subscripts like “ $(x,y) \sim \mathcal{D}$ ” or “ $\mathcal{D}$ ” from probabilities and expectations.
We shall consider two broadly used learning methods for solving the above learning problem, Empirical Risk Minimization (ERM, Section 5), or regularized versions thereof, and Stochastic Gradient Descent (SGD, Section 6). In this latter context, we will consider a parameter space  $\mathcal{W}$  and consider a learner that tries to optimize a loss  $\ell$ :  $\mathcal{W} \times \mathcal{X} \times \mathcal{Y} \rightarrow \mathbb{R}$  iteratively over a collection of bags. Before that, we find it instructive to delve into the theoretical properties of an approach to LLP which is by now belonging to folklore.
3 Proportion Matching Algorithm
We now introduce a simple and very well known algorithm for learning from label proportions. Yu et al. [33] refers to this algorithm as Empirical Proportion Risk Minimization but different versions of the algorithm are discussed in the LLP literature without a clear reference to its origin.
Definition 3.1. Given a loss function  $\ell : \mathbb{R} \times \mathbb{R} \to \mathbb{R}^+$ , a hypothesis set of functions  $\mathcal{H}$ , and a collection  $S = \{(\mathcal{B}_i, \alpha_i), i \in [n]\}$  of  $n$  labeled bags of size  $k$ , the PROPMATCH algorithm minimizes the empirical proportion matching loss, i.e. it solves the following optimization problem
$$
\min  _ {h \in \mathcal {H}} \sum_ {i = 1} ^ {n} \ell \left(\frac {1}{k} \sum_ {j = 1} ^ {k} h \left(x _ {i j}\right), \alpha_ {i}\right). \tag {1}
$$
That is, the PROPMATCH algorithm creates a bag level prediction by simply averaging the predictions of a model on the bag's individual examples. Yu et al. [33] prove a uniform convergence guarantee implying that, given enough data, the minimizer of the empirical proportion matching loss is an approximate minimizer of the population level proportion matching loss:
$$
\min  _ {h \in \mathcal {H}} \underset {\mathcal {B}, \alpha} {\mathbb {E}} \left[ \ell \left(\frac {1}{k} \sum_ {x \in \mathcal {B}} h (x), \alpha\right) \right]. \tag {2}
$$
In general, the minimizer of the population level proportion matching loss may not produce accurate event-level predictions. However, Yu et al. [33] further show that when  $\ell$  is the absolute loss, achieving small population level proportion matching loss also leads to a classifier with small population level event loss.
We now strengthen those results and show that under some general conditions on the loss function, and if the function class  $\mathcal{H}$  is expressive enough, minimizing the population level proportion matching loss is equivalent to minimizing the population level event loss.
2A multiclass extension of all our results is also possible - see Appendix C.
<sup>3</sup>Due to lack of space, all proofs are given in the appendix.
Theorem 3.2. Assume  $\mathcal{H}$  is such that the function  $h^* \colon x \mapsto \mathbb{P}(y = 1|x)$  is in  $\mathcal{H}$ . Let  $\ell : \mathbb{R} \times \mathbb{R} \to \mathbb{R}^+$  be such that, for any random variable  $Z$ ,  $q = \mathbb{E}[Z]$  is the unique solution of  $\min_{r \in \mathbb{R}} \mathbb{E}_Z[\ell(r,Z)]$ . Then,  $h^*$  is a minimizer of (2). Moreover, every other minimizer  $h$  satisfies  $\mathbb{P}(h(x) = h^*(x)) = 1$ .
Corollary 3.3. Let  $\mathcal{H}$  satisfy the conditions of Theorem 3.2. Then  $h^*$  as defined above is the unique minimizer for the proportion matching loss when  $\ell$  is the square loss or the binary cross-entropy loss.
The above corollary provides us with conditions for the proportion matching loss minimizer to also minimize the event level loss. It is natural to ask ourselves, what the behavior of the proportion matching loss would be when those conditions are violated. To our knowledge, this remains an open problem. However, the following example shows that if the model class  $\mathcal{H}$  is not expressive enough, then using the proportion matching loss can in fact lead to an arbitrarily bad event level predictor.
Figure 1: Left: Training loss for different learning approaches. PROPMATCH (using (1)), EASYLLP (using (3), Section 4), and event level using regular cross-entropy loss. Right: Test loss using event level labels to measure cross-entropy loss for all methods. Notice how the PROPMATCH training loss decreases but the test loss continues to increase. EASYLLP tracks the event level loss as suggested by theory.
Let  $\mathcal{X} = \mathbb{R}^2$  and  $\mathcal{Y} = \{0,1\}$ . We define a latent variable  $\eta$  uniformly distributed in  $\{-1,1\}$ . We generate a distribution  $\mathcal{D}$  over pairs  $(x,y)$  as follows: first sample a latent variable  $\eta$ , then sample  $x$  according to the Gaussian distribution  $N((\eta ,\eta),I)$ , where  $I\in \mathbb{R}^{2\times 2}$  is the identity matrix. Finally, sample  $y$  such that  $\mathbb{P}(y = 1) = \frac{e}{e + 1}$  if  $\eta = -1$  and  $\frac{e^4}{e^4 + 1}$  if  $\eta = 1$ . Using SGD, we attempt to fit a logistic regression model parametrized by  $w\in \mathbb{R}^2$  where  $\widehat{p}_w(y = 1|x) = \frac{e^{w\cdot x}}{1 + e^{w\cdot x}}$ . Crucially the hypothesis class used to model  $\mathcal{D}$  does not include the true conditional probability distribution. For the experiment we generate 100000 examples from distribution  $\mathcal{D}$ , and group them randomly into bags of size 500. Figure 1 shows the effect of training a model using proportion matching loss vs. event level loss. Note that even though the empirical proportion matching loss is decreasing, the test loss (on events) increases.
To summarize, we have shown that PROPMATCH can, for the most popular classification and regression losses and under some assumptions on  $\mathcal{H}$ , recover a very good instance level classifier. Nonetheless, when those assumptions are violated the performance of PROPMATCH can drastically degrade. We believe these observations pave the way for new exciting research in fully understanding this simple algorithm.
4 Easy LLP
Based on the results of the previous section, we are interested in defining a robust, theoretically founded algorithm for learning from label proportions. We now introduce the main tool for our approach.
Definition 4.1. Let  $g \colon \mathcal{X} \times \mathcal{Y} \to \mathbb{R}^d$  be any (measurable) function, for some output dimension  $d \geq 1$ . Let also  $p = \mathbb{E}[y]$  be the probability of observing a positive label. We define  $\widetilde{g} \colon \mathcal{X} \times [0,1] \to \mathbb{R}^d$ , the soft-label corrected function associated with  $g$ , as
$$
\widetilde {g} (x, \alpha) = \left(k (\alpha - p) + p\right) g (x, 1) + \left(k (p - \alpha) + (1 - p)\right) g (x, 0). \tag {3}
$$
The main property of  $\widetilde{g}$ , which is relevant to LLP, is that it is as unbiased estimator of  $\mathbb{E}_{(x,y)\sim \mathcal{D}}[g(x,y)]$ , as we next show.
Proposition 4.2. Given a sample  $(x_{1},y_{1}),\ldots ,(x_{k},y_{k})$  drawn i.i.d. according to  $\mathcal{D}$ , let  $(\mathcal{B},\alpha)$  be the corresponding labeled bag of size  $k$ , for some  $k\geq 1$ . Let  $g\colon \mathcal{X}\times \mathcal{Y}\to \mathbb{R}^d$  be any (measurable) function, for some output dimension  $d\geq 1$ , and  $\widetilde{g}\colon \mathcal{X}\times [0,1]\to \mathbb{R}^d$  be its associated soft-label corrected function. Then for every element  $x_{j}\in \mathcal{B}$  we have
$$
\underset {(\mathcal {B}, \alpha)} {\mathbb {E}} \left[ \widetilde {g} (x _ {j}, \alpha) \right] = \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \left[ g (x, y) \right].
$$
For empirical risk minimization (ERM), we shall apply Proposition 4.2 with the function  $g$  that computes the per-example loss of a given model, so that we obtain unbiased estimates of the model's
population level loss. Similarly, for gradient-based optimization, we take  $g$  to be the function that computes the per-example gradient of the loss w.r.t. the model parameters so that we obtain an unbiased gradient estimate. More details of our application of Proposition 4.2 are given at the end of this section.
Proposition 4.2 shows that we can easily obtain an unbiased estimate of the expectation of any function  $g$  by applying a simple linear transformation to the output of  $g(x, y)$ , for  $y = 0, 1$ . We would like to highlight the importance of this simple proposition. Whereas there has been a lot of research in LLP, to our knowledge this is the first expression that shows that one can recover an unbiased estimate of an arbitrary function  $g$  using only information from label proportions.