SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While the above result provides us with a straightforward way to estimate the expectation of a function  $g$  (which can for instance be specialized to a loss function), note that the variance of  $\widetilde{g}$  increases as the number of elements in each bag grows. Indeed, since all terms in the definition of  $\widetilde{g}$  have a factor of  $k$ , we might expect the variance of the estimator to grow as  $k^2$ , which could be prohibitively large even for moderate values of  $k$ . Notice however that because  $\alpha = \frac{1}{k}\sum_{j=1}^{k}y_j$ , and each  $y_j \sim \text{Bernoulli}(p)$ , we expect by standard concentration arguments that  $k(\alpha - p) \in O(\sqrt{k})$  which should imply that the variance scales like  $k$ . The following theorem shows that indeed, the variance of these estimates is of order  $k$  and not  $k^2$ . In addition, since  $\widetilde{g}(x_j, \alpha)$  is unbiased for all  $j \in [k]$ , so is  $\frac{1}{k}\sum_{j=1}^{k}\widetilde{g}(x_j, \alpha)$ . The same theorem also shows that the variance of the latter estimator is always smaller than the variance of the former. That is, using all  $k$  samples in a bag is always better (in terms of variance) than using any single sample.
Theorem 4.3. Let  $g \colon \mathcal{X} \times \mathcal{Y} \to \mathbb{R}^d$  be such that  $\sup_{x,y} \| g(x,y) \|^2 \leq M$ , and denote by  $\widetilde{g}$  its associated soft labeled corrected function. Also, set for brevity  $g_0 = g(x,0)$  and  $g_1 = g(x,1)$  and, for each  $j \in [k]$ ,  $\widetilde{g}_j = \widetilde{g}(x_j, \alpha)$ . Then, for any size  $k \geq 1$  and any  $j \in [k]$ ,
$$
\mathbb {E} [ \| \widetilde {g} _ {j} \| ^ {2} ] = \mathbb {E} [ \| g (x _ {j}, y _ {j}) \| ^ {2} ] + (k - 1) p (1 - p) \mathbb {E} \left[ \| g _ {0} - g _ {1} \| ^ {2} \right]
$$
$$
\mathbb {E} \left[ \left| \left| \frac {1}{k} \sum_ {i = 1} ^ {k} \widetilde {g} _ {i} \right| \right| ^ {2} \right] \leq \mathbb {E} [ | | \widetilde {g} _ {j} | | ^ {2} ]. \tag {4}
$$
Moreover, there exists a universal constant  $C$  such that
$$
\mathbb {E} \left[ \left| \left| \frac {1}{k} \sum_ {i = 1} ^ {k} \widetilde {g} _ {i} \right| \right| ^ {2} \right] \leq C + k p (1 - p) \left| \left| \mathbb {E} [ g _ {0} - g _ {1} ] \right| \right| ^ {2},
$$
where  $p = \mathbb{P}_{(x,y)\sim \mathcal{D}}(y = 1)$
The bound in the above theorem confirms our intuition. Moreover, it shows that the variance grows slower for datasets where  $p$  is close to 1 or 0. This is intuitively clear, for very skewed datasets, we expect label proportions to provide a better description of the true labels. In the extreme cases where  $p = 0$  or  $p = 1$ , LLP becomes equivalent to learning from individual examples.
The results of this section have demonstrated that for any function  $g$ , one can obtain an estimator of its expectation using only label proportions. More importantly the variance of this estimator only scales linearly with the bag size.
Note about knowledge of population level positive rate. At this point the reader is aware that the definition of the soft label corrected function requires knowledge of the population level positive rate  $p$ . While the exact value of  $p$  is unknown, one can easily estimate it from the label proportions itself. Indeed, using the fact that the generated bags are i.i.d. it is easy to see that  $\widehat{p} = \frac{1}{n}\sum_{i=1}^{n}\alpha_i = \frac{1}{nk}\sum_{i,j}y_{ij}$  is a very good estimator for  $p$ .
EASYLLP. We now have all elements to introduce the EASYLLP framework for learning from label proportions. The framework consists of specializing the function  $g$  for particular learning tasks. Two notable instantiations of EASYLLP which we will analyze in further sections are ERM and stochastic gradient descent (SGD). For ERM, given a hypothesis  $h$  and a loss function, we let  $g_{h}(x,y) = \ell(h(x),y)$  and the corresponding soft label corrected loss  $\widetilde{\ell}(h(x),\alpha) = \widetilde{g}_{h}(x,\alpha)$ . To provide regret guarantees using SGD over bags in a parameter space  $\mathcal{W}$  and loss function
$\ell \colon \mathcal{W}\times \mathcal{X}\times \mathcal{Y}\to \mathbb{R},$  we use EASYLLP to estimate the gradient of the loss function with respect to a parameter  $\mathbf{w}\in \mathcal{W}$  by letting  $g_{\mathbf{w}}(x,y) = \nabla_{\mathbf{w}}\ell (\mathbf{w},x,y)$  and its corresponding soft label corrected function  $\widetilde{g}_{\mathbf{w}}(x,\alpha) = \nabla \widetilde{\ell} (\mathbf{w},x,\alpha)$
5 ERM with Label Proportions
Given a hypothesis space  $\mathcal{H}$ , let  $\ell$  be a loss function as defined in Section 2. Given a collection of bags  $S = \{(\mathcal{B}_i, \alpha_i), i \in [n]\}$  of size  $k$ , our learning algorithm simply finds  $h \in \mathcal{H}$  that minimizes the empirical risk constructed via the soft label corrected loss from Eq. (3):
$$
\sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {k} \widetilde {\ell} \left(h \left(x _ {i j}\right), \alpha_ {i}\right). \tag {5}
$$
The main advantage of our algorithm lies in its simplicity and generality. Indeed, our algorithm can be used for any loss function and any hypothesis set. This is in stark contrast, e.g., to the works of [22, 20], whose framework is only applicable to the logistic loss and (generalized) linear models. From a practical standpoint, our approach can also leverage existing learning infrastructures, as the only thing that needs to be specified is a different loss function — which in frameworks like Tensorflow, JAX and PyTorch requires only minimal coding. This differs from other approaches to assigning surrogate labels which may require solving combinatorial optimization problems (or relaxations thereof) like, e.g., [32, 8].
The following theorem provides learning guarantees for minimizing the above empirical loss. Our guarantees are given in terms of the well-known Rademacher complexity of a class of functions.
Definition 5.1. Let  $\mathcal{Z}$  be an arbitrary input space and let  $\mathcal{G} \subset \{g \colon \mathcal{Z} \to \mathbb{R}\}$  be a collection of functions over  $\mathcal{Z}$ . Let  $\mathcal{D}$  be a distribution over  $\mathcal{Z}$  and  $\mathcal{S} = \{z_1, \ldots, z_m\}$  be an i.i.d. sample. The Rademacher complexity of  $G$  is given by  $\Re_n(\mathcal{G}) = \frac{1}{n} \mathbb{E}_{\mathcal{S},\sigma} \left[ \sup_{g \in \mathcal{G}} \sum_{i=1}^{n} g(z_i) \sigma_i \right]$ , where  $\sigma = (\sigma_1, \dots, \sigma_n) \in \{-1,1\}^n$  is uniformly distributed.
Theorem 5.2. Let  $\delta >0$ ,  $\mathcal{S} = \{(\mathcal{B}_i,\alpha_i),i\in [n]\}$  be a collection of  $n$  bags of size  $k$ .<sup>4</sup> Let  $\sup_{\widehat{y},y}\ell (\widehat{y},y)\leq B$ , and  $C_n^k = 2\left(\sqrt{2k\log(kn)} +1\right)$ . Then the following bound holds uniformly for all  $h\in \mathcal{H}$  with probability at least  $1 - \delta$ :
$$
\left| \mathcal {L} (h) - \frac {1}{n k} \sum_ {i, j} \widetilde {\ell} (h (x _ {i j}), \alpha_ {i}) \right| \leq C _ {n} ^ {k} \left(\Re_ {k n} (\mathcal {H} _ {\ell} ^ {(1)}) + \Re_ {k n} (\mathcal {H} _ {\ell} ^ {(0)})\right) + \frac {4 B}{n} + 4 B \sqrt {\frac {k \log (2 / \delta)}{2 n}},
$$
where  $\mathcal{H}_{\ell}^{(1)} = \{x\to \ell (h(x),1)\colon h\in \mathcal{H}\}$  and  $\mathcal{H}_{\ell}^{(0)} = \{x\to \ell (h(x),0)\colon h\in \mathcal{H}\}$ .
Corollary 5.3. With the notation of the previous theorem, let  $\hat{h}$  denote the minimizer of (5). Then with probability at least  $1 - \delta$  over the sampling process we have
$$
\mathcal {L} (\widehat {h}) \leq \min  _ {h \in \mathcal {H}} \mathcal {L} (h) + 2 \left(C _ {n} ^ {k} \left(\Re_ {k n} \left(H _ {\ell} ^ {(1)}\right) + \Re_ {k n} \left(H _ {\ell} ^ {(0)}\right)\right) + \frac {4 B}{n} + 4 B \sqrt {\frac {k \log (2 / \delta)}{2 n}}\right).
$$
Comparison to event level learning. We can now compare the bound from Theorem 5.2 to standard learning bounds for instance level learning like that of [18]. Assuming we had access to a labeled i.i.d. sample  $(x_{ij},y_{ij})$  of size  $kn$ , Theorem 3.3 in [18] ensures that with probability at least  $1 - \delta$  the following bounds hold for all  $h\in \mathcal{H}$ :
$$
\left| \mathcal {L} (h) - \frac {1}{n k} \sum_ {i, j} \ell \left(h \left(x _ {i j}, y _ {i j}\right)\right) \right| \leq 2 \Re_ {n k} \left(\mathcal {H} _ {\ell}\right) + B \sqrt {\frac {\log (2 / \delta)}{2 n k}}, \tag {6}
$$
where  $\mathcal{H}_{\ell} = \{(x,y)\to \ell (h(x),y)\colon h\in \mathcal{H}\}$ . Note that under the weak assumption that the Rademacher complexities  $\Re_{kn}(\mathcal{H}_{\ell}^{(r)})$ ,  $r\in \{0,1\}$ , are of the same order as  $\Re_{kn}(\mathcal{H}_{\ell})$ , the main
Our results can easily be extended to the case where bags have different sizes, provided the items are still i.i.d. The equal size assumption is made here mainly for notational convenience. If the bag sizes are constant with the number of bags  $n$ , then a rate of the form  $1 / \sqrt{n}$  can still be obtained. On the contrary, if the bag sizes scale with  $n$ , the actual rate will depend on the interplay between the bag sizes and  $n$  itself.
difference between the bound in Theorem 5.2 and (6) is simply an extra factor  $C_n^k \in \tilde{O}(\sqrt{k})$  in the complexity term and a factor  $k$  multiplying the confidence term. That is, we achieve similar guarantees to event level learning by increasing the sample size by a factor of roughly  $k$ .
It is worth stressing the difference in flavor of the consistency result contained in Theorem 5.2 above and those in [17, 16]. For instance, in [16] (even with  $m = 2$  bags) the consistency limit has to be interpreted "as the bag size  $n_{tr} = n_1 + n_2$  goes to infinity". In our case, the bag size  $k$  has to remain constant, and it is the number of bags  $n$  that goes to infinity. This further strengthens our claim that the stream of literature that those papers are representative of are by no means subsuming our results. Moreover, unlike our paper, all results in these previous works, as presented, make assumptions about the loss function (e.g., square loss or cross entropy for [16], margin-based losses for [17]). The fact that we make no assumptions allows us to apply our debiasing procedure to any function  $g(x,y)$  of two variables, hence we can debias, e.g., also the gradient of a loss function, enabling the principled usage of stochastic gradient descent procedures with only label proportion information. This is illustrated in the next section.
6 SGD with Label Proportions
We now focus on understanding the effect of label proportions on another very popular learning algorithm: stochastic gradient descent (SGD).
Proposition 4.2 delivers an unbiased estimate of the gradient which can be naturally plugged into any SGD algorithm (e.g., [27]), and one would hope for an upper bound on the excess risk if the learning task at hand leads to a convex optimization problem. The difficulty is that, even if each gradient in a given bag is individually unbiased, the gradients are correlated since they depend on the label proportion computed on the bag. A simple way around it is to pick a single item uniformly at random from the bag to update the model parameters. This is a slight departure from what we considered for ERM, but it both makes our SGD analysis easier and does not affect asymptotic performance.
In order to devise an SGD algorithm that can handle bagged data, first note that the debiasing technique introduced in Proposition 4.2 is very general and it applies to any measurable function, including the gradient of any loss function. Let us denote the gradient of the loss on example  $(x,y)$  by  $g_{\mathbf{w}}(x,y)$  where  $\mathbf{w}$  is the parameter vector of the model and let its soft-label corrected version be  $\widetilde{g}_{\mathbf{w}}$ . We study a version of projected SGD that picks one example per bag and uses the soft-label corrected gradient estimates (pseudocode is given in in Algorithm 1 in the Appendix). The excess risk of this algorithm depends on the squared norm of the debiased gradients, which is at most  $k$  times larger than the variance of the original instance-level gradient based on Theorem 4.3. This observation results in the following risk bound.
Theorem 6.1. Suppose that  $F(\mathbf{w}) = \mathbb{E}[\ell_{\mathbf{w}}(x,y)]$  is convex,  $\mathbb{E}[\| g_{\mathbf{w}_t}(x,y)\|^2] \leq G^2$  for all  $t \in [n]$ , and  $\sup_{\mathbf{w},\mathbf{w}' \in \mathcal{W}} \| \mathbf{w} - \mathbf{w}' \| \leq D$ . Consider Algorithm 1, run with step size  $\eta_t = 1 / \sqrt{k \cdot t}$ . Then for any  $n > 1$  we have
$$
\mathbb {E} \left[ F \left(\mathbf {w} _ {n}\right) - F \left(\mathbf {w} ^ {*}\right) \right] \leq \sqrt {k} \cdot \left(D ^ {2} + 5 G ^ {2}\right) \frac {2 + \log (n)}{\sqrt {n}}.
$$
This theorem quantifies the deterioration in the excess of the risk in terms of bag size when the algorithm is applied with LLP data. Recall that the error of SGD with  $kn$  individually labeled samples decreases like  $O\left(1 / \sqrt{kn}\right)$ . Compared with the above bound, and similar to the ERM case, the regret bound increases by a factor of  $k$ .
The most predominant learning frameworks used today fit deep models to data using SGD-like methods. Even though these problems are typically non-convex, we remark that Proposition 4.2 guarantees that the expected gradient steps of Algorithm 1 are equal to the expected steps of SGD using an instance-level loss. Moreover, Theorem 4.3 guarantees that the variance of the gradient estimates used by Algorithm 1 is not much larger than that of the instance-level case. In particular, it follows that whenever SGD is an effective algorithm for learning from instance-level data, Algorithm 1 should also be effective at learning from LLP data, provided that we have enough data to overcome the increased variance. This is thoroughly demonstrated in the next section. Finally, the only required modifications to any gradient-based training infrastructure to use EASYLLP are to replace the original labels  $y_{ik}$  by label proportions and to implement the soft-label corrected loss. Taken together, these properties of EASYLLP make it especially well suited to modern machine learning pipelines.
In this section we empirically evaluate EASYLLP, PROPMATCH, and two baseline methods to characterize how their performance depends on the bag size for a range of different learning tasks and underlying learning models.
LLP Methods. We evaluate EASYLLP, PROPMATCH, and two baselines. The first baseline is MEANMAP [22], which is a method specialized for learning linear logistic regression models from label proportions. We also compare against the method described in Section 3.2 of [8], which we denote by DA. The DA method computes a loss for each bag by creating synthetic labels for the examples that optimize a combination of the model's loss on the synthetic labels, and a divergence between the label proportion of the synthetic labels and the bag's label proportion. The DA method has an additional hyperparemeter,  $\alpha$ , which controls the weights on these two objectives. Full details of our DA implementation are in Appendix A.3.
Datasets. We carry out experiments on four (binary classification) datasets: Binarized versions of MNIST [13] and CIFAR-10 [12], as well as the Higgs [3] and UCI adult datasets [11]. The labels in MNIST are replaced by labels indicating whether the digit is even or odd, and the CIFAR-10 labels are replaced by whether the original class is an animal or a machine. The MNIST images are resized to have shape  $32 \times 32 \times 3$  to match CIFAR-10. For both image datasets, we normalize the images to have  $\ell_2$  norm equal to one. No preprocessing is performed on the Higgs dataset. Finally, on UCI Adult, we rescale numerical features to the range [0, 1] and one-hot encode categorical features.
Models. We evaluate each LLP method on every dataset using a linear model. Additionally, for the image datasets we also evaluate using a small ConvNet, a large ConvNet, and Resnet-11 [9]. On Higgs we also use the DNN proposed by [3], which has 4 hidden layers, each with 300 units. Finally, on UCI Adult, we also use a NN with one hidden layer. Since the MEANMAP baseline is only applicable to learning linear models, we do not include it in the neural network experiments. Full details of the architectures are provided in Appendix A.2.
Experimental Setup. For each combination of LLP method, dataset, model, and bag size, we train the model 10 times with different seeds and report the average test accuracy. To generate the label proportion data, on each run we shuffle the training data, partition it into consecutive bags of the desired size, and replace the original labels with label proportions computed from the bags. To tune the learning rate for each method, we report the highest accuracy achieved for learning rates in  $\{0.00001, 0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05\}$ . For DA, we tune the learning rate and  $\alpha$  using a grid search with the same learning rates and  $\alpha \in \{0.0, 0.0001, 0.001, 0.01, 0.1, 0.5\}$ . Note that DA is sensitive to the choice of  $\alpha$ , and it is unclear how to tune  $\alpha$  without access to event-level data. In all cases we use the Adam [10] optimizer, binary crossentropy loss, minibatches of size 512, and 20 training passes through the data. Finally, for the two image datasets, we decay the learning rate after  $40\%$ ,  $60\%$ ,  $80\%$ , and  $90\%$  of the training passes by factors 10, 100, 1000, and 5000, respectively.
Results. Figure 2 depicts the accuracy achieved by each method on a selection of datasets and models for a range of bag sizes. The selected dataset and model combinations are representative of the full set of experimental results, which are included in Appendix A.5. Across all datasets, either EASYLLP or PROPMATCH achieves the highest test accuracy for large bag sizes. In agreement with our theory, the performance of EASYLLP decreases predictably with the bag size (due to the increased variance of gradient estimates). On the other hand, we find that DA often performs very well for small bag sizes, but accuracy drops rapidly at a dataset-dependent threshold. PROPMATCH is a competitive baseline, specifically when the underlying learning model is well specified for the task, which is in line with our finding in Section 3. For instance, when using the Resnet-11 model, PROPMATCH has a higher performance than EASYLLP on both MNIST and CIFAR-10. A partial explanation is due to Theorem 4.3. Indeed, it is well known that models trained under logistic regression become overconfident on their predictions. However, our theory suggests that the variance of the gradient increases proportional to  $\| g(x,0) - g(x,1)\|$  which rapidly becomes unbounded as
Due to the differences in problem setup between our work and that of other papers, like [17, 16], it is unclear how a fair experimental comparison to those papers should be performed. For this very reason, we have deliberately avoided to experimentally compare to the algorithms in those papers.
Figure 2: Average test accuracy for each LLP method on selected datasets and models. Error bars show one standard error in the mean.
the model becomes more confident. To test this hypothesis, we trained a model using a double-sided hinge loss, labeled "EASYLLP DHL" (details in Appendix A). This loss induces a small variance term in Theorem 4.3 and indeed we see that this partially closes the gap to PROPMATCH. On the other hand, PROPMATCH's performance degrades more rapidly than EASYLLP's as the bag size increases in the presence of model mis-specification or higher noise in the data (first three plots in Figure 2).
Loss tracking. Another strength of EASYLLP is that the loss estimates computed during training are unbiased estimates of the true training loss. In comparison, the losses minimized by all other methods we consider do not have clear connections to the true training loss and it is unclear how to monitor training performance. In Appendix A.6 we demonstrate this empirically.
8 Conclusions and Limitations
We have studied the problem of learning from label proportions in the case where bags are drawn i.i.d. We have introduced EASYLLP, a novel and flexible approach to LLP for classification that is widely applicable and has well-developed theory. In particular, we have shown how to use EASYLLP to estimate the expected value of any function of  $(x,y)$  pairs from labeled data, and applied these results to proving ERM sample complexity guarantees and convergence guarantees for SGD. In both cases, we have shown that the LLP performance of EASYLLP is only a factor  $k$  worse than when learning from event-level data. We have also elucidated important theoretical properties of the folklore PROPMATCH algorithm which are suggestive of the practical scenarios where it is advisable to make use of it. Finally, we have carried out an extensive empirical evaluation of EASYLLP, PROPMATCH, and two baseline methods on diverse tasks and learning models, and identified relative performance trends.
The results in this work are limited to the special case of LLP where the examples in each bag are drawn i.i.d. from an underlying distribution. While this is a common and important special case, in future work we would like to extend our results to handle cases where the bags need not be i.i.d. Similarly, we hope to study the theoretical properties of EASYLLP more fully in the multiclass case, and to have a complete characterization of when PROPMATCH is consistent. We also believe that it may be possible to further decrease the variance of EASYLLP estimates by replacing the label marginal  $p$  in (3) with bag-specific predictions. Finally, while the LLP framework provides an intuitive form of privacy protection, we would like to explore connections with differential privacy where, for example, each bag's label proportion is computed by a differentially private mechanism.
[1] Private aggregation API, 2022. URL https://developer.chrome.com/docs/privacy-sandbox/private-aggregation/.
[2] E. M. Ardehaly, , and A. Culotta. Co-training for demographic95classification using deep learning from label proportions. In 2017 IEEE International Conference on Data Mining Workshops, pages 1017-1024, 2017.
[3] Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for Exotic Particles in High-Energy Physics with Deep Learning. Nature Commun., 5:4308, 2014. doi: 10.1038/ncomms5308.