SlowGuess/NeurIPS_2023_OCR huggingface dataset:
PU learning. Bekker and Davis [9] groups PU Learning paradigms into three main classes: two step, biased, and class prior incorporation. Biased learning techniques train a classifier on the entire dataset
Table 3: LLP results across different bag sizes. We report the mean and standard deviation of the test AUC over 5 seeds for each setting. The highest metric for each setting is shown in **boldface**.
Dataset	Dist	Method	8	32	128	512
Adult	[0, 1/2]	PL	0.8889 ± 0.0024	0.8782 ± 0.0036	0.8743 ± 0.0039	0.8678 ± 0.0085
Adult	[0, 1/2]	LMMCM	0.8728 ± 0.0019	0.8693 ± 0.0047	0.8669 ± 0.0041	0.8674 ± 0.0040
Adult	[0, 1/2]	CL (Ours)	0.8984 ± 0.0013	0.8848 ± 0.0041	0.8743 ± 0.0052	0.8703 ± 0.0070
Adult	[1/2, 1]	PL	0.8781 ± 0.0038	0.8731 ± 0.0035	0.8699 ± 0.0057	0.8556 ± 0.0180
Adult	[1/2, 1]	LMMCM	0.8584 ± 0.0164	0.8644 ± 0.0052	0.8601 ± 0.0045	0.8500 ± 0.0186
Adult	[1/2, 1]	CL (Ours)	0.8854 ± 0.0022	0.8738 ± 0.0039	0.8675 ± 0.0043	0.8607 ± 0.0056
Adult	[0, 1]	PL	0.8884 ± 0.0030	0.8884 ± 0.0008	0.8879 ± 0.0025	0.8828 ± 0.0051
Adult	[0, 1]	LMMCM	0.8831 ± 0.0026	0.8819 ± 0.0006	0.8821 ± 0.0017	0.8786 ± 0.0052
Adult	[0, 1]	CL (Ours)	0.8985 ± 0.0010	0.8891 ± 0.0013	0.8871 ± 0.0021	0.8790 ± 0.0056
Magic	[0, 1/2]	PL	0.8900 ± 0.0095	0.8510 ± 0.0032	0.8405 ± 0.0110	0.8332 ± 0.0149
Magic	[0, 1/2]	LMMCM	0.8918 ± 0.0077	0.8799 ± 0.0113	0.8753 ± 0.0157	0.8734 ± 0.0092
Magic	[0, 1/2]	CL (Ours)	0.9088 ± 0.0056	0.8830 ± 0.0097	0.8926 ± 0.0049	0.8864 ± 0.0107
Magic	[1/2, 1]	PL	0.9066 ± 0.0016	0.8818 ± 0.0108	0.8769 ± 0.0101	0.8429 ± 0.0443
Magic	[1/2, 1]	LMMCM	0.8911 ± 0.0083	0.8790 ± 0.0091	0.8684 ± 0.0046	0.8567 ± 0.0292
Magic	[1/2, 1]	CL (Ours)	0.9105 ± 0.0020	0.8980 ± 0.0059	0.8851 ± 0.0255	0.8816 ± 0.0083
Magic	[0, 1]	PL	0.9039 ± 0.0029	0.8870 ± 0.0037	0.9002 ± 0.0092	0.8807 ± 0.0200
Magic	[0, 1]	LMMCM	0.9070 ± 0.0026	0.9048 ± 0.0058	0.9113 ± 0.0058	0.8934 ± 0.0097
Magic	[0, 1]	CL (Ours)	0.9173 ± 0.0018	0.9102 ± 0.0057	0.9146 ± 0.0051	0.9088 ± 0.0039
with the understanding that negative samples are subject to noise [9]. We will focus on a subset of biased learning techniques (Risk Estimators) as they are considered state-of-the-art and relevant to us as baselines. The Unbiased Risk Estimator (uPU) provides an alternative to the inefficiencies in manually biasing unlabeled data [18, 37]. Later, Non-negative Risk Estimator (nnPU) [26] accounted for weaknesses in the unbiased risk estimator such as overfitting.
Count Loss. To our knowledge, viewing the computation of the "bag posterior" as probabilistic is new. However, the prior approaches do this implicitly. Many approaches have tried to approximate the "bag posterior" by averaging the instance-level probabilities in a bag [8, 43]. In MIL settings, among instance-level approaches, the MIL-pooling is an implicit "bag posterior" computation. These include mean, max, and log-sum-exp pooling to approximate the likelihood that a bag has at least one positive instance [44]. But again, these are all approximations of what our computation does exactly. In PU Learning, to our best knowledge, the view of unlabeled data as a bag annotated with the mixture proportion is new.
Neuro-Symbolic Losses. In this paper, we have dealt with a specific form of distributional constraint. Conversely, there has been a plethora of work exploring the integration of hard symbolic constraints into the learning of neural networks. This can take the form of enforcing a hard constraint [3], whereby the network's predictions are guaranteed to satisfy the pre-specified constraints. Or it can take the form of a soft constraint [48, 34, 1, 4, 2, 5] whereby the network is trained with an additional loss term that penalizes the network for placing any probability mass on predictions that violate the constraint. While in this work we focus on discrete linear inequality constraints defined over binary variables, there is existing work focusing on hybrid linear inequality constraints defined over both discrete and continuous variables and their tractability [10, 55, 54]. The development of inference algorithms for such constraints and their applications such as Bayesian deep learning remain an active topic [52, 28, 53, 51].
In this section, we present a thorough empirical evaluation of our proposed count loss on the three weakly supervised learning problems,  $LLP$ ,  $MIL$ , and  $PU$  learning. We refer the readers to the appendix for additional experimental details.
$^{1}$ Code and experiments are available at https://github.com/UCLA-StarAI/CountLoss
Table 4: MIL experiment on the MNIST dataset. Each block represents a different distribution from which we draw bag sizes—First Block:  $\mathcal{N}(10,2)$ , Second Block:  $\mathcal{N}(50,10)$ , Third Block:  $\mathcal{N}(100,20)$ . We run each experiment for 3 runs and report mean test AUC with standard error. The highest metric for each setting is shown in **boldface**.
Training Bags	50	100	150	200	300	400	500
Gated Attention	0.775 ± 0.034	0.894 ± 0.012	0.935 ± 0.005	0.939 ± 0.006	0.963 ± 0.002	0.959 ± 0.002	0.966 ± 0.003
Attention	0.807 ± 0.026	0.913 ± 0.006	0.940 ± 0.004	0.942 ± 0.007	0.957 ± 0.002	0.961 ± 0.005	0.965 ± 0.004
CL (Ours)	0.818 ± 0.024	0.906 ± 0.009	0.929 ± 0.005	0.946 ± 0.001	0.952 ± 0.004	0.962 ± 0.002	0.963 ± 0.002
Gated Attention	0.943 ± 0.005	0.949 ± 0.009	0.970 ± 0.005	0.977 ± 0.001	0.983 ± 0.002	0.986 ± 0.004	0.987 ± 0.002
Attention	0.936 ± 0.010	0.962 ± 0.006	0.970 ± 0.001	0.977 ± 0.002	0.981 ± 0.002	0.987 ± 0.001	0.987 ± 0.002
CL (Ours)	0.939 ± 0.010	0.960 ± 0.002	0.964 ± 0.007	0.972 ± 0.002	0.982 ± 0.003	0.982 ± 0.001	0.987 ± 0.002
Gated Attention	0.975 ± 0.003	0.981 ± 0.004	0.992 ± 0.002	0.987 ± 0.004	0.996 ± 0.001	0.998 ± 0.001	0.990 ± 0.004
Attention	0.984 ± 0.001	0.982 ± 0.001	0.996 ± 0.000	0.987 ± 0.007	0.992 ± 0.004	0.994 ± 0.002	0.998 ± 0.000
CL (Ours)	0.981 ± 0.007	0.989 ± 0.000	0.996 ± 0.002	0.995 ± 0.001	0.996 ± 0.002	0.993 ± 0.003	0.999 ± 0.001
Table 5: MIL: We report mean test accuracy, AUC, F1, precision, and recall averaged over 5 runs with std. error on the Colon Cancer dataset. The highest value for each metric is shown in boldface.
Method	Accuracy	AUC	F1	Precision	Recall
Gated Attention	0.909 ± 0.014	0.908 ± 0.013	0.886 ± 0.021	0.916 ± 0.020	0.879 ± 0.020
Attention	0.893 ± 0.015	0.890 ± 0.008	0.876 ± 0.017	0.908 ± 0.016	0.879 ± 0.018
CL (Ours)	0.915 ± 0.008	0.912 ± 0.010	0.903 ± 0.010	0.936 ± 0.014	0.898 ± 0.007
6.1 Learning from Label Proportions
We experiment on two datasets: 1) Adult with 8192 training samples where the task is to predict whether a person makes over  $50k$  a year or not given personal information as input; 2) Magic Gamma Ray Telescope with 6144 training samples where the task is to predict whether the electromagnetic shower is caused by primary gammas or not given information from the atmospheric Cherenkov gamma telescope [19].<sup>2</sup>
We follow Scott and Zhang [40] where two settings are considered: one with label proportions uniformly on  $[0, \frac{1}{2}]$  and the other uniformly on  $[\frac{1}{2}, 1]$ . Additionally, we experiment on a third setting with label proportions distributing uniformly on  $[0, 1]$  which is not considered in Scott and Zhang [40] but is the most natural setting since the label proportion is not biased toward either 0 or 1. We experiment on four bag sizes  $n \in \{8, 32, 128, 512\}$ .
Count loss (CL) denotes our proposed approach using the loss objective defined in Table 2 for LLP. We compare our approach with a mutual contamination framework for LLP (LMMCM) [40] and against Proportion Loss (PL) [43].
Results and Discussions We show our results in Table 3. Our method showcases superior results against the baselines on both datasets and variations in bag sizes. Especially in cases with lower bag sizes, i.e., 8, 32, CL greatly outperforms all other methodologies. Among our baselines are methods that approximate the bag posterior (PL), which we show to be less effective than optimizing the exact bag posterior with CL.
6.2 Multiple Instance Learning
We first experiment on the MNIST dataset [30] and follow the MIL experimental setting in Ilse et al. [24]: the training and test set bags are randomly sampled from the MNIST training and test set respectively; each bag can have images of digits from 0 to 9, and bags with the digit 9 are labeled positive. Moreover, the dataset is constructed in a balanced way such that there is an equal amount of positively and negatively labeled bags as in Ilse et al. [24]. The task is to train a classifier that is able to predict bag labels; the more challenging task is to discover key instances, that is, to train a classifier that identifies images of digit 9. Following Ilse et al. [24], we consider three settings that vary in the bag generation process: in each setting, bags have their sizes generated from a normal distribution being  $\mathcal{N}(10,2),\mathcal{N}(50,10),\mathcal{N}(100,20)$  respectively. The number of bags in
2Publicly available at archive.ics.uci.edu/ml
Figure 2: MIL MNIST dataset experiments with decreased numbers of training bags and lower bag size. Left: bag sizes sampled from  $\mathcal{N}(10,2)$ ; Right: bag sizes sampled from  $\mathcal{N}(5,1)$ . We plot the mean test AUC (aggregated over 3 trials) with standard errors for 4 bag sizes. Best viewed in color.
training set  $n$  is in  $\{50, 100, 150, 200, 300, 400, 500\}$ . Thus, we have  $3 \times 7 = 21$  settings in total. Additionally, we introduce experimental analysis on how the performance of the learning methods would degrade as the number of bags and total samples in training set decreases, by modulating the number of training bags  $n$  to be  $\{10, 20, 30, 40\}$  and selecting bag sizes from  $\mathcal{N}(5, 1)$  and  $\mathcal{N}(10, 2)$ .
We also experiment on the Colon Cancer dataset [41] to simulate a setting where bag instances are not independent. The dataset consists of 100 total hematoxylin-eosin (H&E) stained images, each of which contains images of cell nuclei that are classified as one of: epithelial, inflammatory, fibroblast, and miscellaneous. Each image represents a bag and instances are  $27 \times 27$  patches extracted from the original image. A
Figure 3: A test bag from our MIL experiments, where we set only the digit 9 as a positive instance. Highlighted in red are digits identified to be positive with corresponding probability beneath.
positively labeled bag or image is one that contains the epithelial nuclei. For both datasets, we include the Attention and Gated Attention mechanism [24] as baselines. We also use the MIL objective defined in Table 2.
Results and Discussions For the MNIST experiments, CL is able to outperform all other baselines or exhibit highly comparable performance for bag-level predictions as shown in Table 4. A more interesting setting is to compare how robust the learning methods are if the number of training bags decreases. Wang et al. [44] claim that instance-level classifiers tend to lose against embedding-based methods. However, we show in our experiment that this is not true in all cases as seen in Figure 2. While Attention and Gated Attention are based on embedding, they suffer from a more severe drop in predictive performance than CL when the number of training bags drops from 40 to 10; our method shows great robustness and consistently outperforms all baselines. The rationale we provide is that with a lower number of training instances, we need more supervision over the limited samples we have. Our constraint provides this additional supervision, which accounts for the difference in performance.
We provide an additional investigation in Figure 3 to show that our approach learns effectively and delivers accurate instance-level predictions under bag-level supervision. In Figure 3, we can see that even though the classifier is trained on feedback about whether a bag contains the digit 9 or not, it accurately discovers all images of digit 9. To reinforce this, Table 7 and Table 8, in Appendix B, show that our approach outperforms existing instance-space methods on instance-level classification.
Our experimental results on the Colon Cancer dataset are shown in Table 5. We show that both our proposed objectives are able to consistently outperform baseline methods on all metrics. Interestingly, we do not expect CL to perform well when instances in a bag are dependent; however, the results indicate that our count loss is robust to these settings.
Table 6: PU Learning: We report accuracy and standard deviation on a test set of unlabeled data, which is aggregated over 3 runs. The results from CVIR, nnPU, and uPU are aggregated over 10 epochs, as in Garg et al. [22], while we choose the single best epoch based on validation for our approaches. The highest metric for each setting is shown in **boldface**.
Dataset	Network	CL-expect (Ours)	CL (Ours)	CVIR	nnPU	nPU
Binarized MNIST	MLP	95.9 ± 0.15	96.4 ± 0.01	96.3 ± 0.07	96.1 ± 0.14	95.2 ± 0.19
MNIST17	MLP	98.7 ± 0.17	99.0 ± 0.19	98.7 ± 0.09	98.4 ± 0.20	98.4 ± 0.09
Binarized CIFAR	ResNet	79.2 ± 0.27	80.1 ± 0.34	82.3 ± 0.18	77.2 ± 1.03	76.7 ± 0.74
CIFAR Cat vs. Dog	ResNet	76.5 ± 1.86	74.8 ± 1.64	73.3 ± 0.94	71.8 ± 0.33	68.8 ± 0.53
6.3 Learning from Positive and Unlabeled Data
We experiment on dataset MNIST and CIFAR-10 [29], following the four simulated settings from Garg et al. [22]: 1) Binarized MNIST: the training set consists of images of digits 0 - 9 and images with digits in range [0, 4] are positive instances while others as negative; 2) MNIST17: the training set consists of images of digits 1 and 7 and images with digit 1 are defined as positive while 7 as negative; 3) Binarized CIFAR: the training set consists of images from ten classes and images from the first five classes is defined as positive instances while others as negative; 4) CIFAR Cat vs. Dog: the training set consists of images of cats and dogs and images of cats are defined as positive while dogs as negative. The mixture proportion is 0.5 in all experiments. The performance is evaluated using the accuracy on a test set of unlabeled data.
As shown in Table 2, we propose two objectives
Figure 4: MNIST17 setting for PU Learning: We compute the average discrete distribution for CL and CVIR, over 5 test bags, each of which contain 100 instances. A ground truth binomial distribution of counts is also shown.
for PU learning. Our first objective is denoted by CL whereas the second approach is denoted by CL-expect. We compare against the Conditional Value Ignoring Risk approach (CVIR) [22], nnPU [26], and uPU [37].
Results and Discussions Accuracy results are presented in Table 6 where we can see that our proposed methods perform better than baselines on 3 out of the 4 simulated PU learning settings. CL-expect builds off a similar "exactly-k" count approach, which we have shown to work well in the label proportion setting. The more interesting results are from CL where we fully leverage the information from a distribution as supervision instead of simply using the expectation. We think of this as applying a loss on each count weighted by their probabilities from the binomial distribution. We provide further evidence that our proposed count loss effectively guides the classifier towards predicting a binomial distribution as shown in Figure 4: we plot the count distributions predicted by CL and CVIR as well as the ground-truth binomial distribution. We can see that CL is able to generate the expected distribution, proving the efficacy of our approach.
In this paper, we present a unified approach to several weakly-supervised tasks, i.e., LLP, MIL, PU. We construct our approach based on the idea of using weak labels to constrain count-based probabilities computed from model outputs. A future direction for our work can be to extend to multi-class classification as well as explore the applicability to other weakly-supervised settings, e.g. label noise learning, semi-supervised learning, and partial label learning [15, 36, 58].
We would like to thank Yuhang Fan for helpful discussions. This work was funded in part by the DARPA PTG Program under award HR00112220005, the DARPA ANSR program under award FA8750-23-2-0004, NSF grants #IIS-1943641, #IIS-1956441, #CCF-1837129, and a gift from RelationalAI. GVdB discloses a financial interest in RelationalAI. ZZ is supported by an Amazon Doctoral Student Fellowship.
[1] Kareem Ahmed, Eric Wang, Kai-Wei Chang, and Guy Van den Broeck. Leveraging unlabeled data for entity-relation extraction through probabilistic constraint satisfaction, mar 2021.
[2] Kareem Ahmed, Tao Li, Thy Ton, Quan Guo, Kai-Wei Chang, Parisa Kordjamshidi, Vivek Srikumar, Guy Van den Broeck, and Sameer Singh. Pylon: A pytorch framework for learning with constraints. In Proceedings of the 36th AAAI Conference on Artificial Intelligence (Demo Track), feb 2022.
[3] Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck, and Antonio Vergari. Semantic probabilistic layers for neuro-symbolic learning. In Advances in Neural Information Processing Systems 35 (NeurIPS), dec 2022.
[4] Kareem Ahmed, Eric Wang, Kai-Wei Chang, and Guy Van den Broeck. Neuro-symbolic entropy regularization. In The 38th Conference on Uncertainty in Artificial Intelligence, 2022.
[5] Kareem Ahmed, Kai-Wei Chang, and Guy Van den Broeck. Semantic strengthening of neurosymbolic learning. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS), apr 2023.
[6] Kareem Ahmed, Zhe Zeng, Mathias Niepert, and Guy Van den Broeck. Simple: A gradient estimator for k-subset sampling. In Proceedings of the International Conference on Learning Representations (ICLR), may 2023.
[7] Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann. Support vector machines for multiple-instance learning. In NIPS, 2002.
[8] Ehsan Mohammady Ardehaly and Aron Culotta. Co-training for demographic classification using deep learning from label proportions. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pages 1017-1024. IEEE, 2017.
[9] Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning, 109:719-760, 2020.
[10] Vaishak Belle, Andrea Passerini, and Guy Van den Broeck. Probabilistic inference in hybrid domains by weighted model integration. In Proceedings of IJCAI, pages 2770-2776, 2015.
[11] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, COLT'98, page 92-100, New York, NY, USA, 1998. Association for Computing Machinery. ISBN 1581130570. doi: 10.1145/279943.279962. URL https://doi.org/10.1145/279943.279962.
[12] Gerda Bortsova, Florian Dubost, Silas Ørting, Ioannis Katramados, Laurens Hogeweg, Laura Thomsen, Mathilde Wille, and Marleen de Bruijne. Deep learning from label proportions for emphysema quantification. In Medical Image Computing and Computer Assisted Intervention—MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11, pages 768-776. Springer, 2018.
[13] Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition, 2018.
[14] Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. The Journal of Machine Learning Research, 12:1501-1536, 2011.
[50] Felix X Yu, Krzysztof Choromanski, Sanjiv Kumar, Tony Jebara, and Shih-Fu Chang. On learning from label proportions. arXiv preprint arXiv:1402.5902, 2014.
[51] Zhe Zeng and Guy Van den Broeck. Collapsed inference for bayesian deep learning. arXiv preprint arXiv:2306.09686, 2023.
[52] Zhe Zeng and Guy Van den Broeck. Efficient search-based weighted model integration. Proceedings of UAI, 2019.
[53] Zhe Zeng, Paolo Morettin, Fanqi Yan, Antonio Vergari, and Guy Van den Broeck. Scaling up hybrid probabilistic inference with logical and arithmetic constraints via message passing. In Proceedings of the International Conference of Machine Learning (ICML), 2020.
[54] Zhe Zeng, Paolo Morettin, Fanqi Yan, Antonio Vergari, and Guy Van den Broeck. Probabilistic inference with algebraic constraints: Theoretical limits and practical approximations. Advances in Neural Information Processing Systems, 33:11564-11575, 2020.
[55] Zhe Zeng, Paolo Morettin, Fanqi Yan, Antonio Vergari, and Guy Van den Broeck. Is parameter learning via weighted model integration tractable? In Proceedings of the UAI Workshop on Tractable Probabilistic Modeling (TPM), jul 2021.
[56] Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National science review, 5 (1):44-53, 2018.
[57] Xiaojin Zhu and Andrew B Goldberg. Introduction to semi-supervised learning. Springer Nature, 2022.
[58] Xiaojin Jerry Zhu. Semi-supervised learning literature survey. 2005.
[59] Kaja Zupanc and Jesse Davis. Estimating rule quality for knowledge base completion with the relationship between coverage assumption. In Proceedings of the 2018 World Wide Web Conference, pages 1073-1081, 2018.