A Benchmark Dataset for Learning from Label Proportions
Authors: Anand Paresh Brahmbhatt
Mohith Pokala
Rishi Saket
Aravindan Raghuveer
Venue: Submitted to ICLR 2023 (2023)
Keywords: Learning from Label Proportions
Benchmark Dataset
LLP

Abstract: Learning from label proportions (LLP) has recently emerged as an important technique of weakly supervised learning on aggregated labels. In LLP, a model is trained on groups (a.k.a bags) of feature-vectors and their corresponding label proportions to predict labels for individual feature-vectors. While previous works have developed a variety of techniques for LLP, including novel loss functions, model architectures and their optimization, they typically evaluated their methods on pseudo-synthetically generated LLP training data using common small scale supervised learning datasets by randomly sampling or partitioning their instances into bags.  Despite growing interest in this important task there are no large scale open source LLP benchmarks to compare various approaches. Construction of such a benchmark is hurdled by two challenges a) lack of natural large scale LLP like data, b) large number of mostly artificial methods of forming bags from instance level datasets. 
In this paper we propose LLP-Bench: a large scale LLP benchmark constructed from   the Criteo Kaggle CTR dataset. We do an in-depth, systematic study of the Criteo dataset and propose a methodology to create a  benchmark as a collection of diverse and large scale LLP datasets. We choose the Criteo dataset since it admits multiple natural collections of bags formed by grouping  subsets of its 26 categorical features. We analyze all bag collections obtained through grouping by one or two categorical features, in terms of their bag-level statistics as well as embedding based distance metrics quantifying the geometric separation of bags. We then propose to include in LLP-Bench a few groupings to fairly represent real world bag distributions.
We also measure the performance of state of the art models, loss functions (adapted to LLP) and optimizers on LLP-Bench. We perform a series of ablations and explain the performance of various techniques  on LLP-Bench. To the best of our knowledge LLP-Bench is the first open source benchmark for the LLP task. We hope that the proposed benchmark and the evaluation methodology will be used by ML researchers and practitioners to better understand and hence devise state of art LLP algorithms. 

Under review as a conference paper at ICLR 2023
A BENCHMARK DATASET FOR LEARNING FROM LABEL PROPORTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Learning from label proportions (LLP) has recently emerged an important tech- nique of weakly supervised learning on aggregated labels. In LLP, model is trained on groups (a.ka bags) of feature-vectors and their corresponding label proportions to predict labels for individual feature-vectors. While previous works have developed a variety of techniques for LLP, including novel loss functions, model architectures and their optimization, they typically evaluated their methods pseudo-synte syntheticyyy generated LLP training data using common small scale supervised learning datasets by randomly sampling or partitioning their instances into bags. Despite growing interest in this important task there are large scale open source LLP benchmarks to compare various approaches. Construction of such benchmark is hurdled by two challenges a) lack of natural large scale LLP like data, b) large number of mostly artificial methods of forming bags from in- stance level datasets.
In this paper we propose LLP-Bench: a large scale LLP benchmark constructed from the Criteo Kaggle CTR dataset. We do an in-depth, systematic study of the Criteo dataset and propose methodology to create a benchmark as collection of diverse and large scale LLP datasets. We choose the Criteo dataset since it admits multiple natural collections of bags formed by grouping subsets of its 26 categorical features. We analyze all bag collections obtained through grouping by one or two categorical features, in terms of their bag-level statistics well embedding based distance metrics quantifying the geometric separation of bags. We then propose to include in LLP-Bench few groupings to fairly represent real world bag distributions.
We also measure the performance of state of the art models, loss functions (adapted LLP) and optimizers on LLP-Bench. We perform a series of abla- tions and explain the performance of various techniques on LLP-Bench. To the best of our knowledge LLP-Bench is the first open source benchmark for the LLP task. We hope that the proposed benchmark and the evaluation methodology will used by ML researchers and practitioners to better understand and hence devise state of art I.IP algorithms.
INTRODUCTION
In traditional supervised learning, training data consists of feature-vectors (instances) along with their labels. A model trained using such data is then used during inference to predict the labels of test instances. In recent times, primarily due privacy concerns and relative rarity of high quality supervised data, the weakly supervised framework of learning from label proportions (LLP) has gained importance (Scott & Zhang (2020); [akket et al.) (2022): : O'Brien et al.) (2022)). the training data is available proportion for each bag. The goal is to learn a classification model for predicting the class-labels of individual instances (de Freitas & Kück (205)); Musicant al. (2007))
LLP,, collection of subsets or bags of instances along with the label
Clearly, supervised learning is the special case of LLP when all bags are unit-sized. Unlike super- vised learning however, for which multitude of task-specific real-world datasets are easily avail- able, the same is not true for LLP. While previous works have developed and explored a variety of algorithmic, optimization and deep-neural net based techniques for I.I. (see Sec. 2 for more de-

---

Under review as a conference paper at ICLR 2023
tails), all of them experimentally evaluate their methods on pseudo-syntheeti LLP datasets consisting instances of some supervised learning dataset randomly sampled/partitioned into the different bags. Further, most of the above works use limited scale data, typically small UCI (Dua & Graff 2017 image and social media datasets.
An exception the above is the work of Saket al. (2022) which also uses the Criteo Kaggle CTR (Criteo (2014) and MovieLenss220m (Movielens- 20M Harper & Konstan (2016) which are fairly large in scale, roughly 45 million instances and 20 million instances respectively. In partic- ular, the Criteo dataset has 13 numerical and 26 categorical features whose semantics are undis- closed. Each row an impression and a {0, 13-valued label indicates a click, with in total days of impression-clice cllcck data. The categorical features can be used to create many different bag collections depending on their subset used for grouping, where each choice of the subset's values yields a bag of instances having those feature values. These groupings simulate the typical aggregation scenarios in real-world use-cases, however nowever[Saket et al. (2022) only experimented in a limited manner with one grouping.
In contrast to the above state of affairs, a large number of publicly accessible real-world and large scale supervised-learnin datasets have been studied over the years, whereas there are hardly any datasets which are curated specifically for LLP.
1.1 OUR CONTRIBUTIONS
In this work we address the unavailability of large scale benchmark and standardized evaluation methodologies for LLP. We make the following contributions in this paper towards creating an LLP benchmark building on top of the publicly available Criteo Kaggle CTR dataset.
Bag collections using group-by feature-sets. Typically, for privacy preservation in CTR applica- tions, the impressions are grouped into bags according the values of features such as advertiser-id, product-id, date etc. Thus, we can simulate such aggregations on the Criteo dataset using any sub- set of the categorical feature-set. However, we observe that choosing more than three categorical features likely results in small-sized bags which would be contrary to the goal of large scale LLP datasets. Therefore, our exploration limits the groupings to those obtained using at most two of the categorical features. Below we present the different aspects of our exploration of these groupings. We use a standard preprocessing previously used for training the AutoInt model (Song et al. (2019) on the Criteo dataset at the instance level. More details can be found in Section 4
Analysis, categorization, and filtering groupings. There are 26 categorical features, leading to 26 - (22) = 351 possible groupings using at most two categorical features. Our goal is to curate LLP datasets with not too small or very large bags (as the latter have very weak label supervision), and we always remove bags of size < 50 and those of size > 2500 from these groupings, similar to the work Saket et al. (2022). Post this removal, we identify have at most 500 bags. The remainder 308 groupings are further analyzed in relation to their bag size and label proportion distributions. For each grouping We calculate the threshold bags sizes such that t% of the bags have at most that size, for f = 50, 70, 85, 95. Using normalized vectors of these four values, we use k-Means clustering to the partition the groupings into four subsets typified by increasing bag sizes. More details of these clusters can be found in See [..] Further, modeling the labels as iid Bernoulli with bias given by the average label of the dataset, we compute for each grouping the average of the log likelihoods of the bag label proportions. Using this we also cluster the set of groupings into four subsets indicating how far-from random their label proportions are. Analysis of this characterisation can be found in Section 5.2
outliers those groupings which
In the above removal of bags, a substantial fraction of the original dataset is also removed since there is an abundance of small bags for most groupings. For subsequent analysis involving model training, we further filter out those ones which retain less than 30% of the instances. This is to ensure that only have large-scale LLP bag collections, and we obtain 52 groupings satisfying the retention condition. Details of removals by these filters can be found in Section 4.2
It turns out that these groupings have similar number and average size of bags. We then pro- ceed to estimate the geometric clustering of bags by computing the average inter-bag and intra-bag distances for these groupings. For this use natural definitions of these notions based the
2

---

Under review as a conference paper at ICLR 2023
L distaneee for ease of computation. We also prove certain metric like properties for the inter-bag distances. Details and analysis of these distances is present in Section 5.3
From the above analyzed groupings we select representative subset with diverse statistical proper- ties and include them in LLP-Bench- - our proposed LLP benchmark as a collection of LLP datasets.
LLP model training methodology For each of the 52 grouping create LLP model training and testing setup as follows. We remove the small and large bags as above, and then recreate the instance-level dataset out of the remaining bags. We apply 5-fold split to obtain 5 pairs of train/test splits at the instance-level. For each train/test split, the training bags are recreated using the same grouping the train set. We then train 1-Layer MIP, 2-layer MLP and the AutoInt models using various hyperparameter and optimizer settings, on the training bags and evaluate w.r.t AUC scores on the test set. Details of the training and reported AUC scores are present in Section
Statistically correlating LLP model performance. From the above experimentation we obtain statistics for different groupings based on their bag and label proportion distributions. We calculate the Pearson correlation scores with the model training performance of the LLP data-statistics for the different groupings. The main observations are:
1. A negative correlation with the percentile bag size thresholds indicating that the model performance degrades when the groupings have larger bags. This is intuitively consistent with larger bags having less label supervision (for the same label proportion) that smaller bags.
2. A negative correlation with the label proportion log likelihood. Since bags with label pro- portions deviating from the global label bias have lower likelihood, roughly speaking this means that those groupings where the positive labels are concentrated in fewer bags have better training performance.
3. A positive correlation with the ratio of Average Inter-Bag Distances and Average Intra-Bat Bag Distances. Higher ratio indicates a good separation between the bags. Hence, positive correlation indicates that models perform better when there considerable variation bag distributions.
Some other correlation scores obtained and their interpretation is described in Section 6.2
2 PREVIOUS WORK
The study of LLP is motivated by applications in which only the aggregated labels for groups (hags) of feature vectors are available due to privacy or legal RRúiing (2010) Wojtusiak et al. (20 con- straints or inadequate
costly supervision Dery et al. 20 Chen et al. (2004). It has also been used for several weakly supervised tasks such as IVF prediction (Hernández González Beemández-Goozálezz et al. (018)) and image classification (Bortsova et al. (2018); Orting al. (016)). More recently, LLP has been proposed by O' O'Breen et al. (022) as a framework for privacy preserving conversion prediction.
Several techniques for L.IP have been studied over the years. de Freitas & Küick (2005) Hernández- González et al. (2013) applied trained probabilistic models using Monte-Carlo methods. Subsequent works (Musicant et al. (2007); Rüping (2010) extended supervised learning techniques such neural nets, SVM and k-nearest neighbors to LLP, others adapted clustering based approaches (Chen et al. (2009) Stolpe & Morik (20TT), while Yu et al. (2013) proposed a novel x-SVM method for LLP. Quadrianto et al. (2009) estimated model parameters from label proportions for the exponential gen- erative model and assumptions on label distributions of bags. Their method was further applied by Patrini et al. 2014) for more general models and relaxed distributional assumptions. More recent works have investigated deep neural network based L.IP methods (Bortsova al.) (2018); Arde- haly & Culotta (2017); ; Liu et al. (2019) and techniques using bag combinations (Scott & Zhang (0000): Saket et al. (2022)). Recently, Saket (2021) initiated a thcoretical study of LLP from the computational learning perspective.
All of the previous works in LLP experimentally evaluate their methods on LLP datasets consist- ing of bags randomly created from some real-world supervised learning dataset. In these pseudo- synthetic LLP datasets, instances are randomly sampled/partiiionee into the different bags, where in Patrini et al. (2014) and Saket et al. (2022) this process also clusters Ceature-vectors generate more
3

---

Under review as a conference paper at ICLR 2023
complicated bag distributions. Further, most of the above works use limited scale data, typically small to medium scale UCI datasets (Yu et al. (2013); Patrini et al. (2014); Scott & Zhang (2200)
image datasets (Liu et al. (2119), social media data (Ardehaly & Culotta 2017)) etc. In general, the have been very few natural real-world L.P datasets used for evaluations in previous works. As and Criteo datasets. They used a temporal aggregation into bags for MovieLens-20me while on the Criteo dataset they only used one pair of categorical features to create collection of bags for experimentation.
mentioned earlier earlier[Saket et al. 2022 experimented with Moviel
3 PRELIMINARIES
In our exploration of LLPP shall only consider binary (0, 1) }-valued instance labels.
Notation: X (x(*) R 1m is the dataset of m feature vectors in n-dimensional space with
labels given by Y == (y() E (0.1) 1 We denote by Y == (i(() E [0, 1]]== the corresponding model predictions which are probabilities of the predicted label being 1.
Definition 3.1 (Bag). A bag S [m] consists of feature vectors XB := UiE and with the corresponding label histogram yB == ErEBY The label proportion of the bag is yB/|B).
Definition 3.2 (LLP Dataset). learning from label proportions (LLP) dataset corresponding to collection of bags B = {B is given by {(XB. yB) E B}. The label bias of training datuset is (B Y) = BE UB) /// (EBBB IB).
Since the LLP training dataset lacks instance-level labels we use the dataset label bias to model the label histogram of bag B as the binomial distribution B((B),)) where = ((B, Y). Its log- likelihood LL(B, yB) log f(B yB.p) where f(r, p) == (1)p"(1 p) k is the pdf of B The dataset average bag log- ligeeiill is AvgBagLL = (EBEB LL( yB)) /|3).
3.1 LLP MODEL TRAINING
In order to train model on LLP dataset, apply common loss functions the bag level. In this work we experiment with the binary cross entropy loss Live and the mean-squared error loss Lases which are define for bag B with with average label 2B == ys/ and average label prediction
2n = 0n/lB| as:
Lice (B, zB, 2B) == (2g log &B + (1 zB) log(1 2B)) Lmsee (lnse(B,ye) yB, (B) = lyB 2012. (1)
Note that both the above losses are minimized when OB
In experiments use mini-batch based model training on LLP dataset. A mini-batch here consists of & bags BI...., B& and their corresponding label histograms yn.....: MB. The model predicts all the instances in the bags of the minibatch histograms ÚB,.... LB. The batch-level loss is given by sum of the bag-level losses over the for the mini-batch bags.
aggregated into the predicted label
3.2 BAG-LEVEL DISTANCES
We also analyse the geometric clustering of the feature vectors in bags, by comparing the separation among feature-vectors within bags and their separation across bags. For this, we define natural bag separation.
Definition 3.3 (Bag Separation). For distance d Rd and collection of bags B (B...... B the corresponding separation function is defined as BagSep B, B'.d) == B
We define the M x M matrix BagSepMatrix((3, d) whose (i.j)th element is given by BagSep(B., B,,d)
While BagSep is not a metric since BagSep(B, B) is not necessarily zero, the following lemma shows that it does satisfy the other metric properties. Lemma 3.4. BagSep satisfies non-negativity, symmetry and triangle inequality.
(proved in Appendix A.I A.II]
4

---

Under review as a conference paper at ICLR 2023
We use BagSep compute the average separation between pairs of bags and the average separation within each bag. If the feature-vectors in bags are clustered together and far away from those of other bags, expect the former to be significantly greater than the later.
Definition 3.5 (Inter-Bag Separation for bag). Given B, and metric d on IP, the average inter-bag distance for a bag B E B is defined as InterBagSep(B.. == LB'BB BagSep(B,
For computing the average statistic for the entire dataset we define the following.
Definition 3.6. The mean intra-bag separation of B is defined as MeanlntraBagSep(23 Be BagSep( B, d).
The mean of average inter-bag separation is MeanInterBaggeep(B..r :- 2 EneR lnterBagSep(B.d).
We have the following lemma proved in Appendix A.2
Lemma 3.7. For any bag B. (i) InterBagSep( d) BBagSeppBBBB B,d) 2 1/2 when d is metric, (ii) (iterBagSeppBBd/BBgaggeee (B 2 1/4 when d is the distance.
The following a straightforward corolarry of Lemma 3.7
hatpe///aesssetteooeeeeaaaaaaaaa d) 2 1/2 when metric, (ii) MeanInterBagSeep(1, d) MeanlntraBaggeep(2, / d) 2 1/4 when d is the lg distance.
Corollary 3.8. (i)
We expect this ratio to achieve values substantially less than in adversarial cases. Appendix A.5 provides example of such case. For convenience, for B, MeanInterBaggep at/MMeannttraaageep/p// d) when d = d
InterlntraRatto denote
3.3 CRITEO DATASET
The Criteo CTR dataset (Criteo (2014) has 13 numerical and 26 categorical features and a binary label. Each of the approximately 45 million rows (instances) represents impression (online ad) and the label indicates click. The semantics of all the features is undisclosed and the values of all the categorical features hashed into 32-bits for anonymization. Additionally, the dataset has missing values. We use a preprocessed version of the dataset as done for the AutoInt (Song et al. (2019) model, described and implemented in their provided code"] For convenience we label the numerical and categorical features (in their order of occurrence) N1, N13 and CI,...
The preprocessing applies int(log" (x)) transformation when x > 2 on the numerical feature values a, and we further additively scale so that their values are non-negative integers. The categorical features are encoded as non-negative integers.
4 LLP DATASET: BAG CREATION
We create the LLP dataset by grouping the instances by subsets e C [CI,..., C26) of the categor-
ical columns, where < 2. For each setting of the values of we obtain a bag with instances with those values of Each such grouping yields an LLP datase
Thus, we obtain (36) + 26 = 351 LLP datasets, each referred also as a grouping on e ((e) 2). Note that for any grouping, the set of bags partition the dataset and therefore each instance occurs in exactly one bag.
4.1 CLIPPING GROUPINGS FOR BAG DISTRIBUTION ANALYSIS
As mentioned in Sec. I.I we clip the groupings by discarding all bags of sizes less than 50 and greater than 2500, as our goal is to analyze reasonable LLP datasets. We observe that some group- ings are left with with very few bags or zero bags, while others have a large number of bags. For e.g., the initial grouping on C9 creates only bags and the grouping on C220 creates only 4 bags. Hence, after clipping thse groupings have no bags. The groupings C6, C17, C22, C23 and {C9, C20)
'The is he tat / //g thub. hom./heeprraphhhecc paattsptssssss ng/ Reconmen de rtyst ems/t rree/mast er wewtt//etettee.cooooooo Note that for model training purposes such hags may be created from only the train set portion of the entire dataset
5

---

Under review as a conference paper at ICLR 2023
all contain less than 20 bags. On the other hand, groupings {C10, C16} and {C4, C10) each contain more than x 10 bags.
We compute the mean bag sizes of the clipped groupings. The lowest mean bag size is 62 which we obtain is for the clipped grouping on C23. It manages to retain just one bag after clipping and has 62 instances in that bag. Similarly, the highest mean bag size that obtain is 1292 obtained on clipped grouping on C17. It also retains a single bag after clipping with 1292 instances in that bag. Table 4 provides these statistics for sample of the groupings. Refer Appendix A.8 for statistics of all groupings.
The bag distribution analysis described in Sec. is performed the 308 clipped groupings with least 500 bags remaining.
4.2 FILTERING GROUPINGS FOR MODEL TRAINING
We apply the following filters on the elipped groupings to choose groupings for model training.
Label Information Loss Filter. If the number of bags that remain is less that 10000, discard such groupings to ensure sufficient number training bags. After applying this filter, we are left with 240 groupings.
Instance Information Loss Filter. We drop a grouping if it is left with less than 30% of the original number of instances ( 13.75 10 instances). After applying this filter, groupings. All the groupings in single columns are filtered out as the maximum percentage of instances any of these groupings retains is 21.68% (C4). We finally obtain a set of 52 groupings which satisfy both of the conditions listed above, all of which are emboldened in Table TO in the Appendix.
left with 52
5 BAG DISTRIBUTION ANALYSIS
We perform the bag distribution analysis for all 308 clipped groupings which contain more than 500 bags.
5.1 CHARACTERISING THE DISTRIBUTION OF BAG SIZES
Since have only have a label proportion for each bag. informally speaking, the larger the bag size the lower the amount of label supervision for that bag. The bag sizes for any grouping are characterized by their cumulative distribution function which plots the fraction of bags of size most /. for all f. 2 1. In all the groupings, it is observed that the density of bags drops steeply with the increase in bag size in the histograms of bag sizes. Thus, compute the hag sizes at the 50, 70, 85 and 95 percentile of cumulative distribution plot, for each grouping.
Hence, we can naturally classify the groupings we obtain into long-tailed and short-tailed distribu- tions. Short-tailed distributions have most bags of small size and a very few large sized bags whereas Long-tailed distributions contain many bags of large sizes. Bags of large sizes provide very little label information for a lot of feature level information. Hence, they can be used for learning repre- sentations but are less useful in supervised training.
In order to classify the groupings created into long-tailed and short-tailed, we compute the thresh- old bag sizes at which we attain 50, 70, 85 and 95 percentile of the bags for each clipped grouping. We normalize these values and obtain 4-dimensional vectors for each clipped grouping. Applying k-Means on these vectors we cluster the clipped groupings into clusters. As shown in Table I the mean t- t-percentile bag size, give the same cluster ordering for t = 50, 70, 85, 95. Hence, name the clusters in increasing order of these mean bag sizes as Very Short-tailed, Short-tailed, Long- tailed and Very Long-tailed bag size distributions. Appendix A.9 contains threshold bag sizes and cluster labels based on them for all groupings.
5.2 CHARACTERISING THE DISTRIBUTION OF LABEL HISTOGRAMS
We model the distribution of label histograms in a grouping as a binomial distribution, with bias the label proportion of the grouping. We compute for each grouping its AvgBagLLL value. The

---

Under review as a conference paper at ICLR 2023
Table Mean bag sizes which groupings achieve 50, 70, 85 and percentile in each cluster
Bag Size Dist. Chuster
# Groupings
50 Percentile
Mean bag size:
Mean bag size:
Percentile
85 Percentile
Mean bag size:
Mean bag size:
Percentile
Very Short-toiled
171
107.77
189.05
375.94
905.66
Short suiled
82
155.40
314,83
672.02
1434.77
Long niiled
269.22
1094,90 90
1831.78
Very Long-toited
590.79
1053
1599,57
2145,93
Table 3: Clustering on InterIntraRatio.
Table 2: Clustering on AvgBagLL.
Cluster
# Groupings
Min I.I.
Max I.I.
Cluster
Groupings
Min Ratio
Max
High
26
9..88
.326
Less separated
37
1.02
1.10
Medium
Low
173
-13.98
-2.19
-9.8
16.02
Mediam-
Well-seporated
196
1.12
1.24
1.40
Very Low
36
-38.8
-25.47
For-seporated
1,56
higher the value, the closer the grouping is to having randomly distributed label proportions. Refer to Sec. for the definitions of AvgBagLL and bias of the dataset.
these values and classify Medium, Low and Very Low AvgBagLL. The ranges of AvgBagLL values for each of these clusters listed in Table 2 Appendix A.10 contains the AvgBagLL values well the clusters labels based on them for all groupings.
We perform k-Means
groupings into groupings having High,
5.3 BAG SEPARATION ANALYSIS
As defined in Sec. 3 higher InterIntraRatio indicates bags are clustered. First, obtain an em- bedding of the feature- vectors by training an instance level AutoInt Mode on Criteo dataset and extract it's embedding layer using which transform the instances into this embedding space. We compute the InterIntraRatio and other BagSep quantities for all 308 clipped groupings this embedding space. We then perform k-Means InterIntraRatio classify our groupings into Less- separated, Medium-sseparated Well-separated and Far-separated bag distributions.
The Ranges of InterIntraRatio for each of the clusters are listed in Table 3 Appendix A.1I contains the BagSep quantities for all the groupings along with the clusters they are classified into. Appendix A.4 contains simplification of BagSep computation with & --distancee
5.4 LLP-BENCIE: A REPRESENTATIVE COLLECTION OF LLP DATASETS
provides a representative set of groupings, with their cluster assignments (as per the various analyses above) along with bag-level statisties. Several of these groupings are also used for the LLP model training and analysis presented below. We propose LLP-Bench as collection of LLP datasets with naturally constructed bags which simulate real-world I.I.P use-cases and can be used for evaluating L.P techniques.
Table provides
MODEL PERFORMANCE ON TRAINABLE GROUPINGS
6.1 TRAINING METHODOLOGY
We train on 52 clipped groupings which pass the filters Sec. 4.2 which we create train and test sets follows. For each grouping, recreate the instance-level dataset from the clipped bag- level dataset and the original labels. On this truncated instance- dataset we perform a 5-Fold split, and for each split obtain the training bags dataset by grouping the train set on the same categorical features. The test sets remain at the instance-level.
We train 1-Layer MLP, 2-Layer MLI and the AutoInt model. Preprocessing mentioned in 1.1 ensures that all features have non-negative integers. We use multihot layer whose output is passed
*80% instances used for training and rest for validation. "1-Layer MI.P has 64 hidden units, 2-Layer MI.P has 128 and 64 units in successive layers, tanh activation

---

Under review as a conference paper at ICLR 2023
Table 4: LLP- Groupings. Bold: : Analyzed for model training.
Coll
Col2
No. Bags
After Clipping
Percentage Inst. After
Mean Bag Size
Bag Distribution Clusters
Label Prop Dist.
Inter/Intra Ratio Clusters
Clsuters
Clipping
C3
C8
2486
1.44
265.84
Very Short-waled
High
Lest-separaied
C1
C10
55528
32.44
267.81
Very Short-tailed
High
Medlun-scperated
C19
C22
3893
3.69
434.23
Long-tailed
High
Lest Lest-separoied
C6
C10
46981
31.8
310.24
Short-tailed
Medium
Mediuu--spprrttd
C2
C13
45206
34,87
353.59
Short-tailed
Low
Sedautt-eparated
C7
110
56575
40.52
328.31
Short-tailed
Medium
Per-sepuruted
C10
C13
102841
44.66
199.07
Very Short-taild
Medium
Well-separated
88970
40.43
208.31
Very Short-taledd
Low
Far-separated
C13
1221
2.98
1120.01
Very Long tailed
Very Low
Less seporute
C9
C19
2214
2,6
538.05
Lony tailed
Modium
For seporoted
C7
C10
56575
40.52
328.31
Short-tailed
Medium
Far-sepurated
C0
C11
3221
5.71
812.21
Long-tailed
Very Low
Per-sepprated
C7
C20
30420
31.85
479.96
Lang-tailed
Low
Well-separated
Table 5: AUC Scores of MLP classifiers obtained on LLP-Bench Groupings
Coll
Col2
ET
E2
E3
E4
E5
E6
E7
E8
C7
C70
73.6420.05
73.6720.05
73.640.06
73.6120.04
65.0320.11
63.43a0.1
65.55a0.22
63.9780.23
C6
C10
72 228920.04
72.920.05
72.7950.04
72.120005
65.0120.08
62.8810.31
65,4920.16
63,8420.44
C2
C13
75. 03
75.23+0.03
75.06=0.03
75.08=0.04
68.04=0.07
66.7540.13
68.5740.21
67.21+0.
C7
C10
73.4920.03
73.5a0.02
73.3520.03
73.31a0.03
64.3220.16
62.66:00117
65,2340.27
63.2+0.19
C70
C75
75. 75.004
75.75=0.04
75.58=0.05
75.5620.04
70.07.0.05
68.8340.1
70.6340.08
69.5140.17
C21
76. 79.0990:04
76.91.0003
76.6420.04
76.6720.06
70.8720.08
69.76a0.18
71.48a0.05
70.38a0.13 13
C7
C10
73,4950.03
73.320.02
73.3520.03
73.31=0.03
16
62. 02.00110.17
63,23:0027
63 23..2.01
C7
C20
74.43+0.06
74.46+0.06
74.28+0.05
74.28+0.08
64.65+0.12
63.09-0.2
65.21+0.2
63.6440.1
to the MLP models. AutoInt has a 16-dimensional trainable embedding layer for each feature 5 Output layer has one unit with sigmoid activation.
We perform minibatch gradient descent by sampling minibatches of 8 bags each, and the model predictions are aggregated into the predicted label proportions of the bags. The minibatch loss is the sum over the hags of either the bag-level Lise or Live as described in Sec. [3.1 We then back- propagate this loss and update weights using the optimizer either Adam or SGD. The specifications of experiments are in T Using Adam, we train for 50 epochs with learning rate of Le- 5 for initial 15 epochs and le for the rest. Using SGD, we train for 300 epochs with constant learning rate of le - 5.
We use test AUC scores to qualify the tractability of an LLP dataset. For MLP trained using SGD use the maximum reported AUC score. On the other hand, AutoInt has an increasing trend for both optimizers but it is not (even locally) monotonic, hence for it we use the average of last 5 epochs of training. The AUC score (averaged over the 5 splits) for trainable groupings in LLP-Bench and the various experiments (see in conjunction with Table 7 are listed in S and Table 6 AUC score for all trainable groupings are listed in Table Appendix A.7 contains details of instance-level training which we perform for completeness.
and Adam
Table 6: AUC Scores of AutoInt obtained LLP-Bench Groupings
Table 7: Experiment Legend
Expt.
Model
Opt.
Less
Coll
Col2
E9
E10
E11
E2
L-Layer
Laver MLP
Adua
Adam
MSE
BCE
C
C10
70.420.17
70.2520.18
61.9820.38 38
F3
2-Layer MLP
Adam
BCE
C6
C10
69 84:0.22
69.75=0.22
63.3+1.28
2-Layer
Adum
C2
C13
68 23
68.88+0.42
64.51=1.74
F3
I-Layer MIP
SGD
BCE
C7
C10
110.34
68.77=0.28
62.31=2.02
T-Layer MLP
SGD
MSE
C10
C15
11
71.93:0.02
69.52s0.19
2-Layer MI
SGD
RCF
C7
C21
72 $20.11
72.8420.16
71.49=101
E8
2-Layer MLP
SGD
MSE
C7
C10
69. 110.34
68.77=0.28
62.31=2.00 02
Autolot
Adam
C7
C20
07.9820.82
67.19=0.24
63.00200.8 89
E10
E11
Autolat
Autoiat
5GD
Aduu
MSE
BCE
'Embedding Layer for categorical features, numerical features multiplied by trainable 16-dim vector
8

---

Under review as a conference paper at ICLR 2023
6.2 CORRELATION OF DATASET CHARACTERISTICE WITH AUC SCORES
We compute the Pearson correlation between the AUC scores and the bag level statistics computed in Sec. [ These are visualised Fig. | Some observations from these scores are:
1. Positive correlation with number of bags and number of instances. This is expected as each bag adds to the label information and each instance adds to feature information available the classifier.
2. Negative correlation with the mean bag size and percentile bag size thresholds. This intuitively consistent with larger bags having less label supervision (for the same label proportion) than smaller bags, and typically the model performance would degrade when the groupings have larger bags.
3. Negative correlation with the label proportion log likelihood. A lower log likelihood in- dicates that labels proportions in the dataset are highly skewed. This means that those groupings where the positive labels are concentrated in fewer bags have better training per- formance. In this case, the bag grouping features provide significant supervision which the model can leverage. We can infer the same from highly positive correlation with standard deviation of label proportion.
4. Positive correlation with the InterIntraRatio. Higher ratio indicates good separation between the bags in input space. Hence, positive correlation indicates that models perform better when bags are separable in input space. This can be explained as follows
The distribution of label proportions are skewed the maximum log- wwwlliikiihood hibited is -3.26. Hence, substantial label information is present at the bag-level.
If the InterIntraRatio is high, much of the discriminative information at the bag-level lies in the input space itself. If the InterIntraRatio is low, most this information is in some latent space that the model needs to learn.
Std. of Label Prop.
No. Begs
0.75
No. Inst.
0.50
InterintraRatio
MeaninterBaggSp
0.25
MeanintraBagSep
0.00
Avg LL Label Hist.
Mean Bag Size
Bag Size: Percentile
-0.50
Bag Size: Percentile
Bag Percentile
-0.75
Bag Percentile
E10
E11
Figure Correlation Hcatmap
7 CONCLUSION
Our work conducts an in-depth study of the Criteo CTR dataset for use natural LLP dataset, and provides I.LP-Bench: collection of L.LP datasets from from the Criteo dataset as benchmark for evaluating LLP techniques. In this process, on at most two categorical features, based on their distribution of bags as well as label proportions. We also adopt an evaluation methodology using which train various models appropriately filtered subset of groupings and demonstrate (as well explain) correlation the model performance with the computed statistics.
work analyzes bag collections given by grouping
We believe our work addresses to a great extent the current lack of natural LLP benchmarks, and provides L.P--Bede using which I.IP techniques can be systematically evaluated.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ehsan Mohammady Ardehaly and Aron Culotta. Co-training for demographic classification using deep learning from label proportions. In ICDM, pp. 1017-1024, 2017.
Gerda Bortsova, Florian Dubost, Silas N. Ørting, Ioannis Katramados, Laurens Hogeweg, Laura H. Thomsen, Mathilde M. W. Wille, and Marleen de Bruijne. Deep learning from label proportions for emphysema quantification. In MICCAI, volume 11071 of Lecture Notes in Computer Science, pp. 768-776. Springer, 2018.
http:////wwwww....//.ooo/o/ 1007 086601
Lei Chen, Zheng Huang, and Raghu Ramakrishnan. Cost-based labeling of groups of mass spectra. In Proceedings of the 2004 ACM SIGMOD international conference on Management of data, pp. 167-178, 2004.
Shuo Chen, Bin Liu, Mingjie Qian, and Changshui Zhang. Kernel k-means based framework for aggregate outputs classification. In Yücel Saygin, Jeffrey Xu Yu, Hillol Kargupta, Wei Wang, Sanjay Ranka, Philip S. Yu, and Xindong Wu (eds.), ICDM, pp.
2009.
Criteo. Kaggle display advertising challenge dataset, 2014. htth:t/t://aa er criteo. com/ http://aaaaa.iiipppaaaaaeeeeeaae
Nando de Freitas and Hendrik Kückk. Learning about individuals from group statisties. In UAI, pp. 332-339, 2005.
L. M. Dery, B. Nachman, F. Rubbo, and A. Schwartzman. Weakly supervised classification in high energy physics. Journal of High Energy Physics, 2017(5)11-11, 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL WRL/http://acchive ics. www.cc.iee. edumml
F. Maxwell Harper and Joseph A. Konstan. The movielens datasets: History and context. ACM Trans. Interact. Intell. Syst, 5(4):19:1-19:19, 2016. doi: 10.1145/2827872. URL https://
Jerónimo Hernández-Gonzálee Iñaki Inza, and José Antonio Lozano. Learning bayesian network classifiers from label proportions. Puttern Recognit, 46(12) 2013.
Jerónimo Hernández-Gonzáiee Iñaki Inza, Lorena Crisol-Ortíz, María A. Guembe, María Iñarra, and Jose A Lozano. Fitting the data from embryo implantation prediction: Learning from label proportions. Statistical methods in medical research, 27(4) 2018.
Jiabin Liu, Bo Wang, Zhiquan Qi, Yingjie Tian, and Yong Shi. Learning from label proportions with generative adversarial networks. In NeurlPS, pp. 7167-7177,
Movielens-20M. Movielens 20m dataset.
URL httppe:/ggroopleessoorr///aaaaae
mmvveeeess/20m
David R. Musicant, Janara M. Christensen, and Jamie F. Olson. Supervised learning by training on aggregate outputs. In ICDM, pp. 252-261. IEEE Computer Society, 2007.
Conor O'Brien, Arvind Thiagarajan, Sourav Das, Rafael Barreto, Chetan Verma, Tim Hsu, James Neufeld, and Jonathan Hunt. Challenges and approaches to privacy preserving post-click con- version prediction. CoRR, abs/2201.11666 2022. URL htttps//aaax.v.orrgabb///22 12666
Silas Nyboe Ørting, Jens Petersen, Mathilde Wille, Laura Thomsen, and Marleen de Bruijne. Quan- tifying emphysema extent from weakly labeled scans of the lungs using label proportions learn- ing. In The Sixth International Workshop on Pulmonary Image Analysis, pp. 31 31-42,22 2016.
Giorgio Patrini, Richard Nock. Tibério Cactano, and Paul Rivera. (almost) no label no cry. In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger (eds.), Advances in Neural Information Processing Systems, pp. 190-198, 2014.
Novi Quadrianto, Alexander J. Smola, Tibério S. Cactano, and Quoe V. Le. Estimating labels from label proportions. J. Mach. Learn. Res., 10:2349-223 23744, 2009.
10

---

Under review as a conference paper at ICLR 2023
Stefan Rüping. SVM classifier estimation from group probabilities. In Johannes Fürmkranz and Thorsten Joachims (eds.), ICML. pp. 911-918, 2010.
Rishi Saket. Learnability of linear thresholds from label proportions. In NeurIPS, pp. 6555-6566, 2021.
Rishi Saket, Aravindan Raghuveer, and Balaraman Ravindran. On combining bags to better learn from label proportions. In A/STATS, volume 151 of Proceedings of Machine Learning Re- search, pp. 5913-5927. PMLR, 2022.
press/v111/
saket22a. html
Clayton Scott and Jianxin Zhang. Learning from label proportions: A mutual contamination frame- work. NeurIPS, 2020.
Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang. Autoint: Automatic feature interaction learning via self-attentive neural networks. In CIKM, 2019.
Marco Stolpe and Katharina Morik. Learning from label proportions by optimizing cluster model lection. In Dimitrios Gunopulos, Thomas Hofmann, Donato Malerba, and Michalis Vazirgiannis (eds.), ECML PKDD Proceedings, Part III. volume 6913, pp. 349-364. Springer, 2011.
J. Wojtusiak, K. Irvin, A. Birerdine, and A. V. Baranova. Using published medical results and non- homogenous data rule learning. In Proc. International Conference on Machine Learning and Applications and Workshops, volume 2, pp. 84-89. IEEE, 2011.
Felix X. Yu, Dong Liu, Sanjiv Kumar, Tony Jebara, and Shih-Fu Chang. xSVM for learning with label proportions. In ICML, volume 28 of JMLR Workshop and Conference Proceedings, pp. 504-512, 2013.
Appendix for:
A Benchmark Dataset for Learning from Label Proportions
A PROOFS OF LEMMAS AND ALGORIHTMM
A.1 PROOF OF LEMMA 2.4
Proof. From Def. 3.3 the non-negativity and symmetry properties are obvious.
Triangle Inequality let B1, B2, B3 E B, and we use the following notation for convenience:
B {aralé i [n]). B2 [uylū E [m]]. B3 = {zulk E [4]. As d is a metric, we know that for all i [m].j E [m] and & E [[], d(z,, 2x) S d(æ,, My + d(y), Hence,
>ymm d(z My)
> d(y; 2k)
d(r,, 2x)
dd 2k)
D 5= d(
n
nun
Sk= yi d(zs, =k))
d(z,, v1)
> d(ys zx)
ml BagSep( B1 B3 d) < BagSep( B, B2, d) BagSep B3, d)
In
nmm
11

---

Under review as a conference paper at ICLR 2023
A.2 PROOF OF LEMMA 2.7
Proof. Let B E B. Using triangle inequality and symmetry from Lemma 2.4:
VB' B, BB B,d) < BagSep( B, B¹, d) + BagSep(B's B,, B,d) VB' E B, BagSep (B, BB, < 2BagSep B² B, d)
EBe " Bap B', B,
BagSep( B, B.d) _
BagSep( B, B, 2InterBagSep(B.dl
InterBagSep( B, d) /BagSep(B, B, 2 1/2
The squared euclidean distance is not a metric it follows all properties other than the triangle inequality. Hence, we show the following
Lemma A.1. For any a, b E R" Žlla bll2 < ||a| + 1
Theorem A.2. Given X, Y and B, for any Bi, B2, B3 E B
(BagSep(B1 B1, B3, ( I BagSep(B1. B2, Lg) + BagSep B2, B3, 2g
Proof. Follows by replacing triangle inequality in Lemma 2.4 with inequality in Lemma A.I
L
Corollary A.3. InterBagSep(B, B, BB)g/Baaep(H [B B, 2 1/4
Proof. Follows by replacing inequality in proof of Lemma 2.7 with incquality in Theorem A.2
A.3 PROOF OF COROLLARY 2.8
Proof. Given X, Y and and metric in R Starting with inequality in Lemma 2.7
wB B, BagSep B, B,d) 2InterBagSep(B,, B,
BagsBa B, B, B,d) < 2n
BagSep(B, B, d)
InterBagSep (B,))
B
MeanInterBagSep B, d)/ /MennlntraBaagepp (B, d) 2 1/2
Starting with inequality for didittance in Lemma 2.7, we get
MeanInterBaggeep("
(Beeeee/ Et/MMeanlntraaagSepPP M 2 1/2
A.4 BAG DISTANCE RESULTS USING SQUARED EUCLIDEAN DISTANCE
We use the squared euclidean distance to compute the bag distances it makes the computation faster. Algorithm is used to compute the Bag Separation for any general metric
Theorem A.4. Assuming the Bags to be disjoint, the running time of Algorithm is O(m2n) where m is the number of examples and the dimension the imput space.
Proof. Runtime = EB, E B1 B2In EB, ez B1 EB B2 n m2n
Now, this computation can be simplified due to the following lemma.
Lemma A.5. For any B, B' E B, Ba B', AvgSqNorm(B) + AvgSqNorm(B)) 2DotProduc((Mean( BB) Mean(B')
12

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Compute Bag Separation of dataset
Data: Set bags B, metric on R
Result: BagSepMatrix(B,.d
BagSepMatrix + [0///]1
for B1 E B do
for B2 E B do
for E B₁ do
for e B2 do BagSepMatrix(21, B3] +- BagSepMatrix(3,,55 B2] + d(x((), x(11
end
end
BagSepMatrix(B,, B₂] +- BagSepMatrix(B,, BB) end end
Proof. Let B = {2.li [n]], B' {yylı E [m]}
1
(",
j=1
[ 12l12
-
BagSep [B BB, B, ())
D "== (x Uj)
Di=n -m
f=1
mn
1 >
2
BagSep( [B B", ())
> b.
(> T; Fiy E /!
mn
Algorithm 2 is used to compute the Bag Separation for squared euclidean distance.
Algorithm 2: Compute Bag Separation with squared euclidean distance
Data: Set bags
Result: BagSepMatrix(B,
BagSepMatrix < [0] [1113]]
AvgSqNormm <- [0]
BagMeans + [0]
for E B do
for i E B do
AvgSqNorm(1R B) +- AvgSqNorm(13 (B) + [1
BagMeans(B) (B) + BagMeans(B) + (+)
end
AvgSaNorm(B) B) + (igSsNorm(BB)) (B)
BagMeans(B) (B) + (B
end
for
Bi E B do
for B3 B do
BagSepMatrix(BB, B2] +
AvgSqNorm| [i] AvgSqNorm([B) 2DDoProduuct(aggMMan B1), Bagmeans [2])
end
end
Theorem A.6. Assuming the Bags to be disjoint, the running time of Algorithm is O + n B 2) where m is the number of examples and n is the dimension of the input space.
Proof. Runtime Enea B|n EB, EB EBE (1 + n)
|B |B]2
13

---

Under review as a conference paper at ICLR 2023
A.5 ADVERSARIAL EXAMPLE OF BAGS WITH RATIO OF MEAN INTER TO INTRA BAG SEPARATION AS 1/2
Consider X = (0(1),(1) ,0))) which lie on straight line. The distances are follows:
d(x!t), ,x(1)) = d1
d(s!l), x(())) = d2
. ((((1)xx(0)) d1 d2
We have two bags B₁ {x(1), (0) and B2 = (x(21). The Intra-bag separations for both of them are as follows:
BagSep(B1, B1, B1, d) = (d(( x(1) + ((0)) + d(xx1)), x(11) + d(xl3), x()))) =
1 [d1 + d2)
BagSep( B2, B2, d) =0
Hence, MeanIntraBagSep(35 (B,d) [d] d2)- Now, the bag separation between the bags is as follows:
BagSep(Bi, B2, d) = Tx2 (d(z(1) x 22) + ,(x((), = [d + d2)
InterBagSep( d) 2-1 (BagSep( B1, B2, d)) _ + dz)
-
InterBagSep( = 2-1 (BagSep( (22, B1, d)) = (d1 + d2)
Hence, MeanlnterBaggee((.. (B, d) = (d, + d2).
Hence, MeanInterBagSep(25 B, d) M (B, d) = 1/2
A.6 LLP MODEL TRAINING RESULTS
This table contains the AUC scores of all the experiments in Table 7 Each value represents the mean AUC score in percentage across 5 splits which are created as mentioned in 6.16 The error the standard deviation of mean AUC scores across these 5 splits.
Table 8: AUC Scores obtained after different training configurations for all groupings
Call
Galz
Ri
R3
R3
E4
E5
E6
E7
E8
E9
E10
E11
C7
748810.04
74.9520:33
74,7620,04
74,782003
638800
56
25
69
650020
Cf
Cla
73.644005
78,6740.05
73660006
75.61-0004
65.0++011 11
63.440
55-55
63.97-023
70.40.17
70
61,984034 38
C2
15,6320:02
75,6380:02
75,4230.04
75.4000:33
68.440
6734.0
89 13
675920
N07680
104220016
67,21100
C2
C10
74.45+0.04
74.4540:04
74 340.4
74.2940.55
67.98+0.1
668540114
68.57+011 14
67
20.3340.3
20.274013
66 K840UR
Q
CH
73,5610.02
75.3810:02
12.420002
75. 424002
612000
87 870920
68 08.9950111 15
67822011
69.550024
69.822041
65 65,8620411
C2
C13
75.23:0.04
75.2340:03
05.00..03
04 07
7540 13
67
23
6840.42
6
C3
76,9340.02
16.5500002
76. 76000
70.72002
7083E0.1
59
06
71 310111112
20.220114
12.480.9
12720.11
11411
C
C10
15.0840.04
75.8740:04
75.6480004
75 62
700150.11
69
02202 12
N0.495005
49:47 14
72.436010
72.Me81
70.4440139
C3
Cli
77,0910.2
76
70.792000
703883005
71 09
10.180017
12.010023
12.889010
C3
C33
16.0540004
76.9240.5
76.684003
76.614003
7055700.6
47:00.0.09
71 10.0600.0 03
69
85
72.236033
72.45
74.74
C4
70,9820022
76,9940.05
76,7540004
70.7320022
71.1120.06
700260008
71 71.5720166
13.71
12.0
71 23gl1
C4
C10
76.3140.03
76.11
060000003
06
70
6-0009
71 06
09000000
72.72
726660006
69 944414
C4
CH
76.7720.022
76.7620:02
70.4780:02
700990009
69
71 31000
70.0450
72.5780288
12.5865.19 19
71 148411
C4
76.640.02
76.9940.1
06
70.44-011
26+0.00
09
85+0.07
72.32
71.35+0.91
C4
C13
75.51=0002
75.4740:03
74.90007
743550006
66.6683005
67.09290099
690020012 12
23
71.132003
714126123
66 66.886666
C6
73.56+0.04
73.61-6:04
73.41+0.04
73.4+0.04
648++0.5
62.87+0225
65 65.57-0 17
63.81e0 12
68 18.23+0.3 33
68 68.26+0.48
62,62 1.64
CH
C10
72.89+0.04
72.0+0.05
72,79 72.790
+0.0
65.01+0.08
62.8860331
65 16
(3.8440444
(0.8461124
69.75401
63.541.222
C7
C8
74.55+0.02
74.98+0.03
7..41=0.02
+4.01
67:17: 16
63.35+0 24
66 12:0 17
64.27e0
68.87e0.3
68 37
64 64.8760.54
C7
C10
73.4940.03
73.5e0.02
73.3540.03
73.31
64.32H0.10 16
62.66.0.11
65 27
69.1160
68.77e0
62.51+2.02
C7
C12
76.8720.04
76,8810.03
76.020006
06.600003
70.74:0.0
60 6300
71.37a0
70.020012
72/4860
72.740011
70.8550442
C7
C14
74.66+0.05
74.68+0.05
74, 44.46+0.0
74 74.48+0.06
66.22+0.14
65 1810 11
67 07.52++01 12
65 (5.84+011 12
68 9+0.56
68 14.7440.51
65.31 $1+0..9
C7
C35
76.8910.03
76.8910.03
76.6720022
76.6400004
01000
71.55001
N0350008
72.52006
22.51:021
70.61029
C7
C16
76.8900.05
76.0100.3
66.660003
06.6640.05
7098+005
60
7144+008
72.61+011 31
72.75-0 14
70.9540.1 18
16,5610.03
76.5500:3
7631
70.100
893100007
00352005
197320013
71.870016 16
71 1620000
70.0800.4
C7
C30
74.4840.06
74.4660.06
74 38+0.05
04.3800.0
64.65.011 12
65.0000
63.6400
67 97.880033 83
67. 70H0.44
63.0600.0 89
C7
221
16.8910.04
1691100.3
16.0920.04
76.6720006
708720.08
59 79.76a0 18
71 48
N0.8800
72.520.11
12.5120116
71 11.491100
C7
3810.04
3540.03
110.03
70
78
70 74 12
78
72 360
18:0
910
C7
C20
75.4540.02
75.4240.04
78. 1.1000
75.11
09.6483007
N102001
71 23000
71.
04.2/4444
C10
C12
75.9110.04
75.940.04
75.6740.04
75 6610000
70.0410.07
69 18
40.040000
69
69.440.11
72.436033
22.31
70 70.52.0 22
C70
23.3943.04
73.3320.04
66.77
3620
57 278009
09.64230119
05 05.08337773
C19
C73
75.7640
75.7540.04
75. 75..84
75.5640.04
70
68 68.330111 11
00
69.51-0117
71.880011
69.52+0.11 71.5840022
C70
C76
70.00320.002
70.0340.05
76,003
75.7740:04
70120009
1560.06
70.920006
72.4820113
72.4420111
70. 74.3445555
14

---

Under review as a conference paper at ICLR 2023
C70
C77
73.18
13.12
63.6680.33
62.33
64.55g0.14
02.7869
68.466019
(9.23603
020221.2
15,2010:03
15.28a0:03
73.1220003
75.0920055
05.772045
C10
C20
73.4150044
73,460.04
73.3640004
73.2940.05
65,0700.19 19
63.
65 (5-7560.18 18
64.13
69 3565555
69.960229
62.966666
C27
76,0220003
76,0120:04
78,775005
75-764004
70.T
89 19
70.72000
09
72 72.880008
12:00000
75,4H0.0
75.3740:01
75.1840.02
75
60-72H0:04
68.8100
72.35+0
71,956011
59,52
C19
C36
14,7340.05
74.7240.05
74,08
69 090050011
Ed 42
15
71 11.0660226
70.7860
68 68.0560199
70.72+0.09
71 71.37
96
71.470011
CIT
CT5
76.8510.02
768838001
76.6100.11
76.6630.44
70,7610:08
6880008
71.3720
20.10008
72.3920
72.5200.1
71,12042
CFI
76,87+0.33 C16
76.86+0.44
76.600.02
7K 59+0.03
708
65
71. 31.90+000
70.08+0.66
72,40+0.3
72.5
71,53+0.5
76.5520:66
36.3520:0
70.330.7
C1T
C21
77.0+0.02
76.99-0.01
76 73
76, 02.72-0.02
70.02+0.09
60 10.73+010
71 48.00
70 38e0
72.71
72. 22.78e0. 18
72.04+1 1.54
CIT
C2F
16.3130:03
76.2720:03
26,01
75
70.152007
090090008
77
162008
72.0780.1
12.1520.19
70.9880.44
C26
75,89:0:03
15.6600:04
75 05.680.05
58:0.04
60
68.6760
70.3760 11
60 056000
71.2460 16
714260
69 39
C12
CB
76.8320.04
76.2240.05
76.5520.05
76.3420:04
70440005
89.4120006
71008008
19.71201
72.0100.16
12.884007
71 71,2220422
71.0000
C13
C76
70,7130002
10.7120.02
16.4220055
10.4220:02
7U3123007
09.2200009
71 110000114
09 15
12.368021
12.2120112
C13
C78
16,3580005
76.3240004
76.11 02
10000
89:9
82001
30
1780.1
71.880011 13
11.8000 19
6K 5820066
C21
16.8340:02
76.5720.02
09
15
8250007
72.4660
21.580.25
C13
C27
76, 1340.5
16,0M20:02
25.81
75 830002
89 00.8900.22
58.9 88.9120077
20
69 2910
71 11 11
71.7740 13
35.354
C26
15.96210022
75,9240.05
75.08
75.5920000
0913500077
65.4750.15
29:290005
69.1760119
71.5360.1
71.3420.1
A.7 INSTANCE-LEVEL MODEL TRAINING RESULTS
We perform all the experiments mentioned in 7 on instance-level data. The process remains similar for different configurations as described in Sec. 6.1 We perform a train-test spilt 80:20 on the dataset. We then train using instance level mini-batch gradient descent for the same number of epochs, using the same optimizer, model, learning rate schedule and the instance- variant of the loss function. We again report the AUC scores as described in Sec. 6.1
Table 9: AUC scores obtained by instance level training Criteo
Experiment
Model
Optimizer
Function
Soore
I Layer MLP
Adam
BCB
79.23
MLP
2 Layer MLP
Adam
BCB
30.1
8
Lagerr MLP
Adam
MSE
79.94
Layerr MLL
SGD
MSB
80.54
MLP
SGD
79.17
Layer MLP
SGD
MSB
80.56
A9
Autoint
Adam
BCB
30.66
A11
Ausolat
Adam
MSB
79.02
15

---

Under review as a conference paper at ICLR 2023
A.8 BAG LEVEL STATISTICS
The bag level statistics for all 349 groupings is as follows. The Groupings which are emboldened pass our filters and used for training.
Table 10: Bag Level Statistics of all the Groupings (Emboldened: : Used for Training)
Bags Created
No. of hags left after clipping
Percenttgge Tnst. after elipping
Mean Bag size
Standard Deviation
of Bag cloes
CJ
1443
1261
0.7
256.16
386.61
175781
39052
216.17
316,89
C4
128509
38802
21.88
256.11
35919
C7
11930
7839
12.30
724,69
636.25
531
0.34
295.34
430.38
C10
41224
20252
17.01
184,90 92
482.64
CH
5160
2519
K32.04
683.76
C12
174R35
39444
18.79
218.32
31882
CH
3175
1221
2,98
1120.01
684.56
C33
11254
7.25
312.87
569
C16
169206
40109
20.06
229.24 24
$34.09
CT8
1601
2625
3.32
580,75
60135
C19
2017
1300
1.99
701.55
651
C21
172322
39781
19:28
222.14
5221
56456
21694
309.6 85
431.33
C26
13356
17702
11.57
299.58
604.1
[4H1229
11999
9.12
348.45
463.68
19800966
47251
22,56
216.96
318.0
CJ
C1
1807068
58010
26.87
238.13
328.8
1852
4294
2:21
243.02
57048
C1
9950
2075
1,53
337,95
461.16
7244000
55437
37.91
310.69
417.54
6477
5828
2,95
232.16
12
C
Cy
3001
1501
0.56
261.47
391.39
C10
1034267
55528
32,44
267.81
374.19
CFT
421601
38117
27.5
330.67
CJ
C12
1998608
48523
23.07
217.95
520.44
C13
334925
32966
397.45
448.48
CF
CN
16449
2503
1.87
341.7
462.37
C
C15
614550
39530
26.63
308.8
418.5
19929499
32671
25.52
222.14
321.99
C1
C17
12199
2245
1.7
346.25
460.77
C18
365199
15570
18:07
324,03
43539
C1
[4490]
10476
7.18
314.3
43
220
3772
1732
1.25
332.39
40033
3003740
30168
218.63
3206
7264
1895
1.18
285.31
42219
13043
2353
1.76
343.42
46582
10022/44
44818
24,81
253.74
3593
225
33208
3125
2,34
342.41
458.1 17
C20
HIGNKZ
34315
18.57
245.35
549.66
22
613167
10900
1963
219.98
321.4
C2
588748
446912
24.03
246.51
351.32
41463
5606
400.34
C2
48:34
1907
2,85
684.3
653.76
C2
444591
78261
48.74
285.51
377.68
730220
7834
6.42
373.27
480122
C2
C10
945374
75614
41.23
249.98
345.32
221445
53308
336.11
431.53
C2
649337
41375
1937
221.25
322.15
C2
C13
162788
45206
34,87
353.59
446.07
2102
850
1,33
821.67
693.231
C2
C15
11300
6516
7.28
512.53
96
C2
661655]
43198
21.56
229.8
331.33
4855
2646
3.95
681.16
614
C2
1631
2624
3,32
580.39
80781
4M411
T0064
7,49
350.45
C2
C21
6583009
42128
20.57
225.84
$25.78
1337
1579
1,95
367.74
610.22
C2
3080
2525
3.49
633.74
611.09
22
221
3411266
31123
19.15
282.09
5978
C25
5228
1275
1.52
475.44
364.33
Q2
C25
247348
24100
14:25
271.12
381.3
2470KE3
55817
26.21
215.25
$10.3
11141/9
43935
20.55
217.81
51881
#440001
43641
20.62
216.6
$1698
7358757
80788
37,88
214.92
310.29
1498370
45382
21.51
217.78
319.66
C3
CV
309182
40293
19.2
218.45
320
76999499
71467
205.27
300/33
C3
581612S
73331
35.78
223.64
324.1
C3
C12
187150
19997
18.82
215.65
316.49
5313650
70587
225.86
335.66
C3
C14
413784
40592
19.43
219.46
322.47
C15
1076734
15427
26.22
248.23
36396
16

---

Under review as a conference paper at ICLR 2023
C16
48449
21:91
207.32
303.14
I0N2770
45966
22.1
220.88
325.63
2
736224
43469
22.2
234.16
546.66
C
183084
41788
1999
222,47
327.44
346464
39600)
18.8
217.58
516.56
C2
305311
42557
19.54
210.52
309.43
561157
41869
218.46
523
C3
898254
44976
21:11
218.17
318.12
C#
G34
329746
54467
27.12
228.55 34
529,83
352879
40296
218.52
318.79
C3
C26
406906
53753
26.8
228.57
420.4
C7
955570
51305
26.41
235,97
336.72
C4
649962
53352
27.1
232.85
337.09
7408443
103144
43.82
194.74
275.38
1264446
54197
27.55
233.03
33038
C
297748
42003
22.99
250.9
353:24
S0MIN08
86205
36,08
191.88
273.47
50229443
90009
212.1
299.09
C1
315809
55818
26.26
215.67
51109
5060557
93015
316.15
306-73
C
311943
47208
24.95
282.56
342.7
C4
C15
969150
56468
31,06
252.18
36038
C70
2301290
49211
23.8
221.05
323.03
C4
825745 C17
56219
28.55
232.81
554.87
C4
C18
02W441
47904
26.35
252.14
3008
+4
G79
687309
44217
23.23
240.81
343.11
C4
C20
262150
43501
23,57
246.24
$47.36
342560
54642
215.17
509
G4
C22
436550
46518
24.77
244.09
$48.14
C4
686712
54106
27.73
234.96
333.37
177182
45075
23.31
247.31
35037
C25
250638
43224
23.8
252.44
356 14
C26
325522
1965T
26.9
247.37
35163
C5
2260
579
0.51
401.17
311.77
43
190810
37384
29.42
360.98
4632
3870
2186
1.44
265.61
40535
C10
480175
42678
27.35
293.75
396.83
C1
159530
22921
19.1
381,99
12
1116848
4
21,39
218.6
318.88
CB
122621
18931
16.14
390.76
494.88
C14
1934
780
0.68
399.88
504.78
C3
CJ3
135816
25697
19.44
346.74
454.78
C5
C16
1096716
47775
23.45
224.99
23
C17
2019
649
0,37
403.07
C3
CF8
128596
E5061
12.02
366
472.77
C19
SEMISS
6542
5,04
364.25
467.44
C31
11142267
46016
22.18
220.94
331
65
C23
2904
698
0.62
464.87
524.76
$17424
36088
268.5
375.4
C5
C25
1066
0.91
390.25
492.66
C5
C26
401151
28221
16001
260.12
365.26
82996
38449
35.84
427.51
492.33
C6
C8
4482
1006
0.82
371.25
477.59
C10
2359400
46981
31.8
310.34
411J03
1813
18249
21.90
350.51
57581
86
C2
840420
44683
21.2
217.85
3189
35706
13565
17.92
601.75
595.1
C6
C13
70477
21774
1925
413.78
5094
C10
30R500
47935
23.32
221.89
32191
31-088
10996
10.79
449.9
531.99
C6
C19
14895
6625
6,02
416.33
48981
C2
#33952
46085
218.28
317.88
Co
C24
J01N23
37787
22.33
270.9
3714
86
298457 C26
30115
17.36
262.68
572.55
436540
45042
33.57
339.57
446.14
C7
Cy
11982
7842
12.4
724.75
636.39 39
C7
C10
238182
56575
40.52
338.31
428.4
C7
C11
17182
10090
14.47
657.49
620.35
47
C12
74377639
84716
39.24
212.34
304.78
12348
636.25
C7
C14
91450
38685
34.97
414.38
491.63
47
C15
28K54660
123209
53.78
200.1
280.57
C16
75894773
94444
203.06
287.73
C7
C17
40660
21577
24.25
515.12
590
C1R
1771225
104806
51,31
234.44
C7
41919
29.7
324.8
466
230
47371
30420
31,85
479.96
$2393
75136888
88970
208.31
297.62
C7
C22
70044S
24135
2542
446.7
517.49
C23
69312
21818
23.79
139.37
51991
C24
50336272
103413
46,89
207.86
294.56
27
C25
166155
36959
25.82
439.20
52X34
C26
38014733
71088
35.56
229.28
335.88
C8
1351
066
0.64
301.59
445.11
C8
C10
672470
48179
29.58
281.42
386.09
C11
246919
28835
22,55
360.15
458
17

---

Under review as a conference paper at ICLR 2023
C8
C22
1448335
46342
22,08
218.43
519.2
C73
192815
20392
19,56
368,4
470.4
C8
C14
7603
1321
1,05
364.99
472.07
C8
C35
31255
22.42
329.01
437.19
C76
1426019
49809
24.34
224
$25.08
C8
C17
5354
1163
0.95
374.7
489.44
CMA
204516
19086
345.67
453.35
C8
C19
8,464
7972
5.88
338.4
444.15
C8
C20
2516
798
0.63
360.99
467.7
C21
[-112003
47665
22.57
219.92
32031
C8
C22
3359
901
0.65
330.91
475.51
C8
C23
58:7
1206
0.99
376.07
49287
C8
709533
19791
27.66
261/1
3667
C8
225
11146
1738
1.41
371.65
483.2
548366
30767
253.52
358
C10
37586
23958
1922
367.44
400.48
6317
3221
5.71
812.21
679.5
307826
40733
19.5
220.16
321.44
9
C33
1065
1765
3.95
1033,06
C75
22901
10022
1000
459,94
557.52
C70
193126
02011
20.92
228.26
33212
C78
9681
4547
5,06
509.69
50066
C79
1237
2214
2.0
538.05
58233
C2
304213
41192
20.05
223.11
$22.22
C9
C24
104354
25184
16.51
300.58
411.1
C9
C26
79650
20247
12.76
289.85
594
C10
CH
91642
33364
27-97
384,27
4803
C10
78116021
73831
203.52
296J05
C10
70927
20424
24.97
389.02
4K5.56
C30
268775 C14
47774
32.11
308.12
412
CJ0
102841
199,07
283
C10
C16
6081277
7W773
34.12
195.6
280.63
C10
4F7
312092
44463
30.74
316.96
419.33
C70
3524776
9018
42.31
215.05
M6356
C30
C79
1736492
48258
26.12
266.92
578.75
C10
160915
41714
329.88
C10
C2
79351889
76455
33.33
199.78
288.52
C10
222
169631
33712
24005
327.05
43039
C30
196907
38-64
27.12
323.17
42
C10
22
57407333
87116
38.29
201.48
2858
C10
283759
87918
26.43
319,50
425.56
C10
4449999
65658
31.38
219.1
319.66
CH
C12
5873140
76545
37,53
224.75
324.16
CJ1
C13
3460
2521
4,37
830.07
083.38
C1I
C14
40358
19527
21.43
502.94
556.55
CH
CJ5
184799%
110386
53.59
231.71
306.51
CH
C16
5914
85V9S
41.47
221.08
31566
(1)
C17
31753
16992
20.57
554.95
576.07
CT
87394
249.22
344.12
CTI
C19
947216
31892
20.29
291.63
426.76
CF)
C20
30474
12946
18.01
637.83
604.71
80683
39,4
223.86
$22.08
CH
C22
33530
12068
13.13
498 66
$39
C1
30/667
13380
139
478.04
573.47
C
3654197
91578
45.3
226.75
319/99
C1
C25
91831
14952
1443
442.27
$45.98
CH
C26
27751I8
63466
31.4
226.78
32992
CR
C13
$3502294
73654
36.56
237.52
329.06
C12
C17
4109926
41575
1933
218,65
3203
C12
C73
1078992
49360
26.56
246.12
361.77
C12
C70
127478
48044
21.51
208.34
301.65
C17
1078301
46908
22.01
230.91
52481
C12
C18
728462
43143
22.46
233.28
$44.19
C12
C19
489684
41677
20.36
223.94
$29.07
C12
C20
346040
40127
19.11
218.34
310.20
C12
C21
201953
41802
19.51
213.92
51362
C12
339523
42652
20.45
219.76
524.27
C12
804488
46019
21.95
218.66
31819
G12
C24
218622
54766
27.55
228.95
529144
151278
41326
220.07
319.52
C12
C26
4067799
34177
27.06
220
$29.34
C13
C14
36418
14778
17.58
545.39
576.3
C13
C15
1541997
103804
52.44
231.6
330.76
CB
C16
5369689
82639
40.64
224.87
32163
21711
13068
17.34
605.06
594
C78
891380
80044
45.09
258.23
35454
610
814645
20681
18:36
282.07
416.78
C13
12687
14.33
740
62580
CB
C21
$3797799
77631
38.45
237.05
327.99
C17
C22
23083
8892
10:41
336.3
58378
C13
20833
10425
11.47
504.4
567.35
C3
C24
3331274
87527
44.17
231.33
326.99
C13
69822
11776
11.75
457.51
53989
C33
C26
2466154
61128
30.4
237.99
332.29
C14
C15
11281
6532
7,31
$12.76
568.76
3939913
44415
22.19
229.013
33290
18

---

Under review as a conference paper at ICLR 2023
G14
9799
3079
6,02
543.16
564,07
C3
20329
0913
5,89
250,4
477.3
C14
409726
424912
20.66
222.92
522.38
C14
150980
30844
19,48
289,89
392.99
C14
C26
119197
23964
14,33
279.87
5M691
C13
10925488
53050
28.67
247.75
359.24
C15
28410
397,42
49
CTS
C18
12255
6521
7.29
312.29
565.84
CM
C79
186752
25897
18.35
324.82
434
C13
C20
12824
7268
8.37
528.03
581.62
C13
C2F
1093512
50874
27.06
245.6
559.28
C15
C22
56RIZ
15626
14:28
418.99
5132
CT5
79514
22941
19.8
396.23
489.00
C13
C27
391651
43783
27.07
283.40
394.29
19235
9.96
541.66
C15
C20
431970
36836
23.9
297.37
411.88
C16
1028310
50610
24.88
225.31
328.66
CJ6
727988
46325
24.33
240.57
35021
C16
C79
392868
42613
21445
229.46
33521
131688
42038
20.78
226.00
827.7
C16
C21
220892
15663
21.16
213.33
311.16
C16
C22
538479
44618
22.25
228.89
354.42
CJ6
C23
852001
49312
24.22
225.11
323377
C16
C24
218175
53197
26.54
228.07
$5042
C16
C25
337787
42815
21.46
229.76
33285
C16
C26
38W958
$9940
27.25
231,58
334.5
C17
C78
38525
14374
13.84
441.28
518.66
17150
7252
5.96
378.84
462.68
C17
C21
10650177
48331
25.57
221.64
$25.22
C17
C24
183364
41263
24.18
268.65
376.64
C17
C20
293309
31763
17.97
259.39
566.12
C18
C79
39082
13650
11:21
376.46
4
C18
C20
5016
2769
3.57
390,95
61321
CT8
734247
45194
23009
234,21
34382
C18
C22
25833
7794
7,03
448.02
551.8
36318
12199
442.91
31835
C18
C24
301722
33658
21.14
287.85
403388
C18
11302
3962
4.47
316.88
37531
C38
25295/1
20993
17.63
299.37
414.88
C19
C20
39:16
3305
3,68
$10.17
$46.13
313626
42198
20.78
225.76
529107
C19
12243
3893
3.89
434,25
512.33
C19
18121
6530
5.64
395.95
478.38
C39
C24
313703
30933
18.5
273.92
356.0
C19
C25
312488
7260
5.92
374.09
483.47
C20
319124
25369
14.06
252.04
357.6
342773
40942
19.74
270.99
319.39
C20
C24
122667
27766
17.58
290.31
395.66
21542
390.22
C21
551544
43555
21.03
221.35
524.7
C
C23
885098
47766
22.74
220.57
319.3
C2)
225892
54751
27.33
228.00
327.83
C21
C25
348750
42006
20.41
222.77
T12
403230
54611
27.52
220.35 35
529.42
C22
307779
29960
187
286.2
39664
C22
C20
162373
23738
14.33
276.8
58330
C21
326117
38/944
22.7
270.28
373.9
C21
C26
2461990
29727
17.06
263.12
305.1
C2
L17933
26682
17.68
301.46
41397
193330
30432
20,05
302.077
412.59
C2
C20
96107
21524
13.80
295.26
402.3
130
0.29
1012.17
693.88
C3
305
258
0.17
328.15
44935
C6
19
5
0.01
918.6
709.3
C14
27
1095
562
C17
1292
0
(22
18
193.4
155.1
C25
86
24
0,412
304.54
209.11
1287
452
0.76
766.81
C2
C20
7867
189
0.57
904.18
672.78
C5
654
312
0.24
349.6
474.05
12200
441
0.39
403.22
526.44
C5
1713
471
0.37
360.37
478.7
48
0.02
580.5
54
C6
304
0.11
379.75
62
617
164
44
0.08
317.89
$32.98
0.03
331.51
516.7
C6
111
27
0.04
593.22
587.88
208
0,06
328.42
527.19
186
0.25
631.13
649.91
C9
68
18
0.03
790.28
749.09
28
0.02
893,8
750077
C9
C20
3
0.01
1145.33
535177
(:22
46
10
0.01
548.9
454
C23
0,022
672.18
709.77
19

---

Under review as a conference paper at ICLR 2023
C25
306
-7
0,077
5M0.17
653.06
C2r
C77
0.17
966,99
020,28
C14
C29
80
13
0.002
624.25
582.95
C14
294
0.15
641,21
65066
C14
197
0.11
827.28
746.54
C14
744
214
0.27
577,89
595
C17
328
128.08
C17
133
34
0.04
330.21
477.38
C17
135
23
0.02
480.09
573.73
C17
194
738.01
70307
C20
13
0.03
899, 38
755.05
C20
0.01
1095.6
398.26
C20
0.09
648.51
75085
C2
192
46
0.06
634,89
603.3
522.01
557.99
C23
809
210
0.28
614.03
636.39
A.9
BAG SIZES TO ACIIEVE 50, 70, 85, 95 PERCENTILE OF BAGS
This table contains the threshold bags sizes such that 1% of the bags have at most that size, for [= 50, 70, 85, 95 for all 349 groupings (removing groupings which were left with no bags after clipping). We perform K-Means of 308 of these groupings that have more than 500 bags after clipping. The cluster assigned to each bag is also listed.
Table II: Threshold bag size values below which 50, 70, 85, 95% of bags present and clustering based on this distribution
Coll
Cal2
Bag size below which 50% bags
Bug sizee below which 70% hags
Bag size below which 85% hags
which 95% % hags
Bag sizee below
Clusters assigned on
bag size distribution
70
1130
Very Short-nailed
C2
101
172
335
812
Very Short-taled
G4
117
212
430
1019
Very Shori-tailed
951
1499
20776
Vary Long-ttilled
C8
110
210
310
1383
Shont-talled
C10
17%
365
1539
Short-taild
C1)
637
1155
1690
2104
Very Lomg-tniled
G12
103
175
941
831
Very Shori-talled
C13
1496
2349
Very Long tailed
C13
204
364
1121
1829
Long called
C16
10m
184
364
390
Very Short tailed
C38
697
1271
1983
Long salled
C19
150
917
1454
2153
Very Long lailed
(2)
104
190
835
Very Short-tailed
C24
1356
264
339
1279
Short Italed
C20
136
256
538
1203
Shout-Hailed
131
308
1453
108
171
335
825
Very Shori-taled
1UN
184
872
Very Shore-tailed
190
415
1050
Vary Shori-ailod
142
286
647
1449)
Shout-malled
271
1270
Shont-tilee
C3
CA
70
140
361
1010
Very Short-taited
91
197
449
1168
Very Shori-cailed
C10
122
234
1068
Vary Shor-tailed
145
200
634
1374
Short-iaaled
C12
10%
338
820
Very Short-nalld
209
643
1397
Shoet-talle
142
389
633
1461
Shoet-talled
265
1267
Shont-1alled
C16
104
176
347
854
Very Shori-tileed
1.1
302
655
1413
Short taaled
281
606
1333
Shvet-lailed
140
272
575
1292
Short iaaled
1388
1419
Shout-maled
1035
174
339
830
Very Short tailed
109
224
525
1247
Shout-tliled
293
1456
Short-failed
116
209
402
1006
Very Short-ailed
148
297
1417
Shoot-maled
300
404
964
Vary Shori-ailed
C2
102
176
344
837
Very Short-talle
201
984
Vary Shori-niled
C2
175
377
816
1622
Shost-tilld
419
878
1499
2112
Very Long-tiiled
136
253
503
1102
Vary Shor-tailod
164
344
742
1523
Short-
C10
957
Vary Short-naited
C2
311
637
L329
Shoet- Ialed
02
612
103
177
147
843
Very Shori-oiled
talkd
20

---

Under review as a conference paper at ICLR 2023
C2
C74
613
11.99
1701
2229
Very Long-tailld
C25
304
1121
1829
Long called
C2
C76
105
1383
364
903
Very Short-talle
479
855
1413
2010
Very Long tniled
327
696
1221
1983
Long-tailed
151
304
630
1413
Shoot tziled
853
Very Short-nailed
304
664
1245
2019
Long-sallie
395
788
1351
1992
Long-tailed
232
Very Shor--ulled
C2
C25
210
506
1041
1813
Long-tailed
C2
C26
122
223
158
1084
Very Shor: called
C3
103
175
335
803
Very Shor--uilld
C
1033
124
336
819
Very Short called
837
Vary Shori-uiled
1035
173
334
804
Very Short tailed
103
173
336
832
Very Shore-talee tailed
102
175
827
Very Short catlel
C70
100
164
312
753
Very Shor: called
179
853
Very Shore-called
172
3344
810
Very Shor: tailed
106
180
357
864
Very Shore-tailed
C3
103
125
830
Very called
2
109
193
402
1030
Very Short-taile
C10
100
316
764
Very Short tuiled
108
175
344
846
Vary Shor--ailed
C78
1034
188
371
950
Very Short-ailed
858
Very Short-nailed
C3
C20
103
175
3401
820
Very Short-taled
C21
100
167
332
786
Very Shori-tailed
172
337
Very ualled
C3
C23
103
174
340
834
Very Shori-talldd
107
185
364
875
Very Shor: cailed
C3
103
175
339
826
Very Shori-uille
C20
107
185
365
877
Very Short called
912
Very Shor--uile
C4
110
190
373
895
Very Short tailed
100
161
29h
685
Very Shor: sailed
109
198
372
393
Very Shor--ailed
116
207
4118
987
Very Short called
157
2901
069
Very Shors-tailed
104
174
333
780
Very Shor: tailed
C4
212
1033
125
337
804
Very Short-tailed
C4
C73
106
127
300
Very cailed
C4
C14
114
200
397
945
Very Short-taited
114
203
10300
Very cailed
G4
G16
104
177
349
843
Vary Shor--ailed
C4
217
110
189
394
892
Very tailed
205
1020
Very Short-nailed
C4
C19
196
392
953
Very Shorttailed
C4
C30
114
202
407
965
Very Shori-talldd
G4
C21
10%
175
337
793
Vary Shor-tailed
C22
113
199
400
964
Very Shori-uiled
110
384
904
Very Short-naile
202
4088
976
Very Shori-uiled
24
116
208
418
1012
Very Short cailed
202
S83
Very Shor--unlee
176
358
821
1641
Short itiled
163
331
702
1965
Short itiled
210
420
1210
Very Shor--aaled
133
253
523
1192
Short iaaled
356
1556
Shout-maled
C5
103
174
339
825
Very Short tailed
C5
175
368
791
1582
Shout-tliled
373
825
1638
Short-1aaled
1:5
310
667
1416
Short-uiled
105
180
860
Very Shore-taled
180
392
789
1699
Sheet-talle
63
C18
162
334
730
1498
Shout-tailed
339
1506
Shont-tailed
C5
C21
1044
177
344
K33
Very Short-taleed
C5
174
367
821
1705
Short-tailed
223
1093
Very Shori-talod
C5
175
372
772
1589
Mhart-laale
119
215
1034
Very Short-tailed
C6
463
854
1600
Shoet-falld Italed
165
343
739
1534
Shoet-tailed
221
1246
iaaked
C6
310
640
1172
1972
Long talled
102
171
333
839
Very Short tailed
367
728
1274
1944
Long-tailed
86
190
400
837
1643
Shoot iaaled
180
872
Very Short-tailed
C6
C18
217
464
937
1730
Long called
C6
C19
211
422
822
1578
Shoot-Hiled
C21
173
540
Very Shor--alled
21

---

Under review as a conference paper at ICLR 2023
G6
G24
124
228
459
1091
Very Short-talle
C20
119
210
1006
Very Shor: cailed
C7
154
363
646
1383
Shout-iniled
502
951
1499
2076
Very dailed
296
613
1316
Shoet-triled
419
8.32
1392
2019
Very Long- tailed
797
Very Short-nilled
C7
C13
502
951
1499
2076
Very Long-tille
305
417
833
1577
Shoet-tailed
168
656
Very Shor--ualed
C7
C16
102
168
311
727
Very Shori-uiled
1841
381
1076
1791
Long sailed
C7
C1a
110
188
360
830
Very Shor--ailed
27
C79
133
273
619
1384
Shoet Iaaled
525
1703
Lang-taileed
C7
C21
1035
171
321
761
Very Short tailed
325
467
929
1670
Lang-aaleed
450
918
1070
Long cailed
104
172
322
718
Very Short called
209
450
1700
Long taig-tailed
77
106
181
364
887
Very Short cailed
88
121
220
596
1443
Shoot-Hailed
C79
1288
238
1132
Very Short called
C1f
159
326
705
1486
Shoot-izlle
C22
104
174
339
833
Very Storr-cailed
162
337
727
1516
Sheet-tiled
C8
C34
156
348
719
1305
Shoot-lailed
288
619
1350
Short-uile
C76
105
179
352
860
Very Short-tailed
1.58
344
748
1552
Short-tilld
308
1122
Short iaaled
C8
C19
1.0
306
646
1381
Shoet-talled
C8
C20
161
3330
754
1337
Short tailed
C8
176
342
839
Viry Shori-ailed
C8
121
268
060
1473
Short Italed
343
1580
Shont-iaike
88
C24
119
216
441
1049
Very Short tailed
1:58
3334
743
1529
Short itiled
116
209
426
998
Very Shore-aaled
C70
108
314
709
1468
Short iaaled
610
1119
1671
2185
Very Long--taild
1033
177
346
834
Very Short tailed
925
1394
1903
2298
Very Long-tailld
C33
223
482
906
1749
Long talled
C16
106
188
364
850
Very Short-taile
264
360
1079
1863
Long-tailed
G19
784
620
1162
1923
Long-tailed
C21
1035
181
355
845
Very Short-
256
1245
Shont-taile
C9
C26
1.91
244
512
1156
Very Short-tailed
C10
C1f
177
365
763
1523
Short-tailed
C30
C12
59
163
309
744
Vary Shor--aaled
C10
177
372
777
154K
Short-laaled
272
1238
Short-tried
C10
164
303
705
Very Shori-uiled
C30
988
138
296
097
Very Short cailed
280
1281
Short- lhoee-
C10
105
176
33N
786
Very Short tailed
C10
118
218
454
1081
Very Shor: cailed
C30
131
295
614
1342
Short-1ailed
C10
59
161
302
721
Very Short called
290
1336
Short-taied
C10
149
289
395
1300
Short itiled
C30
1000
165
310
728
Very Short-tailed
C30
144
2911
1294
Short- -iiled
C10
C26
104
176
341
833
Very Shori-taled
C11
106
181
356
857
Very Shore-tailed
CH
C13
636
1151
1698
2197
Vary Lang-tailed
(1)
C14
365
545
1067
1810
bang-tailed
C31
110
810
Vary Shori-tailed
C76
106
180
349
830
Very Shor-tailed
CF)
C17
312
647
1181
1872
Long-tailed
C3)
C18
119
211
417
951
Very Shor-talled
CD
C79
117
223
305
1304
Short-
C11
402
793
1345
1979
Lang-tilld
C1)
106
181
353
852
Very Shori-ailed
351
560
1021
1812
Lang-taited
C11
307
1021
1809
Long Galled
C11
C24
109
187
364
859
Very Short-tatted taited
C11
139
435
937
1783
Long sailed
C12
C20
106
191
356
384
Very Shor--ailed
C12
107
183
362
871
Very Short tailed
101
830
Very Short-tailed
C12
108
191
395
1023
Very Short tailed
C12
C16
100
167
318
772
Very Short-talld
C12
C17
175
344
Very Shor--ailed
22

---

Under review as a conference paper at ICLR 2023
C12
C78
104
1883
369
936
Very Short-talle
C22
C79
103
127
353
308
Very cailed
C12
C29
103
176
342
821
Very Short-taite
C12
101
171
332
798
Very Shori cuiled
C12
105
174
842
Very Shor-talled
C12
103
175
342
8,36
Very Short cailed
C12
873
Vary Shors-nalle
C12
C25
177
342
833
Very Shor--uiled
C12
C24
108
186
366
876
Very Short-tailed
C13
618
1157
1382
Long tailed
C13
C25
113
195
374
868
Very Shori-talled
C13
107
182
3559
351
Very Shor: called
C13
367
733
1279
1960
Long-talled
C13
122
220
436
997
Very Short called
214
1264
Shont-lalld
C13
327
951
1491
2074
Very Long iailed
C19
107
183
361
369
Very Shori-taledd
276
620
1157
1902
Long cailed
352
553
1085
1859
Long talled
191
375
8M0
Very Shore-calee
C13
194
460
1002
1819
Long Galled
C13
C26
107
188
357
890
Very Short-tailed
C34
C25
365
11.0
1929
Long called
C14
E16
107
185
362
882
Very Short-taied
C14
C28
291
618
1180
1885
Long cailed
C14
C19
188
379
761
1507
Sheet-1illd
C14
C21
106
181
349
8,39
Very Short- cailed
C14
248
Very Short-nalled
C14
C26
127
235
485
1134
Very Short-taled
CM
C10
111
196
403
1017
Very Shori-ciled
C13
381
1392
ialed
C15
C18
263
563
1120
1825
Long-tuiled
C15
C19
142
2344
610
L335
Short talled
CT5
372
388
1168
1885
Long-tailed
C13
109
192
390
1011
Very Short called
192
407
1662
Shont-laake
C15
184
3864
789
1580
Short itiled
C15
125
2335
1991
1181
Very Shore-caled
C33
494
993
1746
Long sailed
C13
129
236
534
1252
Short iailed
861
Very Short-taile
C16
108
191
389
966
Very Shor: tailed
C36
1125
182
366
898
Very Short-tailed
CJ6
C29
106
192
300
369
Very cailed
C16
C21
101
171
330
792
Very Short-taited
C16
C22
106
1855
362
890
Very cailed
C16
C21
106
192
359
861
Vary Shor--ailed
C36
C24
107
184
364
Very Short tailed
C16
901
Very Short-nailed
C16
C26
108
186
369
859
Very Short-aaled
C17
215
456
910
1695
Long-tailed
C17
184
367
737
1477
Shoet-tille
C17
104
176
345
842
Ver Shori-ualled
122
233
458
1066
Very Short-naile
C17
11k
213
441
1022
Very Shor--uiled
C18
C29
100
3418
742
1558
Short iailed
C18
712
1292
2002
Long Gaileed
C18
C21
106
185
374
936
Very Short tailed
C18
211
454
9518
1722
Long sailed
C38
217
460
918
1684
Long failed
C18
126
238
497
1204
Very Short called
564
1119
1860
Long-tailed
C18
C20
1.30
251
328
1253
Shoet italed
C19
C20
287
573
1076
1273
Long-tailed
C39
C21
104
130
358
870
Very Short-aated
C19
C22
2066
453
888
1648
Long-tailed
C23
195
391
1526
Short-male
C19
120
225
473
1125
Vary Shori-ailed
C19
C25
166
344
734
1575
Shoot-talld
205
1004
Vary Short-tailed
C20
C21
1044
179
348
830
Very Shor-tailed
C20
C24
130
247
516
1171
Very Shori-talld
C30
130
241
40%
1140
Very Shor-talled
C21
104
177
345
848
Very Shori-ualed
104
346
846
Very Short-naild
C2)
10k
187
368
361
Very Shor--uiled
(3)
105
180
349
845
Very Short-tailed
C21
186
875
Very talled
C22
123
257
304
1170
Very Short tailed
C22
124
230
182
1124
Very Shor: tailed
C21
12.
236
467
1079
Very Shor-tailed
C23
122
231
445
1039
Very Short called
133
255
1257
Shoot-maled
C24
134
256
540
1255
Shoet ttiled
C25
C26
133
250
523
1197
Shoot-Hliled
1443
1346
2383
23

---

Under review as a conference paper at ICLR 2023
G5
140
2010
6:00
1550
573
1302
2079
2079
C14
1687
1657
1687
1657
C17
1292
1292
1292
1292
117
180
496
496
C2)
62
62
62
62
408
730
C2
5:4
993
1692
2277
C2
C20
801
1200
1713
2249
303
C5
C20
168
346
793
1708
C22
135
327
712
1167
439
771
1236
1852
C6
C7
159
608
1595
2127
353
630
864
C6
C29
336
747
1060
1738
113
373
1362
1869
717
1006
1902
86
355
817
1475
2060
1046
1687
23H0
0o0l
1513
1877
2286
E20
773
1902
1902
1902
157
1113
1109
1302
204
917
1526
2323
C9
302
041
1361
2225
C14
G17
872
1320
1792
2134
C14
C20
530
737
1687
1988
C14
958
C14
C24
574
944
1781
2305
C14
C25
209
700
1304
1909
C17
367
313
C17
C22
169
580
830
1655
C17
2417
619
912
1733
C17
585
1691
2262
C20
812
1482
2174
2130
1386
1700
C20
297
834
1605
2243
C22
176
768
1291
1899
C22
536
1147
1903
C23
C25
345
740
1327
2108
A. AVERAGE LoG LIKELIHOOD OF LABEL HISTOGRAM DISTRIBUTIONS FOR ALL GROUPINGS
This table contains the Average Log Likelihood of Label Proportion distributions for all 349 groupings (removing groupings which were left with no bags after clipping). We perform K-Means of 308 of these groupings that have more than 500 bags after clipping. The cluster assigned to each bag is also listed.
Table 12: Average Log Likelihood of Label Distribution of all grouping and it's clusters
Coll
Coll
Standard deviation label propettion
Label bias of the grouping
Average Log likelihood of label histogram distribation
Clusters assignedd on label
histogram distribution
0.05
0.26
-3.31
High
C3
0.17
0.26
17.26
Low
0.16
0.26
Law
C7
0.11
0.23
25.85
Very Low
C8
0.05
0.26
-3.33
High
C10
0.25
11.68
Medtarn
C1
0.1
0.1
-3.13
Very Low
C12
0.17
0.26
17.29
Luw
CB
0.09
0.1
37.14
Very Low
C13
0:10
0.27
34,69
Very Low
0.16
0.26
C18
0:15
0.29
35.47
Very Low
(19
0.07
0.26
-10.9
Madiwan
C2)
0.10
0.26
17.33
Low
C34
0.14
0.26
-9.06
Low
C26
0.14
0:25
-17.44
Low
0.76
-11.58
Modiaan
C)
0.16
0.26
-15.35
Madimas
0.15
-14.82
Madiaam
CJ
0.05
0.26
3.29
High
0.85
0.26
3,86
Fligh
0.12
0.24
-14.11
Madiaas
0:05
0.26
3.26
Hlihh
0.08
0.24
-641
Hlgh
C7
C70
0.26
917
[ligh
611
0:11
0.24
-13.22
Mediten
13.3
24

---

Under review as a conference paper at ICLR 2023
C
C33
0.24
-12.79
Medimas
0.26
10.08
Medruc
C75
614
0.27
-17.64
Low
615
0.26
14.98
Medracn
0.23
-9.44
Madiwa
0:14
0.27
-6..45
Low
C19
0.0%
0.26
-6.54
Flghh
CF
C20
0:05
0.25
-3.74
fligh
CF
0.16
0.26
-15.25
Mediaan
CJ
0:05
0.26
3.47
Digh
C1
C28
0.08
0.25
-8.5 4.35
High
0:14
0.26
13:09
Mediurn
C1
0.06
0.77
4.95
Fligh
0:14
0.25
14.47
Medrum
0.26
Luw
Q2
0.16
0.25
18.75
Low
0.26
13.94
Meshum
0.26
25.47
Very Low
0.21
17.07
Low
0.26
12.51
Mediaa
0.13
0.26
12.55
Medoucn
0.14
0.24
-19.45
Low
0:17
0.26
18.35
Low
0.14
0.24
-19.35
Low
0:13
0:23
38,88
Very Low
0.16
0.27
-34.88
Very Low
C70
0:17
0.26
-18.51
Low
0.22
-30.65
Very Low
0.15
0.29
-35,45
Very Low
0.26
-10.64
Mediann
0.26
18.89
Low
0.12
0.27
-20.9
Low
0:13
0.23
31.79
Very Low
0.55
10.35
Law
0:11
0.27
10.42
Low
0.15
0.24
17.85
0.16
0.26
16.04
Low
0
0.26
16.29
Luw
0.17
0.26
Low
0:18
0.25
17.95
Low
0.16
0.26
15.91
Mediaas
0.17
0.26
18.36
Low
C10
0:16
0.26
-14.44
Mediaa
Cii
0.18
0.25
18.37
Low
C12
0.17
0.26
-17.24
Low
0.18
0.24
-1..5
Low
0.18
0.26
-19668
Law
C15
0:18
0.26
21,58
Low
C16
0.16
0.26
-16.08
Low
C3
C17
0.18
0.26
-1.35
Low
C18
0.17
0.26
-19.4
Low
0.1G
0.26
-6.79
Low
C20
0.17
0.26
18.1
Low
0:17
0.26
0.26
17.05
Low
0:18
0.26
19.2
Low
0.16
0.26
16.82
0.17
0.26
17.98
Low
0
0.26
17.65
Low
0.15
0.26
13.94
Mediwas
0.10
0.26
10.45
Low
0.18
0.26
-17.89
Low
0.15
0.26
15.44
Medsusn
0.16
0.25
-19177
Low
C4
C70
0.10
0.27
14.9
Mediia
C4
Cff
0.17
0.26
-17.2
Low
C22
0.16
0.26
-15988
Mediaas
G4
G13
0.17
0.76
-17.9
Law
C4
C14
0:16
0.26
-19.6
Low
C4
015
0.17
0.26
-20.63
Law
C4
C76
0.16
0.26
-16.51
Low
C4
017
0.16
0.26
-1879
Low
G4
0.16
0.26
-9.2
Law
C79
0:15
0.26
-17
Low
0.16
0.26
-18.48
0.16
0.26
15.96
Medoven
24
0.16
0.26
-1.34
Faw
0.17
0.26
19.01
Luw
C4
C24
0.15
0.26
-7.38
Low
0.16
0.6
18.55
Low
C4
C20
0.1G
0.26
18.41
Low
C9
0:03
0.26
4.1
High
0.12
0.23
-15.77
Mediaa
C5
Ca
0.05
0.26
3.29
High
C3
C10
0.1
0.26
.98
Mediww
C11
0.11
0.23
-13.15
Mextia
25

---

Under review as a conference paper at ICLR 2023
G5
C22
0.16
0.26
-16.24
Low
0.23
13.99
Medrum
011
0.26
-13.34
Madium
G15
0.27
20.47
Low
C76
D:T6
0.26
-16.02
Law
C5
0:09
0.25
-11.8
Medowa
0:14
0.28
-19.16
Low
CS
C19
0:07
0.26
-7.35
Flgh
0.16
0.26
-16.28
Fow
0:08
0.24
-9.9
Medsum
C5
C34
0.14
0.26
-16.28
Low
0:06
0.27
3.33
High
C5
C26
0.14
0.25
15.58
Modiurn
C6
0:12
0.23
18.8
Low
0:05
0.26
Flghh
C6
C70
0.1
0.26
10.84
Mediuma
025
21.32
Luw
C72
0.10
0.26
10.94
Low
0.1
0.23
30.65
Low
0.16
0.27
22.42
Vey Low
C6
C76
0.16
0.26
16.72
Low
C6
C18
0:16
0.29
26.28
Vey Low
Co
C79
0:08
0.26
3,39
High
C6
0.16
0.26
-1699
Low
0:15
0.26
17.50
Low
0:15
0.25
-16.74
Law
0:12
0.24
-14.95
Medima
0.22
-25.84
Very Low
C19
0.11
0.24
-13.29
Madiusa
0.22
-25.75
Very Low
0.25
17.65
Low
C7
0.22
-3585
Very Low
0.23
21.35
Low
C7
0.5
17,15
Law
0.25
17.37
Low
021
Luw
C7
0.16
0.24
17.44
Low
0.44
16.04
Luwe
0.22
20.59
Low
0:18
0.25
17.65
Low
0.24
-1966
0.12
0.23
20
Low
0.17
0.26
-1685
Low
0:12
0.23
19.37
Low
0.17
0.25
-17.49
Low
0:08
0.23
-3.1
High
G10
0.26
J9:48
High
Ci1
0:11
0.24
-14.45
Medira
0.26
-15.84
Maliunn
C8
C13
0:11
0.24
-13.45
Madiai
C8
0.1
0.26
-12.16
Mediarn
0.14
0.27
-1911
Law
0.16
0.26
15.6
Mediamm
0.09
0.23
-00.55
Madiaas
C8
0.14
0.28
1788
Low
0.08
0.26
699
High
0:05
0.25
High
88
C2F
0.16
0.26
13.87
Mediusa
0:055
0.26
High
0:08
0.24
9
High
0:14
0.26
15.74
Medrum
0.066
0.27
High
88
0.14
0.25
15.18
Medsuma
C10
0.1
0.24
-12.2
Mediwa
Cii
0.1
0.2
30.29
Very Low
C12
0.17
0.26
-18.55
Low
0.1
0.19
33,86
Very Low
G15
0.16
0.25
-34.98
Vary Low
C9
Ci6
0:17
0.26
-18.45
Low
C18
0.16
0.26
-35.17
Vary Low
C9
C19
0.1
0.28
-1595
Madiwa
0.17
0.26
-18.2
Low
0.15
0.25
-01.5
Law
0.14
0.4
18.95
Low
C10
0.24
-1361
Madiiun
C10
0.16
0.26
14.31
Medowrn
C10
0.1
0.24
-188
Modiiern
C10
0.12
0.26
13.33
Medruc
C10
C35
0.15
0.26
14.08
Mediuss
C10
0
0.66
14.35
Mediarr
C10
C77
0.1
0.25
11.75
Modiaas
C10
C13
0.13
0.26
13,55
Medium
C10
C19
0.12
0.26
-11.26
Modiuas
C10
C20
0.1
0.26
11.45
Medoum
C30
C21
0.16
0.26
-14.4
Madiwan
C30
C22
0.1
0.26
-1.19
Mediua
26

---

Under review as a conference paper at ICLR 2023
C10
C2
011
0.26
-12.46
Medimas
C10
C21
0,15
0.27
13..88
Medrum
C10
625
011
0.26
-11.28
Madius
C10
0.26
13.96
Medrach
C1)
0.88
0.25
-18.45
Law
C11
0.1
0.21
29.1
Very Low
C11
0114
0.21
-25.74
Vey Low
C11
0.16
0.24
-17.9
Low
CF)
C16
0.17
0.25
-18.17
Low
C11
0.21
20.9
Low
CHI
C1R
0.16
0.24
-1861
Low
C11
0:33
0:25
14.39
Mesburn
C1)
0.2
Law
C1
0:18
0.25
18.6
Low
025
Luw
C1
0.2
0.23
21
Low
017
0:25
17.55
Luw
CH
0:12
0.23
18.45
Low
0.17
0.25
17.87
Low
0.17
0:25
11.42
C12
0.18
0.26
19.52
Low
C12
0.18
0.26
21.37
Low
C12
C70
0,10
0.26
10.1
Low
C12
0.17
0.26
-1.44
Low
C12
0:17
0.26
19.37
C12
0.16
0.26
-1682
Law
C12
C20
0:17
0.26
11.09
Low
C12
C21
0.16
0.26
-16.99
Faw
C12
0.16
0.26
-17.05
Low
C12
0.18
0.26
-19.21
Low
C12
0.16
0.26
16.4
Low
C12
C25
0.17
0.26
18.00
Low
C12
0.16
0.26
17.02
Low
CR3
0:13
023
16.15
Very Low
C13
0:10
0.24
18.38
Low
0.25
C13
0.1
0.11
20.48
Low
0.16
0.24
18.75
Luw
0.13
0.25
13.32
Mediwas
0.1
0.22
21.19
Low
0.17
0:25
-14.50
Low
C13
0.23
19.35
Low
0:12
0.23
-00.99
Low
C33
C24
0.10
0.25
-1744
Low
C9
0.12
0.23
-17.36
Low
C13
C20
0.17
0:25
-1.96
Low
C14
G75
0.16
0.27
-34.64
Very Low
C14
C36
0.17
0.26
-19.62
Low
C14
C18
0.15
0.27
-34.77
Vary Low
C14
C19
0:11
0.25
-13.26
Madiaa
C14
0.18
0.26
-19.7
Low
C14
0.1G
0.26
-21
Low
C14
C26
0.15
0.25
-1.33
Low
0.17
0.26
-3115
Low
CTS
0.16
0.25
29.05
Very Low
C13
0.10
0.27
34,66
Very Low
0.13
0.26
17.46
Luw
C13
C20
0.16
0.27
34.75
Very Low
0.18
0.26
21.28
Lowe
C33
0.16
0.28
26.58
Very Low
C13
0.16
0.26
30,28
Very Low
0.16
0:25
-1.78
Low
C15
0.15
0.27
28.49
Very Low
C15
C26
0.16
0.25
-2.99
Low
CJ6
C17
0.17
0.26
-1.97
Low
C16
C18
0.17
0.26
-19.3
Low
C16
C19
0.16
0.26
-16.85
Low
C16
C20
0.17
0.76
-18.15
Law
C16
0:16
0.26
-16.45
Low
C16
C22
0.16
0.26
-17.16
Law
C16
C28
0.17
0.26
-9.005
Low
C16
0.16
0.26
-1629
Low
C16
0.16
0.26
17.99
Law
C16
0.16
0.26
17.68
Low
0.16
0.26
-0052
Very Low
C17
0.1
0.24
Medswrn
0.17
0.26
19.25
Faw
C17
C21
0.16
0.25
Low
C17
C25
0.15
0.24
18.47
Low
C18
C79
0:12
0.7
13.5
Mediurr
C38
C20
0.15
0.28
-3.7
Vary Low
C18
C21
0.17
0.26
19.29
Low
C18
C32
0.15
0.29
-2565
Vay Low
C18
C25
0.16
0.26
32.27
Very Low
C18
C24
0.15
0.25
-30,08
Low
C38
C25
0.14
0.28
23.31
Very Low
27

---

Under review as a conference paper at ICLR 2023
C18
C26
0:15
0.25
-30
Low
C39
C20
0:08
0.26
10.34
Medrum
C19
C31
0.16
0.26
-6.92
Low
C19
C22
0:0N
0.26
8.90
High
0.25
-12.29
Madiwa
C19
0:14
0.25
16.3
Low
0.09
0.26
-10.25
C79
C26
0.14
0.25
-14.99
Medrucn
C30
G21
0.17
0.26
-18.19
Low
C20
0:15
0.25
19.38
Luw
C20
C26
0:15
0.25
-18.09
Low
C21
0.16
0.26
17.08
Low
C2)
6.18
0.26
10.2
Low
C21
0:16
0.26
10.34
Low
0.26
C21
C20
0.16
0.26
17.85
Low
C22
0:15
0.26
18.31
C22
0:14
0.25
Low
C23
0.16
0.26
20.12
Low
0.15
0:24
C2
0.15
0.25
19.27
Low
0:15
0.25
20.44
Low
C20
0:14
0.24
Low
0.13
0.28
47.05
0:05
0.26
3.41
C6
0:09
0.25
-12.4
C14
0:12
0.29
33.6
C22
0.05
0.26
-3.44
C27
0.11
-1 14
C23
0:08
0.26
C2
C9
0:12
0.21
-6684
C20
0:13
0.27
0.08
0.22
C20
0.04
0.25
3.93
0:05
0.27
C6
0.09
0.19
10.45
0:13
0.23
0:09
0.2
11.09
86
C20
0.07
0.23
5.80
0.15
0.27
C6
0:11
0.2
21.74
C6
0.088
0.28
2:44
0:11
0.22
36.01
0.066
0.11
-16.34
0:03
0.11
10.55
0.08
0.18
-7.58
0:014
0.12
-10.5
C14
C17
0.14
0.22
465.9
C14
C20
0.13
0.2
-38.31
0.15
0.26
-37.84
C14
0.
0.24
66.83
0:13
0.26
C17
0.02
0.06
3.74
C17
0:11
0.25
13.32
0.0h
0.09
10.64
C17
C25
0.11
0.21
25,85
220
0:055
0.29
4.9
C20
C23
0.16
0.11
-8.7
C20
C25
0.08
0.25
11.92
C22
C23
0.26
C22
C25
0.08
0.31
9:01
C21
C25
0.12
0.24
.27.2
A.11 BAG SEPARATION STATISTICS FOR ALL THE GROUPINGS
This table contains MeanInterBaggep. MeanIntraBagSer and their ratio for all 349 clipped groupings (removing groupings which were left with no bags after clipping). We perform K-Means of 308 of these groupings that have more than 500 bags after clipping. The cluster assigned to each bag is also listed.
Table 13: Bag Separation Statistics and their clusters on all groupings
BagSep
McanIntraBagNep
InterlatraKatio
Clusters assigned based InterintraRati distribation
CJ
0.83
0.82
1.02
Lesi-sjuratis
0.81
Medium-separated
C4
0.81
1.16
Modium-sepurated
C7
0.81
0.61
1.33
Well-eeparated
0.83
C10
0.82
0.7
1.18
Medium-sejuratst
28

---

Under review as a conference paper at ICLR 2023
0:75
1.2
Medium-separated
C22
0.81
0.
1.16
Medium sepurated
CB
0.72
0.66
1.1
Lesi-sparraed
C15
0.8
0.7
1:15
Medium separated
C16
0.81
0.7
1.16
Mediun-seurated
C88
0.79
0.71
1.12
Medium separatte
C19
0.77
0.76
1.03
Lev-separaais
C21
0.81
0.7
1.16
Mechum--epurated
C74
0.82
0.72
1.14
Meliem-sepurated
C26
0:79
1.12
Methum sepurated
CJ
0.81
0.74
1.1
Lev-seuurated
CJ
0.83
0.72
1:15
Medium sepuraled
CJ
0.82
0.73
114
Mechun-spprated
CJ
0.83
0.8
102
Less sepuraled
0.82
0.79
100
Los-wpurated
C
0.81
0.6
1.34
sepaaaee
0.83
0.1
100
Less Les--epuraled
0.9
05
1.12
Medium separated
0.82
0.2
1.14
Medium sepurated
0.79
000
Medium Medjum--epprattes
0.83
0.72
1.15
Medhum sepurated
0.79
0.
1.33
Mediiu-ssqurated
C7
0.82
0.7
1.05
Less separated
0.82
0.72
1.14
Medium-separated
0.82
0.72
1.15
Medium separated
C17
0.86
0.73
1.17
Mediun-seuratel
C18
0.81
0.73
1.12
Medium-separated
C19
0.77
1.04
eev-sepuratil
C1
C20
0.85
0.81
1.03
Lest-separated
0.83
0.72
1:55
Melium--spurated
CJ
0.83
031
100
Less sepurated
C1
C23
0.84
0.8
1.05
Lev-sequratee
0.83
0.23
1:33
Medium sepuraled
C1
0.83
08
104
Las-sepurated
0.81
0.22
1.12
Medium separated
0.82
0.69
1.19
Methim sepurated
Q2
0.82
0.09
1.38
Medium separated
0.81
0.33
1.1
Less sepuraepal
0.74
0.00
1.33
Medium-scppratee
0.78
0.55
1.44
Far sepuraled
0.81
0.74
Lest-spaaraeed
0.81
0.81
121
Medium sepurated
0.76
0.6
1.26
Well-separated
C2
C72
0.82
0.09
1.19
Medium separated
0.75
0.62
1.22
Medium-sejurates
0.76
0.68
1:13
Medium separated
G75
0.8
0.7
1.15
Modiun-squratol
0.82
0.09
1.18
Modium-eeparated
C17
0.81
1.34
Wall-spiraned
C2
Cr8
0.79
0.71
1.12
Medinm-separaeed
0.78
0.72
1.09
Les-scpurated
C2
0.82
0.09
1.19
Mediun-seuuaaed
0.77
0.69
112
Medium-srjuratel
0.79
0.66
12
Medium-separatsi
0.83
0.1
117
Melhun-sepraled
C2
0.85
0.78
1.00
Less seprsepardted
07
1.4
Methum sepurated
0.8
0.09
1.30
Medium separated
0.82
0.1
1.66
Mediium sepurated
C3
0.82
07
1.88
Mediun-searated
0.84
0.54
136
Far sepurated
0.84
0:72
1.35
Medium--eparaate
0.83
0.64
13
Well selrraed
C3
C10
0.83
0.67
1.25
Well-ceparaeed
C3
Ci
0.34
0.02
135
Well-separaeed
C12
0.81
0.7
1.16
Medium-squrated
C33
0.84
0.03
133
C3
G14
0.82
0.09
1.19
Medium-squratel
C15
0.83
0.09
1.21
Modium-segarated
C16
0.69
1.16
C3
C17
0.85
0.64
1.28
Woll-sepuraeed
C3
C18
0.82
0.69
119
Medium-sepurated
C3
G19
0.82
07
137
Mediun-squratel
C20
0.82
0.69
118
Medium sepurated
0.81
Medium-sejaratti
C3
0.82
07
117
Medhum-sepuraieed
@
0.83
0.68
122
Modiun-sepurtell
C3
C21
0.81
07
1.6
Methum sepuraled
C25
0.82
077
1.37
Medium separated
C25
0.81
0.89
1.17
Mediuu sepurated
C4
0.82
0.1
1.35
Mediun-separated
C9
0.81
0.09
1.7
Medium sepiuasep
0.83
0.59
1.55
Far-sopanmad
G4
C8
0.82
0.71
1.15
Medium separated
C4
C7
0.85
0.64
1.3
Well-aeparted
C4
C70
0:33
0.06
126
Well-ssparaee
29

---

Under review as a conference paper at ICLR 2023
G4
C7
0.82
0.22
1.34
Well-separated
C7
G2
0.09
1.16
Medium sepurated
C4
0.22
0.68
1.31
Well-separated
0:81
0.09
1:18
Medium separated
C4
C75
0.82
0.08
1.2
Mediun-seurated
C4
0.81
0.7
1.16
Medium separated
C17
0.82
0.64
Wall-separated
C7
C18
0.82
0.69
1.18
Mechim--epurated
C4
C10
0.81
0.7
1.16
Melium-separated
C7
C20
0.81
0.69
1.17
Methum sepurated
G4
C21
0.8
0.7
1.16
Medium-eeuurtted
C23
0.82
0.7
1.16
Medium sepuraled
C4
0.82
0.68
121
Meclin--equratel
0.81
0.
1.35
Medium separated
C4
0.82
1.66
Methim Mechim--eprrate sepurated
C
0.81
0.09
1.37
Medium sepurated
0.81
0.79
100
Less Jepuraeprleed
C
0.81
0.0
134
Well-separzed
0.83
0.8
102
Less sepurated
0.82
0.72
Medium Medjum--eparatee separated
0.79
0.67
1.17
Medhum sepurated
C5
0.82
0.71
1.16
Medium-sepurated
C3
C23
0.78
0.7
1.12
Medium separated
0.82
0.77
1.05
Levissparated
0.82
0.71
1.14
Medium separated
Ci6
0.82
0.71
1.15
Mediun-seuratol
65
C17
0.88
0.73
1.21
Medium-separates
C18
0.81
0.72
1.12
Medium-ssparattl
C3
C19
0.79
0.76
1.04
Lesi-separaeed
0.82
0.71
1.16
Melium--epuratel
C23
0.81
1.06
Less sepurated
C5
C34
0.83
0.71
1.14
Mellim--equrated
0.82
0.79
101
Less sepuraled
C5
0.8
0.72
112
Medhin-sepurated
86
0.78
0.58
133
Well Wellaseprrredd
0.82
0.79
100
Lesy-wpurated
Co
C10
0.81
07
1.37
Medium separated
0.75
063
1.19
Medhum sepurated
Co
0.82
0.09
1.18
Medium
C6
0.73
0.61
1.14
Medium sepurated
0.79
0.07
1.37
Medium- Meddum--spparasss
Co
0.81
0.09
1.18
Medium sepurated
C6
0.77
0.67
1.35
Moditn-sequrated
Co
C19
0.77
0.22
1.06
Less separated
C5
C2
0.82
0.09
1.18
Medium-sequrated
0.82
0.
1.16
Medium separated
C16
0.8
0.00
1.15
Modiun-squratel
0.81
0.6
1.34
Well-separated
0.81
9.61
1.33
Woll-srprrned
C7
C70
0.84
0.6
1.41
Far-sepurated
C11
0.81
0.6
135
Woll-weprrted
0.34
0.54
1.56
Fat-sepuratad
0.81
0.61
133
Woll-seprraed
0.79
0.57
1.58
Woll-seprrned
C7
0.81
0.53
134
27
C20
0.83
0.54
135
Far sepuraled
C7
0.83
0.3
143
For sepurated
C7
C18
0.8
0.53
1.31
Far separated
0.82
0.3
14
Well sepratee
C7
0.8
0.58
1338
Well-separated
0.84
0.54
136
Far sepurated
0.81
1336
Well-separated
C7
0.82
0.39
1.4
Well separaed
C7
0.83
0.54
1.59
Par-soganated
C7
C25
0.8
0.39
1.36
Well-separaee
C7
C26
0.84
0.55
1.54
Far-segurattd
C8
0.92
0.81
1.14
Medium Medium-separatee
C8
C10
0.83
0.72
1.14
Medium-sepurttsl
C8
Cil
0.79
0.08
1.37
Modium-segarated
C12
0.82
0.71
1.15
C8
Cy3
0.78
0.7
1.12
Medium-sejarated
C8
0.82
0.78
1.05
Lew-sopuratel
C8
0.82
0.72
134
Mediun-seuratee
C8
0.82
0.1
115
Meliom-
0.87
0:73
1.19
Medium-sejaratti
C8
0.81
0.72
113
Methun-sepurited
88
C79
0.79
0.76
1004
Le4-seuuratol
C8
C20
0.83
0.
100
Less sepuraled
C8
C2F
0.82
0.7
1335
Medium-separaaee separated
C8
C22
0.83
0.1
102
Less sepurated
C22
0.84
0.5
1.05
Less-sparrtod
C8
C21
0.83
0.2
1.33
Medium Mediuunsppurrted
0.82
0.79
Lewt-separatsi
C8
C25
0.81
0.72
1.12
Medium separated
C19
0.88
0.6
1.46
Far-sogunated
C11
0.36
0.58
1.48
Far tur-separated
30

---

Under review as a conference paper at ICLR 2023
C9
C22
0.83
0.44
1.3
Well-separated
C9
C73
0,89
0.0
1:48
Far repurated
C9
C75
0.77
0.08
1.44
Far-separaned
C70
0,64
0.64
1.31
sepluued
0:98
0.71
1.38
Woll-aeparated
C9
C29
0,99
0.09
1.44
Far separated
C21
0.89
0.64
1.3
Wall-separaed
C9
C24
0.9
0.66
1366
Well-sepuraeed
CV
C26
0.86
0.33
1.37
Woll-eepirated
C10
C71
0.83
0.68
122
Medhum sepurated
C10
C12
0.83
0.66
1.25
Woll-separated
C10
0.83
0.69
121
Medium sepurated
C10
0.82
0.7
117
Mechon-sepratedd
C10
0.82
0.65
125
Well Wellasee
C10
0.83
0.66
Wall-wpurated
C10
0.34
0.08
123
Medium separated
C10
0.81
0.66
123
Medhum sepuratedd
C10
0.84
0.09
12
Medium separated
C10
0.83
0.7
118
Medium sepurated
0.84
0.06
Well-separues
C10
0.83
0.71
1.16
Medium sepurated
C30
0.84
0.08
1224
Medium--eeuaaate
C10
C24
0.83
0.07
124
Medium separated
C10
0.82
0.7
1.18
Medium-separated
C10
0.84
0.67
1.26
Well-seprruss
C1)
0.83
0.62
1.35
Woll-cepurate
C11
C33
0:75
0.03
1.2
Medium-separatee
CF1
0.75
0.62
12
C11
CF5
0.8
0.6
1.33
Well-separaeed
CF)
0.83
0.62
1.34
Woll-srparated
C11
0.78
0.6
124
Medhum sepurated
CFI
0.78
0.6
132
Woll-sepurated
C11
0.79
0.65
12
Medium sepurated
C1)
0.76
0.1
121
Mechin-spurated
C11
0.83
0.63
1.35
Well repuraed
C11
0:77
0.65
1.19
Medhum sepurated
C11
0.81
0.04
1.20
Well-separased
0.82
062
131
Well-separated
C1
0.78
0.04
12
Medium separated
C1
0.83
0.63
131
Well separated
0.84
0.4
1332
Well-separuned
C12
0.82
0.09
1.38
Medium sepurated
C12
0.83
0.09
122
Meditm-sequrated
C12
C20
0.8
0.09
1.16
Medium separated
C12
0.82
0.64
1.28
Woll-separated
C12
0.82
0.09
1.19
Medium separatee
C12
G19
0.81
07
1.17
Modiun-squratol
C12
C20
0.81
0.09
1.38
Medium-sepurated
C12
C2
0.81
1.16
C12
C22
0.82
0.7
1.17
Medimm-separated
C12
0.83
0.68
122
Mediun--spuratee
C12
0.81
07
133
Mediun-squrated
C12
0.82
0.7
117
Medium-sepuratee
0.81
0.09
Medium-separatei
0.74
0.61
1115
Medhun-spuraled
C13
0.79
0.61
133
Well Well-sepuraee
0.83
063
132
Wall-wparzsed
C13
0.78
0.04
1221
Medium separated
0.78
0.61
128
Well-seprried
C13
0.78
0.00
1.18
Medium-scppratee
C13
0.74
0.65
1.14
Medium sepurated
0.84
0.43
Well-separated
C13
0.76
0.69
1.33
Medium sepurated
C1
0.8
0.68
1.23
Modiun-scqarated
C33
C24
0.82
0.04
1228
Well-separaeed
CB
C25
0.77
0.67
1.15
Medium-squratsd
C13
C20
0.85
0.04
1229
Well-eepruee
C14
G75
0.8
0.7
1.15
Medium-squratel
C14
C16
0.82
0.09
1.38
Medium-segarated
C14
C18
0.79
0.69
C14
C79
0.78
0.72
1.07
Lew-separated
C14
C21
0.82
0.69
1.19
Medium-sepurated
C14
C24
0.82
07
137
Mediun-squratel
C14
C26
0.8
0.69
115
Melium
CR5
0.82
0.68
121
Medium-seprrttsl
CTS
0.82
061
135
Woll-oeparned
C15
C18
0.8
0.
1.55
Modium-sepuratel
C13
1.14
Methum separaled
C13
C20
0.81
077
1.35
Medium- separated
C15
0.83
0.68
121
Mediuu sepurated
C33
C22
0.81
077
1.1G
Mediun-separated
C13
C23
0.82
0.00
124
Medium separated
C15
C24
0.83
1.19
Mediun--eppaaned
C13
C25
0.82
0.72
1.14
Medium separated
C15
C26
0.82
0.7
1.18
Medium-eeqarated
CJ6
C17
0.2
0.04
128
Well-separned
31

---

Under review as a conference paper at ICLR 2023
C16
C78
0.82
0.09
1.39
Medium--eparated
CJ6
0.82
1.17
Medium sepurated
C16
C20
0:1
0.09
1.18
Medium-separanss
C16
123
0:81
1.16
Medium sepurated
C16
C2
0.82
1.16
Mediun-seurated
C16
C23
0,82
0.08
1.22
Medium-separatts
C16
1.15
Medium-ssparated
C16
C25
0.82
0.7
1.17
Mechum-sppuaated
C16
0.81
0.60
1.17
Modium-sepurate
C17
0.81
1334
Wall-wearrned
C17
C79
0.81
0.7
1.16
Medium-sepuated
C17
0.82
0.64
1228
Well-seprraed
C17
0.83
0.66
127
Wollowepratee
C17
0.81
0.00
124
Medium separated
C18
0:79
0.1
111
Mechum--epuraai
C18
0.79
0.71
1.32
Medium separated
C18
0.82
0.69
1.19
Melhim--epuraal
C38
0.8
0.1
1.33
Mediun-sepurate separated
CR8
0.61
0.6
121
Medium sepurated
0.82
Medium separated
C18
0.83
0.74
1.12
Medhum sepurated
C18
C26
0.81
07
1.35
Medium-sepurated
C39
C20
0.78
0.33
1.06
Less separated
C19
C28
0.82
0.7
1.37
Medium-separated
0.8
0.76
1005
Lest-separated
C19
0.7%
0.73
1.07
Loi-sepuratod
C19
C24
0.82
0.71
1.35
Medium-separated
0.78
0.72
Lev-separatil
C19
C26
0.8
0.7
1:13
Medinm-separated
C30
0.81
0.40
1.18
Melium-sepurated
C20
C21
0.82
1.16
Methum sepurated
C20
C26
0.79
0.7
1.14
Medlim-sepuratel
C21
0.82
1.37
Medium sepurated
C2)
0.83
0.68
122
Mechin-squrated
C21
0.81
0.7
1.35
Medium separated
C21
0.82
117
Methum sepurated
C21
C20
0.81
0.09
1.37
Medium separated
C22
0.83
0.72
1:15
Medhum sepurated
C22
0.71
1.34
Medium separated
C23
0.83
0.7
1.19
Medium sepurated
0.08
Medium separnm-eeparaeed
C24
0.83
0.92
1.15
Medium sepurated
C24
0.82
0.71
1.36
Meditm-sepurate
C20
C20
0.7
1:33
Medium separated
0.73
0.64
1.14
0,82
1002
0.85
0.65
1.32
C/4
0.78
0.71
1.31
C17
0.55
C22
0:87
0.84
1.04
C27
0.53
C25
0.82
0.78
1.05
C2
0.96
0.68
1.42
C20
0:76
0.67
0.94
0.3
6
C20
0.83
0.81
100
0.82
0:1
C6
0.77
1.29
0.78
0.89
1:13
C6
C77
1.15
0.07
1.71
86
C20
0.83
0.74
1.12
0.92
0.3
Co
0.87
0.76
1.35
0.76
0.71
1.08
C9
C14
0.82
0.31
1.6
9
C17
0.98
0.56
1.75
C20
0.63
1.04
1.06
0.01
1.14
0.73
0.57
1.28
C25
1.08
0.87
C14
C17
0.99
0.6
1.64
C14
C20
0.87
0.77
1:13
C14
C12
0.79
0.1
1.31
C14
C23
0.83
0.67
123
C14
0.74
C17
C20
0.55
0:33
C17
C22
1.33
0.5
1:75
C17
1.39
0.39
235
C17
C25
0.88
0.04
1.38
220
0.88
0.3
101
C20
C23
0.86
0.74
1.16
C20
C25
0.82
0.57
100
C22
0.85
0.77
C22
C25
0.8
0.75
1.00
(21
C25
0.81
0.73
1.11
32


Reviews:
[{"review_id": "unmUQV8FEm", "reviewer": "Reviewer_w1Wh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors generated some artificial benchmarks for the tasks of learning from label proportions (LLP) based on the publicly-available Criteo Kaggle CTR dataset.", "summary_of_the_review": "As I have stated before, the main contribution of this submission is to generating some artificial benchmarks for the tasks of learning from label proportions (LLP) based on the publicly-available Criteo Kaggle CTR dataset. Maybe the generated benchmarks are large, but they are still artificial data sets (though generated from real-world binary classification data set). I am not sure that such contribution is deserved to be accepted by ICLR. I'd love to hear from other reviewers.", "strength_and_weaknesses": "Strength:\n1. According to Table 10 in Appendix A.8, most of the generated LLP benchmarks contains many bags (more than one million for some of them) and their characteristics are diversified.\n2. The details of the generation procedure are provided which is convenient for readers.\n3. Some analysis and experiments on these generated LLP benchmarks are also provided.\n\nWeaknesses:\nThe task of learning from label proportions (LLP) is interesting. However, as also stated in this submission: \" they typically evaluated their methods on pseudo-synthetically generated LLP training data using common small scale supervised learning datasets by randomly sampling or partitioning their instances into bags.\".  This submission claims that a large scale LLP benchmark constructed from the Criteo Kaggle CTR dataset is proposed. Maybe the benchmark is really large, but it is also artificially generated from binary classification data set (i.e., Criteo Kaggle CTR dataset), not a real-world LLP data set. I don't know whether this will make sense to generate LLP artificial benchmarks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "pMDyO-qHC1", "reviewer": "Reviewer_A9So", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper created a benchmark dataset for LLP problem, where people would like to train an instance predictor based on bags of features and positive ratios in the bag. The main contribution is to split bags with feature sets, analyze the statistics of bags, and benchmark the performances of different LLP methods.", "summary_of_the_review": "In summary, it's not well-motivated to construct such a dataset. The proposed dataset didn't provide any instruction on what kind of model will be better for LLP. ", "strength_and_weaknesses": "Strength:\n\n- A detailed analysis of the differences among grouping criterion\n\n\nWeakness: \n\n- The paper is incremental from Saket et al. (2022). Basically, it computes some statistics on the different groups with the same dataset in previous work.\n- A key question on how to balance the bag size and performance is not answered. The paper only reports that smaller bag size generally leads to better performance, but didn't quantify how larger bag size harms privacy on the benchmark dataset. Therefore, it's hard to be motivated to choose larger bag sizes.\n- The numbers/ thresholds used in 5.1 seems to be picked without ground.\n- The dataset is too toy. The method performed on this 39-dimensional dataset may not be able to generalize on image/text data.\n- Is there any proof showing that the method performing better on this dataset can be generalized to other datasets?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YehevEf3wUv", "reviewer": "Reviewer_AGL5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submission presents a collection of benchmark datasets for learning from label proportions. It carefully describes how all these datasets were constructed from the original data in the Criteo dataset, by computing bags of instances based on groupings induced by categorical variables in the data. The submission presents a substantial number summary statistics for the constructed benchmark datasets. It also compares a small number of learning algorithms on this data.\n", "summary_of_the_review": "Using a dataset with unknown semantics of the features to create an LLP problem by grouping instances according to these features does not yield a compelling benchmark. Also, only a small number of LLP methods are compared on the benchmark.\n", "strength_and_weaknesses": "The submission purports to introduce a natural large-scale dataset for learning from label proportions. However, given the absence of any information involving the categorical features used to establish bags, regardless of the summary statistics produced, it seems entirely unclear how natural or realistic the constructed data is.\n\nThe number of LLP methods considered in the experiments seems quite limited.\n\nOther comments and questions:\n\n\"large number of mostly artificial methods of forming bags from instance level datasets.\" - why is this a drawback?\n\nPlease define what CTR means.\n\n\"to the partition the groupings \"\n\nIs the squared Euclidean distance calculated based on the numeric features only (for inter- and intra-bag distances)?\n\nWhat is the rationale for performing the 5-fold cross-validation at the instance-level before re-creating bags?\n\n\"techniques such neural\"\n\n\"The model predicts on all the instances in the bags of the minibatch are aggregated\"\n\n\" over the for the\"\n\nWhy not use a distance metric for bags that has been used in the past in machine learning, based on the Hausdorff distance?\n\n\"The categorical features are encoded as non-negative integers.\" - this implies an order\n\n\" it’s embedding layer \"\n\n\"which we transform the instances into this embedding space\"\n\nThe results for SGD do not seem to add anything to the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9_4taP0WsqG", "reviewer": "Reviewer_fQUW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on creating a benchmark as a collection of diverse and large-scale LLP datasets based on the Criteo Kaggle CTR dataset. The authors obtained all bag sets grouped by one or two categorical features and applied two filters on the clipped groupings to choose groupings for model training. They estimated the geometric clustering of bags and qualified the tractability of an LLP dataset by the test AUC scores.", "summary_of_the_review": "The novelty of this paper is insufficient. It still has room for improvement.", "strength_and_weaknesses": "Pros:\n- The authors considered establishing a benchmark dataset for LLP, which did not exist in previous studies.\n\nCons:\n - It is a bit confusing why to choose the Criteo Kaggle CTR dataset and why the dataset groups in this way. First of all, in the Criteo Kaggle CTR dataset, the semantics of these 26 category features are undisclosed and missing for some samples, so the direct use of category features for subcontracting requires a more reasonable explanation. Secondly, why using at most two categorical features for grouping also needs a reasonable explanation.\nIn LLP, the natural grouping of bags can usually be based on geographic location, time point, age group, etc. It may be more appropriate to select a dataset with these kinds of information.\n- In the study of LLP, the performance of various techniques on bags of different sizes is also very important. The reason for removing bags with a size greater than 2500 or less than 50 needs to be explained.\n- Grouping based on category features for the Criteo Kaggle CTR dataset has existed in the work of Saket et al. (2022). This is slightly insufficient as the innovation point of this paper.\n- The authors used the test AUC score to qualify the tractability of the LLP dataset, but the result did not seem to be ideal (AUC scores were centered around 60-80%). It would be nice to add some experiments that use SOTA methods, such as LLP-VAT (Tsai et al. 2020) or MCM (Scott et al. 2020), to illustrate that the dataset makes sense.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject