Title: A Unified Approach to Count-Based Weakly Supervised Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: weakly supervised learning

constraint

label proportion

learning from positive and unlabeled data

multiple instance learning
Abstract: High-quality labels are often very scarce, whereas unlabeled data with inferred weak labels occurs more naturally. In many cases, these weak labels dictate the frequency of each respective class over a set of instances. In this paper, we develop a unified approach to learning from such weakly-labeled data, which we call *count-based weakly-supervised learning*. At the heart of our approach is the ability to compute the probability of exactly $k$ out of $n$ outputs being set to true. This computation is differentiable, exact, and efficient. Building upon the previous computation, we derive a *count loss* penalizing the model for deviations in its distribution from an arithmetic constraint defined over label counts.

A Unified Approach to Count-Based Weakly-Supervised Learning

Abstract

High-quality labels are often very scarce, whereas unlabeled data with inferred weak labels occurs more naturally. In many cases, these weak labels dictate the frequency of each respective class over a set of instances. In this paper, we develop a unified approach to learning from such weakly-labeled data, which we call countbased weakly-supervised learning . At the heart of our approach is the ability to compute the probability of exactly k out of n outputs being set to true. This computation is differentiable, exact, and efficient. Building upon the previous computation, we derive a count loss penalizing the model for deviations in its distribution from an arithmetic constraint defined over label counts. We evaluate our approach on three common weakly-supervised learning paradigms and observe that our proposed approach achieves state-of-the-art or highly competitive results across all three of the paradigms.

1 Introduction

Weakly supervised learning [56] enables a model to learn from data with restricted, partial or inaccurate labels, often known as weakly-labeled data . Weakly supervised learning fulfills a need arising in many real-world settings that are subject to privacy or budget constraints, such as privacy sensitive data [45], medical image analysis [12], clinical practice [39], personalized advertisement [9] and knowledge base completion [21, 59], to name a few. In some settings, instance-level labels are unavailable. Instead, instances are grouped into bags with corresponding bag-level labels that are a function of the instance labels, e.g., the proportion of positive labels in a bag. A key insight that we bring forth is that such weak supervision can very often be construed as enforcing constraints on label counts of data .

More concretely, we consider three prominent weakly supervised learning paradigms. The first paradigm is known as learning from label proportions [38]. Here the weak supervision consists in the proportion of positive labels in a given bag, which can be interpreted as the count of positive instances in such a bag. The second paradigm, whose supervision is strictly weaker than the former, is multiple instance learning [35, 17]. Here the bag labels only indicate the existence of at least one positive instance in a bag, which can be recast as to whether the count of positive instances is greater than zero. The third paradigm, learning from positive and unlabeled data [16, 31], grants access to

x

y

0

0

1

1

(a) Classical

{

x

i

}

k

i

=1

˜

y

=

∑

y

i

/k

0

1/3

3/5

(b) LLP

{

x

i

}

k

i

=1

˜

y

{

y

i

}

0

1

1

(c) MIL

x

˜

y

?

1

?

?

(d) PU Learning

the ground truth labels for a subset of only the positive instances , providing only a class prior for what remains. We can recast the class prior as a distribution of the count of positive labels .

Leveraging the view of weak supervision as a constraint on label counts, we utilize a simple, efficient and probabilistically sound approach to weakly-supervised learning. More precisely, we train a neural network to make instance-level predictions that conform to the desired label counts. To this end, we propose a differentiable count loss that characterizes how close the network's distribution comes to the label counts; a loss which is surprisingly tractable. Compared to prior methods, this approach does not approximate probabilities but computes them exactly . Our empirical evaluation demonstrates that our proposed count loss significantly boosts the classification performance on all three aforementioned settings.

2 Problem Formulations

In this section, we formally introduce the aforementioned weakly supervised learning paradigms. For notation, let X ∈ R d be the input feature space over d features and Y = { 0 , 1 } be a binary label space. We write x ∈ X and y ∈ Y for the input and output random variables respectively. Recall that in fully-supervised binary classification, it is assumed that each feature and label pair ( x , y ) ∈ X ×Y is sampled independently from a joint distribution p ( x , y ) . A classifier f is learned to minimize the risk R ( f ) = E ( x ,y ) ∼ p [ ℓ ( f ( x ) , y )] where ℓ : [0 , 1] ×Y → R ≥ 0 is the cross entropy loss function. Typically, the true distribution p ( x , y ) is implicit and cannot be observed. Therefore, a set of n training samples, D = { ( x i , y i ) } n i =1 , is used and the empirical risk, ˆ R ( f ) = 1 n ∑ n i =1 ℓ ( f ( x i ) , y i ) , is minimized in practice. In the count-based weakly supervised learning settings, the supervision is given at a bag level instead of an instance level. We formally introduce these settings as below.

2.1 Learning from Label Proportions

Learning from label proportions (LLP) [38] assumes that each instance in the training set is assigned to bags and only the proportion of positive instances in each bag is known. One example is in light of the coronavirus pandemic, where infection rates were typically reported based on geographical boundaries such as states and counties. Each boundary can be treated as a bag with the infection rate as the proportion annotation.

The goal of LLP is to learn an instance-level classifier f : X → [0 , 1] even though it is trained on baglevel labeled data. Formally, the training dataset consists of m bags, denoted by D = { ( B i , ˜ y i ) } i m =1 where each bag B i = { x j } k j =1 consist of k instances and this k could vary among different bags. The bag proportions are defined as ˜ y i = ∑ k j =1 y j /k with y j being the instance label that cannot be accessed and only ˜ y i is available during training. An example is shown in Figure 1b. We do not assume that the bags are non-overlapping while some existing work suffers from this limitation including Scott and Zhang [40].

2.2 Multiple Instance Learning

Multiple instance learning (MIL) [35, 17] refers to the scenario where the training dataset consists of bags of instances, and labels are provided at bag level. However, in MIL, the bag label is a single binary label indicating whether there is a positive instance in the bag or not as opposed to a bag proportion defined in LLP. A real-world application of MIL lies in the field of drug activity [17]. We can observe the effects of a group of conformations but not for any specific molecule, motivating a MIL setting. Formally, in MIL, the training dataset consists of m bags, denoted by D = { ( B i , ˜ y i ) } i m =1 , with a bag consisting of k instances, i.e., B i = { x j } k j =1 . The size k can vary among different bags. For each instance x j , there exists an instance-level label y j which is not accessible. The bag-level label is defined as ˜ y i = max j { y j } . An example is shown in Figure 1c.

The main goal of MIL is to learn a model that predicts a bag label while a more challenging goal is to learn an instance-level predictor that is able to discover positive instances in a bag. In this work, we aim to tackle both by training an instance-level classifier whose predictions can be combined into a bag-level prediction as the last step.

2.3 Learning from Positive and Unlabeled Data

Learning from positive and unlabeled data or PU learning [16, 31] refers to the setting where the training dataset consists of only positive instances and unlabeled data, and the unlabeled data can contain both positive and negative instances. A motivation of PU learning is persistence in the case of shifts to the negative-class distribution [37], for example, a spam filter. An attacker may alter the properties of a spam email, making a traditional classifier require a new negative dataset [37]. We note that taking a new unlabeled sample would be more efficient, motivating PU learning. Formally, in PU learning, the training dataset D = D p ∪ D u where D p = { ( x i , ˜ y i = 1) } n p i =1 is the set of positive instances with x i from p ( x | y = 1) and ˜ y denoting whether the instance is labeled, and D u = { ( x i , ˜ y i = 0) } n u i =1 the unlabeled set with x i from

where the mixture proportion β := p ( y = 1 | ˜ y = 0) is the fraction of positive instances among the unlabeled population. Although the instance label y is not accessible, its information can be inferred from the binary selection label ˜ y : if the selection label ˜ y = 1 , it belongs to the positively labeled set, i.e., p ( y = 1 | ˜ y = 1) = 1 ; otherwise, the instance x can be either positive or negative. An example of such a dataset is shown in Figure 1d.

The goal of PU learning is to train an instance-level classifier. However, it is not straightforward to learn from PU data and it is necessary to make assumptions to enable learning with positive and unlabeled data [9]. In this work, we make a commonly-used assumption for PU learning, selected completely at random (SCAR) , which lies at the basis of many PU learning methods.

Definition 2.1 ( SCAR ) . Labeled instances are selected completely at random, independent from input features, from the positive distribution p ( x | y = 1) , that is, p (˜ y = 1 | x , y = 1) = p (˜ y = 1 | y = 1) .

3 A Unified Approach: Count Loss

In this section, we derive objectives for the three weakly supervised settings, LLP, MIL, and PU learning, from first principles. Our proposed objectives bridge between neural outputs, which can be observed as counts, and arithmetic constraints derived from the weakly supervised labels. The idea is to capture how close the classifier is to satisfying the arithmetic constraints on its outputs.

Algorithm 1 Count Probability p ( ∑ k i =1 ˆ y i = s )

Input: A set of k log probabilities { t i } k i =1 with t i := log p (ˆ y i = 1) , the number of instances k , and a label sum s

Output:

k

i

=1

i

log probabilities log

p

(

y

ˆ

=

s

∑

Initialize an array A to be -Inf everywhere

Compute log1mexp

for

i

= 1

for return

′

i

t

to

k

do

= 0

s

m

to do

a

-

=

A

[

i

1

, m

] +

t

-

A

[

i, m

A

] =

]

or

[

k, s

′

i

A

[

k,

:]

They can be easily integrated with deep learning models, and allow them to be trained end-to-end. For the three objectives, we show that they share the same computational building block: given k instances { x i } k i =1 and an instance-level classifier f that predicts p (ˆ y i | x i ) with ˆ y denoting the prediction variable, the problem of inferring the probability of the constraint on counts ∑ k i =1 ˆ y i = s is to compute the count probability defined below:

where J · K denotes the indicator function and ˆ y denotes the vector (ˆ y 1 , · · · , ˆ y k ) . For succinctness, we omit the dependency on the input and simply write the count probability as p ( ∑ k i =1 ˆ y i = s ) . Next, we show how the objectives derived from first principles can be solved by using the count probability as an oracle. We summarize all proposed objectives in Table 2. Later, we will show how this seemingly intractable count probability can be efficiently computed by our proposed algorithm.

LLP setting. Given a bag B = { x i } k i =1 of size k and its weakly supervised label ˜ y , by definition, it can be inferred that the number of positive instances (count) in the bag is k ˜ y . Our objective is to minimize the negative log probability -log p ( ∑ i ˆ y i = k ˜ y ) . Notice that when each bag consists of only one instance, that is, when the bag-level supervisions are reduced to instance-level ones, this objective is exactly cross-entropy loss. We further show that our method is risk-consistent, that is, the optimal classifier under our proposed loss provides predictions consistent with the underlying risk as in the supervised learning setting. Details of the risk analysis can be found in Appendix A.

MIL setting. Given a bag B = { x i } k i =1 of size k and a single binary label ˜ y as its weakly supervised label, we propose a cross-entropy loss as below

Notice that in the above loss, the probability term p ( ∑ ˆ y i = 0) is accessible to the oracle for computing count probability, and the other probability term p ( ∑ ˆ y i ≥ 1) can simply be obtained from 1 -p ( ∑ ˆ y i = 0) , i.e., the same call to the oracle since all prediction variables ˆ y i are binary.

PU Learning setting. Recall that for the unlabeled data D u in the training dataset, an unlabeled instance x i is drawn from a mixture distribution as shown in Equation 1 parameterized by a mixture proportion β = p ( y = 1 | ˜ y = 0) . Under the SCAR assumption, even though only a class prior is given, we show that the mixture proportion can be estimated from the dataset.

Proposition 3.1. With SCAR assumption and a class prior α := p ( y = 1) , the mixture proportion β := p ( y = 1 | ˜ y = 0) can be estimated from dataset D .

Proof. First, the label frequency p (˜ y = 1 | y = 1) denoted by c can be obtained by

(

a

+

, a

(

t

i

)

//

log

p

(

-

y

i

)

= 0)

)

or a set of log probability

{

log

p

(

∑

ˆ

y

i

=

s

)

}

k

s

=0

k

i

=1

that is, c = p (˜ y = 1) /α . Notice that p (˜ y = 1) can be estimated from the dataset D by counting the proportion of the labeled instances. Thus, we can estimate the mixture proportion as below,

The probabilistic semantic of the mixture proportion is that if we randomly draw an instance x i from the unlabeled population, the probability that the true label y i is positive would be β . Further, if we randomly draw k instances, the distribution of the summation of the true labels ∑ k i =1 y i conforms to a binomial distribution Bin ( k, β ) parameterized by the mixture proportion β , i.e.,

Based on this observation, we propose an objective to minimize the KL divergence between the distribution of predicted label sum and the binomial distribution parameterized by the mixture proportion for a random subset drawn from the unlabeled population, that is,

where Bin ( s ; k, β ) denotes the probability mass function of the binomial distribution Bin ( k, β ) . Again, the KL divergence can be obtained by k +1 calls to the oracle for computing count probability p ( ∑ k i =1 ˆ y i = s ) . The KL divergence is further combined with a cross entropy defined over labeled data D p as in the classical binary classification training as the overall objective.

As an alternative, we propose an objective for the unlabeled data that requires fewer calls to the oracle: instead of matching the distribution of the predicted label sum with the binomial distribution, this objective matches only the expectations of the two distributions, that is, to maximize p ( ∑ k i =1 ˆ y i = kβ ) where kβ is the expectation of the binomial distribution Bin ( k, β ) . We present empirical evaluations of both proposed objectives in the experimental section.

4 Tractable Computation of Count Probability

In the previous section, we show how the count probability p ( ∑ k i =1 ˆ y i = s ) serves as a computational building block for the objectives derived from first principles for the three weakly supervised learning settings. With a closer look at the count probability, we can see that given a set of instances, the classifier predicts an instance-level probability for each and it requires further manipulation to obtain

count information; actually, the number of joint labelings for the set can be exponential in the number of instances. Intractable as it seems, we show that it is indeed possible to derive a tractable computation for the count probability based on a result from Ahmed et al. [6].

Proposition 4.1. The count probability p ( ∑ k i =1 ˆ y i = s ) of sampling k prediction variables that sums to s from an unconstrained distribution p ( y ) = ∏ k i =1 p (ˆ y i ) can be computed exactly in time O ( ks ) . Moreover, the set { p ( ∑ k i =1 ˆ y i = s ) } k s =0 can also be computed in time O ( k 2 ) .

The above proposition can be proved in a constructive way where we show that the count probability p ( ∑ k i =1 ˆ y i = s ) can be computed in a dynamic programming manner. We provide an illustrative example of this computation in Figure 1. In practice, we implement this computation in log space for numeric stability which we summarized as Algorithm 1, where function log1mexp provides a numerically stable way to compute log1mexp ( x ) = log(1 -exp( x )) and function logsumexp a numerically stable way to compute logsumexp ( x, y ) = log(exp( x ) + exp( y )) . Notice that since we show it is tractable to compute the set { p ( ∑ k i =1 ˆ y i = s ) } k s =0 , for any two given label sum s 1 and s 2 , a count probability p ( s 1 ≤ ∑ i ˆ y i ≤ s 2 ) where the count lies in an interval, can also be exactly and tractably computed. This implies that our tractable computation of count probabilities can potentially be leveraged by other count-based applications besides the three weakly supervised learning settings in the last section.

5 Related Work

Weakly Supervised Learning. Besides settings explored in our work there are many other weaklysupervised settings. One of which is semi-supervised learning, a close relative to PU Learning with the difference being that labeled samples can be both positive and negative [57, 58]. Another is label noise learning, which occurs when our instances are mislabeled. Two common variations involve whether noise is independent or dependent on the instance [20, 42]. A third setting is partial label learning, where each instance is provided a set of labels of which exactly one is true [14]. An extension of this is partial multi-label learning, where among a set of labels, a subset is true [46].

Unified Approaches. There exists some literature in regards to 'general" approaches for weakly supervised learning. One example being the method proposed in Hüllermeier [23], which provides a procedure that minimizes the empirical risk on 'fuzzy' sets of data. The paper also establishes guarantees for model identification and instance-level recognition. Co-Training and Self-Training are also examples of similar techniques that are applicable to a wide variety of weakly supervised settings [11, 49]. Self-training involves progressively incorporating more unlabeled data via our model's prediction (with pseudo-label) and then training a model on more data as an iterative algorithm [25]. Co-Training leverages two models that have different 'views' of the data and iteratively augment each other's training set with samples they deem as 'well-classified'. They are traditionally applied to semi-supervised learning but can extend to multiple instance learning settings [33, 47, 32].

LLP. Quadrianto et al. [38] first introduced an exponential family based approach that used an estimation of mean for each class. Others seek to minimize 'empirical proportion risk' or EPR as in Yu et al. [50], which is centered around creating an instance-level classifier that is able to reproduce the label proportions of each bag. As mentioned previously, more recent methods use bag posterior approximation and neural-based approaches [8, 43]. One such method is Proportion Loss (PL) [43], which we contrast to our approach. This is computed by binary cross entropy between the averaged instance-level probabilities and ground-truth bag proportion.

MIL. MIL finds its earlier approaches with SVMs, which have been used quite prolifically and still remain one of the most common baselines. We start with MI-SVM/mi-SVM [7] which are examples of transductive SVMs [13] that seek a stable instance classification through repeated retraining iterations. MI-SVM is an example of an instance space method [13], which identifies methods that classify instances as a preliminary step in the problem. This is in contrast to bag-space or embedded-space methods that omit the instance classification step. Furthermore, Wang et al. [44] remains one of the hallmarks of the use of neural networks for Multi-Instance Learning. Ilse et al. [24], utilize a similar approach but with attention-based mechanisms.

PU learning. Bekker and Davis [9] groups PU Learning paradigms into three main classes: two step, biased, and class prior incorporation. Biased learning techniques train a classifier on the entire dataset

with the understanding that negative samples are subject to noise [9]. We will focus on a subset of biased learning techniques (Risk Estimators) as they are considered state-of-the-art and relevant to us as baselines. The Unbiased Risk Estimator (uPU) provides an alternative to the inefficiencies in manually biasing unlabeled data [18, 37]. Later, Non-negative Risk Estimator (nnPU) [26] accounted for weaknesses in the unbiased risk estimator such as overfitting.

Count Loss. To our knowledge, viewing the computation of the 'bag posterior' as probabilistic is new. However, the prior approaches do this implicitly. Many approaches have tried to approximate the 'bag posterior' by averaging the instance-level probabilities in a bag [8, 43]. In MIL settings, among instance-level approaches, the MIL-pooling is an implicit 'bag posterior' computation. These include mean, max, and log-sum-exp pooling to approximate the likelihood that a bag has at least one positive instance [44]. But again, these are all approximations of what our computation does exactly . In PU Learning, to our best knowledge, the view of unlabeled data as a bag annotated with the mixture proportion is new.

Neuro-Symbolic Losses. In this paper, we have dealt with a specific form of distributional constraint. Conversely, there has been a plethora of work exploring the integration of hard symbolic constraints into the learning of neural networks. This can take the form of enforcing a hard constraint [3], whereby the network's predictions are guaranteed to satisfy the pre-specified constraints. Or it can take the form of a soft constraint [48, 34, 1, 4, 2, 5] whereby the network is trained with an additional loss term that penalizes the network for placing any probability mass on predictions that violate the constraint. While in this work we focus on discrete linear inequality constraints defined over binary variables, there is existing work focusing on hybrid linear inequality constraints defined over both discrete and continuous variables and their tractability [10, 55, 54]. The development of inference algorithms for such constraints and their applications such as Bayesian deep learning remain an active topic [52, 28, 53, 51].

6 Experiments

In this section, we present a thorough empirical evaluation of our proposed count loss on the three weakly supervised learning problems, LLP , MIL , and PU learning . 1 We refer the readers to the appendix for additional experimental details.

6.1 Learning from Label Proportions

We experiment on two datasets: 1) Adult with 8192 training samples where the task is to predict whether a person makes over 50 k a year or not given personal information as input; 2) Magic Gamma Ray Telescope with 6144 training samples where the task is to predict whether the electromagnetic shower is caused by primary gammas or not given information from the atmospheric Cherenkov gamma telescope [19]. 2

We follow Scott and Zhang [40] where two settings are considered: one with label proportions uniformly on [0 , 1 2 ] and the other uniformly on [ 1 2 , 1] . Additionally, we experiment on a third setting with label proportions distributing uniformly on [0 , 1] which is not considered in Scott and Zhang [40] but is the most natural setting since the label proportion is not biased toward either 0 or 1 . We experiment on four bag sizes n ∈ { 8 , 32 , 128 , 512 } .

Count loss (CL) denotes our proposed approach using the loss objective defined in Table 2 for LLP. We compare our approach with a mutual contamination framework for LLP (LMMCM) [40] and against Proportion Loss (PL) [43].

Results and Discussions We show our results in Table 3. Our method showcases superior results against the baselines on both datasets and variations in bag sizes. Especially in cases with lower bag sizes, i.e., 8 , 32 , CL greatly outperforms all other methodologies. Among our baselines are methods that approximate the bag posterior (PL), which we show to be less effective than optimizing the exact bag posterior with CL.

6.2 Multiple Instance Learning

We first experiment on the MNIST dataset [30] and follow the MIL experimental setting in Ilse et al. [24]: the training and test set bags are randomly sampled from the MNIST training and test set respectively; each bag can have images of digits from 0 to 9 , and bags with the digit 9 are labeled positive. Moreover, the dataset is constructed in a balanced way such that there is an equal amount of positively and negatively labeled bags as in Ilse et al. [24]. The task is to train a classifier that is able to predict bag labels; the more challenging task is to discover key instances , that is, to train a classifier that identifies images of digit 9 . Following Ilse et al. [24], we consider three settings that vary in the bag generation process: in each setting, bags have their sizes generated from a normal distribution being N (10 , 2) , N (50 , 10) , N (100 , 20) respectively. The number of bags in

T

0.80

0.80

0.75

T

0.75

C

AU

0.70

AU

C

0.70

王

0.65

t

e

S

0.65

0.60

0.60

0.55

CL (Ours)

Gated Attention

0.55

CL (Ours)

Gated Attention

0.50

Attention

0.50

Attention

10

15

20

25

30

35

40

10

15

20

25

30

35

40

Number of Bags

Number of Bags

training set n is in { 50 , 100 , 150 , 200 , 300 , 400 , 500 } . Thus, we have 3 × 7 = 21 settings in total. Additionally, we introduce experimental analysis on how the performance of the learning methods would degrade as the number of bags and total samples in training set decreases , by modulating the number of training bags n to be { 10 , 20 , 30 , 40 } and selecting bag sizes from N (5 , 1) and N (10 , 2) .

We also experiment on the Colon Cancer dataset [41] to simulate a setting where bag instances are not independent. The dataset consists of 100 total hematoxylin-eosin (H&E) stained images, each of which contains images of cell nuclei that are classified as one of: epithelial, inflammatory, fibroblast, and miscellaneous. Each image represents a bag and instances are 27 × 27 patches extracted from the original image. A

<

10

-

6

0

.

9997

<

10

-

6

<

10

-

6

6

×

10

-

6

<

10

-

6

<

10

-

6

<

10

-

6

<

10

-

6

<

10

-

6

<

10

-

6

1

.

0000

<

10

-

6

7

positively labeled bag or image is one that contains the epithelial nuclei. For both datasets, we include the Attention and Gated Attention mechanism [24] as baselines. We also use the MIL objective defined in Table 2.

Results and Discussions For the MNIST experiments, CL is able to outperform all other baselines or exhibit highly comparable performance for bag-level predictions as shown in Table 4. A more interesting setting is to compare how robust the learning methods are if the number of training bags decreases. Wang et al. [44] claim that instance-level classifiers tend to lose against embedding-based methods. However, we show in our experiment that this is not true in all cases as seen in Figure 2. While Attention and Gated Attention are based on embedding, they suffer from a more severe drop in predictive performance than CL when the number of training bags drops from 40 to 10 ; our method shows great robustness and consistently outperforms all baselines. The rationale we provide is that with a lower number of training instances, we need more supervision over the limited samples we have. Our constraint provides this additional supervision, which accounts for the difference in performance.

We provide an additional investigation in Figure 3 to show that our approach learns effectively and delivers accurate instance-level predictions under bag-level supervision. In Figure 3, we can see that even though the classifier is trained on feedback about whether a bag contains the digit 9 or not, it accurately discovers all images of digit 9 . To reinforce this, Table 7 and Table 8, in Appendix B, show that our approach outperforms existing instance-space methods on instance-level classification.

Our experimental results on the Colon Cancer dataset are shown in Table 5. We show that both our proposed objectives are able to consistently outperform baseline methods on all metrics. Interestingly, we do not expect CL to perform well when instances in a bag are dependent; however, the results indicate that our count loss is robust to these settings.

6.3 Learning from Positive and Unlabeled Data

We experiment on dataset MNIST and CIFAR10 [29], following the four simulated settings from Garg et al. [22]: 1) Binarized MNIST: the training set consist of images of digits 0 -9 and images with digits in range [0 , 4] are positive instances while others as negative; 2) MNIST17: the training set consist of images of digits 1 and 7 and images with digit 1 are defined as positive while 7 as negative; 3) Binarized CIFAR: the training set consists of images from ten classes and images from the first five classes is defined as positive instances while others as negative; 4) CIFAR Cat vs. Dog: the training set consist of images of cats and dogs and images of cats are defined as positive while dogs as negative. The mixture proportion is 0 . 5 in all experiments. The performance is evaluated using the accuracy on a test set of unlabeled data.

CVIR

0.25

CL(Ours)

Binomial(100,0.5)

0.20

ability

Probab

0.15

0.10

0.05

0.00

30

35

40

45

50

55

60

65

70

Count

of counts is also shown. As shown in Table 2, we propose two objectives for PU learning. Our first objective is denoted by CL whereas the second approach is denoted by CL-expect. We compare against the Conditional Value Ignoring Risk approach (CVIR) [22], nnPU [26], and uPU [37].

Results and Discussions Accuracy results are presented in Table 6 where we can see that our proposed methods perform better than baselines on 3 out of the 4 simulated PU learning settings. CL-expect builds off a similar 'exactly-k' count approach, which we have shown to work well in the label proportion setting. The more interesting results are from CL where we fully leverage the information from a distribution as supervision instead of simply using the expectation. We think of this as applying a loss on each count weighted by their probabilities from the binomial distribution. We provide further evidence that our proposed count loss effectively guides the classifier towards predicting a binomial distribution as shown in Figure 4: we plot the count distributions predicted by CL and CVIR as well as the ground-truth binomial distribution. We can see that CL is able to generate the expected distribution, proving the efficacy of our approach.

7 Conclusions

In this paper, we present a unified approach to several weakly-supervised tasks, i.e., LLP, MIL, PU. We construct our approach based on the idea of using weak labels to constrain count-based probabilities computed from model outputs. A future direction for our work can be to extend to multi-class classification as well as explore the applicability to other weakly-supervised settings, e.g. label noise learning, semi-supervised learning, and partial label learning [15, 36, 58].

Acknowledgments

We would like to thank Yuhang Fan for helpful discussions. This work was funded in part by the DARPA PTG Program under award HR00112220005, the DARPA ANSR program under award FA8750-23-2-0004, NSF grants #IIS-1943641, #IIS-1956441, #CCF-1837129, and a gift from RelationalAI. GVdB discloses a financial interest in RelationalAI. ZZ is supported by an Amazon Doctoral Student Fellowship.

References

Kareem Ahmed, Eric Wang, Kai-Wei Chang, and Guy Van den Broeck. Leveraging unlabeled data for entity-relation extraction through probabilistic constraint satisfaction, mar 2021.

Kareem Ahmed, Tao Li, Thy Ton, Quan Guo, Kai-Wei Chang, Parisa Kordjamshidi, Vivek Srikumar, Guy Van den Broeck, and Sameer Singh. Pylon: A pytorch framework for learning with constraints. In Proceedings of the 36th AAAI Conference on Artificial Intelligence (Demo Track) , feb 2022.

Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Broeck, and Antonio Vergari. Semantic probabilistic layers for neuro-symbolic learning. In Advances in Neural Information Processing Systems 35 (NeurIPS) , dec 2022.

Kareem Ahmed, Eric Wang, Kai-Wei Chang, and Guy Van den Broeck. Neuro-symbolic entropy regularization. In The 38th Conference on Uncertainty in Artificial Intelligence , 2022.

Kareem Ahmed, Kai-Wei Chang, and Guy Van den Broeck. Semantic strengthening of neurosymbolic learning. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS) , apr 2023.

Kareem Ahmed, Zhe Zeng, Mathias Niepert, and Guy Van den Broeck. Simple: A gradient estimator for k-subset sampling. In Proceedings of the International Conference on Learning Representations (ICLR) , may 2023.

Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann. Support vector machines for multiple-instance learning. In NIPS , 2002.

Ehsan Mohammady Ardehaly and Aron Culotta. Co-training for demographic classification using deep learning from label proportions. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW) , pages 1017-1024. IEEE, 2017.

Jessa Bekker and Jesse Davis. Learning from positive and unlabeled data: A survey. Machine Learning , 109:719-760, 2020.

Vaishak Belle, Andrea Passerini, and Guy Van den Broeck. Probabilistic inference in hybrid domains by weighted model integration. In Proceedings of IJCAI , pages 2770-2776, 2015.

Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory , COLT' 98, page 92-100, New York, NY, USA, 1998. Association for Computing Machinery. ISBN 1581130570. doi: 10.1145/279943.279962. URL https://doi.org/10.1145/279943. 279962 .

Gerda Bortsova, Florian Dubost, Silas Ørting, Ioannis Katramados, Laurens Hogeweg, Laura Thomsen, Mathilde Wille, and Marleen de Bruijne. Deep learning from label proportions for emphysema quantification. In Medical Image Computing and Computer Assisted InterventionMICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11 , pages 768-776. Springer, 2018.

Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition , 2018.

Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. The Journal of Machine Learning Research , 12:1501-1536, 2011.

Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. J. Mach. Learn. Res. , 12(null):1501-1536, jul 2011. ISSN 1532-4435.

Francesco De Comité, François Denis, Rémi Gilleron, and Fabien Letouzey. Positive and unlabeled examples help learning. pages 219-230, 12 1999. ISBN 978-3-540-66748-3. doi: 10.1007/3-540-46769-6_18.

Thomas Dietterich, Richard Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis-parallel rectangles. Artificial Intelligence , 89:31-71, 03 2001. doi: 10.1016/ S0004-3702(96)00034-3.

Marthinus C du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014.

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive. ics.uci.edu/ml .

Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems , 25(5):845-869, 2013.

Luis Galárraga, Christina Teflioudi, Katja Hose, and Fabian M Suchanek. Fast rule mining in ontological knowledge bases with amie++. The VLDB Journal , 24(6):707-730, 2015.

Saurabh Garg, Yifan Wu, Alexander J Smola, Sivaraman Balakrishnan, and Zachary Lipton. Mixture proportion estimation and pu learning: a modern approach. Advances in Neural Information Processing Systems , 34:8532-8544, 2021.

Eyke Hüllermeier. Learning from imprecise and fuzzy observations: Data disambiguation through generalized loss minimization. International Journal of Approximate Reasoning , 55(7): 1519-1534, 2014.

Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning , pages 2127-2136. PMLR, 2018.

Giannis Karamanolakis, Subhabrata Mukherjee, Guoqing Zheng, and Ahmed Hassan Awadallah. Self-training with weak supervision. arXiv preprint arXiv:2104.05514 , 2021.

Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. Positive-unlabeled learning with non-negative risk estimator. Advances in neural information processing systems , 30, 2017.

Ryoma Kobayashi, Yusuke Mukuta, and Tatsuya Harada. Learning from label proportions with instance-wise consistency. arXiv preprint arXiv:2203.12836 , 2022.

Samuel Kolb, Paolo Morettin, Pedro Zuidberg Dos Martires, Francesco Sommavilla, Andrea Passerini, Roberto Sebastiani, and Luc De Raedt. The pywmi framework and toolbox for probabilistic inference using weighted model integration. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI , pages 6530-6532, 7 2019. doi: 10.24963/ijcai.2019/946.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.

Yann LeCun. The mnist database of handwritten digits. 1998.

Fabien Letouzey, François Denis, and Rémi Gilleron. Learning From Positive and Unlabeled examples. In Proceedings of the 11th International Conference on Algorithmic Learning Theory, ALT'00 , pages 71-85, Sydney, Australia, 2000. Springer Verlag.

Kangning Liu, Weicheng Zhu, Yiqiu Shen, Sheng Liu, Narges Razavian, Krzysztof J Geras, and Carlos Fernandez-Granda. Multiple instance learning via iterative self-paced supervised contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3355-3365, 2023.

Huchuan Lu, Qiuhong Zhou, Dong Wang, and Ruan Xiang. A co-training framework for visual tracking with multiple instance learning. In 2011 IEEE International Conference on Automatic Face & Gesture Recognition (FG) , pages 539-544, 2011. doi: 10.1109/FG.2011.5771455.

Robin Manhaeve, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. Deepproblog: Neural probabilistic logic programming. In NeurIPS , 2018.

Oded Maron and Tomás Lozano-Pérez. A framework for multiple-instance learning. In M. Jordan, M. Kearns, and S. Solla, editors, Advances in Neural Information Processing Systems , volume 10. MIT Press, 1997.

Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013.

Marthinus Du Plessis, Gang Niu, and Masashi Sugiyama. Convex formulation for learning from positive and unlabeled data. In Proceedings of the 32nd International Conference on Machine Learning , volume 37, 2015.

Novi Quadrianto, Alex Smola, Tibério Caetano, and Quoc Le. Estimating labels from label proportions. 2008.

Gwenolé Quellec, Guy Cazuguel, Béatrice Cochener, and Mathieu Lamard. Multiple-instance learning for medical image and video analysis. IEEE reviews in biomedical engineering .

Clayton Scott and Jianxin Zhang. Learning from label proportions: A mutual contamination framework. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 22256-22267. Curran Associates, Inc., 2020.

Korsuk Sirinukunwattana, Shan e Ahmed Raza, Yee Tsang, David Snead, Ian Cree, and Nasir Rajpoot. Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Transactions on Medical Imaging , 35:1-1, 02 2016. doi: 10.1109/TMI.2016.2525803.

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems , 2022.

Kuen-Han Tsai and Hsuan-Tien Lin. Learning from label proportions with consistency regularization. In Asian Conference on Machine Learning , pages 513-528. PMLR, 2020.

Xinggang Wang, Yongluan Yan, Peng Tang, Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks. Pattern Recognition , 74:15-24, 2018.

Janusz Wojtusiak, Katherine Irvin, Aybike Birerdinc, and Ancha V Baranova. Using published medical results and non-homogenous data in rule learning. In 2011 10th International Conference on Machine Learning and Applications and Workshops , volume 2, pages 84-89. IEEE, 2011.

Ming-Kun Xie and Sheng-Jun Huang. Partial multi-label learning. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Chang Xu, Dacheng Tao, and Chao Xu. A survey on multi-view learning. arXiv preprint arXiv:1304.5634 , 2013.

Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Van den Broeck. A semantic loss function for deep learning with symbolic knowledge. In International conference on machine learning , pages 5502-5511. PMLR, 2018.

David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In Proceedings of the 33rd Annual Meeting on Association for Computational Linguistics , ACL '95, page 189-196, USA, 1995. Association for Computational Linguistics. doi: 10.3115/ 981658.981684. URL https://doi.org/10.3115/981658.981684 .

Felix X Yu, Krzysztof Choromanski, Sanjiv Kumar, Tony Jebara, and Shih-Fu Chang. On learning from label proportions. arXiv preprint arXiv:1402.5902 , 2014.

Zhe Zeng and Guy Van den Broeck. Collapsed inference for bayesian deep learning. arXiv preprint arXiv:2306.09686 , 2023.

Zhe Zeng and Guy Van den Broeck. Efficient search-based weighted model integration. Proceedings of UAI , 2019.

Zhe Zeng, Paolo Morettin, Fanqi Yan, Antonio Vergari, and Guy Van den Broeck. Scaling up hybrid probabilistic inference with logical and arithmetic constraints via message passing. In Proceedings of the International Conference of Machine Learning (ICML) , 2020.

Zhe Zeng, Paolo Morettin, Fanqi Yan, Antonio Vergari, and Guy Van den Broeck. Probabilistic inference with algebraic constraints: Theoretical limits and practical approximations. Advances in Neural Information Processing Systems , 33:11564-11575, 2020.

Zhe Zeng, Paolo Morettin, Fanqi Yan, Antonio Vergari, and Guy Van den Broeck. Is parameter learning via weighted model integration tractable? In Proceedings of the UAI Workshop on Tractable Probabilistic Modeling (TPM) , jul 2021.

Zhi-Hua Zhou. A brief introduction to weakly supervised learning. National science review , 5 (1):44-53, 2018.

Xiaojin Zhu and Andrew B Goldberg. Introduction to semi-supervised learning . Springer Nature, 2022.

Xiaojin Jerry Zhu. Semi-supervised learning literature survey. 2005.

Kaja Zupanc and Jesse Davis. Estimating rule quality for knowledge base completion with the relationship between coverage assumption. In Proceedings of the 2018 World Wide Web Conference , pages 1073-1081, 2018.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a unified loss for different weakly supervised paradigms: learning from label proportions (LLP), multiple instance learning (MIL) and learning from positive and unlabeled data (PU).\nThe main idea is to compute the probability of a count sum constraint by summing over all possible combination of events compatible with the constraint (e.g.: exactly k out of n outputs being set to true). An efficient dynamic programming algorithm is given to compute the distribution of the count. It is demonstrated empirically that the count loss can be used to train models in the mentioned weakly supervised settings.\n\nSTRENGTHS:\nThe unification of different weakly supervised paradigms under the count loss is a novel contribution. The dynamic programming algorithm provided make the efficient computation of the loss possible.\nThe paper is easy to follow. The three weakly supervised problems addressed by the paper in details (and potentially more, where the loss can be applied) are well researched and have many real world application.\n\nWEAKNESSES:\nIn its current form the loss is defined only for binary classification problems, while alternatives like probabilistic logic based solutions are applicable to multi class settings as well.\nFor example we know that on an image (our bag) there are 5 object, 2 cats, 2 dogs, and 1 apple. In defense of the paper however, It does not seem too complicated  to extend to this case.\n\nThe novelty of the loss can be questioned. Probabilistic logic can be used to formulate this and more general losses, and this is actually used in deep learning see [1,2]. For example [1] describes the supervision of MNIST digit bags with their sum, this is clearly more general, as in binary case will give back the count loss. Naively evaluating the probabilistic program computing this probability is of course inefficient, however there are techniques (tabled execuition, memoization) to reuse results of sub-problems quite similarly to dynamic programming. I am not sure if your dynamical programming method exactly corresponds to any of the methods used in probabilistic logic programming, but claiming superiority in computational efficiency would require comparison. \n\nNote that I do not question the novelty of the unified view on LLP, MIL and PU. \n\n[1] Manhaeve, Robin, Sebastijan Dumancic, Angelika Kimmig, Thomas Demeester, and Luc De Raedt. \"Deepproblog: Neural probabilistic logic programming.\" Advances in neural information processing systems 31 (2018).\n\n[2] Oldenhof, Martijn, Adam Arany, Yves Moreau, and Edward De Brouwer. \"Weakly Supervised Knowledge Transfer with Probabilistic Logical Reasoning for Object Detection.\" In The Eleventh International Conference on Learning Representations. 2023. (**NOTE:** this is contemporaneous with your submission, and as such you are not required to be aware of it, **I am ignoring the content of this specific paper when evaluating novelty.**)\n\nQUESTIONS:\nYou defined the problem for binary classification,  how difficult would it be to generalize to multiple classes? I assume the main question is the applicability of the algorithm for this case.\n\nHow much time the loss computation and the backpropagation through the dynamics programming graph takes proportionally in the experiments. Is it significantly slower to compute than the traditional losses you compare with?\n\nPlease do not only bold numerically equal mean values as with 3 repeats there is no difference for example between 0.818 ± 0.024 and 0.807 ± 0.026 (in Table 4).  Check the significance of the results, but at least as a shortcut method bold multiple results if they are so close that their variance significantly overlaps not only if their mean precisely equal. This would considerably help the job of the reviewer and the reader.\n\nCount probability is sometimes computed exactly with probabilistic logic reasoning systems. Would be quite valuable to compare runtime with a probabilistic logic engine using tabled execution. I would execute simply a few case with high $n$ to see computational times.\nI also suggest to mention the connection with logical inference in the paper. This gives for example a \"possible world\" semantic for the sum in the objective, containing the indicator variable.\n\nNOTE: After the authors rebuttal: the clarification of the contribution. and the discussion on the logic programming solution, I raised the score by one.\n\nLIMITATIONS:\nThe author acknowledges some limitations, like that it is not currently working for multi-class cases, but there are no detailed discussion on limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a unified framework for weakly-supervised binary classificasion, where instances can be considered as bags and their weak supervision can be seen as count of positive instances in each bag.\nThe framework is based on the probability of the count and the authors derived loss functions based on it for three different weakly-supervised settings, Learning from label proportions (LLP), multiple instance learning (MLP), and Positive and unlabeled learning (PU). The direct handling count is reasonable and the information of bag size is incorporated as supervision in contrast to Proportion Loss (PL).\nExperimental results on multiple datasets demonstrated that the proposed method performs better than existing methods.\n\nSTRENGTHS:\n- The proposed unified approach for count-based weakly-supervised learning is novel, and the direct handling count in their formulation is reasonable. The solution is exact, not approximation.\n\n- The paper is well-written and easy to follow.\n\n- Experimental results on multiple datasets demonstrated the effectiveness of the proposed method.\n\nWEAKNESSES:\n### Originality:\n- It is better to discuss concrete descriptions of how the existing work trains models.\n\n### Clarity:\n- In p.2, for the definition of a set of training data, n should be defined explicitly.\n- Table 1 should be Figure 1.\n- Table 1-a should be described in the main text.\n- In eq at l.4 from the bottom in p.5, there is a redundant bracket.\n- On p.8, it is better to clarify how to generate discrete bag sizes from the normal distribution.\n- Figures are better to place on top.\n- Positions of proposed methods are inconsistent among tables.\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nThis work focuses on binary classification, and enhancement for multi-class cases is discussed as their future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a unified approach for three problems of weak supervision: LLP, MIL, and PU learning. The approach is based on estimation of a common computational block – count probability. For its estimation, the dynamic programming method is used. The method is compared in three problem scenarios with different baselines.\n\nSTRENGTHS:\nDeveloping accurate models for learning with weak supervision (LLP, MIL) is a challenging and important problem. Different weak supervision scenarios share commonalities in their problem formulation, and this paper clearly outlines the common part, and the main bottleneck — the count probability. Having count probability estimated, one can plug it into the scenario-dependent objective.\n\nOverall, the paper is written very clearly and easy to follow, provides a good overview of weak supervision scenarios and unifies them under the same umbrella of optimization problems.\n\nWEAKNESSES:\nThe main part I missed in the paper is the training algorithm. The paper describes in detail the inference part of it: given the estimations of instance-probabilities $p(y_i|x_i)$ it shows how one can compute the count probability. But what I missed is how this is used in the optimization part for training, how are the gradients propagated from the objective to the instance level? It might be straightforward, but in my opinion including the pseudocode/explanation of the training method is essential to understand the details. Without that, I failed to understand the training method, and therefore the value of the paper.\n\n----\nRaised my score after reading the rebuttal.\n\nQUESTIONS:\nMy main question is how the training algorithm actually works, and can the authors provide a pseudocode for that?\n\nLIMITATIONS:\nIt would be great to see the runtime depending on the bag size, in other words is there any computational limitation to the approach related to the parameters of the problem?", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposes a differentiable count loss for three weak supervision settings: LLP, MIL, PU learning. The paper provides a clear explanation of the proposed loss, demonstrates strong empirical performance, and highlights the efficiency of its implementation using dynamic programming.\n\nSTRENGTHS:\n1. The paper provides a clear explanation of the proposed differentiable count loss, making it easier for readers to understand the methodology and its application in weak supervision settings.\n\n2. The paper demonstrates strong empirical performance across various tasks and datasets. This provides evidence of the effectiveness of the proposed loss and enhances the credibility of the research.\n\n3. The authors showcase the efficiency of their implementation by utilizing dynamic programming. This highlights the practicality and computational advantages of the proposed approach.\n\nWEAKNESSES:\n1. The paper could benefit from a clearer articulation of its contribution. Providing a more explicit comparison and emphasizing the novelty of the proposed method would strengthen the paper; The author mentioned that prior works (Ardehaly&Culotta and Tsi&Lin) are motivated by the same principle. More comparison to the prior work would be helpful.\n\n2. The paper lacks sufficient citations to related works; in the related work section “there exists some literature …, for example …” Expanding the reference list and acknowledging existing literature, would enhance the paper's academic rigor.\n\nQUESTIONS:\n1. \"We also perform risk analysis with details in Appendix?\" Could you specify which appendix contains the details of the risk analysis?\n\n2.  Bottom of page 5: There appears to be a typo in the expression \"p(s_1 ≤ ∑_i y_i) ≤ s_2)\". Please correct this typo.\n\n3. In Proposition 4.1, it is unclear whether the dynamic programming approach is proposed or if it is directly using the result from Ahmed[1]. Please clarify this point.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a unification of three different kinds of weak supervision, viz., Learning from label proportions (LLP), Multi-instance learning (MIL), and learning from positive and unlabeled data (PU learning). They all involve supervision at the bag level instead of the instance level (except for PU learning, where some positive instances are also given). \nThe unification comes from the fact that in all three cases, we need to compute the probability distribution of a sum of binary variables using the probability distribution over each binary variable. To do so efficiently, the authors use the dynamic-programming algorithm proposed by Ahmed et al. 23. Once the distribution over sum is computed, it can easily be used to define the likelihood-based loss function for each of the three scenarios. For example, for LLP, one would directly maximize $log Pr(\\sum_{j \\in B_i} y_j = k_i)$ for a given $(B_i, k_i)$ train sample, and efficient computation of $log Pr(\\sum_{j \\in B_i} y_j = k_i)$ is done using DP algorithm proposed by Ahmed et al. 23\n\nSTRENGTHS:\nThe paper is very well written, and the technique is clearly explained with all the assumptions and proofs.\n\nI like the idea of computing the exact probabilities for the sum of binary variables and using it to define the loss function for the three scenarios of weak supervision considered in the paper.\n\nWEAKNESSES:\n1. The crux of the paper lies in computing the probability over the sum of binary variables, which has been borrowed from Ahmed et al., 23. Extending the idea to apply it to the scenario of LLP, MIL, and PU is natural and not that novel.\n\n2. The related works section should also cover other paradigms of weak supervision, e.g. partial label learning.\n\nQUESTIONS:\n1. The paper considers the scenario of only binary classification under weak supervision, whereas the evaluation is done against a more generic method (Tsai et al., 20). Is it possible to extend count loss to multi-class classification? Maybe, by considering it as a binary classification task with an additional one-hot constraint that can be enforced using the semantic loss function defined in Xu et al.,17, or other similar techniques.\n\n2. In the MIL experiment, observing the instance-level accuracy obtained by your method will be interesting. \n\n[Xu et al 17]: A Semantic Loss Function for Deep Learning with Symbolic Knowledge\n\nLIMITATIONS:\nThe authors have sufficiently addressed the limitations of their work and also discuss the positive impact of weak supervision on the privacy of the subjects whose data is collected for training.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}