Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: [Summary]
The paper presents a solution to the problem of learning from label proportion (LLP) by incorporating regularization by data generation motivated by semi-supervised learning.   
It is argued that, by consistency assumption, the classification function defined on the data manifold should be a locally consistent mapping (in the sense of bag label proportion) such that the discrepancy between two bags of data points within a small neighborhood should be constraint. This is further combined with the conventional LLP loss (cross entropy between bag label proportion and classification results) to produce the final loss function for learning. For bag data generation, image attributes are leveraged to group similar data points in the feature space. Evaluation is performed on three benchmarks against vanilla solution and ROT loss. 

[Comments]
I’m not sure if I fully follow the contribution and several technical details.

 It looks to me that the major contribution claimed is the novel loss function that combines the proportion loss and the consistency loss, but both seem to be from off-the-shelf solutions from literature with slight variation. E.g., J_prop is the standard cross entropy loss (Ardehaly & Culotta (2017) and Dulac-Arnold et al. (2019)), and J_cons is from the vanilla consistency definition. I had a hard time getting the novelty here. 

The notion and definition are somehow confusing to me too. What is K in the second line of page 3? Shouldn’t J_prop in page 4 defined as sum of all per-bag losses? In J_cons of pge 4, shouldn’t x under the summation be x_\hat as x_\hat is sampled? The equations should be properly numbered for easy reference.

The use of image attributes in 4.4 for K-means bag generation seems a strong requirement? What kinds of image attributes are used and how are they generated? Looks like this strategy only applies to image classification?

The results reported could be more clear. The gap between the proposed method and vanilla or ROT does not seem quite big in many cases (less than 1% in the best cases in table 1). I’m not sure if these results are convincing or not as statistical significance is unclear.   

With all of the above uncertainty, I do not have confidence to have the paper accepted in the current format based on my preliminary assessment.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary:
This paper proposes using Consistency Regularization and a new bag generation technique to better learn classification decision boundaries in a Label Proportion setting.  The consistency regularization works to make sure that examples in the local neighbourhood have similar outputs. The authors further use K-means clustering to create a new bagging scenario they use to mimic real-world LLP settings. 

Overall, this paper sheds light on how we can use the underlying structure to better be able to make predictions on labels even when we don't have original labels but proportional bags. This is a very interesting problem area and will be interesting to the field.

# For the experiments, a few notes and comments?
1. It's a bit harder to appreciate some of the performance gains here without understanding the error rates around the accuracy. This would be especially good to know for the larger bag sizes where one expects more uncertainty.
2. For each of the datasets, it would have been also enlightening to provide what the normal labelled performance would be as it would give an indication of the limit when the bag sizes get smaller.
3. I am a bit confused by the k-means scenario. With bag sizes similar to the uniform bagging scenario, would the performance not be more as we are already capturing latent structure with the k-means algorithm? 
4. Following up on 3, Does this not have implications for actually preserving some privacy?


# Other Comments
5: It would be interesting to see an approach like mixup can be combined with the consistency concept in this case. So what do you expect when you now combine examples and train the underlying mixed bag objective.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Summary of the paper:  Learning from label proportions (LLP) is an area in machine learning that tries to learn a classifier that predicts labels of instances, with only bag-level aggregated labels given at the training stage.  Instead of proposing a loss specialized for this problem, this paper proposes a regularization term for the LLP problem. The core contribution of this paper is to use the idea of consistency regularization, which has become very popular in semi-supervised learning in the recent years.  The regularization term takes a perturbation of an input sample, and then force the output of the original and perturbed sample to be similar by minimizing  a KL divergence of the two output distributions.  Experiments show the performance of the proposed method under two bag generation settings.  The paper also finds empirically that the hard L_1 has high correlation with the test error rate, which makes it an ideal candidate when the user splits the validation data from training data (meaning there are no ground truth labels for each instances).

Reasons for the decision of the paper:  The proposed consistency regularization term seems to be borrowed directly from semi-supervised learning (SSL) research, and is not really specialized for the LLP problem.  Although the discovery that adding this regularization term for the LLP problem increases generalization performance is novel, using this term for data without ground truth labels in image datasets such as CIFAR10/100 and SVHN itself has low novelty, since these datasets satisfy the smoothness assumption used by consistency regularization methods.  Another issue is that there are only one trial for every experiment, making it hard for the reader to see if the results are statistically significant or not.  For these two reasons, it was hard to give a high score decision for this paper.  However, the paper’s motivation is interesting, and the direction to explore regularization techniques for the label proportion learning problem is important and seems novel for this area.  If the regularization method can be extended in a way that relates to the LLP problem, it would make the paper much stronger.

Other minor comments:

In the experiments, the average test accuracy of the last 10 epochs are reported.  I was curious if the last epochs have better models than earlier epochs.  In many weakly-supervised areas such as noisy labels, it is common that the accuracy goes up very quickly but then gradually decreases.  Does this also happen for the LLP problem?

In J_cons, does the expectation need to be over p?


***After author response:
Thank you for answering my questions!  Although I have a better understanding of the paper now, I still have the same concerns and would like to keep my score.
review_title: Official Blind Review #2

Meta-review: After reading the author's rebuttal, the reviewer still hold that the main contribution is just the simple combination of already known losses. And the paper need to pay more attention on the clarity of the paper.
Decision: Reject