Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper discussed an approach to take priors into account in machine learning with a slightly different formula, which I have doubts on its correctness. In the experiments, they showed that performance in some tasks can be significantly improved by cleverly choosing class priors.

main_review: This paper discussed an approach to take priors into account in machine learning with a slightly different formula, which I have doubts on its correctness. In the experiments, they showed that performance in some tasks can be significantly improved by cleverly choosing class priors. The experiment results look good, but I do not think it fully justifies their algorithm because of the lack of baselines and inappropriate interpretations of an equation.

The authors mentioned a lot of different types of priors, however many of them (including most of those used in their experiments) could be characterized as cost-sensitive learning, a.k.a. taking priors into account in classification problems. In this regard, the citations to cost-sensitive learning literature and comparisons are vastly insufficient. e.g. papers that should be cited include:

Zadrozny et al. Cost-sensitive learning by cost-proportionate example weighting. ICDM 2003

Branco et al. A survey of predictive modeling on imbalanced domains. ACM Computing Surveys 2016 is a recent survey paper on cost-sensitive learning.

They did not compare against any prior work on cost-sensitive classification, which is a major problem that lead me to the decision of rejection. A recent popular general approach to deal with class priors is focal loss, which should be cited and compared as well.
 
Lin et al. Focal Loss for Dense Object Detection. ICCV 2017

Despite the authors claims, eq. (4) still seems fundamentally flawed in that in the case of 50%/50% 2-class classification problems, the classifier could have entirely flipped one class with another and still optimize eq. (4). I believe what is working is NOT eq. (4), but the practice of applying eq. (4) on mini-batches. Because each mini-batch is much smaller, it becomes difficult to keep the balance in a mini-batch and the multiple resampling of each mini-batch from different epochs may lead to the effect of normalization, similar to the effects that we can see in minibatch regularization in the training of GANs. Hence, I have strong doubts that similar balance-breaking would not necessarily happen so reliably if mini-batches are not used.

The choice between QR and RQ is quite arbitrary in different experiments. Is there any principle about it? Or just trial-and-error?

There is an equation typo in eq. (4), the log-sum term should be log(\sum_i q_i(l)), not log(\sum_l q_i(l)). The interpretation in the subsequent paragraph is although correct in that it favors predicting a high-confidence label.

Minor: The authors should mention that q_i(l) is an energy function (logits) instead of a probabilistic distribution, otherwise they should have \sum_j q_j(l) = 1 and the derivation would be in a different place (e.g. the \sum_j q_j term won't exist).

summary_of_the_review: This paper discussed an approach to take priors into account in machine learning with a slightly different formula, which I have doubts on its correctness. In the experiments, they showed that performance in some tasks can be significantly improved by cleverly choosing class priors. The experiment results look good, but I do not think it fully justifies their algorithm because of the lack of baselines and inappropriate interpretations of an equation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a weakly supervised learning strategy, which exploits instance labels in a form of label prior distributions for training classifiers. The main idea of this work is to build an implicit generative model from a probabilistic label prediction network, which is then trained with an ELBO loss. The paper applies this framework to several types of weakly supervised learning scenarios, including classification with negative labels or labels from ranking, semantic segmentation and text classification from coarse labels, and other tasks with structured label priors, etc.  The experimental evaluation validates the efficacy of the proposed learning strategy on the aforementioned tasks with comparisons to the prior works. 
main_review: Pros:
- This work considers a general form of weakly-supervised problems, which can be applied to a wide range of scenarios, including learning from partial or coarse labels. 
- The paper proposes a new strategy by constructing an implicit generative model that incorporates the weak label priors in a unified manner.
- On a range of vision and text classification tasks, the proposed method seems to achieve competitive performances with higher efficiency. 

Concerns:
- The proposed implicit generative model seems expensive to compute due to its normalization factor. As shown in Equation 2, the forward probability requires a summation over the entire training dataset, which is also used in the loss in Equation 4. It is unclear how to implement this in a mini-batch mode. Note the summation within the log should be over i instead of l. 
- As discussed in the paper, there are potentially different degenerate solutions for the loss function, and it is unclear in which conditions those solutions can be avoided. The strategy of breaking symmetries used in this work seems a bit ad hoc and no general guideline is provided. 
-  The motivation for adopting the RQ loss is unclear. Unlike the QR objective, the RQ loss does not correspond to any properly-defined loss function. More importantly, it seems difficult to choose which one to use in a specific task. In this paper, RQ is adopted for partial labels and text classification, while QR is more effective for land cover segmentation.
- The experimental evaluation seems less convincing due to the following reasons:
1) For the partial label task in 4.1, the improvement over a simple NLL baseline is marginal, as shown in Figure 3 top. 
2) For the learning from rank task in 4.2, it lacks comparisons with other learning to rank learning objectives.
3) For the land cover segmentation task in 4.3, it also lacks comparisons with a simple baseline trained with pseudo/inferred labeling. 
4) For the learned prior task in 4.4, the pre-training seems to play a key role and it is unclear how important the proposed strategy is. The performance is much worse if the network is trained from scratch.  
5) For the text classification in 4.5, it lacks detailed ablative study for the proposed model. It is unclear whether the improvement is due to a better prior adopted in this work or the RQ loss. 
summary_of_the_review: The proposed implicit generative model seems interesting and achieves good performance for a range of weakly-supervised tasks. However, I am slightly leaning towards the negative side due to the lack of clarity in the model design and missing comparisons in the experimental evaluation.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents an approach base on implicit generative models to address a number of existing (and potentially new) scenarios, including weak labels (noisy or incomplete), likelihoods given by a different prediction mechanism on auxiliary input,
or priors reflecting knowledge about the structure of the problem. The experiment presents a selection of very diverse case studies.
main_review: The paper addresses an impressive range of learning settings under the same formulation. The paper is clear and well-presented.

On the positive side, I like how the authors connect fields that are in general not address under the same umbrella frameworks, which makes their approach very complete and appealing. 

Also, I wanted to highlight the very thorough experimental design and analysis. The choice of case studies is very adequate and in some cases, inspiring.

On the negative side, I would have liked to have seen a more elaborate discussion of the relationship with existing methods for each of the problems the paper addresses. The literature review lacks a bit and it makes it difficult to assess the overlap with previous works. 

Also, although the experimental work is very good, I was missing any form of theoretical analysis of the proposed approach.
summary_of_the_review: I like the paper and its aim, particularly on the experimental front. There is room for improvement on the theoretical analysis and related work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors present implicit generative models in a free energy criterion with a combination of both the training of neural networks for label prediction and modeling of a label prior. They also discuss multiple sources of label priors to handle label uncertainty for coarse and imprecise data input. The extensive experiments conducted on five different tasks and the experimental results seem to well validate the efficacy of the proposed method.
main_review: Strength:

+The idea of implicit generative models with label priors to handle coarse and imprecise data input is interesting and easy to follow.

+The experiment validation covers five different tasks in both CV and NLP fields, which shows a potential good and positive impact for implicit generative models to the CV and NLP communities.

Weakness:

-The paper is not well written and organized. Especially in the introduction section, the background of the problem and the motivation of introducing implicit generative models are not well stated. I  suggest the authors to move Section A in the Appendix into the Section 2 as a separative subsection.

-The experiment validation still has room to improve. As the goal of this paper is to resolve label uncertainty, the authors should compare the proposed methods with some state-of-the-art approaches like [35; 54; 53; 55; 58] to make the experiment validation more solid.

-It should be better if the authors can provide discussion on limit and failure cases of the proposed method existing in different tasks.

summary_of_the_review: The idea is interesting and easy to follow. It has a very positive impact to the AI community. However, the paper writing and organization, experiment design and presentation still have room to improve. Therefore, I initially tend to marginally accept this paper.
review_title: null

Meta-review: The authors have addressed several of the issues raised by the reviewers, and they are strongly encouraged in include the additional experiments, and sections, that they propose, in a revised submission. The reviewers also recognized the novelty and extend of applications the proposed methodology has. Nevertheless, the paper would significantly benefit from a rigorous and thorough comparison to related work, placing it well within the context of the literature brought up by reviewers. Experimental comparisons to competitors, even if the latter address more restrictive settings, would strengthen the paper. Most importantly, the authors should consider including a comprehensive related work section, that convincingly discusses and compares to related/adjacent methods.
Decision: Reject