Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel framework for semi-supervised learning, that solves two issues of previous methods: 1) over-reliance on labeled data and 2) error accumulation. It shows that jointly solving the main task together with another task (that discriminates whether the data label is real or not) leads to better performance.
main_review: Strengths
- The proposed framework seems to be novel.
- It works well in experiments, on a wide range of tasks (classification, label propagation, and data imputation).
- It seems to be potentially beneficial for many domains, since it does not have domain-restrictions, while many previous SSL methods rely on certain image domain techniques such as consistency regularization (and data augmentation).

Weaknesses
- Since the proposed method is only compared with the original pseudo-label method, comparing with other extensions of pseudo-labelling methods that are mentioned in Section 5 will make the contributions more clear.
- In addition to the papers mentioned in Section 5, there are a few papers that try to address the error accumulation in semi-supervised learning methods that is observed in pseudo-labelling. For example: "In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning" from ICLR 2021 and "Repetitive Reprediction Deep Decipher for Semi-Supervised Learning" from AAAI2020.

Questions
- I am not sure if I understood the experiments correctly. As the missing rate goes higher, do we have more unlabeled samples (as explained in the last paragraph of page 6), or do we have more noisy-labelled samples (as explained in 1st paragraph of Section 4.1)?
- Can we show the 3rd task (data imputation) in Figures 2 to 4?
- One of the benefits of the method seems to be that it can be incorporated into a wide range of SSL algorithms. I think the paper demonstrated that it can be used to enhance pseudo-labelling method, but what kind of other SSL algorithms can SCL incorporate?

Minor questions and comments
- SSL is a very hot topic and there has recently been many advances. Since the experiments do not compare with many of the recent works, it would be better to emphasize why they were not compared. (For example, Section 1 has a discussion on how recent SSL methods utilize consistency regularization, which relies on heavy data augmentation techniques that is only available in certain domains.)
- What kind of value for parameter alpha is used in the image classification? (For the other two tasks, I think the appendix explains that alpha is 1).
- If we are given a labeled dataset L and unlabeled dataset U, it seems we can automatically construct vector M (which is explained in end of page 2). If this is correct, then why do we need M as an input in Algorithm 1 in page 6?
- What is P introduced in the beginning of Section 2.2? It seems like it is a set from the $p \in P$ notation but since it compares with M in the loss function, it also looks like a vector.
- typo "perforamnce" in page 6
- Should $m_i, m_j$ in the beginning of page 3 be $M_i, M_j$?
- Is $Y$ a label space ($y \in Y$), or is it the full set of labels in the training dataset ($Y = Y_L \cup Y_U$)?
- Ideally it would be better to perform several trials and report mean/standard error in Table 1.

=========== after rebuttal

Thank you for answering my questions. The additional experiments are helpful to have a better understanding about the proposed method. It looks like the advatangeous points of the proposed method is now about the low computational costs, according to the new experiments including UPS, rather than better performance. Although this still may be beneficial for the research community, it seems to be slightly less significant and also may affect the storyline. I would like to also recommend to put the new experiments with UPS in the main paper instead of the appendix.
summary_of_the_review: The proposed method seems to have some nice benefits, but I feel there are a few weaknesses that should be addressed. I also have a few questions and it would be helpful if the authors can take a look at the previous section (main review).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces Self-interested Coalitional Learning (SCL), which is a novel approach to semi-supervised learning. SCL combines the traditional self-training approach to semi-supervised learning with an auxiliary task that infers label observability. The empirical results show that, in a variety of scenarios, SCL outperforms both self-training and the original model.   
main_review: This is an interesting paper on a topic with important practical applications: semi-supervised learning. The contribution appears to be original, and it is likely to influence future work in the field. The authors are explicitly calling out and addressing the two main weaknesses of traditional self-learning approaches: error accumulation and over-reliance on the labeled data. 

The paper would greatly benefit from an additional section that would provide an intuitive, illustrative example of how and why the proposed approach outperforms self-training. Ideally, it should compare and contrast the convergence of (1) self training, (2) the auxiliary task, and (3) SCL. 

The paper would also benefit by tightening the narrative around the ALPHA parameter, which, in the main paper, is only discussed in the theoretical framework. Appendix A provides no value of ALPHA for the first dataset, and it proposed (without any justification) a value of 1 for the other two domains. Appendix B is extremely brief and not very helpful. The authors make no recommendation on how to tune alpha, and the argument that even the worst alpha (in the 0.1 - 0.9 range) is better than the original model is fairly weak, given the wide variations of the accuracy due to changes the value of alpha.  

OTHER COMMENTS:
- for Table 1, please add three more rows: 0%, 90%, and 99%.  The former is critical to understanding the upper-bound performance, while the later two will bring SCL into a more realistic semi-supervised regime, where unlabeled data is one or two orders of magnitude more abundant than the labeled data  
- please add to Figure 6 the horizontal lines with the accuracy of the original model for each of the three missing rates 
- it is still unclear why did you choose to use only 10%of the data for image classification (page 6); is scalability to large datasets a concern?
- please spell-check the paper - eg, "perforamnce" on page 4

- page 2:  please replace "more sufficient"
- page 3: "jointly solving above two tasks"   -->  "jointly solving THE above two tasks"
- page 3: "there are some other works embody" --> "there are some other works THAT embody"
- page 4: "are impacted the influence" -->  "are impacted BY the influence"
- page 7: please replace "well learn"
summary_of_the_review: Overall, this paper uses a novel idea to improve the state of the art for semi-supervised training.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new semi-supervised learning method. Motivated by the error accumulation problem of typical self-training paradigms, the authors propose to explicitly model the confidence of pseudo labels as an auxiliary task. They come up with a self-interested coalitional learning (SCL) strategy to solve both tasks jointly. Under the new framework, the main task is transformed into a cost-sensitive learning problem. Experiments demonstrate that pseudo labels are substantially more accurate with the new method and better performance of the main tasks at different label missing rates.
main_review: Pros:
- Overall the paper is well-structured and easy to follow.
- The new method achieves its original goals and improves SSL effectiveness by jointly solving the main and the auxiliary tasks.
- The authors introduce a new SCL strategy to solve the problems, which can be applied to a broader class of learning problems.
 
Cons:
- Lack of experiments
    - The proposed method is only compared with the self-learning method (with the same base learner). While this demonstrates how the model is improved with SCL, it is also necessary to compare with state-of-art SSL methods.
    - It's also valuable to include the supervised method with fully-labeled dataset as a reference in all experiments.
    - For data imputation, a more common case is that missing state is correlated with input/output instead of simply random missing. It also checks the method robustness against labeled/unlabeled distribution shift.
    - Compared with original self-learning method, the new method has an extra discriminator model, which are based on the same base learners as for the main tasks. It's meaningful and more fair to compare with supervised models of higher capacity.
- The paper doesn't cover how SCL can work together with consistency regularization, which is commonly used together with self-learning.


Besides, I have a few questions:
- Although Table 1 doesn't have a row for Missing rate = 0% (full dataset), it seems SCL methods have better accuracy than model trained with full dataset for the first two tasks. Is this because the SCL has double model capacity due to the extra discriminator?
- Why is the test accuracy of pseudo-labels 100% for SCL method in Figure 4? Are they calculated differently?
summary_of_the_review: This is an interesting paper from technique perspective. But it definitely needs more empirical studies to demonstrate practical value.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new semi-supervised learning framework by introducing an auxiliary task that distinguishes whether the pseudo-labels are truly labeled or not. Then, this information is used to add a reweighting loss to the main objective. Experiments on several simple benchmark datasets show that the proposed method outperforms some naive baselines.
main_review: The idea of introducing the auxiliary task that discriminates whether an instance is labeled is quite interesting. In effect, such a strategy is first introduced in active learning [1]. In the VAAL method [1], a similar discriminator is introduced to identify where an example is labeled or not, which is then used to indicate the uncertainty of an example for active selection. Therefore, the proposed method has a close connection to a recent work in SSL [2] that also employs the uncertainty measure to select high-quality pseudo-labels.

I have the following concerns.

1. The derivation in section 3.2 is confusing. For example, in Eq. (3), the second equality is incorrect and the term $\frac{dd}{dx}$ should be added. Also, it would be better to change the notation of $d$ (discriminator) to another one, since derivative dx also uses the notation d. Besides, I actually did not understand why $\mathcal{L}_B$ depends on $f$, since $f$ and $\mathcal{L}_B$ are from two different branches without sharing network blocks (Figure 1).
2. The experimental section is not convincing and this is my main concern. The datasets and the baselines are too simple. State-of-the-art SSL methods should be employed to support the claims. In particular, the uncertainty-based SSL method [2] should be compared. As I have discussed above, the proposed method can implicitly be equal to existing techniques in SSL. Is the proposed method complementary to existing methods? Or it is contradictory to some techniques? These questions require an in-depth empirical analysis.

Overall, this work is below the bar of an ICLR paper regarding its poor experiments.


[1] Sinha S, Ebrahimi S, Darrell T. Variational adversarial active learning[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019: 5972-5981.

[2] Rizve M N, Duarte K, Rawat Y S, et al. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning[J]. arXiv preprint arXiv:2101.06329, 2021.
summary_of_the_review: Interesting idea, poor experiments and confusing derivation.
review_title: null

Meta-review: This paper proposes a new method for the important problem of semi-supervised learning. This method relies on an auxiliary task, label observability prediction, to weight the examples according to the confidence in their pseudo-labels, so as to avoid the propagation of errors encountered in self-training. Limited experiments show that the proposed method can compete with other methods in terms of performance or training time. On the positive side, all evaluators agree on the potential value of the proposed approach, which is generic in nature. On the negative side, the experimental evaluation, although strengthened during the discussion, is not yet strong enough to have really convinced of the real merits of the method. In particular, comparisons with the state of the art still need to be improved. In addition, the paper would benefit from some rewriting, in particular of the mathematics (e.g. the d notation for task B should be avoided as suggested by one reviewer, there is a misplaced partial derivative in equation 6). The authors could also simplify their derivation by using the envelope theorem. I therefore recommend rejection, with an encouragement to strengthen the experimental part, and to improve the derivation of the proposed method.
Decision: Reject