Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a technique for optimizing non-decomposable metrics in self-supervised learning settings. The method is based on the hybrid loss for optimizing non-decomposable metrics in supervised learning settings (Narashiman & Menon, 2021). The authors apply the reduction of the non-decomposable metric to cost-sensitive learning and form a weighted consistency regularizer. The framework is then used to improve the FixMatch algorithm for self-supervised training. Finally, the authors demonstrate the benefit of the models in real-world datasets.
strengths_and_weaknesses: Strengths:
- Optimizing non-decomposable metrics are relevant to many real-world problems
- The authors provide theoretical justifications for the proposed models.
- The experiments show the benefit of the proposed algorithm compared to baselines.

Weaknesses:
- My biggest complaint about the paper is the clarity of the paper. The paper is hard to understand for someone that is not already familiar with the related works. In many places, the authors refer to the related works without providing enough explanation for the reader to understand. In short, I do not think the authors have made the paper self-contained.
- Some terminologies in the paper are used before even defining them, for example, Assumption 1 in Sec 3.2 uses F_pl, but it is defined only after Sec 3.4.

----------- post rebuttal -----------

Thanks to the authors for providing the detailed explanation. 
The authors has added more explanation in the appendix. 
However, I still think the paper need more explanation on the main paper to add clarity and help reader that is not already familiar with the related works. As also mentioned by another reviewer, the paper is very hard to read.
For the technical contribution of the paper, I could not asses it thoroughly due to the difficulty of understanding the contribution of the paper.


questions: Please answer my concerns and questions in the previous section.

limitations: The authors have not discussed the limitation of the proposed model. A discussion on the model's limitations is suggested.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper propose a new loss function for self learning that can take into account also non-decomposable cost functions. The work both analyzes the performance when using this new cost function and show empirically its advantage over previous solutions for self-training. In the theoretical part, the author claim that their loss function leads to a lower error compared to standard optimization over the loss. In the practical part, the authors show that their approach, which combines also a smart thresholding for keeping only part of the examples for the self training, leads to improved results over previous solutions. Both vision and NLP benchmarks are being considered. 
strengths_and_weaknesses: The strength of the paper is that it combines a theoretical analysis with some strong empirical results compared to existing baselines. 
A new loss function is being proposed and a thresholding approach. Both seems to improve the performance for the task at hand

Weaknesses: 

1. The paper is very (very very) hard to read. The notation are not well presented. Some symbols are defined only after they are being first used. The writing is quite bad and it is very hard to follow. Few examples:
a.  the recall and precision on line 65-66 are defined the same. 
b. F_{pl =} is used in Assumption 1 but defined only in Section 3.4
c. The notation is also not very consistent. Sometimes the authors use P_w where w is the weight and in the same equation use P_i where i is an index. It is very unclear and confusing. 
d. Less critical compared to the above but part of the things that are defined should be better defined. For example c that is used in assumption 4 is mentioned without saying what it is. Indeed, it is mentioned in the c-expansion but the statement should start with mentioning that there is a function c and then say what is done with it. The comment here, is indeed a matter of style (unlike point a-c, which are not a matter of style but just make the paper very hard to follow)

2. The assumption made in theoretical part are not justified. Specifically, in assumption 1 if we assume beta is negligible then it means that R_B,w is basically equal to zero. Is it really the constraint we want to use? 
Also, if we assume that the error of the found function F^* is smaller than the one of F_{pl}, why Theorem 5 is surprising? Is not it just saying that if the error is smaller than the error is smaller? The authors assume something and then claim something more complicated that more or less states the same. At the end of the day, they want to show that F^* is better. So they assume that and then claim they proved that. So what is the point? 

3. The experiments are interesting but I wonder whether the main benefit is just from the new thresholding being used. There is an ablation that show the impact of this new threshoding approach on the proposed CSST approach. Yet, the real missing experiment is checking what is the impact of this approach on existing baselines (e.g., FixMatch). Clearly, the problem with the threshold has already been mentioned in previous works as the authors admit. So proposing a solution for it should be checked also with previous works. 

4. Finally, why only one baseline is being compared to in each task?  Is FixMatch/UDA the only work that was proposed sine 2020 for this problem (Clearly there are and the authors should compare to them). Note that this is the least important problem in the paper that I find compared to the other points raised above. 
questions: see above 
limitations: Didn't see limitations being mentioned
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Non-decomposable metrics take an important role in machine learning. This paper proposes to improve self-training by optimizing non-decomposable objectives for semi-supervised learning. This strategy can bring a significant average improvement in desired metric of minimizing worst-case recall while maintaining similar accuracy compared with SOTA methods. Empirical results show that proposed Cost-Sensitive Self-Training (CSST) framework help improve the performace of baselines, e.g., FixMatch, UDA.
strengths_and_weaknesses: Pros
- Propose to improve self-training method by optimizing non-decomposable metrics that utilizes unlabeled data in addition to labeled data.
- Propose a weighted consistency regularizer for Cost-Sensitive Self-Training.
- The superiority of desired non-decomposable metric is theoretically justified.
- Empirical results have validated the effectness of the proposed method, and bring improvement over baselines.


I am overall satisfied with this paper and would like to give some comments that may make this paper better.
- It is expected to release the source code of all experiments to benifit the community.
- It is promising to compare with methods that improves the recall of Fixmatch, e.g., Flexmatch.
questions: See above.
limitations: Yes.
review_title: null

Meta-review: metareview: The paper received two negative scores 3/4 (another is 8 with high confidence 5) and the main critcism is that the writing is vague especially for the topic of the paper may not be very popular to the community. The authors have made good efforts in improving their presentation and they also provide additional clarifications as well as new experimental results point to point. Hence in my opinion, the new pdf version is more readable.

For its significance and novelty, the proposed new loss with regularization is theoretically sound and empirically effective. It also addresses the self-training for non-decomposable metrics which is the first time in literature to our knowledge. There are many applications for this method and there are little related work in the community which highlights its potential impact.

I suggest to accept this paper for its significance, quality and strong results. The writing is also improved during the rebuttal.
confidence: Certain
Decision: Accept