Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper introduces a graduated-non-convexity-based method to improve the fine-tuning performance on NLP tasks. Based on a gradually convexified loss, the models are assigned different weights to the examples in self-training to mitigate the effect of incorrect labels. Experiments indicate the proposed method outperforms standard self-training.
strength_and_weaknesses: The paper is overall well-written, with clearly stated contributions. Applying the GNC in self-training is an interesting idea. 

Weaknesses:    
1.There seems to be many ad-hoc factors introduced into the proposed algorithm. First, changing the GM to SGM cannot ensure that the Black-Rangarajan Duality still holds. In this sense, it is hard to say the SGM loss and the reweighted classification loss are still related. Where does decreasing μ by μ = μ/2.8 come from? Is it 2.8 is the number with the best performance found with experiments?    
2.Lacking more comparison with baselines. It is well known that standard self-training is not the perfect solution. There are a lot of works such as FixMatch, which can really improve standard self-training by a large margin. The authors should also include these methods in their comparison.    
3.The experiment setup section is confusing. Why some tasks have 16 labeled examples while some have 12? The backbones of different methods are also different, make it very hard to compare except the last two rows.     
4.The authors mentioned running each experiment for 20 times and report the mean. Why not report the standard deviation? Is it the stddev is larger than the margin (I’m inferring this from table 4).

clarity,_quality,_novelty_and_reproducibility: Apart from some confusing points, the paper is overall clear. I believe the method can be reproduced from the code.
summary_of_the_review: Although the idea in this paper is interesting, the weaknesses make it unsatisfactory for iclr as a top venue. Thus, I tend to reject this work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a self-training method based on graduated non-convexity in order to be robust to outliers with wrong pseudo labels and over-confident training samples. Intuitively the core idea of the method is to penalize these two kinds of samples by assigning low loss weights to them. Experimental results show that the proposed method improves the performance and stability of self-training on many different tasks.
strength_and_weaknesses: Strength

1. The paper is well-written and the method is clearly presented.

2. Though the main technique is mostly adapted from previous work [1], the use of graduated non-convexity in the self-training setting looks novel to me.

3. The experiments are extensive and the ablation study looks solid.

Weaknesses

1. The empirical improvements compared to baselines are not very stable and to some extent marginal in some datasets.

[1] Heng Yang, Pasquale Antonante, Vasileios Tzoumas, and Luca Carlone. Graduated non-convexity for robust spatial perception: From non-minimal solvers to global outlier rejection. IEEE Robotics and Automation Letters, 5(2):1127–1134, 2020.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and the use of graduated non-convexity in self-training looks novel. The source code is available.
summary_of_the_review: Given the strength and weaknesses, I tend to rate the paper as marginally above the acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Facing the problem that self-trained models usually exhibit unstable performance influenced by data distribution and pseudo-label accuracy, this paper firstly applies the graduated non-convexity in self-trainng.
The authors firstly formulate the NLU tasks to classification tasks and construct supposition for different tasks. Then the suppositions are fed into the pre-trained model to generate pseudo-labels for each data example. As the core, authors proposed the convexity controlled training. The method achieves great improvement on downstream tasks.

strength_and_weaknesses: Strengths:

(1) Novelty of firsly introducing the GNC into self-training to achieve robustness;

(2) Easy understanding for the suppositions construtions which is aimed at formulate different tasks as uniform classification task.

(3) Great performance which validates the effectiveness of the method.

Weakness:

(1) why the coefficient μ is decreased by 2.8 in Eq. 8? As the preliminary introduced that "A more convex iteration provides a guess for the following optimization." Thus the setting of the decreasing of coefficient should be discussed.

(2) The robustness against incorrect pseudo-labels, imbalanced training data, overfitting, and adversarial evaluation data have not been clearly verified in the experiments.

clarity,_quality,_novelty_and_reproducibility: The clarity is good. The Quality is fair. The Novelty is good. The Reproducibility is good.
summary_of_the_review: Please see the comments in Strength And Weaknesses, but I am not an expert in NLP domain, some of my adjustment maybe not correct
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose an outlier robust self-training method based on graduated non-convexity (GNC) to mitigate the instability issue in conventional self-training. Specifically, they propose new robust cost functions according to Black-Rangarajan Duality. Experimental results on natural language understanding demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: Strength:

* This paper considers self-training, which is a very important topic and has potentially wide usage.

* The proposed method is easy to understand, although there are still some presentation issues.

* Experimental results seem to demonstrate the effectiveness of the proposed method.


Weaknesses:

* Some parts of the paper are not clear.

  * In Section 4.1, the authors mentioned they pre-train a model on MNLI. How is the pre-training conducted? Do you train a model with 3-way classification loss (entailment, neutral, contradiction) using the provided labels in MNLI? Or do you train a model to predict $s^1_{\text{mnli}}$?

  * Continuing on the previous comment, how is $s^2_{\text{mnli}}$ related to $s^1_{\text{mnli}}$? You mentioned these two are contradictory, could you give a concrete example? Also, how do you use $s^2_{\text{mnli}}$ in the pre-training stage?

  * What is $\ell_{\text{max}}$ in the paragraph below Eq. 8?


* Some heuristics seem rather arbitrary.

  * For example, in the paragraph below Eq. 8, the schedule of $\mu$ is defined as $\mu \leftarrow \mu/2.8$. Could the authors provide some intuition about this $2.8$ factor? Or is this purely based on trial and error? And why is training stopped when $\mu<1$?

  * Also, in Section 5.2, could the authors provide some intuition about the schedule of $\epsilon$? In particular the additional hyper-parameter $\alpha \in [1.2, 2.8]$?

* The technical novelty and contribution is quite limited. The authors simply introduce an additional hyper-parameter $\epsilon$ into the GNC method. The design of this hyper-parameter is based on intuition, and absolutely no theoretical guarantee is provided,  (which I think is fine if strong enough empirical evidence is provided). 

* The authors mention that “we train BERT and DeBERTa models as the pretrained entailment classifiers”. During the entailment pre-training stage, is the model initialized from pre-trained BERT/DeBERTa? If so, the model already contains information that can help downstream tasks such as SST-2.

* The authors mentioned that “we investigate weakly supervised domain and task adaption of textual entailment models”. The authors could run experiments in a more general setting. For example, directly fine-tune on GLUE where only a small proportion (e.g., 10%) of the labels are considered available. 

I will raise my score if the authors can address my concerns and run more experiments.
clarity,_quality,_novelty_and_reproducibility: See above
summary_of_the_review: See above
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Reviewers found the paper well-written and the introduction of Graduated Non-Convexity to self-learning novel and interesting.
They (uzE6, py5v) found the empirical results to be weak or inconsistent over baselines. The authors did report standard deviations, but they show that the improvement in the mean is rarely greater than 1 standard deviation. The authors did not address this main objection.

Several reviewers also objected to the use of heuristic adjustments for $\mu$ and the authors provided a citation but also did not seem to fully address this.

justification_for_why_not_higher_score: by reviewer scores and comments. The reviewers provided valid points that were not fully addressed and no reviewers felt strongly about accepting.
Decision: Reject