LLM-Informed Semi-Supervised Learning for Text Classification

Authors: Tiberiu Sosea, Cornelia Caragea
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: semi-supervised learning, LLM

Abstract:
Large Language Models (LLMs) have shown impressive zero-shot and few-shot capabilities in many NLP tasks including text classification. While these models outperform others in terms of raw performance when few examples are available, they are expensive to use in practice and may lag behind traditional approaches when data (labeled or unlabeled) is plentiful. Semi-supervised learning (SSL) on the other hand can utilize large amounts of unlabeled data in combination with labeled data to improve a model's performance. In this paper, we propose to unify LLM and SSL under a common framework which effectively leverages the few-shot capabilities of LLMs in combination with SSL's ability to extract valuable information from unlabeled data to improve the model capabilities in text classification. Our approach, entitled LLM-SSL, utilizes LLMs to generate predictions on unlabeled examples and uses these predictions to guide the SSL training and improve the quality of pseudo-labels during training. We show that LLM-SSL outperforms both prior SSL approaches as well as few-shot LLMs on six text classification benchmarks.

Decision: reject
Reviewer scores: 3, 1, 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1729330180662

thread_type:
official_review

<thread_object>
creator: reviewer_hxcx

<content>
summary:
Summary:

This work proposes a method to make an LLM to generate pseudo labels in a semi-supervised way to train another models on text-classification tasks.

Contribution:

The experimental results of this method on text-classification tasks are good compared to some settings such as the few-shot in-context learning of LLMs.

strengths:
The experimental results of this method on text-classification tasks are good compared to some settings, such as the few-shot in-context learning of LLMs.

weaknesses:
1. The cost of this work can be quite large compared to the non-training method of utilizing unlabeled texts by language models.

2. The backbone is not good enough to capture complex semantics of different downstream tasks which is likely to fail in the text-generation tasks.

3. The tasks are quite simple which involves largely emotion classification.

4. There are many advanced few-shot methods for in-context learning. However, the author has not mentioned them.

5. The captions of tables are not up to the standard of the ICLR submission format.

questions:
1. I wonder about the results of advocating more advanced few-shot methods, such as better retrieval or chain-of-thought methods. I am unsure whether we need to train an encoder-only model to solve the emotion classification task.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622030265

thread_type:
official_review

<thread_object>
creator: reviewer_k6wr

<content>
summary:
The paper porposed LLM-SSL, a novel approach that combines Large Language Models (LLMs) with Semi-Supervised Learning (SSL) for text classification tasks. The primary goal is to improve the quality of pseudo-labels generated during SSL by leveraging the vast knowledge of LLMs, particularly in the early stages of training when the model's predictions are less reliable. Additionally, the approach introduces Average Pseudo-Margin (APM) metric to monitor the training dynamics of unlabeled examples, ensuring high-quality pseudo-labels throughout training. Experiments on six text classification benchmarks show significant improvements over strong baselines.

strengths:
1. The proposed method (including APM and LLM) is well-motivated and novel.

1. The experimental results seems very competitive.

weaknesses:
1. Some details in experiments in unclear to me, e.g., what's weak augmentation and strong augmentation?
1. I did not see discussion about APM threshold $\gamma$. In my understanding, it should be a important hyperparam to control our confidence of pseudo labels. I want to see more evidence to show its robustness.

questions:
Please refer to weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665999375

thread_type:
official_review

<thread_object>
creator: reviewer_t5cc

<content>
summary:
The paper presents LLM-SSL, a novel approach that combines semi-supervised learning (SSL) with few-shot prompting from large language models (LLMs) for text classification. In this method, the LLM generates pseudo-labels for unlabeled data, while SSL techniques, relying on training dynamics, leverage these pseudo-labels to enhance model training. Specifically, the authors propose using a class-specific exponential moving average of pseudo-margins over training iterations to estimate the model's certainty in each pseudo-label and refine these labels by combining predictions from both the LLM and the model.

Experiments were conducted on six datasets across 20- to 200-shot cases, comparing several baselines. These experiments showed consistent improvements over the baselines. An additional ablation study further explores the contributions of each component within the method.

strengths:
1. Overall, the paper is well-written and clear, though minor edits are needed.
2. The motivation is well-established.
3. The proposed method is novel, building on established SSL literature, including works like FixMatch, FlexMatch, SoftMatch, and MarginMatch.
4. Results demonstrate consistent improvements over the baselines.
5. The method shows faster convergence.
6. The ablation study reveals that the different components are needed for achieving the overall performance.

weaknesses:
1. The paper does not compare itself to several key methods in the field of few-shot text classification, which would provide important context. These include:
A. Basic few-shot text classification methods, such as SetFit and FastFit.
B. Self- and semi-supervised methods specifically designed for classification. For instance, the paper "Zero-Shot Text Classification with Self-Training" achieves strong results on datasets like IMDB and GoEmo in a zero-shot setup, although by using unlabeled training data without augmentation. Several other methods in this category could also be relevant for comparison.
C. LLM-based approaches for enhancing classification. For example, "Selective In-Context Data Augmentation for Intent Detection using Pointwise V-Information" leverages LLM-generated examples and employs smaller classifiers training dynamic to to determine whether to include these examples in training. "In-Context Learning for Text Classification with Many Labels" uses retrieval models to select the most relevant examples for LLM few-shot contexts.

Including comparisons or at least discussing key distinctions between LLM-SSL and these methods is essential for demonstrating the relevance and advantages of the proposed approach in few-shot text classification.

2. Although much of the method presentation focuses on the SSL aspect, ablation results show that LLM-SSL without the APM component still performs well, closely matching LLM-SSL’s overall performance and outperforming most baselines. This raises questions about the specific impact of the SSL component, as much of the improvement may stem from the integration of the LLM with FlexMatch, rather than the SSL itself.

3. Regarding the experimental setup, only Mistral is used as the LLM throughout the experiments. Given that an essential part of the method involves the LLM, it would be valuable to assess its impact by experimenting with different LLMs. This would clarify the LLM’s role within the method, which is currently underexplored.

4. Although less critical, the use of only one encoder model, such as BERT (which is relatively dated), is a limitation. Including newer models like RoBERTa, DeBERTa, or other recent alternatives could demonstrate the robustness of the method across various base architectures.

questions:
1. Why LLM ZS and FS are the same across all shots (in all cases the shots exceed the context!?) and don't have STD results?
2. The SSL extension of the method can be very general (not necessarily for languages, and not for classification) and can target many different tasks. How will it perform on other tasks?
3. In the pseudo-code you describe the algorithm as changing the data, is it not easier to describe it by changing the data?

Comments:

1. No explanation on the way the STD was calculated, no STD for ablation, no significant testing.

2. Small points:

a. row 167: grammar, confidence at an iteration

b. row 182: \neq can be a better annotation in eq. 3

c. row 219:1 in the pseudo-code:  pseudo-labels "for the every unlabeled example" -> for every

d. pseudo-labels line 8, avreage PM, why not APM, why eq. 4 and not 5?

e. pseudo-labels line 12, the loss function H is not defined.

f. row 253: superior to what? which specific model version?

g. you mention that the pseudo-label can be a two-hot vector but didn't explain how it affects the loss function.

h. row 331: why n Edunov is not in brackets, same 333, verify for all the paper.

Probably will be good to proofread the whole paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694456356

thread_type:
official_review

<thread_object>
creator: reviewer_mfpv

<content>
summary:
This paper studies the semi-supervised text classification problem with the help of LLMs. The method is built upon FlexMatch (Zhang et al., 2021) and further introduces hard pseudo-labels from LLMs for the unlabeled texts. The LLM's pseudo-labels and traditional pseudo-labels from FlexMatch are blended together during the iterations and then used as the supervision for the model training. The experiment results on 6 datasets show that the proposed method outperforms both FlexMatch and LLM, showing that the integration of both methods is effective.

strengths:
- This paper works on a well-defined problem by introducing LLM as help.
- The proposed method is intuitive and easy-to-follow.
- Six datasets are comprehensive.

weaknesses:
- Integrating LLM into the semi-supervised text classification problem is a natural idea, however, the way proposed in this paper is a little straightforward. Based on the line 1 in Algorithm 1, we will have to send all the unlabeled texts to the LLM, which can be very expensive. We can of course expect the performance improvement as the classifier is trained with additional signals from LLM, however, perhaps at a huge cost. An analysis between cost and improvement will be very useful. I hope to see a cost-effective method to improve the classification.

- Many weakly supervised text classification works have been ignored in this paper. Their problem setting often involves very limited supervised data, and sometimes, even zero annotated documents. The final classification performance can be very strong too (e.g., 90% F1 or higher). For example, some recent works [1, 2] in this direction also discussed how to leverage language models to better generate pseudo-labels and put them into a bootstrap framework for classifier training. More recently, label selection and debiasing have been studied too [3]. I believe some of these methods can be easily adapted as baselines here.

[1] Mekala, Dheeraj, and Jingbo Shang. "Contextualized weak supervision for text classification." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

[2] Wang, Zihan, Dheeraj Mekala, and Jingbo Shang. "X-Class: Text Classification with Extremely Weak Supervision." Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

[3] Dong, Chengyu, Zihan Wang, and Jingbo Shang. "Debiasing Made State-of-the-art: Revisiting the Simple Seed-based Weak Supervision for Text Classification." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

questions:
- This is a minor format issue. I think the authors somehow didn't know how to use the citation properly in this ICLR template. Please explore \citet and \citep and check their differences. In many cases, I believe you may want to use (NAME et al., YEAR) instead of NAME et al., (YEAR).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721226693

thread_type:
official_review

<thread_object>
creator: reviewer_hc3t

<content>
summary:
This paper introduces LLM-SSL, a method that combines FixMatch with LLMs for semi-supervised text classification. The authors claim that incorporating LLMs improves semi-supervised text classification performance compared to FixMatch and other SSL baselines.

strengths:
The paper is clearly written and easy to follow.

weaknesses:
1. Missing Important Baselines:  
The paper omits several key baselines for semi-supervised text classification, such as PET [1], which also uses large amounts of unlabeled data for training. Additionally, it would be useful to evaluate the performance of using LLMs like Mistral-7B-Instruct directly for text classification without any fine-tuning. With few-shot examples, in-context learning could serve as an alternative for semi-supervised text classification. Recent work [2] also explores similar applications of LLMs in text classification and could provide additional comparative insights.

2. Contribution:  
The use of LLMs for labeling is presented as a primary contribution of this paper. However, this approach is already well-established in NLP and may not suffice as a novel contribution.

3. Unfair Comparison:  
The comparisons may not be entirely fair, as none of the baselines incorporate LLMs. Adding LLMs for labeling naturally boosts performance, making it difficult to isolate the effectiveness of the proposed SSL method. To present a fair comparison, it would be better to exclude LLMs. Without them, the performance of the proposed method appears to be close to that of other SSL baselines.

References:  
[1] Schick et al., Exploiting Cloze Questions for Few-Shot Text Classification and Natural Language Inference  
[2] Zhang et al., Generation-driven Contrastive Self-Training for Zero-Shot Text Classification with Instruction-Tuned GPT

questions:
Why not use LLMs to estimate the quality of model-labeled examples?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates semi-supervised text classification by leveraging large language models (LLMs). The proposed method utilizes LLMs to generate predictions on unlabeled examples and uses these predictions to guide the SSL training and improve the quality of pseudo-labels during training.  The experimental results show that the proposed method outperforms baseline methods, demonstrating that the integration of both methods is effective.

Pros:
- This paper is easy-to-follow.
- The motivation of the proposed method is intuitive.

Reasons to reject:
- Simply integrating LLMs into semi-supervised text classification is intuitive and somewhat straightforward, which makes the proposed method lack novelty, and thus the contributions are not significant enough.
- This paper did not compare with many other powerful semi-supervised learning methods. Although the authors argue that their goal is to compare the proposed method with semi-supervised learning methods that are similar (consistency regularization-based) and contemporary to their work. I do not think this is reasonable and fair.