JointMatch: A Unified Approach for Diverse and Collaborative Pseudo-Labeling to Semi-Supervised Text Classification
Authors: Henry Peng Zou
Cornelia Caragea
Venue: EMNLP 2023 Main (2023)
Keywords: Semi-supervised learning
text classification
pseudo-labeling
adaptive local thresholding
cross-labeling
weighted disagreement and agreement update.

Abstract: Semi-supervised text classification (SSTC) has gained increasing attention due to its ability to leverage unlabeled data. However, existing approaches based on pseudo-labeling suffer from the issues of pseudo-label bias and error accumulation. In this paper, we propose JointMatch, a holistic approach for SSTC that addresses these challenges by unifying ideas from recent semi-supervised learning and the task of learning with noise. JointMatch adaptively adjusts classwise thresholds based on the learning status of different classes to mitigate model bias towards current easy classes. Additionally, JointMatch alleviates error accumulation by utilizing two differently initialized networks to teach each other in a cross-labeling manner. To maintain divergence between the two networks for mutual learning, we introduce a strategy that weighs more disagreement data while also allowing the utilization of high-quality agreement data for training. Experimental results on benchmark datasets demonstrate the superior performance of JointMatch, achieving a significant 5.13\% improvement on average. Notably, JointMatch delivers impressive results even in the extremely-scarce-label setting, obtaining 86\% accuracy on AG News with only 5 labels per class. We make our code available at https://github.com/HenryPengZou/JointMatch.

JointMatch: A Unified Approach for Diverse and Collaborative Pseudo-Labeling to Semi-Supervised Text Classification
Henry Peng Zou
Cornelia Caragea
Computer Science
University of Illinois Chicago
pzou3@uic.edu
cornel ia@uic. edu
Abstract
large amounts of unlabeled data (Berthelot et al., 2019; Sohn et al., 2020).
Semi-supervised text classification (SSTC) has gained increasing attention due to its ability to leverage unlabeled data. However, exist- ing approaches based on labuli-la suffer from the issues of pseudo-lahel bias and er- ror accumulation. In this paper, we propose JointMatch, a holistic approach for SSTC that addresses these challenges by unifying ideas from recent semi-supervised learning and task of learning with noise. JointMatch adap- tively adjusts classwise thresholds based on the learning status different classes mit- igate model bias towards current easy classes. Additionally, JointMatch alleviates error accu- mulation by utilizing two differently initial- ized networks to teach each other in cross- labeling manner. To maintain divergence be- tween the two networks for mutual learning, we introduce a strategy that weighs more dis- agreement data while also allowing the utiliza- tion of high-quality agreement data for training. Experimental results on benchmark datasets demonstrate the superior performance of Joint- Match, achieving significant 5.13%, improve- ment on average. Notably, JointMatch delivers impressive results even in the extremely-scarre- label setting, obtaining 86% accuracy AG News with only labels per class. We make code available https: //gi comb CenryPennzoo/JointMatttt
Recently, the combination of pseudo-labeling and consistency regularization has become a pop- ular paradigm for semi-supervised learning (Sohn et al., 2020; Zhang et al., 2021; Sosea and Caragea, 2022). Pseudo-labeling (Lee et al., 2013) uses fixed threshold to select the model's high- confidence predictions pseudo-labels for further training, whereas consistency regularization (Saj- jadi al., 2016) enforces the model make similar predictions for perturbed versions of the same data. For example, UDA (Xic et al., 2020) applics strong data augmentations, such as bock-translation, to un- labeled data and minimizes the divergence between model predictions of input and its augmented views. FixMatch (Sohn et al., 2020) uses the pseudo-label generated from to supervise the strongly augmented version of the same data. SAT (Chen et al., 2022) improves Fix- Match by training meta-learner to re-rank dif- ferent augmentations based on their similarities with the original data. These methods require a pre-defined eigh--oonfddenc threshold to generate high-quality pseudo-label labels for good performance. However, there some potential limitations: (1) Setting fixed threshold for pseudo-label selection neglects the varied difficulties of learning different classes (Zhang al., 2021; Wang et al., 2023b,a). This can cause the model bias toward casy classes, as more pseudo-labels will be generated for cur- rent easy classes (see Figure la); and (2) If these pseudo-labels are incorrect and used to train the model, the model can be worse and produce more inaccurate pseudo-labels, progressively accumulat- ing its error and degenerating its performance (see Figure Ic) (Arazo al., 2020).
unlabeled data
1 Introduction
The success of deep learning models often heavily depends on the availability of large amounts of la- beled data (He et al., 2016; Vaswani et al., 2017). However, the labeled data for many tasks are often expensive, difficult, and time-consuming to obtain. By contrast, acquiring unlabeled data is more cost- effective and convenient in many scenarios. This has led to surge of interest in semi-supervised learning, which aims to enhance learning perfor- mance with limited labeled samples by leveraging
To address these issues, we propose Joint- Match, a diverse and collaborative pseudo-labeling approach for semi-supervised text classification (SSTC). JointMatch is a holistic framework that

---

1030
2560
12000
Number Label
clos3
Hknbee of Picuro Labet
33000
8000
10009
800
8.9
5330
60
X00
htalach jerstasch
taratons
norators
(a) FixMatch
JointMatch
(c) Error Accumulation
Figure I: Number of pseudo labels (a, b) and validation accuracy (c) of FixMatch and JointMatch AG News with 10 labels per category. Pseudo-labeling approaches use a fixed threshold, such as FixMatch, suffer from pseudo-label bias towards easy classes and error accumulation along the training. In contrast, JointMatch effectively balances pseudo-labels and avoids degenerated performance caused by error accumulation.
unifies ideas from recent semi-supervised learn- ing and the lask of learning with noise for SSTC. Inspired by FlexMatch (Zhang et al., 2021) and FreeMatch (Wang et al., 2023b), we adaptively ad- just classwise thresholds based on the estimated learning status for different classes at different times. This enables difficult classes at the current iteration to have lower local thresholds, thereby facilitating pseudo-labels to be produced and used for learning these classes (see Figure 1b). Fol- lowing the idea of co-training (Blum and Mitchell, 1998), JointMatch simultaneously trains two differ- ently initialized networks and uses them to teach each other in a cross-labeling manner to alleviate етrоr accumulation, as different networks can filter different types of noise.
in the extremely-scarcc--lahe setting, achieving 86% accuracy on AG News with only labels per class. We provide comprehensive ablation studies and analysis to understand each part of JointMatch.
JointMatch
2.1 Overview
In this section, we introduce our proposed Joint- Match for semi-supervised text classification. Joint- Match is a unified approach that integrates ideas and components from recent semi-supervised learn- ing and the task of learning with noise. Specifically, utilize three key techniques, i.e. (i) adaptive lo- cal thresholding; (ii) cross-labeling; (iii) weighted disagreement & agreement update, to address the limitations we have identified: (a) bias towards easy classes; (b) error accumulation; (c) tradeoff between divergence & consensus.
Nevertheless, with the increase of training iter- ations, those networks will slowly converge and reduce one network and thus will again suffer from the issue error accumulation. Inspired by Co-Teaching+ (Yu et al., 2019) and Decoupling (Malach and Shalev-Shward to give more weight to disagreement data to keep the two networks diverged while also allowing the utilization of high-quality agreement data for train- ing. The relationship and difference between our JointMatch and related techniques are discussed in detail Section 2.5.
The main pipeline for JointMatch is shown in Figure 2. For each batch of unlabeled data 24 we first apply both weak data augmentation a(.)) and strong data augmentation A(.), such as synonym replacement and back translation, respectively. The weakly augmented data are then forwarded to two differently initialized models f and g to make pre- dictions. High-confidence predictions that pass the adaptive local threshold Tr(c) are selected as pseudo-labels. Especially, the pseudo-labels gen- erated by one model are used to teach its peer network, i.e., cross-labeling. The unlabeled loss C, is computed between generated pscudo-labels and model predictions of strongly augmented data. Here, we weigh more disagreement data (where both networks have different label predictions) keep the two networks diverged but also allow the utilization of agreement data that are more likely
2017). we propose
We evaluate the proposed JointMatch on three commonly studied SSTC benchmark datasets. Ex- perimental results indicate the superior perfor- mance of JointMatch, obtaining significant 5.13% average improvement over the latest work SAT. We also analyze the performance of JointMatch with varying numbers labeled data. The results show that JointMatch can deliver impressive results even

---

Unlabeled Data
Cross Labeling
Model f
Pseudo Label
L.
Model f
Weakly-Augmented a(4)
ra(c)
StronglyAugmented AQ4)
Madel
Piscuda Label
wh
Model g
Adoptive Thresholding
Weghted Disagroement Update
Figure Pipeline JointMatch. Unlabeled data undergo both weak and strong data augmentation. Weakly augmented data are first fed into two differently initialized models. Predictions with confidence surpassing adaptive local thresholds are selected as pseudo labels for training. Notably, pseudo labels from one model are used to teach its peer network, i.e, cross-labeling. Unlaheled data loss is then computed between pseudo labels from data and model predictions from strongly augmented data. Here, we weigh more disagreement data to keep the two models diverged for effective mutual learning.
to be correct. A complete algorithm for JointMatch is presented in Algorithm 1. Next, we explain our key components in detail.
Pr(c)
(c() MaxNorm(f((c)).. max, (P:
(2)
By doing this, difficult classes at the current itera- tion will have lower local thresholds, encouraging more pseudo-labels to be generated and utilized for training these classes.
2.2 Adaptive Local Thresholding
gneudo-labeling-aaee based semi supervspere text classi- fication algorithms use a fixed threshold to select high-confidence unlabeled data as pscudo-labels for training. However, this ignores the varied dif- ficulties of learning different classes at different time steps, and may result in that easier classes can generate more pseudo-labels than hard classes for learning and can cause incressing model bias along training (see Figure la). Inspired by Flex- Match (Zhang et al., 2021) and FreeMatch (Wang et al., 2023b), instead of using a fixed threshold, our JointMatch estimates the learning status of different classes and adaptively adjusts the local thresholds for different classes to produce more diversified pseudo-labels (see Figure 1b). Specifically, each time step first estimate the classwise learning status it ())) (C)) via the expo- nential moving average of model's ' predicted prob- ability on unlabeled data:
2.3 Cross-Labeling
Another issue of pseudo-labeling, erally, self-training, is error accumulation: If the generated predictions are incorrect and the model is trained on them, the model can become worse and worse, continually producing more noisy pseudo-labels and accumulating its error (see Figure Ic). Inspired by Co-Training (Blum and Mitchell, 1998), our JointMatch involves the si- multaneous training of two networks with different initializations. These networks are utilized to tually instruct each other through tross-labeling. This strategy mitigates error accumulation because different networks can filter out different noises.
gen-
More formally, given two differently initialized networks f and each network first selects its own high-confidence predictions of unlabeled data as pseudo-labels for the other network. The selected pseudo-labels are then used compute unlabeled data loss Cu update the parameters for its peer network:
aB pB
= Ap:-1 + (1 X)
(1)
b=1
pB
where Po = 1/C,,//C..,,//C C' is the num- ber of classes, B is the batch size of labeled data, I is the ratio of unlabeled data to labeled data, A E 1) is the momentum parameter, Ø Pmn (a(g)) denotes model's predicted class distri- bution on weak-augmented unlabeled data. Then we normalize the estimated learning status and ad- just the local threshold for each class e from the pre-defined threshold 7:
C/
1 max ((xx(g)) 2
B
T (arg max (2 Q()
C9 BB
max(())))
T/ (arg max(g !
(3)

---

7: end for Algorithm JointMatch algorithm.
I: Input: Differently initialized model f. 9. number of classes C, labeled batch X {(r6, Mb) b (1,2..... B) unlabeled batch U {un b (1, .........BB))...... pB)), where B is the batch size of labeled data, 1 is the ratio of unlabeled data to labeled data, unsupervised loss weight Wh. EMA decay X. fixed threshold т, disagreement weight 6, weak and strong data augmentations ax-). A(-).
2: for model f. do
3: Le - B b=1 My,,, (a(*)))) {Compute loss for labeled data, (.) denotes for model prediction)
4: Pt = Ap--1 + (1 2) nT En 9b (Estimate learning status for different classes, qu is an abbreviation of
5: for c - 1 to C do
6: TC(C)= = MaxNorm(f(()) T
[Adjust the local threshold for each class based on its learning stonus
8: end for
9: for to μB do
wh 81 (gh +
+ (1-
(.)
(Compute sample weight based on prediction disagreement, 4
denotes the generated hard label)
11: end for
12: C
RB p b=1 wi 1 (max (1g) 2 7i aar max (d:)) 26)
Ca 1 D wpl max (%) 2 oe (arg max (1))) H H(gb Q%) (Compute loss for unlabeled data, the
pB
pseudo- nheudollabel generated by one
used supervise another network, Qu abbreviation of pm (A(us)))
Return: L. + wyLl, L + w.
where and Qs denote model's predicted class distributions on weakly and strongly augmented data, respectively, Z is the one-hot label that is converted from (b. and H refers to cross-entropy loss. Compared to self-training this approach dis- plays a Eegzag-shapel error flow, which helps to avoid direct error accumulation within single net- work.
labels and should also be utilized. To this end, we propose to compute a loss weight wy for each unla- beled data that weighs more disagreement data to keep two networks diverged while also allowing the utilization of agreement data:
wy 51 (0g # # + (1 -1-5)1(Gp 5)
4
(4)
where ô (0.5, 1) the disagreement weight. The unlabeled data loss E,, thus becomes:
2.4 Weighted Disagreement & Agreement Update
Two differently initialized networks can have var- ied learning abilities to filter different types of er- ror in the initial training stage, allowing them to learn from its peer. However, the training goes on, those networks will gradually converge, and the co-training approach will degenerate self- training and thus again suffer from error accumula- tion. To address this problem, Decoupling (Malach and Shalev-Shwartz, 2017) and Co-Teaching+ (Yu et al., 2019) propose to update networks only by data with disagreed network predictions and find that this strategy is effective in keeping two net- works diverged. Nevertheless, their approaches completely ignore the agreement data, where both networks have the same prediction. We argue that those agreement data are a valuable learning signal, as they are more likely to receive correct pseudo-
C/
Wp I max(g]))
uB
Tf (arg max (g)))))((G)) Q,)
C4
Σ wo 1 max(()))
μB
of (arg max(g ()) Q (5)
Our experiment result in Section 4.2 shows that this approach is more effeetive than the method using (i) only disagreement data; (2) only agreement data; (3) weighting both kinds of data equally. Note that this method can be further improved by adaptively adjusting the disagreement weight based on the disagreement degree of two networks or based on

---

Semi-Supervised
Fully-Supervised
SSTC
FixMatch/
FlexMatch/
Co- Co-Training
Decoupling
Co-Teaching+
JointMatch
SAT
FreeMatch
Pscudo-Labeling
/
Weeak-Strong Augmenlation
/
Adaptive Local Threshold
Double Networks
X
x
Cross Laheling
X
x
Disagreement Update
X
X
Weighted Disagree & Agree Update
X
X
Table I: Comparison closely related techniques with our JointMatch approach. In the first column, "Pseudo- Labeling": select high-confidence predictions pseudo-labels for training; "Weak- -trong Augmentation": use the pseudo-labels generated by weakly-angmeentd sample to supervise the strongly-augment Update": update networks by only samples receives different pseudo-label from the two networks.
sample; "Disagreement
different time steps for curriculum learning. How- ever, this topic is outside the focus and scope of this paper, and we leave it to interested researchers for future exploration.
1998) trains two networks simultaneously and su- pervises each network by the generated pseudo- labels from its peer network.
Decoupling (Malach and and Co-Tcaching+ (Yu et al., 2019) observe that update by disagreement data can make two net- works in Co-Training diverged and thus more ef- fective. However, when updating their network, they completely ignore agreement data, which is also more likely receive correct pseudo-labels in semi-supervised learning. Therefore, propose utilizing both disagreement and agreement data for training, but weighting disagreement data higher to keep the two networks diverged while also exploit- ing the high-quality pseudo-labels from agreement data.
2017)
The overall objective for training one network in JointMatch is:
== Wufa
(6)
where wa represents the loss weight for unlabeled data loss L, and the supervised loss L. is given
by:
B
4.
Hlyb, Pm Pm (a(x))))
B
Our JoinMatch framework unifies ideas from the approaches described above into a single, effective framework for semi-supervised text classification. The three key techniques in JointMatch are: (i) adaptive local thresholding: (ii) cross-labeling: and (iii) weighted disagreement & agreement update.
2.5 Relation to Other Approaches
this section, we compare our JointMatch algo- rithm with closely related approaches in Table 1. Our goal is to identify the connections among them and point out the key techniques that make Joint- Match effective in semi-supervised text classifi- cation. FixMatch (Sohn et al., 2020) and SAT (Chen al., 2022) employ the consistency reg- ularization between weak and strong augmentation with pseudo-labeling to leverage unlabeled data. However, this approach uses a fixed threshold for lseudo-labeling, and often suffers from model bias towards easy class and error accumulation along their training. FlexMatch (Zhang et al., 2021) and FreeMatch (Wang al., 2023b) propose adap- tively adjust local thresholds based on the estimated learning status of each class, which alleviates the bias towards easy classes. To address the issue er- ror accumulation, Co- Training (Blum and Mitchell,
Experiments
3.1 Experimental Setup
Datasets and Metrics. We evaluate JointMatch on three standard semi-supervised text classification benchmarks: AG News (Zhang et al., 2015), Yahoo! Answers (Chang et al., 2008) and IMDB (Maas et al., 201 1). Following (Chen et al., 2022. 2020; Li al., 2021), use the original test and randomly sample from the training construct our training unlabeled and validation set. Table 2 presents the dataset statistics and split information. We report the mean and standard

---

Dataset
Label Type
# Classes
Unlabeled
# Validation
Test
AG News
News Topic
5000
2000
1900
Yahoo! Answer
AQ Topic
10
5000
2000
6000
IMDB
Review Sentiment
2
5000
1000
12500
Table 2: Dataset statistics and split information. The number of unlabeled data, validation data and test data in the table means the number of data per class.
AG News (c=4)
Yahoo! (c=10)
IMDB (c=2)
Methods
Accuracy
Macro-F1
Accuracy
Macro-FI
Accuracy
Macro-F1 F1
Average
BERT
69.18 1.18 3.7
68.27 # 3,5
58.11 + 1.6
57.38 + 1,9
63.16 1.4
62.93 1.6
63.17
UDA
76.69 3.2
76.51
3.0
59.32 a
58.47
2.3
64.88 + 1.7
64.57 1.5
66.74
TixText
78.07 + 2.8
77.23 + 3.5
59.93 + 1.9
59.24 + 1.8
65,22
65.78 1.2
67.58
FixMatch
00.22222:
78.98 2.1
60.17 1.7
59.86 1.5
64.52
64.31 1.4
68,18
SAT
85.43 # 1.2
85.30 + 1.5
61.33 + 1.5
60.96 # 1.4
68.96
68.92
71.82
JointMtch (Ours)
87.68 1 0.5
87.64 + 0.5
66.58 + 0.7 66.09 + 0.8 76.91 + 4.5 76.79
4.6
76.95
Table 3: Performance (accuracy (%) and macro-F1 (%)) comparison with baselines different text classification datasets. JointMatch delivers best results on all benchmark datasets, surpassing the latest work SAT (Chen et al., 2022) SSTC by 5.13% average. The best results shown in blue. c: number of classes.
deviation accuracy and macro-Fl from five runs with different model parameter initialization.
ahels/Class
100
FixMatch
56.13
65.07
63.85
68.07
Ensemhlle
65.92
68.40
Jointatth
64.08
65.52
67.51
67.24
69.74
71.39
Baselines. We compare JointMatch with several popular and recent approaches: UDA (Xie al., 2020), MixText (Chen al., 2020), FixMatch (Sohn al., 2020) and SAT (Chen et al., 2022). We also compare to the vanilla ensemble of FixMatch and FreeMatch (Wang et al., 2023b) to further demonstrate the effectiveness of JointMatch.
Table 4: Accuracy Yahoo! Answer with varying
numbers of labeled data.
#LabelsCCaass
10
15
100
1000
FixMarch
70.29
80.25
82.32
85.96
87.79
89.57
FixMatch-Essenl Ensemble
72.01
80.86
84.14
85.33
86.86
88.61
FreeMatch Ensemble
87.79
JoimMatch
86.00
87.68
88.33
88.50
89.07
90.29
Implementation Details. Following (Chen et al., 2022, 2020), we use the BERRTbased-uncase model as our backbone model and the Hugging- Face Transformers (Wolf al., 2020) library for the implementation. We adopt the same data aug- mentation techniques for fair comparisons, i.e., synonym replacement for weak augmentation and Buck-translation for strong augmentation, in all baselines. In detail, for back translation, we trans- late texts into German and then translate them back ran-
Table 5: Accuracy on AG News with varying numbers of labeled data. JointMatch delivers significant improve- ments over FixMatch, especially for extremely low-shot settings, indicating its effectiveness in leveraging unla- beled data.
Emputhetic Dialognes
Average
Dataset
Hurricane
Emotions
Accuracy
Classes
32
27
8
BERT (2019)
21.63
28.86
71.72
40.74
FixMatch (2020)
24.40
30.06
73.44
42.63
into English; for synonym replacement, domly substitute 30% of words with WordNet syn- onyms. A complete list of our hyper-paaee parameters is provided in Appendix A, and our code is released.
SAT (2022)
29.25
30.36
74.06
44.56
FreeMaich (2023)
30.15
32,74
77.03
46.64
SoftMathh (2023)
29.96
31.84
74.38
45.39
JointMatch (Ours)
38.40
Table 6: Accuracy results diverse datasets with larger number of classes. JointMatch consistently out- performs other methods, showing that our approach also works well in settings with larger number of classes.
3.2 Comparison with Baselines
We summarize the comparison with basclines on different text classification datasets in Table Fol- lowing SAT (Chen et al., 2022), we randomly sam- ple N₂ samples per class as labeled data for train-
ing. No is set to 10 for both the AG News and IMDB datasets, and 20 for the Yahoo! dataset.

---

Datasets
# Classes
Total
Train
Val
Test
Method
AG News
Yahoo!
Empathetic Dialogues
32
24,850
19,533
2,770
2,547
JointMatch
88,39
68.32
Goflmotions
27
29,425
23,485
2,956
2,984
Hurricane
8
12,800
10,240
1,280
1,280
Adaptive Threshold
82.97
66.42
Cross Labcling
82.84
65.62
Table 7: Statistics and split information of the added datasets, All numbers shown here are the total number of data in all classes.
Disagree Weights
87.89
67.47
All (FixMatch)
80.25
63.80
Table Accuracy after removing different parts of JointMatch on AG News with 10 labels per class, Yahoo! Answer with 30 labels per class.
The results are averaged over 5 runs with different model parameter initialization. We observe that our JointMatch consistently demonstrates the best performances across all datasets, surpassing the best baseline by 5.13% on average. On the most challenging Yahoo! Answer dataset, JointMatch also significantly outperforms the best baseline by 5.25% accuracy and 5.13% macro-F1. These sub- stantial improvements indicate the effectiveness of our diverse and collaborative eseudo-labeling framework JoinMatch. We further provide detailed ablation studies and analysis in the next section.
include one more recently published and compet- itive cemi-supervised learning method SoftMatch (Chen et al., 2023) for comparison. JointMatch consistently delivers improvement over other meth- ods, indicating that our approach also works well in settings with a larger number of classes. Table 7 provides the statistics and split information of the added datasets. We provide short descriptions of these datasets in Appendix C.
Ablation Study and Analysis
3.3 Varying the Number of Labeled Data
We also conduct experiments with varying num- bers of labeled data to demonstrate the accuracy improvement of our JointMatch over FixMatch on Yahoo! Answers and AG News datasets. To en- sure fair comparison, we also include the perfor- mance of the vanilla ensemble of FixMatch, where we train two FixMatch models with different ini- tializations and average their predicted probability distributions to obtain the result for the ensemble model. As shown in Table 4 and Table 5, Joint- Match outperforms the vanilla ensemble of Fix- Match with different numbers of labeled data on both datasets. This indicates that our improvement does not come simply from the model ensemble. Noteworthy is that, JointMatch offers remarkable improvements over this vanilla ensemble model, especially with extremely limited labeled data: 7% on Yahoo! Answer with labels per class and sur- prisingly 13.99% on AG News with 5 labels per class. This further validates the effectiveness of JointMatch in utilizing unlabeled data and demon- strates its capability and potential to be used in real-world scenarios.
4.1 Effectiveness of Each Component
To show the effectiveness of each component JointMatch, we also measure the accuracy perfor- mance of JointMatch after removing different parts in Table 8. We observe that the performance de- creases after stripping each component, suggest- ing that all components in JointMatch contribute to the final performance. The performance of JointMatch drops most significantly after removing cross-labeling on both datasets, justifying the bene- fits of two models teaching each other and its help in alleviating error accumulation. Adaptive local thresholding performs a similarly vital role in the final performance of JointMatch. This indicates the effectiveness of adjusting local thresholds based on current classwise learning status.
Removing disagree weights, i.e., weighted dis- agreement and agreement updates, also gives a per- formance drop, although smaller compared to the other two components. We hypothesize that the rea- son for this is that the current fixed disagreement weights are not optimal. They should be adaptively adjusted based on the degree of disagreement be- tween the two networks at different times. This topic is outside the focus and scope of this paper and we leave it for future exploration. Despite this, we show the benefit of disagreement weights and provide an ablation study on its values in the next section.
3.4 Generalizability Results
To show the generalizability of JointMatch, we add experiments on three datasets with larger number classes (c=32, 27, 8). Table summarizes the accuracy results the 10-shot setting. We also

---

MoMatt'ss Mairh sinn rrows lahwing
0.8 JaintMatich
0.8
0.6
Accurscy
0.6
0.4
02
0.2
jointMatcch wio cross labeling
jointhltch
Iterations
Iteracions
(a) Quantity
(b) Quality
Figure 3: The quantity and quality of pseudo labels AG News with 10 labels per class. Ratio refers the fraction of pscudo-labeled data that participate in the current training iteration. JointMatch improves the quantity of pscudo labels at the early training stage, while maintaining a high quality of pseudo labels throughout the whole training process.
0.3
0.5
0.7
0.9
to learn from each other.
Accuracy
87.22
87.55
87.89
88.22
88.39
87.71
4.3 Quantity & Quality of Pseudo Labels
Table 9: Accuracy on AG News with 10 labels per class and various disagreement weights. Updating networks by only agreement samples (& = 0) and only disagree- 1) perform worse than updating
JointMatch improves both the quality and quan- tity of pseudo labels used for training, As shown in Figure 3a, FixMatch gradually leverages more pseudo labels as training advances, ultimately uti- lizing about 80% of the available pseudo labels. However, the cost is that the accuracy of pseudo la- bels keeps dropping, eventually falling below 40%, as indicated in Figure 3b. This consistent with our motivation in Section 2.3 that directly using noisy pseudo labels will cause models to accumu- late errors and gradually produce even more noisy pscudo labels. On the contrary, JointMatch main- tains high accuracy of pseudo labels throughout the entire training process (Figure 3b), which vali- dates the effectiveness of cross-labeling prevent- ing error accumulation. Furthermore, JoinMatch generates a greater number of pseudo labels than FixMatch during the early stages of training (Fig- ure 3a) without sacrificing their quality.
ment samples by both types of data. Weighting more disagreement samples further delivers better results.
4.2 Influence of Disagreement Weights
We now analyze the benefits of the weighted dis- agreement & agreement update. Table 9 shows how our approach performs with different disagreement weights 5 on AG News with 10 labels per class. Note that (1) 8 0 means updating networks by only agreement samples, 5 1 means updating by only disagreement samples and 8 0.5 means updating by both types of samples and weighting them equally; (2) All samples used for updating networks are firstly high confidence samples.
We observe that updating by either only agree- ment samples or only disagreement samples per- forms worse than updating by both kinds of sam- ples. This validates our assumption both agreement samples and disagreement samples are beneficial for training, as agreement samples provide high- quality pscudo-labels while disagrcement samples offer sample diversity and keep the two networks diverse. In addition, weighting more disagreement samples can obtain better results than equal weight- ing, which emphasizes the importance of keeping the two networks diverged and thus enabling them
Related Work
Semi-Supervised Text Classification. Semi- supervised learning has attracted a lot of attention in the field of text classification due to its abil- ity to leverage large amounts of unlabeled data with limited labels. UDA (Xie al., 2020) in- troduces strong data augmentation and proposes a consistency loss to minimize the distance of pre- dicted distributions between differently perturbed data. Text (Chen et al., 2020) leverages Mixup

---

(Zhang et al., 2018) to interpolate unlabeled data and labeled data in hidden space avoid overfit- ting the limited labeled data. S2TC-BDD (Li et al., 2021) 1) notices the margin bias issue and addresses it by balancing label angle variances. PCM (Xu et al., 2022) exploits the inherent semantic match- ing capability inside pre-trained language models to benefit SSTC. AUM-ST (Sosea and Caragea, 2022) builds self-training and uses Area Under the Margin (Pleiss al., 2020) filter possibly inaccurate pseudo-labels, Hosseini and Caragea (2023) combine two models in a co-training fash- ion, one being trained using unsupervised domain adaptation from source to a target domain and the other using semi-supervised learning in the target domain where the two models iteratively teach each other by interchanging their high confi- dent predictions. CrisisMatch (Zou et al., 2023) studies several popular coni-supervised compo- nents and integrates Mixup with pseudo-labeling (Lee et al., 2013) for disaster tweet classification. Building upon the idea of FixMatch (Sohn et al., 2020) that uses pseudo-labels generated by weakly- augmented data to teach the recent work SAT (Chen et al., 2022) proposes re-rank different weak and strong augmentations according to their similarity with the original in- put. However, these works ignore the difficulties learning different classes at different time steps. Inspired by recent FlexMatch (Zhang et al., 2021) and FreeMatch (Wang et al.. 2023b), our Joint- Match introduces adaptive thresholding to SSTC to dynamically adjust classwise thresholds based the learning status of each class.
Match integrates their ideas into semi-supervised text classification. We provide a comprehensive comparison between closely related approaches with our JointMatch in Table 1.
Conclusion
In this paper, we propose JointMatch, semi- supervised text classification approach that inte- grates ideas from recent semi-supervised learning and learning with noise. Our method is motivated by observed problems in cemi-supervised text clas- sification (SSTC); model bias towards easy classes and error accumulation. JointMatch utilizes adap- tive thresholding, cross-labeling, and weighted dis- agreement & agreement updates to address these issues effectively. Through extensive experiments three standard SSTC benchmark datasets, we found that JointMatch significantly outperforms previous works on all datasets and demonstrates surprising improvement over FixMatch on an ex- tremely scarce label setting. We also show that JointMatch can generalize well to datasets that have large number of classes.
data,
Limitations
There several limitations our work. First, our method trains two networks simultaneously, re- sulting in a higher computational cost and longer training time. For example, each experiment on AG News takes approximately 2.5 hours on an A5000 GPU for JointMatch, while Fix only requires 1.25 hours. However, the additional computational cost is not significant issue in low-resource set- tings, as experiments in such settings typically do not take much time. Second, our weighted dis- agreement & agreement update could be further improved by adaptively adjusting the disagreement weight based on the degree of mutual agreement among networks and their confidence at different times. This will be explored in the future.
Learning with Noise. The task of learning with noise aims to train robust networks against label noise. Co- Co-Teaching (Han et al., 2018) proposes to simultaneously train two networks and mutually select small-loss instances teach its peer, thus avoiding direct eиror accumulation within one net- work. Nevertheless, as the training goes on, the two networks may converge, reduce to one iden- tical network, and the problem of error accumula- tion resurfaces. Decoupling (Malach and Shalev- Shwartz, 2017) proposes to update models only by the data receiving different predictions from the networks. Co-Teaching+ (Yu et al., 2019) ob- serves that this strategy can keep the two networks diverged and further boost the performance of Co- Teaching. While these approaches are originally designed for fully supervised settings, our Joint-
Acknowledgements
This research is partially supported by National Science Foundation (NSF) grants IIS-1912887 IIS- 2107487, and ITE-2137846. Any opinions, find- ings, and conclusions expressed here are those of the authors and do not necessarily reflect the views of NSF. We thank our reviewers for their insightful feedback and comments which helped improve the quality of our paper.

---

References
2018. Co-teaching: Robust training of deep neural networks with extremely noisy labels. Advances in neural information processing systems, 31.
Firoj Alam, Umair Qazi, Muhammad Imran, and Ferda disaster in- cidents data from twitter with deep learning bench- marks. In Proceedings of the International AAAI Conference on Web and social media, volume 15, pages 933-942.
Ofli, 2021. Humaid:
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778.
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. 2020. Pseudo- labeling and confirmation bias in deep semi- supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IGEE.
Mahshid Hosseini and Cornelia Caragea. 2023. Semi- supervised domain adaptation for tasks. In Findings of the Association for Compu- national Linguistics: ACL 2023, pages 5402-5410, Toronto, Canada, Association for Computational Lin- guistics.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raf- fel. 2019. Mixmatch: A holistic approach to semi- supervised learning, Advances in neural information processing systems, 32.
Dong-Hyun Lee et al. 2013. Pscudo-label: The simple and efficient semi-supervised learning method for deep neural networks. Workshop on challenges in representation learning, ICMI., volume 3, page 896.
Avrim Blum and Tom Mitchell. 1998. Combining la- beled and unlabeled data with co-training. In Pro- ceedings of the eleventh amnual conference on Com- putational learning theory, pages 92-100.
Changchun Li, Ximing Li, and Jihong Ouyang. 2021, Semi-supervised text classification with balanced deep representation distributions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages Association for Computational Linguistics.
Ming-Wae BeiggWeiiCe Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representa- tion: Dataless classification. In Proceedings of the 23rd National Conference on Artificial Intelligence. Volume 2, AAAI'08, page 830-835. AAAI Press.
Online.
Andrew L. Maas. Raymond E. Daly, Peter T. Pham. Dan Huang, Andrew Y. Ng. and Christopher Potts. 2011 1. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang. Bernt Schicle, Xing Xie, Bhiksha Raj. and Marios Savvides. 2023. Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learn- ing. In The Eleventh International Conference on Learning Representations.
Hui Chen, Wei Han, and Soujanya Poria. 2022. SAT: Improving semi-supervised text classification with self self-training In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6141-6146, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Eran Malach and Shai Shalev-Shwartz. 2017. Decou- pling" when to update" from" how to update". Ad- vances in neural information processing systems, 30.
simple mnstance-adaptin
Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kil- ian Q. Weinberger. 2020. Identifying mislabeled data using the area under the margin ranking. In Proceed- ings of the 34th International Conference on Neu- ral Information Processing Systems, NIPS'20 20, Red Hook, NY, USA. Curran Associates Inc.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix- Text: Linguistically-inormes
interpolation of hid- den space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2147- 2157, Online. Association for Computational Lin- guistics.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open- domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 5370-5381, Florence, Italy. Association for Computational Linguistics.
Dorottya Demszky, Dana Movshovitz-Atiias Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine- grained emo- tions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040-40544 Online. Association for Computational Linguisties.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. 2016. Regularivation with stochastic transformations and perturbations for deep semi-supervised learning. Advances neural information processing systems. 29.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama.

---

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shi- nozaki. 2021. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34: 18088- 18419.
Tiberiu Sosea and Comelia Caragea. 2022. Leveraging training dynamics and self-training for text classifi- cation. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 4760-4762, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. International Conference on Learning Representations.
Xiang Zhang, Jumbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. Advances in neural information processing systems, 30.
Henry Peng Zou, Yue Zhou, Cornelia Caragea, and Doina Caragea. 2023. Semi-supervised few-shot learning for fine-grained disaster tweet classification. In Proceedings of the 20th International ISCRAM Conference, pages 385 395.
Fangxin Wang. Lu Cheng, Ruocheng Guo, Kay Liu, and Philip Yu. 2023a. Equal opportunity of coverage in fair regression. In Advances in Neural Information Processing Systems.
Yidong Wang. Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang. Marios Savvides, Takahiro Shinozaki, Bhiksha Raj. Bernt Schiele, and Xing Xie. 2023b. Freematch: Self-adaptive thresh- olding for supi-supe learning, In The Eleventh International Conference on Learning Representa- tions.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Seao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, Association for Computational I Linguistics.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. Advances in neural information procexsing systems, 33:6256-6268.
Haiming Xu, Lingqiao Liu, and Ehsan Abbasnejad. 2022. Progressive class semantic matching for semi- supervised text classification. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. pages 3003-3013, Seat- tle. United States. Association for Computational Linguistics.
Xingrui Yu, Bo Han, Jiangchao Yao. Gang Niu, Ivor Tsang, and Masashi Sugiyama. 2019. How does disagreement help generalization against label cor- ruption? In International Conference on Machine Learning, pages 7164-7173. PMLR.

---

A Hyperparameters
Fixed Threshold
Accuracy
Macro-Fl
A complete list JointMatch hyperparameters on all evaluated datasets are provided in Table 10.
0
83.13
83.32
0.25
83.13
83.32
0.5
85.21
85.13
AG News
Yahoo! Answer
IMDB
0.75
87.32
87.37
Batch Size B
4
0.9
87.47
87.49
Learning Rate Ir
le-5
2e:5
2e-5
0,95
87.37
87.3
Unsupervises Loss Weight
0.05
0.98
88.39
88.32
EMA decay A
0.9
0.99
87.84
87.82
Fixed Threshold
0.98
Disagreement Weight
Table 12: Ablation study on the fixed threshold.
Unlabeled
Table 10: Complete list of JointMatch hyperparameters on AG News, Yahoo! Answer, IMDB.
performance drop, as less number of unlabeled can be utilized.
B.3 Unlabeled Data Ratio in MiniBatch
B Additional Ablation Study Results
In Table 13, we present the results of JointMatch with different unlabeled data to labeled data ratio μ. One can observe that using large amount of unlabeled data help increase performance. A very high value of μ is encouraged since it slows down the training while just giving marginal improvements.
This section provides additional ablation study re- sults on several hyperparameters as complementary to the main paper. All experimental results here are obtained from JointMatch on AG News with 10 labels per class.
B.1 EMA Decay
Table 11 shows the result of JointMatch with differ- ent EMA decay À for estimating classwise learning status. Note that À = 1 means the estimated learn- ing status for all classes always equals their initial value 1/C and adjusting local thresholds (C is the number of classes). We observe that adjust- ing local thresholds based on estimated learning (A E [0, 1)) is significantly better than not adjusting local thresholds (A = 1).
Unlabeled Data Ratio
Accuracy
Macro-FF
86.43
86.39
3
86.74
86.71
5
88.08
88.04
10
88.39
88.32
15
88.22
88.22
20
88.45
88,46
30
88.49
88.50
Table 13: Ablation Study Unlabeled Data Labeled
EMA Decay
Accuracy
Macro-F1
Data Ratio.
87.13
87.06
0.25
86.93
86.95
C Short Descriptions of Added Datasets
0.5
87.61
87.59
0.9
88.39
88.32
We provide short description of these datasets here: (1) GoEmotions (Demszky al., 2020) a dataset of Reddit comments labeled with 27 emo- tions, such as amusement, fear, and gratitude. (2) Empathetic Dialogues (Rashkin et al., 2019) con- sists of conversations between a speaker and lis- tener and is labeled with 32 fine-grained emotions. (3) Hurricane is crisis tweet dataset sampled from HumAID (Alam et al., 2021). It contains human- labeled tweets collected during hurricane disasters and includes crisis-related classes, such as infras- tructure and utility damage, displaced people and evacuations.
0.99
87.28
87.28
83.68
83.8
Table 11: Ablation study EMA decay.
B.2 Fixed Confidence Threshold
In Table 12, show the performance of Joint- Match with varying fixed confidence threshold It can be seen that setting a threshold higher than 0.75 is generally beneficial for good performance, but setting it to a too high value will lead to some


Reviews:
[{"review_id": "lwzbd8sTkg", "reviewer": "Reviewer_3ZiN", "summary": "", "questions": "Refer above.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your valuable comments and assessment of our paper! \n\n> Q1: All the three techniques of adaptive local thresholding, cross-labeling and weighted disagreement/agreement update are based on existing work. In that sense, the paper is weak on the novelty aspect.\n\nWe agree that our work combines techniques from existing works, but so do other works, e.g., FixMatch which combines consistency regularization and pseudo-labeling. However, our novelty exhibits in many aspects: (1) Our JointMatch is innovative at proposing a holistic framework that effectively unifies ideas from recent semi-supervised learning and learning with noise for semi-supervised text classification, which is also pointed out by Reviewer zxwi. The relation and difference with related approaches are also summarized in Table 1 and Section 2.5. (2) Previous works Decoupling and Co-Teaching+ completely ignore agreement data when updating their network. We argue that those agreement data are valuable learning signals and propose a new strategy that utilizes both disagreement and agreement data in a weighted manner. (3) To the best of our knowledge, our work is the **first in semi-supervised text classification** that **highlights** and **effectively addresses both** the **issues** of **error accumulation** and **bias toward easy classes** in **one unified framework**, by combining cross-labeling, weighted agreement update and adaptive thresholds.\n\n> Q2: There are some hypotheses that are not backed by sufficient analytical explanation. For instance, \"different networks can filter out different noises\" - it is unclear why two networks are enough to guarantee this? What if both networks tend to make similar errors?\n\nThank you for giving us the opportunity to clarify. We present statistics about the filtered *incorrect pseudo-label* (noise) of every 50 iterations and 4K predictions in the table below. (1) We can observe that there is a considerable amount/percentage of non-overlapping noises between two networks in different iterations, validating that different networks can filter out different noises. This hypothesis is also supported by previous work Co-Training (Blum & Mitchell, 1998) and Co-Teaching (Han et al., 2018b). (2) For those overlapping noises, i.e., the same errors made by both networks, we can see that models will reduce their percentages and improve upon them over iterations.\n\n\n| Iteration | \\# Non-Overlap Noise | \\# Overlap Noise | Non-Overlap Percentage | Overlap Percentage |\n| --------- | -------------------- | ---------------- | ---------------------- | ------------------ |\n| 50        | 594                  | 1328             | 0.145                  | 0.335              |\n| 100       | 502                  | 1044             | 0.125                  | 0.260              |\n| 150       | 509                  | 890              | 0.130                  | 0.220              |\n| 200       | 401                  | 848              | 0.100                  | 0.210              |\n| 250       | 380                  | 905              | 0.095                  | 0.225              |\n| 300       | 364                  | 819              | 0.095                  | 0.205              |\n| 350       | 281                  | 776              | 0.075                  | 0.190              |\n\n\n> Q3: The pseudo labels generated for classes with low confidence threshold - won't these have higher probabilities of being incorrect?\n\nIt is correct that pseudo-labels generated from lower confidence thresholds are more likely to be incorrect. (1) However, adaptively lowering confidential thresholds for hard classes allows for pseudo-label diversity and can help alleviate aggressive bias towards easy classes. (2) Our method is designed to deal with the case where the pseudo-labels are potentially incorrect. First, as the training goes on, the model will improve its accuracy in these hard classes and progressively increase their confidence thresholds to filter out low-quality pseudo-labels. Second, double-networks, cross-labeling and weighted-agreement-update can effectively maintain a high quality of pseudo-labels throughout the whole training process, as visualized in our Figure 3.\n\nHope our response could resolve your concern. Please let us know if you have further questions. Thanks again!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While deep learning has become ubiquitous, training these models often requires large amounts of data that has cost and environmental impact. Data efficient learning, including semi-supervised learning that this paper deals with, is an important research direction. The authors highlight two important challenges with the current semi-supervised classification approaches and propose ways to address these. The proposed techniques, although inspired from existing work, are gracefully combined into a holistic approach. \nThey also make the code available which will help practitioners leverage this technique. Researchers can replicate the work and potentially extend it further.\nThe approach is well described and the paper is generally well written.\nThe evaluation is thorough and offers sufficient support to the claims around addressing the challenges highlighted earlier in the paper.", "reasons_to_reject": "All the three techniques of adaptive local thresholding, cross-labelling and weighted disagreement / agreement update are based on existing work. In that sense, the paper is weak on the novelty aspect.\nThere are some hypotheses that are not backed by sufficient analytical explanation. For instance, \"different networks can filter out different noises\" - it is unclear why two networks are enough to guarantee this? What if both the networks tend to make similar errors?\nThe pseudo labels generated for classes with low confidence threshold - won't these have higher probability of being incorrect?\n\nWhile the evaluation suggests that this works well in practise, the paper can benefit from a more sound theoretical backing.", "questions_to_authors": "Refer above.", "paper_topic_and_main_contributions": "In this paper, the authors consider the semi-supervised text classification setting which has gained a lot of interest due to its promise of enhancing classification performance with limited labeled data. Specifically, they focus on collaborative pseudo labelling and consistency regularization citing two limitations of existing approaches - (1) fixed threshold for pseudo labelling leading to biased label generation for easy to learn classes; (2) progressive accumulation of pseudo labelling errors leading to eventual degradation of classification performance. \nThey then propose an approach that they refer to as JointMatch that makes use of adaptive per class thresholding to address the biased label generation and cross-labelling with two classification models trained on divergence/consensus weighted loss. Adaptive local thresholding adjusts the per class threshold based on the learning status of the class and progressively updates it to prevent starvation of hard-to-learn classes. Cross-labelling makes use of two different classification networks that are separately initialized and leverages the thesis that different networks filter out different noises. This helps in alleviating the accumulation of errors. However, co-training, the driving idea behind cross-labelling, eventually converges and degenerates to self-training. In order to retain the divergence of these networks, the authors train them on an objective that is weighted by the divergence / convergence of these networks.\nEvaluation on multiple datasets across different baseline models and ablation study supports the efficacy of the proposed approach.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Equation 2 --> missing bracket\n\"local threshold\" is introduced in section 2.1 but the notations for it come later in section 2.2"}, "extra_text": {}}, {"review_id": "hu8cVYixQt", "reviewer": "Reviewer_Tckf", "summary": "", "questions": "Would it be possible to apply the approach also in a dynamic setting, e.g. when new topics are showing up?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your valuable comments and assessment of our paper!\n\n> Q1: One major drawback of the paper is in my opinion that the number of classes of the datasets included in the experiment are relatively small (c=4, c=10, c=2). It has to be proven that the approach also works well in settings with a large number of classes.\n\nThank you for this constructive suggestion. We have **added** experiments on **three datasets** with a larger number of classes (c=32, 27, 8), and the accuracy results are shown in the table below. JointMatch consistently delivers improvement over other methods, showing that our approach also **works well in settings with a larger number of classes**. We provide short descriptions of the added datasets below. Please let us know if you are interested in any other specific datasets, we will be happy to conduct experiments immediately and add results on them.\n\n| Dataset           | Empathetic | Dialogues |  (c=32) |       | GoEmotions | (c=27) |       |       | Hurricane  | (c=8) |       |       | Average |\n|-----------------|:----------:|:---------:|:-------:|:-----:|:----------:|:------:|:-----:|:-----:|:----------:|:-----:|:-----:|:-----:|:-------:|\n| #Labels/Class     | 5          | 10        | 15      | 25    | 5          | 10     | 15    | 25    | 5          | 10    | 15    | 25    | -       |\n| BERT              | 15.54      | 21.63     | 29.25   | 33.80 | 18.26      | 28.86  | 37.86 | 44.33 | 24.92      | 53.98 | 66.25 | 71.72 | 37.20   |\n| FixMatch          | 16.53      | 24.40     | 34.75   | 37.81 | 24.60      | 30.06  | 34.65 | 42.06 | 34.30      | 51.09 | 65.62 | 73.44 | 39.11   |\n| FreeMatch         | 17.86      | 30.15     | 36.08   | 39.14 | 24.50      | 32.74  | 35.76 | 44.34 | 35.78      | 52.03 | 67.50 | 77.03 | 41.08   |\n| FixMatch-Ensemble | 20.22      | 30.31     | 37.50   | 42.25 | 27.45      | 36.16  | 39.21 | 45.04 | 35.23      | 56.48 | 67.50 | 73.67 | 42.59   |\n| JointMatch        | **22.69**      | **34.67**     | **40.40**   | **44.80** | **30.97**      | **38.40**  | **40.11** | **45.64** | **45.23**      | **73.59** | **76.41** | **78.98** | **47.66**   |\n\n\n\n\n> Q2: Concerning the performance: the standard deviation (SD) seems to be rather high for the IMDB dataset. In general: if we take e.g. for the accuracy the \"average +/- 2 times the SD\", the proposed method several times overlaps with the SAT method. So the performance improvements look less impressive from this point of view.\n\nWe conducted the **statistical t-test** to show the significant difference between the accuracy result of JointMatch (76.91 ± 4.5) and SAT (68.96 ± 1.7) on IMDB dataset. The computed t-value = 2.698 is more extreme than the critical t-value 2.306 ($\\alpha = 0.05$). This test shows that **JointMatch is significantly better than SAT on IMDB dataset**. Besides, to further address the concern about the performance, we report the average accuracy results across 6 diverse datasets in the table below. We can see that the total average performance **improvement** of JointMatch **is significant across 7 diverse datasets**.\n\n\n| Dataset           | Empathetic Dialogues | GoEmotions | Yahoo! Answers | Hurricane | AG News | IMDB   | Total Average |\n| ----------------- | -------------------- | ---------- | -------------- | --------- | ------- | ------ | ------------- |\n| \\# Classes        | 32                   | 27         | 10             | 8         | 4       | 2      | \\-            |\n| \\# Total Data     | 24,850               | 29,425     | 130,000        | 12,800    | 35,600  | 37,000 | \\-            |\n| BERT (2019)       | 21.63                | 28.86      | 58.11          | 71.72     | 69.18   | 63.16  | 52.11         |\n| FixMatch (2020)   | 24.4                 | 30.06      | 60.17          | 73.44     | 80.22   | 64.52  | 55.47         |\n| SAT (2022)        | 29.25                | 30.36      | 61.51          | 74.06     | 86.38   | 65.43  | 57.83         |\n| SoftMatch (2023)  | 29.96                | 31.84      | 64.83          | 74.38     | 82.68   | 67.92  | 58.60         |\n| FreeMatch (2023)  | 30.15                | 32.74      | 62.11          | 77.03     | 85.96   | 70.10  | 59.68         |\n| JointMatch (Ours) | **34.67**                | **38.4**       | **66.58**          | **78.98**     | **87.68**   | **76.91**  | **63.87**         |\n\n\n> Q3: The details on the data augmentation are not provided.\n\nThanks for bringing this to our attention. Following previous work SAT (Chen et al., 2022), we use back translation as strong data augmentation and employ synonym replacement as weak data augmentation. In detail, for back translation, we translate texts into German and then translate them back into English; for synonym replacement, we randomly substitute 30% of words with WordNet synonyms. The code of our augmentation techniques will be provided.\n\n> Q4: Would it be possible to apply the approach also in a dynamic setting, e.g. when new topics are showing up?\n\nThank you for pointing it out. This is an interesting direction and deserves another study on different scenarios: specifically, when new topics arise, we may have different scenarios, for example, (1) limited labeled data and limited unlabeled data, or (2) limited labeled data along with large amounts of unlabeled data. Given our encouraging results on the extremely low-shot setting, we believe our method has the potential to be effective in these scenarios. We will explore these in the future and will include a discussion on this in our final version of the paper.\n\nWe sincerely hope our response can resolve your concerns. But if you have any further concerns or questions, please let us know, and we will do our best to address them as soon as we can.\n\n---\n**Short Descriptions of the Added Datasets**: [1] GoEmotions (Demszky et al., 2020) is a dataset of Reddit comments labeled with 27 emotions, such as amusement, fear, and gratitude. [2] Empathetic Dialogues (Rashkin et al., 2019) consists of conversations between a speaker and listener and is labeled with 32 fine-grained emotions. [3] Hurricane is a crisis tweet dataset sampled from HumAID (Alam et al., 2021). It contains human-labeled tweets collected during hurricane disasters and includes 8 crisis-related classes, such as infrastructure_and_utility_damage, displaced_people_and_evacuations. We will provide a more comprehensive description of datasets, statistics and split information in our paper.\n\n[1] Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, Online. Association for Computational Linguistics.\n\n[2] Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Computational Linguistics.\n\n[3] Firoj Alam, Umair Qazi, Muhammad Imran, and Ferda Ofli. 2021. Humaid: human-annotated disaster incidents data from twitter with deep learning benchmarks. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 933–942.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written. It contains most of the relevant information to follow the ideas and method setup. In the experimental study the approach shows better performance than the other methods with which it is compared. The method seems to work well even with a very small number on labeled cases per class.", "reasons_to_reject": "One major drawback of the paper is in my opinion that the number of classes of the datasets included in the experiment are relatively small (c=4, c=10, c=2). It has to be proven that the approach also works well in settings with a large number of classes.\n\nConcerning the performance: the standard deviation (SD) seems to be rather high for the IMDB dataset. In general: if we take e.g. for the accuracy the \"average +/- 2 times the SD\", the proposed method several times overlaps with the SAT method. So the performance improvements look less impressive from this point of view.\n\nThe details on the data augmentation are not provided.", "questions_to_authors": "Would it be possible to apply the approach also in a dynamic setting, e.g. when new topics are showing up?", "paper_topic_and_main_contributions": "The paper introduces a new approach called JointMatch for semi-supervided text classification. It follows the idea of pseudo-labeling while avoiding the well-known problem of bias and error accumulation in simple pseudo-labeling approaches with a fixed threshold.It gives a good overview of how the new approach can be compared to closely related methods (table 1).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}, {"review_id": "GehiO5IM7Z", "reviewer": "Reviewer_zxwi", "summary": "", "questions": "Please add more experiments and I will reconsider my score.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your valuable comments and assessment of our paper!\n\n> Q1: It would be interesting to see how JointMatch performs compared to other semi-supervised learning methods, not just the baseline methods. A broader comparison could have been beneficial.\n\n> Q2: Depending on the experimental results and discussions, there could be limitations in the evaluation of the approach. For instance, the performance of the approach might not be tested across a sufficiently diverse range of datasets or problem domains. Please add more experiments and I will reconsider my score.\n\nThank you for these constructive suggestions. We have **added more** recent semi-supervised methods and diverse datasets. The table below compares the accuracy of JointMatch with **recently published and competitive semi-supervised learning methods**: SoftMatch (Chen et al., 2023 ICLR), FreeMatch (Wang et al., 2023 ICLR), SAT (Chen et al., 2022 EMNLP) and baseline FixMatch (Sohn et al., 2020 NeurIPS). We conducted experiments on **six different datasets**, covering a **diverse** range of problem **domains**. The results show that JointMatch consistently outperforms these powerful semi-supervised learning methods on diverse datasets and tasks. A more detailed table will be added to our paper. Furthermore, we provide a short description of these datasets below. If you are interested in any other specific methods, datasets or experiments, please let us know, and we will be happy to conduct experiments immediately and include them in our paper. Thanks again for being willing to reconsider your scores.\n\n\n| Dataset           | Empathetic Dialogues | GoEmotions | Yahoo! Answers | Hurricane | AG News | IMDB   | Total Average |\n| ----------------- | -------------------- | ---------- | -------------- | --------- | ------- | ------ | ------------- |\n| \\# Classes        | 32                   | 27         | 10             | 8         | 4       | 2      | \\-            |\n| \\# Total Data     | 24,850               | 29,425     | 130,000        | 12,800    | 35,600  | 37,000 | \\-            |\n| BERT (2019)       | 21.63                | 28.86      | 58.11          | 71.72     | 69.18   | 63.16  | 52.11         |\n| FixMatch (2020)   | 24.4                 | 30.06      | 60.17          | 73.44     | 80.22   | 64.52  | 55.47         |\n| SAT (2022)        | 29.25                | 30.36      | 61.51          | 74.06     | 86.38   | 65.43  | 57.83         |\n| SoftMatch (2023)  | 29.96                | 31.84      | 64.83          | 74.38     | 82.68   | 67.92  | 58.60         |\n| FreeMatch (2023)  | 30.15                | 32.74      | 62.11          | 77.03     | 85.96   | 70.10  | 59.68         |\n| JointMatch (Ours) | **34.67**                | **38.4**       | **66.58**          | **78.98**     | **87.68**   | **76.91**  | **63.87**         |\n\n\n> Q3: The complexity of the JointMatch approach, which might limit its applicability or use in certain scenarios.\n\nWe acknowledge the complexity of JointMatch since it unifies ideas from different domains to address two critical issues in semi-supervised text classification. However, it yields significant improvements in diverse datasets and settings. If the user does not want to deal with the full complexity of the model, they can remove some modules, such as disagreement and agreement weights. The model will still perform well ( though not as well as the full model), as shown in the ablation study from Table 5 in the paper. Besides, we provide a legible algorithm of JointMatch in the paper and release our code to help users understand and utilize our method.\n\n---\n\n**Short Descriptions of Datasets:** [1] GoEmotions (Demszky et al., 2020) is a dataset of Reddit comments labeled with 27 emotions, such as amusement, fear, and gratitude. [2] Empathetic Dialogues (Rashkin et al., 2019) consists of conversations between a speaker and listener and is labeled with 32 fine-grained emotions. [3] Hurricane is a crisis tweet dataset sampled from HumAID (Alam et al., 2021). It contains human-labeled tweets collected during hurricane disasters and includes 8 crisis-related classes, such as infrastructure_and_utility_damage, displaced_people_and_evacuations. Next are the datasets that are already in the paper: [4] Yahoo! Answers classifies question-answer pairs into 10 topics, such as Sports, Health, and Education. [5] AG News categorizes news articles into 4 categories, such as World, Sci/Tech, and Business. [6] IMDB is a binary sentiment analysis dataset consisting of movie reviews. We will provide a more comprehensive description of datasets, statistics, and split information in our paper.\n\n[1] Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, Online. Association for Computational Linguistics.\n\n[2] Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards empathetic open domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Computational Linguistics.\n\n[3] Firoj Alam, Umair Qazi, Muhammad Imran, and Ferda Ofli. 2021. Humaid: human-annotated disaster incidents data from twitter with deep learning benchmarks. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 933–942.\n\n[4] Ming-Wei Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dataless classification. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2, AAAI’08, page 830–835. AAAI Press.\n\n[5] Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.\n\n[6] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an important issue in semi-supervised text classification, i.e., pseudo-label bias and error accumulation.\n2. The JointMatch approach is innovative, combining recent ideas from semi-supervised learning and learning with noise.\n3. The paper provides experimental evidence of the effectiveness of JointMatch, including in scenarios with very few labels.\n4. The approach introduced in the paper is robust and can handle disagreement data and high-quality agreement data effectively.", "reasons_to_reject": "1. The complexity of the JointMatch approach, which might limit its applicability or use in certain scenarios.\n2. It would be interesting to see how JointMatch performs compared to other semi-supervised learning methods, not just the baseline methods. A broader comparison could have been beneficial.\n3. Depending on the experimental results and discussions, there could be limitations in the evaluation of the approach. For instance, the performance of the approach might not be tested across a sufficiently diverse range of datasets or problem domains.", "questions_to_authors": "Please add more experiments and I will reconsider my score.", "paper_topic_and_main_contributions": "The paper deals with the topic of Semi-supervised Text Classification (SSTC), an area that has been receiving increasing attention because of its ability to leverage unlabeled data. The central issue that the paper addresses is the problems of pseudo-label bias and error accumulation in SSTC, which are common issues in existing approaches that use pseudo-labeling.\n\nThe main contribution of the paper is the proposal of a new approach, JointMatch, which unifies ideas from recent semi-supervised learning and learning with noise. The approach adjusts class-wise thresholds based on the learning status of different classes to mitigate model bias towards currently easy classes. It also alleviates error accumulation by utilizing two differently initialized networks to teach each other in a cross-labeling manner.\n\nAdditionally, JointMatch introduces a strategy that weights more disagreement data while also allowing the utilization of high-quality agreement data for training. The paper presents experimental results on benchmark datasets that demonstrate the superior performance of JointMatch, achieving a significant improvement. Remarkably, JointMatch delivers impressive results even in the extremely scarce label setting.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes JointMatch which addresses the challenges (pseudo-label bias and error accumulation) in semi-supervised text classification (SSTC). 

JointMatch combines three existing techniques (cross-labeling, per-class adaptive thresholding, and weighted (dis) agreement update) in order to mitigate two important problems in SSTC: model bias towards easy-to-learn classes and error accumulation. Experiments on multiple datasets indicate the potential efficacy of JointMatch.

There were worries that the proposed method is only a combination of existing work and not novel for that reason, however, the holistic combination of previous methods and experimental analyses of Jointmatch were recognized as innovative by multiple reviewers. Additionally, the authors added additional results: recent semi-supervised learning models as baselines and experiments of JointMatch on larger label sizes (>10). 

I recommend authors reflect new experiment results and clarifications in the manuscript.
Decision: Accept-Main