DeCrisisMB: Debiased Semi-Supervised Learning for Crisis Tweet Classification via Memory Bank
Authors: Henry Peng Zou
Yue Zhou
Weizhi Zhang
Cornelia Caragea
Venue: EMNLP 2023 Findings (2023)
Keywords: Semi-Supervised Learning
Debiasing
Crisis Tweet Classification

Abstract: During crisis events, people often use social media platforms such as Twitter to disseminate information about the situation, warnings, advice, and support. Emergency relief organizations leverage such information to acquire timely crisis circumstances and expedite rescue operations. While existing works utilize such information to build models for crisis event analysis, fully-supervised approaches require annotating vast amounts of data and are impractical due to limited response time. On the other hand, semi-supervised models can be biased, performing moderately well for certain classes while performing extremely poorly for others, resulting in substantially negative effects on disaster monitoring and rescue. In this paper, we first study two recent debiasing methods on semi-supervised crisis tweet classification. Then we propose a simple but effective debiasing method, DeCrisisMB, that utilizes a Memory Bank to store and perform equal sampling for generated pseudo-labels from each class at each training iteration. Extensive experiments are conducted to compare different debiasing methods' performance and generalization ability in both in-distribution and out-of-distribution settings. The results demonstrate the superior performance of our proposed method. Our code is available at https://github.com/HenryPengZou/DeCrisisMB.

DeCrisisMB: Debiased Semi-Supervised Learning for Crisis Tweet Classification via Memory Bank
Henry Peng Zou
Yue Zhou
Weizhi Zhang
Cornelia Caragea
Computer Science
University of Illinois Chicago
(pzou3,yzhou23) yzhou232, wzhan42, corneliajeuic. edu
Abstract
2016; Li al., 2017, 2018; Neppalli al., 2018: Ray Chowdhury et al., 2020; Sosea al., 2021). On the other hand, current semi- semi-superrised mod- els can be biased, performing moderately well for certain classes while extremely worse for others, resulting in detrimentally negative effect on dis- aster monitoring and analysis (Alam et al., 2018 8; Ghosh and Desarkar, 2020; Sirbu al., 2022: Zou et al., 2023; Wang et al., 2023a). For instance, ne- glecting life-essential classes, such as requests or urgent needs, displaced people evacuations and injured or dead people, can have severely adverse consequences for relief efforts. Therefore, is cru- cial to mitigate bias in semi- sumi-supervieed approaches for crisis event analysis.
During crisis events, people often use social media platforms such as Twitter to dissemi- nate information about the situation, warnings, advice, and support. Emergency relief organi- zations leverage such information to acquire timely crisis circumstances and expedite rescue operations. While existing works utilize such information build models for crisis event analysis. fully- sully-supervised approaches require annotating vast amounts data and are practical due to limited response time. On the other hand, semi-supervised models can be bi- ased, performing moderately well for certain classes while performing extremely poorly for others, resulting in substantially negative ef- fects on disaster monitoring and rescue. In this paper, we first study two recent dehiasing meth- ods on supervised crisis tweet classifica- tion. Then propose simple but effective debiasing method, DeCrisisMB, that utilizes Memory Bank to store and perform equal sam- pling for generated pseudo-labels from each class at each training iteration. Extensive ex- periments are conducted compare different debiasing methods' performance and general- ization ability in both in-distribution and out- of-distribution settings. The results demon- strate the superior performance of our pro- posed method. Our code is available at https: //gi thub. com/ cwwm//eeerrPegggoo/eerriiii
In this paper, we investigate and observe that bias in semi-supervised learning can be related to inter- class imbalances in terms of numbers and accura- cies of pscudo-labela produced during training. We do this analysis using representative work in semi- supervised learning, Pseudo-Labeling (PSL) (Lee et al., 2013; Xie et al., 2020a; Sohn al., 2020). We then study two different debiasing methods for semi-supervised learning on the task of crisis tweet classification. These two state-of-the-art semi- supervised debiasing approaches are: Debiasing via Logits Adjustment (LogitAdjust) (Wang et al.. 2022) and Dehiasing via Self-Adaptive Threshold- ing (SAT) (Wang et al., 2023b). Our analysis show that although these methods have effects in debi- asing and balancing pseudo labels across classes, their debiasing performance is still unsatisfying and there are drawbacks that need to be addressed. LogitAdjust debiases the pseudo-labeling process by explicitly adjusting predicted logits based on the average probability distribution over all unla- beled data. However, we observe that this explicit adjustment makes it difficult for models fit data, leading to unstable training, and their classwise pseudo-labele are still highly imbalanced. SAT proposes to dynamically adjust global and local
1 Introduction
During natural disasters, real-time sharing of cri- sis situations, warnings, advice and support on so- cial media platforms is critical in aiding response organizations and volunteers to enhance their sit- uational awareness and rescue operations (Varga et al., 2013; Vieweg et al., 2014). Although exist- ing works utilize such information build models for crisis event analysis, standard supervised ap- proaches require annotating vast amounts of data during disasters, which is impractical due to lim- ited response time (Li et al., 2015; Caragea et al.,

---

thresholds of pseudo-labeling to enforce poorly- learned calegories generating more pseudo-labels. This approach does help produce more balanced pseudo-labell but comes at the cost of sacrificing the accuracy of the pseudo-labels (in-depth anal- ysis and visualized comparisons are provided in Section 6.)
significant progress in improving disaster relief ef- forts (Imran al., 2013; Li et al., 2015; Imran et al., 2015; Li et al., 2017. 2018; Neppalli et al., 2018; Mazloom et al., 2018; Ray Chowdhury et al.,
2020; Sosea et al., 2021). For example, Imran et al. (2013, 2015) propose to classify crisis-related tweets to obtain useful information for better disas- ter understanding and rescue operations. Caragea et al. (2016) and Nguyen et al. (2017) Convolu- tional Neural Networks (CNNs) for classifying and extracting informative disaster-related tweets. Li et al. (2021) combines self-training with BERT pre- trained language models to boost the performance of classifying disaster tweets when only unlabeled data is available. Alam et al. (2018); Ghosh and De- sarkar (2020); Sirbu et al. (2022); Zou et al. (2023) leverage both labeled and unlabeled data to develop more effective crisis tweet classifiers. However, while these studies are effective in incorporating information from unlabeled data, they do not ac- count for the fact that the trained semi-supervised models can be biased and neglect certain difficult- to-learn classes, which can have severely negative consequences for disaster relief efforts.
To address these issues, we propose simple but effective debiasing method, DeCrisisMB, that utilizes memory bank store generated pseudo- labels. We then use this memory bank to sample an equal number pseudo-labels from each class per training iteration for debiasing semi-supervised learning. Extensive experiments are conducted to compare the three debiasing methods for semi- supervised crisis tweet classification. Additionally, we evaluate and compare their generalization abil- ity in outoo--distribution datasets and visualize their performance in debiasing semi-supervised models. Our results and analyses demonstrate the substan- tially superior performance of our debiasing meth- ods.
The contributions of this work are summarized follows:
We provide an analysis which shows that im- balanced pseudo-label quantity and quality can cause bias in semi-supervised learning. We investigate their influence by demonstrat- ing the model improvement after equal sam- pling and removing erroneous pseudo-labels.
2.2 Seni-Supervised Learning and Debiasing
Semi-supervised learning aims to reduce the re- liance labeled data in machine learning mod- cls by utilizing unlabeled data to improve their performance (Lee et al., 2013: Berthelot al., 2019; Xie et al., 2020b; Zhang et al., 2021). Self- training involves using the model's prediction prob- ability as a soft label for unlabeled data (Scud- der, 1965; McLachlan, 1975; Lee et al., 2013; Xie et al., 2020b). Pseudo-labeling is a modification of self-training that reduces confirmation bias by using hard labels and confidence thresholding to select high-quality pseudo-labels (Lee et al., 2013; Zhang et al., 2021). Mcan Teacher (Tarvainen and Valpola, 2017) proposes to use the exponential moving average of model weights for predictions. MixMatch (Berthelot et al., 2019) employs sharp- ening to promote low-entropy prediction on unla- beled data and utilizes MixUp (Zhang et al., 2018) to mix and combine labeled and unlabeled data. MixText (Chen et al., 2020) introduces MixUp to text domains by performing interpolation on hidden representations of texts. Recently, Wang et al. (2022) observe that pseudo-labels generated by semi-supervised models are naturally imbal- anced even their training datasets are balanced.
We study two recent semi-supervised debias- ing methods crisis tweet classification and propose DeCrisisMB, simple but effective debiasing method based on memory bank and equal sampling.
We conduct extensive experiments to compare different debiasing methods and provide out- of-distribution results and analysis of their de- biasing performance. Experimental results demonstrate our superior performance com- pared to other methods.
2 Related Work
2.1 Disaster Tweet Classification
Analyzing social media information shared during natural disasters and crises is crucial for enhanc- ing emergency response operations and mitigat- ing the adverse effects of such events, leading to more resilient and sustainable communitics. In recent years, crisis tweet classification has made

---

Clsr
0355
0.2
6880
0355
0.6
Nuttoo Poouabs
2000
0.5
(80
97
(965
desc
10:0
ACC
380
bersioos
tentiore
Ieral ons
(a) classwise accuracy
(b) pseudo-label quantity
(c) pseudo-label quality
Figure I: Classwise model accuracy (a), pseudo-label quantity and quality (b,c) of the representative semi-supervised method Pseudo-Land Labeling (Lee et al., 2013; Berthelot et al., 2019; Xie et al., 2020b) on Hurricane dataset. Semi- supervised models can be highly biased and ignore some classes. We assume this is because the model generates biased pseudo-labels, and training with these biased pseudo-labels can further exacerbate model bias.
The authors first introduce logit adjustment from long-tail learning (Menon al., 2021) to debias semi-supervised learning. The recent state-of- the- art approach in semi-supervised classification by Wang et al. (2023b) designs self-adaptive thresh- olding strategy to lower confidence thresholds of pseudo-labeling for poorly-learned classes. Both of them are helpful in creating more pseudo-labels for difficult classes and debiasing semi-supervised models, but are the expense of sacrificing pseudo- labels accuracy and causing the training process to be unstable. To this end, we investigate their de- biasing effect on the semi-supervised crisis tweet classification task. Then we propose neat debias- ing method that utilizes memory bank for equal sampling. and analyze its effectiveness compared with other debiasing methods.
Dataset Classes
Statistics
[0] rescue volunteering or L_.donatineefios
Hurricane
Threerrises
infrastrutture unility damage
Toral: 12800
Total: 7120
[2] sympahy_annd __ddndd support
Train: 1280
Train: 712
[3] caution and advice
160
[4] not homanitarian
Validation: 160
Validation: 89
[5] injured or or_deed_ppepppe people
displaced_peopl..ee people_amn evacuntions displaced
per Class
[7] requests urgent neads
1600
890
Table 1: Dataset statistics for Hurricane and ThreeCrises dataset.
wildfires and earthquake disasters that occurred be- tween 2016 and 2019. As shown in Table 1, both datasets include the same 8 crisis-related classes, and the number of their total labels is balanced. Our train, test, and validation sets are split in a ratio of 0.8:0.1:0.1.
3 Analysis of Inter-class Bias
3.2 Inter-Class Biases and Pseudo-Label Imbalances
Even with balanced datasets, semi-supervised learning may exhibit bias towards certain classes while ignoring others. this section, we demon- strate that such bias can be related to inter-class imbalances, including the numbers and accuracies of pscudo-labels generated during training. This motivates debias semi-super learning by balancing pseudo-label, during training.
We conduct a pilot experiment on the crisis datasets to demonstrate the bias in semi-supervised mod- els and their pscudo-labels. Here we use Pscudo- Labeling (Lee et al., 2013; Xie et al., 2020a; Sohn et al., 2020), representative work in semi- supervised learning, for analysis. Figure 1(a) illus- trates the model validation accuracy for different classes during training on the Hurricane dataset. can be observed that the model favors certain classes while other classes' performance worsens as training proceeds. We assume this is because the model generates biased pseudo-labels, and train- ing with these biased pseudo-labels will further increase the model's bias.
3.1 Dataset
In this work, we use two crisis tweet datasets: Hur- ricane and ThreeCrises, both sampled from Hu- mAID (Alam et al., 2021). The Hurricane dataset contains 12.800 human-labeled tweets collected during hurricane disasters that happened between 2016 and 2019. The ThreeCrises dataset consists 7,120 annotated tweets collected during floods,
Consistent with our assumption, Figure 1(b) shows the total number of generated pseudo-labels

---

Preliminary Experiment
Investigation
Correct
Balanced PL
Accuracy
Macro-FI
Datusct: Hurricane
(a) Baseline
No
No
63.1 66.7
(b) Delete Incorrect
Accuracy
Yes
No
73.7 1.5
70.2
(c) Equal Sampling
Number
No
Yes
73.4
70.2
(d) Delete Incorrect- Sampling
Both
Yes
Yes
79.6 0.4
77.4
Datusct: ThreeCrises
(a) Buseline
No
No
6,95
6.34
(b) Delete Incoreet
Accuracy
Yes
No
74.7
71.9 + 0.7
(c) Equal Sampling
Number
No
Yes
68.6
6.22
(d) Delete Incurrect+ Equall Sampling
Both
Yes
Yes
78.3 1.6
74.7 = 1.8
Table Investigation on the influence of pseudo-label accuracy and number Hurricane and ThreeCrises Dataset. In both datasets, we observe that the bias can be mitigated by balancing pseudo-label quality and quantity, and the relative quantity pseudo-labels is equally important as the quality pseudo-labels. All results are averaged over 3 runs.
in each category as the training progresses. We can see that the number of pseudo-labels is highly im- balanced across different categories and becomes increasingly biased toward leading classes. Fig- ure (cc) shows the class-wise accuracy of pseudo- labels. We observe that the accuracy of pseudo- labels also varies between classes. One interesting finding is that some classes with higher pseudo label accuracy but lower pseudo label numbers per- form worse Figure ((a) than classes with lower pseudo label accuracy but higher pseudo label num- bers. This implies that the quantity or diversity of pseudo labels in one class might play an equally crucial role the quality of pscudo labels in the learning process.
use the memory bank to sample an equal number of pseudo-labels from each class for training.
Table 2 shows accuracy and macro-FI results for different settings. Trivally, deleting incorrect pscudo labels boosts model performance. However, intriguing that sampling the same number of pseudo-labels per iteration for training also signifi- cantly increases model performance, although the sampled pseudo-labels are nol necessarily correct with no oracle information provided. This further indicates that the relevant number of pseudo-labels has equal importance with pscudo-label accuracy for training unbiased semi-supervised models. Fi- nally, deleting incorrect pseudo labels and then per- forming equal sampling can further increase model performance and achieve the best result. This moti- vates us to alleviate the bias terms of inter-class pseudo-label numbers and accuracies.
3.3 Effect of Equal-Sampling and Erroneous Label Removal
Another perspective for examining how inter-class pseudo-label numbers and accuracies can impact semi-supervised learning performance is to investi- gate model improvement after equal sampling and deleting incorrect pseudo-labels. To this end, conduct the following experiments:
Debiasing Methods
In this section, we first introduce our baseline Pseudo-Labeling (PSL) (Lee et al., 2013; Xie al., 2020a; Sohn et al., 2020), representative semi- supervised method. We then present and discuss two state-of-the-art semi-supervised debiasing ap- proaches: Debiasing Logits Adjustment (Log- itAdjust) in Wang et al. (2022) and Debiasing via Self-Adaptive Thresholding (SAT) in Wang et al. (2023b). Finally, we introduce our debias- ing method DeCrisisMB.
(a) Baseline: Pseudo-Labeling that utilizes high- confidence model predictions unlabeled data as pseudo-labels for iterative training; (b) Delete In- correct: Delete all incorrect pseudo-labels and use only correct ones for training; (c) Equal Sampling: Sample and use an equal number of pseudo-labels from each class for each training iteration. Note guaranteed to be cor-
that the pseudo-labels rect. We implement this through equal sampling in memory bank, which described in detail in the next section; (d) Delete Encorrect+Equal Sam- pling: Remove incorrect pseudo-labed -abels and then
4.1 Pseudo-Labeling
Semi-supervised learning aims reduce the re- liance on labeled data by allowing models to lever- age unlabeled data effectively for training bet-

---

ter models. Suppose we have labeled batch X {0b, yb) b E (1, .I,,,....B). B)} and an unla- beled batch U (up : b (1, (B)). where I is the ratio of unlabeled data to labeled data, B is the batch size of labeled data. The objective of semi-superrised learning often consists of two terms: a supervised loss for labeled data and an un- supervised loss for unlabeled data. The supervised loss L, is computed by cross-entropy between pre- dictions and ground truth labels of labeled data Tb.
where 2b, 2b refers to the logits of the unlabeled data up after and before adjustment, p is the approx- imated average probability distribution on unla- beled data, m E (0. 1) is the momentum parameter of EMA, Ps is the model prediction on an unlabeled sample, X is the debias factor, which controls the strength of the debias effect. The logits adjustment in Eq. 3 alleviate the bias by making false majority classes harder generate pseudo labels, while false minority classes easier to produce pseudo labels.
4.3 Debiasing via Self-Adaptive Thresholding
f- Σ H(y., p(ylm)))
(1)
Another recent method to debias imbalanced pseudo labels is Self-Adatiive Thresholding (SAT) (Wang et al., 2023b). It adaptively adjusts each class's global and local confidence threshold based on the model's overall and class-wise learn- ing status. The model's 's overall learning status is estimated by the EMA of confidence unlabeled data, and the classwise learning status is estimated by the EMA of the probability distribution of un- labeled data, similarly Eq. Formally, at time step L, the self-adaptive global threshold Ty and self- adaptive local threshold (c) for class c are defined as:
6=1
where p denotes the model's probability predic- tion. Pseudo-labeling (PSL) (Lee et al., 2013; Xie et al., 2020a; Sohn et al., 2020) takes advantage of unlabeled data by using the model's prediction of unlabeled data as pseudo-labels to optimize the unsupervised loss:
uB Lu
1 (max((p) > ()6 P6)
(2)
B. b=1
where Ps is the model prediction of unlabeled data us. is the confidence threshold to generate pseudo-labels and 26 is the hard/one-hot pseudo- label of ub, L is L2 loss or cross-entropy loss func- tion. Note that only confident predictions are used to generate pseudo-labels and compute the unsu- pervised loss.
Ti = ===mT-1 (1-m)
Σ
max(p))
(5)
BB b=1
Fe(c)
Te(c) max(p:(c) DE [C]] T
(6)
4.2 Debiasing via Logits Adjustment
where (c(c) is the EMA of probability prediction for class e on all unlabeled data, as in Eqq 4. The insight is that the global threshold т, is low at the beginning of training to utilize more unlabeled data, and grows progressively to eliminate possibly in- correct pseudo-labele confident during the training process. Mcanwhile, the self-adaptive local threshold adjusts classwise local thresholds based on the learning status of each class. This self-adaptive thresholding strat- egy helps debias the semi-supervised model by enforcing the model to create more pseudo-labed for -behavb classes, but is at the expense of lowering their pseudo-label quality, as shown in Section
The first debiasing method for semi-supervised learning we investigate here is Debiasing via Logits Adjustment (LogitAdjust) in Wang et al. (2022). It is claimed that the average probability distribu- tion unlabeled data can be utilized to reflect the model and pseudo-label bias: the higher the average probability one class receives, the more pseudo-labels are usually generated in this class. LogitAdjust proposes to debias pseudo-labeling by adjusting logits based on estimated averaged prob- ability distributions. Since computing the average probability distribution of all unlabeled samples at every iteration is very time-consuming. LogitAd- just uses its exponential moving average (EMA) as approximation. These can be formulated as:
the model becomes more
4.4 Proposed Approach: Debiasing via Memory Bank
2b = 2b - Alogp
aB
Motivated by our analysis in Section 3, we pro- pose DeCrisisMB, an equal-sampling strategy via memory bank to debias semi supervised models.
p - mp + (1 m)
Pb

---

Dataset: Hurricane
Methods
Accuracy
Macro-FT F1
Accuracy
Macro- F1
Accuracy
Macro-F1
Accuracy
Macro F1
Accuracy
Macro F1
al alhelloClass
10
20
50
PSL
39.3 1.6
1.
6.80
49.8
66.7 +47
63.1
73.2
69.6 1.4
0.2
76.0 0.5
MixMMtcch
47.4
4..
17
12
67.6226
64.3
14.1
1.
78.5 0.6
75.4
FlexMarch
7.0
7.9
53.9
3..
70.5
68.2
1.
70.3
78
7..0
LdgitAdjust
4.8
1.4
58.0
53.5
6%.6
6.71
13
7.
0.
2.4
SAT
4.9
4..5
60.0+
5.2
71.4+
68.5+ 1.0
74.7
71,3+04
7.44
7..0
DeCrisisMB
5.8
3.5
65.6772
1..9779
73.4
70.2
77.0 1.6
74.1
7
7.
Datnset: ThreeCrises
#Labela/Class
3
5
10
20
50
PSL
4..
0.7
50.0
13
64.5
44.9
60.4
71.9
6.2
2.
2.0
MixMatth
0.7
04
43.5
483
64.9
6..1
726
6.1
06.
73.5
FlexMarch
4.9
14
5388555
49.4
64.5
61.3#53
7188009
6.4
7..2
1.22
Logit Adjust
3..6+11
47.2+30
61.1+36
72.0
6..7+12
77.3
SAT
4.75
3.
55.8444
50.445.1
68
65.0
12
68.4222
76
7..4
DeCrisisMB
5.55
2.5
4.08
56.0
6.96
64.2
738
70.9
77.0
Table 3: Accuracy and Macro-Fl results of different debiasing methods on Hurricane and ThreeCrises datasets. All results are averaged over runs. Best results are shown in bold.
Memory Bank
quality. In such manner, the bias from the dif- ferent numbers of pseudo-labels produced in each class can be significantly alleviated removed.
Update
Sample
Note that our method very different from the standard samer-g approach: For different training iterations, the generated pseudo-labels can be extremely skewed. In many cases/iterations, there are no pseudo-labels generated for some classes, especially hard classes. The standard under-sampling approach cannot promote learn- ing for these ignored classes during these iterations and will lead the model to increasingly ignore them; The standard wver-sampling approach also cannot handle these cases because there are no pseudo- labels to be oversampled. In contrast, our method stores previously generated high-quality pseudo- labels for each class in memory bank and then we can sample equal numbers of pseudo-labels in each class per training iterations. Results and analysis show that this simple approach very powerful in debiasing since we effectively balance pseudo- labels each training iteration while maintaining the high quality of pseudo-labels.
Unlabaled data
Model
Prediction
Peeudo-label
Figure 2: Illustration of debiasing via Memory Bank.
As illustrated in Figure 2, the memory bank con- sists of C' independent queues, where C is the number of classes. For batch of unlabeled data, the data that receive high-confidence model pre- diction will be assigned pseudo-labels. Since each class may receive a highly imbalanced number of pseudo-labels, we first push these pseudo-labeled data to the corresponding queue in the memory bank. At each training iteration, we randomly sam- ple equal number of pseudo-labeled data from the memory bank for each class. Those rebalanced samples and pseudo-labels are then passed to the model and used for training. We repeat these steps until the model converges. The key using the memory bank to store and rebalance pscudo-labels for each training iteration without sacrificing their
Experiments
5.1 Experimental Setup
Following Chen et al. (2020); Li et al. (2021); Chen al. (2022), use the BERT based-uncased model
backbone model and the Hugging- Face Transformers (Wolf et al., 2020) library for the implementation. We provide a complete list of our parameerrssssess in Appendix A. Our code is released.

---

Source: ThreeCrises, Target: Hurricane
Methods
Accuracy
Macro-F1 3
Accuracy
Macro-Fl
Accuracy
Macro-F1
Accuracy
Macro-F
#Labels/Class
10
20
PSL
37.3
2.5
45.8 1.3
40.6 1.4
61.6 1.664.4
56.9
69.7
65.9 + 2.5
LogitAdjust
42.7 6.2
35,9 6.4
49.7 1.5
44.3 1.3
59.8 1.8
0.5
65.4 0.5
SAT
0.8
3.2
51.1
6.87
66.0
6.59
66.3 69.7
DeCrisisMB
3.1
45.2
58.2 6.2
53,9
68.3 = 1.0
64.6
72.7 1.3
70.0 1.4
Source: Hurricane, Target: ThreeCrises
#Labels/Class
10
PSL
3.5
33.4
63.4 6,9
58.6
68.5
63.6
LogitAdjust
4.5
3..
33.
4.6
5..1 6..
68.3
64.2
SAT
3.2
2.0
572
2.84
67.1 3.6
62.9
66.4 1.7
DeCrisisMB
52.0
2.8
64.9 # 2.3
61.8
67.6
64.3 2 1.7
71.3
67.6
Table 4: Out-of-distribution results. Average over 3 runs.
Dataset
AG News
Yahoo! Answers
Hurricane
ThreeCrises
Average
Methods
Accuracy
Macro-FI
Accuracy
Macro-FI
Accuracy
Macro-F1
Accuracy
Macro-F1
PSI.
67.96
66.04
49.34
42,43
55.00)
49.80
50.00
37.40
52.25
AdgitAdjust
75.54
74.40
52.10
43.93
58.00)
53.50
52.80)
37.60
55.98
SAT
77.13
75,94
56.43
51.03
60.00
56.00
55.80
37.80
58.77
DeCrisisMB
80.55
79.50
59.25
54.09
65.60
62.50
59.80
48.60
63.74
Table 5: Generalizability results on different domains and diverse datasets in the 5-shot setting.
5.2 Main Results
5.3 Out-of-Distribution Results
In Table 4, among all the competitors, our DeCri- sisMB model achieves the best out-of-distribution performance in all evaluation settings, particularly exceeding the second-best SAT approach by the accuracy of 10.45% and Macro-Fl of 11.6% on average when the number of labels is limited to 3. All the above results prove the effectiveness of the DeCrisisMB under distribution shift and also further reveal its potential to be deployed into more realistic and challenging application scenarios.
The comprehensive evaluation results on Hurricane and ThreeCrises are reported in Table 3. In addition the debiasing methods discussed in Section 4, we also compare our method with two other prior and competitive approaches semi-supervised learning and pscudo-label debiasing: MixMatch (Berthelot et al., 2019) and Flex (Zhang et al., 2021). We can see that DeCrisisMB significantly outperforms these approaches in most cases, indi- cating its effectiveness in leveraging unlabeled data and debiasing pseudo-labels. When there are 50 labels for each class, all the baseline models and the DeCrisisMB produce very close results. With the decreased number of labels, it is noteworthy that the DeCrisisMB achieves the best performance by yielding a larger accuracy margin in compari- son with the other three baseline methods. Under the extremely reakly-superviiee setting (with 3 labels for each class), the debiasing process of De- CrisisMB is surprisingly efficient, bringing 13.7% and 10.8% Macro-Fl improvement than the second- best method SAT. All these results imply the strong debiasing capability of the proposed DeCrisisMB and justify its superiority.
5.4 Generalizability Results
To demonstrate the generalizability of our method, we further test our method on two standard semi- supervised learning benchmark datasets: AG News and Yahoo! Answers, both of which are in differ- ent domains and provide larger test and validation sets. A detailed breakdown and statistics of these additional datasets provided in Appendix B. The performance comparisons on the 5-shot setting are presented in Table 5. It can be observed that DeCrisisMB consistently outperforms other meth- ods across different domains and diverse datasets, demonstrating its strong generalizability.

---

064
02
Berctoor
Banutorn
Ianaiiora
(a) PSL
(b) LogitAdjust
(c) SAT
(d) DeCrisisMB
Figure 3: Comparison of classwise accuracy on the validation set of Hurricane dataset between different debiasing methods. All three debiasing methods demonstrate some debiasing effects, i.e., improving the performance of ignored classes in PSL.. However, the training of Logit becomes unstable since its explicit logit adjustment makes it difficult for the model to fit the training data. DeCrisisMB achieves the best debiasing effect and training is also more stable. detailed analysis is provided Section 6.
5000
2006
pass
2553
500
2000
2043
3000
1991
Tumber
3900
200
200
Baratons
Teratora
(a) PSL
(b) LogitAdjust
(c) SAT
(d) DeCrisisMB
Figure 4: Comparison of total numbers of generated pseudo-labels for all classes between different debiasing methods. All three methods have shown some effect in balancing pseudo-labele Nevertheless, the generated pseudo-labels from LogitAdjust pseudo-label numbers but is at the cost of sacrificing their accuracy. DeCrisisMB most effectively balances both the quantity (Figure 4(d)) and quality (Figure 5) of pseudo-labels A detailed analysis is provided in Section 6.
still highly imbalanced. SAT shows great performance in balancing classwise
PS
of produced pseudo-labels. Figure 5 presents the average pseudo-label accuracy over the worst 1, 4 and classes. Overall, all three debiasing meth- ods have some effects in debiasing and balancing pseudo-lahel numbers across the class, as shown in Figure 3, However, there are some differences:
LogiAciust
LogitAdjust can be observed to be unstable over the training process (Figure 3(b) and its generated pscudo-labels are still highly im- balanced (Figure 4(b)). One potential reason behind this is that explicit logit adjustments make the model difficult to fit data and make training unstable.
Number Warst Classes
Figure 5: Pseudo-label quality of worst classes, demon- strated by the average pseudo-label accuracy of worst 1, 4 and 8 classes. DeCrisisMB significantly improve the performance of the worst classes.
SAT does improve the performance of hard- to-learn classes but is still slightly unstable in the training process (Figure 3(c)) and has lower lasuds-labels accuracy than the PSL baseline (Figure 5). Our assumption is that although lowering thresholds helps generate more pseudo-labels for the poorly-learned classes (Figure 4(c)), this comes the ex- pense reducing their pseudo-labed accu- racy (Figure 5).
6 Analysis
To further understand each debiasing approach and why the proposed method achieves the best debas- ing results, provide several visualizations on Hurricane datasets. Figure demonstrates quali- tative comparisons of different debiasing methods. Figure 4 shows the comparison on total numbers

---

Our proposed DeCrisisMB achieves the best results and is powerful in debiasing the semi- supervised models (Figure 3(d) since we ef- fectively balance pseudo-labels in each train- ing iteration (Figure 4(d)) while maintaining the high quality of pseudo-labels (Figure 5), and thus the training is also more stable.
witness information offered in real-time by those who are able to make phone calls. Our model can be integrated into systems that can help response organizations to have a real-time situational aware- ness. In time, such systems could pinpoint the joy of having survived a falling tree, the horror of a bridge washing out or the fear of looters in action. Responders might be able to use such system to provide real-time alerts of the situation on the ground and the status of the affected population. Thus, our research is aimed at having positive impact on sustainable citics and communitics.
7 Conclusion
this work, we demonstrate that semi-supervised models and their pscudo-labels generated on so- cial media data posted during crisis events can be biased, and balancing pscudo-labels used in train- ing can effectively debias semi-supervised mod- els. We then study and compare two recent debi- asing approaches in semi-supervised learning with our proposed debiasing method for crisis tweet classification, Experimental results show that our method based on memory bank and equal sampling achieves the best debiasing results quantitatively on both in-distribution and dittr--diitributinn set- tings. We believe our work can serve as a universal and effective adds-on debiasing module for semi- supervised learning in different domains.
A Acknowledgements
This research is partially supported by National Science Foundation (NSF) grants 191-191, IIS- 2 2107487, ITE-2137846. Any opinions, findings, and conclusions expressed here are those of the authors and do not necessarily reflect the views of NSF. We thank our reviewers for their insightful feedback and comments.
References
Firoj Alam, Shafiq Joty, and Muhammad Imran. 2018. Graph based semi-supervised learning with convolu- tion neural networks to classify crisis related tweets. In Proceedings of the international AAAI conference on web and social media, volume 12.
Limitations
This work examines various debiasing methods, primarily in the context of classification settings. However, it should be worthwhile to investigate the debiasing effects of these methods in other settings, such as in generative tasks and large language mod- els. Such exploration would help further demon- strate the generality of these methods. We plan to conduct such exploration in the future.
Firoj Alam, Umair Qazi, Muhammad Imran, and Ferda Olli, 2021. Humaid: human-annotated disaster in- cidents data from twitter with deep learning bench- marks. Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 933-942.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raf- fel. 2019. Mixmatch: A holistic approach to semi- supervised learning. Advances in neural information processing systems, 32.
Broader Impact
For our crisis domain, the strongest contribution of this paper is the debiasing strategy that helps alleviate the negative effect of models being biased towards the more frequent classes. Using Twitter data, we reliably improve the performance of life- essential classes such as requests or urgent needs, displaced people and evacuations, and injured or dead people. Currently, crisis responders can track weather data to know where hurricane hits an affected population what are potentially flooded areas in rainy seasons, but they cannot know in real time the effect that a disaster is having on the population. They often ask, "How bad is it out there?". Traditionally, they rely on either cyewit- ness accounts after the fact from survivors, or eye-
Coreelaa Caragea, Adrian Silvescu, and Andrea H Tapia. 2016. Identifying informative messages disasters using convolutional neural networks. In Proceedings of the 13th International Conference on Information Systems for Crisis Response and Management.
Ming-Wes Wei Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representa- tion: Dataless classification. In Proceedings of the 23rd National Conference on Artificial Intelligence Volume 2, AAAI'0,, page 830-835. AAAI Press.
Hui Chen, Wei Han, and Soujanya Poria. 2022, SAT: Improving sumi-supervised text classification with simple instance- adaptive self self-training. In Findings of the Association for Computational Linguistics:

---

EMNLP 2022, pages 6141-6146, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Reza Mazloom, HongMin Li, Doina Caragea, Muham- mad Imran, and Cornelia Caragea. 2018. Classifica- tion of twitter disaster data using a hybrid feature- instance adaptation approach. In Proceedings of the 15th International Conference on Information Sys- temx for Crisis Response and Management, Rochexter, NY, USA, May 20-23, 2018. ISCRAM Association.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix- interpolation of hid-
Text: Linguistically-inforee den space for sumi-super text classification. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 2147- 2157, Online. Association for Computational Lin- guistics.
Geoffrey J McLachlan. 1975. Iterative reclassification procedure for constructing asymptotically optimal rule of allocation in discriminant analysis. Journal of the American Statistical Association, 706350):365- 369.
Samujjwal Ghosh and Maunendra Sankar Desarkar. 2020. Semi-supervised granular classification frame- work for resource constrained short-texts: Towards re- trieving situational information during disaster events. In 12th ACM Confereenee on Web Science, pages 29- 38.
Aditya Krishma Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. 2021. Long-tail learning via logit adjustment. In International Conference on Learning Representations.
Muhammad Imran, Carlos Castillo, Fernando Diaz, and Sarah Vieweg. 2015. Processing social media mes- sages in mass emergency; A survey. ACM Comput- ing Surveys (CSUR), --348
Venkata Kishore Neppalli, Coreelia Caragea, and Doina Caragea. 2018. Deep neural networks versus naive bayes classifiers for identifying informative tweets during disasters. In Proceedings of the 15th Inter- national Conference Information Systems for Cri- sis Rexponse and Management, Rochester, NY, USA, May 20-23, 2018, ISCRAM Association.
Muhammad Imran, Shady Elbassuoni, Carlos Castillo, Fernando Diaz. and Patrick Meier. 2013. Practical extraction of disaster-relevant information from so- cial media. In Proceedings of the 22nd international conference on world wide web. pages 1021-1024.
Dat Tien Nguyen, Kamela Ali AI Mannai, Shafiq Joty. Hassan Sajjad, Muhammad Imran, and Prasenjit Mi- tra. 2017. Robust classification of crisis-related data on social networks using convolutional neural net- works. In Eleventh international AAAI conference web and social media.
Dong-Hyun Lee et al. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 896.
Hongmin Li, Doina Caragea, and Cornelia Caragea. 2017. Towards practical usage of a domain adapta- tion algorithm in the early hours of a disaster. In 14th Proceedings of the International Conference on Information Systems for Crisis Response and Man- agement, Albi, France, May 21-24, 2017. ISCRAM Association.
Jishnu Ray Chowdhury, Cornelia Caragea, and Doina Caragea. 2020. Cross-lingual disaster-related multi- label tweet classification with manifold mixup. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguisties: Student Re- search Workshop, pages 292-298, Online. Associa- tion for Computational Linguistics.
Hongmin Li, Doina Caragea, and Cornelia Caragea. 2021. Combining self-training with deep learning for disaster tweet classification. In 18th International Conference on Information Systems for Crisis Re- sponse and Management, ISCRAM 2021, Blacksburg. VA, USA, May 2021, pages 719-730. ISCRAM Digi- tal Library.
Henry Scudder. 1965. Probability of error of some adaptive nnttern-recognitionn machines. IEEE Trans- actions on Information Theory, 11(3):363-371.
Justin Sirbu, Tiberiu Sosea, Comella Caragea, Doina Caragea, and Traian Rebedea. 2022. Multimodal semi-supervised learning for disaster tweet classifica- tion. In Proceedings of the 29th International Con- ference on Computational Linguistics, pages 2711- 2723, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Hongmin Li, Doina Caragea, Cornelaa Caragea, and Nic Herndon. 2018. Disaster response aided by tweet classification with a domain adaptation approach. Journal of Contingencies and Crisis Management, 26(1):16-27.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608.
Hongmin Li. Nicolais Guevara, Nic Herndon, Doina Caragea, Kishore Neppalli, Cornelia Caragea, Anna Cinzia Squicciarini, and Andrea H. Tapia. 2015, Twitter mining for disaster response: A domain adap- tation approach. In 12th Proceedings of the Interna- tional Conference on Information Systems for Crisis Respouse and Munagement, Krystiansand, Norvay, May 24-27, 2015. ISCRAM Association.
Tiberiu Sosea, lustin Sirbu, Cornelia Caragea, Doina Caragea, and Traian Rebedea. 2021. Using the

---

image-text relationship to improve multimodal disas- ter tweet classification. In 18th International Confer- ence on Information Systems for Crisis Response and Management, ISCRAM 2021, Blacksburg, VA, USA, May 2021, pages 691-704. ISCRAM Digital Library.
Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoe Le. 2020b. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10687-10698.
Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve seni-supervised deep learning re- sults. Advances in neural information processing systems, 30.
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang. Manabu Okumura, and Takahiro Shi- nozaki. 2021. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems, 34:1 18488- 18419.
István Varga, Motoki Sano, Kentaro Torisawa, Chikara Hashimoto, Kiyonori Ohtake, Takao Kawai, Jong- Hoon Oh, and Stijn De Saeger. 2013. Aid is out there: Looking for help from tweets during large scale disaster. In Proceedings of the S1st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 1619-1629, Sofia, Bulgaria. Association for Computational Lin- guistics.
Hongyi Zhang. Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
Xiang Zhang. Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances neural information processing systems, 28.
Henry Peng Zou, Yue Zhou, Comella Caragea, and Doina Caragea. 2023. Crisismatch: Supe-supe few-shot learning for fine-grained disaster tweet clas- sification. In Proceedings of the 20th International ISCRAM Conference, pages 385-395.
Sarah Vieweg, Carlos Castillo, and Muhammad Imran. 2014. Integrating social media communications into the rapid assessment of sudden onset disasters. In In- ternational Conference on Social Informatics, pages 444-661. Springer.
A Hyperparameters
Fangxin Wang. Lu Cheng, Ruocheng Guo, Kay Liu, and Philip Yu. 2023a. Equal opportunity of coverage in fair regression. In Advances in Neural Information Processing Systems.
A complete list of hyperparameters of evaluated methods is provided in Table 6. We use the same hyperparameters and Adam optimizer across all datasets.
Xudong Wang, Zhirong Wu, Long Lian, and Stella X Yu. 2022. Debiased learning from naturally imbalanced pseudo-labels. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 14647-14657.
PSL LogitAdjust
DeCrisisMB
Learning Rate
le4
Batch Size
32
Unsuperivaed Loss Weight
20
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang. Marios Savvides, Takahiro Shinozaki, Bhiksha Raj. Bernt Schiele, and Xing Xie. 2023b. Freematch: Self-adaptive thresh- olding for semi-supervised learning. In The Eleventh International Conference on Learning Representa- tions.
EMA Momentum
0.9
Confidence Threshold
0.9
Unlabeled Data Ratio
Length of Queue in DeCrisisMB
Equal Sampling Number Debias Strength À
0.4
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of oftte-of-the-arr natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Table 6: A complete list of hyperparameters of all eval- uated methods in this study.
B Statistics of Added Datasets
The detailed breakdown and statistics of the addi- tional datasets, AG News (Zhang al., 2015) and Yahoo! Answers (Chang al., 2008), are provided in Table 7.
C Further Augment DeCrisisMB
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoe Lc. 2020a. Unsupervised data augmen- tation for consistency training. Advances neural information processing systems, 33:6256-6268.
As inspired by the two previous debiasing methods discussed in Section 4, different classes have var- ied learning statuses; thus, it might be beneficial

---

Dataset
Total
# Training
# Validation
# Test
Domain
# Classes
Class Example
AG News
35,600
20,000
8,000
7,600
News Topic
4
Sei/Tech, World, Business
Yahoo! Answers
130,000
50,000
20,000
60,000
QA Topic
Sports, Health, Education
Table 7: Detailed breakdown and statistics of added datasets.
to sample more pscudo-labels for poorly- loarly-lcarned classes and sample fewer pseudo-labels for leading classes. To this end, we propose an adaptive sam- pling strategy (AdSampling) to explore way of augmenting our DeCrisisMB method. Specifically, the number of pseudo-labels to be sampled from different classes depends on their learning status, which is estimated by (c(c) as in Eq. 6 and Eq. 4. Formally, the sampling number for class at time I is defined
et Labeled Data Per Class
Accuracy
20
DeCrisisMBB
58.
65.6
73.4 1.0
77.0 1.6
78.9 + 1.0
AdSampling
Macro-FI F1
3
20
50
DeCrisisMB
54
62
102212
74.1 1.5
75.7 = 1.3
AdSampling
Table 8: Accuracy and Macro-FI results of DeCrisisMB with and without adaptive sampling. Average over 3 runs.
Ni(c)
(c(c) * N
(7)
where C' is the number of classes, and N is the original sampling number for each class queue in the memory bank. This helps in over-balancing pseudo-labels used per iteration for poorly- behaved classes and speeding up the debiasing process.
Table 8 indicates the accuracy and Macro-FI results of DeCrisisMB with and without AdSam- pling on the Hurricane dataset. AdSampling, the sampling strategy to prioritize the poorly-learned classes, further boosts the DeCrisisMB perfor- mance and achieves better debiasing results in most cases. Note that adaptive sampling is just simple add-on and exploration inspired by the two baseline methods and can be further optimized.


Reviews:
[{"review_id": "YOT6Imm6KD", "reviewer": "Reviewer_MKfM", "summary": "", "questions": "Please provide a proper breakdown for each dataset, class, and tweet counts", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your valuable comments and assessment of our paper! \n\n> W1: Small test and validation sets, probably not generalizable.\n\nWe have **added experiments** on **two standard** semi-supervised learning **benchmarking datasets**: AG News and Yahoo! Answers, both of which provide **larger test and validation sets**. The statistics of all our datasets are provided in the first table below. The result comparisons on the 5-shot setting are presented in the second table below. Our **MemoryBank** method also **works well on these larger datasets**, demonstrating its generalizability.\n\n> W2: The method should work in other domains, this is not tested\n\nThank you for this constructive suggestion. We have conducted additional experiments and **tested our method on different domains and four diverse datasets**. The dataset statistics and performance comparisons are shown in the table below. The results demonstrate that MemoryBank consistently outperforms other methods across different domains. A more comprehensive comparison will be included in our paper. \n\n> W3: Please provide a proper breakdown for each dataset, class, and tweet counts\n\nThanks for giving us the opportunity to clarify this. **A detailed breakdown and statistics of each dataset are provided** in the following table. Furthermore, we provide a short description and citation of these datasets below. \n\n\n| Dataset        | # Total Data | # Training | # Validation | # Test | Domain       | # Classes | Class Example                                          |\n|----------------|--------------|------------|--------------|--------|--------------|-----------|--------------------------------------------------------|\n| AG News        | 35,600       | 20,000     | 8,000        | 7,600  | News Topic   | 4         | Sci/Tech, World, Business                              |\n| Yahoo! Answers | 130,000      | 50,000     | 20,000       | 60,000 | QA Topic     | 10        | Sports, Health, Education                              |\n| Hurricane      | 12,800       | 10,240     | 1,280        | 1,280  | Crisis Tweet | 8         | Infrastructure and Utility Damage, Causion and Advice  |\n| ThreeCrises    | 7,120        | 5,696      | 712          | 712    | Crisis Tweet | 8         | Displaced People and  Evacuation, Sympathy and Support |\n\n\n| Dataset     | AG News  |          | Yahoo! Answers |          | Hurricane |          | ThreeCrises |          | Average |\n|-------------|----------|----------|----------------|----------|-----------|----------|-------------|----------|---------|\n| Methods     | Accuracy | Macro-F1 | Accuracy       | Macro-F1 | Accuracy  | Macro-F1 | Accuracy    | Macro-F1 | -       |\n| PSL         | 67.96    | 66.04    | 49.34          | 42.43    | 55.00     | 49.80    | 50.00       | 37.40    | 52.25   |\n| LogitAdjust | 75.54    | 74.40    | 52.10          | 43.93    | 58.00     | 53.50    | 52.80       | 37.60    | 55.98   |\n| SAT         | 77.13    | 75.94    | 56.43          | 51.03    | 60.00     | 56.00    | 55.80       | 37.80    | 58.77   |\n| MemoryBank  | **80.55**    | **79.50**    | **59.25**          | **54.09**    | **65.60**     | **62.50**    | **59.80**       | **48.60**    | **63.74**   |\n\n\nIf you are interested in any other specific methods, datasets or experiments, please let us know, and we will be happy to conduct experiments immediately and include them in our paper. Thanks again for your very constructive feedback and suggestions.\n\n---\n**Short Descriptions of Datasets:** [1] Hurricane and ThreeCrises are two crisis tweet datasets sampled from HumAID (Alam et al., 2021). It contains human-labeled tweets collected during hurricane disasters and includes 8 crisis-related classes, such as infrastructure_and_utility_damage, displaced_people_and_evacuations. [2] Yahoo! Answers classifies question-answer pairs into 10 topics, such as Sports, Health, and Education. [3] AG News categorizes news articles into 4 categories, such as World, Sci/Tech, and Business. We will provide a more comprehensive description of datasets, statistics, and split information in our paper.\n\n[1] Firoj Alam, Umair Qazi, Muhammad Imran, and Ferda Ofli. 2021. Humaid: human-annotated disaster incidents data from twitter with deep learning benchmarks. In Proceedings of the International AAAI Conference on Web and Social Media, volume 15, pages 933–942.\n\n[2] Ming-Wei Chang, Lev Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dataless classification. In Proceedings of the 23rd National Conference on Artificial Intelligence - Volume 2, AAAI’08, page 830–835. AAAI Press.\n\n[3] Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Apparently solid debiasing method\n- Interesting domain the method is applied to", "reasons_to_reject": "- Small test and validation sets, probably not generalizable\n- The method should work in other domains, this is not tested", "questions_to_authors": "Please provide a proper breakdown for each dataset, class, and tweet counts", "paper_topic_and_main_contributions": "In this work the authors introduce a debiasing method for semi-supervised learning of tweets topics, with a focus on crisis tweets. The contribution is very well motivated and the method developed seems promising. Unfortunately, the test and validation sets are very small to clearly demonstrate any generalizability of the method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dOFdGnoYly", "reviewer": "Reviewer_Kir9", "summary": "", "questions": "Could the authors provide justification for their choice to compare their algorithm with Logit adjustment and the SAT algorithm, while excluding other previous approaches in their analysis?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your valuable comments and assessment of our paper! \n\n>  W1: Could the authors provide justification for their choice to compare their algorithm with Logit adjustment and the SAT algorithm, while excluding other previous approaches in their analysis?\n\nYes! We compared our work with SAT and LogitAdjust because their corresponding works were recently published (ICLR 2023, CVPR 2022) and they are the main modules that focus on debiasing pseudo-labels in two recent state-of-the-art semi-supervised approaches, namely FreeMatch and DebiasPL. They are very competitive baselines and have been shown to outperform previous debiased semi-supervised approaches. That’s the main reason we compared with them and didn’t include earlier works. We are sorry that we don’t have enough time and computation resources to add comparisons with broader previous approaches due to the limited rebuttal time. However, if you would like to see comparisons with more previous approaches, we will add them in our final paper, and we are currently conducting experiments on them. \n\n> W2: Missing References: Wang et al, Long-tailed Recognition by Routing Diverse Distribution-Aware Experts (ICLR'21)\n\nThanks for bringing this work to our attention! We will discuss this work and add its reference in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors explore the realm of semi-supervised learning within the context of language classification tasks. They also illustrate that in cases of imbalanced datasets with only a few labeled samples — essentially, scenarios involving semi-supervised learning and imbalance — the quantity of pseudo-labels can exhibit bias.", "reasons_to_reject": "Although the authors conduct a comparison between the proposed algorithm and previous methods like logit adjustment (designed for imbalanced datasets) and the self-adaptive thresholding method (intended for semi-supervised learning), I believe the comparative analysis lacks strength. To my knowledge, there exists a multitude of approaches for addressing imbalanced data and semi-supervised learning, such as the utilization of multiple experts (as seen in RIDE) and semi-supervised learning methods (like FlexMix, as referenced by the authors). Therefore, in order to gain a more comprehensive understanding of the interplay between class imbalance and semi-supervised learning within the language classification task, a more thorough exploration of these prior approaches is essential.", "questions_to_authors": "Could the authors provide justification for their choice to compare their algorithm with Logit adjustment and the SAT algorithm, while excluding other previous approaches in their analysis?", "paper_topic_and_main_contributions": "This paper presents an analysis demonstrating how an imbalanced pseudo-labeling approach can introduce bias in semi-supervised learning. Additionally, the authors suggest a memory-bank based approach to address this issue. The proposed approach is straightforward: it ensures class balance while generating pseudo-labels. The authors assess the effectiveness of this method through a simple algorithm, comparing its performance against previous techniques such as logit-adjustment and self-adaptive thresholding methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Wang et al, Long-tailed Recognition by Routing Diverse Distribution-Aware Experts  (ICLR'21)", "Typos_Grammar_Style_and_Presentation_Improvements": "-\n"}, "extra_text": {}}, {"review_id": "76UNpKskwv", "reviewer": "Reviewer_LWNG", "summary": "", "questions": "Please refer to W1 and W2 of my detailed comments above.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your valuable comments and assessment of our paper! \n\n> W1: While the proposed MemoryBank method offers good performance, the novelty of the method itself is fairly limited in my opinion. Based on the description in Section 4.4, the MemoryBank method is similar to the fairly standard under-sampling approach typically used to train imbalanced datasets, albeit with some modification to the training loop.\n\nThanks for giving us the chance to clarify our novelty. (1) Our method is **very different from the standard under-sampling approach**: For different training iterations, the generated pseudo-labels can be highly imbalanced. In many cases/iterations, there are no pseudo-labels generated for some classes, especially hard classes. The standard under-sampling approach cannot promote learning for these ignored classes during these iterations and will lead the model to increasingly ignore them; The standard over-sampling approach also cannot handle these cases because there are no pseudo-labels to be oversampled. In contrast, our method stores previously generated high-quality pseudo-labels for each class in a memory bank and then we can sample equal numbers of pseudo-labels in each class per training iterations. Results and analysis show that this simple approach is very powerful in debiasing since we **effectively balance pseudo-labels in *each training iteration*** while maintaining the high quality of pseudo-labels. (2) Moreover, to the best of our knowledge, our work is the **first** that **introduces Memory Bank** to **debias pseudo-labels** and semi-supervised models. In this sense, it is very **novel**. \n\n> W2: Some parts of the paper seem contradictory to each other. For example, the main motivation of the paper is “…current semi-supervised models can be biased towards more frequent classes, performing moderately well for certain classes while extremely worse for others…” but the dataset chosen is one where “both datasets include the same 8 crisis-related classes, and the number of their total labels is balanced”. I would think experimenting on a more imbalanced dataset would better support the claims of the paper.\n\nThanks for pointing this out. We would like to refine our expression as follows: “...current semi-supervised models can be biased, performing moderately well for certain classes while extremely worse for others”. Notice that this is **not contradictory** to our experimental setting of using balanced datasets. **Even with balanced datasets**, semi-supervised models can be **highly biased** towards easy-to-learn classes and **gradually ignore hard classes**, as demonstrated in our Figure 1, 3 and previous works SAT/FreeMatch  (Wang et al., 2023 ICLR), LogitAdjust/DebiasPL (Wang et at., 2022 CVPR). \n\n> W3: For tables 3 and 5, it can be more clearly indicated that the 3, 5, 10, 20 refers to the number of labels for each class.\n\nThank you for helping us improve the table presentation. We have added ‘# labels per class’ in the corresponding rows of these tables.  \n\nHope our response could resolve your concern. Please let us know if you have further questions or concerns and we will try our best to address them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There are a few strong points about this paper that I particularly like, which are:\n\nS1.\tThis paper studies the important problem of crisis detection, more specifically on how semi-supervised debiasing methods can be used for crisis detection from tweets.\n\nS2.\tThe proposed MemoryBank method is a simple approach that also offers good performance over the various baselines evaluated. There is also a comprehensive set of experiments and discussion/analysis of the results.\n\nS3.\tThe paper is overall well-written and a pleasure to read.", "reasons_to_reject": "There are a few weak aspects of the paper that the authors can clarify or improve on, which are:\n\nW1.\tWhile the proposed MemoryBank method offers good performance, the novelty of the method itself is fairly limited in my opinion. Based on the description in Section 4.4, the MemoryBank method is similar to the fairly standard under-sampling approach typically used to train imbalanced datasets, albeit with some modification to the training loop.\n\nW2.\tSome parts of the paper seem contradictory to each other. For example, the main motivation of the paper is “…current semi-supervised models can be biased towards more frequent classes, performing moderately well for certain classes while extremely worse for others…” but the dataset chosen is one where “both datasets include the same 8 crisis-related classes, and the number of their total labels is balanced”. I would think experimenting on a more imbalanced dataset would better support the claims of the paper.\n\nW3.\tThis is only a very minor weak point in my opinion and not a basis for rejection. The context of the study is placed in the domain of crisis detection but the proposed semi-supervised debiasing method, various baselines and even the experiments are for more generic text classification type of tasks and the link to crisis detection is not so strong, apart from the labels being crisis-related.", "questions_to_authors": "Please refer to W1 and W2 of my detailed comments above.", "paper_topic_and_main_contributions": "This paper studies the problem of how imbalanced labels can cause biases in semi-supervised methods and investigates how semi-supervised debiasing methods are used for crisis tweet detection. The paper presents MemoryBank, a semi-supervised debiasing method that performs an equal-sized pseudo-label sampling at each iteration of model training. There is also a large set of experiments to compare the proposed MemoryBank method against three baselines, and various analysis on the performance of MemoryBank.\n\nI wish the authors all the best as they proceed with this work.\n\nPost-rebuttal: I thank the authors for their responses and acknowledge that they have been read. My scores remain unchanged.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "For tables 3 and 5, it can be more clearly indicated that the 3, 5, 10, 20 refers to the number of labels for each class."}, "extra_text": {}}, {"review_id": "fbe44lsiR6", "reviewer": "Reviewer_msjn", "summary": "", "questions": "1. Eqn. 2 does not seem to be correct. \n\n2. Does adaptive sampling contribute anything to the performance of the model? Results in Table 4 depict that most of the benefit comes from memory bank rather than adaptive sampling.\n\n3. In Table 3, the authors report results for ThreeCrises. It is better to report the result for individual event types. Does it perform equally well for all the events?\n\n4. Are the improvements statistically significant? For ThreeCrises, an out-of-domain setup shows marginal improvement for 20 instances per class (Table 5). In that case, the utility might be limited.\n\n5. It is better to give some details about the learning rate, optimizer details, etc.\n\n6. The idea is fine but the novelty is limited.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your valuable comments and assessment of our paper! \n\n> W1: Does adaptive sampling contribute anything to the performance of the model? Results in Table 4 depict that most of the benefit comes from memory bank rather than adaptive sampling.\n\nThanks for giving us the opportunity to clarify this. The original MemoryBank method provides the most performance improvement. The adaptive sampling is just an add-on and exploration inspired by the two baseline methods. It turns out that it only contributes marginally to the original MemoryBank method, + 0.44% accuracy and +0.6% macro-F1 score on average. Note that only Table 4 contains results with adaptive sampling, whereas all other tables (Table 3,5) report the results from the original MemoryBank method. We will make this clearer in our paper. If reviewers deem it necessary, we can remove adaptive sampling results from the formal paper and place them in the appendix instead.\n\n> W2: Are the improvements statistically significant? For ThreeCrises, an out-of-domain setup shows marginal improvement for 20 instances per class (Table 5). In that case, the utility might be limited.\n\nMemoryBank actually delivers very impressive improvements over other methods. (1) The following table summarizes the average accuracy and macro-F1 results comparisons in our out-of-domain setup. We can see that **MemoryBank consistently significantly outperforms other methods on average**. On the extreme 3-shot settings, MemoryBank can achieve +17.1% accuracy improvement on the ThreeCrises dataset. (2) The improvements become **gradually marginal when increasing the number of labels** and enough amounts of labels are given. For example, in the mentioned 20-shot setting, the difference between MemoryBank and SAT is not statistically significant. However, this is a **common** issue/**observation for semi-supervised learning**. This paper aims to emphasize MemoryBank’s impressive results in the low-shot semi-supervised setting.\n\n\n|             | S: ThreeCrises, | T: Hurricane;  | S: Hurricane, | T: ThreeCrises |\n| ----------- | --------------- | ------------- | ------------- | -------------- |\n|             | Avg. Accuracy   | Avg. Macro-F1 | Avg. Accuracy | Avg. Macro-F1  |\n| PSL         | 53.60           | 49.03         | 55.78         | 50.70          |\n| LogitAdjust | 56.18           | 51.35         | 55.75         | 51.05          |\n| SAT         | 57.23           | 52.48         | 57.65         | 52.73          |\n| MemoryBank  | **62.13**           | **58.43**         | **63.95**         | **60.28**          |\n\n\n> W3: The idea is fine but the novelty is limited.\n\nOur idea may seem **simple**, but it is **novel and** also very **effective**. To the best of our knowledge, our work is the **first to introduce MemoryBank to debias pseudo-labels** and semi-supervised models. Additionally, our work is unique in that it investigates different semi-supervised debiasing methods within the context of crisis domains.\n\n> W4: In Table 3, the authors report results for ThreeCrises. It is better to report the result for individual event types. Does it perform equally well for all the events?\n\nWe appreciate this proposal. ThreeCrises is a very small dataset, and if we decompose it into individual events, the resulting individual datasets will be too small for proper evaluation. We expect that our method will perform slightly differently for different events. However, if you are interested in how well our method performs in different domains/datasets, we are happy to address this. The following table presents our results on different domains and four diverse datasets. The results show that MemoryBank delivers different but consistent improvements on other diverse domains and datasets.\n\n| Dataset     | AG News  |          | Yahoo! Answers |          | Hurricane |          | ThreeCrises |          | Average |\n|-------------|----------|----------|----------------|----------|-----------|----------|-------------|----------|---------|\n| Methods     | Accuracy | Macro-F1 | Accuracy       | Macro-F1 | Accuracy  | Macro-F1 | Accuracy    | Macro-F1 | -       |\n| PSL         | 67.96    | 66.04    | 49.34          | 42.43    | 55.00     | 49.80    | 50.00       | 37.40    | 52.25   |\n| LogitAdjust | 75.54    | 74.40    | 52.10          | 43.93    | 58.00     | 53.50    | 52.80       | 37.60    | 55.98   |\n| SAT         | 77.13    | 75.94    | 56.43          | 51.03    | 60.00     | 56.00    | 55.80       | 37.80    | 58.77   |\n| MemoryBank  | **80.55**    | **79.50**    | **59.25**          | **54.09**    | **65.60**     | **62.50**    | **59.80**       | **48.60**    | **63.74**   |\n\n\n> W5: It is better to give some details about the learning rate, optimizer details, etc.\n\nThank you for this suggestion. We have provided a complete list of hyperparameters (including learning rate) of evaluated methods in Appendix A. We use AdamW from the huggingface library as our optimizer. A more comprehensive description of our experiment details, including optimizer details, will be provided in our paper. Our code is also released.\n\n> W6: Eqn. 2 does not seem to be correct.\n\nThanks for pointing this out. We will fix it and remove the two redundant brackets in Eqn.2 of our paper.\n\nWe sincerely hope our response can resolve your concerns. But if you have any further concerns or questions, please let us know, and we will do our best to address them as soon as we can.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Debasing is important in the current scenario to ensure the fairness of classifiers across different classes.\n\n2. The authors sampled an equal number of tweets from each class and this helps in removing the bias of semi-supervised classifier.", "reasons_to_reject": "1. Eqn. 2 does not seem to be correct. \n\n2. Does adaptive sampling contribute anything to the performance of the model? Results in Table 4 depict that most of the benefit comes from memory bank rather than adaptive sampling.\n\n3. In Table 3, the authors report results for ThreeCrises. It is better to report the result for individual event types. Does it perform equally well for all the events?\n\n4. Are the improvements statistically significant? For ThreeCrises, an out-of-domain setup shows marginal improvement for 20 instances per class (Table 5). In that case, the utility might be limited.\n\n5. It is better to give some details about the learning rate, optimizer details, etc.\n\n6. The idea is fine but the novelty is limited.", "questions_to_authors": "1. Eqn. 2 does not seem to be correct. \n\n2. Does adaptive sampling contribute anything to the performance of the model? Results in Table 4 depict that most of the benefit comes from memory bank rather than adaptive sampling.\n\n3. In Table 3, the authors report results for ThreeCrises. It is better to report the result for individual event types. Does it perform equally well for all the events?\n\n4. Are the improvements statistically significant? For ThreeCrises, an out-of-domain setup shows marginal improvement for 20 instances per class (Table 5). In that case, the utility might be limited.\n\n5. It is better to give some details about the learning rate, optimizer details, etc.\n\n6. The idea is fine but the novelty is limited.", "paper_topic_and_main_contributions": "In this paper, authors analyzed the performance of some debasing strategies on the performance of semi-supervised crisis tweet classifier. Further, they proposed memory banking and adaptive sampling based debasing strategies to overcome the limitations of existing debasing methods I.e., logits adjustment and self-adaptive thresholding. Through memory banking, authors ensured to take the same number of new training instances from each crisis class. Memory bank helps to ensure an equal number of samples from different classes in different iterations. The authors tested their method over Hurricane and three crisis events. The memory bank based debasing approach works well when the number of labeled data per class is less in number.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an approach to address biases in semi-supervised learning within the context of crisis detection from tweets. While the paper has interesting ideas, there are also areas of concern that warrant attention before it can be considered for publication.
Reasons to Accept:
* 		Solid Debiasing Method: The paper introduces a debiasing method, MemoryBank, which offers competitive performance in comparison to baselines. The simplicity of this approach is a merit, as it presents a viable solution for addressing biases in semi-supervised learning scenarios.
* 		Interesting Domain Application: The choice of crisis detection from tweets as the application domain is noteworthy. This context offers a unique and compelling real-world problem, which adds value to the paper's contributions.
* 		Exploration of Semi-Supervised Learning in Imbalanced Data: The paper discusses semi-supervised learning with imbalanced datasets, shedding light on how the quantity of pseudo-labels can introduce bias. This exploration is interesting, given the practical challenges of handling imbalanced data in real-world applications.
* 		Comprehensive Experimental Evaluation: The paper includes a set of experiments that contributes to the understanding of the proposed method's effectiveness.
Reasons to Reject:
* 		Generalizability Concerns: The paper raises concerns about generalizability due to the use of small test and validation sets. It is crucial to demonstrate that the proposed method is effective beyond the specific dataset and domain tested.
* 		Lack of Cross-Domain Testing: While the method shows promise within the crisis detection domain, its applicability to other domains remains unexplored. A broader assessment of its effectiveness in diverse domains would strengthen the paper's contributions.
* 		Comparative Analysis Needs Strengthening: The comparative analysis between the proposed MemoryBank method and existing approaches appears to lack depth. A more thorough exploration of prior approaches for addressing imbalanced data and semi-supervised learning is essential to provide a comprehensive understanding of the proposed method's advantages and limitations.
* 		Needs Deeper Discussion of the Method: The MemoryBank method has lots of aspects very similar to standard under-sampling techniques with modifications to the training loop. The paper should clearly establish how MemoryBank significantly advances the state of the art in addressing bias in semi-supervised learning.
* 		Contradictions in Dataset Choice: The paper's choice of a balanced dataset appears contradictory to its main motivation, which is to address biases in semi-supervised models when classes are imbalanced. Using a more imbalanced dataset would better support the paper's claims and motivations. More discussions are needed on this aspect.
In conclusion, while the paper presents an interesting approach to address biases in semi-supervised learning within the crisis detection domain, there are concerns related to generalizability, comparative analysis, and the novelty of the approach that need to be addressed. All those aspects were discussed during the rebuttal phase and there was convergence on the discussions. I encourage the authors to carefully revise the paper considering the aspects discussed with the reviewers.
Decision: Accept-Findings