VIBE: Topic-Driven Temporal Adaptation for Twitter Classification
Authors: Yuji Zhang
Jing Li
Wenjie Li
Venue: EMNLP 2023 Main (2023)
Keywords: temporal adaptation; neural topic model; social media; twitter classification

Abstract: Language features are evolving in real-world social media, resulting in the deteriorating performance of text classification in dynamics. To address this challenge, we study temporal adaptation, where models trained on past data are tested in the future. Most prior work focused on continued pretraining or knowledge updating, which may compromise their performance on noisy social media data. To tackle this issue, we reflect feature change via modeling latent topic evolution and propose a novel model, VIBE: Variational Information Bottleneck for Evolutions. Concretely, we first employ two Information Bottleneck (IB) regularizers to distinguish past and future topics. Then, the distinguished topics work as adaptive features via multi-task training with timestamp and class label prediction. In adaptive learning, VIBE utilizes retrieved unlabeled data from online streams created posterior to training data time. Substantial Twitter experiments on three classification tasks show that our model, with only 3% of data, significantly outperforms previous state-of-the-art continued-pretraining methods.

VIBE: Topic-Driven Temporal Adaptation for Twitter Classification
Yuji Zhang' Jing Li Wenjie Li
"Department of Computing. The Hong Kong Polytechnic University, HKSAR, China Ressaacch Centre on Data Science & Artificial Intelligence
yu-ji. zhangeconnect. polyu. (jing-amelia, Ii, wenjie. 1i @épolyu. edu. hk
Abstract
Evolving Topics Over Time
T 0: Covid, NoMask, ReturnToWork Dread
Language features are evolving in worldworld social media, resulting in the deteriorating per- formance of text classification in dynamics. To address this challenge, we study temporal adap- tation, where models trained past data tested the future. Most prior work focused on continued pretraining or knowledge updat- ing, which may compromise their performance on noisy social media data. To tackle this is- sue, reflect feature change via modeling la- tent topic evolution and propose a novel model, VIBE: Variational Information Bottleneck for Evolutions. Concretely, we first employ two Information Bottleneck (IB) regularizers to dis- tinguish past and future topics. Then, the dis- tinguished topics work adaptive features via multi-task training with timestamp and class- label prediction. In adaptive learning, VIBE utilizes retrieved unlabeled data from online streams created posterior to training data time. Substantial Twitter experiments three clas- sification tasks show that our model, with only 3% of data, significantly outperforms previous state-of ofte--oftthee the-art art continued
T + 1: WearAMask, LockDown, ImmunnCoomrrmmise
T 2: Symptom, PanicBuy, ImWithFauci, TrumpKillUs
T + 3: Therapy. TrampisANationalDisgee Lying
T 4: RedStates, TrumpLandSlid, TrampLiesAmericans
Figure I: Evolving topies in COVID-19 Stance dataset over time. T+7 (r=0, 1..,, 4) is in chronological order.
tures (Hendrycks et al., 2020), when there is a time gap between training and test data. Röttger and Pierrehumbert (2021) and Luu et al. (2022) then defined temporal adaptation task to adapt model trained the past data to the shifted future data.
To tackle this task, Röttger and Pierrchumbert (2021) and Luu et al. (2022) apply continued pre- training on the temporally-update data to catch the distribution shift yet observe limited enhance- ment with noisy contexts, meanwhile relying heav- ily computational resources. Others (Shang et al., 2022; Li et al., 2022: Chen et al., 2022) re- sort to knowledge-basee methods, which require high-quality data unavailable on social media.
Given these concerns, how shall we effectively approach temporal adaptation in noisy social me- dia contexts? Inspired by the success of latent topics in many social media tasks (Zhang et al., 2021; Pathak et al., 2019), our solution is to ex- plore latent topic evolution in representing feature change (Zhang and Li, 2022). To better illustrate our idea, we exemplify the COVID-19 stance detec- tion dataset (Glandt et al., 2021). Here employ the Neural Topic Model (NTM) (Miao et al., 2017) to capture the latent topics in five sequential peri- ods and show the top topic words by likelihoods in Figure As can be seen from the topic evolution, users changed discussion points rapidly over time, where they shifted focus from concerns to the virus itself (e.g.. "Immune Compromise", "Symptom") to the disappointment to the former US President Trump (e.g.. "Trump Land Slid", "Lying Trump"). From this example, we can learn that topics can help characterize evolving social media environ-
continued-peetraining pretraining methods.
1 Introduction
Our ever-changing world leads to the continuous data distribution shifting in social media. As result, language features, formed by word pat- terns, will consequently evolve over time. There- fore, many text classification models, including the state-of-the-art ones based on pretraining, demon- strate compromised results facing shifted fea-
"oode will later https://githi com/ For technical questionn, contact zyu-ji connect.polyu..ik
CelestinezYuuV1BB. Mata---epprall--daaaaaa
'This work is supported by the NSFC Young Scientists Fund (Project No. 62006203). a grant from the Research Grants Council of the Hong Kong Special Administrative the Innova- tion and Technology Fund (Project No. PRP/047/22FX), and PolyU Internal Fund from RC-DSAI (Project No. 1-CE1E). Jing Li is corresponding author. The authors would like to thank the anonymous reviewers from EMNLP P 2023 for their insightful suggestions of various aspects of this work.
Region, China (Project No. PolyU/25200822

---

ments, which motivates to propose topic-driven temporal adaptation. Specifically, we employ an NTM for featuring latent topies, which under the control of Information Bottleneck (IB) regulariz- ers to distinguish past-exclusive, future-exclusee and time-invarien invariant latent topics to represent evolu- tion. Here the time-invariant topic is to encode the task-relevant semantics unchanged over time; the past- and future-exclusive latent topies capture the time-variant topics by modeling the data from the past and future. To further align the topic evolution to a specific classification task, we leverage multi- task training to infer timestamps and class labels jointly. Our model is then named VIBE, short for Variational Information Bottleneck for Evolutions.
2 Related Work
2.1 Temporal Adaptation
Our task is in line with temporal adaptation, a sub- task of domain adaptation (DA). While prior DA studies focus on domain gaps exhibiting intermit- tent change, temporal distribution shift usually hap- pens step by step, forming successive process. Liu et al. (2020a) showed the impracticality of ap- plying traditional DA on temporal adaptation be- cause the former aims to eliminate the distribution shift (Xie et al., 2021: Shen et al., 2021; Liu et al., 2020b; Hendrycks et al., 2020; Chen et al., 2021) while the adaptive objective for the latter is to track the shifts. To track the shifts, Huang and Paul (2019) utilized more labeled data to adapt, result- ing in high costs in the ever-evolving environments. Unsupervised temporal adaptation was therefore advanced in two lines. One is based on contin- ued pretraining on large-scale future data (Röttger and Pierrehumbert, 2021; Luu et al., 2022; Huang et al., 2022), relying on extensive computational re- sources whereas interior with noisy context. The other adopted external knowledge resources (Shang et al., 2022; Li et al., 2022; Chen et al., 2022, 2023; Song and King, 2022; Liang al., 2022), requir- ing high-quality data unavailable in social media. Differently, we feature evolving latent topics with less reliance on data quality and quantity.
To study how VIBE functions on social media data, we utilize the data temporally posterior to training data to learn adaptation (henceforth adap- live data). Here we consider two scenarios. First, in an ideal circumstance, the data between training and test time from the original dataset can serve as the golden adaptive data. Second, when it is in- accessible in a realistic setup, we retrieve relevant online streams as the retrieved adaptive data.
For experiments, tion setup on three text classification tasks with data from Twitter platform and draw the following re- sults. First, the main comparison shows that VIBE outperforms the previous stateof--the-ar (SOTA) models with either golden or retrieved adaptive data. For example. VIBE brings relative increase of 5.5% over the baseline for all datasets, using only 3% of the continued pretraining data. Then, an ablation study exhibits the positive contribution of VIBE's sub-modules, and VIBE is insensitive to retrieved adaptive data quality. Next, we exam- ine the effects of adaptive data quantity on VIBE, and the results show that VIBE relies much less on data volume compared to continued pretraining. At last, we present case study to interpret how VIBE learns the evolving topic patterns for adaption.
conduct a temporal adapta-
2.2 Topic Modeling for Social Media
A topic model is a statistical model for discov- ering abstract topics occurring in a collection of texts (Miao et al., 2017), whose potential for deal- ing with dynamics comes from its capability to clus- ter texts exhibiting similar word statistics. Many previous studies (Bak et al., 2014; Naskar et al., 2016; Srivatsan al., 2018) have demonstrated the effectiveness of topic models to encode social me- dia texts. Furthermore, Saha and Sindhwani (2012); Pathak et al. (2019); Balasubramaniam et al. (2021) analyzed the evolving topic semantics via topic ma- trix factorization. Nevertheless, none of the above studies explore how latent topics help social media temporal adaptation, which is gap we mitigate.
To sum up, we present three-fold contributions:
To the best of our knowledge, we are the first to explore how to model latent topic evolution as temporal adaptive features from noisy contexts.
3 Preliminaries
We propose a novel model VIBE coupling the information bottleneck principle and neural topic model for temporal adaptation in social media.
To prepare readers for understanding VIBE's model design (to appear in $4), we present a preliminary section here with the problem formulation in 13.1, information bottleneck principle (VIBE's thcoreti- cal basis) in $3.2, and the model overview in $3.3.
Substantial Experiments show that VIBE sig- nificantly outperforms previous SOTA models, and is insensitive to adaptive data quantity.

---

BERT
C
past tweet
BERT
Cask
(x*11e*)
Recordmacial BoW
post fweet
pxt
post
time
(=*
2()
projection
Juntro
Clack
future tweet t
p(
projection
Cime
future tweet
(<>"
BERT
Recoustracted future BoW
Figure 2: The framework of our model VIBE. Taking past and future tweets t", t the input, VIBE derives the latent topics they conditioned on, as past-exclusive, future-exclusive, and time-invariant topics x", 2", and x". Clask is MLP for first stage training on predicting class label. Clack and C2 Cime are second stage multi-task task training.
3.1 Problem Formulation
Here I(:.)) is the mutual information of two ran- dom variables. Minimizing LıB enables Z to be maximally expressive for Y with a tradeoff of min- imally relying on X (Lin et al., 2020; Wu ct al.. 2020). In VIBE, take time-invariant topics 2" example, for the past tweets 8², the IB regularizers force the time-invariant topics 2 to be expressive for t", meanwhile penalizing disturbance from fu- ture tweets t For the future tweets tV, vice versa. Thus, the representation of = is distinguished from time-variant = and z". For tractability of mutual information, the variational approximation (Alemi et al., 2017) is adopted to optimize the IB objective function by constructing lower bound.
We explore temporal adaptation in text classifica- tion on Twitter, where an input tweet sample de- notes f, and its class label c. Following Luu et al. (2022), the training data is from the past period X with labels (denoted Dx = (t", ));; In contrast, adaptive and test data are unlabeled and from the future period Y (denoted Dy = (1V J). Our goal here is enable VIBE to infer accurate futuree label e via exploring the topic evolution from t" t
To that end, VIBE explicitly captures time- invariant, past-exclusive, and future-exclusive top- ics, respectively denoted 2", 2", and 2 which are extracted and differentiated from the past and fu- ture tweets, t and tV z" represents the past and only be learned from t², not from tV; likewise, 2 reflects the future and exclusively originates from t without information from t" The shared part of 1 and 1 modeled by 2², unchanged over time yet helpful in indicating class labels.
3.3 Model Overview
Based on $3.1 and 3.2, we then brief an overview of VIBE before detailing its design in 84. For modeling latent topics, we adopt NTM (the vanilla NTM is elaborated in A. to take bag-of- word (BoW) vectors of past tweets 1 and future tweets t as input to derive past-exclusive, future- exclusive, and time-invariant latent topic variables. During the learning process, the two BoW vectors are first encoded into latent topics 2², =², and 2", then decoded reconstruct the BoW vectors. Their disentanglement handled by the IB regularizers, which will be claborated in $4.2.
For adaption, VIBE should be fed with the adap- tive data, N samples of unlabeled 1d from the future time Y (from either golden retrieved adaptive data) are selected on relevance each past tweet t (from training data). Taking t (past) and tr (future) as input, VIBE adopts the information bot- tleneck principle (introduced in $3.2) to distinguish topics =", z", and 2", which will be detailed in $4.2.
In addition, to encode more useful features, 2² is endowed with task specific semantics by using (2* t²) to infer the class label c" of past tweet tt. We then employ the trained classifier Clask to pseudo-label the future tweets dh With the constructed BoW of t and 1 carrying the tempo- ral topie patterns from the input, we employ multi- task training to infer timestamps and class labels to align topic evolution features to classification.
3.2 Information Bottleneck Principle
Information Bottleneck (IB) Principal (TISHBY, 1999; Tishby and Zaslavsky, 2015) was designed for pursuing a maximally informative represen- tation with a tradeoff for general predictive abil- ity (Alemi et al., 2017). Its objective function is:
Li = ((; I(Z;Y).
(1)

---

4
VIBE Model
differentiate them for modeling topic evolution, we apply IB regularizers to force disentanglement of topics =², > and = following Hwang et al. (2020) (see $4.2). Additionally, we endow latent topics 2 with time-invariant task-specific semantics by training 2d tt to infer t"'s label c" (see 84.3).
Here $4.1 first describes how NTM explores the distribution of past and future tweets. Then, we discuss the design of time-oriented ($4.2) and task- oriented training ($4.3) to feature the time-invariant and time-variam latent topic variables by IB. Next, $4.4 presents how jointly optimize the NTM and classifier, followed by the application of the topic evolution features for final prediction $4.5.
4.2 Time-Oriented Training for NTM
In the following, we elaborate on how the IB reg- ularizers control NTM to learn the disentangled past, future-exclusive and time-invariant topics.
4.1 NTM with Multiple Latent Variables
As discussed in 33.1, NTM is used to capture the latent topic evolution by generating the past and fu- ture tweets t", 1 based on latent topics. We assume that paired past-future tweet samples (L", (8) come from the joint distribution (t", t (")""p(t"",t" ;t)) The distribution PD( t²) is jointly conditioned on vame-variant topics (past-exclusive topics 2" and future-exclusive topics z²) and time-invarian top- ics 2 To learn the latent topics, we pursue the maximization of the marginal likelihood (Diederik et al., 2014) of the distribution:
IB Regularizer. This regular- izer designed to capture time-invariant topics =² shared by the past and future tweets, whose design follows the interaction information theory (McGill, 1954). Interaction information refers to generaliz- ing mutual information to three or more random variables, representing the amount of shared infor- mation between them. Concretely, we denote the shared information between past time X and fu- ture time Y as 2 and present the formulation for interaction information [(X; Y; 2 as follows:
(f ,tt) f do da day (t"|=",
I(X;Y; I(x; I(X; (
(5)
Pay (0" ".",,"") =>(((=>))))))>))>)))))))) Ip(=" pp==))
= =x;;);))=(((;)))))) 100-1100) 1() 10:221x)
(6)
The above two equations represent / [(;; Y; Z through a symmetrical lens. As can be seen, in Eq. 5, we maximize the interaction information by maximizing the first term [X; 2 to make 2 expressive for past time X and minimizing the sec- ond term T(X: 8 Y) to penalize the disturbance from future time Y. Symmetrically, Eq. 6 encour- ages z be expressive for future time and avoid being affected by past time X. Optimizing these two functions simultancously would allow z to be time-invariantly expressive of X and Y.
Here the involved parameter 0=[6x: f mod- els the conditional distributions; our training objec- tive is to maximize the generative model Po (t", t²). To allow tractable optimization in training, we substitue Pa(a*, 2", 22",," ts, (2)) as the approximated posterior (=(2", =M | (", (V) based the varia- tional inference method (Diederik al., 2014) and the approximated posterior is factorized into:
90(=". = 2"1"" (t) 9ex (t"))))s((( ("))a (2 (P)
(3)
where e = ox, dbs Φy denotes the entire encoder parameter 1ax and Jay reflect the past-exclusive and future-exclusive topies X (past time) and Y (future time), and Løs represents time-invariant topics. For inference, we derive evidence lower bound (ELBO) of pt ,,)) (More in A.2):
VariaVa IB Regularizer. Our goal is to minimize the mutual information x and ;7)) thus past,, future-exclusive topics 2", nd are statistically independent of the time- invariant topics 2², and vice versa. Here we only present the formulation for the past time period X. and that for future time Y can be inferred anoma- lously. The mutual information of Z and 2 is: 1( (0=x(===-10 -1x== 2 1(x: 1(x: 23)
log 2 Byg [lgg
( [="))))]
+ 11 loog (t" [""""")]]
-D la (z [Ix()))] - DRL la (2"1t") [[p[=")] (4)
DxI. ig (= 1 [Ip())))
I(X; ,25) is maximized to force z and 2 to be jointly informative of past time X, while penalizes last two terms to avoid either 2 or x to be individually expressive for X (more in A.2).
However, three encoders 7(= 1 1 g(=01 | (",,t) (V), and q(>0| t²) possibly encodes 2", nd and 2n with substantial overlaps in their captured semantics. To

---

Joint Regularization for IB Regularizers. Time-variant and time-invariant representations are jointly regularized. We combine Eq. 5 and for time X's regularization, and time Y's is analogous.
and future. Then we dive deep into how to inject the task-specific features to 2 by inferring class labels. Specifically, we adopt the posterior latent topics =* t" (instead of directly using =*) to leverage the task-specific factor reflected by the class label cf and its input t². Here we concatenate tweet (s BERT embedding bt and the topic features r" r= (t)) yielded by NTM encoder (x"()), shown in Eq. 12) for predicting the class label c", At the output layer, the learned classification features are mapped to a label 2t with the formula below:
max I(X:Y: ((()))))
=I(X;Z* (,X;2*22>)-00 1(x;2*) - ((X;
(8)
max I(x;Y;2)) (2) =1(Y - ) -I()
9
(9) Tractable Optimization for IB Regularizers. Since there are intractable integrals like unknown distribution pD (t", (*) in mutual information terms mentioned above, we maximize generative distribu- tions' lower bounds for traceability concerns. More details for Eq. 10- 10-Eq. 12 are discussed in A.2.
6º Souu (Wout U. + boue)
(14)
(our(.) is the activation function for classification output (c.g., softmax). Wout and b are learn- able parameters for training. Ug couples the BERT- encoded semantics (b"") and the implicit task- specific topics via multi-layer perceptron (MLP):
For [X;; Z ) ), we derive its lower bound with the generative distribution p(t²
| 12" =*):
log (t")=""=")
I (x:Z*, x 23)
Egtar (tom))t)))
(t()
= -/MLP(W ML "rr"""""""""""" (x"(e")))) + bsErr) (15)
2 ((X) + EppitF
(*)gee ayy [log (t"la", "1"""""""" 20)g(= (0)
(10)
4.4 Joint Training for NTM and Classification
For intractable
(unknow distribu-
To coordinate NTM and classifier training, we com- bine classification loss and the NTM reconstruction loss together. For classification, the loss Lpast is based on cross-entropy (trained labeled past data) Adding LNTM (Eq. 13) to Lpast. we get the joint optimization loss function for training the cemporally-adaptivy classifier C Clask whose joint- training loss is the weighted sum of classification (task learning) and NTM (topic evolution) losses:
tion (t")), the generative distribution p(=*) is adopted as the standard Gaussian, known as the Variational Information Bottleneck (Tishby and Za- slavsky, 2015). Its lower bound goes as follows:
I(x;2*> 2 (xpoon)Dxxllaa=
(11)
Likewise, the lower bound for - (X; z)( is:
- -I(x22 = (eppee/.owaeeerrr....... ((="|t", .
((>1(1))
2 (Ep [KL lg (=""cc" ||v (=")("))]
L Epast + p Lytm
(16)
(12)
Substituting Eq. 10-12 to Eq. 8-9, we derive the lower bound for disentangling past time X and future time Y. For joint regularization on disentan- glement and the maximum likelihood objective in Eq. 4, we combine them for joint maximization:
Here f trades off the effects of task and topic evo- lution learning. In this way, their parameters are updated together. After the training, we employ the trained Clask to pseudo label future tweets (2 from adaptive data to form Dy = (1", clt)) for the next stage training, which will be discussed in $4.5.
Ey av (log p(t".e".=" 2""
max
5.0
7(2 2" A(2 I(X:1; 1(2) -1>" 112") (2))
4.5 Topic Space Projection
In the original design of NTM decoding (Miao et al., 2017), 1t and 1 are reconstructed based on a ruthless vocabulary dictionary. We tailor-make it to map the reconstructed t and t into a sphere space to align tweets with similar semantics to exhibit closer distance in topic space. Meanwhile, it is con- ducted via multi-task learning on predicting times- tamps and class labels to inject time-awareness senses into the classification training:
max(1 + A) B (0 (.L. P..."
+ A- Eppitt 1v [Dxz[q(>" (")'bo"""
E an) [Dxclo(="
DKrll(e" e, (s*("))
(13)
This objective is denoted as Lutm (the NTM loss).
4.3 Task-Oriented Training for NTM
We have discussed controlling the latent topics = to represent tmme-invarian features shared by past
ÿ) = =f (W your Ur + byout)
(17)

---

Dataset
Scale
Adaptive
Time Span
starce: Emain Sain-goiden
Stance: from retricve
Stance
7.122
35,610
2020FFch-2020/Sep
0.4
beab-ratreive
Hate
8,773
43,860
2018/Oct-2220/uly
Hash
22,415
65,820
201//ee--011//ee
NHN
0.3
Table 1: Data statisties. The "Scale" column shows the dataset's tweet number and the "Adaptive" column shows the number of tweets as retrieved adaptive data.
0.2
0.1
Figure 4: MMD score quantify distribution shifts. The gap between the test and retrieved adaptive data is larger than the cross-task one between Stance and Hate.
1st SToet ( Tout byout)
(18)
(-) is the activation function for classification out- put. W and are learnable parameters for training. uy concatenates tweet t's BERT embedding and its reconstructed BoW vector for class label y' and timestamp I" prediction via a MLP. The two classi- fiers are named C lank and C The cross-entropy losses for y' and 7' prediction are denoted as Ltask and Liime- We optimize their joint loss as follows:
Nov for adaptation and validation, Dec for the test. In setup, training data (with labels) are from the past; adaptive and test data are from the future. For gathering the retrieved adaptive data, we first analyzed the top 100 trendy words (excluding stopwords) in training data. Then, we searched their related tweets and sampled those posted in the same time period of test data (50 per day for each search word). Finally, retrieval method (e.g.. DPR) adopted for ranking and selecting tweets with the top-N highest similarity to each training sample to form the adaptive data.
Exphere
Crask + Leime.
(19)
5 Experimental Setup and Data Analysis
In the following, we introduce data (including its analysis) and baselines for the experimental setup.
Data Analysis. As previously shown, Twitter usually challenges NLP models with its noisy data (Wang et al., 2019). We then exemplify the Stance dataset and provide analysis to quantify unique challenges from noisy data to temporal adaptation. Following Gururangan et al. (2020)'s practice, we first analyze the vocabulary overlap of four sub- sets training set, golden adaptive data (from the original dataset), retrieved adaptive data (exter- nal data retrieved from the wild), and test set, As shown in Figure 3, the vocabulary's inter-subset overlap ratio is low, around 40-50%, in contrast to 60-90% non- non-Twitter data (Luu et al., 2022). It indicates the grand challenge of capturing temporal adaptive features on Twitter, echoing Huang and Paul (2019)'s findings that Twitter data tends to generate vocabulary gap faster than other data.
Data Collection. As displayed Table 1, we experiment with VIBE on three datasets for stance detection (Stance) (Glandt et al., 2021). hate speech (Hate) (Mathew et al., 2021), and hashtag (Hash) prediction. Their data is all from Twitter, and each sample is tweet with classification label. We show more details of datasets in A.3.
Data Setup. For the Stance and Hate dataset, tweets were sorted by time order, where the earliest 50% samples are for training and the latest 30% for test. The middle 20% tweets posted between training and test time were adopted for validation (5% random samples) and the golden adaptive data (the rest 15%). For Hash dataset whose data is unevenly distributed temporally, we adopted the ab- solute temporal split: Sep and Oct data for training.
51.98
45.5
53.1
100.0
Model ERM
Stance
Hate
Hash
0.6603 0.8014 0.795920.448
0.8153 72778
DANN
0.667410 d639
0639
0.808313.228 2849
retrieve
55.38
48.92
100.0
53.1
Mixup
+0.077
0.6580+a. 9159
0.7915+2.0a51
MMD
0.7836=0.3799
0.661 1 698.
6981
0.8114+2..035
golden 47.8
48.92
P+CTd
0.8028=0.646d
0.6708 +a. 1637
0.8171++a.ca
UDALM
0.8120=0.454
0.6755
$250 5250
0,8341 17.4
DPT
0.8173=002200
0.6712.1 1731
2.834061:30004
train 100.0
47.8
55.38
51.98
VIBE
0.8368 10.1914
0.690010.5501
0.8551 1:2.4603
train golden retrieve test
Table 2: Classification accuracy with golden adaptive data. We report mean standard deviation (std.) in ten runs. The order of magnitude for std. is 10-5
Figure 3: Pairwise vocabulary overlap (%): the vocabu- lary gathers the top 5K most frequent words (excluding stop words) from each subset.

---

Datasets
Stance
Hate
Hash
Continued Pretraining Models
UDALM
DPT
UDALM
DPT
UDALM
DPT
RANDOM
0.8043-0A62
0.8
0.661320789
0.
0.82019
0.8217-22-68
DPR
0.8127 0,7706
0.810
0.6708 20.3764
0.6732.00250
0.8283 22,3412
0.831343..00
Retrievers (Nm10)
BM25
0.8100
0.813510.75
0.669610.4929
0.6724_00310
0.8251 126470
0.2594
Spider
0.8105-0.5992
0.8112+0.3552
0.6704=02253
0.6663+0.296
0.8246-3.520
0.8275 +3.1308
Condense
0.8099=0,114
0.8076+0.600
0.6709-1.127
0.0681+0880
0.8262 +2.812
0,8289 +-30750
ERM (buselloue)
0.8014go.4w
(+0.0%)
0.6603_7.
(+0.0%)
0.8153 (+0.01%)
DPT (SOTA, N=300)
00:547 (+2.9%)
0.6734_1000 (+2.0%)
0.8301 1:0s (+1.8%)
VIBE (O=10)
0.8422 9,300
(+5.1 %)
0.6950=14545 (+5,3%)
0,8617141660 (+5.7%)
Table 3: Classification accuracy on retrieved adaptive data. Percentages in brackets indicate the relative increase compared to ERM. N values mean we retrieve the top-N retrieved data as adaptive data. VIBE outperforms all comparisons significantly (measured by the paired t-test with p-value< 0.01) and achieves an increase of more than 5% for three datasets, using adaptive data scale 3% of the DPT (the SOTA based on continued pretraining).
We then quantify the distribution shifts of train- ing, test, and retrieved adaptive data with the commonly-used MMD criterion for measuring the feature difference (higher scores indicate a larger gap) (Gretton et al., 2012). In Figure 4, the MMD score of "stance: retrieve-test" is higher than that of "stance-hate: train". It means the distribution gap between the retrieved adaptive data and test data is even larger than the cross-task, cross-dataset distri- bution gap between stance and hate dataset.
Results on Golden Adaptive Data. As shown in Table 2, we draw the observations as follows:
Performances of all DA baselines are not com- parative to the ERM, which minimizes the predic- tion losses on past training data only. echos find- ings of previous studies (Rosenfeld et al., 2022; Ye et al., 2021) with two-fold reasons. First, DA meth- ods tend to minimize the distribution shift or extract the domain-invariant features instead of adapting to the temporal shift. Second, adapting to tempo- ral distribution shifts among noisy data presents challenge unable to be addressed by DA baselines.
Baselines. We first employ the ERM (Vapnik and Vapnik, 1998) baseline trained with past data only. Then, the Domain Adaptation (DA) base- lines- - adversarial- DANN (Ganin et al., based Mixup (Yan et al., 2020), distribution-aligniin based MMD (Li et al., 2018), and self-training based P+CFd (Ye et al., 2020) are compared. Be- sides, we consider state-of-the-arr (SOTA) Con- tinued Prefraining methods: UDALM (Karouzos et al., 2021) and DPT (Luu et al., 2022). To acquire large-scale adaptive data, we adopt Retrieval meth- ods via RANDOM, DPR (Karpukhin et al., 2020), BM25 (Robertson et al., 2009), Spider (Ram et al., 2022), and Condense (Gao and Callan, 2021). All models' implementation details are shown in A.4.
Continued pretraining methods are effective while VIBE performs the best. The better results from UDALM and DPT than other DA baselines indicate that continued pretraining on high-quality data may partially help. They are outperformed by VIBE, implying that VIBE can better utilize the adaptive data for temporal adaptation by modeling topic evolution. Nevertheless, it is concerned the small data scales may limit poatinue-pretraining 's performance, so we will then discuss how they work with large-scale retrieved adaptive data.
2016).
Results on Retrieved Adaptive Data. We have demonstrated VIBE's effectiveness with golden adaptive data split from the original dataset. How- ever, in the real world, well-collected, high-quality data may be unavailable. Therefore, we explore the potential of utilizing retrieved adaptive data from online Twitter streams. We compare VIBE (us- ing DPR retrieved data) to UDALM and DPT, the SOTA continued pretraining methods, with data collected by varying retrievers. The results are shown in Table 3, and we observe the following:
6 Experimental Results
In this section, we first discuss the main results (56.1). Then, an ablation study (56.2) discusses the contributions of various VIBE modules. Next, we quantify VIBE's sensitivity to adaptive data scales in $6.3. Lastly, case study ($6.4) interprets the features learned by VIBE in temporal adaptation.
Data quantity malters more to continued pre- training than quality. UDALM and DPT perform comparably. For them both, retrieval slightly helps, yet the selection of retrievers barely affects their results. Meanwhile, scaling up the data quantity
6.1 Main Results
We first compare classification accuracy with golden adaptive data and choose the best baselines to experiment with retrieved adaptive data.

---

Dataset
Stance
tive samples. pseudo-labeling still yields promising results. In fact, the performance of pseudo-labeling is only marginally worse than the upper-bound re- sults achieved with correctly-labeles data alone.
Vanilla NTM
0.8200-00:200
Module Ablation
II-NTM
4.8115
VIBE (Full)
0.8422
16
0.821320.5800
32
0.828620.4550
Topic Number (K)
64
0.8379+#0.499
6.3 Effects of Adaptive Data Scale
128
0.8422:t0.2220
256
1.8.66++0.242
As shown in $6.1, adaptive data scale (N) is crucial for DPT. We are then interested in how it affects VIBE and show the comparison of VIBE and DPT in Figure 5. As shown, VIBE consistently performs better and cxhibits much less sensitivity to adaptive data quantity than DPT, where the former peaks with very small N and the latter grows slowly.
512
0.83400=0.1807
DPR
0.8422.10.20 2036
Retriever (N=10)
Spider
0.8352++0.290
BM25
0.8309
Table 4: Ablation experimental results. Vanilla NTM:
simple concatenation with BERT and NTM features. IB-NTM: VIBE without topic space projection ($4.5).
0.86
substantially boost DPT's results. This observation is consistent to Luu et al. (2022) that continued pre-training relies on large-scale data to work well. VIBE exhibits the largest performance gain com- pared to baseline (ERM). VIBE uses 3% of the re- trieved data while outperforming DPT with million- scale data. It implies that VIBE can utilize the data efficiently and its learned topic evolution is useful.
visE (Dur Model) DPT (BOTA)
Accuracy
0.82
0.80
40
60
80
100
Volume Adaptive Date
Figure Accuracy (y-axis) over varying (adaptive data with the top-N retrieved samples). VIBE shows much smaller reliance on adaptive data scales than DPT.
6.2 Ablation Study
We have shown the overall superiority of VIBE, To provide more insight, we further discuss it with Stance (other datasets exhibit similar trends). Here, we investigate the efficacy of its modules through the experimental ablation results in Table 4.
6,4 Case Study on Evolving Topic Patterns
To further examine how VIBE works, we qualita- tively analyze the learned topics on Stance, where the changing stance towards Dr. Anthony Fauci is analyzed. We examine the stance labels of all tweets containing the word "Fauci" and find a ris- ing number of tweets in favor of Fauci over time, By taking a closer look at these tweets, we find that word "Fauci" often co-occurs with "Trump", where specifically, in the tweets against Trump, people tend to support Fauci. In other words, the "usually- against" stance gained from "Trump" "Trmmp"-tweets might, in return signal the "favor" stance for "Fauci"
First, compare module ablation results and find that either IB or topic-space projection con- tributes positively to VIBE, with their joint effects leading to the best results. Then we quantify the ef- feets of the NTM's topic number (K) and observe first increase- then decrease trends. This is because small K may cause underfitting, while large K may lead to sparse topics, both hindering VIBE's capability of capturing high-quality topics. Finally, we examine the VIBE with varying retrievers and observe that DPR works the best in finding suitable adaptive data thanks to its large-scale pretraining. In the second training stage of VIBE, future tweets are pseudo-labeled by the classifier trained in the first training stage. Thus we further explore the effects of label distribution of pseudo labels on model performance. We compare prediction accu- racy on Stance with either ground-truth pscudo la- bels false pseudo labels and observe the accuracy of 84.72% and 83.06% respectively. While pseudo- labeling self-training can introduce errors, which can potentially undermine its effectiveness, our findings suggest that when combined with posi-
Training: Against-Fauci
Test: Support-Fauci
Adaptive: Against-Trump
Figure 6: Tweets distribution in sphere topic space. Blue dots: against-Fauce tweets past training data; Pink dots: support-Fauci tweets in future test data; Green dots: against- egainst-Trump tweets in adaptive data.

---

tweets where they are both discussed. Interestingly, we do observe rising "for-Fauci" stance rate over time in tweets mentioning both Trump and Fauci.
generalize over more complex tasks with varying data resources (Zeng et al., 2018; Sun et al., 2023). Many social media applications suffer from tem- poral misalignment caused by distribution shifts over time. For instance, in a dynamic environ- ment, how to recommend users items that align with their evolving interests (Zhang et al., 2021)? Furthermore, harmful texts may contain mantics over time, making non-adapted models fail detect them. In the context of rumor de- tection, as rumors are tested by facts over time, learning new facts to reinforce rumor detection is necessary. More language understanding tasks also face the challenge of comprehending changing texts in dynamic language environments, resulting in compromised end-task performance, which is explored by Zhang and Li (2022)'s empirical study on the effects of temporal misalignment on social media downstream tasks when there is a time gap between training and test data. We will explore our model VIBE in more application scenarios like the aforementioned tasks in the future to enhance performance of language models in the real world.
We then visualize VIBE's sphere topic space of Fauci- and Trump-related samples in Figure 6. Recall that VIBE projects the reconstructed BoW vectors of tweets into sphere space via multi-task training on predicting time and class labels to ob- tain time-aware coordinates of tweets (as shown in 4.5). In the left sphere, the distribution of against- and support-Fauci tweets from the past and fu- ture are different, which presents challenges for the model to infer the changing stance. Neverthe- less, in the right sphere, the against-Trump tweets from adaptive data exhibit similar distribution with support-Fauci tweets from future test data, allowing VIBE to adapt the change. These ob- servations show VIBE can effectively capture topic evolution, largely benefiting temporal adaptation.
7 Discussion on Application Scenarios
In this section, we envision more application sce- narios where our model VIBE can potentially be applied for dynamic language understanding in temporally evolving environment.
Conclusion
We have presented the first study on the effects of latent topic evolution for temporal adaptation in noisy contexts. A novel model VIBE has been proposed leveraging the information bottleneck capture topic evolution. In experiments, VIBE sig- nificantly outperforms the continued pretraining methods with much smaller scales adaptive data.
In the real world, the rapid changes in language environments challenge language models. Previ- ous approaches often resort to continuously updat- ing models to cope with shifted distributions and newly emerging knowledge, resulting in extensive resource consumption. Then how about taking ad- vantage of large language models that are highly knowledgeable? We have explored applying a zero- shot large language model (LLM) LLaMa (Touvron et al., 2023), while cxhibiting compromised perfor- mance compared with the transformer- VIBE, showing that purely scaled-up language models can not tackle temporal misalignment. Moreover, it may be prohibitively expensive to keep updating LLM's knowledge via continual training, and the challenge will be exacerbated as models continue scale up (Bender et al., 2021; Ziems et al., 2023). Inspired by that topic evolution can indicate lan- guage evolution in a temporally changing environ- ment (Zhang and Li, 2022), VIBE unveils a low- cost temporal adaptation approach by reflecting on past and future topic statistics simultaneously to infer evolution, which is efficient and practical in the real world. VIBE is built upon the general text classification task, and it encodes sentence-Iever temporal semantics, hence having the potential to
Limitations
In the following, we summarize limitations of our study, which should be considered for future work. For the data we employ for adaptation, the re- trieved adaptive data was first crawled online and then retrieved by DPR. In the ablation study, we analyzed the difference of employing different re- trievers to retrieve future data for adaptation. The results demonstrate that although adaptive data col- lected by different retrievers all benefit temporal adaptation, the four retrievers exhibit varying in- fluences on the quality of adaptive data, which indicates the requirement for retrievers to select suitable adaptive data. Thus the DPR should be continued-trainel in the future keep pace with the evolving online environments.
For the research scope, our paper focuses on temporal adaptation for text classification on so-

---

cial media data, In the future, we will broaden our research scope to more applications like recommen- dation or generalization on dynamic contexts.
Emily M Bender, Timmit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM confer- ence on fairness, accountability, and transparency. pages 623-
Ethics Statement
In our empirical study, datasets Stance and Hate are publicly available with well-reputed previous work. These two datasets in general does not create direct societal consequence. The stance detection dataset helps us understand how the public opinions changed during the COVID-19, shedding lights for future work on public opinions study. The hate speech dataset is for defending against hate speech by training models to detect and prevent them. Our model aims to enhance model performance to bet- ter detect hate speech and climinate them in evolv- ing environments, which is for social benefit. The Stance and Hate datasets are for research purpose only. For experimental results, all the qualitative results we discuss are output by machine learn- ing models, which do not represent the authors' personal views. Mcanwhile, user information was excluded from these two datasets.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha, 2021. ATOM: robustifying out-of- distribution detection using outlier mining. In Machine Learning and Knowledge Discovery in Databases, Research Track European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part III, volume 12977 of Lecture Notes in Computer Science, pages 430-445. Springer.
Kai Chen. Ye Wang, Yitong Li, and Aiping Li. 2022. RotateQVS: Representing temporal information as rotations in quaternion vector space for temporal knowledge graph completion. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 5843-5857, Dublin, Ireland. Association for Compu- tational Linguistics.
Yankai Chen, Yifei Zhang, Menglin Yang. Zixing Song, Chen Ma, and Irwin King. 2023. WSFE: wasserstein sub-graph feature encoder for effective user segmentation in collaborative filtering. CORR. abs/2305.044| 10..
For the data collection of Hash dataset, Twit- ter's official API was employed strictly following the Twitter terms of use. The newly gathered data was thoroughly examined exclude any possi- ble ethical risks, e.g., toxic language and user pri- vacy. We also conducted data anonymization in pre-processing by removing user identities and re- placing @ @enttion with a generic tag. We ran a similar process for adaptive data's auto-collection.
P Kingma Diederík, Max Welling, et al. 2014. Auto- encoding variational bayes. In Proceedings of the International Conference on Learning Representa- tions (ICLR), volume 1.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario Marchand, and Victor Lempitsky. 2016. Nomain-adversane training of neural networks. The journal of machine learning research. 17(1):2096- 2030.
References
Luyu Gao and Jamie Callan. 2021. Condenser: a pre- training architecture for dense retrieval. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 981-993. Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Alexander A. Alemi, lan Fischer, Joshua V. Dillon, and Kevin Murphy, 2017. Deep variational information bottleneck. In International Conference on Learning Representations.
Jin Yeong Bak, Chin- Yew Lin, and Alice Oh. 2014. Self- disclosure topic model for classifying and analyz- ing Twitter conversations. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1986-1999 1986.1996. Doha, Qatar. Association for Computational Linguis- tics.
Kyle Glandt, Sarthak Khanal, Yingjie Li, Doina Caragea, and Cornelia Caragea. 2021. Stance detec- tion in COVID-19 tweets. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers). pages 19961111, Online. Association for Computational Linguistics.
Thirunavukarasu Balasubramaniam. Richi Nayak, Khanh Luong, and Md Abul Bashar. 2021. Iden- tifying covid-19 misinformation tweets and learning their spatio-temporal topic dynamics using nonnega- tive coupled matrix tensor factorization. Social Nei- work Analysis and Mining, 11()):57.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander J. Smola. 2012. A kernel two-sample test. Mach. Learn, Res., 13:723-773.

---

Suchin Gururangan, Ana Marasovie, Swabha Swayandipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah A. Smith, 2020. Don't i stop pretraining: Adapt language models domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: a benchmark for question answering research. Trans. Assoc. Comput. Linguistics, 7:452- 466.
Haoliang Li, Sinno Jialin Pan, Shiqi Wang. and Alex C Kot. 2018. Domain generalization with adversarial feature learning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition. pages 5400-5409.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishman, and Dawn Song. 2020. Pretrained transformers improve out-of disdrribibttonn robustness. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 2744-2751, Online. Association for Computa- tional Linguistics.
Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng, Yajuan Lyu, Yong Zhu, Long Bai, Wei Li, Jiafeng Guo, and Xueqi Cheng. 2022. Complex evolutional pattern learning for temporal knowledge graph rea- soning. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 290-296, Dublin, Ireland. Association for Computational Linguistics.
Hao Huang, Xiubo Geng, Guodong Long, and Daxin Jiang. 2022. Understand before answer: Improve temporal reading comprehension via precise question understanding. In Proceedings of the 2022 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 375-384, Seattle, United States. Association for Computational Linguisties.
Langzhang Liang, Zenglin Xu, Zixing Song, Irwin King, and Jieping Ye. 2022. Resnorm: Tackling long-tailed degree distribution issue in graph neural networks via normalization. CoRR, abs/2206.08 0811..
Xiao Lin, Indranil Sur, Samuel A. Nastase, Uri Hasson, Ajay Divakaran, and Mohamed R. Amer. 2020. A data-efficient mutual information neural estimator for statistical dependency lesting.
Xiaolei Huang and Michael J. Paul, 2019. Neural tem- porality adaptation for document classification: Di- achronic word embeddings and domain adaptation models. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 4113-4123, Florence, Italy, Association for Computational Linguistics.
Hong Liu, Mingsheng Long, Jianmin Wang, and Yu Wang. 2020a. Learning to adapt to evolving domains. In Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
HycongJoo Hwang, Geon- Heon-H Kim, Seunghoon Hong, and Kee-Eung Kim. 2020. Variational inter- action information maximization for cross-domain disentanglement. In Advances in Neural Information Processing Systems 33: Annual Conference on Neu- rul Information P Processing Sysiems 2020, NeurIPS 2020, December 6-1 2020, virtual.
Xiaodong Liu, Hao Cheng, Pengeheng Hc, Weizhu Chen, Yu Wang, Hoifung Poon, and Jianfeng Gao. 2020b. Adversarial training for large neural language models. CoRR, abs/2004,08994,
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Kar- ishma Mandyam, and Noah A. Smith. 2022. Time waits for no one! analysis and challenges of tem- poral misalignment. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5944-5958, Seattle, United States. Association for Computational Lin- guistics.
Constantinos Karouzos, Georgios Paraskevopoulos, and Alexandros Potamianos. 2021. UDALM: Unsuper- vised domain adaptation through language modeling. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 2579-2590, Online. Association for Computa- tional Linguisties.
Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukher- jee. 2021. Hatexplain: A benchmark dataset for ex- plainable hate speech detection. In Thirty-Fifih AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial In- telligence, EAAl 2021, Virtual Event, February 2-9, 2021, pages 14867- AAAI Press.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Wei Chang, Andrew M. Dai, Jakob
W McGill. 1954. Multivariate information transmis- sion. Transactions of the IRE Professional Group on Information Theory, 4(4):93-111.

---

Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th In- ternational Conference on Machine Learning, ICML 2017, Sydney, NSW. Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Re- search, pages 2410-2419. PMI.R
Chao Shang, Guangtao Wang. Peng Qi. and Jing Huang. 2022. Improving time sensitivity for question answer- ing over temporal knowledge graphs. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Pupers), pages 8017-8026, Dublin, Ireland. Association for Computational Linguistics.
Debashis Naskar, Sidahmed Mokaddem, Miguel Re- bollo, and Eva Onaindia. 2016. Sentiment analysis in social networks through topic modeling. In Proceed- ings of the Tenth International Conference on Lan- guage Resources and Evaluation (LREC' 16), pages 46-53, Portoro7, Slovenia. European Language Re- sources Association (ELRA).
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang. Renzhe Xu, Han Yu, and Peng Cui. 2021. Towards out-of-distribution generalization: A survey. CoRR. abs/2108. 13624.
Zixing Song and Irwin King. 2022. Hierarchical het- erogeneous graph attention network for syntax-awart summarization. In AAAI, pages 11300-1348, AAAI Press.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English twects. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 14, On- line. Association for Computational Linguistics.
Nikita Srivatsan, Zachary Wojtowicz, and Taylor Berg- Kirkpatrick. 2018. Modeling online discourse with coupled distributed topics. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4673-4682, Brussels. Belgium. Association for Computational Linguistics.
Ajeet Ram Pathak, Manjusha Pandey, and Siddharth Rautaray. 2019. Adaptive model for dynamic and temporal topic modeling from big data using deep learning architecture. International Journal of Intel- ligent Systems and Applications, (6):33
Chenkai Sun, Jinning Li, Yi Fung, Hou P. Chan, Tarek Abdelzaher, Chengxiang Zhai, and Heng Ji. 2023. Decoding the silent majority: Inducing belief aug- mented social graph with large language model for response forecasting. The 2023 Conference on Em- pirical Methods in Natural Language Processing (EMNLP).
Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. 2022. Learning to retrieve pas- sages without supervision. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 2687-2700 Seat- tee. United States. Association for Computational Linguistics.
N TISHBY. 1999. The information bottleneck method. In Proc. of the 37th Allerton Conference on Commu- nication and Computation, 1999.
Naftali Tishby and Noga Zaslavsky. 2015. Deep learn- ing and the information bottleneck principle. In 2015 ieee information theory workshop (itw), pages 1-5. IERE.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: Bm25 and be- yond. Foundations and Trends@ Information Re- Irieval, 3(4)3333-38.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint 1rXiv:2302.13971.
Elan Rosenfeld, Pradeep Kumar Ravikumar, and An- drej Risteski. 2022. Domain-adjuss regression or: ERM may already learn features sufficient for out-of- distribution generalization. In NeurlPS 2022 Work- shop on Distribution Shifts: Connecting Methods and Applications.
Vladimir Vapnik and Vlamimir Vapnik. 1998. Statisti- cal learning theory wiley. New York, 1(624):2.
Paul Rötteer and Janet Pierrehumbert. 2021. Temporal adaptation of BERT and performance on downstream document classification: Insights from social media. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2400-2412, Punta Cana. Dominican Republic. Association for Compu- tational Linguistics.
Yue Wang, Jing Li, Hou Pong Chan, Irwin King. Michael R. Lyu, and Shuming Shi. 2019. Topic- aware neural keyphrase generation for social media language. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguisties. pages 2516-2526, Florence, Italy. Association for Computational Linguistics.
Ankan Saha and Vikas Sindhwani. 2012. Learning evolving and emerging topics in social media: a dy- namic nmf approach with temporal regularization. Proceedings of the filk ACM international confer- ence on Web search and data mining, pages 693-702.
Tailin WU, Hongyu Ren, Pan Li, and Jure Leskovec. 2020. Graph information bottleneck. In Advances in Neural Information Processing Systems, volume 33, pages 20437-20448. Curran Associates, Inc.

---

Sang Michael Xie. Ananya Kumar, Robbie Jones, Fereshte Khani, Tengyu Ma, and Percy Liang. 2021. In-n-out: Pre-training and self self-training using auxil- iary information for out-of-distribution robustness. In 9th International Conference on Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. Oveeeeve...net.
in the generation time, reflects the implicit seman- tic consistency throughout time and enables the learning of underlying past-to-future connections.
The NTM includes an encoder and a decoder. For the input, a tweet is firstly mapped into the bag-of-word (BoW) vector form, then fed into the auto-encoder. Given the BoW input t, the clus- tering is conducted through auto-encoding, which contains encoding process to map t into latent topic vetcor 2t, followed by decoding to rebuild conditioned on the topic (z,). During this process, Ze a K dimensional vector; each entry reflects the chance of t clustering into a certain topic and K is a hyper-paramedt representing the total topic number in corpus. Below presents concrete steps. For encoding, is embedded into the latent topic space to generate z, via Gaussian sampling, where the mean μ and standard deviation o are learned with the following formula:
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. 2020. Improve unsupervised domain adap- tation with mixup training. star, 1050:3.
Hai Yc. Qingyu Tan, Ruidan He, Juntao Li, Hwee Tou Ng. and Lidong Bing. 2020. Feature adaptation of pre-trained language models across languages and do- mains with robust self traininn. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7386-7399, Online. Association for Computational Linguistics.
Nanyang Ye, Kaican Li, Lanqing Hong. Haoyue Bai, Yiting Chen, Fengwei Zhou, and Zhenguo Li. 2021. Ood-bench: Benchmarking and understanding out- of-distribution generalization datasets and algorithms. arXiv preprint arXiv:2 106.00721. 1(3):5.
= fo(fe(t)) o (o))((t)
(20)
Jichuan Zeng, Jing Li, Yan Song, Cuiyun Gao. Michael R. Lyu, and Irwin King. 2018. Topic mem- ory networks for short text classification. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3120-3131, Brussels, Belgium. Association for Computational Linguistics.
(-) is a Rel.U-activated neural perceptron. Then Z, is drawn from the normal distribution below:
N (H4 o))
(21)
It is later transformed to a distributional vector via softmax function yield 0, representing the topic mixture of It initiates decoding step re-construct t by predicting é below:
Yuji Zhang and Jing Li. 2022. Time will change things: An empirical study on dynamic language understand- ing in social media classification. arXiv preprint arX.v:2210.02857.
i sofima((so00.))
(22)
Yuji Zhang, Yubo Zhang, Chunpu Xu, Jing Li, Ziyan Jiang, and Baolin Peng. 2021. HHowYouTar Twwees: Learning user hashtagging preferences via person- alized topic attention. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 7811-7820 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
fo(-) is another ReLU-activated perceptron map- ping information in topic space back to the BoW. The weights of f-(-) (after softmax normalization) are employed to represent the topic-wort distribu- tions and the latent topic vector (with cross-time views gained in clustering) can be engaged in clas- sification (Eqq. 15) to capture dynamics over time.
Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2023. Can large lan- guage models transform computational social sci- ence? arXiv preprint arX1v:2305.03514.
A.2 Proofs
In this section, we explain some derivations from our formulations in detail.
A
Appendix
ELBO for pa(t*, We explain how the ELBO of po( (8) is derived, as shown in Eq. 23.
A.1
Vanilla Neural Topic Model
In this section, we elaborate on the vanilla neural topic model, which is the basc of our model VIBE. The potential of NTM to deal with dynamics comes from its capability of clustering posts ex- hibiting similar word statisties and forming la- tent topics to reflect their shared discussion point. Therefore, the intrarrluster content, though varying
IB Regularization on Mutual Information. We explain how Eq. 5-Eq. 7 are derived here. Interac- tion Information between X. Y, and Z is:
I(X; Y; 2) = I(X;Y) I(X;Y|Z)
= =][X - -(X;Z]))
(24)
= =({:2)) - -(()2)x)

---

Dogp(t, =log| p p(t" xx"x"")(("" 22",2))
=log p(t (t"
24, 2222)) 22.2")
( 9 (9) do d2"
((2 19)g((" (3,0)
p( (2"""))p(( (())))
-lo Baterrleetttteeeeeeo
(23)
( (1))g))(( (1 (9)
2egger e (o log (Cc 1=",="p((" ( p"2=")((")pp((")
7/=" ()) (")(""r,,))) (2g (2))
Eageeeeee (5,0) [logp (f" 12" + (e [ x*",
- Duu (2" [1)))= 1K4 la (=" tt" (n")] - DRu (2" (t") [nn)))))))
Eq. and Eq. 6 are derived similarly. Morcover, given T(X; 2) f(x; 2[x I(Y: -I(Y; 2(x),
Thus we built Hash dataset for studying the user- generated labels instead of well-annotated clean data. In this dataset, the classification label is a hashtag the author annotates to indicate tweet's s topic. Its data was gathered following Nguyen et al. (2020) with English tweets posted from September to December 2011, 1, to allow balanced year cov- erage across different datasets. Following Zhang et al. (2021), the top 10 hashtags with the high- est frequency were selected to be the labels. Then tweets containing these hashtag labels were pre- served to group the Hash dataset.
the mutual information between 2 and 2 is:
x 1(2 Ix)
(25)
Since (x=11z) = q(=1 rr ²)), the last term in Eq. 25 equals zero, whose calculation goes as:
H |x, 1x(
=H(z*Ix) ((Z* (26)
Thus eliminate the zero term and get:
(27)
A.4 Implementation Details
We discuss the implementation details of our model VIBE and comparison baselines in this section.
As Eq. 24,
simi-
larly - = 1(X: 1(X: (x,;
Thus:
For retrievers, we adopted the DPR2 model of the Natural Questions BERT Base check- point (Karpukhin et al., 2020). This model has been pretrained on the Natural Questions dataset (Kwiatkowski al., 2019) for open-domain question answer. For other retrievers, we also adopted their pretrained checkpoint for retrieval.
1 I(X; z*) I(X; I(X;"* (xx))
(28)
Tractaility for Joint Regularization. Hwang et al. (2020) elaborates how the lower bounds of I(X: )2 I(x; YYY and [X;; x are de- rived detailedly via variational approximation.
For domain adaptation baselines, we employed the official implementation" of them. Since these domain adaptation baselines were implemented on image processing tasks, we modified the part of the image processing code to natural language process- ing code. We trained each model for 40 epochs and fixed parameters via grid search validation data. For continued pretraining, UDALM fine-tunes itself using mixed classification and Masked Lan- guage Model loss on past training data and future adaptive data together. For DPT, we used unlabeled future adaptive data for continued pretraining of BERT with MLM loss and then fine-tuned the re-
A.3 Dataset
For the three datasets, we adopted the Twitter API for recovery' of tweets with missing information (c.g., the timestump). The Stance detection dataset contains tweets with annotated stances about var- ious COVID-19 topics (Glandt et al., 2021). The Hate Speech data was released by Mathew et al. (2021) with labels indicating whether not the hate speech exists in tweet. These two datasets from publicly available benchmarks with relatively clean annotations. However, many social media applications are built upon noisy user-generated labels (Wang et al., 2019; Zhang et al., 2021).
DPR 'https://g) //ttps:://gthuub.co//// http://hhh...c/maccccaaaeaaaaaee
oper. twi tter. cwwee////////////////
DomainBed

---

sulting model labeled past training data, We implemented the based on official code for BERT (base, uncased). UDALM was trained for 40 epochs. DPT was trained for 30 epochs for pretraining and 10 epochs for fine-tuning.
For our model VIBE, we took the BERT (base, uncased) for the BERT encoding, ensure fair comparison with the continued pretraining base- lines. For all the MIPs, the hidden layer size was set to 2048. For NTM, the topic number for each latent topic including 2", 2 and 2 is set to 128. The corpus size is set to 20,000. VIBE is trained in two stages. For the first stage of training, The Latm in Eq. 13 is first optimized via updating NTM parameters for one epoch for warm-up, then past NNTM are jointly optimized via updating NTM, BERT, and MLP parameters for 10 epochs. The parameters
fixed via grid search on val- idation data. Then the pretrained classifier was used to pseudo-label the unlabeled adaptive data. Then VIBE was trained in second stage, where the reconstructed BoW vectors of past and future tweets were projected to the sphere topic space via multi-task training on inferring time and class la- bels simultaneously. During this, all parameters of VIBE were updated together and fixed via grid search on validation data for 10 epochs.
'nttps://g/ com.b.commoooe cob/huggl i acce transformers


Reviews:
[{"review_id": "3diFxs4w9T", "reviewer": "Reviewer_MQvq", "summary": "", "questions": "- Have the authors considered using a different input representation other than bag-of-words for the topic model learning?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\nQ1.1: What about using a different input representation other than bag-of-words for the topic modeling?\n\nA1.1: Thanks! The reason for adopting the Bag-of-Words (BoW) representation is that it reflects word statistics from past to future and provides a sparse representation that complements the dense representation captured by transformer encoders. Previous studies have also demonstrated the advantages of using BoW representations in modeling noisy social media text [1, 2]. We will further explore different forms of representation for topic modeling in future research.\n\nReferences:\n\n[1] Dieng, Adji B., Francisco JR Ruiz, and David M. Blei. \"Topic modeling in embedding spaces.\" Transactions of the Association for Computational Linguistics 8 (2020): 439-453.\n\n[2] Lixing Zhu, Gabriele Pergola, Lin Gui, Deyu Zhou, and Yulan He. 2021. Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1571–1582, Online. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is very well-written and easy to understand. I have not read topic model papers in some time, but I could readily follow the mathematical derivations and ideas. \n- Authors use topic models to combine the latest transformer-based representations with traditional Bag-of-words (BoW) representations very elegantly in the proposed method. It's rare to see topic model papers, and exciting to see one.\n- The experimental results are convincing; the authors compare the proposed method with domain adaptation baselines, with continued pretraining baselines covering multiple baselines that tackle the feature drift problem.", "reasons_to_reject": "- I have no serious concerns with the paper, it's very well-written, proposes a solid technical contribution, and good empirical results. ", "questions_to_authors": "- Have the authors considered using a different input representation other than bag-of-words for the topic model learning?", "paper_topic_and_main_contributions": "The paper deals with the problem of Temporal Adaptation for Tweets classification. The authors model the latent topic evolution using a modified Neural Topic Model by learning time-dependent and time-invariant topics and later using the topics for label classification. \nAuthors adapt the Information Bottleneck principle to time-invariant and time-variant IB regularizers to learn the topics. The time-invariant representation is injected with the class information by training it to predict the class labels. Experiments on Tweet classification datasets demonstrate the efficacy of the proposed method over well-established baselines. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "No missing reference as far as I know. ", "Typos_Grammar_Style_and_Presentation_Improvements": "No grammatical mistakes I could find in the paper, it's very well-written."}, "extra_text": {}}, {"review_id": "S75qhgushM", "reviewer": "Reviewer_QHXk", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\nQ2.1: The presentation of the paper can be improved.\n\nA2.1: Thanks! We will refine the presentation of this paper in the revised version, including the equation and the standard deviation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method VIBE is effective. It outperforms the state-of-the-art continued pretraining methods with a smaller amount of adaptive data. The paper consists of a comprehensive set of preliminary analyses, classification experiments, ablation studies, and a case study to show the effectiveness of VIBE. \n\n2. The authors conduct extensive theoretical derivation to obtain the information bottleneck regularization objectives. \n\n3. The paper is clearly written and easy to follow.", "reasons_to_reject": "1. The presentation of the paper can be improved.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes Variational Information Bottleneck for Evolutions (VIBE) for temporal adaptation in tweet classification tasks. VIBE models latent topic evolution by explicitly distinguishing past-exclusive, future-exclusive, and time-invariant latent topics via information bottleneck regularization. The authors further leverage multi-task learning to infer timestamps and class labels jointly to inject time-awareness senses into the classification training. The authors conduct experiments on three tweets classification datasets and demonstrate that VIBE outperforms state-of-the-art continued pertaining methods with a smaller amount of adaptive data. The authors also conduct comprehensive ablation studies to investigate the effectiveness of VIBE's different modules. A case study of changing stance towards Dr. Anthony Fauci is presented to demonstrate VIBE can effectively capture topic evolution.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 267, Equation (3), $q_{\\Phi_X}(z^s|t^x)$ -> $q_{\\Phi_X}(z^x|t^x)$.\n\nIn the tables, I would suggest using a better way to show the standard deviation. Although the authors stated in Table 2 that the order of magnitude for std. is 10−5, it is still confusing to see a classification accuracy like 0.8109±1.1388. "}}, {"review_id": "OmQ8SvLCU2", "reviewer": "Reviewer_vJ1w", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\nQ3.1: Why does this paper propose a complex solution?\n\nA3.1: Thanks! Our paper addresses the challenging research problem of the deteriorating model performance over time in real-world applications. This issue is more of a temporal adaptation task[3, 4, 5, 6] than a text classification task. Our model design is necessary as it is challenging to capture the rapid evolution of language features in social media environments as time progresses. Previous studies on domain adaptation and pretraining baselines all exhibited compromised performance when faced with changing environments, highlighting the urgent need for a more advanced framework. Additionally, we follow the previous temporal adaptation benchmark [3, 4] to experiment with various text classification tasks and examine the capability of NLU models to comprehend temporal dynamics. In the future, we plan to explore temporal adaptation in more complex scenarios, such as text generation.\n\nQ3.2: Can scaling up language models like GPT3.5 tackle the temporal influences on models? How about adapting GPT3.5 to the changing future?\n\nA3.2: Thanks! In recent papers [7, 8, 9], it has been shown that the scaled-up language models exhibit compromised performance on out-of-distribution data. This highlights the inferiority of directly using extra-large language models to solve the temporal adaptation problem, which is especially challenging when faced with rapid, synchronous analysis of breaking events. Updating LLM’s knowledge of temporal events may be prohibitively expensive via continued pretraining [10], and this challenge will only be exacerbated as models continue to scale up. Even the official publisher for GPT3.5 has not updated it since 2021, which further demonstrates the impracticality of adapting these models for researchers who have limited computational resources. \n\nWhile continued pretraining GPT3.5 for temporal adaptation is extremely expensive, we experimented on continued pretraining BERT. However, we observed that this approach still requires a significant amount of training samples and computational resources, yet it exhibits lower-than-expected performance (as shown in Table 3). In contrast, our proposed model VIBE offers a compelling alternative. It can adapt models to the future and achieve state-of-the-art performance at a much lower cost. With only 3% of the data used in continued pretraining, VIBE shows promising potential for addressing temporal adaptation challenges in the rapidly changing real world. For more interpretation, we will report the performance of extra-large language models in the revised version.\n\nQ3.3: Why adopt accuracy as the evaluation metric?\n\nA3.3: Thanks! For the evaluation metric, we follow previous work’s practice of utilizing prediction accuracy in evaluating model performance in the temporal setting, which directly reflects the model’s ability to adapt to changing data [11, 12]. To provide more insights, We will add the label distribution analysis in the revised version. \n\nReferences:\n\n[3] Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah A. Smith. 2022. Time Waits for No One! Analysis and Challenges of Temporal Misalignment. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5944–5958, Seattle, United States. Association for Computational Linguistics.\n\n[4] Paul Röttger and Janet Pierrehumbert. 2021. Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2400–2412, Punta Cana, Dominican Republic. Association for Computational Linguistics.\n\n[5] Mourao, Fernando, Leonardo Rocha, Renata Araújo, Thierson Couto, Marcos Gonçalves, and Wagner Meira Jr. \"Understanding temporal aspects in document classification.\" In Proceedings of the 2008 International Conference on Web Search and Data Mining, pp. 159-170. 2008.\n\n[6] Alkhalifa, Rabab, Elena Kochkina, and Arkaitz Zubiaga. \"Building for tomorrow: Assessing the temporal persistence of text classifiers.\" Information Processing & Management 60, no. 2 (2023): 103200.\n\n[7] Linyi Yang, Shuibai Zhang, Libo Qin, Yafu Li, Yidong Wang, Hanmeng Liu, Jindong Wang, Xing Xie, and Yue Zhang. 2023. GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-Distribution Generalization Perspective. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12731–12750, Toronto, Canada. Association for Computational Linguistics.\n\n[8] Wang, Jindong, H. U. Xixu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang et al. \"On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective.\" In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models. 2023.\n\n[9] Kocoń, Jan, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz et al. \"ChatGPT: Jack of all trades, master of none.\" Information Fusion (2023): 101861.\n\n[10] Bender, Emily M., Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. \"On the dangers of stochastic parrots: Can language models be too big?🦜.\" In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610-623. 2021.\n\n[11] Kasai, Jungo, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. \"RealTime QA: What's the Answer Right Now?.\" arXiv preprint arXiv:2207.13332 (2022).\n\n[12] Zaporojets, Klim, Lucie-Aimée Kaffee, Johannes Deleu, Thomas Demeester, Chris Develder, and Isabelle Augenstein. \"TempEL: Linking dynamically evolving and newly emerging entities.\" Advances in Neural Information Processing Systems 35 (2022): 1850-1866.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThis paper introduces an approach to address the problem of deteriorating text classification performance in dynamic real-world social media. This innovative perspective allows the model to adapt effectively to changing language features over time, improving performance.\n2.\tVIBE leverages two Information Bottleneck regularizers to distinguish past and future topics. Using these distinguished topics as adaptive features and incorporating unlabeled data from online streams, the model achieves significant performance gains in text classification tasks on Twitter data.\n3.\tThe paper presents substantial experiments conducted on real-world social media data for three different text classification tasks. The results demonstrate the superior performance of VIBE.", "reasons_to_reject": "1.\tThe model looks like a complex solution to a simple text classification task.\n2.\tDo the authors think that when the language model scale is large enough, it can ignore the impact of data temporal differences on classification performance? Why didn't the authors try to use large language models like GPT3.5 as their baseline models?\n3.\tHow are the labels distributed in each dataset? Why did the author only apply accuracy as an evaluation metric?", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces VIBE, a model designed to address the challenge of deteriorating text classification performance in dynamic real-world social media. VIBE utilizes two Information Bottleneck regularizers to distinguish past and future topics, enabling it to adapt to changing language features. Through multi-task training and leveraging unlabeled data from online streams, VIBE significantly outperforms previous state-of-the-art continued pretraining methods with 3% of the data in experiments on Twitter for three classification tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uAxeydZvH4", "reviewer": "Reviewer_7trR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments!\n\nQ4.1: Why choose Twitter datasets for experiments, how about more datasets like news and reports?\n\nA4.1: Thanks! This paper’s scope is tweet classification because Twitter data presents a unique challenge of data noisiness in temporally evolving environments. Twitter data contains millions of varying topics discussed by the public, which change very fast as time goes by, making it challenging for models to keep pace with the content evolution (as discussed by our paper in lines 449-475). The rapidly evolving Twitter data reflects a realistic scenario for temporal adaptation study. For further study, we will expand our research scope to more applications like news and reports in the future.\n\nQ4.2: The application scenarios should be included.\n\nA4.2: Thanks! Many real-world applications are challenged by the evolving language features in social media platforms. For example, as time passes, the hate speech detection classifier may fail to recognize some new cases of hate speech due to the outdated training data, our method can adapt the outdated hate speech detectors to the future, contributing to a higher detection accuracy of new hate speech. Additionally, our approach can be generalized to more scenarios like making accurate recommendations for users when facing changing environments. We will add more discussion on application scenarios in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They propose a new task which would be beneficial to researchers working on temporal information extraction and topic modeling.\n2. They have conducted extensive experiments on three datasets, which makes their conclusions solid and sound. Additionally, the detailed ablation study and case study also provide insights into the future research direction.", "reasons_to_reject": "1. All three datasets they used are tweets. It would be beneficial to conduct experiments on some formal text datasets, such as news and reports.\n2. The application scenarios should be included.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies temporal adaptation, where models trained on past data are tested in the future. To do so, they propose a new model, VIBE: Variational Information Bottleneck for Evolutions. Specifically, they first employ two Information Bottleneck (IB) regularizers to distinguish past and future topics. Then, the distinguished topics work as adaptive features via multi-task training with the timestamp and class label prediction. In adaptive learning, VIBE utilizes retrieved unlabeled data from Twitter streams created posterior to training data time.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the problem of temporal adaptation for tweet classification and introduces the Variational Information Bottleneck for Evolutions (VIBE) model. The paper explores temporal adaptation in tweet classification, which is a valuable perspective for handling dynamic social media data. 

Most reviewers find the paper well-written, technically solid, and empirically convincing (MQvq and vJ1w). Reviewers have recognized the use of Information Bottleneck regularizers to distinguish past and future topics as an innovative approach to adapting to changing language features over time. The extensive experiments conducted on real-world social media data, along with ablation studies and case studies, are commended for demonstrating the effectiveness of the VIBE model. The clear and well-structured presentation of the paper, along with its readability, is appreciated by reviewers.

Reviewer QHXk suggests that the paper's presentation could be improved without specifying particular areas for enhancement. Reviewer vJ1w raises questions about the complexity of the VIBE model for text classification and suggests exploring the impact of data temporal differences on classification performance, especially in the context of larger language models (like GPT 3.5). More information is also asked on labels in each dataset and the choice of evaluation metrics. Additionally, conducting experiments on formal text datasets, such as news and reports, in addition to tweets, will be helpful (7trR).

In summary, the paper constitutes an innovative approach, comprehensive experiments, and a clear presentation. There are no major concerns with the paper's technical contributions. As suggested by reviewers, authors should consider including LLMs as a baseline, temporal differences in performance, and experiments on different text sources.
Decision: Accept-Main