Simple Temporal Adaptation to Changing Label Sets: Hashtag Prediction via Dense KNN
Authors: Niloofar Mireshghallah
Nikolai Vogler
Junxian He
Omar Florez
Ahmed El-Kishky
Taylor Berg-Kirkpatrick
Venue: EMNLP 2023 Main (2023)
Keywords: Retrieval Based Models
Ethics
Deletion
Temporal Adaptation
non-parametric

Abstract: User-generated social media data is constantly changing as new trends influence online discussion and personal information is deleted due to privacy concerns.
However, traditional NLP models rely on fixed training datasets, which means they are unable to adapt to temporal change---both test distribution shift and deleted training data---without frequent, costly re-training.
In this paper, we study temporal adaptation through the task of longitudinal hashtag prediction and propose a non-parametric dense retrieval technique, which does not require re-training, as a simple but effective solution.
In experiments on a newly collected, publicly available, year-long Twitter dataset exhibiting temporal distribution shift, our method improves by 64% over the best static parametric baseline while avoiding costly gradient-based re-training.
Our approach is also particularly well-suited to dynamically deleted user data in line with data privacy laws, with negligible computational cost/performance loss.

Simple Temporal Adaptation to Changing Label Sets: Hashtag Prediction via Dense KNN
Niloofar Mireshghallah's Nikolai Vogler' Junxian He
Omar Florez Ahmed El-Kishky Taylor Berg-Kirkpatrick
'University of California San Diego CCarnegie Mellon University, Twitter, "LatinX in Al (fmireshg.nvoglee..tukl @ @ucsd.edu, jumxianh @cs.cmu.edu www.:reescarcemmmil.ccooom gmail.com, aclkishky @twitter.com
omarflorez.essarcch @
Abstract
Step 1: Encode/Index Datastores
Temponni Distrismti Shift
lser-generated social media data is constantly changing as new trends influence online dis- cussion and personal information is deleted due to privacy concerns. However, traditional NLP models rely on fixed training datasets, which means they are unable to adapt to tem- poral change- both test distribution shift and deleted training data frequent, costly re-training. In this paper, we study tempo- ral adaptation through the task of longitudi- nal hashtag prediction and propose non- parametric dense retrieval technique, which does not require re-training, as simple but effective solution. In experiments newly collected, publicly available, year-long Twit- ter dataset exhibiting temporal distribution shift, our method improves by 64% over the best static parametric baseline while avoid- ing costly gradient-based re-training. Our ap- proach is also particularly well-suited to dy- namically deleted user data in line with data privacy laws, with negligible computational cost/performance loss.
Month
allcudiat
IF Thaass
Non
Herddyy
Hedlayy
Month
Exooder
Step 2: Inference via KNN Dense Retrieval
New Tret Toues
tetch most resent datastore
Most Recent
Month Datasiore
quenyy
Hedlday
KNUN
compare
Encoder
encode
qyary
rerank
Figure I: An overview of the KNN Dense Retrieval model components. First, month's tweets are encoded into em- beddings and added to external, indexed datastore with their associated hashtags- process that could be automated in production setting. Second, test tweet is encoded and used query the most recent datastore, which contains the latest month's training data for temporal adaptation. The top-k tweets from this datastore are then retrieved and re-ranked.
Freotiuc-Pietro, 2020) identifies the advantages of continual exposure late-breaking training data. Given the ever-shifting nature of user-generated data, deployed models must be explicitly designed to (1) adapt to dynamic test distributions to pre- vent temporal performance degradation, and (2) abide by data privacy laws, such as GDPR, CCPA, etc. (Voigt and Von dem Bussche, 2017), that man- date prompt removal of user data.
1 Introduction
Distribution shift, particularly in the target labels classification tasks, presents a serious challenge for the deployment of NLP systems in real-world scenarios. The distribution of text data changes over time due to new language usage, events, or trends (Eisenstein et al., 2014; Ryskina et al., 2020; Jaidka al., 2018), which causes statically-trained models to become stale without further re-training on new data (Lazaridou et al., 2021; Dhingra et al., 2022; Luu et al., 2022). Temporal shift is most evi- dent in social media data, which we study through multi-label tweet-hashtag prediction, where tweets are classified into rapidly changing iser-generated trends. Recent temporal analysis of named entity recognition models for social media (Rijhwani and
In this paper, our main contribution is an empiri- cal analysis of a known, but understudied learning paradigm in the context of temporal adaptation: non-parann con-parametric classification via dense retrieval from datastore. By using K-ncarest neighbor (KNN) classifier in conjunction with static neu- ral text encoder and simple reranker (as depicted in Fig 1), we demonstrate that continual and au- tomatic updates of the training datastore facilitate quick adaptation and deletion. In contrast to con-
"Equal contribution.
Work done while at Twitter.

---

temporary neural paradigms, which require gradi- ent descent during fine-tuning, our approach re- quires no gradient updates-bott adaptation and retroactive deletion use minimal compute.
K-nearest neighbor (KNN) classifier with static neural text encoder and simple reranker, as depicted in Fig. I, we enable simple but effective updating of an external datastore when train/test data changes, in contrast to full re-training of parametric models. Our model first retrieves the top-K nearest hash- tag labels from a datastore based on L2 distance in neural embedding space,
In order to analyze our proposed temporal adap- tation method, we introduce new supervised dataset ($3) focused on hashtag prediction, multi- label classification task in the social media domain that exhibits label shift over time. Hashtag predic- tion, in which a system must predict the set of hash- tags to be assigned to a given input tweet, is partic- ularly amenable to our non con-parametric method due to the intrinsic, user-generated categorization. In other words, the latest supervised training data can be automatically scraped and updated periodically on-demand. We release the dataset as tweet IDs. In addition to comparing several text encoding strategies and re-ranking methods on the dataset ($5), we compare against two state-of-the-ar para- metric baselines based on BART (Lewis et al., 2020), large pre-trained requence-to-sequenee model. These variants include a fine-tuned BART basede classifier, along with the full BART seq2seq model fine-tuned on tweet-hashaag pairs. We find that our best non-oornnnnd parametric approach out- performs static parametric models with an aver- age relative gain of 64% recall under test distri- bution shift-and, even outperforms conventional gradient-based lemporal adaptation of parametric baselines with an average relative gain of 12% recall. Together, our empirical analyses high- light non- parameetiri techniques as a practical and promising direction for handling distribution shift and user-deletion, and may facilitate future work on real-world, temporal NLP system deployment.
Sy(maa) = {211 91 .n),.... (Tk: 3k)] argmin top-k || zall2
(I)
(=a.aa)
where 24 is a neural encoding of a test/query tweet and zy is the neural encoding of a train tweet from train datastore D. Both z. and ya, which represents one (of the potentially many) associated tweet's hashtag labels, are stored "unrolled" in D, such that tweets with h. hashtags occur h times with only one hashtag per entry. After obtaining the top-K K tweet-hashter pairs (xz), we add final re-ranking step to return final hashtag predictions Dr, where << k:
D, {u : Dr} re-rrnk(S& (2)
Based on this, our dense KNN model has three main components: (1) A static neural tweet en- coder, such as the fine-tuned BART model (Lewis et al., 2020) that we use, which is only trained once and does not require updates, (2) a datastore that enables fast nearest neighbor search, and (3) a re- ranker to boost recall performance. We explain components (2) and (3) in more detail next.
We follow prior work (Khandelwal et al., 2020, 2021) and use FAISS (Johnson et al., 2019), which enables effi- cient indexing, clustering, and approximate nearest neighbor similarity scarch for dense vectors using quantization, which is faster than exact search, but does not precisely correspond to exact L.2 distance. See S 4.1 1. for implementation details.
Datastore and Search.
2 Dense KNN Hashtag Prediction Model
Language in user-generated social media data con- stantly changes, which makes the prediction of new trend categories, or hashtags, challenging for NLP models. Fundamentally, hashtag prediction is multi-label classification problem, in which a set of hashtags must be suggested for a single tweet. As new events influence online discussion, the clas- sification labels change drastically: roughly half of hashtag types in our dataset are newly replaced after 4-5 months (see details in App. Fig. 3).
Re-ranking Retrieved Hashtags. For optimal performance, we explore several methods for re- ranking the top- K retrieved tweets to return the "unique" top-R (R << K) hashtags, after remov- ing repetitions. First, default distance ranking involves re-ranking based on tweet embedding L.2 distance returned by FAISS, which are approxi- mate as FAISS quantizes vectors for search ef- ficiency. Second, our actual distance ranking method uses the actual distance between the en- coded query tweet and its retrieved neighbors. Third, frequency-based ranking prioritizes the
For this task, we consider known, but under- studied learning paradigm in the context of tem- poral adaptation: non non-ponnan -orammemmtrric classification via dense retrieval from a datastore, By combining a

---

R hashtags with the greatest support in the initial retrieved K. We count the number of occurrences of each hashtag in the top- retrieved tags, and then rank them from most repeated to least, and return the top-R most common ones.
conditional probability of the tokenized sequence of concatenated hashtags, given the tweet text as input. During test time, we force the model to gen- erate 120 tokens, and then select the first 5 hashtags for calculating recall independent of their order.
For our model's s neural encoder for KNN, we use the same BART architecture as tweet text encoder. In this setup, we use the neural classifier fine-tuned once on the initial 3-week training set for hashtag classification, and then freeze and re-use for encoding every subsequent train/test week in the year. That is, instead of continuously re- training our neural encoder for KNN, we simply re- use it and swap out the encoded datastores during updating. We describe other encoder variants and their results in $5.
3 Longitudinal Hashtag Dataset
Since we require a dataset with fine-grained tem- poral annotations and aim to study longitudinal effects of temporal distribution shift, we collect a new large-scale benchmark dataset for temporal hashtag prediction on Twitter. We scrape tweets from the entire 2021 calendar year organized by week published, keeping only tweet-hashtags in the top-10k 10K most frequent hashtags that week. We observe large label distribution shift over time, as roughly 50% of labels are new after 4-5 months (for detail, see Fig. 3).
4.1 Training Details
For the neural classifier experiments, we train each classifier for 30 epochs, and choose the best check- point based on validation recall. We use learning rate of 3e-5 with a polynomial scheduler and train- ing batch size of 36 for both the neural classifier and the seq2seq models. For the Tw/oA setting in- troduced in Section 5, we train one model each for classífier/seeq/sq. For the classifier, this results in fixed label set size of 16,886 for all test sets. For the Tw/A A setting, we train 12 different models on each of the 3-week alternating train splits, testing on each of the 1-week future test sets with the most recent models.
For temporal evaluation, we divide our dataset into alternating 3-week train/l-week test splits (nearly, but not exactly aligned with month bound- aries), such that the test data is always from future date. We chose to divide our dataset into weeks be- cause we (a) needed a fine-grained enough division to investigate how model performance decays over time, yet (b) enough data points in each division for supervised training/fine-tuningg of high-capacity neural baselines for comparison. Of course, these two decisions are at odds with each other since very fine- line-grained divisions will reduce the amount of training data in each division. Finally, (c) desired a temporal setup that would be practical for real-world use (i.e. avoid excessively frequent updates). For comparison, we also consider the setting with no temporal shift, where each week contains its own train/test splits and all of the 3- week test splits are aggregated for evaluation. To facilitate models like ours that require frozen his- torical training corpus (e.g. to train our static tweet encoder), we reserve the first 3-week time split.
For the datastores, we encode all tweets into an N E memory maps, where N is the number of unrolled tweets ($2) in the training set and E is the embedding size (F = 1024). Separately, we construct an N x V memory map for hashtags, where V 280 is the max allowable length of tokens. We use the FAISS IndexFlatIP quantizer with 7.2 as the distance metric. For retrieval, we use their search function as well, which retrieves the nearest K neighbors but using the approximate (quantized) distances which means the retrieval and order is not exact. We also experimented with Cosine distance and found it to under-perform com- pared to L2 in our case. We use K = 1204 for our experiments (see App. Table 3).
4 Experimental Setup
For the parametric neural classifier, we use BART- large (Lewis et al., 2020) with multi-label classifi- cation head. To explore if performance degradation over time is simply due to fixed label set, we pro- vide another parametric model, neural seq2seq baseline based on BART with conditional genera- tion head, which can generate hashtags unseen dur- ing training. The model is trained maximize the
5 Results
Our experimental results consist of performance comparisons between (1) different models and tem- poral settings and (2) user-deletion. We present model ablations in Appendix A.
Code/data (tweet IDs) available here: https://githhbb hwwiniiresgghlllhhteenngeeo alnn

---

Temporal Adaptation. We perform experiments with three different temporal dataset ($3) settings: (1) Non-temporal (NT), where there no tempo- ral distribution shift between train and test; each week contains its own train/test splits and the 3- week test splits that correspond with each 3-week train split are aggregated for evaluation. (2) Tem- poral w/o Adaptation (TwooA) where model's training data consists of only the first 3-weeks of the year and its test data is from every later test week (weeks 4, 8, ..... 48). (3) Temporal w/ Adaptation (Tw/A), where the model is re-trained throughout the year on the 3-week split immedi- ately before the given test week (e.g. for test week 8, the model is trained on weeks 5-7).
model over time. To understand why our approach does not suffer performance degradation despite using a tweet encoder trained on stale data, we can compare our "Dense KNN Tw/A" results in Fig 2 (b) with the "Classifier Tw/oA" setting. As a re- minder, these two methods share the same underly- ing BART encoder model (except for the classifica- tion head), which is trained on the first three-weck training dataset. While the neural encoder stays the same for "Dense KNN Tw/A" and "Classifier Tw/oA" settings, the "Dense KNN Tw/A' model gets updated with the temporally closest datastores over time, while the "Classifier Tw/oA" must only rely on the stale neural encoder/classifier head. If we contrast "Dense KNN Tw/A"'s purple solid line in Fig 2 (b) with the "Classifier Tw/oA"'s orange dashed line, we observe sharp performance drop for the stale classifier. However, performance stays relatively constant over time for the "Dense KNNN Tw/A" model despite having the same encoder as the stale classifier. We understand this to show that the learned representations for comparing tweets, the component shared by both methods, are not negatively impacted by the passage of time, but it is instead the classification head that becomes outdated tions. Since "Dense KNN Tw/A" is able to see the most recent, empirical label set distribution when we update the datastore over time, this would explain its high performance. These findings are further supported by the increase in performance from the "Classifier Tw/oA" (dashed orange line) to "Classifier Tw/A" (solid orange line) settings, the "Lifetime of Hashtag" label set overlap over time (App- Fig 3), and App. Fig 4, in which we show the yearlong hashtag prediction performance breakdown for the best temporally adapted settings.
In Table (a), we summarize results across these 3 settings. In terms of comparing the two paramet- ric baselines, we can see that in the temporal setups (Tw/oA, Tw/A) the Seq2Seq model outperforms the Classifier, potentially due to its capability of generating unseen hashtags, unlike the Classifier's fixed training label set. The Classifier is able to per- form well exclusively for the NT evaluation, which does not exhibit any distribution shift, and there- fore is not representative setting for production. Dense KNN outperforms Seq2Seq and Classifier for both temporal sctups, which shows how sim- ple, yet robust non parametric models are under distribution shift.
result of changing label set distribu-
In Fig 2 (b), we compare year-long performance degradation of Tw/oA & Tw/A models for a bet- ter understanding of the effects of temporal shift. As the test week progresses further along in the year, we see a marked decrease in performance (as much as a 34% relative decrease) for Tw/oA mod- els (dashed line), which is indicative of major dis- tribution shift between the old data the model was trained on the new test data. Alteraatiely,, the solid lines, representing the continuously reairrined models with the Tw/A setting, tend to maintain their performance much better. Our dense KNN model performs the best across time, despite being the simplest and most efficient model to update since it requires no mrudient-based training.
Deletion Adaptation. As explained in $1, efficient model adaptation facilitates selective deletion of training data, which can be an important feature for removing harmful training examples or obeying user deletion requests. For nonpparam meth- ods like dense KNN, deletion involves straightfor- ward removal of the given tweet from the datas- tore, followed by re-indexing operation, which takes minutes on CPU. The parametric classifier, however, needs be fully re-trained without the deleted tweets, which takes about week on our dataset with single A A100 GPU. In Fig 2 (c), we compare our dense KNN with training data from weeks 9-11 and test data from future weeks 12,
Does distribution shift affect the tweet encoder? While our neural tweet encoder- the component of our dense KNN model that encodes tweets as keys for our datastores requiress historical data to learn how to embed tweets for nearest hashtag retrieval, results in Fig 2 (b) show that this older data has no impact on the performance our full "Dense KNN"

---

24
Dadeet
Temporal
29
Non: Temporal
weo Adapt
Adape
500
Method
65
61
@5
65
@1
Cass Daros kie
Recel.g
-0
200
MCTiriin
0.5
18
Classifier
39.9
14.8
14,8
5.4
23.3
8.3
Tust
Seg2Seg
13.0
23.5
16
Dense KNN 39.5
13.2
18.5
6.7"
26.2"
9.67
14
14
12
Figure 2: (a, left) Comparing average test recall (1, 5) performance over the year. "Most Common Train" (MC Train) uses the most frequent hashtag label from the corresponding training set for cach setting. For instance, "Non-temporal" and "Temporal Adaptation" settings use the most common hashtag label from three weeks right before test week, unlike "Temporal w/o Adaptation", which uses the most common hashtag label from the first three weeks the year. denotes statistical significance via paired bootstrap test (Koehn, 2004). (b, center) Comparison of our KNN approach with the neural seq2seq and classifier models. Dashed lines show evaluation results a model trained/datastore created on only time bucket (first weeks). Solid lines show the performance adapted/re-train deletion (thicker line means higher deletion percentage) under Temporal w/o Adaptation (Tw/oA) setting, We have plotted the optimal classitier, without any deletion, as its the upperbound of classifier performance.
models on the most recent data. (c, right) Performance effects training data
16, 20, .... 18 under 4 different training dataset deletion percentages: 1.7% 20%, 50% and 80%. The 1.7% scenario represents the real-world user- deleted data percentage (due to account suspension, user deletion, or content violations) that is obtained after re-seraping the tweets 5 months after initial collection, while other percentages are chosen for interest and deleted randomly. We compare the behavior of dense KNN with updated datastores with that of the parametric classifier without any deletion in Fig 2 (c). We observe that the KNN model decays gracefully as more data is deleted, and even with the deletion, it still outperforms the upper bound classifier model.
et al., 2022: Loureiro et al., 2022) as their re- training cost continues to grow. While in-context learning enables updating model posteriors without gradient updates, static models are still unable to update their temporal world knowledge (Akyürek et al., 2022; Xie et al., 2021). The temporal distri- bution shift problem on Twitter has been studied in Preetr and Cohn (2013); Rijhwani and Preotiuc-Piette (2020); Luu et al. (2022); Kowald et al. (2017); Kamath and Caverlee (2013).
User Data Deletion. While several methods have been proposed to partially mitigate compu- tational costs of unlearning through techniques to speed up re-training, substantial compute is still required (Bourtoule et al., 2021; Wu et al., 2020). Decremental learning (Cauwenberghs and Poggio. 2000; Karasuyama and Takeuchi, 2009) has been proposed for removing specific training samples. Different from previous methods, we explore sim- ple, theoretically guaranteed, and effective way to delete user data by modifying the datastore.
6 Related Work
Hashtag Recommendation. Earlier works on hashtag recommendation compute tf-idf vectors from extracted keywords and apply classifiers on these vectors (Jeon al., 2014; Sedhai and Sun, 2014), while late works incorporate neural nets (Li al., 2016; Shen al., 2019; Gong and Zhang, 2016; Ma et al., 2019). Feng and Wang (2014) collected a dataset of millions of tweets/hashtags across two months of 2012 and designed a personal hashtag recommendation model using user meta- data. Instead, we release a public dataset of tweets over an entire year, show how continually update a non-parametric datastore handle temporal shift. Another line of research models semantic word shift as one form of temporal change (Wijaya and Yeniterzi, 2011; Kulkarni et al., 2015; Hamilton et al., 2016; Kutuzov et al., 2018) Temporal Adaptation. More recently, the tempo- ral generalization problem has been re-emphasized in pretrained language models (Lazaridou et al., 2021; Dhingra et al., 2022; Luu et al., 2022; Jin
7 Conclusion
In this paper. we show that a non-parametric dense KNN retriever model is particularly well-suited for the task of temporal adaptation as it can effi- ciently adapt to changing test distributions over time by easily updating and swapping out datas- tores. We find improves by 64% over the best static parametric baselines while avoiding their costly gradient-based re- re-training. Our model also performs well under selective deletion of training data, which is an important feature for removing harmful training examples, or obeying user dele- tion requests in line with data privacy laws, such as GDPR and CCPA. We hope the release of our dataset will encourage more work on temporal model adaptation in the future.

---

Limitations
Wei Feng and Jianyong Wang. 2014. We can learn your# hashtags: Connecting tweets to explicit topics. In 2014 IEFF 30th International Conference on Data Engineering, pages 856-867. IEEE
Our proposed dense retrieval method relies on a collection of "historic data" which is used to train an encoding/embedding model, to produce the rep- resentations that would be used as keys in the data- store. As the KNN method relies on these represen- tations for finding neighbors, it is crucial to train the embeddings on in-domain historic data, that would also be unlikely to receive deletion requests, However, as we show our experiments, this data does not go stale for even a year out, and still per- forms well and provided appropriate embeddings, as the performance of the KNN method does not degrade with the passage of time (as seen in Fig 2 (b) and discussed in Section 5).
Yuyun Gong and Qi Zhang. 2016. Hashtag recommen- dation using attention based convolutional neural net- work. In Proceedings of the Twenty-Fifth Interna- tional Joint Conference on Artificial Intelligence, IJ- CAI 2016, New York, NY. USA, 9-15 July 2016, pages 2782-2788. IJCAT/AAI Press.
William I. Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic word embeddings reveal statisti- cal laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Compu- rational Linguistics (Volume I: Long Papers), pages 1489-1501 1501, Berlin, Germany, Association for Com- putational Linguistics.
Kokil Jaidka, Niyati Chhaya, and Lyle Ungar. 2018. Di- achronic degradation of language models: Insights from social media. In Proceedings of the 56th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 195-200, Melbourne, Australia. Association for Computational Linguisties.
Ethics Statement
We abide by the Twitter development and data usage agreement (https://develog: twitter. twwtter.com//ttt comp////ee/deeeelppeeeeeessss agreement-and-pollen
for the collection and usage of the Twitter data. Also, upon release of the dataset, we will only make the tweet IDs available for data re-hydration (and not the user ID or the tweet body), to protect the privacy of the users, which is inline with the overall goal of this paper as well.
Mina Jeon, Sanghoon Jun, and Eenjun Hwang. 2014. Hashtag recommendation based on user tweet and hashtag classification on twitter. In International Conference on Web-Age Information Management. pages 325-336 Springer.
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-War Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2022. Lifelong pretraining: Continually adapting language models to emerging corpora. In Proceedings of BigScience Episode #5- Workshop on Challenges & Perspectives in Creating Large Lan- guage Models, pages 1-16, virtual+Dublin. Associa- tion for Computational Linguistics.
References
Ekin Akyüreek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2022. What learning algorithm is in-context learning? investigations with linear models. arXiv preprint arXiv:2211.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data. 73)55555-547.
Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Ma- chine unlearning. 2021 IEEE Symposium on Secu- rity and Privacy (SP), pages 141-159. IEEE
Krishna Y Kamath and James Caverlee. 2013. Spatio- temporal meme prediction: learning what hashtags will be popular where. Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 1341-1350.
Gert Cauwenberghs and Tomaso A. Poggio. 2000. In- cremental and decremental support vector machine learning. In Advances in Neural Information Pro- cessing Systems 13, Papers from Neural Information Processing Systems (NIPS) 2000, Denver, CO, USA, pages 415- MIT Press.
Masayuki Karasuyama and Ichiro Takeuchi. 2009. Mul- tiple incremental decremental learning of support vector machines. In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems 2009. Pro- ceedings of meeting held 7-10 December 2009, Van- couver, British Columbia, Canada, pages 907-915. Curran Associates, Inc.
Bhuwan Dhingra, Jeremy Cole, Julian Fisenschlos, Daniel Gillick, Jacob Eisenstein, and William Co- hen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257-273.
Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2021. Nearest neigh- bor machine translation. In 9th International Confer- ence on Learning Representations, ICLR 2021, Vir- tual Event, Austria, May 3-7, 2021. OpenReview.net.
Jacob Eisenstein, Brendan O' O'Conoo, Noah A Smith, and Eric Xing. 2014. Diffusion of lexical change in social media. PloS one, 9(1 11:e1111144 13114.

---

Urvashi Khandelwal, Levy, Dan Jurafsky, Luke Zeytlemoyer, and Mike Lewis. 2020. Generalization through memorization: Nearest neighbor language models. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net.
waits for no one! analysis and challenges of tem- poral misalignment. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5944-5958, Seattle, United States. Association for Computational Lin- guistics.
Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 388-39 395, Barcelona, Spain, Association for Computational Linguistics.
Renfeng Ma, Xipeng Qiu, Qi Zhang, Xiangkun Hu, Yu- Gang Jiang, and Xuanjing Huang. 2019. Co-attention memory network for multimodal microblog's hashtag recommendation. IEÉE Transactions on Knowledge and Data Engineering, 33(2):388-400.
Dominik Kowald, Subhash Chandra Pujari, and Elisa- beth Lex. 2017. Temporal effects on hashtag reuse twitter: cognitive-inspired hashtag recommen- dation approach. In Proceedings of the 26th Interna- tional Conference World Wide Web, pages 1401- 1410.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. weRTweet: A pre-trained language model for English tweets. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9-14, On- line. Association for Computational Linguistics.
Vivek Kulkarni, Rami AI-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detec- tion of linguistic change. In Proceedings of the 24th International Conference on World Wide Web, WWW 2015, Florence, Italy, May 18-22, 2015, pages 625- 635. ACM
Daniel Treotiue-Piette and Trevor Cohn. 2013. A tem- poral model text periodicities using Gaussian pro- cesses. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 977-988, Seattle, Washington, USA. Associa- tion for Computational Linguistics.
Andrey Kutuzov, Lilja Øvrelid, Terrence Szymanski, and Erik Velldal. 2018. Diachronic word embeddings and semantic shifts: a survey. In Proceedings of the 27th International Conference on Computational Lin- guistics, pages 1 1384-1397 Santa Fe, New Mexico, USA. Association for Computational Linguisties.
Shruti Rijhwani and Daniel ProotuucPieerro 2020. Temporally-informes analysis of named entity recog- nition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7605-7617, Online. Association for Computational Linguisties.
Angeliki Lazaridou, Adhi Kuncoro, Flena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez. Cyprien de Masson d' Autume, Tomas Ko- cisky, Sebastian Ruder, et al. 2021. Mind the gap: Assessing temporal generalization in neural language models. Advances in Neural Information Processing Systems, 24-29348-29363.
Maria Ryskina, Ella Rabinovich, Taylor Berg- Kirkpatrick, David R Mortensen, and Yulia Tsvetkov. 2020. Where new words are born: Distributional semantic analysis of neologisms and their semantic neighborhoods. arXiv preprint arXiv:2001.077
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequencc pre-training for natural language generation, translation, and com- prehension. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Surendra Sedhai and Aixin Sun. 2014. Hashtag rec- ommendation for hyperlinked tweets. In The 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR "14 Gold Coast, OLD, Australia - July 06 11, 2014. pages 831-834, ACM.
Yatian Shen, Yan Li, Jun Sun, Wenke Ding, Xianjin Shi, Lei Zhang, Xiajiong Shen, and Jing He. 2019. Hashtag recommendation using Istm networks with self-attention. Comput. Mater. Continua, 61(3):1261- 1269.
Jia Li, Hua Xu, Xingwei Hc, Junhui Deng, and Xiaomin Sun. 2016. Tweet modeling with 1stm recurrent neu- ral networks for hashtag recommendation. In 2016 International Joint Conference on Neural Networks (IJCNN), pages 1570-1577 IEEE
Paul Voigt and Axel Von dem Bussche. 2017. The eu general data protection regulation (gdpr). A Prac- tical Guide, 1st Ed., Cham: Springer International Publishing, 103152676;:00-5555
Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Samacho-Colladee 2022. Timelms: Diachromic language models from twitter. arXiv preprini 221.2200.032229.
Derry Tanti Wijaya and Reyyan Yeniterzi. 2011. Un- derstanding semantic change of words over centuries. In Proceedings of the 2011 international workshop on DETecting and Exploiting Cultural diversiTy on the social web, pages 45-
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Kar- ishma Mandyam, and Noah A. Smith. 2022. Time

---

Yinjun Wu, Edgar Dobriban, and Susan B. Davidson. 2020. Deltagrad: Rapid retraining of machine learn- ing models. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 10355-10366. PMIR.
Sang Michael Xie. Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-coneextt learn- ing implicit bayesian inference. arXiv preprint ar..v22111.02000.

---

Lifetime of Hashtag
1.0
Temporal
Jan
Non-temporal
Wio Adaptacion
W/ Adaptation
Feb
655
011
615
611
415
Mar
0.9
MC Train
1.69
0.46
1.11
0.26
1.68
0.47
KNN-Soq
35.21
10.35
3.32
May
KNN- Bertweet
33.07
11:00
7.55
2.30
15:46
4.59
0.8
KNN-CIsf
39.54
13:15
18.45
6.68
26.21
9.60
Jun
Table 2: Ablation study the effect of different encoders encoding tweets the datastore and retrieving the nearest neighbors.
Aug
0.7
Sep
0.6
Nov
encoder could also require updates, although only across years as previously described.
Jan Feb Mar May Jun Jul Aug Sep Oct Nov Dec
Ablating K and re-ranking methods. In Ta- ble 3, we show ablation results for top-k and re- ranking averaged over the 12 test weeks. Over- all, frequency-based re-ranking outperforms the distance-basse methods by large margin, which we hypothesize is due to the robustness added by the repetitions in hashtags. We
Figure 3: On Twitter, hashtag usage exhibits significant tem- poral distribution shift, which is challenging for current NLP models. We show the hashtag lahel set overlap, computed as recall between each month's hashtags, for our longitudinal dataset of 7. 7.13M tweets over the 2021 calendar year. We note that the recall not symmetric because monthly hashtag have different sizes.
that for the distance-based methods, the overall trend is that higher K is better, due to fewer cascading er- rors (1024 optimal on average). The frequency- based re-ranking, however, degrades significantly the number of retrieved neighbors is large (1024 and 2048), which is expected, as more irrelevant but common hashtags are suggested. When K ap- proaches the datastore size, we approach random frequency- -based classifier.
Longitudinal Hashtag Dataset Statistics
Number of Train Tweets Per Week
475,437
Number of Val-Test Tweets per Week
59,430
Number of Avg Tags per Tweet
2.9
Number of Unique Tags per Week
10,000
Avg Hashtag Length (tokens)
3.2
Avg Tweet Length (tokens)
30.3
Table I:
Summary statistics of new longitudinal dataset.
OOV tag prediction performance. Finally, we want to investigate how updating the datastore helps capture new, out-of- cit-of-voccbulary (OOV) hashtags, that would not be predicted if we kept ing the datastore from time bucket 1. Fig 5 shows the results for this experiment, where the OOV refers to out-of-vocabulary with respect to bucket I's hashtag vocabulary. IV refers in-vocabulary hashtags, which means the tags that appear both in the given test week, and the train data of time bucket 1. We report the recall over the IV and OOV tags separately. We can see that the updates in the datastores help predict 19% of the hashtags that would otherwise not be predicted, on average across the test weeks. We see that as we proceed with time buckets, the OOV recall grows, eventu- ally overtaking the IV recall. It is worth noting that the number of IV tags is substantially smaller in later time buckets. We suspect the superior OOV performance the later time buckets is related to content shift, i.e., the meanings of IV tags may have shifted by the later time buckets.
A Ablation Studies
In this section, we perform an ablation of several model components, including the encoding of the tweets for building the datastore, the K in KNN, and the re-ranking of the retrieved K nearest neigh- bors. Finally, we break-down the recall @5 results of the KNN over different time buckets, to see how well the updated datastore helps capture out-of- vocabulary tags that a datastorc/model from the first time bucket would not have captured.
Ablating encoder for datastore. Apart from classifier encoder used in all previous experiments, we also tried using both the seq2seq model as the encoder, and a generic encoder trained on tweets, named Bertweet (Nguyen et al., 2020), compared in Table 2. The classifier encoder performs the best, followed by the seq2seq, and Bertweet. One poten- tial explanation for Bertweet's poor performance is that training set, which consists of tweets from 2012 to 2019, is outdated, creating a sigmificant dis- tribution mismatch. This would hint that the KNN

---

Temperal
Prediction Performance Degradation Seq25eq Baseline (Recall 5)
Temporal Hashtag Prediction Performance Degradation 2021
KMi--Cis (Recall 5)
Ed2
SG
1:22
5.24
Ed44
2:14
6:44
6-22
2:24
21
331
627
R25
224
0G4
3.1
022
022
3,28
R22
Train Detesiio Bucket
Tran Dataseo Bucket
9.25
3.
8
F2
22
222
324
3.00
635
655
0.2
3.15
021
0.34
0.1
2:35
631
6:5
121
9.27
Weak Bucket
Week Bucket
Figure 4: Evaluation matrix using all possible combinations of previous whree-week span train buckets (roughly corresponding to months) to predict hashtags for all possible combinations of test week buckets (occurring directly after each week--s train bucket throughout the year). We compare recall@5 results for the best performing seq2seq temporal baseline from Table 2 (a), and our Dense KNN model. We include using future months to predict past test weeks, although unrealistic in practice, for completeness.
Method / K
20
50
100
1024
2048
Frequency-basee
25.23
26.26
26.21
23.12
21.44
Default Dist.
24.00
24.09
24.32
24.27
24.21
Actual Dist.
24.22
24.50
24.77
24.97
24.92
Table 3: Effect of K on KNN retrieval and comparing dif- ferent re-ranking methods from $2. Recall @ is reported, averaged over the 12 test weeks the year under the Tw/A evaluation scheme.
24
IV
22
Recal 15
18
16
14
12
12
16
20
36
00
44
Testt Weekk
Figure 5: Studying the effect of updating the datastore on
capturing out-of-vocabularg (OOV) vs. in vocabullyry (IV) hashtags. OOV recall is reported with respect to the vocabu- lary from the initial three- week training set.


Reviews:
[{"review_id": "7Wigltrzph", "reviewer": "Reviewer_sW7e", "summary": "", "questions": "A. If a new Tweet significantly differs from historical Tweet data, or if the content of the historical data is unrelated to the retrieval content, how will the model rerank them and infer this situation?\n\nB. We suggest the authors provide reasons or explanations for the proposed approach, such as why they chose the KNN model over other models, dataset preparation, and illustrations for parameter settings, like top-k and top-R.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful feedback on our submission. We hope to answer your questions below.\n\n\n**A.** While our neural tweet encoder---the component of our dense KNN model that encodes tweets as keys for our datastores---requires historical data to learn how to embed tweets for nearest hashtag retrieval, results in Figure 2 show that this older data has no impact on the performance of our full “Dense KNN” model over time. To understand why our approach does not suffer performance degradation despite using a tweet encoder trained on stale data, we can compare our “Dense KNN Tw/A” results in Figure 2 with the “Classifier Tw/oA” setting. As a reminder, these two methods share the same underlying BART encoder model (except for the classification head), which is trained on the first three-week training dataset. While the neural encoder stays the same for “Dense KNN Tw/A” and “Classifier Tw/oA” settings, the “Dense KNN Tw/A” model gets updated with the temporally closest datastores over time, while the “Classifier Tw/oA” must only rely on the stale neural encoder/classifier head. If we contrast “Dense KNN Tw/A”’s purple solid line in Fig 2 (center) with the “Classifier Tw/oA”’s orange dashed line, we observe a sharp performance drop for the stale classifier. However, performance stays relatively constant over time for the “Dense KNN Tw/A” model despite having the same encoder as the stale classifier. We understand this to show that the learned representations for comparing tweets, the component shared by both methods, are not negatively impacted by the passage of time, but it is instead the classification head that becomes outdated as a result of changing label set distributions. Since “Dense KNN Tw/A” is able to see the most recent, empirical label set distribution when we update the datastore over time, this would explain its high performance. These findings are further supported by the increase in performance from the “Classifier Tw/oA” (dashed orange line) to “Classifier Tw/A” (solid orange line) settings, and the “Lifetime of a Hashtag” label set overlap over time presented in Figure 3.\n\n\n**B.** We initially choose the dense KNN model over other model classes for two main reasons: (1) it can efficiently adapt to changing test distributions over time by easily updating and swapping out datastores, and (2) it abides by data privacy laws, such as GDPR, CCPA, etc. by removing individual entries from the training data without costly gradient-based re-training of the model. We find that our results in Figure 2 support these desiderata. \n\n\nRegarding our dataset preparation decisions, we elaborate on our  choices in the rebuttal to Reviewer #3 in response 2. To summarize, for temporal division of the dataset, we decided on 3-week train sets/1-week test sets to balance (a) how fine-grained the divisions could be for detailed temporal analysis with (b) the need for large enough divisions to train supervised neural models. We also required (c) a realistic setup for an industrial hashtag recommendation engine that would not require excessively frequent updates. \nFinally, our parameter settings for top-*K* and top-*R* were determined through grid search on a separate validation set. We show the effect of *K* on different tweet encoders in the Appendix Table 3. We will expand on this and also bring the “Training Details” from the Appendix into the main paper with the extra page afforded to us in the camera ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Critical issue: social-media data is continually increasing, changing, and shifting, making the cost of annotating or maintaining such data over time relatively expensive. Most current research explores the technology of supervised learning models, but the cost of labeling datasets and retraining models needs to be considered.\n\nNovel ideas: this paper applies the dense KNN retrieval model to reduce the cost of model retraining and maintain the model's adaptability based on shifts in time distribution. \nThe proposed method appears to be easy to implement and innovative, showing good performance in the Twitter dataset. We believe that such research, which can reduce annotation costs and reduce the need for retraining models, is worthy of further study for exploration.", "reasons_to_reject": "Perhaps due to the limitation of a short paper, the authors did not provide sufficient explanations for the proposed method, such as the reason for choosing the KNN model, the parameters setting of top-k and top-R, etc. But we think these are minor weaknesses that can be revised.", "questions_to_authors": "A. If a new Tweet significantly differs from historical Tweet data, or if the content of the historical data is unrelated to the retrieval content, how will the model rerank them and infer this situation?\n\nB. We suggest the authors provide reasons or explanations for the proposed approach, such as why they chose the KNN model over other models, dataset preparation, and illustrations for parameter settings, like top-k and top-R.", "paper_topic_and_main_contributions": "Challenges: the high cost of maintaining training data for model retraining over time. User-generated social-media data is influenced by new trends that shape online discussions and altered by deleting personal information for privacy considerations. Traditional NLP models rely on fixed training datasets and cannot adapt to time variations. Most of the current research explores the adaptation of traditional fields and requires retraining models.\n\nContributions: The authors investigate temporal adaptation through a task of longitudinal hashtag prediction, introducing a dense non-parametric retrieval technique that eliminates the need for retraining. This method improves by 64% recall compared to previous methods and avoids the cost of retraining models.\n\nContribution types: approaches for data- and compute efficiency.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7p67lJCVnQ", "reviewer": "Reviewer_aHrb", "summary": "", "questions": "A. What does the “MC Train” in Figure 2 mean?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful feedback on our submission. We hope to address your questions and concerns below. In summary, we’ve fixed the error you pointed out in our ablation table (we had accidentally swapped two columns, which led to the strange results you pointed out), added a full temporal matrix of train/test results for our best method vs. the best baseline method, and performed significance testing to measure the statistical reliability of our results.\n\n\n**Reasons to Reject:**\n\n\n**1.** Thanks so much for noticing this discrepancy in the encoder ablation results in Table 2. We checked our results and found that this was actually a data entry typo due to transferring our ablation results from the model output into the table. Specifically, the recall@1 columns for “W/o Adaptation” and “W/ Adaptation” are actually swapped for the KNN-Seq row and the KNN-Bertweet row. While double-checking this ablation table against the main results table in Figure 2 (left), we also found that the “Frequency baseline” result in Table 2 had its numbers swapped as well. In particular, both the recall@5 and recall@1 columns were swapped for the “W/o Adaptation” and “W/ Adaptation” settings. Here’s the updated table formatted in Markdown with these **fixes marked in bold**:\n\n| Method | NT R@5 | NT R@1 | Tw/oA R@5 | Two/A R@1 | Tw/A R@5 | Tw/A R@1 |\n| :--- | :---: | :---: |  :---: | :---: | :---: | :---: |\n| Frequency baseline | 1.69 | 0.46 | **1.11** | **0.26** | **1.68** | **0.47** |\n| KNN-Seq | 35.21 | 11.83 | 10.35 | **3.32** |19.64 | **5.88** |\n| KNN-Bertweet | 33.07 | 11.00 | 7.55 | **2.30** | 15.46 | **4.59** |\n| KNN-Clsf | 39.54 | 13.15 | 18.45 | 6.65 | 26.21 | 9.60 |\n\n\nAfter fixing this data entry error, the recall@1 numbers now have better agreement with the recall@5 settings across each row. Also, the “Frequency baseline” is correct now, since it’s actually the same baseline as the “MC Train” frequency baseline from Figure 2 (left) but with an extra digit of precision. We would like to emphasize that these numbers are from our additional experimental ablation table added for completeness in the Appendix Table 2 and not from our main results in the main paper (located in Figure 2 (left)). We apologize for the confusion, and will release all scripts to verify these results along with the other code/data.\n\n\n**2.** Thanks for your suggestion, we think a full evaluation result matrix would be a great addition to the paper. For this rebuttal, we performed an exhaustive evaluation setup using all possible combinations of previous three-week-span train buckets (roughly corresponding to months) to predict hashtags for all possible combinations of future test week buckets (occurring directly after each three-week-span train bucket throughout the year). We compare recall@5 results for the best performing temporal baseline from Figure 2 (Seq2seq), and our Dense KNN model in separate 12x12 Markdown tables below:\n\n**Seq2seq w/ Adaptation Baseline Results**\n\n\n|Train Datastore Bucket/Test Week Bucket|  4  |  8  |  12  |  16  |  20  |  24  |  28  |  32  |  36  | 40  | 44  | 48  |\n|------------------------------:|----:|----:|----:|----:|----:|----:|----:|----:|----:|----:|----:|----:|\n|                              **1**|18.7|16.9|16.7|14.4|17.1|16.8|14.4|15.3|13.9|14.8|15.4|14.3|\n|                              **2**||21.0|20.3|15.6|17.1|16.8|13.2|17.6|14.6|13.0|15.4|12.8|\n|                              **3**|||25.3|16.8|19.1|18.4|14.0|18.6|16.6|15.0|16.7|13.8|\n|                              **4**||||21.7|20.9|19.8|18.2|15.8|17.0|14.1|14.9|13.5|\n|                              **5**|||||23.9|20.1|16.3|17.8|17.4|17.9|17.8|17.4|\n|                              **6**||||||22.7|17.8|17.1|17.9|15.9|17.6|16.0|\n|                              **7**|||||||24.1|19.1|19.7|15.3|18.1|16.4|\n|                              **8**||||||||22.4|22.9|17.5|19.3|17.0|\n|                              **9**|||||||||26.9|19.0|21.0|17.9|\n|                             **10**||||||||||21.2|23.2|18.6|\n|                             **11**|||||||||||27.8|21.1|\n|                             **12**||||||||||||26.4|\n\n\n\n**Dense KNN w/ Adaptation Results**\n\n\n|Train Datastore Bucket/Test Week Bucket|  4  |  8  |  12  |  16  |  20  |  24  |  28  |  32  |  36  | 40  | 44  | 48  |\n|------------------------------:|----:|----:|----:|----:|----:|----:|----:|----:|----:|----:|----:|----:|\n|                              **1**|24.7|21.0|19.9|18.3|16.1|15.5|14.0|13.5|14.5|12.6|12.0|13.1|\n|                              **2**||24.4|21.2|17.2|16.9|15.3|14.9|14.5|13.9|12.3|12.3|12.5|\n|                              **3**|||25.4|20.6|16.8|17.6|15.9|15.1|14.8|13.2|12.1|14.1|\n|                              **4**||||25.0|19.8|17.7|15.8|15.5|15.2|13.3|13.1|13.5|\n|                              **5**|||||24.1|19.2|16.5|14.9|16.3|13.6|14.2|13.6|\n|                              **6**||||||23.5|18.7|17.2|16.4|14.9|15.1|15.1|\n|                              **7**|||||||25.4|19.7|18.9|15.1|15.3|15.5|\n|                              **8**||||||||24.1|20.9|16.8|17.0|15.8|\n|                              **9**|||||||||26.3|18.3|18.4|17.2|\n|                             **10**||||||||||24.1|19.0|17.8|\n|                             **11**|||||||||||25.3|20.8|\n|                             **12**||||||||||||26.8|\n\n\nIn these results, we can get a better understanding of how both methods generalize to longer time horizons. We plan to include all results in this format in future revisions.\n\n**3.** Thank you for pointing this out – we agree that significance testing is needed in order to establish that these results are generalizable. In future revisions, we will add confidence intervals for systems, and paired tests for the most important comparisons. For time and space during rebuttal, we report the results of a paired significance test for the most critical comparison: the comparison of the average performance of our best model (Dense KNN w/ Adaptation) with that of the next best model (Seq2seq w/ Adaptation). We performed a paired bootstrap that resampled the test set 50,000 times with replacement ([Statistical Significance Tests for Machine Translation Evaluation](https://aclanthology.org/W04-3250); Koehn, EMNLP 2004). We find that the gain of Dense KNN over Seq2seq is statistically significant at the p<0.001 level – indicating that the superior performance of our proposed model is likely statistically reliable. Further, since all other baselines perform worse than Seq2seq, we have every expectation that our approach has statistically significant gains over the other baselines as well.\n\n**Questions:**\n\n\n**A.** “MC Train” is an abbreviation for “Most Common Train”, which refers to a baseline where we return the most frequent hashtag label from the corresponding training set for each setting. For instance, in the “Non-temporal” and “Temporal w/ Adaptation” settings, we always return the most common hashtag label from the three weeks right before the test week. For the “Temporal w/o Adaptation” setting, we always return the most common hashtag label from the first three weeks of the year. We included this baseline to show what performance looks like for a method that relies simply on common hashtag usage statistics to recommend hashtags to users. We chose this “most common” approach because it performed best for our Dense KNN re-ranker. However, we realize we did not include an in-depth explanation of this and will do so in future versions of the paper. Thanks for pointing this out.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a new task focusing that revolves around hashtag prediction, considering the impact of temporal shifts.\n2. The authors propose a retrieval-based method that enables the model to predict hashtags without the burdensome requirement of re-training.", "reasons_to_reject": "1. The experimental results derived from the collected dataset fail to convincingly demonstrate the universal effectiveness of the proposed method. Specifically, as indicated in Table 2, the performance of KNN-Seq and KNN-Bertweet with adaptation exhibits a lower recall @1 compared to their counterparts without adaptation, raising concerns about the method's reliability and applicability.\n\n2. The decision to split the data based on a weekly interval for real-world scenario evaluation appears somewhat impractical, as it results in data sets with closely aligned training and testing dates. To enhance the validity of the experiments, it is suggested to adopt a more reasonable time interval, such as training on data from January, February, and March, while testing on data from July, to simulate a more realistic temporal separation.\n\n3. The paper lacks a significant test to ascertain the statistical significance of the reported results.", "questions_to_authors": "A. What does the “MC Train” in Figure 2 mean?", "paper_topic_and_main_contributions": "The authors' contribution lies in introducing a novel task that focuses on predicting hashtags for tweets using temporal adaptation, taking into consideration the real-time nature of social media. Traditional models trained solely on historical data might not be suitable for handling new data in this context. To address this challenge, the authors propose a method that leverages a constructed datastore to retrieve similar tweets, thus providing valuable contextual information to the model. Consequently, the proposed model demonstrates the ability to predict hashtags for new tweets without the need for re-training. The evaluation of their approach is conducted using tweet data gathered from Twitter, and the dataset is thoughtfully partitioned into weekly segments to perform comprehensive experiments.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CXrR1EGXhT", "reviewer": "Reviewer_9dfY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful feedback on our submission. We hope to address your questions and concerns below:\n\n\n**1.** We appreciate the suggestion to include a more detailed dataset description in the paper. To address this, we will (a) move our dataset statistics table that is currently in the Appendix into the main body of the paper in all future revisions, and (b) elaborate on our dataset construction and provide examples of its contents in the main paper, which we were unable to include due to the short paper format’s space constraints. We hope that this will encourage others to use the dataset for future research on temporal model adaptation.\n\n\n**2.** In addition to providing more details on the dataset in future revisions, we will also elaborate on how we decided the temporal division. In order to test temporal effects at-scale, we chose to divide our dataset into weeks because we (a) needed a fine-grained enough division to investigate how model performance decays over time, yet (b) enough data points in each division for supervised training/fine-tuning of high-capacity neural baselines for comparison. Of course, these two decisions are at odds with each other since very fine-grained divisions will reduce the amount of training data in each division. Finally, (c) we desired a temporal setup that would be practical for real-world use (i.e. avoid excessively frequent updates). While, in our paper, we showed that updating a model every three weeks provides noticeable gains in performance, we note that due to the hashtag prediction task’s inherent user-generated labels, more fine-grained (i.e., day-to-day) splits could just as easily be gathered in an industrial setting for a production system to further reduce temporal label shift.\n\n\n**3.** Good question – this is a key feature of our approach. Because our proposed model (Dense KNN w/ temporal adaptation) is non-parametric, temporal updates can be performed simply by replacing the datastore with the most recent batch of tweets and hashtags. As a side effect, replacing the datastore updates our hashtag label inventory: a dense KNN retriever’s label set is fully determined by its datastore. The BART Classifier baseline can only predict the tags seen in the training data as well due to the fixed output layer size, but requires costly gradient updates to do so. This is because the BART Classifier is parametric: the number of parameters is fixed when it is trained, so new hashtag labels require new parameters.  Finally, in order to have a complete experimental comparison, we wanted to also include a parametric baseline that had at least the possibility of predicting new hashtags. For this purpose, we included the BART seq2seq baseline. Because it generates hashtags token by token, this baseline has an open-vocabulary: it has the ability to place probability mass on hashtags it never saw during initial training. In our experimental results, we show that neither the Classifier nor the Seq2seq model perform as well as our temporally updated Dense KNN retrieval model, *even when they are temporally updated with new supervised training data.* This demonstrates that our Dense KNN model is an effective approach to label shift in temporal classification despite requiring no costly gradient-based re-training like the Classifier/Seq2seq model. Its efficiency also suggests that it could be automatically updated much more often than other methods (i.e., at daily intervals) to reduce the occurrence of unseen hashtags.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a non-parametric dense retrieval technique that offers a simple and effective solution to adapt NLP models to temporal changes.\n2. The experiments conducted on a year-long Twitter dataset demonstrate an improvement over the baselines.\n3. The paper's focus on user-generated social media data and its adaptability challenges is relevant in the current digital landscape.\n", "reasons_to_reject": "1.\tLack of detailed description of the dataset in the main sections, such as data volume.\n2.\tWhat is the basis for setting time division?\n3.\tMy understanding is that the author classifies new tweets into the static hashtag of the training set, but the hashtag itself changes with time. How does the author deal with this problem?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses a challenge in NLP related to the adaptability of traditional models to the constantly changing nature of user-generated social media data. By focusing on temporal adaptation through longitudinal hashtag prediction, the authors propose a non-parametric dense retrieval technique to handle temporal distribution shifts and deleted training data without the need for costly re-training. The paper presents an approach with practical implications. However, some aspects deserve further attention and refinement.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper develops a new approach for tackling the challenge of predicting hashtags for tweets in real-time social media by leveraging temporal adaptation. The method utilizes a constructed datastore to retrieve similar tweets, providing informative context and allowing the model to predict hashtags for new tweets without the need for re-training. The evaluation employs Twitter data partitioned into weekly segments, offering comprehensive insights into their approach's effectiveness. While this paper addresses the crucial issue of adaptability in handling evolving social media data, there are some aspects that could benefit from further attention and refinement, as mentioned in the second review.

The reviewers acknowledge the merits of this work and the authors provided additional experiments to address some of the raised concerns. The reviewer agree that the work is sound and has a decent excitement level.
Decision: Accept-Main