Predict the Future from the Past? On the Temporal Data Distribution Shift in Financial Sentiment Classifications
Authors: Yue Guo
Chenxi Hu
Yi Yang
Venue: EMNLP 2023 Main (2023)
Keywords: Distribution Shift
Financial Sentiment Classifications
Language Models Robustness
TL;DR: An analysis and mitigation method on the temporal data distribution shift problem in financial sentiment classifications

Abstract: Temporal data distribution shift is prevalent in the financial text. How can a financial sentiment analysis system be trained in a volatile market environment that can accurately infer sentiment and be robust to temporal data distribution shifts? In this paper, we conduct an empirical study on the financial sentiment analysis system under temporal data distribution shifts using a real-world financial social media dataset that spans three years. We find that the fine-tuned models suffer from general performance degradation in the presence of temporal distribution shifts. Furthermore, motivated by the unique temporal nature of the financial text, we propose a novel method that combines out-of-distribution detection with time series modeling for temporal financial sentiment analysis. Experimental results show that the proposed method enhances the model's capability to adapt to evolving temporal shifts in a volatile financial market.

Predict the Future from the Past? On the Temporal Data Distribution Shift in Financial Sentiment Classifications
Yue Guo Chenxi Hu Yi Yang
The Hong Kong University of Science and Technology
yguoarêconnect.uss. ust.cctt................ ust.hk chuak@connect .st. hk imyiyang@ust.he hk
Abstract
There has been limited research on the tempo- ral dataset shift in the financial context. Existing NLP works on financial sentiment analysis follow the conventional approach that randomly splits a dataset into training and testing so that there is no distribution shift between training and testing (Malo et al., 2014; Cortis et al., 2017). However, in real-world financial sentiment analysis sys- tem, there could be unpredictable distribution shifts between the data that is used to build the model (in-sample data) and the data that the model runs inference on (out-of-sample data). As a result, the practitioners often face a dilemma. If the model fits too well to the in-sample data, it may experi- ence drastic drop in the out-of-sample data if a distribution shift happens (such regime shift from bull market bear market); if the model is built minimize performance disruption, its per- formance may be unsatisfactory on the in-sample data as well as the out-of sammpl data.
Temporal data distribution shift is prevalent in the financial text. How financial senti- ment analysis system be trained a volatile market environment that can accurately infer sentiment and be robust to temporal data dis- tribution shifts? In this paper, we conduct an empirical study the financial sentiment anal- ysis system under temporal data distribution shifts using real-world financial social media dataset that spans three years. We find that the fine-tuned models suffer from general perfor- mance degradation in the presence of tempo- ral distribution shifts. Furthermore, motivated by the unique temporal nature of the financial text, we propose novel method that combines out-of cut-of-distribution detection with time series modeling for temporal financial sentiment anal- ysis. Experimental results show that the pro- posed method enhances the model's capability adapt evolving temporal shifts a volatile financial market.
In this paper, we raise our first research ques- tion RQ1: how does temporal data shift affect the robustness of financial sentiment analysis? The question is not trivial as it seems. For exam- ple, Guo et al. (2023a) find that language models are robust to temporal shifts in healthcare predic- tion tasks. However, financial markets may exhibit even more drastic changes. To answer this question, we systematically assess several language models, from BERT to GPT-3.5, with metrics that mea- sure both the model capacity and robustness under temporal distribution shifts. In our monthly rolling- based empirical analysis, this dilemma between in-sample performance and out-of-sample perfor- mance is confirmed. We find that fine-tuning pre- trained language model (such BERT) fails to pro- duce robust sentiment classification performance in the presence of temporal distribution shifts.
1 Introduction
Natural language processing (NLP) techniques have been widely adopted in financial applications, such as financial sentiment analysis, to facilitate investment decision-making and risk management (Loughran and McDonald, 2016; Kazemian et al., 2016; Bochkay et al., 2023). However, the non- stationary financial market environment can bring about significant changes the data distribution be- tween model development and deployment, which can degrade the model's performance over time and, consequently, its practical value. For exam- ple, regime shift in the stock market refers to significant change in the underlying economic or financial conditions. A regime shift, which may be triggered by changes in interest rates or political events, can significantly affect the market behav- ior and investor sentiment (Kritzman et al., 2012; Nystrup et al., 2018).
Moreover, we are interested in RQ2: how to mitigate the performance degradation of financial sentiment analysis in the existence of temporal dis-

---

tribution shift? Motivated by the unique temporal nature of financial text data, we propose a novel method that combines out-of-distribution (OOD) detection with autoregressive (AR) time series mod- eling. Experiments show that OOD detection can effectively identify the samples causing the model performance degradation (we refer to those sam- ples as OOD data). Furthermore, the model per- formance the OOD data is improved by an au- toregressive time series modeling on the historical model predictions. As a result, the model perfor- mance degradation from in sampee data to out-of- sample data is alleviated.
(X,, Yi) denotes set of text and associated sen- timent label collected from time t. Here could be at various time horizons, such as hourly, daily, monthly, or even longer horizon.
In the real-world scenarios, at time t, the sentiment classification model can only be trained with the data that is up to 4, i.e., {(X1, Y1), .).,. (Xe, Yt))]- We denote the model trained with data up period i as Mt. In continu- ous production system, the model is applied to the data (X++1, 1() the next time period +
The non-stationary financial market environment leads to different data distributions at different pe- riods, i.e., there is temporal distribution shift. However, the non-stationary nature of the financial market makes it difficult to predict how data will be distributed in the next period, Without loss of generality, we assume p(X,, Y2) A p(X++1, Y(+1) for any time
This work makes two contributions to the lit- erature. First, while sentiment analysis is very well-studied problem in the financial context, the long-neglected problem is how to build robust financial sentiment analysis model under the per- vasive distribution shift. To our knowledge, this paper provides the first empirical evidence of the impact of temporal distribution shifts on financial sentiment analysis. Second, we propose novel ap- proach to mitigate the out-of-sample performance degradation while maintaining in sampe sentiment analysis utility. We hope this study contributes to the continuing efforts build more robust and accountable financial NLP system.
2.2 Evaluation Metrics
Unlike traditional financial sentiment analysis, tem- poral financial sentiment analysis trains the model in sample data and applies the model to the out-of-sampil data. Therefore, in addition the in-sample sentiment analysis performance, we also care about its generalization performance on out- of-sample data. In other words, we hope the model experiences minimal performance degrada- tion even under significant temporal distribution shifts. Specifically, we use the standard classifica- tion metric F1-Score to measure the model perfor- mance. To measure the model generalization, we use AF F Flout, where Fli. and Floud are the F1-Score on in-sample and oft-of-sample data respectively. An ideal financial sentiment anal- ysis model would achieve high Flin and Flout and low F1 at the same time.
2 Temporal Distribution Shift in Financial Sentiment Analysis
In this section, first define the task of financial sentiment analysis on temporal data. We then in- troduce two metrics for model evaluation under the data distribution shift.
2.1 Problem Formulation
The financial sentiment analysis model aims to clas- sify a text input, such as social media post or financial news, into positive or negative classes It can be expressed as text classification model M (X) + Y. Conventionally, this task is modeled and evaluated on non-tenn temporal dataset, i.e. (X, Y) consists of independent examples un- related to each other in chronological order.
3 Experiment Setup
This section describes the evaluation setups on the dataset and models for temporal model analysis.
3.1 Dataset
We collect time-stamped real-worke world financial text dataset from StockTwits? Twitter-like social me- dia platform for the financial and investing com- munity. StockTwits data is also used in prior NLP work for financial sentiment analysis (Cortis et al., 2017), though the data is used in a conventional
In the real world, financial text data usually exhibits temporal patterns corresponding to its occurrence time. To show this pattern, we de- note (X ((X,, (X YN)}, where
"We consider binary positive/negative prediction in this paper. Other financial analysis systems may have an additional neutral lahel (Huang et al., 2022).
hhtps://ttocktwitsccom//

---

(a) distribution StockTwits Dataset:
monthly stock price return of Apple Inc, and Tesla Inc. in the dashed line in Figure (b) and (c). shows that the sentiment and the stock price movement are highly correlated, and the Spear- man Correlation between the sentiment and the monthly return is 0.397 for Apple and 0.459 for Tesla. Moreover, the empirical conditional proba- bility of labels given the specific token (i.e., "Ap- ple", "Tesla") varies in different months. Taking together, we observe a temporal distribution shift in the financial social media text.
0.90
20000
ventiment poultivity
17500
serdmment goutbvte
0.55
15000
1250
0.75
5000
posttive
negative data
2500
0.70
14-01
15:03
16-01
17:01
(b) Apple Inc.
centimont pontmmottooosuee posttvity
1.0
cantimmnt posttivitty
stock
0.15
0.10
0.05
monthy stock
0.00
0.8
-0.05
3.2
Sentiment Classification Models
-0.20
24-03
Tesla
17-01
We choose several standard text classification meth- ods for sentiment classification, including (1) a sim- ple logistic regression classifier that uses bag-of- words features of the input sentences, (2) an LSTM model with a linear classification layer on top of the LSTM hidden output, (3) three pretrained lan- guage models: BERT (base, uncased) (Devlin al., 2019), RoBERTa (hase) (Liuu et al., 2019) and fi- nance domain specific pretrained model FinBERT Yang et al., 2020); (4) the large language model GPT-3.5 (text-davinci-003) (Brown al., 2020) with two-shot in-context learning.
0.9
posttivity
SeeRtieet positviity
monthly stock return
0.7-
0.5
14-03
35-01 16-01 17-01
Figure (a) A temporal view of the "sentiment posi- tivity" (line) of the dataset and the number of examples with positive and negative labels (bar). (b) and (c): The "sentiment positivity" score (solid line) and the monthly stock price return (dashed line) of Apple Inc and Tesla Inc, respectively.
Empirical Evaluation of Temporal Data Shift in Financial Sentiment Analysis
non-temporal setting. In experiment, collect all posts from StockTwits spanning from 2014-1-1 2016-12-2 12-31 3. We then filter the dataset by select- ing those posts that contain a user-generated senti- ment label bullish ("1") or bearish ("0"). The final dataset contains 418,893 messages, each associated with publish date, providing helpful information for our temporal-based based empirical analysis.
Our first research question aims to empirically ex- amine temporal data shift affects the robustness of financial sentiment analysis and to what extent. Prior literature has studied temporal distribution shifts in the healthcare domain and finds that pre- trained language models are robust in the pres- ence of temporal distribution shifts for healtheare- related prediction tasks such hospital readmis- sion (Guo et al., 2023a). However, financial mar- kets and the financial text temporal shift are much more volatile. We empirically answer this research question using the experiment setup discussed in Section 3.
We provide some model-free evidence on the temporal distribution shift. First, we plot the "sentiment positivity" score for the monthly sen- timent label distributions of the whole dataset in Figure 1 (a). The sentiment positivity is defined the percentage of the positive samples, i.c., #pos/ (#pos #neg). It shows that while most messages are positive each month, the ratio be- tween positive and negative samples fluctuates.
4.1 Training Strategy
Training a sentiment classification model on the time-series data is not trivial, as different utiliza- tion of the historical data and models would lead different results in model performance and gen- eralization. To comprehensively understand the model behavior under various settings, we summa- rize three training strategies by the different incor- poration of the historical data and models.
We then choose two representative companies, Apple Inc. and Tesla Inc., and filter the messages with the token "Apple" or "Tesla". We plot their sentiment positivity score in the solid lines in Fig- ure (b) and (c), respectively. We also plot the
'More recent year data is not easily accossible due to API restriction.

---

Old Data, New Model (ODNM): This train- ing strategy uses all the available data up to time I, i.e ( YD), (X,, Yt)} to train a new model Mr. With this training strategy, the sentiment analy- sis model is trained with the most diverse financial text data that is not restricted to the most recent period.
4.3 Empirical Results
We evaluate different sentiment classification mod- els using different training strategies 4. We present the main empirical results on the original imbal- anced dataset in Table 1, averaged over the 35 monthly results. An experiment using the balanced dataset after up-sampling the minor examples is presented in Appendix A, from which similar con- clusions can be drawn. To better understand the model performance over time, we plot the results by month in Figure 2. using BERT and NDOM strat- egy as an example. The monthly results of NDNM and ODNM on BERT are shown in Appendix B. Furthermore, to compare the performance and the robustness against data distribution shift among the training strategies, we plot the in-sample per- formance Flin Flin(aug) (avg) in Figure 3, and the perfor- mance drop AF AFl((vg) in Figure 4 by the training strategies. The F1 (argg is supplemented in Ap- pendix B.
New Data, New Model (NDNM); For each time period f, a new model M, is trained with the latest data (X,, Yt) collected in time I. This training strat- egy fits the model to the most recent data, which may have the most similar distribution to the out- of sampee data if there is no sudden change in the market environment.
New Data, Old Model (NDOM): Instead of training a new model from scratch every time, we update the model trained at the previous time with the latest data. Specifically, in time t, the param- eters of the model M, are initialized with the pa- rameters from M, and continuously learn from (X,, Yi). This training strategy inherits the knowl- edge from past data but still adapts to more recent data.
We have the following observations from the experimental results: The performance drop in the out-of-sample prediction is prevalent, espe- cially for the negative label. All AF1 of the fine- tuned models in table 1 are positive, indicating that all fine-tuned models suffer from performance degradation when serving the models to the next month's 's data. Such performance drop is especially significant for the negative label, indicating that the minority label suffers even more in the model gen- eralization. The results remain after up-sampling the minority label, as shown in Appendix A.
For GPT-3.5, we use two-shot in-context learn- ing to prompt the model. The in-context examples are randomly selected from (X,, Yt). The prompt is "Perform financial sentiment classification: text: | positive example) label:positive: text: (a negative example) label:negative; text: (testttning example) la- bel:"..
4.2 Rolling-based Empirical Test
NDOM training strategy achieves the best in- sample performance, yet fails to generalize on out- of-sample. Table and Figure 3 show that NDOM has the highest F1-score among the three training strategies, especially for the in-sample prediction. Training with ODNM strategy is most robust against data distribution shift. As shown in Table I and Figure 4, among the three training strategies, ODNM has the smallest performance drop in out- of-sample predictions. It suggests that using long range of historical data can even out the possible distribution shift in the dataset and improve the model's Iss robustness.
To empirically examine temporal sentiment classi- fication models, we take a rolling-basee based approach. We divide the dataset by month based on the times- tamp of each text. Since we have three years of StockTwitss data, we obtain 36 monthly subsets. For each month t, we train a sentiment classification model M, (Section 3.2, except GPT-3.5) using a training strategy in Section 4.1 that uses data up to month f. For evaluation, we consider the test- ing samples from (X₃: Yt) as the in-sample and the testing samples from (X++1, Y(41) as out-of- sample. This roaee approach simulates a real-world continuous production setup. Since have 36 monthly datasets, our temporal-based em- pirical analysis is evaluated on 36-1 '35 monthly datasets (except the last month). We report the aver- age performance in F' Fl-scorr and A Fl as 2.2. The train/validate/test split is by 7:1.5:1.5 randomly.
There is a trade-off between in-sample per- formance and out-of-sample performance. As stated above, the NDOM strategy achieves the best
"Since LR and LSTM are trained using datasct-specific vocabulary, the NDOM training strategy, which uses the old model's 's vocabulary, is not applicable.

---

Fl.:(po9)
Plae(pps)
Fluulang)
Logisllckegression
ODNM
89.9
59.7
34.8
2.10
61.2
1.18
NDNM
91.2
90.1
1.07
46.2
36.1
10.04
68.7
63.1
5.55
91.4
55.8
4.51
LSTM
NONM
90.4
$9.1
1.32
44.5
35.2
9:42
67.5
62.2
5.37
41.9
BERT
NONM
91.5
90.4
1.13
53.5
45.2
8.27
72.5
67.8
4.70
NDOM
93.1
92.1
0.95
65.1
59.5
5.67
79.1
75,8
3.31
ODNM
91.0
5038
0.24
48.8
41.1
1.38
09.9
65.9
0.97
RoBERTH
NDNM
91.7
90.6
1.12
56.8
90.0
6.80
74.3
70.3
3.96
NOOM
93.3
92.6
67.4
4.18
77.9
GONM
89.4
80.2
0.25
40.3
58.5
1.76
64.9
63.9
1.00
PiaBERT
NDNM
91.2
XDOM
92.6
91.7
0.83
60.9
$4.9
6,05
76.7
73.3
3.44
G
81.8
$2.5
51.0
67.5
Table I: Performance of different models under New Data New Model (NDNM), New Data Old Model (NDOM), and Old Data New Model(ODNM) training strategies. The numbers averaged over the monthly evaluation over three years.
BERT fine-tuned and NDOM training strategy
in-sample performance but suffers significant out- of-sample degradation. ODNM, conversely, has the slightest performance degradation, yet its overall prediction capability limited. From a practical perspective, both strategies are not idcal. First, ac- curately identifying the market sentiments is essen- tial for a financial sentiment classification model to build portfolio construction. Second, it is also im- portant that financial sentiment classification mod- els produce stable prediction performance that the subsequent trading portfolio, driven by the pre- diction outcomes, can have the least disruption.
0.9
0.8
F1-Score
0.7
0.6
0.5
out (Neg)
F1_in (Neg)
F1_ out (Pos)
F1_in in (Pos)
14-01
15-01
16-01
17-01
Figure 2: BERT's in-sample and out-of-sample predic- tion using the NDOM strategy. The performance gaps between the iampe and out-of-sample prediction are significant, especially for the negative label.
In GPT-3.5, the performance drop is not ob- served, but the in-sample and out-of-sample per- formance falls behind the fine-tuned models. It indicates that increasing the model size and train- ing corpus may reduce the performance drop when facing a potential distribution shift. However, GPT- 3.5 not ideal, as its performance on the in-sample and out- sample data is significantly worse than the fine-tuned models, which aligns with the find- ings from (Guo et al., 2023b). Moreover, as the training corpus in GPT-3.5 covers the data from 2014 to 2017, our test set may not be genuinely -itt---diitribttion regarding GPT-3.5, which may also result in alleviating the performance drop.
BERT fine-tuned
ODNM
0.80
NDNM
NDOM
Macro Fl.n
0.75
0.70
0.65
14-01
15-01
16-01
17-01
Figure 3: The performance comparison Fl (avg) us- ing three training strategies in BERT.
BERT fine-tuned
4.4 Additional Analysis
ODNM
M
We conduct additional analysis to understand the problem of performance degradation further. We examine the relationship between distribution shift and performance drop. To measure the distribution shift between in-sample data (X,, Y;) in month and -f--lle data (Xt 1, YY in month t+1, we follow the steps: 1) For each token e in the vo- cabulary V. we estimate the empirical probability
NDOM
0.05
0.00
-0.05
14-01
15-01
16-01
17-01
Figure 4: The performance drop A (Fl(g using three training strategies BERT.
'1 is the vocabulary from the pretrained tokenizer.

---

Spearmanr (p-value)
the sentiment label change in the volatile financial market. For example, suppose the model makes predictions based on the spurious correlated words p(y "Tesla"). When the market sentiment regard- ing Tesla fluctuates, the model's robustness will be affected.
BERT
0.385 (0.020)
RoBERTa
0.345 (0.039)
FinBERT
0.281 (0.096)
Table 2: Spearman correlations between the perfor- mance drop (AF1) and in-sample to out-of-sample dis- tribution shift.
We use the most explainable logistic regression model as an example to provide evidence for spu- rious correlations. The logistic regression model assigns coefficient each word in the vocabulary, suggesting the importance of the word contributed to the prediction. We fit a logistic regression model to our dataset and then extract the top 30 words with the highest coefficients (absolute value). The extracted words are listed in Table 3, with bold words indicating the sentiment words and the un- bolded words as the spurious words. We can that most words the model regards as important are not directly connected the sentiments. As a result, the performance of model prediction p(ylv) is likely influenced by the changing financial senti- ment in the volatile markets.
Important Features
devices, inchr, inchry, nakd, incnn, bears, pharnacenticals, patience, incara, pix, bulls, release, inesla, paper, fall, dump, purp,, strong, advanced, inephs, ride, impp, addeed, cateriillar, kellogg, shorts, plx, owens, dilution, squeeze
Table 3: Top 30 important words identified by the Lo- gistic Regression model. The bold words the finan- cial sentiment words, and the unbolded words are the spurious correlated words. The model assigns high im- portance to the many spurious words and is prone to be influenced by spurious correlations.
Pe(ylu), (vle), (v)),, (v). 2) We measure the distribution shift from (Xe, Yt) to (X++1, Y4+1) as the weighted sum of the KL-divergence (KLD) between the two conditional probability distribu- Pt+1
Mitigating Model Degradation under Temporal Data Shift
tions: Pz L (1)l), (ty))))))
We then compute the Spearman correlation be- tween the performance drop AF1 and the distribu- tion shift from in-sample data (Xe, Yt) to out-of- sample data (Xe 1, I)) The result is shown in Table 2, showing significant positive correlation between performance drop and distribution shift, Therefore, more significant distribution shift, pri- marily caused by financial market turbulence, can lead to more severe model performance degrada- tion. This problem especially problematic be- cause the performance degradation may exacerbate sentiment-basee trading strategy during a volatile market environment, leading to significant invest- ment losses.
In the previous section, our analysis reveals a con- sistent performance degradation financial senti- ment classification models on out-of-sample data, In this section, we explore possible ways to mit- igate the degradation. As our previous analysis shows that the performance degradation is corre- lated with the distribution shift, it is reasonable to infer that the performance degradation is caused by the failure of the outtrf-distribution (OOD) exam- ples. This observation motivates two-stage method to improve the model robustness under temporal data distribution shifts. Firstly, we train an OOD sample detector to detect whether an upcoming sample is out of distribution. If not, we still use the model trained on in-sample data on this sample. If yes, we propose using an autoregressive model from time series analysis simulate the model prediction towards the future.
propose a
Second, we provide empirical evidence that the models are prone be influenced by spurious correlations. Generally, a sentiment classification model makes predictions on the conditional prob- ability p(ylv) based on some sentiment words v. Ideally, such predictions are effective if there is no distribution shift and the model can success- fully capture the sentiment words (c.g., bear- ish, bullish, and so on). However, if model is affected by spurious correlations, undesirably as- sociates the words with no sentiment with the label. The model generalization will be affected when the correlations between the spurious words and
5.1 Mitigation Method
This subsection introduces the mitigation method for temporal financial sentiment analysis.
5.1.1 Detecting OOD Samples
As the model trained on the historical data experi- ences performance degradation on the future data under distribution shift, we first employ an OOD

---

detection mechanism to determine the ineffective future data. To train the OOD detector, we collect a new dataset that contains the in-distribution (ID) data (label=0) and OOD data (label=1) regarding a model M, The labeling of the OOD dataset is based on whether M2 can correctly classify the sam- ple, given the condition that the in-sample classifier can correctly classify the sample.
also be expressed by the combination of the past models' predictions. Specifically, given an OOD sample rt+1 detected by the OOD classifier, the prediction Øt+1 given by linear regression on the predictions from the past models Mr..., Mr pil. i.c.,
D1+1 1++1=Eashoe agM, x(xx+1) t
Ea
(3)
Specifically, let i.j denote the indexes of time satisfying < j. given a target sentiment classifier M₁, and a sample (Ir).9z) which can be correctly classified by the in-sample sentiment model Mj. the OOD dataset assigns the label by the rule
where 0g is the regression coefficient and e is the error term estimated from the past data. More- over, is the order of the autoregressive model determined empirically.
For temporal data in financial sentiment classi- fication, the future distribution is influenced by an aggregation of recent distributions and stochastic term. Using an AR model on previous models' pre- dictions can capture this feature. The AR modeling differs from weighted ensemble method that as- signs each model a fixed weight. In our method, weights assigned to past models are determined by how recently they were trained used, with more recent models receiving higher weights.
0, if M((z;) Vj OOD(M = - 1, M((1)) + y
After collecting the OOD dataset, we train an OOD classifier ((, on the OOD dataset to detect whether a sample x is OOD concerning the model M. The classifier is two-layer multi-layer perceptron (MLP) on the [CLS] token of (z). i.e.,
5.2 Experiment Setup
f(M,x)= =
bi)) tb2 (2)
To train the OOD detector and estimate the parame- ters in the AR model, use data from 2014-01 to 2015-06. We split the data each month by 7:1.5:1.5 for sentiment model training, detector/AR model training, and model testing, respectively. The data from 2015-07 to 2016-12 is used to evaluate the effectiveness of the mitigation method.
The classifier is optimized through the cross- entropy loss on the OOD dataset. During train- ing, the sentiment model M parameters are fixed, and only the parameters of the MLP (i.e., Wi, W2,b,,by b2) are updated by gradient descent. The parameters of the OOD classifier are used across all sentiment models M E (M,...,,yy regardless of time.
To train the OOD detector, we use Adam op- timizer and grid search for the learning rate in [2x10 10
3 2x 10 2 10 51. batch size in [32, 64].
During inference, given sentiment model Mt and an out-of-sample F++1 we compute [MM, 2++11 to detect whether It+1 is OOD re- garding to Mt. If Tt+1 is not an OOD sample, we infer the sentiment of It by Mc. Otherwise, xt is regarded as an OOD sample, and M, may suffer from model degradation. Therefore, we predict the sentiment of rx+1 by the autoregressive model from time-series analysis to avoid potential ineffec- tiveness.
When building the OOD dataset regarding Mc. we use the data that happened within three months starting from t.
To estimate the parameters in the AR model, for training sample (z,, Vt), we collect the predic- tions of x; from M,-1. Mc-p. and train a regres- sion model to predict yı, for t from 2014-01+p 2015-06. We empirically set the order of the AR model p as the experiment.
5.1.2 Autoregressive Modeling
5.3 Baselines
In time series analysis, an autoregressive (AR) model assumes the future variable can be expressed by a linear combination of its previous values and on a stochastic term. Motivated by this, the distri- bution in the future difficult to estimate directly, we assume the prediction from a future model can
Existing NLP work has examined model robust- ness on out-of-domain data in a non-temporal shift setting. We experiment with two popular meth- ods, spurious tokens masking (Wang et al., 2022) and counterfactual data augmentation (Wang and Culotta, 2021), to examine their capability in miti-

---

BERT
Pln((ugg)
(aeg)
AF AFl(avy) J
Precision
Recall
F1
BERT
80.13
78.07
2.05
+STM
78.04
75.87
2.18
ID
0.99
0.8
0.88
+CDA
76.91
74.88
2.02
OOD
0.23
0.86
0.37
+Ours
80.13
78.70
1.42
Accuracy
0.8
RoBERTa
81.87
80.25
1.62
+STM
81.12
79.14
1.98
FinBERT
+CDA
79.68
77.59
2.09
Precision
Recall
F1
+Ouns
81.87
80.61
1.26
ID
0.81
0.9
FinBERT
77.46
75.44
2.01
+STM
76.57
74.49
2.08
OOD
0.15
0.91
0.26
+CDA
73.92
72.02
1.91
Accuracy
0.82
+Ours
77.46
76.09
1.36
Table 4: The performance of the OOD detector of BERT and FinBERT. The most crucial indicator is the recall of OOD data, want to retrieve much OOD data as possible.
Table The mitigation results. Our method improves the performance of the sentiment model on out-of- sample data and reduces the performance drop.
gating the performance drop under temporal data distribution shift.
achieve the recall of 0.86 and 0.91 for OOD data in BERT and FinBERT, respectively, indicating the adequate capability to identify the data that the sen- timent models will wrongly predict. As the dataset is highly unbalanced towards the ID data, the rela- tively low precision in OOD data is expected. Nev- ertheless, the detector can achieve an accuracy of around 0.8 the OOD dataset.
Spurious Tokens Masking (Wang et al., 2022) (STM) is motivated to improve the model robust- ness by reducing the spurious correlations between some tokens and labels. STM identifies the spu- rious tokens by conducting cross-datasct stability analysis. While genuine and spurious tokens have high importance, "spurious" tokens tend to be im- portant for one dataset but fail to generalize to oth- ers. Therefore, We identify the "spurious tokens" those with high volatility in the attention score across different months from 2014-01 to 2015-06. Then, mask the identified spurious tokens dur- ing training and inference the data from 2015-07 2016-12.
Table shows the results of the mitigation meth- ods under the NDOM training strategy. We only apply our mitigation method to the oft-of-sample prediction. For BERT, RoBERTa, and FinBERT, our method reduces the performance drop by 31 %2 26%, and 32%, respectively. Our results show that The AR model can improve the model performance OOD data. As result, the overall out-of-sample performance improved, and the model degrada- tion alleviated.
Counterfactual Data Augmentation (Wang and Culotta, 2021) (CDA) improves the model robustness by reinforcing the impact of the causal clues. It first identifies the causal words by match- ing algorithm and then generates the counterfactual data by replacing the identified causal word with its antonym. Like STM, we identify causal words on the monthly datasets from 2014-01 to 2015-06. More details of the setup of the two baseline methods are presented in Appendix C.
Another advantage our methods that, unlike the bascline methods, our method does not require re-training the sentiment models. Both baseline methods re-train the sentiment models on the newly generated datasets, either by data augmentation spurious tokens masking, the cost of influencing the model performance. Our proposed methods avoid re- re-training the sentiment models and improve the out-of samppl prediction with aggregation on the past models.
5.4 Mitigation Results
First, we analyze the performance of the OOD de- tector. Table 4 shows the classification reports of the OOD detector of BERT and FinBERT the test set. The recall of OOD data is the most crucial indicator of the model performance, as we want to retrieve much OOD data as possible before ap- plying it to the AR model to avoid potential model degradation. As shown in table 4, the detector can
Related Works
Temporal Distribution Shift. While temporal dis- tribution shift has been studied in other contexts such as healtheare (Guo al., 2022), there is no systematic empirical study of temporal distribution shift in the finance domain. Moreover, although
a prior study in the healthcare domain has shown

---

that the large language models can significantly mitigate temporal distribution shifts on healthcare- related tasks such as readmission prediction (Guo et al., 2023a), financial markets are more volatile, so is the financial text temporal shift. Our em- pirical study finds that fine-tuned models suffer from model degradation under temporal distribu- tion shifts.
robust to temporal data distribution shifts.
Limitations
This paper has several limitations to improve future research. First, our temporal analysis is based on the monthly time horizon, so we only analyze the performance degradation between the model built in the current month and the follow- ing month t + 1. Future analysis can investigate other time interval granularity, such weekly annual. Second, our data is collected from social media platform. The data distribution on social media platforms may differ from other financial data sources, such as financial news articles or an- alyst reports. Thus, the robustness of language models on those types of financial text data needs to be examined, and the effectiveness of our pro- posed method on other types of financial text data warrants attention. Future research can follow our analysis pipeline to explore the impact of data dis- tribution shifts on financial news or analyst reports textual analysis. Third, our analysis focuses on sentiment classification performance degradation. How performance degradation translates into eco- nomic losses is yet to be explored. Trading sim- ulation can be implemented in future study to understand the economic impact of the problem better.
Financial Sentiment Analysis. NLP techniques have gained widespread adoption the finance do- main (Loughran and McDonald, 2016; Yang et al.. 2020; Bochkay et al., 2023; Shah et al., 2022; Wu et al., 2023; Chuang and Yang, 2022). One of the essential upplications is financial sentiment clas- sification, where the inferred sentiment is used to guide trading strategies and financial risk manage- ment (Kazemian et al., 2016). However, prior NLP work on financial sentiment classification has not explored the temporal distribution shift problem, a common phenomenon in financial text. This work aims to investigate the financial temporal distribu- tion shift empirically and proposes mitigation method.
7 Conclusions
this paper. we empirically study the problem of distribution shift over time and its adverse im- pacts on financial sentiment classification models. We find consistent yet significant performance degradation when applying a sentiment classifica- tion model trained using in samle (past) data to the oft-ol-sample (future) data. The degradation is driven by the data distribution shift, which, unfor- tunately, is the nature of dynamic financial markets. To improve the model's robustness against the ubiq- uitous distribution shift over time, we propose a novel method that combines out-of-distribution de- tection with autoregressive (AR) time series mod- eling. Our method is effective in alleviating the out-of-sample performance drop.
References
Khrystyna Bochkay. Stephen V. Brown, Andrew Leone, and Jennifer Wu Tucker. 2023. Textual anal- ysis in accounting: What's next?". Contemporary Accounting Research, 40(2):765 805.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voon Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Given the importance of NLP in real-world finan- cial applications and investment decision-making, there is an urgent need to understand the weak- nesses, safety, and robustness of NLP systems. We raise awareness of this problem in the context of financial sentiment classification and conduct temporal-based empirical analysis from practi- cal perspective. The awareness of the problem can help practitioners improve the robustness and ac- countability of their financial NLP systems and also calls for developing effective NLP systems that are
Chengyu Chuang and Yi Yang. 2022. Buy tesla, sell ford: Assessing implicit stock market preference in pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Shori Papers), ACL

---

2022, Dublin, Ireland, May 22-27, 2022, pages 100- 105. Association for Computational Linguistics.
Tim Loughran and Bill McDonald. 2016. Textual anal- ysis in accounting and finance: A survey. Journal of Accounting Research, 54(4): 187-1230.
Keith Cortis, André Freitas, Tobias Daudert, Manuela Hürlimann, Manel Zarrouk, Siegfried Handschuh, and Brian Davis. 2017. Semeval-2017 task 5: Fine- grained sentiment analysis on financial microblogs and news. In Proceedings of the 11th International Workshop on Semantic Evaluation, SemBval@ACL 2017, Vancouver, Canada, August 3-4, 2017, pages 519-535. Association for Computational Linguistics.
Pekka Malo, Ankur Sinha, Pekka J. Korhonen, Jyrki Wallemius, and Pyry Takala. 2014. Good debt bad debt: Detecting semantic orientations in economic texts. J. Assoc. Inf. Sci. Technol., 65(4):782-796,
Peter Nystrup. Henrik Madsen, and Erik Lindström. 2018. Dynamic portfolio optimization across hidden market regimes. Quantitarive Finance, 18(1 3:83-5..
Jacob Devlin, Ming-Way Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Tech- nologies, NAACL-HLT 2019. Minneapolis, MN, USA. June 2-7, 2019, Volume I (Long and Short Papers), pages 4171-4186. Association for Computational Linguisties.
Raj Shah, Kunal Chawla, Dheeraj Eidnami, Agam Shah, Wendi Du Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. When FLUE meets FLANG: Benchmarks and large pre- trained language model for financial domain. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 2322- 2335, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Tianlu Wang, Rohit Sridhar, Diyi Yang, and Xuezhi Wang. 2022. Identifying and mitigating spurious correlations for improving robustness in NLP models. In Findings of the Association for Computational Linguistics: NAACL 2022, Seattle WA, United States, July 10-15, 2022, pages 1719-1729. Association for Computational Linguistics.
Lin Lawrence Guo, Stephen R Pfohl, Jason Fries, Alis- tair EW Johnson, Jose Posada, Catherine Aftandilian, Nigam Shah, and Lillian Sung. 2022. Evaluation of domain generalization and adaptation on improving model robustness to temporal dataset shift in clinical medicine. Scientific reports. 12(1):1-10.
Lin Lawrence Guo, Ethan Steinberg, Scott Lanyon Fleming, Jose Posada, Joshua Lemmon, Stephen R Pfohl, Nigam Shah, Jason Fries, and Lillian Sung. 2023a. Ehr foundation models improve robustness in the presence of temporal distribution shift. Scientific Reports, 13(1):3767.
Zhao Wang and Aron Culotta. 2021. Robustness to spu- rious correlations in text classification via automati- cally generated counterfactuals. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial In- telligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 14024 14031. AAAI Press.
Yue Guo, Zian Xu, and Yi Yang. 2023b. Is chat- gpt financial expert? evaluating language mod- els on financial natural language processing, CoRR, abs/2310. 16664
Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravol- ski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David S. Rosenberg, and Gideon Mann. 2023. Bloomberggpt: large language model for finance. CoRR, abs/2303. 17564.
Allen Huang, Hui Wang, and Yi Yang. 2022. Finbert: A large language model for extracting information from financial text. Contemporary Accounting Re- search.
Yi Yang. Mark Christopher Siy Uy, and Allen Huang. 2020. Finbert: A pretrained language model for financial communications. CoRR, 2ww/2000600097,,
Siavash Kazemian, Shunan Zhao, and Gerald Penn. 2016. Evaluating sentiment analysis in the context of securities trading. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics, ACI. 2016, August 7-12, 2016, Berlin, Germany, Volume I: Long Papers. The Association for Computer Linguistics.
Mark Kritzman, Sebastien Page, and David Turking- ton. 2012. Regime shifts: Implications for dynamic strategies (corrected). Financial Analysts Journal, 68(3):22-35 39.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 16692.


Reviews:
[{"review_id": "Fm1wZkPkTF", "reviewer": "Reviewer_MD99", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer MD99:\n \nThank you very much for your thoughtful and positive review of our work. We truly appreciate the time you took to evaluate our work on analyzing and addressing the temporal data distribution shift problem in financial sentiment classifications. Your insights and feedback are invaluable to us.\n \nWe are delighted that you recognize our empirical study of the model degradation phenomenon in a volatile financial market is timely and practical, and the proposed method is effective in this environment with temporal data distribution shifts. Your recognition of our efforts in this area means a lot to our team.\n \nIf you have any further questions or suggestions for collaboration, please feel free to reach out. We are always open to fruitful discussions and potential partnerships.\n \nOnce again, thank you for your support and positive feedback. We look forward to sharing more of our work with you in the future.\n \nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper tackles the timely and challenging problem of applying sentiment analysis to financial text in the presence of temporal data distribution shifts. The empirical study and proposed method contribute valuable insights into improving the performance of financial sentiment analysis models in volatile market environments. ", "reasons_to_reject": "No significant concerns or issues were identified.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores financial sentiment analysis in the presence of temporal data distribution shifts using a real-world financial social media dataset. The main contributions include:\n\n- Conducting an empirical study of the performance of fine-tuned models in a volatile financial market with temporal data distribution shifts.\n- Proposing a method that combines out-of-distribution detection with time-series modeling to improve the adaptability of financial sentiment analysis systems in the face of evolving temporal shifts.\n", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bT5zNGeADq", "reviewer": "Reviewer_mkMy", "summary": "", "questions": "Question 1: What is the calculation formula of alpha_k (regression coefficient)? It refers to “weights are determined by how recently they were trained”", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mkMy:\n \nThank you very much for your thoughtful and positive review of our work. We truly appreciate the time you took to evaluate our work on analyzing and addressing the temporal data distribution shift problem in financial sentiment classifications. Your insights and feedback are invaluable to us.\n \nFor question 1, we use the data from 2014-01 to 2015-06 to train the mitigation models, and then use the data from 2015-07 to 2016-12 to test the method's effectiveness (the same setting for all baselines). Rolling from 2014-01+p to 2015-06 (p is the order of the autoregressive model), for each example $x_t$ on time stamp $t$, we collect the past model predictions $M_{t-p}(x_t), M_{t-p+1}(x_t), …, M_{t-1}(x_t)$, and the ground-truth label $y_t$. We train a linear regression model to predict $y_t$ based on past model predictions. $\\alpha_k$ is the regression coefficient learned by the linear regression model. Empirically, we find that the well-trained regression model assigns a higher weight to the model closer to the time $t$. We will explain this experiment setting more clearly in the revised version of the paper.\n \nIf you have any further questions or suggestions for collaboration, please feel free to reach out. We are always open to fruitful discussions and potential partnerships.\n \nOnce again, thank you for your support and positive feedback. We look forward to sharing more of our work with you in the future.\n \nBest regards,\n\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a novel method that combines out-of-distribution (OOD) detection with autoregressive (AR) time series modeling to alleviate model performance degradation.\n\nThe analysis of Temporal Data Shift in Financial Sentiment (Section 4) is valuable.", "reasons_to_reject": "N/A", "questions_to_authors": "Question 1: What is the calculation formula of alpha_k (regression coefficient)? It refers to “weights are determined by how recently they were trained”", "paper_topic_and_main_contributions": "Topic:\n\nThis paper empirically studies the financial sentiment analysis system under temporal data distribution shifts.\nAnd this paper proposes a novel method that combines out-of-distribution detection with time series modeling for temporal financial sentiment analysis.\n\nContributions:\n\nThis paper builds a robust financial sentiment analysis model under the pervasive distribution shift.\n\nThis paper provides the first empirical evidence of the impact of temporal distribution shifts on financial sentiment analysis.\n\nThis paper proposes a novel approach to mitigate the out-of-sample performance degradation while maintaining in-sample sentiment analysis utility.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "92gpY1cqkO", "reviewer": "Reviewer_EjqB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer EjqB:\n \nThank you very much for your thoughtful and positive review of our work. We truly appreciate the time you took to evaluate our work on analyzing and addressing the temporal data distribution shift problem in financial sentiment classifications. Your insights and feedback are invaluable to us.\n \nWe would like to make some clarifications on the reviews:\n\nRegarding reason 1 to reject, we agree that semantic words convey the sentiment. In other words, the model makes predictions on the conditional probability $p(y|v)$ based on some sentiment words $v$. Ideally, such predictions are effective if there is no distribution shift and the model can successfully capture the sentiment words (e.g., $v=$ bearish, bullish, etc.). However, we also find that the models are prone to be influenced by spurious correlations. For example, if the model makes predictions based on the spurious correlated words $p(y|\\text{\"Microsoft\"})$ when the market sentiment regarding Microsoft fluctuates, the model robustness will also be affected.  \nWe use the most explainable Logistic Regression model as an example to provide evidence for such spurious correlations. After fitting the model to our dataset, we extract the top 30 words with the highest coefficients (absolute value), which are listed in the table below:\n\n|                                                                                                                                        Important Features                                                                                                                                        |\n|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------:|\n|     devices, incbr, incbry, nakd, incnn, **bears**, pharmaceuticals, **patience**, incara, ptx, **bulls**, **release**, incsla, paper, **fall**, **dump**, pump, **strong**, **advanced**, incphs, ride, imnp, added, caterpillar, kellogg, **shorts**, plx, owens, **dilution**, **squeeze**    |\n\nWe can see that most of the features the model regards as important are not directly connected to the sentiments except the bolded words that convey financial sentiments. As a result, the model prediction $p(y|v)$ is likely to be influenced by the changing financial sentiment in the volatile markets. This experiment further confirms that the findings of our work are non-trivial and counter-intuitive, raising awareness and addressing the importance of the negative impact of temporal distribution shifts on financial sentiment analysis.\n\nRegarding reason 2 to reject, we would like to explain that our work is not simply stacking and permutation of various models. Our contributions are twofold: (1) a novel OOD detection method: facing the challenge of the unknown and unpredictable distribution shifts, we propose a new method to conduct OOD detection by training a classifier on the OOD dataset generated on the historical data. The OOD detection is effective and avoids the requirement to know the future distribution. (2) A method to improve model performance. After identifying the OOD data, as the in-domain model is expected to be ineffective in the OOD data, we propose a method to improve model performance with the autoregressive model from the time-series analysis. The intuition of using a time-series analysis model is that the main objective of the time-series analysis is to predict the future value from the past values; therefore, it is very suitable for our scenario to improve the model’s future performance. Experiments prove that our method is effective.\n\nRegarding reason 3 to reject, we agree that designing specific fine-tuning methods could be a solution to improve the model's robustness. However, the empirical study in section 4.3 shows a trade-off between model absolute performance (in-sample performance) and model robustness to distribution shift (out-of-sample performance). As a result, it is not easy for a fine-tuning method to improve in-sample and out-of-sample performance simultaneously (see the results of the baseline models). Considering these difficulties, we do not use a new model or a fine-tuning method for our task. Instead, we propose to predict the future label with a time-series analysis method. To our knowledge, we are the first to propose to apply the time-series analysis method to improve model robustness on NLP tasks. Experiments show that our method is effective in both the in-sample and out-of-sample performance.\n \nIf you have any further questions or suggestions for collaboration, please feel free to reach out. We are always open to fruitful discussions and potential partnerships.\n \nOnce again, thank you for your support and feedback. We look forward to sharing more of our work with you in the future.\n \n \nBest regards,\nAuthors", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper is well structured.\n2.\tThe paper addresses a significant problem in the field of financial sentiment analysis. Temporal data distribution shifts are prevalent in financial text, and developing robust models to handle such shifts is crucial for accurate sentiment analysis in volatile markets.\n3.\tThe authors conduct the study using a real-world financial social media dataset spanning three years. This approach ensures the findings are grounded in real data, making the results more applicable and reliable.", "reasons_to_reject": "1.\tI understand that sentiment will change with financial trends, but the carrier of sentiment features is the semantics expressed in words. Why does the author think that the semantic features corresponding to sentiment at different times are significantly different?\n2.\tThis work seems like a stacking and permutation of various models, lacking innovation in models.\n3.\tSince the authors mention the volatility of financial sentiment data, they should emphasize how to design models or specific fine-tune methods to address this issue.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper investigates temporal data distribution shifts in financial sentiment analysis. It identifies performance degradation in fine-tuned models under such shifts. To address this, a novel method combining out-of-distribution detection with time series modeling is proposed. Experimental results demonstrate its effectiveness in adapting to evolving temporal shifts in a volatile financial market. Overall, the study offers valuable insights and a promising solution to improve sentiment analysis in dynamic financial environments.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are in general agreement that this paper could be accepted in EMNLP-2023. The reviewers also raised some concerns about this submission. The authors have provided a rebuttal that appeared to alleviate some concerns. A suggestion was made to accept the paper as Main Conference. The reviewers' concerns are also suggested to be addressed in the final version.
Decision: Accept-Main