KeFVP: Knowledge-enhanced Financial Volatility Prediction
Authors: Hao Niu
Yun Xiong
Xiaosu Wang
Wenjing Yu
Yao Zhang
Weizu Yang
Venue: EMNLP 2023 Findings (2023)
Keywords: Volatility forecasting
Finance
Text mining
TL;DR: Leveraging financial metric knowledge for financial volatility prediction.

Abstract: Financial volatility prediction is vital for indicating a company's risk profile. Transcripts of companies' earnings calls are important unstructured data sources to be utilized to access companies' performance and risk profiles. However, current works ignore the role of financial metrics knowledge (such as EBIT, EPS, and ROI) in transcripts, which is crucial for understanding companies' performance, and little consideration is given to integrating text and price information. In this work, we statistic common financial metrics and make a special dataset based on these metrics. Then, we introduce a knowledge-enhanced financial volatility prediction method (KeFVP) to inject knowledge of financial metrics into text comprehension by knowledge-enhanced adaptive pre-training (KePt) and effectively incorporating text and price information by introducing a conditional time series prediction module. We conduct extensive experiments on three real-world public datasets, and the results indicate that KeFVP is effective and outperforms all the state-of-the-art methods.

KeFVP: Knowledge-enhanced Financial Volatility Prediction
Hao Niu', Yun Xiong' Xixosu Wang', Wenjing Yu', Yao Zhang'
Weizu Yang²
Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University SShangaii Yuanlu Jiajia Information and Technology Co., Ltd.
"(hniu18, yunx, xswang19, yaozhang) wuu.eeedgeuuuuuuuuuuuu..m. cn
"(wjyu21 Jem. fudan. edu. cn
2 (wweizu..ynng)ejjajjaarrooor net
Abstract
should be aware of. One such unstructured data source earnings calls, which are quarterly con- ferences held by public company management to explain the latest performance, offer guidance on their expectation for the coming future, and answer questions raised by investors and analysts (Qin and Yang. 2019). The information conveyed by the conference provides investors and analysts with valuable insights into the company's current state and future prospects. Hence, the goal of this task is to predict future stock price volatility after the earn- ings call announcement by combining historical prices and earnings call transcripts.
Financial volatility prediction is vital for char- acterizing company's risk profile. Transcripts companies' carnings calls serve as valu- able, yet unstructured, data sources be uti- lized to access companies" performance and risk profiles. Despite their importance, cur- rent works ignore the role of financial metrics knowledge (such as EBIT, EPS, and ROI) in transcripts, which is crucial for understanding companies' performance, and little considera- tion is given to integrating text and price in- formation. In this work, statistically alyze common financial metrics and create special dataset centered on these metrics. Then, we propose wewwwlede-enhancee financial volatility prediction method (KeFVP) to inject knowledge of financial metrics into text com- prehension by knowledge-enhancet adaptive mre-trannin (KePt) and effectively integrating text and price information by introducing con- ditional time series prediction module. Exten- sive experiments are conducted on three real- world public datasets, and the results indicate that KcFVP is effective and outperforms all the state-of-ae of-the-art methods.
Recent works (Qin and Yang. 2019; Sawhney et al., 2020c; Yang al., 2020; Sawhney al., 2020b) have also embarked on exploring the ap- proaches of utilizing these carnings calls to im- prove financial volatility predictions. Prior works based on earnings calls pay attention to multi- task architecture for predicting volatility and price movement (Sawhney et al., 2020c; Yang et al., 2020), correlations between stocks (Sawhney et al., 2020b), and the impact of numeric features (Yang et al., 2022). However, these studies either com- pletely overlooked the role of price (Yang et al., 2020, 2022: Qin and Yang. 2019), or did con- sider the combination of price and text information elaborately (Sawhney et al., 2020b,c). For instance, VoITAGE (Sawhney et al., 2020b) encodes price information by vanilla L.TM,, while Ensemble (Sawhney et al., 2020c) uses Support Vector Re- gression predict volatility based historical price information. Thus, based the preceding review, we propose two existential challenges in existing studies: (1) financial metric (FM) knowl- edge not concerned; (2) combining text and price information is rarely considered elaborately.
1 Introduction
The volatility of financial asset prices is typically considered valid proxy for the risk of financial assets and plays an essential role in evaluating the risk of financial assets and their derivatives (Yang et al., 2020). Predicting the volatility of financial assets is therefore of great significance to market participants. Mcanwhile, in addition to asset price information, wealth of unstructured data (c.g., news, social media, etc.) (Ding et al., 2014, 2015; Xu and Cohen, 2018; Duan al., 2018; Yang et al., 2019; Feng al., 2019) can also reflect potential changes in the future volatility of assets, which is also vital information that the market participants
Specifically, there are large number of FMs (e.g., EPS, EBITDA?, etc. shown in Table 1)
*Corresponding author
Pllasse refer to Table 9 in Appendix B for full names and descriptions
'The code is at http://gthhh.com//aaanni/////

---

Table 1: Statistics on datasets. #Ave. Sent. denotes the average number of sentences in each transcript. The column #Total TS gives the overall number of transcripts. Total Sent. denotes the overall count of sentences, while FM Sent. denotes the number of sentences containing FM. And FM Ratio is the ratio of the previous two.
Firstly, the text information is directed into the IE module, composed of multiple Transformer blocks, Following the module's processing, the refined text information, along with historical prices, is fed into the CTSP module to carry out volatility predictions. Our main contributions are as follows:
Datnett
ttm Sent.
Flaial TS
#Total Sext.
#FM Sent.
FM Ratio (%)
We first highlight the overlooked issue of dis- regarding FMs in existing studies. To counter- act this challenge, develop knowledge- enhanced adaptive pre-training (KcPt) method inject FMs knowledge into PLMs and con- struct a specific KePt dataset centered on FMs for adaptive pre-training.
291
61434
38150
17559
10576
$7963
54320
Tias
25532
MABC-15
Taxl
2571
2367
Oond
12457
16518
$0.44
Dnin
101
080
19246
S1622
$1.01
Tol
84
250
23557
11436
48.55
MARE-16
18cx
100
140
19338
7597
$4.04
Overal
58
1400
13686]
70655
51.63
We proposed an FVP model, equipped with IE and CTSP modules, designed to effectively amalgamate price and text information.
in the financial text to describe companies' perfor- mance in terms of earnings, cash flow, and assets and liabilities, which offer important assistance in analyzing companies' performance. However, to the best of our knowledge, there is rarely work that aims to use such FM knowledge to enhance finan- cial predictions. Secondly, the Efficient Market Hypothesis (EMH) (Malkiel, 1989; Sawhney et al., 2020a) suggests that financial markets are informa- tionally efficient, meaning that stock prices reflect all available market information. Consequently, in addition to historical prices, text information also affects stock prices. Thus, the concurrent integra- tion of price and text of paramount importance.
We perform evaluations using three real-world earnings call datasets, and our results establish new state-of-the-art (SOTA) benchmarks.
Related Work
2.1 Integrate Financial Knowledge into PLMs
In finance, financial metrics (FMs) serve as cru- cial indicators of understanding companies' perfor- mance, financial, and operating status when execu- tives or analysts read financial texts. Nevertheless, vanilla PLMs ignore the processing of FMs, and few researchers have recognized such challenge. Meanwhile, in the general field, injecting knowl- edge into PLMs during pre-training has been in- vestigated (Yu et al., 2022; Sun et al., 2020; Wang et al., 2021a) to some extent. Hence, we extract FM descriptions from Wikidata as knowledge and propose the KePt method to infuse such financial knowledge into PLMs during pre-training. To our best knowledge, it is the first attempt to focus on FM knowledge in the processing of financial texts.
In this work, we propose a novel approach, knowledge-enbanced financial volatility prediction (KeFVP), to tackle the challenges mentioned above. The overview of KeFVP is shown in Figure 1. Ini- tially, introduce a tive pre-training (KePt) method, designed to inject FM knowledge into Pre-trained Language Models (PLMs). To facilitate this, we construct a KePt dataset that merges financial corpora (e.g., TRC2- 3 and FiQA", etc.) and extracts specific descriptions of FMs from Wikidata for use in the KePt process. Subsequently, we employ the PLM Post-KePt to extract representations of each sen- tence from the earnings call transcripts. These representations are then incorporated into an end- to-end financial volatility prediction (FVP) model along with historical prices for volatility prediction. The FVP model consists of two major components: an information enhancement (IE) module and a conditional time series prediction (CTSP) module.
adap-
2.2 Earnings Call Data
Earnings calls present explanations of companies' performance, guidance for the upcoming quarter, and opportunities for in-depth Q&A, which pro- vides good window of communication between investors, brokerage analysts, and company man- agement (Keith and Stent, 2019). The earnings call datasets we used were released by (Qin and Yang, 2019; Li et al., 2020). Existing studies pri- marily focus on exploring three main aspects: (a) multimodal fusion, some works (Qin and Yang,
"hups:///stess google.co com/view/fiiu/hoome wwwp:/wwww......giiiiiiii.aaaaaa Page
"htps:///www.t httpps///wwww.oooaaaaaaaaaaoooe

---

2019; Sawhney al., 2020c; Yang et al., 2020) are dedicated to exploring the combination of text and audio modalities and its benefits for volatility predictions: (b) inter-company relationship, such as VoITAGE (Sawhney et al., 2020b), which uti- lizes graph neural networks to incorporate stock interdependence into prediction; and (c) the charac- teristics of financial texts, for instance, NumHTML (Yang et al., 2022) explores the importance of meric structure conveyed by numbers in texts for financial prediction. However, these works do not integrate price and text information elaborately, nor do they recognize the significance of FMs in texts.
jad, 2017; Zhang et al., 2022: Audrino et al., 2020; Chen al., 2020; Wang et al., 2021b). For instance, (Nonejad, 2017; Zhang et al., 2022) incorporates macroeconomic indicators into the volatility predic- tions; and (Audrino et al., 2020) finds that analyz- ing market emotions can enhance the effectiveness of volatility prediction.
Approach
The overview of KeFVV is shown in Figure 1, which is made up of two components: (1) knowledge-enhanced adaptive pre-training (KePt) (top); and (2) financial volatility prediction (FVP) model (bottom). The FVP model consists of two major modules: the (i.) information enhancement (IE) module (bottom(a)); the (ii.) conditional time series prediction (CTSP) module (bottom(b)).
2.3 Financial Prediction with Text
Financial predictions have been greatly enhanced by the incorporation of text information, such as financial news and social media. Current research be grouped into four distinct categories. (1) Event-based Prediction. These approaches leverage event information extracted from financial news to guide financial predictions. Significant works this domain include (Ding et al., 2014, 2015) and more recent developments (Yang et al., 2019; Deng et al., 2019). (2) Plain Text-based Prediction. Such methods (Duan et al., 2018; Xu and Cohen, 2018) involve learning directly from unstructured data such tweets or news documents, without pre-extracting structured events. (3) Inter-company Relationships. This genre of studies (Ang and Lim, 2022; Sawhney et al., 2020a; Xu al., 2021; Cheng and Li, 2021) takes into account the compayy relationships while considering text information. (4) Portfolio Management. This type of works (Liang et al., 2021; Sawhney et al., 2021a; Du and Tanaka-Ishir 2020; Sawhney al., 2021b) targets portfolio management problems instead of financial predictions by exploiting textual information.
3.1 Knowledge-enhanced Adaptive Pre-training (KePt)
We introduce the KePt method and the KePt dataset constructed answer the challenge of disre- garding financial metric (FM) knowledge.
3.1.1 KePt Dataset
To construct the KePt dataset, we collect Finan- cial PhraseBank 6, FIQA (both Taskl and Task 2), and EC dataset as financial corpora. Then, we ex- tract descriptions of frequent FMs from Wikidata. The key FMs, along with their descriptions and the frequency of their appearances across earnings call datasets and the KePt dataset, are displayed in Table 9 in Appendix B. Next, guided by our pre- compiled list of frequent FMs, we sift through the financial corpora to isolate sentences that include these metrics. These sentences are subsequently integrated into the KePt dataset. Accompanying descriptions relevant to these FMs are also incor- porated. A comprehensive statistic is provided in Table 8 in Appendix B. Please note that each FM is accompanied by a corresponding description, and every sentence comprises at least one FM. To foster future research endeavors, we will make the KePt dataset publicly available with our source code.
2.4 Stock Market Volatility Prediction
the stock market, volatility prediction plays a central role in risk management, asset allocation, and derivative pricing (Liang et al., 2022; Ma et al., 2019; Bollerslev et al., 2009; Epstein and Ji, 2013). In the field of finance, research on volatility pre- dictions primarily focuses on two aspects. Firstly, works in this aspect aim to construct forecasting methods based on widely used financial models such as GARCH and ARIMA (Dai al.. 2022; Spyridon D. Vrontos and Vrontos, 2021; Wang et al., 2016; Engle and Patton, 2007). The sec- ond aspect is innovation on the data side (None-
3.1.2 KePt Method
The KePt method is illustrated in Figure 1(top). We commence with the published BERTBASE (bert- pase-uncased") as our foundation and proceed
FinancialPhraseBank-vii
"httss//hugginnfacccooorrrt http://ggggggaacccoooooaaaaaaaaa

---

We demonstrate the formulation of the financial volatility prediction task based on earnings calls. For each stock, there exist multiple earnings calls, which are held quarterly in cycles. In this study, we focus solely on the impact of one of earnings calls on subsequent stock price volatility. Each carnings call transcript X7 (I1,..., xx)) con- sists of numerous sentences, where is the total number of sentences, detailed statistics are in Table Knnwwlegge-enbbeee enhanced Adaptive Pre- traininn (KePt)
Organic shine aquused usand EPG both coubllediiiee.
(LM11
Language Model
Kin
AK
SWiT
sent
FPG (oR poius warming aosonding
Kite
Language
Mosked Tasen Prodktin
Finannal Matr (910 Keowledge
Concal
LM2
Trand Rey Plae
Seueonal
Ad
Autaeer
Cundlt
KcPi
(uuzDec)
MARD-15, MABG-10
Frand
K
Trarforrer Bocks
Decompoeder
Azcbore: Croader MablCn
Inbirato
Financial Volatility Prediction (FVP)
Figure 1: The overall architecture of KeFVP, KePt at the top, FVP at the bottom.
conduct training using the KePt method. We desig- nate the embedding layer and the initial encoders the BERT BASE as Language Model (LM1) and assign the remaining layers to Language Model (LM2). LM1, which is used to encode both sen- tences and FMs, shares its weight parameters.
where [ha, hs] E Rxx2D is the stacked matrix of ha and h Next, we use h2 instead of the relevant part the FM words h2 in H. to form H., and feed H', into LM2 to continue pre-training.
Similarly, we employ the masked language model (MLM) objective to provide supervision sig- nals for training both LMI and LM2 simultane- ously. Specifically, randomly mask 15% of the tokens. When a token is masked, we substitute it with (1) the [MASK] token 80% of the time, (2) a randomly drawn token from the default glossary 10% of the time, (3) the unchanged token 10% the time. We use the KePt dataset to further pre- train the BERTBASE model and save the parameters of this model for future utilization. For more im- plementation details, refer Appendix C.
Concretely, the input sentence is represented S ([CLS], W1, [MASK] wN, [SEP]), while FMs in this sentence, and their descriptions, are represented as Kn ([CLS], SPym. [S [SEP]). Here, w denotes words in the sentence, SPym and Sdessc are FMs and their descriptions. We obtain representations H2 E N and ha E D by feeding S and Kn into LMI,
H2 (S).
h (K)) (1)
where N is the length of the input sentence, & is the number of FMs contained in this sentence and D is the hidden dimension. It should be pointed out that we treat the representation of [CLS] token of select FM word repre- sentations from H2 as h, E k which are from plain sentences S. To integrate knowledge during pre-training, we coalesce the representations of FM words from both plain sentences S and metric de- scriptions Kn. Specifically, we employ a Icarnable weight W E R2DxD to adaptively adjust the con- tributions of both b, and h, obtain the fused representation h. which is as follows.
3.2 Financial Volatility Prediction (FVP)
Kn as hg. Subsequently,
Given the input transcript Xy,, we first employ a text encoder to map them into proper representa- (2) tion space. We combine LMI and LM2 to act as
h = -W(hs he)),

---

the text encoder TEXTENC. Specifically,
As shown in Figure (bottom(b)), we employ N Autoformer encoders to model historical volatility series. Specifically, for the i-th Autoformer en- coder AUTOENC;()) the calculation is as follows:
Hr TEXTENC(XT),
where Hy RR denotes the representation of transcript, and d the dimension of hidden layers. Meanwhile, there also exists an adjusted clos- ing price* for each stock on each trading day. As- suming that an earnings call is announced on the day d, we collect adjusted closing price series P = (pa-1,.... Pa) for the time window / prior to the announcement of earnings calls as historical price data. Subsequently, we calculate historical av- erage log volatility series V (U1....,.)) Uz) based on Py (refer to the Appendix A for details).
V V,, V AUTOENC, (ORDEEN,,VVi... En
(8)
The output feature V En E /R xx will be provided to Autoformer decoder AUTODEC(-) and conditional attention module CONDATT(-) for prediction. We encapsulate AUTODEC(-) and CONDATT(-) as a conditional decoder, and we employ M conditional decoders. Following (Wu et al., 2021), we also employ a series decomposition module to decom- pose the time series into trend and scasonal items (Vt, V, E IR /xxd) (refer to Appendix D for details). For j-th Autoformer decoder AUTODEC,()). we use H H
Given the transcript representation Hy and his- torical volatility series V, for a specific stock, the objective of this task is to predict the average log volatility Vid,d r] at the future time period т.
cd to fill in the part to be predicted in the seasonal item &, form the conditional sea- sonal item v 8.C Then, we feed v Va,o along with Vi En into AUTODEC,()).
3.2.1 Information Enhancement (IE)
We adopt Transformer blocks to compose the in- formation enhancement module TRANSIE. Con- cretely, feed H. Hy attached with sentence mask Msent into L Transformer blocks.
V--
S.C. = -V H D,De
(9)
cd
V
vi AUTODEC, 1
&C VEn).
(10)
H TRANSIE Msent,
(4)
De 5,
V
v 1 v
(11)
D.De
where Myent is to indicate whether the position is real feature padding. For the 1-th Transformer block, the computation conducted follows:
where 0 De = V.. vo V,, and [:] is the concate- nation operation. To further capture the interaction between text and historical volatility information,
V,,
- = SELFATT 11 Msent),
(5) apply the CONDATT,() module to fuse H²11 and V beside AUTODEC,().
H 11 (H H 1 H = (H ie + FEEDFOWARD(H (
(6)
(7)
H = CONDATT , (Hi-1 Vin. V Vin)-
(12)
where SELFATT denotes self-attention mechanism and LN denotes layer normalization (Vaswani et al., 2017). Then, we obtain the transcript representa- tion 44-1 C Raax serving as the input to CTSP.
Specifically,
Q Lincar, (Ha (HiT)),
K. Hy Linears s.n) Linear (V'sn) (13)
3.2.2 Conditional Time Series Prediction (CTSP)
QK
H H Ill
Hy.
Vd
To predict future volatility, we input historical volatility information V, into the model, where I is the total number of timestamps. To jointly pro- cess time series and text information H after the IE module, we utilize a CTSP module based on Autoformer (Wu et al., 2021) to treat ! as a condition when making predictions. Here we de- fine Ho = - L- to carry out the later operation.
where H is the output of CONDATT,()) Then, also use H. to fill in the part to be predicted in the seasonal item v De of AUTODEC,
V
H Hy
(14)
s.C.
s.De
where V &.C is the conditional seasonal item for the next Autoformer decoder.
price.asp

---

Table The overall performance. The results with f. i and 5 are retrieved from (Yang et al., 2022), (Li et al., 2020) and (Sawhney al., 2020c) respectively, and the remainder except for KeFVP are from (Sawhney al., 2020b). KcFVP is the average result across 10 runs. KeFVP(Best) reports the best result in these 10 runs. M3 are MSE scores of different time periods, and M MSEI is the average over above. The form of A(B) denotes mean (for A) and standard deviation (for B). The best results are in bold, and the second-best restuls are underlined.
BC:
MAFE-IS
MARC-16
Model
MSS
MSSE:
MSS];
MSI
MS
MSS
MSB
MSN:
MSB
MSS
MSS
MSD,
MSD:
MS
MSI
Ypas
1.12
2.99
0.85
0.42
0:23
Price LSTM
0.75
1.97
0.46
0.32
0.24
BILSTM AFT
0.34
1.96
0:44
0.30
0.23
0.995
1.3507
0.5427
0:331
0.281
0.001
1.554
0L521
D.302
0.2867
HANiGllove)
0.60
1.4
0.45
0.31
0.20
AUMMMAAadioo
0.60
1.41
0:44
0.12
0.22
MDRMKTexHHAuddi
0.58
1.37
0.42
0.630
L425
0,4884
D:220F
0.255
0.18
1.426
0.478F
0:3:1
0.250
WTML
0:21
(TTML/eecAAudio)
0.40
088
0.4167
0.272
0.107
0.556
1.100
0.5157
0.314
VATAGE
0.63
0.29
0:31
KeFVP
0,308
(3.31e2)
0,610
(1.33e-2)
9.291
(0.59e-2)
0.183
(8.63-2)
0.114
0.364
(1.23--2)
0,418
(0.27e-2)
0.387
(8.30-2)
0.123
(0.17-22)
0,887
1.318
(6.36-2)
0.445
(4.42-2)
0.279
(3.66-2
0.363
(3.36-2)
0.377
SVMITF-DE7
1.70
0.50
0.34
0.25
0.59
1,42
0.41
0.30
Moln Plccor CNN
0.41
0.73
0.35
0.29
0.24
NunHTMLLTEaa Audaa
0.31
0,202
01601
0.308
0.181
0.114
KaFVFellett
0.276
11388
9.36
0.301
1.19%
0,400
6,182
0.117
0,84
1:318
0.347
0.194
0.23
6.126
3.3 Model Training and Inference
Price and text: HAN (Hu et al.. 2021), HTMLL(Text) (Yang et al., 2020), SVM(TF-IDF) (Tsai and Wang, 2014; Ding et al., 2014);
In the end, we add up the final obtained VM-1 and V.M-1, and employ fully connected layer as the prediction layer. Specifically,
Price, text, and audio: be-LSTM (Poria et al.. 2017), Multi-Fusion CNN (Sebastian and Pierucci, 2019), MDRM(Tex(+-Auddo)) (Qin and Yang, 2019), Ensemble(TexttAAudi)) (Sawhney et al., 2020c), NunHHTMLLLTTexttAAuddd (Yang et al., 2022), HTML(Tex(AAudo)) (Yang et al., 2020), VoITAGE (Sawhney al., 2020b).
I'n = Wp V-M-1) bp: (15)
where W E Rdxi and by E I are the weight matrix and bias, respectively. The objective is:
C E(0m 3mm)):
(16)
4.3 Main Results
where ym is the predicted volatility, and ym is the ground truth.
As shown in Table we report the main results compared with baselines. Following (Sawhney et al., 2020b; Yang et al., 2020; Qin and Yang, 2019), we chose MSE as the comparative metric (refer to the Appendix C for details). For a fair comparison, we report the average results of 10 runs and the best results of those 10 runs, as some baselines (Sawhney et al., 2020b) report average results, while others (Yang al., 2022; Li et al., 2020; Sawhney al., 2020c) do not. We make pre- dictions for 3, 7, 15, and 30-day periods following previous works. Although on the EC dataset, for the MS results, our method did not perform as well as expected, KeFVP outperformed all base- lines in terms of both average and best results for other time periods. For day 7 of the EC dataset and day 15 of the MAEC-16 dataset, KeFVP is basically on par with VoITAGE in terms of average results but exceeded its performance in terms of best results. In addition, it is worth noting that Ke- FVP outperforms all models that incorporate both text and audio information when only text is uti-
4 Experiments
4.1 Datasets
Following previous works, our experiments are conducted on EC (Qin and Yang, 2019), MAEC- 15, and MAEC-16 (Li et al., 2020) datasets (refer Appendix B for details). The statistics are dis- played in Table We partition each dataset into training, validation, and testing sets in a 7:1:2 ra- tio, consistent with prior works. We experiment on settings of n E {3, 7, 15, 30) days to explore short- to medium- and long-term performance. For implementation details, refer to Appendix C.
4.2 Baselines
We group baselines according to the information (price, text, and audio) they used for prediction.
Vpast (Qin and Yang, 2019), Price LSTM (Kim and Won, 2018) and BiLSTM + ATT (Siami- Namini et al., 2019);
Price:

---

Table 3: Results ablation study on the BC dataset. KeFVP denotes our overall method: KcFVP(wo CTSP & IE) denotes removing CTSP and IE, and only text remains; KeFVP(w/ Text) denotes removing text only historical price remains; KeFVP(w/ IE) denotes removing information enhancement (IE); KeFVP(w/o X) denotes using text embedding from raw BART (X KePt) and adaptively pre-trained PLMs without knowledge injection (X K Knowledge).
Pattern
Model
MSE
MSE3
MSE;
MSE15
MSE
KeFVP(wlo CTSP & IF))
0.392
0.743(2.94e-2)
0.380(1.72e22
0.258(1
0. 0,185(1.17e) 0.252().43e-2)
KeFVP(w/o Text)
0.331
0.647(2.55e-2)
(.328(1.52e-2)
0.210(1.32e-2)
0.138(1 0.138(1.35-22
FVP
KeFVP(w/o IE)
0.319
0.642(4480e-2)
0.308
0.198(1.60es
0.129(1.88e-2
KeFVP(Wo KePt)
0.318
0.644((.522-22
0.120(0.60e-2 0.196(1.23e-2) (..66()))))22)
KePt
KeFVP(wlo Knowledge)
0.319
0.650(3.11e-2
0.310(1.
0. (95((.)33))2)
0.120(0.69e2)
Overall
KcFVP
0.300
0.610(3.31e22)
0.291(1.33e22)
(.183(0.89c-2) 0.114(0.63e-2)
lized, which proves the superiority of our method.
Table 4:
Comparison with financial PLMs on the EC dataset. FVP(PLMS) denotes using text embedding from the corresponding PLMs. Corpus Size of FVP(KePt- BERT) and FVP(FinBERT) are counted by the number of sentences in each corpus, while CVP(FLANG-BER' is counted by the number of documents in each corpus.
4.4 Ablation Study
For KePt. As shown in Table 3, we conduct abla- tion studies to substitute KePt with text embedding counterparts: KeFVP(WWo KePt) and Ke/V Knowledge). Relative KeFVP, the effects of both KeFVP(W/o Knowledge) and KeFVP(w/o KePt) decrease, which indicates that neither the adaptive pre-training without knowledge nor the direct use of the raw published BERT is as effective as that using KePt.
Ciosa Sta
Mik
436-10
65221472-22
Wizn 14021
4233
1026
1199331631
01144155211
4.5 Financial PLMs
In this section, compare the performance of our KePt-BERT with two popular BERT-based finan- cial PLMs (FinBERT (Araci, 2019) and FLANG- BERT (Shah al., 2022)) on the volatility predic- tion task. The results are presented in Table FVP(FinBERT) is worse than FVP(KePt-BER) -BRRT) in this task. We denote KeFVP FVP(KePt- BERT) to clearly indicate the PLMs it utilizes. Note that FVP(FinBERT) is built on FinBERT (Araci, 2019), further training BERTBASE (Devlin et al., 2019) on large financial corpora (consists of finaniila,,ial, Financial PhraseBank, and FiQA dataset) (containing approximately 406,019 sen- tences in total) but neglecting FM knowledge, whereas FVP(KePt-BERT using KePt is based on much smaller corpus (KePt dataset, containing 8,732 sentences) with FM knowledge. This obser- vation underscores the potential of enhancing the efficacy of volatility prediction through the utiliza- tion of KePt to inject knowledge.
For FVP. To illustrate the effects of various com- ponents, ablation experiments are carried out on the EC dataset as shown in Table 3. KeFVP(w/o IE) drops significantly when we remove IE. It can be concluded that IE plays a significant role in en- hancing performance. For KeFVP(w/o Text), we exclude the effect of text information, meaning that only time series information is used. As can be observed, KeFVP(w/o Text) becomes significantly worse compared to KeFVP. Therefore, only histor- ical information is insufficient, and combining it with text information will help prediction. This also illuminates the capability of CTSP to effec- tively amalgamate text and time series information, yielding robust predictions. Furthermore, we in- vestigate the prediction performance relying solely on text information. Concretely, we substitute the CTSP and IE modules with fully connected layer and exclusively text information as input for prediction (i.e. KeFVP((Wo CTSP & IE)). It can be observed that KeFVP(W/o CTSP & IE) decreases a lot compared to KeFVP. Also, KeFVP(w/o CTSP & IE) drops substantially in contrast to KcFVP(w/o IE). This demonstrates the significance of CTSP.
Moreover, we conducted experiments involv- ing FLANG-BERT, designated as FVP (FLANG- BERT). Notably, it's important to acknowledge that the dataset used for training KePt-BERT is also notably smaller than that of FLANG-BERT. The specific corpus sizes are provided in detail within the Table 4. It is apparent that in this task, both Fin- BERT and FLANG- BEAN---BETTTT exhibit less favorable

---

Table 3: Comparison with non-text baselines.
SISE
MSE,
SSE,
MISE
Lineer Regresiion
0.622
(1995E3 2)
0.995..836 83655
SJR563.908 3)
0,40303 50e
GARCH
1.76
7884 88)
1,551 187)
ARIMA
GATI
(1
040000 1Ow
0:80119 78e-1)
0.311
1A 3)
27
KaFVP
0.300
6.1330.9--2
0.1546LA36-2)
performance compared to KcPt-BERT.
Figure 2:
Impact of text embeddings. The form of P(Q)
denotes predicted results (for P) and margins (for Q)
4.6 Non-text Baselines
Furthermore, we extend our experiments on the EC dataset to include non-text baselines, namely the classical Linear Regression, GARCH, and ARIMA models. These additional baselines are presented in the Table 5. Due data availability constraints and the need to ensure comparability, we have main- tained the use of historical price information as non-textual data, consistent with previous works. Notably, we have refrained from incorporating tex- tual data from earnings calls in this context.
prediction, and the preceding period representing the historical volatility series. Figure 2(b-e) demon- strate the predicted results of KcFVP, KeFVP(w/o Knowledge), KeFVP(w/o KePt), and Price Only (relying solely on the historical price), respectively, The prediction of KcFVP (in Figure 2(b)) is the closest the golden label. This proves that KePt is the most efficacious of all counterparts. The margin between the predicted result (-3. 3. 3.6089) and golden label (-4.2025) in Figure 2(c) is 0.5936, which is substantially greater than KeFVP (the margin is 0.2016), and also larger than the other counter- parts (KeFVP(W/o Knowledge) (0.5035) and Ke- FVP(w/o KePt) (0.5133)). This indicates that com- bining information from time series and text data can help predictions. Still, more importantly, the different levels of understanding of text data (differ- ent text embeddings) play a great role prediction.
Regarding the interpretability of transformer- based models, regression models offer intuitive explanations due to their fewer parameters. How- ever, their limited fitting capacity restricts their effectiveness. Transformer-basee models exhibit robust fitting capabilities, albeit with a larger pa- rameter count. These models can be explained vi- sually through methods like attention visualization to some extent, which aids in the intuitive under- standing of a substantial number of parameters and thereby contributes to result comprehension.
4.8 Evaluation on KePt
Simultaneously, we introduced KeFVP (w/o Text) as a point of comparison. This outcome rep- resents the results derived solely from the Auto- former model's treatment of time series data. In comparison to KeFVP, it is evident that the incor- poration of textual information yields significant enhancement in predictive performance compared with dealing with time series data in isolation.
Financial sentiment analysis serves as corner- stone in the realm of financial text mining, This task is dedicated to scrutinizing the emotional dy- namics within the financial market, and its efficacy is anchored in a deep understanding of financial texts. In this section, we apply our KePt method to financial sentiment analysis as a further testa- ment to its effectiveness. We employ three standard datasets for this purpose: heQAlneadline, FiQA- post, and PhraseBank. Comprehensive statistics along with experimental settings elaborated in Appendix E. For the FiQA-headline and FiQA- post datasets, our performance metrics are Mean Squared Error (MSE) and Square (R2). For the PhraseBank dataset, we gauge performance using Accuracy (Ace) and Macro-fl (F1) scores. The ensuing results are encapsulated in Table 6.
4.7
Case Study
As shown in Figure 2, we visually exhibit the in- fluence of different text embeddings for volatility predictions. We select the case related to Fidelity National Information Services (FIS Inc), an Ameri- can multinational corporation that offers financial products and services, from the EC dataset. All analyses are based on the 3-day prediction. Fig- ure 2(a) is the golden label of this case. Day 28 (the light gray vertical dashed line in the chart) marks the release date of the carnings call, with the subsequent period constituting the target of our
We compare KePt with two baselines: KePt(w/o Knowledge) and BERT all datasets, KePt outperforms, further attesting to its potency. When comparing KePt and KePt(w/o Knowledge) on the

---

Table 6: The results of KePt on financial sentiment analysis. KePt(w/o Knowledge) denotes adaptively pre- training on the KePt dataset without knowledge injec- tion.
Table 7: Comparison of time series models on the EC dataset. KeFVP(Y) denotes the KeFVP equipped with different time series models (CondTF = Conditional Transformer, CondLSTM Conditional LSTM).
FiQA-headine
FiQA-ppot
ParaseBank
MSl
Model
MSE 1 R²
MSE 1 R'
FI
KaVRRcodm 0.33
0.71C 36-3
BERTRASE
0.115
0.297
0.101
0.342
KI1.54
82.97
KeVVICCTPP
0,227
0.30
6418331e22
06616-55-21
0.291
0.200:2.866
0JJ83 (8:96-2)
0.1550 466-21
0.114 (1362)
0.1130:00: 3)
KePoowlo Knowledgo)
0.100
0.297
0.084
0.438
76.40
82.56
0.080
0.435
0.077
0.488
82.73
83,80
4.10 Impact of Different Time Series Models
To highlight the effect of CTSP, we conduct exper- iments employing different time series models in- stead of Autoformer. We replace Autoformer with Transformer by removing the series decomposition and replacing auto-correlation (Wu et al., 2021) with self-attention, and the remaining operations are in line with the rest of CTSP. We refer to this model as the conditional Transformer (CondTF), As demonstrated in Table 7, there is noticeable performance gap between KcFVP(CondTF) and KeFVP(C which illustrates merging price and text information under Autoformer is more effec- tive. addition, we borrow the Conditional LSTM (Sawhney et al., 2020b) as another countcrpart, which we called KeFVP(CondLSTMM) Overall, Ke- FVP(CTSP) is better than KeFVP(ComdLSTTM) For E KeFVP(CondL.STM is comparable to KeFVP(CTSP), while the standard deviation of KeFVP(CondI.STM) is much larger than Ke- FVP(CTSP), which indicates that the stability of KeFVP(CTSP) is superior to KeFVP(CondL We also add further case studies in Appendix to explore the impact of textual information on time series predictions.
Figure 3: Visualization of sentence embeddings. Con- taining FM denotes sentences with FMs while Missing FM refers to those without them.
and FiQA-post, the improvement for MSE and R² are 0.020 and 0.138 on FiQA- headline, and the improvement for these metrics are 0.007 and 0.05 on FoQst These improve- ments align with the FM Ratio (17.89% for FiQA- headline and 9.19% for FiQA-post) presented in Table 10 in Appendix E. KePt exerts more pro- nounced effect on datasets with higher FM Ratio, indicating the positive influence of incorporating FM knowledge on text understanding.
4.9 Sentence Embedding Visualization
To further elucidate the effect of KePt, this sec- tion provides visual demonstrations of sentence em- beddings from three ablation models: (a) KeFVP, (b) KeFVP(W/o Knowledge), and (c) KeFVP KePt). As shown in Figure 3, this case is from the EC dataset, the earnings call transcript issued by Fidelity National Information Services (FIS Inc) on February 7, 2017. The total sentence number of this transcript is 147, and there are 58 sentences con- taining FM and 89 sentences not containing. We apply PCA (Principal Component Analysis) to re- duce the dimensionality of sentence embeddings to represent them in a swo-dimensional space. Com- paring Figure 3(a) with (b) and (c), we can observe that Figure 3(a) has a more dispersed distribution whether for sentences containing FM sentences missing FM. So it can be inferred that injecting FM knowledge can not only improve the representation of sentences containing FM but also improve the representation ability of sentences missing FM.
5 Conclusion and Future Work
In this work, we present KeFVP and systematically illustrate the FM knowledge introduction into pre- dictions. The KePt method is proposed and the KePt dataset is concurrently constructed to serve it. The FVP equipped with IE and CTSP modules to integrate text and price information drives results to SOTA, Further, this method is also informative for research based on other financial text (e.g. news, tweets, etc.). In the future, we will explore ways to incorporate other financial knowledge (e.g., finan- cial analysis formulas) into financial applications.
Limitations
While our method underscores the significance of incorporating FM knowledge to enhance volatility

---

predictions and financial sentiment analysis, it is not without limitations that should be addressed. Primarily, the scope of our training data has con- strained the extent of performance improvement. Despite the advancements, the limited size of our dataset has inevitably affected our model's capac- ity to perform optimally, Some future work will focus augmenting the volume of training data to further enhance model performance.
Conference on Artificial Intelligence, AAAI 2021, Thirty-Thine Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Fleventh Symposium on Educational Advances in Artificial In- telligence, EAAI 2021. Virtual Event, February 2-9, 2021, pages 55 55-62. AAAI Press.
Zhifeng Dai, Tingyu Li, and Mi Yang. 2022. Fore- casting stock return volatility: The role of shrinkage approaches in data-rich environment. Journal of Forecasting, 41(5):980-99)
Secondly, our method is fundamentally built on encoder-based pre-trained models (BERT). How- ever, the applicability of our approach to other model architectures like decoder- based (e.g., GPT series) and seoodder-dccoder-baseee (e.g.. 15 series) models remains unexplored. We postulate that the introduction of FM knowledge may still be benefi- cial for these models, and hence, exploring this is promising direction for future research.
Shumin Deng, Ningyu Zhang, Wen Zhang, Jiaoyan Chen, Jeff Z. Pan, and Huajun Chen. 2019. Knowledge-dring stock trend prediction and expla- nation via temporal convolutional network. In Com- panion of The 2019 World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pages 678-685. ACM.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In NAACI. (1), pages 4171-4188 As- sociation for Computational Linguisties.
Acknowledgements
This work is funded in part by the National Natural Science Foundation of China Project (No.U1936213), the Shanghai Science and Tech- nology Development Fund (No.22dz) www.22ddz22070444 2007044, and CNKLSTISS.
Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. 2014. Using structured events to predict stock price movement: An empirical investigation. In Proceed- ings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, Octo- ber 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, Special Interest Group of the ACL. pages 1415- 1425. ACL
References
Xiao Ding, Yue Zhang, Ting Liu, and Junwen Duan. 2015. Deep learning for event-driven stock predic- tion. In Proceedings of the Twenty-Fourth Interna- tional Joint Conference on Artificial Intelligence, IJ- CAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pages 2327-2333. AAAI Press.
Gary Ang and Ee-Peng Lim. 2022. Guided attention multimodal multitask financial forecasting with inter- company relationships and global and local news. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume Long Papers), ACI. 2022, Dublin, Treland, May 22-27, 2022, pages 2313-6326. Association for Com- putational Linguistics.
Xin Du and Kumiko Tanaka-Ishii. 2020. Stock em- beddings acquired from news articles and price his- tory. and application to portfolio optimization. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 3353-3363. Associa- tion for Computational Linguistics.
Dogu Araci. 2019. Finbert: Financial sentiment analy- sis with pre-trained language models. arXiv preprint arXiv:1908.1000 10063.
Francesco Audrino, Fabio Sigrist, and Daniele Balli- nari. 2020. The impact of sentiment and attention measures stock market volatility. International Journal of Forecasting, 36(2)334-357.
Junwen Duan, Yue Zhang, Xiao Ding, Ching-Yio Yun Chang, and Ting Liu. 2018. Learning target-specific representations of financial news documents for mulative abnormal return prediction. In Proceedings of the 27th International Conference on Computa- tional Linguistics, COLING 2018, Sania Fe, New Mexico, USA, August 20-26, 2018, pages 2823-2833. Association for Computational Linguistics.
Tim Bollerslev, George Tauchen, and Hao Zhou. 2009, Expected Stock Returns and Variance Risk Premia. The Review of Financial Studies, 22(1 (2(11):4433 4492.
Liming Chen, Ziqing Du, and Zhibao Hu. 2020. Im- pact of economic policy uncertainty on exchange rate volatility of china. Finance Research Letters, 12:101266.
Robert F. Engle and Andrew J. Patton. 2007. 2 what good is a volatility model?*. John Knight and Stephen Satchell, editors, Forecasting Volatility in the Financial Markets (Third Edition), third edition edi- tion, Quantitative Finance, pages 47 -63, Butterworth- Heinemann, Oxford
Rui Cheng and Qing Li. 2021. Modeling the momen- tum spillover effect for stock prediction attribute- driven graph attention networks. In Thirty-Fifih AAAI

---

Larry G. Epstein and Shaolin Ji. 2013. Ambiguous Volatility and Asset Pricing in Continuous Time. The Review of Financial Studies, 26(7):1740-11)
Nima Nonejad. 2017. Forecasting aggregate stock mar- ket volatility using financial and macroeconomic pre- dictors: Which models forecast best, when and why? Journal of Empirical Finance, 42:131-154.
Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, and Tat-Seng Chua. 2019. Enhancing stock movement prediction with adversarial train- ing. In Proceedings of the Twenty-Eighth Interna- tional Joint Conference on Artificial Intelligence, IJ- CAI 2019, Macao, China, August 10-16, 2019, pages 5843-5849. ijeai.org.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Context-dependent sentiment anal- ysis in user essergeenrrated videos. In Proc. ACL, pages 873-883.
Yu Qin and Yi Yang. 2019. What you say and how you say it matters: Predicting stock volatility using verbal and vocal cues. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 390-401. Association for Computational Linguistics.
Yongli Hu, Puman Chen, Tengfei Liu, Junbin Gao, Yan- feng Sun, and Baocai Yin. 2021. Hierarchical atten- tion transformer networks for long document classifi- cation. In International Joint Conference on Neural Networks, IJCNN 2021, Shenchen, China, July 18-22, 2021, pages 1-7. IEEF.
Katherine A. Keith and Amanda Stent. 2019. Modeling financial analysts' decision making via the pragmat- ies and semantics of earnings calls. In Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 493-503. Association for Computational Linguistics.
Ramit Sawhney, Shivam Agarwal, Arnav Wadhwa, and Rajiv Ratn Shah. 2020a. Deep attentive learning for stock movement prediction from social media text and company correlations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, FMNLP 2020, Ouline, Novem- ber 16-20, 2020, pages 8415-8426. Association for Computational Linguistics.
Ha Young Kim and Chang Hyun Won. 2018. Fore- casting the volatility of stock price index: A hybrid model integrating LSTM with multiple garch-type models. Expert Syst. Appl., 103:25-37.
Ramit Sawhney, Piyush Khanna, Arshiya Aggarwal, Taru Jain, Puncet Mathur, and Rajiv Ratn Shah. 2020b. Voltage: Volatility forecasting via text audio fusion with graph convolution networks for earnings calls. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages Association for Computational Linguis-
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster).
Jiazheng Li, Linyi Yang. Barry Smyth, and Ruihai Dong. 2020. Maec: A multimodal aligned earnings confer- ence call dataset for financial risk prediction. In Proceedings of the 29th ACM International Confer- ence on Information amp; Knowledge Management, CIKM '20, page 3063-3070, New York, NY, USA. Association for Computing Machinery.
tics.
Ramit Sawhney, Puneet Mathur, Ayush Mangal, Piyush Khanna, Rajiv Rain Shah, and Roger Zimmermann. 2020c. Multimodal multi-task financial risk forecast- ing. In MM '20: The 28th ACM International Con- ference on Multimedia, Virtual Event / Seattle, WA, USA, October 12- 16, 2020, pages 456-465. ACM.
Chao Liang, Yan Li, Feng Ma. and Yaojie Zhang. 2022. Forecasting international equity market volatility: new approach. Journal of Forecasting, 41(7))1433- 1457.
Ramit Sawhney, Arnay Wadhwa, Shivam Agarwal, and Rajiv Rain Shah. 2021a. FAST: financial news and tweet based time aware network for stock trading. In Proceedings of the 16th Conference the the European Chapter of the Association for Computational Lin- guistics: Main Volume, EACL 2021, Online, April 19 23, 2021, pages 2164-2175 Association for Com- putational Linguistics.
Qianqiao Liang, Mengying Zhu, Xiaolin Zheng, and Yan Wang. 2021, An adaptive news-driven method for cvar-sensitive online portfolio selection in non- stationary financial markets. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 2708-2715. ij- cai.org.
Ramit Sawhney, Arnav Wadhwa. Shivam Agarwal, and Rajiv Ratn Shah. 2021b. Quantitative day trading from natural language using reinforcement learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 4018 4030. Association for Computational Linguis- ties.
Feng Ma, Yaojie Zhang, M. I. M. Wahab, and Xiaodong Lai. 2019. The role of jumps in the agricultural fu- tures market on forecasting stock market volatility: New evidence. Journal of Forecasting, 38(5):400- 414.
Burton Malkiel. 1989 Efficient market hypothesis. In Finance, pages 127-134. Springer.

---

Jilt Sebastian and Piero Pierucci. 2019. Fusion tech- niques for utterance-lever emotion recognition com- bining speech and transcripts. In Proc. Interspeech, pages 51-55.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition transform- ers with auto-correlation for long-term series fore- casting. In Advances in Neural Information Pro- cessing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pages 22419-22430.
Raj Sanjay Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Ra- man, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. When flue meets flang: Benchmarks and large pretrained language model for financial domain. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties.
Wentao Xu, Weiqing Liu. Chang Xu, Jiang Bian, Jian Yin, and Yan Liu. 2021. REST: relational event- driven stock trend forecasting. In WWW '21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19.23, 2021, pages 1-10. ACM IW3C2.
Sima Siami-Naming Neda Tavakoli, and Akbar Siami Namin. 2019. A comparative analysis of forecasting financial time series using arima, Istm, and bilstm. CoRR, abs//911..00 1.09512.
Yumo Xu and Shay B. Cohen, 2018. Stock movement prediction from tweets and historical prices. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics, ACL 2018, Mel- bourne, Austrulia, July 15-20, 2018, Volume I: Long Papers, pages 1970-1979. Association for Computa- tional Linguisties.
John Galakis Spyridon D. Vrontos and Toannis D. Vron- tos. 2021. Implied volatility directional forecast- ing: aÅ machine learning approach. Quantitative Finance, 21(10):1687-1766
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang. and Zheng Zhang. 2020. Colake: Contextualized language and knowledge em- hedding. In Proceedings of the 28th International Conference Computational Linguistics, COLING 2020, Barcelona. Spain (Online). December 8-13, 2020, pages 3660-3670. International Committee on Computational Linguistics.
Linyi Yang. Jiazheng Li, Ruihai Dong, Yue Zhang, and Barry Smyth. 2022. Numhtml: Numeric-orienter hierarchical transformer model for multi-task finan- cial forecasting. In Thirty-Sixthh AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty- Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium Educutional Advances in Artificial Inielligence, EAAI 2022 Virtual Event, February 22 MMarch 1, 2022, pages 11604-11612. AAAI Press.
Ming-Feng Tsai and Chuan-Ju Wang. 2014. Financial keyword expansion via continuous word vector repre- sentations. In P Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, Special Interest Group of the ACL, pages 1453-1458. ACL.
Linyi Yang. Tin Lok James Ng. Barry Smyth, and Rui- hai Dong. 2020. HTML: hierarchical transformer- based multi-task learning for volatility prediction. In www 20: The Web Conference 2020. Taipei, Taiwan, April 20-24, 2020, pages 441 441-451.4 ACM IW3C2.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9. 2017, Long Beach, CA, USA, pages 1998-60088
Linyi Yang, Zheng Zhang, Su Xiong, Lirui Wei, James Ng. Lina Xu, and Ruihai Dong 2019. Explain- able text-driven neural network for stock prediction. CoRR, htt//992.04999/
Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. 2022. JAKET: joint pre-training of knowledge graph and language understanding. Thirty-Sixth AAAI Conference on Artificial Intelli- gence, AAAI 2022, Thirty-Fourth Conference In- novative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Ad- vances in Artificial Intelligence. EAAI 2022 Virtual Event, February 22 March 1, 2022, pages 11630- 11638, AAAI Press.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang. Zhiyuan Liu, Juanzi Li, and Jian Tang. 202 la KEPLER: A unified model for knowledge embed- ding and pre-trained language representation. Trans. Assoc. Comput. Linguistics, 9:176-194.
Xinyu Wang, Yi Luo. Zhuqing Wang. Yan Xu, and Congxin Wu. 2021b. The impact of economic pol- icy uncertainty on volatility of chinaå financial stocks: An empirical analysis. Finance Research Letters, 10:601.
Lixia Zhang, Qin Luo, Xiaozhu Guo, and Muhammad Umar. 2022. Medium-term and long-term volatility forecasts for eua futures with country-specific eco- nomic policy uncertainty indices. Resources Policy. 77:102644.
Yudong Wang, Feng Ma, Yu Wei, and Chongfeng Wu. 2016. Forecasting realized volatility in changing world: A dynamic model averaging approach. Jour- nal of Banking Finance, 14-199..

---

Table Statistics on dataset KePt. The overall number of sentences in KePt is under #Total Sent., and the Ave. Length denotes the average length of sentences. The overall number of FMs KePt and the average number of FMs in each sentence are under #Total FM. and # Ave. FM., respectively.
transcript is divided into series of sentences, and the detailed statisties are displayed in Table 1. In addition, intuitively, we also illustrate the statisties of major financial metrics (FMs) for this dataset in Table 9. The stock prices (time series data) are extracted from Yahoo Finance in the time frame from January' 17 to 31 December's 17.
Dataset
#Total Sent.
Ave. Length
#Total FM. #Ave. FM.
KePi
8732
156.19
11442
MAEC-15, MAEC-16 Following the previous work (Li et al., 2020), we also conduct experiments on the datasets published in their work, which we refer as MAEC-15 and MAEC-16. As they do not publish processed historical price data, we grabbed the corresponding historical prices for the period when the earnings call was issued via Yahoo Finance (we took time window of 30 days be- fore and after). We then use the volatility formula (17) to calculate the volatility of these prices. We will also release the processed price dataset in our source code.
Mayo
Figure 4: Occurrence frequency of main FMs the KePt dataset.
We also provide detailed statistics of the KePt dataset and main FMs for both the three carnings call datasets and KePt dataset. The detailed statistic of KePt dataset is displayed in Table 8. The occur- rence frequency and detailed description of each main FM for both earnings call datasets and the KePt dataset are listed in Table 9. Also, for visual illustration,
A Volatility Calculation
Following (Sawhney al., 2020b), for specific stock, its adjusted closing price on the trading day is Pj. and the average log volatility over trading day det to trading day dst + ô is calculated as follows:
provide distribution maps of the main FMs over these datasets (Figure and Figure 4). It can be found that the frequency distribution of FMs is basically the same all datasets.
Edatt (T3 r)2
Vldat, dot 15] = In(1
(17)
where the return of trading day j is defined as r Pi-1 and F is the mean of return over trading day dst to trading day dat + 8.
C Implementation Details
Pj
The KePt is based the pre-trained BERT BASE (Devlin al., 2019), the BERT containing 12 hid- den layers, and 768 hidden dimensions for each layer. The number of the epoch is 60 for KePt. We use the AdamW optimizer while training with the learning rate initialized by 2e-5. For FVP, we still employ the Adam optimizer (Kingma and Ba, 2015) and initialize the learning rate to 2e-4, weight decay to 0.05, and the number of training epochs is 200. We conduct experiments GeForce GTX 1080Ti, and our codes are imple- mented based on Pytorch. The average computa- tion time on the GPU (1080Ti) is 5 hours for KePt and 15 minutes for FVP, Our source code for both KePt and FVP will be released later.
For the historical price data P,
we follow previous works (Qin and Yang. 2019; Sawhney et al., 2020c; Yang et al., 2020; Sawhney al., 2020b) to calculate the historical average log volatility U/a.r,4 for the time window according Equation (17), where det - I and E (1.2..... I}. For simplicity, we abbreviate V, as V, (vi,..., vy) taking the values of ó as subscripts.
(pa-I....,Pal
VI
NVIDIA
B Dataset Analysis
EC Following previous works, our experiments are conducted on the EC dataset, a publicly avail- able earnings call dataset released by (Qin and Yang, 2019). The dataset contains 559 carnings call transcripts for 277 S&P 500 companies. Each
Following (Sawhney et al., 2020b; Yang et al., 2020: Qin and Yang. 2019), treat the volatility
Phttps//financcyyaaoo.om

---

Table 9: Statistics of the main FMs for three public datasets and our KePt dataset. The Description is the explanation extracted from Wikidata for specific FM, and the Frequency denotes the number of times the FMs appear in the dataset. Here we only list the FMs that appear more frequently.
FM
Doseriptions
BC
MABC-15
Freguency
MADC-16
Kelt
EPS
Farsings per shone, valae of carinngs per ousoirndnn shons of commen stock for compuny
976
337
628
1194
EDIT
25
175
FATTISA
dipersisionn amoriistion like rornings rabio, the ratlo of company s store price to the company a cornings per shore
exponees,
377
Fi
16
238
RUI
Retuan Evvetisent, roto betwgen the net poods and post of invrssimett resultngg from invesment of some resources
34
24
25
402
CO0S
Cve gools sold,
posls soul derin putticalar company 35sets generating tocome
31
95
Ketum assets, ratio to expeess the protiability
Leverage
The use of borzowed funds zuhher than fresh equity in the purciase of assct
552
414
674
1898
Georing
Levenge, the use bornowed lunds rither than fresh equiy the purchase of an 35901 words.
13
50
Profit Margin
Pront margin the
poodi.
39
192
percentage of tumower comains profit for the compuny
Income Statemeni
Financial statenest of comgaey: alows the company rivemues and expenies chrring particulir period
401
26
39
204
Balance Sheet
Accousting fasacial summacy
440
283
606
1154
pediiahili busines
210
630
Crsh Flow
Menement muney into out hosness, Pkoted, linacial prodinct
451
Tax Rate
Ratio (isually expreseed is pereentage) which buiiees or person is usedd Rabning speraingg pullis oond uile Resning prus
597
195
354
2085
Operating Margin
527
137
207
545
Mingin
321
Table 10: Statistics on the three datasets for financial sentiment analysis. #Sent. denotes the total number of sentences in each dataset. The overall number of FMs in each dataset is under # FM, and FM Ratio is the ratio of the previous two. In addition. FM Sent. denotes the number sentences containing FMs, and FM / FM Sent. indicates the average number of FMs in each FM Sent..
Datnent
Sent.
FM
FM Ratio (5)
IPM Sent.
FMI PM Seet.
Tisss
21110
zon.com Inc daily volatility series from 2017/04/01 to 2017/12/31. Subfigure (a) is the volatility series with- out decomposition, and (b) and (c) are the decomposed seasonal and trend series, respectively.
Figure 5: Time series decomposition of Ama-
FIJA headline
Oveudl
Dex
Tes:
436
44
57
78
22:73
17.89
5.85
75
1.84
1.11
Traie
473
50
10.57
45
1.11
Test
135
15
11.11
13
1:35
Dex.
67
5.97
Overall
625
5.19
62
1,11
Triie
3190
764
23.11
Test
969
205
21.16
176
1314
prediction as a regression task, and we also chose the Mean Square Error (MSE) as the comparative metric in previous works. The MSE is computed follows:
seasonal series excluding trend factors are more reflective of volatility, and their integration with text information will likely amplify this advantage. For the sake of such reasons, we use H to fill in the part to be predicted in the seasonal item v s,De to form the conditional seasonal item v for the j-th Autoformer decoder AUTODEC()). s,C
(M)-
MSE-
(18)
where ÿ is the predicted volatility, y1 the ground truth.
D Time Series Decomposition
E
Financial Sentiment Analysis
The decomposition process is:
E.1 Dataset Statisties
V, = AVGPOOL
(19)
As shown in Table 10, we performed statistics on the three financial sentiment analysis datasets FFQAA-headlig FiQA-post 10 and PhraseBank "1) we used. We analyzed the FM Ratio in each dataset, there are higher FM Ratios in FiQA-headline and PhraseBank, but a lower Ratio in FiQA-post. This
V, V1 V,,
where V,, V. E /xd denote trend and seasonal items, respectively. AvGPOOL(-) denotes the mov- ing average pooling, and PADDING(.) is to keep the time series length constant.
As shown in Figure 5, such time series decompo- sition makes seasonality more apparent and facili- tates the merging with text information. Intuitively,
ll FinancialPhruseBanke/l/

---

Distribution metrics datasets
metrics
three datasets
LEA
HETTA
Haw
Nut bure
ERTOA
Prott
Exlanee Short
Met incaia
Cosh Teee
Orn Margi
Margi
Margh
Figure 6: Occurrence frequency of main FMs the three earnings call datasets.
Table 11: The impact of different text information on time series modeling. These cases are from the EC dataset, and the results based on 3-day predictions.
Case
Model
Top-8 Similar Points
Predictions
Absolute Difference
Ground Truth
BERT BASI:
3,
14, 22, 26]
-4,4576
0.1055
-4.3521
Curp (20171031)
Knowledge)
3,
26]
-4,4224
KePt
30,
11,
22]
-4,4121
0,0600
.4.3521
BERTRSSE
[1. 11,1,
18,
25, 28)
-.3885
0.1673
-4,5558
Martin Marietta (201 71102)
Knowledge)
21,
28]
KePt
14.14.11
18,
24,
28
-4,4080
0.1478
-4.5558
BERT
[4.
12,
15, 19,
24,
28]
4.1397
0.7434
.333963
Apache Corporation (20171102)
Kel Knowledge)
14,9, 12,
15,
28]
-4.1442
0,7479
-3.3963
KePt
7. 12,
15,
19,
25,
28]
-4,0600
0.6897
.3.3963
statistical result FiQA-headline and FiQA-post is consistent with the effect of KePt on the two datasets, which shown in Table
F Text Information and Time Series
Within the CTSP module, we introduced the Con- ditional Attention (CondAtt) module atop the Aut- oformer framework, aimed at incorporating text information. Within the CondAtt module, diverse text inputs yield varied impacts time series mod- eling. Our analysis is presented in the Table 11. We delved into different text models (BERT BASE, KePt (w/o Knowledge), and KePt) within CondAtt, and examined their conditional attention with historical time series information. We show the Top-8 data points of the time series according to text-to-time series attention weights (the overall length of the historical time series data point is 29) for analysis. We also provide the Ground Truth, predicted results (Predictions). and the absolute difference between the Ground Truth and Predictions (Absolute Dif- ference) for each case. evident that distinct textual inputs capture diverse time series informa- tion, thereby influencing time-series predictions. It is worth noting that KePt's text input improves the modeling of time series information, resulting in Predictions that are closer the Ground Truth and have smaller Absolute Difference.
We partitioned the dataset into training, vali- dation, and test sets following 7:1:2 ratio for each dataset. In the FiQA-headline and FiQA-post datasets, due to the unavailability of golden labels in the official test set, we partitioned 20% of the training set as a dedicated test set. In addition, We adopted the version of the dataset for which more reached the compre-
than 50% agreement hensive dataset for PhraseBank. This dataset was then partitioned according to the aforementioned ratios for training, validation, and testing.
E.2 Experiment Settings
We fine-tuned each type of pre-trained model (BERTBASE, KePt(w/o Knowledge) and KePt), along with an additional classification head, sepa- rately on the three datasets. The number of epochs 10, 10, and respectively for FiQA-head, Foott and ParaseBank. The learning rate is 2e-5 by using the Adam optimizer with the default settings. The batch size is 32. We conduct experiments on the NVIDIA Tesla V100.
for fine-tuning


Reviews:
[{"review_id": "EXG71vq93A", "reviewer": "Reviewer_QcTY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you so much for your detailed comments and affirmation of our approach.\n\nWe have compiled a vocabulary of 253 Financial Metrics (FM) along with their corresponding knowledge descriptions. Utilizing Spacy's EntityRuler functionality, we performed Named Entity Recognition (NER) to identify FM terms within sentences. This process enabled us to filter sentences containing FM terms, thereby forming the KePt dataset. In the camera-ready version, we will augment this detail and provide a more detailed description of the dataset construction process.\n\nIn fact, in the manuscript of this paper, we presented the average results (Mean(Std)) of 10 runs along with their corresponding standard deviations in Tables 2, 3, and 5.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well-written and easy to follow.\n- The proposed solution is novel and sound, particularly, constructing the KePt dataset, which is useful to enhance PLMs for financial tasks and will make the dataset publicly available.\n- The analysis is comprehensive, including ablation study and case study.\n\n", "reasons_to_reject": "- A few parts lack details. E.g., Line 220, we sift through the financial corpora to isolate sentences that include these metrics. (how to identify these metrics).\n\n- It is good to run the experiments multiple times and report the mean and std.", "questions_to_authors": "", "paper_topic_and_main_contributions": "To address the challenges of financial volatility prediction tasks, this paper presents a novel method called FVP (Financial Volatility Prediction). In specific, the authors develop a method called KePt to enhance PLMs with financial features, such as EPS, EBIT ROI etc. Second, the FVP model considers historical price and text information. The experiments are conducted on three real-world datasets (Earings Call Data) and show that the proposed solution has significant improvement compared with a number of SOTA benchmarks. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1CkHIVaCFc", "reviewer": "Reviewer_HZuA", "summary": "", "questions": "Question 1: Can you explain the motivation of the Information Enhancement module?\n\nQuestion 2: Both Autoformer Decoder(AutoDec) and Conditional Attention(CondAtt) capture the interaction between text and historical volatility information. What’s the differences? Why use these two components in conditional decoder?\n\nQuestion 3:\n\nLine 489, in Figure 2, the light gray vertical dashed line should be Day 28;\n\nLine 536, not 0.161, from Table 4, FiQA-post on MSE (0.084-0.077=0.007);\n\nIs that right?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your comments and affirmation of our method.\n\n## __1: Relationship with ERNIE__\n\nWe have identified two distinct aspects of our approach:\n\n### Divergence in Motivation from ERNIE:\nOur motivation diverges from ERNIE's focus. We aim to enhance Volatility Prediction by enabling pre-trained models to better comprehend the content of earnings calls. During our investigation, we observed that over 50\\% of sentences in earnings calls encompass financial metric words (FM), such as EBIT and EPS, as shown in Table 1 in our manuscript. These FMs hold paramount importance in understanding a company's performance. Hence, our strategy involves incorporating detailed descriptions pertaining to these FMs as knowledge. This augmentation seeks to elevate the effect of models in Volatility Prediction tasks based on earnings call text.\n\n### Incorporating Sentence-level Information:\nThe FM knowledge we incorporate consists of elaborate descriptions rather than spans, in contrast to ERNIE. These descriptions are full sentences. Our methodology indirectly validates the significance of incorporating sentence-level information. This is also effective in improving the performance of the model, which makes it possible to add more information in our way.\n\n## __2: About Conditional Time Series Prediction (CTSP) module and Question 2__\n\nWithin the CTSP module, we introduced the Conditional Attention (CondAtt) module atop the Autoformer framework, aimed at incorporating text information. CondAtt is designed to enable textual data to influence time-series predictions. Unlike the Autoformer decoder (AutoDec) (consistent with the Autoformer ), which only captures time series information, the addition of CondAtt enables textual information to influence time series modeling. Our experiments involving CondAtt are shown in the table below.\n\nIn addition to the Autoformer model, we extended the implementation of CondAtt to various other transformer-based time series models (CTFM = Conditional Transformer, CInFM = Conditional Informer, CFedFM = Conditional FEDformer, CAtFM = Conditional Autoformer). DeFVP(Y w/o CondAtt) denotes the variant without CondAtt(·). We conducted ablation studies on the EC dataset, revealing that the inclusion of CondAtt significantly enhances the performance of these transformer-based models when combining textual information for improved time series modeling.\n\n* Informer: Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong,\nand Wancai Zhang. 2021. Informer: Beyond Efficient Transformer for Long 1034 Sequence Time-Series Forecasting. In Thirty-Fifth AAAI Conference on Artificial 1035 Intelligence, AAAI 2021.\n\n* FEDformer: Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. 1039 FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. In International Conference on Machine Learning, ICML 2022.\n\n| Model                     | MSE(Average) | MSE_3          | MSE_7          | MSE_15       | MSE_30       |\n|---------------------------|---------------|----------------|----------------|----------------|----------------|\n| DeFVP(CTFM w/o CondAtt)   | 0.845         | 1.184 (1.88e-1)| 0.829 (1.69e-1)| 0.722 (1.87e-1)| 0.646 (1.86e-1)|\n| **DeFVP(CTFM)**            | 0.381         | **0.725 (3.66e-2)**| **0.369 (1.44e-2)**| **0.247 (1.14e-2)**| **0.182 (1.12e-2)**|\n|---------------------------|---------------|----------------|----------------|----------------|----------------|\n| DeFVP(CInFM w/o CondAtt)  | 0.989         | 1.334 (3.49e-1)| 0.953 (3.24e-1)| 0.840 (3.37e-1)| 0.830 (3.46e-1)|\n| **DeFVP(CInFM)**           | 0.369         | **0.705 (3.07e-2)**| **0.351 (1.71e-2)**| **0.240 (1.03e-2)**| **0.178 (1.31e-2)**|\n|---------------------------|---------------|----------------|----------------|----------------|----------------|\n| DeFVP(CFedFM w/o CondAtt) | 0.353         | 0.670 (8.09e-2)| 0.367 (4.26e-2)| 0.242 (5.30e-2)| 0.134 (0.93e-2)|\n| **DeFVP(CFedFM)**          | 0.311         | **0.610 (3.00e-2)**| **0.304 (1.92e-2)**| **0.202 (2.38e-2)**| **0.126 (0.89e-2)**|\n|---------------------------|---------------|----------------|----------------|----------------|----------------|\n| DeFVP(CAtFM w/o CondAtt)  | 0.331         | 0.647 (2.55e-2)| 0.328 (1.52e-2)| 0.210 (1.32e-2)| 0.138 (1.35e-2)|\n| **DeFVP(CAtFM)**           | **0.300**     | **0.610 (3.31e-2)**| **0.291 (1.33e-2)**| **0.183 (0.89e-2)**| **0.114 (0.63e-2)**|\n\n\nMoreover, within the CondAtt module, diverse text inputs yield varied impacts on time series modeling. Our analysis is presented in the following Table. We delved into different text models (BERT(base), KePt (w/o Knowledge), and KePt) within CondAtt, and examined their conditional attention with historical time series information. We show the Top-8 data points of the time series according to text-to-time series attention weights (the overall length of the historical time series data point is 29) for analysis. We also provide the Ground Truth, predicted results (Predictions), and the absolute difference between the Ground Truth and Predictions (Absolute Difference) for each case. It is evident that distinct textual inputs capture diverse time series information, thereby influencing time-series predictions. It is worth noting that KePt's text input improves the modeling of time series information, resulting in predictions (Predictions) that are closer to the Ground Truth and have a smaller Absolute Difference.\n\nWe intend to incorporate these two sets of experiments into the camera-ready version of the paper.\n\n| Case                                    | Model               | Top-8 similar points            | Predictions | Absolute Difference | Ground Truth |\n|-----------------------------------------|---------------------|----------------------------------|-------------|--------------------|--------------|\n| American Tower Corp A (20171031)        | BERT(base)           | [0, 3, 7, 8, 11, 14, 22, 26]    | -4.4576     | 0.1055             | -4.3521      |\n|                                         | KePt(w/o Knowledge) | [0, 3, 7, 8, 11, 14, 22, 26]    | -4.4624     | 0.1103             |    -4.3521          | \n|                                         | KePt                | [0, 3, 7, 8, 11, 14, **18**, **22**]    | -4.4121     | **0.0600**             |      -4.3521        |\n|-----------------------------------------|---------------------|----------------------------------|-------------|--------------------|--------------|\n| Martin Marietta Materials (20171102)    | BERT(base)           | [1, 11, 15, 18, 21, 24, 25, 28] | -4.3885     | 0.1673             | -4.5558      |\n|                                         | KePt(w/o Knowledge) | [1, 11, **14**, 18, 21, 24, 25, 28] | -4.3939     | 0.1619             |      -4.5558        |\n|                                         | KePt                | [**4**, **14**, 15, 18, 21, 24, 25, 28] | -4.4080     | **0.1478**             |      -4.5558        |\n|-----------------------------------------|---------------------|----------------------------------|-------------|--------------------|--------------|\n| Apache Corporation (20171102)           | BERT(base)           | [4, 9, 12, 15, 19, 24, 25, 28]  | -4.1397     | 0.4734             | -3.3963      |\n|                                         | KePt(w/o Knowledge) | [4, 9, 12, 15, 19, 24, 25, 28]  | -4.1442     | 0.7479             |        -3.3963       |\n|                                         | KePt                | [**2**, **7**, 12, 15, **16**, **19**, 25, 28]  | -4.0860     | **0.6897**             |       -3.3963        |\n\n## __3: About Information Enhancement (IE) module__\n\nAs demonstrated in Table 1 (Statistics on datasets) in our manuscript, the datasets EC, MAEC-15, and MAEC-16 encompass a substantial number of sentences within their earnings call transcripts, averaging around 100 sentences per transcript. Consequently, prior to integrating textual information into time series prediction modules like CTSP, it becomes essential to further model the wealth of sentence-level information to capture crucial insights. In this regard, we introduce the Information Enhancement (IE) module. This module employs attention mechanisms at the sentence dimension. The objective is to distill the vital elements from the multitude of sentences in each transcript, thus better serving the CTSP module.\n\nIn Table 3 of our manuscript, the ablation studies on the IE module are presented. The results underscore the significant role of the IE module in performance enhancement. In the camera-ready version, we intend to visualize the attention weights within the IE module, offering a more intuitive depiction of how the IE module aids in capturing key information from the numerous sentences.\n\nResults of ablation study on EC dataset. **KeFVP** denotes our overall method; KeFVP(w/o CTSP \\& IE) denotes removing CTSP and IE, and only text remains; KeFVP(w/o IE) denotes removing information enhancement (IE).\n| Pattern | Model                 | MSE(Average) | MSE_3          | MSE_7          | MSE_15       | MSE_30       |\n|---------|-----------------------|---------------|----------------|----------------|----------------|----------------|\n| FVP     | KeFVP(w/o CTSP & IE)  | 0.392         | 0.743 (2.94e-2)| 0.380 (1.72e-2)| 0.258 (1.43e-2)| 0.185 (1.17e-2)|\n|         | KeFVP(w/o IE)         | 0.319         | 0.642 (4.80e-2)| 0.308 (2.08e-2)| 0.198 (1.60e-2)| 0.129 (1.38e-2)|\n|---------|-----------------------|---------------|----------------|----------------|----------------|----------------|\n|         | **KeFVP**              | **0.300**     | **0.610 (3.31e-2)**| **0.291 (1.33e-2)**| **0.183 (0.89e-2)**| **0.114 (0.63e-2)**|\n\n\n## __4: Other Concerns__\n\nThanks for pointing out these issues. I understand that minor oversights can happen during the writing process. We are planning to address these issues in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Inject financial metrics knowledge into a pre-trained language model.\n\n2. Design a model to combine price and text information for financial volatility prediction.", "reasons_to_reject": "1. Injecting knowledge into a pre-trained language model is similar to ERNIE（Enhanced Language Representation with Informative Entities ）.\n\n2. The Conditional Time Series Prediction module（bottom(b) of figure 1） seems a little complicated.", "questions_to_authors": "Question 1: Can you explain the motivation of the Information Enhancement module?\n\nQuestion 2: Both Autoformer Decoder(AutoDec) and Conditional Attention(CondAtt) capture the interaction between text and historical volatility information. What’s the differences? Why use these two components in conditional decoder?\n\nQuestion 3:\n\nLine 489, in Figure 2, the light gray vertical dashed line should be Day 28;\n\nLine 536, not 0.161, from Table 4, FiQA-post on MSE (0.084-0.077=0.007);\n\nIs that right?", "paper_topic_and_main_contributions": "Topic:\n\nKnowledge-enhanced Financial Volatility Prediction. This paper proposes a knowledge-enhanced financial volatility prediction method (KeFVP) to inject knowledge of financial metrics into text comprehension by knowledge-enhanced adaptive pre-training (KePt) and effectively integrating text and price information by introducing a conditional time series prediction module.\n\nContribution:\n\nThis paper develops a knowledge-enhanced adaptive pre-training (KePt) method to inject FMs knowledge into PLMs and construct a specific KePt dataset centered on FMs for adaptive pre-training.\n\nThis paper proposes an FVP model, equipped with IE and CTSP modules, designed to amalgamate price and text information effectively.\n\nThis paper performs evaluations using three real-world earnings call datasets, and the results establish new state-of-the-art (SOTA) benchmarks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "U3iE42ybxa", "reviewer": "Reviewer_8Q4Z", "summary": "", "questions": "Q: Why preferential masking of financial words was not performed? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Dear reviewer, thank you for your comments and affirmation of our method.\n\n## __1: Use of weak baselines 1: FinBERT, FLANG-BERT baselines, and preferential masking__\n\nIn regard to FinBERT, Table 3 in the manuscript presents the performance of FinBERT on the Volatility Prediction task. Additionally, based on the description in the FinBERT paper, FinBERT was pre-trained using the MLM approach on financial datasets. Consequently, the pretraining methodology of FinBERT aligns with that of KeP(w/o Knowledge), as outlined in Table 4 in our manuscript. Therefore, the pre-training approach of FinBERT can be inferred from the outcomes of KePt (w/o Knowledge).\n\nMoreover, we conducted experiments involving FLANG-BERT, designated as FVP (FLANG-BERT) in the following Table. Notably, it's important to acknowledge that the dataset used for training KePt-BERT is notably smaller than that of FinBERT and FLANG-BERT. The specific corpus sizes are provided in detail within this Table. This work primarily focuses on the task of Volatility Prediction. It is apparent that in this task, both FinBERT and FLANG-BERT exhibit less favorable performance compared to KePt-BERT.\n\n(Corpus Size of KeFVP and FVP(FinBERT) are counted by the number of sentences in each corpus, while FVP(FLANG-BERT) are counted by the number of documents in each corpus.)\n\n| Model                     | Corpus Size | MSE(Average) | MSE_3          | MSE_7          | MSE_15       | MSE_30       |\n|---------------------------|-------------|---------------|----------------|----------------|----------------|----------------|\n| FVP(BERT(base))            | -           | 0.318         | 0.644 (2.52e-2)| 0.311 (0.66e-2)| 0.196 (1.23e-2)| 0.120 (0.60e-2)|\n| FVP(FinBERT)              | 406,019     | 0.318         | 0.631 (2.22e-2)| 0.322 (0.72e-2)| 0.195 (1.19e-2)| 0.122 (0.74e-2)|\n| FVP(FLANG-BERT)           | 696,001     | 0.313         | 0.644 (5.75e-2)| 0.293 (1.86e-2)| 0.190 (0.78e-2)| 0.124 (0.59e-2)|\n| **FVP(KePt-BERT) (KeFVP)** | **8,732**       | **0.300**     | **0.610 (3.31e-2)**| **0.291 (1.33e-2)**| **0.183 (0.89e-2)**| **0.114 (0.63e-2)**|\n\nFurthermore, to ensure fairness, we adopted the preferential masking of financial words method from FLANG-BERT and applied it to our KePt dataset. This led to the training of the FLANG-BERT(KePt) model from the bert-base-uncased checkpoint. The corresponding results are exhibited in the following Table under the FLANG-BERT(KePt).\n\n(The results of KePt on financial sentiment analysis. KePt(w/o Knowledge) denotes adaptively pre-training on the KePt dataset without knowledge injection.)\n\n| **Model**               | **FiQA-headline**  | **FiQA-post** | **PhraseBank** |\n|                      | MSE ↓         | R^2 ↑         | MSE ↓     | R^2 ↑     | Acc (%) ↑ | F1 (%) ↑  |\n|----------------------|---------------|---------------|-----------|-----------|-----------|-----------|\n| BERT(base)            | 0.115         | 0.297         | 0.101     | 0.342     | 80.54     | 82.97     |\n| KePt(w/o Knowledge)  | 0.100         | 0.297         | 0.084     | 0.438     | 78.40     | 82.56     |\n| FLANG-BERT(KePT)     | 0.091         | 0.396         | 0.100     | 0.393     | 81.61     | **85.04**     |\n| KePt                 | **0.080**     | **0.435**     | **0.077** | **0.488** | **82.73** | 83.80     |\n\n\nIt is evident that for the Volatility Prediction task, even with a relatively smaller volume of data used for fine-tuning, the performance of KePt-BERT remains superior to that of FinBERT and FLANG-BERT. Furthermore, in the context of financial sentiment analysis, KePt outperforms BERT, FinBERT, and FLANG-BERT in general under the same training corpus (KePt dataset). This observation underscores the effectiveness of KePt.\n\nAs described in the FLANG paper, FLANG employs over 8000 financial words and utilizes preferential masking of financial words during model training. However, FLANG does not incorporate any knowledge descriptions about these financial words. In contrast, our approach, KePt, leverages a subset of 253 financial metric (FM) vocabulary along with corresponding descriptions, intentionally excluding commonly used terms in general language to prevent potential semantic confusion caused by the ubiquitous lexicon. This selection is aimed at maintaining the specificity of financial language in the context of our method.\n\n\n## __2: Use of weak baselines 2: Non-text Baselines__\n\nFurthermore, we have extended our experiments to include non-text baselines, namely the classical linear regression, GARCH, and ARIMA models. These additional baselines are presented in the following Table. Due to data availability constraints and the need to ensure comparability, we have maintained the use of historical price information as non-textual data, consistent with previous works. Notably, we have refrained from incorporating textual data from earnings calls in this context.\n\n| Model                           | MSE(Average) | MSE_3          | MSE_7          | MSE_15       | MSE_30       |\n|---------------------------------|---------------|----------------|----------------|----------------|----------------|\n| **LinearRegression**            | 0.622         | 0.995 (3.20e-2)| 0.595 (5.83e-2)| 0.495 (3.90e-2)| 0.402 (3.80e-2)|\n| **GARCH**                       | 1.756         | 2.084 (1.88)   | 1.729 (1.78)   | 1.657 (1.85)   | 1.555 (1.83)   |\n| **ARIMA**                       | 0.611         | 0.944 (6.94e-1)| 0.596 (6.66e-1)| 0.497 (7.10e-1)| 0.407 (5.78e-2)|\n| **KeFVP(w/o Text)**                 | 0.331         | 0.647 (2.55e-2)| 0.328 (1.52e-2)| 0.210 (1.32e-2)| 0.138 (1.35e-2)|\n| **KeFVP**                       | **0.300**     | **0.610 (3.31e-2)**| **0.291 (1.33e-2)**| **0.183 (0.89e-2)**| **0.114 (0.63e-2)**|\n\n\nWe also agree with the potential benefits of leveraging variables such as CDS spread, implied volatility from option data, and financial metrics like market cap, book-to-market ratio, and cost of capital for enhancing Volatility Prediction, but we have deferred exploration of these variables to future works. This decision is consistent with the main focus of this paper on mining earnings call text, as well as the integration of text and time series data.\n\nRegarding the interpretability of transformer-based models, regression models offer intuitive explanations due to their fewer parameters. However, their limited fitting capacity can restrict their effectiveness. Transformer-based models exhibit robust fitting capabilities, albeit with a larger parameter count. These models can be explained visually through methods like attention visualization to some extent, which aids in the intuitive understanding of a substantial number of parameters and thereby contributes to result comprehension.\n\nSimultaneously, we introduced KeFVP (w/o Text) as a point of comparison. This outcome represents the results derived solely from the Autoformer model's treatment of time series data. In comparison to KeFVP, it is evident that the incorporation of textual information yields a significant enhancement in predictive performance compared with dealing with time series data in isolation.\n\n## __3: Dataset Time Period__\n\nSince our work follows the precedent of using the EC and MAEC datasets, all experiments are conducted within the scope of these two types of datasets. Examining high volatility periods such as the COVID-19 crisis or the 2008 financial crisis presents a more challenging research problem, but no one in the community has compiled relevant datasets for algorithm research. With your prompt, we also have an algorithm for this problem research is of great interest. We will sort out the data during the relevant high volatility period and conduct corresponding algorithm research in future work. In fact, the data sorting work is already in progress.\n\n## __4: Performance Improvement__\n\nRelative to VolTAGE, KeFVP employs a narrower set of data types. While VolTAGE utilizes earnings call text, historical price data, audio data from earnings calls, and a graph constructed from inter-company relationships within the dataset, KeFVP solely relies on textual and price data. Notably, despite this narrower scope, KeFVP achieves superior performance compared to VolTAGE.\n\nIt's interesting to note that following VolTAGE's approach by introducing audio and graph data could potentially lead to further performance improvements in KeFVP. Integrating audio and graph data might provide additional context and insights that could enhance the model's predictive capabilities. This is indeed a promising avenue for future work and development.\n\n\n## __5: Other Concerns__\n\n* Font Size: We will adjust the font size in the camera-ready version to enhance readability.\n\n* Reference: In fact, we have cited the work (Araci, Dogu. 'FinBERT: Financial sentiment analysis with pre-trained language models.' arXiv preprint arXiv:1908.10063 (2019)) in the paper. In addition, we also cite this important related work (Raj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. 'When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain.' In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2322–2335, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics) in the camera-ready version. \n\nFollowing your guidance, we intend to enrich the camera-ready version by including references from financial journals pertaining to Volatility Prediction. This addition aims to provide a comprehensive literature overview.\n\n* Typos Grammar Style And Presentation Improvements: We will revise the presentation in Figure 1 to align with the descriptions in the text. Any additional issues related to phrasing, typos, or other concerns will also be addressed in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written in general. The paper attempt to enhance the NLP methods for an important task in the financial domain. For the same, it tries to incorporate price data with text and enhance the pre-trained model. It also provides ablation studies, and a case study to show the value addition coming from KePt and KeFVP. ", "reasons_to_reject": "Use of weak baselines 1: The paper uses the BERT-base-uncased model as a foundation and enhances it with the financial text. Some existing pre-trained language models (FinBERT, FLANG-BERT) do something similar. Pre-training corpus of the FLANG includes a detailed description of 8000+ financial words from Investopedia including all the financial metrics used here. It also uses the dictionary for preferential masking. It will be good to see FinBERT or FLANG-BERT as a baseline against KePt-BERT. \n\nUse of weak baselines 2: A more suitable non-text baseline for the task can be added. For example, what will be the performance of a simple logistics regression model for volatility prediction with independent variables like CDS spread, market cap, book-to-market ratio, implied volatility from options, cost of capital, etc? The better question to ask will be what additional value text adds over the other financial variable available. The model like simple regression also provides interpretability over the transformer-based models. Also, variables like CDS and implied volatility are updated at a much higher frequency compared to the release of earnings calls once a quarter. Other models like GARCH can be also used as a baseline. \n\nDataset time period: EC dataset covers only 1 year of data while MAEC data covers 2015 to 2018. While the proposed model outperforms suggested baselines the period of study is not sufficient to make a strong conclusion. It also doesn’t include any market-wide high volatility periods like COVID-crisis or the 2008 financial crisis. \n\nPerformance improvement is very small over the baselines used. MSE of Voltage for EC data is 0.31 and KeFVP is 0.30. Which in my opinion is not a significant improvement given the small time period of study. \n\nIt is not a reason to reject, but the paper abuses the ACL formatting guidelines which doesn’t require a minimum font size for tables and figure. Some of the figures and tables in both the main paper and appendix (which doesn’t have a page limit) have font sizes of less than 5. In some cases maybe 2 or 3. The paper could easily move some information in the appendix and use a readable font size. ", "questions_to_authors": "Q: Why preferential masking of financial words was not performed? ", "paper_topic_and_main_contributions": "The paper applies various natural language processing methods/techniques available for an important financial task of volatility prediction. For this task, the paper first introduces the Knowledge-enhanced Adaptive Pre-training module (KePt) which takes advantage and tries to put emphasis on the financial metrics (FMs). Later it performs a Knowledge-enhanced financial volatility prediction (KeFVP) task by combining embedding from KePt transformer with past volatility data. It achieves SOTA over the baselines by a small margin for the volatility prediction task. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Figure 1: Hs’ is used while in text Hd is used"}, "extra_text": {"Missing_References": "1.\tAraci, Dogu. \"Finbert: Financial sentiment analysis with pre-trained language models.\" arXiv preprint arXiv:1908.10063 (2019).\n\n2.\tRaj Shah, Kunal Chawla, Dheeraj Eidnani, Agam Shah, Wendi Du, Sudheer Chava, Natraj Raman, Charese Smiley, Jiaao Chen, and Diyi Yang. 2022. When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2322–2335, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. \n\n\nAlso, the paper ignores literature in financial journals (Journal of Finance, Journal of Financial Economics, and Review of Financial Studies) on volatility prediction while doing a good literature review of top NLP venues. It will be better to have a balanced literature overview while writing an interdisciplinary paper like this one. This is in context with a strong statement on line #83. \n"}}]

Author rebuttal:

Meta-review: The reviewers agree that the authors present a new method for FVP and show SOTA performance on three benchmark datasets. The paper is generally well-written with comprehensive analysis. The authors did a good job in making a detailed response to the reviewer's queries and clarifying their doubts. The new study on non-text baselines w.r.t. the reviewer comments forms an important artifact to the paper and should be included in the subsequent versions. The same goes for the relevant references from non-NLP journals (financial) to provide more context in the literature review section for this type of interdisciplinary research. In light of these arguments, the paper has merit to feature in the conference.
Decision: Accept-Findings