言語処理学会 第30回年次大会 発表論文集（2024年3月）

Enhancing Economic Time Series Prediction with News Text
Data and Numerical Data: A Transformer-Based Approach
Shangyang Mou1 Wenting Zhang1 Takuji Kinkyo1 Shigeyuki Hamori1
Jinhui Chen2 Tetsuya Takiguchi1 Yasuo Ariki1
1 Kobe University
2 Wakayama University
205x610x@stu.kobe-u.ac.jp

Abstract
Financial time series such as stock prices can be hard to
predict as it is diﬃcult to model short-term and long-term
temporal dependencies between data points. In addition,
they are inﬂuenced by underlying fundamentals that areconveyed through various news feeds. Existing methods
for predicting ﬁnancial time series typically focus on either
numerical data or textual information. When considering
textual data, the prevailing approach is to utilize sentiment information extracted from news sources as features.
This paper introduces a novel methodology that directly
integrates ﬁnancial news content with corresponding timeseries data, to enhance the accuracy of ﬁnancial time series
forecasting.

1

Introduction

Financial time series forecasting is a challenging signal
processing problem, largely because the underlying ﬁnancial system that generates such data is enormously complex, resulting in data that can exhibit both non-linear andnon-stationary characteristics [1]. However, ﬁnancial data
prediction plays a pivotal role in various aspects of the ﬁnancial domain, from individual investment decisions tomacroeconomic policy making [2].
With the growing availability of big data, there has been
an increase in research into ﬁnancial date analysis. For
example, from automated trading systems [3, 4] to stock
price prediction [5, 6], the availability and analysis of large
ﬁnancial datasets have enhanced the accuracy and speed ofmarket predictions and investment strategies, signiﬁcantly
impacting the ﬁeld of ﬁnancial technology.
In ﬁnancial time series forecasting, unlike data fromweather, traﬃc, or electricity [7, 8], ﬁnancial time series

are inﬂuenced by a variety of complex factors. These include historical data as well as macroeconomic policies,
global events, and related economic indices [9, 10]. Existing approaches in ﬁnancial series prediction have utilized
either textual or numerical data, with some models considering both types concurrently [11, 12]. The most methods
employing ﬁnancial text have predominantly relied on sentiment analysis, using sentiment scores as features to assist
model predictions [13, 14]. These models typically focus
on predicting the direction in which the data will fall or
rise, rather than directly predicting the data series. However, inaccuracies in sentiment analysis, especially mispredicted sentiment labels, can notably aﬀect the quality
and accuracy of ﬁnancial predictions, leading to signiﬁcantdeviations in forecasted outcomes.
Recently, transformer models have become increasingly
popular for time series forecasting tasks, showing impressive outcomes [8, 7, 15, 16, 17]. However, [18] claims thattransformers may not be as eﬀective as anticipated, with results showing that linear models outperform the more complex transformer-based models. Despite this, considering
the transformers’ robust language processing capabilities[19], we explore whether it can eﬀectively handle textual
features combined with numerical data for ﬁnancial data
forecasting.

2

Proposed Method

This paper proposes a methodology that directly uses
news text as a feature combined with numerical data forforecasting. The approach, implemented on transformerbased models, has yielded promising results. The eﬀectiveness of this method illustrates the potential of integratingcomplex language models with traditional numerical timeseries analysis, oﬀering a more comprehensive understand-― 1975 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Figure 1

FRAMEWORK ARCHITECTURE.

ing of the ﬁnancial data. For the time series forecasting

partitioned into smaller sub-sequences known as patches.

tasks, we consider the following problem: given a col-Self-attention is then computed between the patches. This

lection of multivariate time series samples with look-backapproach enables the model to handle longer sequences

window 𝐿 : (𝑥1 , ..., 𝑥 𝐿 ) where each 𝑥 𝑡 at time step 𝑡 is awhile avoiding memory constraints and facilitating quicker

vector of dimension M, we would like to forecast 𝑇 future

inference. Additionally, patching enables the capture of lo-

values (𝑥 𝐿+1 , ..., 𝑥 𝐿+𝑇 ).

calized semantic information that can not be available when

Forward Process. We deﬁne the 𝑖-th univariate series ofusing individual point-wise input tokens.

(𝑖)length 𝐿, starting at time index 1, as 𝑥1:𝐿= (𝑥 1(𝑖) , ..., 𝑥 𝐿(𝑖) ),where 𝑖 = 1, ..., 𝑀. The input numerical data (𝑥 1 , ..., 𝑥 𝐿 )univariate series 𝑥 (𝑖)channel independently, the model can eﬃciently capture

is divided into 𝑀∈
andeach is processed independently through the transformer

complex patterns and relationships across the entire sequence. Inspired by this, we extend the input to combine

backbone according to PatchTST [16] model’s channel-

with news text, as shown in Figure 1, for each channel

independence setting. Then the corresponding news text

we concatenate the correspond news text embedding to thedata denoted as 𝑎 𝑡 = (𝑤 1 , ..., 𝑤 𝑒 )𝑡 𝑤 𝑡 is concatenated totime series.

each univariate time series. Since we only use the lat-

Transformer Encoder. PatchTST uses the vanilla trans-

est day’s news text embedding 𝑤 𝐿 , for each time series

former encoder that maps the observed signals to the latent(𝑥1 , ..., 𝑥 𝐿 ), the ﬁnal input to the transformer becomes a

representations. Initially, the numerical data and textual

concatenation of the numerical series and the textual em-

feature embeddings are concatenated, forming an enriched(𝑖)(𝑥 1:𝐿, 𝑎 𝐿 ), with our objective being to

input sequence. This combined sequence is then mapped

bedding:𝑧 (𝑖)=ℝ1×𝐿 ,

By dividing the input into patches and processing each

forecast 𝑇 future values (𝑥 𝐿+1 , ..., 𝑥 𝐿+𝑇 ).

2.1

to the transformer’s latent space of dimension 𝐷 via a trainable linear projection 𝑊 𝑝 ∈ ℝ𝐷× 𝑃 , where 𝑃 is the dimen-

Model Architecturesion after concatenation. Additionally, a learnable additive

In the study by Nie et al. [16], a novel transformer architecture, named PatchTST, was introduced, innovatively
incorporating two pivotal concepts - channel independence
and patching - speciﬁcally designed for enhancing time se-position encoding 𝑊 𝑝𝑜𝑠 ∈ ℝ𝐷× 𝑁 is applied to maintainthe temporal order of the concatenated patches: 𝑧 𝑑(𝑖) =(𝑖)
𝐷× 𝑁 denote the combined
𝑊 𝑝 𝑧 (𝑖)𝑝 + 𝑊 𝑝𝑜𝑠 , where 𝑧 𝑑 ∈ ℝ
input fed into the transformer encoder. In the multi-headattention mechanism, each head ℎ = 1, ..., 𝐻 transforms theries forecasting.
Channel independence. Channel independence is a property of the PatchTST model that allows diﬀerent channels

input into query matrices 𝑄 ℎ(𝑖) = (𝑧 𝑑(𝑖) ) 𝑇 𝑊ℎ𝑄 , key matrices

𝐾 ℎ(𝑖) = (𝑧 𝑑(𝑖) ) 𝑇 𝑊ℎ𝐾 , and value matrices 𝑉ℎ(𝑖) = (𝑧 𝑑(𝑖) ) 𝑇 𝑊ℎ𝑉 ,of the input to be processed independently. In traditional

where 𝑊ℎ𝑄 , 𝑊ℎ𝐾 ∈ ℝ𝐷×𝑑𝑘 and 𝑊ℎ𝑉 ∈ ℝ𝐷×𝐷 . The scaledtransformers, the same set of attention weights is used for

dot-product attention mechanism then generates the atten-

all channels, which limits the model’s ability to capturetion output 𝑂 ℎ(𝑖) ∈ ℝ𝐷× 𝑁 as:

ﬁne-grained information in each channel. In contrast, the(𝑂 ℎ(𝑖) ) 𝑇 = 𝐴𝑡𝑡𝑒𝑛𝑡𝑖𝑜𝑛(𝑄 ℎ(𝑖) , 𝐾 ℎ(𝑖) , 𝑉ℎ(𝑖) )PatchTST model applies attention weights separately to

𝑄 ℎ(𝑖) (𝐾 ℎ(𝑖) ) 𝑇 (𝑖))𝑉ℎ= 𝑆𝑜 𝑓 𝑡𝑚𝑎𝑥(√𝑑𝑘each channel, allowing it to better capture the unique features and patterns in each channel.

(1)

Patching. Patching is a technique to alleviate the com-

The multi-head attention block incorporates BatchNorm

putational burden of self-attention. Rather than attending
to every position in the sequence, the input sequence islayers and a feed-forward network with residual connections. Finally, a ﬂatten layer followed by a linear head

― 1976 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

is used after the encoder to obtain the prediction result

Considering that the U.S. stock market typically has 256

(𝑖)
(𝑖)
(𝑖)
𝑥ˆ 𝐿+1:𝐿+𝑇= ( 𝑥ˆ 𝐿+1, ..., 𝑥ˆ 𝐿+𝑇) ∈ ℝ1×𝑇 .

trading days per year, coupled with the scarcity of ﬁnan-

FinBert. To eﬀectively process news text, we employ thecial news datasets, we constructed a dataset containing 611

FinBERT model [20], a domain-speciﬁc pre-trained BERT

samples. The dataset consists of economic data for thearchitecture specialized for ﬁnancial contexts. FinBERT isperiod 2018-03-01 to 2020-07-02 and corresponding news

adept at extracting features from ﬁnancial texts, enablingtexts. The choice of this time period and the resulting

it to capture the complex sentiment and thematic content

size of the dataset is dictated by the amount of relevant

inherent in ﬁnancial news. In our approach, FinBERT isnews content available for analysis. The dataset thus pro-utilized to generate feature embeddings which serve as avides a comprehensive view of the economic landscaperich, contextual representation of the news data.

over a given period, combining quantitative market data

The integration of textual features and quantitative timewith qualitative news narratives to provide an integrated

series data signiﬁcantly enhances the model’s ability toapproach to ﬁnancial analysis.

forecast time series with greater accuracy, leveraging the3rich contextual information present in the news text to provide a more comprehensive understanding of the factors
inﬂuencing the time series trends.

2.2

Experiments

In this section we describe the details of the training
process and the experiment setting.
Loss Function.

Model Input

Consistent with PatchTST [16] we use

the Mean Squared Error (MSE) loss to measure the dis-

For our study, we constructed a dataset focused on thecrepancy between the prediction and the ground truth. The

U.S. economy by utilizing a collection of news headlines

loss in each channel is gathered and averaged over 𝑀 time

scraped from the oﬃcial websites of CNBC, the Guardian,series to get the overall objective loss:
𝐿 = 𝐸𝑥the U.S. economy and stock market trends over the past one
to two years. To complement this textual data, we aligned
it with corresponding period economic time series data,
including indices such as the SP 500, MSCI US, NYSE
American Composite, Nasdaq Composite, Dow Jones Industrial Average, and the US Dollar Index.
Given the signiﬁcantly higher dimensionality of textual
data compared to numerical data, and the fact that news is
not available every day, we only select latest day’s news.
On days without news, we inserted a ’no news’ marker. In
our transformer-based approach, we added a masking layer
to handle these ’no news’ instances by masking them, so themodel do not pay attention to them. To align the news datawith the corresponding index in dimension and retain asmuch textual information as possible, we concatenate thelatest day’s news with a historical window of time series
data, as the latest updates tend to have the greatest impact.
For instance, to forecast the time series for the upcoming
12 days, we utilise the previous 96-day time series and thelatest day’s news, reﬂecting the immediacy and relevanceof the latest news in inﬂuencing economic trends.
1） https://www.kaggle.com/datasets/notlucasp/ﬁnancial-newsheadlines

1 ∑ (𝑖)(𝑖)|| 𝑥ˆ
− 𝑥 𝐿+1:𝐿+𝑇|| 22
𝑀 𝑖=1 𝐿+1:𝐿+𝑇
𝑀

and Reuters1）. These headlines provided an overview of(2)Model Parameters. In Table 1, we compare our dataset
US Economy with other popular datasets dedicated to multivariate time series forecasting tasks. Notably, our dataset
appears to be relatively small, primarily due to the scarcityof news text data. This peculiarity prompted speciﬁc adjustments in our model’s parameters. Original PatchTST
model conﬁguration consists of 3 encoder layers with thehead number 𝐻 = 16 and the latent space dimension
𝐷 = 128. However, in our adapted model that incorporates
news text data, we found it necessary to modify the architecture to better capture the complexity introduced by theadditional textual information. To this end, we increased
the number of encoder layers to 12, providing a deeper
and more nuanced processing capability for the combined
numerical and textual data. Additionally, with our dataset
variables, we optimized the head number to 4, ensuring
eﬃcient processing while capturing the intricate patterns
in the combined numerical and textual inputs.
Results Analysis.

Based on our experiments, inte-

grating news text as a feature into the corresponding economic data time series has demonstrated signiﬁcant im-― 1977 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Datasets

Weather Traﬃc Electricity

Features21

Timesteps 52696

ETTm1

US Economy(ours)

862

321

7

6

17544

26304

69680

611

Table 1

Dataset Comparison.

provements in multivariate time series forecasting. Specifically, as shown in Table 2 when only economic data wasused, MSE = 0.5779. However, upon incorporating thecorresponding news text as an additional feature, there wasa notable reduction in the MSE to 0.4773. This substantial
decrease underscores the eﬃcacy of including news text in

Figure 2

Comparison of MSE Loss with & without News Con-tent.

enhancing the accuracy of our forecasting model. The re-data. The improved MSE with textual data integration

sults clearly indicate that the contextual insights provided

demonstrates the model’s enhanced capability to discernby the news text play a crucial role in predicting economic

and synthesize the latent factors aﬀecting economic ten-trends more accurately, thereby validating our approach ofdencies, resulting in wiser and more precise forecasts.
These observations underscore the comprehensive na-

combining qualitative textual data with quantitative time

ture of forecasts that incorporate qualitative insights fromseries data.
L

N

MSE

96

12

0.5779

Num+News 96

120.4773Numtext data, aﬃrming the value of combining diverse datatypes for a deeper understanding of economic movements.
This approach aligns with the intricate realities of ﬁnancial

Table 2Multivariate time series forecasting results withlook-back window L and number of input tokens N are predicted.

In our comparative analysis of model performance withand without news content shown in Figure 2, we observeda marked contrast in the optimization trajectories. Using
solely economic data, the model rapidly converged to its
minimum MSE of 0.5779 within just 7 epochs, after which
it began to overﬁt, as indicated by the subsequent increasein validation loss. This swift convergence suggests thatthe model ﬁnds it relatively straightforward to learn fromnumerical data, which, although indicative of economic
trends, lacks the complexity inherent in textual information.
Conversely, when we incorporated news text into theforecasting model, the optimal MSE of 0.4773 wasachieved only after a considerably longer training periodof 70 epochs. This prolonged learning curve highlights thecomplexity that textual data introduces to the model, necessitating a more nuanced and detailed learning process.
Notably, the model trained with text data did not exhibit
signs of overﬁtting within the observed epochs, suggesting that it was able to continue learning and integrating
insights from the textual content eﬀectively.
The extended training required for the text-enhanced
model is indicative of its process to assimilate and interpretthe multifaceted inﬂuences captured in textual data, whichmarkets, where qualitative factors often steer the quantitative indicators, suggesting a promising direction for advancing ﬁnancial time series forecasting methodologies.

4

Conclusions
In conclusion, our method integrates economic news text

with economic data for multivariate time series forecasting.
This approach adds inﬂuential news content to the prediction process, enhancing traditional numerical data analysis.
Our experiments demonstrated that features extracted fromnews text can improve economic sequence forecasting.
However, a key limitation is the limited availability ofﬁnancial news that high-related to speciﬁc economic indicators. The sheer volume of market news makes it challenging to identify news that directly aﬀects certain economic metrics. Moreover, aligning them temporally witheconomic data is a non-trivial task. Therefore, our validation is conducted on a relatively small dataset.
Moving forward, we plan to extend the scope of our dataset
to further validate our ﬁndings. A larger dataset will allow
a more thorough evaluation of our model and the broader
impact of text-enhanced time series forecasting. This direction of research holds the potential to signiﬁcantly improve
the precision of economic predictions by incorporating aricher spectrum of qualitative information.

are often non-linear and abstract compared to numerical― 1978 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

15th International Conference on Computer and Information Science (ICIS), pp. 1–6. IEEE, 2016.

References[1] Adam Atkins, Mahesan Niranjan, and Enrico Gerding.
Financial news predicts stock market volatility better thanclose price. The Journal of Finance and Data Science,Vol. 4, No. 2, pp. 120–137, 2018.
[2] Daniel Broby. The use of predictive analytics in ﬁnance.
The Journal of Finance and Data Science, Vol. 8, pp.
145–161, 2022.
[3] Yang Liu, Qi Liu, Hongke Zhao, Zhen Pan, and Chuanren
Liu. Adaptive quantitative trading: An imitative deep
reinforcement learning approach. In Proceedings of the
AAAI conference on artiﬁcial intelligence, Vol. 34, pp.
2128–2135, 2020.
[4] Boming Huang, Yuxiang Huan, Li Da Xu, Lirong Zheng,
and Zhuo Zou. Automated trading systems statistical andmachine learning methods and hardware implementation:a survey. Enterprise Information Systems, Vol. 13,
No. 1, pp. 132–144, 2019.
[5] Sreelekshmy Selvin, R Vinayakumar, EA Gopalakrishnan,
Vijay Krishna Menon, and KP Soman. Stock price prediction using lstm, rnn and cnn-sliding window model. In
2017 international conference on advances in computing, communications and informatics (icacci), pp.

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

1643–1647. IEEE, 2017.
Mehar Vijh, Deeksha Chandola, Vinay Anand Tikkiwal,
and Arun Kumar. Stock closing price prediction using machine learning techniques. Procedia computer science,Vol. 167, pp. 599–606, 2020.
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang,
Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond eﬃcient transformer for long sequence time-series
forecasting. In Proceedings of the AAAI conference on
artiﬁcial intelligence, Vol. 35, pp. 11106–11115, 2021.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng
Long. Autoformer: Decomposition transformers with
auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems,
Vol. 34, pp. 22419–22430, 2021.
Rakesh Kumar Verma and Rohit Bansal. Impact of macroeconomic variables on the performance of stock exchange:a systematic review. International Journal of Emerging
Markets, Vol. 16, No. 7, pp. 1291–1329, 2021.
Torben G Anderson, Tim Bollerslev, Francis X Diebold,
and Clara Vega. Micro eﬀects of macro announcements:
Real-time price discovery in foreign exchange. American
economic review, Vol. 93, No. 1, pp. 38–62, 2003.
Mark Cecchini, Haldun Aytug, Gary J Koehler, andPraveen Pathak. Making words work: Using ﬁnancial
text as a predictor of ﬁnancial events. Decision support
systems, Vol. 50, No. 1, pp. 164–175, 2010.
Renáta Myšková and Petr Hájek. Mining risk-related sentiment in corporate annual reports and its eﬀect on ﬁnancial
performance. Technological and Economic Development of Economy, Vol. 26, No. 6, pp. 1422–1443, 2020.
Ryo Akita, Akira Yoshihara, Takashi Matsubara, and Kuniaki Uehara. Deep learning for stock prediction using
numerical and textual information. In 2016 IEEE/ACIS[14] Bledar Fazlija and Pedro Harder. Using ﬁnancial news sentiment for stock price direction prediction. Mathematics,
Vol. 10, No. 13, p. 2156, 2022.
[15] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang
Sun, and Rong Jin. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In
International Conference on Machine Learning, pp.
27268–27286. PMLR, 2022.
[16] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, andJayant Kalagnanam. A time series is worth 64 words:
Long-term forecasting with transformers.
In TheEleventh International Conference on Learning Representations, 2022.

[17] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu
Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are eﬀective for time series forecasting.
arXiv preprint arXiv:2310.06625, 2023.
[18] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu. Are
transformers eﬀective for time series forecasting? In Proceedings of the AAAI conference on artiﬁcial intelligence, Vol. 37, pp. 11121–11128, 2023.

[19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser,
and Illia Polosukhin. Attention is all you need. Advancesin neural information processing systems, Vol. 30, ,2017.
[20] Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and
Jun Zhao. Finbert: A pre-trained ﬁnancial language representation model for ﬁnancial text mining. In Proceedings― 1979 ―of the Twenty-Ninth International Joint Conferenceon Artiﬁcial Intelligence, IJCAI’20, 2021.

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).