# MM-iTransformer: A Multimodal Approach to Economic Time Series Forecasting with Textual Data
## 3 Proposed Method
### Proposed MM-iTransformer Framework  
We propose MM-iTransformer, where MM stands for multimodal, to overcome the limitations of existing time series forecasting models. This framework integrates textual embeddings derived from financial news and numerical embeddings from historical prices, providing a robust cross-modal approach to enhance predictive performance in economic applications (see Figure 3). FinBERT for Textual Embeddings: To incorporate rich semantic information from financial news, we utilize FinBERT [37], a language model fine-tuned specifically for financial text. Unlike traditional sentiment analysis approaches, FinBERT captures nuanced financial terminology and contextual insights from complete news articles. Given a set of historical news articles \(x_{1},x_{2},...,x_{L}\) within a historical window of length \(L\), each article \(x_{t}\), which is composed of a word sequence, is processed through FinBERT to obtain its contextualized embedding. Specifically, we extract the [CLS] token's last hidden state from FinBERT's output, resulting in a d-dimensional vector \(e_{t}\) that represents the overall context of the news at time \(t\). This process yields a sequence of textual embeddings, as defined in Equation (1):  
\[T_{1:L}=\{e_{1},e_{2},...,e_{L}\}\in\mathbb{R}^{L\times d} \tag{1}\]  
Figure 3: The workflow of MM-iTransformer: FinBERT Embed generates textual embeddings, Inverted Embed represents price data as variate tokens, and the Multivariate Correlations Map captures interactions for forecasting. Each embedding \(e_{t}\) encapsulates the semantic and contextual information of the corresponding news article, effectively capturing critical financial insights that may influence market movements. Integration with Time Series Data: The core of MM-iTransformer lies in effectively integrating the textual embeddings \(T_{1:L}\) with the historical time series data. The historical price data \(Y_{1:L}\) are embedded using Inverted Embed of iTransformer. Specifically, we describe these data as \(Y_{1:L}=\{y_{1},y_{2},...,y_{L}\}\in\mathbb{R}^{L\times c}\), where c is the number of numerical variables (e.g., different financial indicators). Our goal is to learn a function F that maps these inputs to future forecasts as shown in Equation (2):  
\[\hat{Y}_{L+1:L+H}=F(Y_{1:L},T_{1:L}) \tag{2}\]  
Here, \(\hat{Y}_{L+1:L+H}\in\mathbb{R}^{H\times c}\) represents the predicted future values over a forecast horizon \(H\). To integrate textual and numerical data, we treat the textual embeddings as additional variate tokens, similar to how variables are represented in the inverted perspective of the iTransformer framework. This alignment allows us to leverage the attention mechanism to model cross-modal relationships effectively. Cross-Attention Mechanism: The combined input consists of both numerical and textual embeddings concatenated along the variate (feature) dimension:  
\[X=[Y_{1:L};T_{1:L}] \tag{3}\]  
In the cross-attention mechanism, we compute the query (\(Q\)), key (\(K\)), and value (\(V\)) matrices using learned weight matrices as shown in Equations (4), (5), and (6), respectively:  
\[Q_{h} =XW_{h}^{Q} \tag{4}\] \[K_{h} =XW_{h}^{K}\] (5) \[V_{h} =XW_{h}^{V} \tag{6}\]  
Here, \(W^{Q},W^{K}\in\mathbb{R}^{(c+1)\times d_{v}}\) and \(W^{V}\in\mathbb{R}^{(c+1)\times d_{v}}\) are learnable weight matrices for the queries, keys, and values, respectively. These matrices project the input features into appropriate representation spaces for attention computations. Note that \(d_{k}\) and \(d_{v}\) are distinct from the dimension d used for textual embeddings, as they specifically define the sizes of the query, key, and value projections within the attention mechanism:  
* \(c+1\): The total number of variates, where \(c\) represents the numerical variables, and \(1\) represents the additional textual variable.