# MM-iTransformer: A Multimodal Approach to Economic Time Series Forecasting with Textual Data
## 4 Experiment
### Cross-Modality Correlation Interpretation  
We conduct detailed case studies on both the Gold-price dataset and the Forex dataset. These studies illustrate how our model, through its attention mechanism, adeptly captures and quantifies the interdependencies between textual embeddings and time series data.  
As visualized in the attention heatmaps (Figure 8), the model utilizes the attention mechanism to pinpoint and quantify the interdependencies that exist between diverse data modalities. In the Gold-price dataset, the heat map reveals substantial attention weights on interactions between gold-related economic news and the price movements of gold itself. A notable correlation of 0.45 underscores the substantial influence of economic news on market prices as shown in Figure 8a. This high correlation coefficient indicates that our model not only recognizes relevant economic news but also assesses its impact on market fluctuations significantly. Similarly, the heat map for the Forex dataset shows focused attention on economic narratives associated with specific currency pairs. For instance, the attention weight of 0.31 for the GBP/USD exchange rate highlights the significant impact of economic narratives on currency valuation, as demonstrated in Figure 8b. This focused attention reflects the model's ability to discern and emphasize the economic indicators that are the most predictive of currency movements.  
Figure 8: Comparison of attention heatmaps illustrating the interactions between textual and time series data. Values indicate the similarity between the high-dimensional representations of time series and textual data, where higher values suggest a stronger correlation. (**a**) Attention Heatmap of Head 0, Layer 1 on the Gold-price Dataset. Each row and column corresponds to features including gold price, gold-related economic news, and temporal features (Day of Week, Day of Month, Day of Year). (**b**) Attention Heatmap of Head 0, Layer 1 on the Forex Dataset. Features represented include three currency exchange rates (EURO/USD, GBP/USD, JPY/USD), forex-related economic news, and temporal features (Day of Week, Day of Month, Day of Year).  
\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Dataset** & \multicolumn{2}{c}{**Forex**} & \multicolumn{2}{c}{**Gold-Price**} \\ \hline  & **Training Speed (ms/iter)** & **Memory Footprint** & **Training Speed (ms/iter)** & **Memory Footprint** \\ \hline MM-iTransformer & 232.89 ms & 1.02 GB & 235.23 ms & 1.02 GB \\ iTransformer & 49.91 ms & 0.59 GB & 47.72 ms & 0.44 GB \\ \hline \hline \end{tabular}
\end{table}
Table 5: Model efficiency comparison under input-96-predict-96 of weather and traffic.  
These findings provide quantitative evidence of meaningful patterns and relationships in different types of data. The ability of our model to discern these complex relationships not only demonstrates advanced multimodal comprehension capabilities but also significantly enhances the model's forecast performance. By effectively integrating and analyzing multimodal data, the model offers a robust tool to capture complex market dynamics that are often missed by more traditional models.