# Dual-Modality Transformer with Time Series Imaging for Robust Epileptic Seizure Prediction
## 2 Related Work
To overcome the challenges of gradient instability with long input sequences, LSTMs implement a circular memory structure where hidden layers maintain connections to both adjacent nodes and themselves. This architecture enables LSTMs to effectively map historical input sequences to outputs, theoretically approximating arbitrary temporal sequence transformations. Contemporary research increasingly favors hybrid architectures that combine complementary strengths of different deep learning models. Notable examples include CNN-LSTM [44] and RNN-LSTM [45] architectures, which effectively address challenges such as limited temporal memory and gradient instability. These integrated approaches demonstrate enhanced feature extraction capabilities and superior classification performance. For processing time series data, several imaging transformation techniques have emerged as powerful tools, including recursive graphs [23], Gramian Angular Fields (both summation and difference variants, GASF/GADF) [46], and Markov transition fields [47]. [PERSON] et al. converted EEG to images using time series image conversion algorithm, Recurrence Plot (RP), and Gramian Angular Field (GAF) [48]. The Convolutional Neural Network (CNN) model based on VGGNet was used to learn the converted EEG images. The results show that the use of GAF and CNN models based on EEG results can effectively improve the objectivity and efficiency of the diagnosis of various mental disorders, including schizophrenia. [PERSON] et al. converted one-dimensional EEG signals into two-dimensional images using a Gram Angular Difference Field (GADF) and, based on transfer learning, employed a Domain Adaptive Network with Channel Attention (CADAN) composed of a channel attention mechanism and domain adaptive network for cross-subject emotion recognition [49]. It also achieved a breakthrough in the accuracy of emotion recognition tasks. [PERSON] et al. used two different signal inputs (e.g., EEG signals and their instantaneous power) for image generation in two different ways--the Gram Angle Summation Field (GASF) and the Gram Angle Differential Field (GADF) [50]. An EEG dataset from the University of Bonn was used for experimental verification. The experimental results show that the classification accuracy reaches a high record, and the efficiency of the proposed method is evaluated. [PERSON] et al. 's experiment directly applied time series imaging to the epilepsy classification research, demonstrating the excellent effect of time series imaging in epilepsy classification research, which revealed the usefulness of GAF in the deep learning model for the classification of epilepsy. These studies provide a novel research perspective where deep learning is combined with time series imaging. In the above studies, the EEG signals were imaged by using the Gramian Angle field and then classified by a deep learning model. This study proves the robustness of time series imaging in data processing without using traditional data processing methods, such as wavelet transform or Fourier transform; deep learning models can also accurately classify different types of data using the images. This inspired us to use the Markov transition field to image EEG data. Prior works have made significant contributions to EEG-based epilepsy detection, yet our approach addresses several fundamental limitations in existing methods. Traditional preprocessing approaches like wavelet transforms and short-time Fourier transforms require careful parameter tuning and often lose critical temporal information. Current deep learning methods typically focus on either temporal or spatial features independently, missing the complex interplay between these modalities. Additionally, existing multi-modal approaches often employ simple concatenation or weighted averaging for feature fusion, which fails to capture the dynamic relationships between different feature representations. Our work advances the state-of-the-art method in several key aspects.