## 2 Methods
### Memory-Driven Cross-Modal Attention Model  
Clinical practice shows that patients' clinical notes and health monitoring time series data have significant clinical correlation. When making diagnosis and treatment decisions, clinicians usually need to combine data from these two different modalities to comprehensively analyze the patient's health status. Therefore, we propose a memory-driven cross-modal attention model (MDCA), which can adaptively learn fine-grained correlation features between cross-modal data. Specifically, MDCA enables cases with similar clinical patterns to share critical information during feature fusion, thereby enhancing disease prediction performance through dynamic cross-modal interactions. The architecture of the MDCA model is illustrated in Figure 2.  
The MDCA model is mainly composed of a multi-subspace attention model, a dynamic memory model (DM), and a memory-aware constrained LN model (MacLN). We first feed the time series data and clinical text embedding into the Dense layer for linear mapping, and then the feature sequence is mapped to the same dimensional space after the LeakyReLU function, which can be expressed as Equation (5). \[ \begin{array}{l}\widetilde{X}_{t}=\text{LeakyReLU}(\lambda_{a}\cdot X_{t})\\ \widetilde{C}_{e}=\text{LeakyReLU}(\lambda_{e}\cdot C_{e})\end{array}  \tag{5}\]  
Here, \(\lambda_{a},\lambda_{e}\) represents the learnable mapping parameter, and \(\widetilde{X}\) represents the time series feature representation after mapping. Furthermore, the multi-modal features will be fed into the multi-subspace attention layer to learn the correlation between different modal features. \(Q,K,V\) represent three different feature channels, and \(K\cdot\theta_{K}=V\cdot\theta_{V}=\widetilde{X}_{t}\), \(Q\cdot\theta_{Q}=\widetilde{C}_{e}\). Different feature channels have independent mapping parameters, which increases the expressiveness of the features. Then, the model further learns the interdependence between cross-modal features by calculating the attention weights between different modal features in each head. The principle is shown in Equation (6). \[ \begin{array}{l}\widetilde{H}_{1}=V\cdot\sigma(\frac{Q\cdot K^{T}}{\sqrt{d _{k}}})\\ \widetilde{H}_{m}=[\widetilde{H}_{1}\oplus\widetilde{H}_{2}\oplus,\ldots, \oplus\widetilde{H}_{n}]\cdot\lambda_{m}\end{array}  \tag{6}\]  
where \(\sigma(*)\) represents the softmax function, \(d_{k}\) represents the dimension of \(K\), \(\oplus\) represents the concatenation operation, \(\lambda_{m}\) represents the learnable linear mapping parameter, \(\widetilde{H}_{1}\) represents the output of the single-head attention mechanism, and \(\widetilde{H}_{m}\) represents the output of the final multi-head attention mechanism. This design enables the model to learn the dynamic association between time series signals and clinical text in parallel in different feature subspaces, significantly enhancing the cross-modal feature interaction capability. Figure 2: The architecture of the MDCA model. In addition, this paper introduces a residual connection structure to further improve the model's ability to capture multi-scale features and alleviate the gradient vanishing problem, as shown in Figure 2. Furthermore, the feature processed by the residual connection will be fed into the MacLN for feature optimization. The MacLN integrates patient's historical information into the normalization process to improve the feature representation capabilities. Due to the heterogeneity of cross-modal data, classic layer normalization will encounter scale constraints and parameter offset problems during feature fusion. The designed MacLN has the following advantages: (1) dynamically adjusts the normalization parameters based on the memory state, which can adapt to different input features; (2) considers historical pattern information and enhance the joint expression cross-modal features.