## 1 Introduction  
In recent years, deep learning-based medical assistance has attracted significant attention due to its powerful capabilities in data fusion and feature extraction [1, 2, 3, 4, 5]. Within the expansive domain of healthcare big data, there has been an accumulation of substantial volumes of multi-modal medical data, including medical images, electronic health records(EHRs), and so on. In-depth analysis of these data helps to uncover their core value and can provide auxiliary support for medical decision-making. EHRs, serving as the primary repository of patients' health conditions and treatment processes, mainly consist of time series data generated by vital signs monitoring devices and clinical text notes written by clinicians, playing a crucial role in clinical decision support systems [6; 7; 8; 9; 10; 11; 12]. Recently, AI-assisted clinical prediction based on EHRs has achieved remarkable progress [13; 14; 15; 16; 17; 18; 19]. The approach of constructing predictive models by integrating patients' multi-source medical features is conducive to continuously monitoring patients' health trends and providing decision support for clinicians [20; 21; 22; 23; 24; 25; 26]. Previous approaches mainly focused on single-modality modeling, utilizing temporal data or clinical notes for clinical prediction. Some studies have used time series data to predict ICU mortality or length of stay [27; 28; 29; 30]. However, these methods fail to exploit the complementarity of multi-modal data, resulting in limited prediction performance. Advances in multi-modal learning have led to new paradigms in clinical prediction. Studies show that by learning the joint representation of patients' temporal and clinical text features, the accuracy of clinical predictions can be significantly enhanced, thereby providing stronger support for clinical decision-making [31; 32; 33]. Although existing research has made significant progress, there are still some limitations. Firstly, the unstructured nature of clinical notes, combined with the spatiotemporal heterogeneity of diverse data modalities, increases the complexity of data processing and limits the model's ability to capture cross-modal feature correlations. Second, the existing methods mostly rely on simplistic feature concatenation strategies, thus failing to account for the latent semantic associations among multi-modal features. Third, clinical diagnosis requires a comprehensive analysis of the contextual relationships between symptom descriptions. However, current research mostly focuses on the word-level analysis, lacking modeling of semantic associations between sentences. Fourth, EHRs data display significant information redundancy, and directly utilizing all clinical knowledge as input may introduce noise interference. Lastly, existing methods usually adopt static data for modeling, fail to consider the dynamic evolution trajectory of patients' health status, and ignore the impact of historical status. To mitigate the aforementioned problems, we propose D4 Care, a novel clinical outcome prediction model. First, we use a BiGRU network to capture time series features. Second, we adopt a dual-view text feature extraction strategy to learn the global semantic feature associations of clinical notes through a sentence-aware attention model and obtain fine-grained structured features by extracting medical entities in clinical notes. This approach not only maintains the semantic integrity of the text but also enhances the model's capacity to identify critical information, thus improving model's performance. Furthermore, we design a memory-driven cross-modal attention mechanism to dynamically learn the deep associations between the features of different modalities. At the same time, we introduce a memory-aware constrained layer normalization to handle heterogeneity of data and model the dependencies of historical-current patterns. In addition, we use gating mechanisms and dynamic memory components to learn and update critical features.