## 2 Related Studies
### Deep Learning and Reinforcement Learning in Earth Science and Data Assimilation
#### 2.2.2 Reinforcement Learning for Data Assimilation and Forecasting  
Reinforcement learning (RL) is an AI approach designed to optimize decision-making by enabling agents to interact with their environment and learn policies that maximize cumulative rewards [26]. Specifically, RL agents refine their strategies through trial and error, continuously adapting to dynamic conditions in order to achieve optimal outcomes. In Earth science and DA, RL applications remain in their early stages. However, emergingresearch suggests that integrating RL with DL could provide effective solutions for tackling complex decision-making challenges in nonlinear systems.  
Unlike the ensemble Kalman filter (EnKF), which relies on Gaussian assumptions and linear updates, [PERSON] et al. (2024) proposed an RL-based DA approach [40]. In their framework, an RL agent is employed within each assimilation window to generate a correction vector based on the current model state and observational innovation. This correction is added to the background forecast to update the state estimation of chaotic dynamical systems. The proposed method exhibits advantages in handling nonlinear and non-Gaussian systems, improving forecast accuracy [41]. Experiments on Lorenz-63 and Lorenz-96 models demonstrated superior performance over EnKF, particularly under high noise levels. However, because the correction vector shares the same dimensionality as the numerical model state, scalability to high-dimensional systems may impose substantial computational burdens. Furthermore, the correction strategy may have overlooked physical consistency among state variables, which could potentially introduce spurious errors. Similarly, [PERSON] et al. (2023) introduced an RL-based automatic assimilation model (SWMM-RL) to improve the accuracy of urban stormwater runoff and water quality simulations [42]. In this approach, during each storm event, an RL agent outputs optimized input parameters for the Storm Water Management Model (SWMM) based on environmental states, with the goal of minimizing prediction errors. This framework can be viewed as an event-driven, offline-trained, data-driven parameter calibration strategy that enhances the adaptability of the model to varying environmental conditions. SWMM-RL substantially improves the prediction of runoff volumes and pollutant loads--especially under extreme rainfall scenarios--compared to traditional fixed-parameter approaches. Nevertheless, the model exhibits instability in low-rainfall scenarios, and its generalization capability requires further validation across broader and more complex conditions. However, applying a single RL agent to the entire study area may overlook localized watershed variations.  
Beyond direct DA applications, RL has demonstrated its potential in handling nonlinear forecasting problems in Earth system sciences by framing predictive tasks as sequential decision-making processes, enabling adaptive model refinement and improved uncertainty estimation in complex, dynamic environments. [PERSON] et al. (2024) proposed a dynamic ensemble forecasting method (DDPG-WRF-TBL) that integrates NWP, deep deterministic policy gradient (DDPG), and error sequence correction (TCN-BiLSTM) to enhance multi-step wind speed forecasting [43]. The key innovation lies in dynamically adjusting ensemble weights to avoid the low-performing member and optimizing forecast accuracy by prioritizing reliable simulations. However, the method's effectiveness remains dependent on the quality of NWP outputs. [PERSON] et al. (2025) developed an RL-based ensemble forecasting framework (RL-EFF) for renewable energy prediction, employing a deep Q-network (DQN) to dynamically select the most suitable base models based on meteorological conditions [44]. This approach overcomes the limitations of traditional ensemble methods that rely on fixed model combinations, thereby improving adaptability and predictive accuracy. Despite its advantages, the discrete action space of RL-EFF may limit its ability to fully integrate base model outputs.