# Balancing Depth for Robustness: A Study on Reincarnating Reinforcement Learning Models
## 2 Materials and Methods  
Reinforcement Learning (RL) is a machine learning paradigm that learns how to make decisions by interacting with an environment. In RL, an agent performs actions to influence the environment, receives rewards, and improves its behavior strategy based on these feedbacks to maximize the cumulative rewards. Most RL research is based on the absence of prior knowledge, but large-scale RL systems usually need to run for a very long time and constantly interact with new data. Starting them from scratch will take weeks or even months, which makes methods that do not exploit prior knowledge inefficient. For example, a system that plays Dota 2 like a human underwent months of RL training during its development and constantly changed (e.g., model architecture, environment, etc. ), which required building on previously trained systems to avoid retraining from scratch. Deep Q-Network (DQN) is a DRL technique that approximates Q-values using neural networks instead of Q-tables. Q-values are computed using the Bellman equation [38], as shown in Equation (1):  
\[Q(s,a)=r+\gamma\underset{a}{max}\,Q\big{(}s^{\prime},a^{\prime}\big{)} \tag{1}\]  
where \(\gamma\) is the discount factor controlling the weight of the next state, with values between 0 and 1. \(Q(s,a)\) represents the value of taking action a in state \(s\), \(r\) is the immediate reward received after executing action a in state s, \(s^{\prime}\) is the new state reached, and \(a^{\prime}\) represents all possible actions in state \(s^{\prime}\). The concept of reincarnating RL addresses the need to quickly iterate on a problem or test new strategies without restarting from scratch, as depicted in Figure 1.  
This paper presents an adaptive deep Q-Dagger network architecture based on RRL (Figure 2), which aims to improve model robustness and generalization ability by dynamically adjusting the network depth configuration. Unlike traditional RL models with fixed network depth, our method customizes the network depth according to task complexity and explores the best architecture to enhance multi-task performance flexibility. Task complexity can be calculated by the following indicators:  
\[C_{task}=w_{1}\,\mathrm{S}\,+w_{2}\,\mathrm{R}+w_{3}\mathrm{A} \tag{2}\]  
Among them, \(w_{1}\), \(w_{2}\), and \(w_{3}\) are weight coefficients used to reflect the impact of different task characteristics on complexity. S is the state space dimension, R is the reward sparsity, and A is the action space dimension. Adaptive Network Design: RRL seeks to reduce training time and computational resource usage by reusing previous results and training experiences. However, the network's structural design--particularly its depth--plays a crucial role in determining learning outcomes and model stability. To address this, this study introduce a depth-adaptive Q-Dagger network capable of adjusting its architecture according to task complexity, with the following key design steps:  
Figure 1: Interaction between agent and environment in Reincarnating Reinforcement Learning. 1. Network Parameter Initialization: Network weights are initialized using the Xavier uniform distribution to maintain training stability. 2. Convolutional Layer Generation: The number of convolutional layers is adaptively chosen based on task complexity, ranging from 1 to 5 layers. The number of filters per layer is randomly selected from [32; 64; 128; 256], with a fixed kernel size of 3 \(\times\) 3 and a stride of 1 \(\times\) 1. ReLU activation and batch normalization are applied to enhance stability and improve learning dynamics. 3. Fully Connected Layer Generation: The number of fully connected layers is adaptively selected between 1 to 3 layers, with the number of neurons per layer randomly chosen from [64; 128; 256]. ReLU activation is applied to increase the network's non-linear representation capacity. 4.