## 3 System Model
### DRL Module
\[\
abla_{\theta}J(\theta)=\mathbb{E}_{s_{t}}\Big{[}\mathbb{E}_{a_{t}\sim\pi_{ \theta}}\Big{[}\alpha\log\pi_{\theta}(a_{t}|s_{t})-\min_{i}Q_{i}(s_{t},a_{t}) \Big{]}\Big{]} \tag{12}\]  
TQC is an algorithm that extends SAC within the distributional reinforcement learning paradigm. It leverages an ensemble of critic networks to estimate quantile-based return distributions and applies a truncation-and-averaging mechanism to both the critic and target critic by discarding excessively high quantile values and averaging the remaining ones. This approach is motivated by the tendency of upper quantiles to induce overestimation, and thus TQC has improved policy stability. The target value \(y_{t}\) for \(Q\)-learning can be formulated as follows:  
\[y_{t}=r_{t}+\gamma\cdot\text{mean of bottom }(K-M)\text{ quantiles of }\{Z_{j}(s_{t+1},a_{t+1})\} \tag{13}\]  
where \(Z_{j}\) denotes the quantile-based \(Q\)-values, representing the predicted return values for a given state-action pair, as a set of \(K\) quantiles. In other words, among the total \(K\) quantiles, the top \(M\) quantiles prone to overestimation are discarded, and the remaining \(K-M\) lower quantiles are retained. The mean of these retained quantiles is then used to compute the target \(Q\)-value. Furthermore, the loss function for the \(Q\)-update is formulated as follows:  
\[L(\phi)=\frac{1}{NK}\sum_{i=1}^{N}\sum_{j=1}^{K}\rho_{\tau}^{\text{x}}\Big{(}y _{t}^{(i)}-z_{j}(s_{t},a_{t};\phi_{i})\Big{)}, \tag{14}\]  
where \(\rho_{\tau}^{\text{x}}\) represents the Quantile Huber Loss, \(N\) is the number of critic networks, and \(z_{j}(s_{t},a_{t};\phi_{i})\) denotes the \(j\)-th quantile output of the \(i\)-th critic network evaluated in the state-action pair \((s_{t},a_{t})\). The policy update follows the same principle as that in SAC, incorporating the entropy term \(\alpha\log\pi_{\theta}(a_{t}|s_{t})\) along with multiple \(Q\)-networks. The update is performed using the mean of the truncated quantile \(Q\)-values as follows:  
\[y_{t}=r_{t}+\gamma\min_{i=1,2}Q_{i}^{\prime}(s_{t+1},\mu^{\prime}(s_{t+1})+ \epsilon),\quad\epsilon\sim\text{clip}(\mathcal{N}(0,\sigma),-c,c), \tag{7}\]0  
CrossQ extends the conventional DDPG and TD3 architectures by introducing a conservative \(Q\)-update scheme based on a cross-validation mechanism. Specifically, it constructs multiple actor-critic pairs, where each actor-critic pair participates in target generation and value estimation for the others in a crosswise manner. This design mitigates the overestimation bias by avoiding self-referential updates, thus promoting more conservative and stable learning dynamics. Furthermore, to enhance training stability, CrossQ integrates batch renormalization into the critic architecture. This can be formulated as follows:  
\[y_{t}=r_{t}+\gamma\min_{i=1,2}Q_{i}^{\prime}(s_{t+1},\mu^{\prime}(s_{t+1})+ \epsilon),\quad\epsilon\sim\text{clip}(\mathcal{N}(0,\sigma),-c,c), \tag{7}\]1  
where \(\mu_{i}(s_{t+1})\) denotes the action generated by the \(i\)-th actor for the next state \(s_{t+1}\) and \(Q_{j}(s_{t+1})\) represents the predicted \(Q\)-value from the \(j\)-th critic corresponding to that action. In Equation (16), the condition \(i\
eq j\) implies that the action used to compute the target \(Q\)-value is generated by actor \(i\)'s policy, while the corresponding \(Q\)-value is evaluated using critic \(j\). By selecting the minimum of multiple \(Q\)-values, the approach enables a more conservative Q estimation. This cross-validation mechanism prevents an actor from evaluating the \(Q\)-value of its own generated actions using its corresponding critic, thereby mitigating overfitting and overly optimistic value estimates.