# Entropy-Guided Distributional Reinforcement Learning with Controlling Uncertainty in Robotic Tasks
## 2 Background
### Truncated Quantile Critics  
The TQC algorithm employs quantile regression to approximate the target distribution in the same manner as the QR-DQN. \[Z(s,a)=\frac{1}{M}\sum_{m=1}^{M}\delta(\theta^{m}(s,a))\]  
TQC employs an ensemble of \(N\) networks where each network \(n\) predicts the distribution \(Z_{\psi_{n}}\):  
\[Z_{\psi_{n}}(s,a):=\frac{1}{M}\sum_{m=1}^{M}\delta\Big{(}\theta^{m}_{\psi_{n}}( s,a)\Big{)}.\]  
A single Bellman target distribution was computed for all networks. This is achieved by first computing all targets for all networks, pooling all targets \(Z_{\psi_{1}}(s^{\prime},a^{\prime}),\ldots,Z_{\psi_{N}}(s^{\prime},a^{\prime})\) into one set, and then sorting them in ascending order. The \(kN\) smallest of these targets \(y_{i}\) are used to define the target distribution. TQC approximates the locations for quantile fractions \(\tau_{m}\), where \(m\in[1...M]\) with \(\theta^{1}_{\psi_{n}}(s,a),\ldots,\theta^{M}_{\psi_{n}}(s,a)\) by minimizing  
\[J_{Z}(\psi_{n})=\mathbb{E}_{D,\pi}\Big{[}\mathcal{L}^{k}(s_{t},a_{t};\psi_{n} )\Big{]}.\]  
The parameters \(\psi_{n}\) are trained by minimizing the quantile Huber loss given by the following:  
\[\mathcal{L}^{k}(s,a;\psi_{n})=\frac{1}{kNM}\sum_{m=1}^{M}\sum_{i=1}^{kN}\rho^ {H}_{\tau_{m}}\Big{(}y_{i}(s,a)-\theta^{m}_{\psi_{n}}(s,a)\Big{)},\]  
where \(\rho^{H}_{\tau}(u)=|\tau-\mathbb{I}(u<0)|\mathcal{L}^{1}_{H}(u)\) is the Huber quantile loss and \(\mathcal{L}^{1}_{H}(u)\) is the Huber loss with parameter 1. The rationale behind truncating some quantiles from the target distribution was to prevent overestimation bias. In TQC, the number of dropped targets per network \(d=M-k\) is a hyperparameter that must be tuned per environment, allowing for fine-grained control of the bias. The TQC algorithm utilizes multiple critic networks to effectively manage uncertainty. Each network is trained on distinct data subsets, leading to diverse predictions. This ensemble technique improved the stability of the model and prevented overfitting. Moreover, employing multiple networks aids in capturing environmental variability more effectively, thereby enabling more reliable decision-making in unexpected scenarios  
The policy parameter \(\phi\) is trained similarly to SAC by maximizing the entropy-penalized estimate of the Q-value, which is the expectation over the distribution obtained from the critic:  
\[J_{\pi}(\phi)=\mathbb{E}_{D,\pi}\bigg{[}\alpha\log\tau_{\phi}(a\mid s)-\frac{ 1}{NM}\sum_{m,n=1}^{M,N}\theta^{m}_{\psi_{n}}(s,a)\bigg{]}.\]  
This method encourages the agent to explore more by maximizing entropy. This allows the agent to explore the environment better and reach new states, ultimately leading to the development of a better long-term strategy. Another important advantage of the TQC is its stability. The TQC algorithm gathers and uses more information for each state-action pair, improving the sample efficiency, which is crucial in data-limited scenarios. By explicitly modeling uncertainty, the TQC algorithm enables agents to make more reliable decisions and better handle the complexity and variability of the environment. Thus, TQC improves upon existing methods by explicitly modeling the distribution of returns and using an ensemble of networks to better handle inherent uncertainty and variability in RL environments. This helps reduce the overestimation bias and ultimately enhances the performance and stability of the agent. The TQC algorithm is particularly effective in high-dimensional and complex environments and can significantly improve the performance of RL agents in various applications. However, since the target distribution is adjusted by removing extreme quantiles, some useful training data may be lost, reducing sample efficiency. This issue becomes more pronounced, especially in the early stages of training when data are limited, and it can cause TQC to require more data compared to conventional RL methods.