# Entropy-Guided Distributional Reinforcement Learning with Controlling Uncertainty in Robotic Tasks
## 2 Background  
### Distributional Reinforcement Learning  
Standard RL models the expected value of a random return for a specific action in a given state using Q-learning. An agent is modeled as a Markov Decision Process (MDP) consisting of \((\mathcal{S},\mathcal{A},R,P,\gamma)\). \(\mathcal{S}\) and \(\mathcal{A}\) represent the state and action spaces, respectively. \(R\) is the reward, \(P\) is the transition probability, and \(\gamma\) is the discount factor. The agent makes decisions based on a policy \(\pi(a|s)\) that maps state \(s\in S\) to actions \(a\in A\). The Q-learning method is to model the value function \(Q_{\pi}\) as the following Bellman equation:  
\[Q(s,a)=\mathbb{E}[R(s,a)]+\gamma\mathbb{E}\big{[}Q(S^{\prime},A^{\prime}) \big{]},\]  
where the random variables \(S^{\prime}\) and \(A^{\prime}\) are distributed according to \(P(\cdot\mid s,a)\) and \(\pi(\cdot\mid s^{\prime})\). RL aims to determine the optimal policy \(\pi^{*}\) by maximizing the cumulative rewards, which in turn involves finding the optimal \(Q^{*}\) function. This can be performed by iteratively applying the Bellman update to improve the estimated \(Q_{\theta}\) as the parameterized optimal \(Q^{*}\) function. Figure 1: Architectures of distributional reinforcement learning algorithms. \[Q_{\theta}(s,a)\leftarrow\mathbb{E}[R(s,a)]+\gamma\mathbb{E}\bigg{[}\max_{a^{\prime}} Q_{\theta}(s^{\prime},a^{\prime})\bigg{]}\]  
The distributional RL models the entire probability distribution of the random return rather than its expected value, showing improvements in sample diversity and performance. Rather than using the Q-function, the Z-distribution is used to model the return distribution for the state-action pairs. \[Z^{\pi}(s,a)\overset{D}{=}R(s,a)+\gamma Z^{\pi}(S^{\prime},A^{\prime}),\]  
where A \(\overset{D}{=}\) B denotes the equality of the probability laws. The Q function is calculated as follows:  
\[Q^{\pi}(s,a)=\mathbb{E}[Z^{\pi}(s,a)]=\mathbb{E}\bigg{[}\sum_{t=0}^{\infty} \gamma^{t}R(s_{t},a_{t})\bigg{]},\]  
\[x_{t}\sim P(\cdot\mid s_{t-1},a_{t-1}),a_{t}\sim\pi(\cdot\mid s_{t}),s_{0}=s,a _{0}=a.\]  
Modeling the entire probability distribution of returns can provide more information than before. The C51 algorithm [11] utilizes a categorical distribution to represent the return distribution over a fixed set of equidistant points and aims to minimize the KL divergence to the projected distributional Bellman target. \[Z^{\pi}(s,a):=\sum_{i=1}p_{i}\delta_{z_{i}},\]  
where \(z_{i}\) is a fixed set of return values and \(\delta\) is the Dirac delta function. The C51 outperformed previous DQN variants on the Atari-57 benchmark. QR-DQN [12] addresses the limitation of C51 by approximating the return distribution \(Z^{\pi}(s,a)\) as follows:  
\[Z_{\varphi}(s,a):=\frac{1}{M}\sum_{m=1}^{M}\delta(\theta_{\psi}^{m}(s,a)),\]  
where \(M\) denotes the number of atoms in the distribution and \(\theta_{\psi}^{m}(s,a)\) is the Dirac delta function at locations. This is a mixture of atoms provided by a parametric model \(\theta_{\psi}:S\times A\rightarrow\mathbb{R}^{M}\). The \(\psi\) is the parameter of target network and can be optimized by minimizing the 1-Wasserstein distance [26] between the temporal difference target distribution \(T^{\pi}Z_{\psi}\) and \(Z_{\psi}\). The distributional RL models the distribution of returns, enabling an agent to consider various potential returns. This increases the diversity of the sample data and improved the learning performance of the agent [27]. According to [PERSON] et al. [1], the overall uncertainty of an agent can be divided into epistemic uncertainty and aleatoric uncertainty. By modeling the reward distribution, the agent gains a better understanding of aleatoric uncertainty. This involves comparing the return distributions using various probability metrics and studying the convergence properties of the distributional Bellman operator.