# Entropy-Guided Distributional Reinforcement Learning with Controlling Uncertainty in Robotic Tasks
## 4 Experiments  
In this study, extensive experiments were conducted on robotic control to explore complex movements and interactions in unpredictable environments. These experiments are essential for understanding how well RL algorithms can adapt and perform in real-world applications. To achieve this, we tested our algorithms in the DeepMind Control Suite [37] and fetch environments from Gymnasium robotics [38]. The DeepMind control suite offers a set of standard tasks for continuous control that are commonly used to assess the performance of RL algorithms. The Fetch environment in the Gymnasium robotics provides a variety of robotic tasks, including manipulation and navigation, creating diverse and challenging scenarios for algorithm evaluation. Both environments use the MuJoCo physics engine (version 2.1.0, developed by DeepMind),which is ideal for accurately modeling the physical interactions and dynamics of complex robotic systems. By utilizing the same physics engine across different environments, we ensured consistent simulations, allowing for a more reliable comparison of the proposed algorithm's performance. The robotic tasks in the experimental environments are defined as MDP, which consist of a state space \(\mathcal{S}\), an action space \(\mathcal{A}\), a reward function \(R\), state transition probabilities \(P\), and a discount factor \(\gamma\). For instance, in the Hopper task of the DeepMind Control Suite, the state space \(\mathcal{S}\) includes the positions, velocities, and balance states of the robot's joints. The action space \(\mathcal{A}\) consists of the magnitude and direction of the forces applied to these joints. The reward function \(R\) is designed to encourage specific behaviors, such as maintaining balance or reaching a target location within a designated time frame. The state transitions \(P\) are governed by the dynamics modeled by the MuJoCo physics engine, which predicts the next state based on the robot's actions. The discount factor \(\gamma\) determines the importance of future rewards, which is dynamically adjusted in the proposed algorithm based on entropy. In the FetchPickAndPlace task from the Gymnasium robotics environment, the state space includes the positions of the robotic arm, the target object, and the goal location. The action space consists of the movement directions for the robotic arm and grip control actions, such as opening or closing the gripper. The reward function provides feedback based on how closely the object approaches or successfully reaches the target location. These components highlight the complexity and physical interactions involved in robotic tasks, enabling the agent to learn effectively in various environments. We utilized the TQC algorithm in all our experiments. The TQC aims to enhance the stability and performance of RL agents by better handling the uncertainty and variability in dynamic environments. To ensure a fair comparison, we kept all other codes and parameters identical across different experimental setups. This involved using the same network architectures, learning rates, and other hyperparameters as well as maintaining consistent training protocols and evaluation metrics. The hyperparameter settings according to our experiments are listed in Table 1. Our algorithm dynamically adjusts the discount factor based on the entropy loss. The discount factor is a crucial parameter in RL, because it determines the importance of future rewards. Traditional approaches often use a fixed discount factor, which can lead to suboptimal performance in environments with varying levels of uncertainty. In contrast, our approach allows the discount factor to vary in response to the entropy loss, providing a more adaptive and flexible learning process. This dynamic adjustment aims to balance the trade-off between exploration and exploitation more effectively, thereby enhancing an agent's ability to learn from its environment.