## 3 Training Process  
### Simulation Environment  
An overview of the robot training and simulation process is shown in Figure 4. ROS is used to transmit signals between the Gazebo simulation environment and the RL algorithm. The model receives the state from Gazebo when the robot moves in the environment, then calculates the reward corresponding to that state to put the input state into the GCTD3 algorithm and find the appropriate action for the robot in the next state. Nodes and topics communicate with each other thanks to ROS which is shown in Figure 5. In Figure 5, when the robot walks in the Gazebo simulation environment, /gazebo node publishes data about the robot's transformation and the ground contact state of the feet to topics /Joint_states, /FootR_contact and /FootL_contact. After that, the /walker_controller node subscribes data provided from the three topics so that RL algorithms can process and find a proper action. These found values are published to six topics corresponding to six robot joints and /gazebo node continues getting these six values from the six topics for the bipedal robot to perform in the next state. The components used in testing the algorithm are detailed in this section. Figure 3: The actor network and two critic networks structure in GCTD3 algorithm: N is the number of nodes in the graph; “In” is the number of features of each node at a state; and “His” is the number of most recent consecutive states used during training, the activation function is ReLu. The robot model has the shape and size as shown in Figure 6. The bipedal robot consists of seven links (the left and right thigh, the left and right shank, the left and right foot and a body), we selected Acrylonitrile butadiene styrene (ABS) material for all links of the robot. The weight and height of each link are presented in Table 1.  
Figure 4: Diagram of the main components in the training and simulation process of the bipedal robot. Figure 5: Communication between nodes and topics in ROS. Ovals represent nodes and rectangles represent topics. The software used for the simulation is Gazebo and the Robot Operating System is used to get state information from the environment for computational processing to find the appropriate action and then send it back to the environment. * Environment: Gazebo simulation environment receives control signals through the ROS with a frequency of 50 Hz, i.e., ROS will take the state from the environment and calculate the reward of that state to store data into a replay buffer every 0.02 s;
* State: At each joint of the robot model, sensors are placed to measure the information received from the environment for the agent to process. The sent signals contain information of the angular position and angular velocity of the joints, the coordinates of the robot's body in the vertical direction and the robot's speed according to the robot's movement direction, and the signals from the sensor located at the feet to know whether the robot touches the ground or not, five consecutive states from \(t-5\) to \(t\) are concatenated again to form the input state of the neural network during training;
* Action: Before sending the robot control signals, the action corresponding to the desired angle coordinates of the joints is calculated by our proposed network, the action consists of six values corresponding to the desired angle of six joints: left hip joint, right hip joint, left knee joint, right knee joint, left ankle joint and right ankle joint. The angle ranges of these joints are limited as shown in Table 2;
* Reward: We use a dense reward function based on the actual human gait to make the algorithm easier to converge. While walking, the robot always has two states: one-foot contacts with the ground and two feet contact with the ground (Figure 7). In each state, the height of the body robot has different values. In the paper [25], only an average