## 2 Network Architecture
### Gctd3  
In the paper [20], the actor and critic networks used contiguous Fully Connected (FC) layers, so the weights learned only from these FC layers made the model more difficult to achieve convergence. Thus, the paper [9] proposed a method of combining RNN into TD3 to increase input information from many previous consecutive states, however that made the neural network more complex and difficult to train with weak hardware. In this paper, we propose to use the GCTD3 network for two purposes:* The graph convolutional layers of our GCTD3 show efficiency with data in the form of the graph structure of the bipedal robot. The joints will learn each other's relationships and constraints through these layers, the joints that are not connected will not affect each other. For example, the right knee joint and the right hip joint can share their attributes with each other, but the left knee joint and the right knee joint are not directly related;
* GCNs do not need to use a large number of weights to increase the joint features of the robot joints, so the computational volume is not too large even when using many previous states combined as the input of the neural network. According to Equation (7), the number of weights \(\Theta\) used for a joint to increase the number of its features from \(F\) to \(F^{\prime}\) is only equal to \(F\times F^{\prime}\).  
In this paper, we modelled a 2-legged robot with six independent joint coordinates to test the algorithm (details on the robot model are presented in Section 3.1). We created a graph with seven nodes located at the joints of the robot and one node located at the ground, thereby building the adjacency matrix for the bipedal robot. The order of the robot's nodes is shown in Figure 2, the adjacency matrix \(\hat{A}\) and the diagonal degree matrix \(\hat{D}\) built according to the principle in Section 2.1 are also shown in Figure 2.  
The characteristics of a node in the next state will be calculated based on the characteristics of that node in the previous state and the neighboring nodes affecting that node. This relationship is modeled in the form of Graph Convolutional layers and based on Equation (7), the layers of graph convolution network are represented as (14):  
\[H^{l+1}=\sigma\left(\hat{D}^{-\frac{1}{2}}\hat{A}\hat{D}^{-\frac{1}{2}}H^{l}W^ {l}\right) \tag{14}\]  
where \(H^{l+1}\), \(H^{l}\) are the features matrices of the nodes at \((l+1)^{\text{th}}\) and \(l^{\text{th}}\) layers, \(W^{l}\) is the weights' matrix at the \(l^{\text{th}}\) layer trained by the neural network and \(\sigma()\) is the activation function, in this paper we used ReLu function.  
The neural network structure used is shown in Figure 3. The input of the algorithm is the sensor signals placed at eight positions consisting of the joints of the robot, the body of the robot, and the ground. In order to increase the input information for the neural network, the most recent five consecutive states are used (His). At a state, each node is characterized by a 2-D input feature vector (In) consisting of the position on the z-axis and the velocity on the y-axis of the body robot node, angular velocity and the angular position of the joints of the robot and the ground contact states of two feet. Since the last five states are used, each node will have a total of 10 features, so the input dimension of the GCTD3  
Figure 2: The order of nodes, linking edges for the bipedal robot; the adjacency matrix \(\hat{A}\) and the diagonal degree matrix \(\hat{D}\) built based on the defined nodes and edges.  
is (8, 10). After three layers of the GCN with the number of hidden features 32, 32 and 16, respectively, each node will be represented by a 16-features vector, these features are the relationships between the nodes affecting each other. Then, we concatenate the features of eight nodes before introducing two fully connected layers and a last linear layer to give the desired angles for the joints of the robot for the next state.