# Planar Delaunay Mesh Smoothing Method Based on Angle and a Deep Q-Network
## 2 Smoothing Method Based on Angle and Deep Q-Network
### Workflow of the Algorithm
#### 2.3.3 Step 2: Training of DQN Model  
In this paper, the reinforcement learning model with inner polygon constraints is designed as follows, with a design reference to [PERSON] [27]. Agent: the central node of the polygon. Environment: polygon and inner polygon. Action vector: action = {up, down, left, right}, move step reference to [PERSON] [16], take \(Move\ step=0.02\cdot l_{min}\), where \(l_{min}\) is the shortest side length of the polygon. State vector: \(State=\{\delta_{c},\delta_{1},\ \cdots,\ \delta_{k},\theta_{1},\ \cdots,\ \theta_{v_{2}}\}\), where \(\delta_{c}\) is the coordinate of the agent, \(\delta_{k}\) is the vertex coordinate of the polygon, \(k\) is the number of polygon nodes, \(\theta_{v_{2}}\) is the vertex coordinate of the inner polygon, and \(v_{2}\) is the number of inner polygon nodes. Reward function: According to the design in [27], a new function is designed, as shown in Table 1. In the new reward function, a weak penalty is applied when the intelligence moves out of the inner polygon, and a strong penalty is applied when it moves out of the polygon. Figure 5: The data structure of mesh data information. We set the hyperparameter learning rate to four levels: 0.1, 0.01, 0.0001, and 0.0001. The training results show that when the learning rate is 0.01, the learning efficiency and quality are relatively balanced, and the mesh optimization effect is optimal. During the training process, the agent adopts the Epsilon-Greedy algorithm. The epsilon is 0.9, which means that the agent has a 90% probability of randomly taking action during the training process and a 10% probability of taking action based on the maximum Q value. The remaining hyperparameters are set regarding Zhang [27]. The parameter settings of the neural network are shown in Table 2.  
Neural networks act as actuators in deep reinforcement learning to compute and store Q-values. The agent observes the state and takes action, and the state of the environment is updated to the next moment. The environment feeds the agent with a reward based on the node quality and constraints at the next moment. The agent recalculates the Q-value based on the state and reward of the current moment and the delayed moment. The agent iterates the above steps during training to find the optimal policy that maximizes the cumulative Q-value. The DQN model is saved at the end of the training, and the update process of the neural network parameter \(\theta\) is as follows:  
Input: state vector \(S\), action vector \(A\), learning rate \(\alpha\), decay rate \(\gamma\), epsilon \(\epsilon\), current Q-network Q, target Q-network Q ', batch size \(m\), maximum round \(M\), maximum step \(T\), and update frequency \(C\) of the target Q-network. Output: parameters \(\theta\) of the current Q-network. 1. The program reads the information about the dataset stored in memory. Normalize the dataset features by scaling the polygon to be within a square region with a side length of 1;
2. Initialize experience playback pool D, randomly initialize the parameters of the real Q-network \(\theta\), and initialize the parameters \(\theta^{\prime}\) of the target Q-network equal to \(\theta\);
3. The program reads the state vector of the dataset and obtains the eigenvector \(\Phi(S_{t})\). Using \(\Phi(S_{t})\) as the input in the Q-network, calculate the Q-values corresponding to all actions. \(\epsilon\)--the greedy algorithm selects the corresponding action from the current Q-value. The agent has a probability of 0.9 to randomly select actions to search for other strategies to maximize rewards and benefits. There is a 10% probability of selecting the action corresponding to the maximum Q-value;
4. Execute action \(A_{t}\) in the current state \(S_{t}\), to obtain the feature vector corresponding to the next state \(\Phi(S_{t+1})\), reward \(R_{t}\), and terminate status conducted;