## 4 Simulation Analysis
### Simulation Result
#### 4.3.1 Task 1  
Take the task of controlling PCM to reach the target point (0, 80, 740) mm as the new task. To evaluate the generalization performance, both the policy initialized using the MAML meta-reinforcement learning and the baseline policy are trained to adapt to new tasks using the REINFORCE algorithm, and their generalization performances are compared. The training processes for both policies employ the same hyperparameters. The training result is shown in Figure 4.  
As illustrated in Figure 4, when faced with a new task, the policy initialized with MAML meta-reinforcement learning reaches convergence in the excepted reward after 60 updates with a training time of 5 min. The baseline policy reaches convergence in the excepted reward after 180 updates with a training time of 15 min. By comparison, it is evident that the MAML meta-reinforcement learning algorithm has learned effective initial weights for the policy network from historical experiences. These initial weights enable the policy to adapt to new tasks with minimal updates, improving the adaptive capability for end position control of the PCM.  
After the training of the policy initialized by MAML meta-reinforcement learning is completed, we use the trained policy to control the PCM. The position changes and the position error changes of the PCM are shown in Figures 5 and 6, respectively.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Values** \\ \hline Learning rate \(\alpha\) & \(1\times 10^{-14}\) \\ Size of the trust-region \(\delta\) & \(1\times 10^{-2}\) \\ Conjugate gradient iterations & 10 \\ Maximum line search iterations & 15 \\ Line search backtrack coefficient & 0.8 \\ Inner loop task samples & 20 \\ Number of trajectories collected per task & 20 \\ Inner loop optimization iterations & 1 \\ Outer loop task samples & 20 \\ Outer loop optimization iterations & 1000 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Hyperparameters of MAML meta-reinforcement learning.  
Figure 4: Result of Task 1.  
The position error is (2.77, \(-\)2.82, \(-\)2.78) mm, and the maximum error is within 1.4%. Thus, the policy initialized with MAML meta-reinforcement learning can effectively control the PCM to reach the new target point after training.  
To compare the performance with other reinforcement learning algorithms, the control policies for PCM are trained using the PPO algorithm, the Actor-Critic algorithm, and the PILCO algorithm. The hyperparameters of the PPO algorithm are as follows: the clipping parameter \(\varepsilon\ =0.2\), the learning rate for the policy network \(l_{a}=1\times 10^{-4}\), and the learning rate for the value network \(l_{c}=5\times 10^{-3}\). Both the policy network and the value network have one hidden layer, and the number of neurons in the hidden layer is 256. The hyperparameters of the Actor-Critic algorithm are as follows: the learning rate for the policy network \(l_{a}=1\times 10^{-4}\), and the learning rate for the value network \(l_{c}=5\times 10^{-3}\). Both the policy network and the value network have one hidden layer, and the number of neurons in the hidden layer is 256. After the training of the policy, we use the trained policy to control the PCM. The position changes and position error changes of the PCM are shown in Figures 7 and 8, respectively.  
Figure 5: The position changes of the PCM after policy adaptation in Task 1.  
Figure 6: The position error changes of the PCM after policy adaptation in Task 1.  
The training times and errors of these algorithms are shown in Table 2.  
From Table 2, it can be seen that the policy initialized with MAML meta-reinforcement learning requires the shortest training time and achieves higher control precision during new task training. Compared to other algorithms, this approach enhances both training speed and control accuracy.