## 4 Experiments
### Comparison of HER and HER-CA  
Firstly, we configured the environment. We set the joint limits for the initial two joints of the robotic arm to \(\pm 45^{\circ}\), and applied a \(45^{\circ}\) offset to them, creating a two-dimensional joint space where both dimensions ranged from 0 to \(90^{\circ}\). During the environment's initialization, the starting point, obstacle point, and target point were indiscriminately positioned within the joint space. The path is judged to be a collision when it is within \(3^{\circ}\) of an obstacle, and a reach when it is within \(2^{\circ}\) of the target point. The actions given by the action model plus a Gaussian noise were explored for each game, and the maximum number of steps in each game was five. Both the action model and the value model use a fully connected neural network with 3 hidden layers, each having 256 neurons. The action model uses the ReLU activation function and the network output is processed using the Tanh function with the learning rate set to \(3\times 10^{-5}\). The value model uses the ReLU activation function and the learning rate is set to \(3\times 10^{-4}\). All networks use the Adam optimizer. Each epoch plays 200 games and stores the data from each game in the data pool, and then performs 20 network trainings. Each network training is randomly collected from the data pool for 50 steps, where each step has an 80% probability of being processed by HER or HER-CA. In addition a soft update of the network model is performed at the end of each epoch. Data acquisition was achieved by testing preserved action network models. Specifically, each action model was saved at intervals of every 10 epochs, and the success rate of each model was ascertained by running 5000 games with every individual action model. The initial experiment sought to evaluate the effectiveness between obstacle resetting and obstacle moving away in the context of using the HER-CA method. As depicted in Figure 6, using the move away strategy slightly outperforms the resetting method, exhibiting a marginally higher average and peak success rate. Subsequent experiments compared the effects of HER and HER-CA, with setups involving one obstacle and two obstacles for model training scenarios. The experimental results are shown in Figure 7, which indicates that the model trained by HER-CA exhibits an average success rate improvement of about 10% compared to the model trained by HER. We used the results of the success test for the full training process to calculate the average success rate, and the success rate for the 5000 epochs at the end of the training was used for interval estimation to calculate the confidence level. The average success rate improvement of HER-CA in the case of one obstacle was 9.4%, and the confidence level for an improvement greater than 5% was 71.1%. The average success rate of HER-CA in the case of two obstacles was improved by 10.1%, and the confidence level for an improvement greater than 5% was 88.5%. The one-obstacle case HER-CA carried less effect enhancement than the two-obstacle case, and the reason for this phenomenon may be that one obstacle is inherently less difficult to avoid, leading to the fact that HER can also be trained slightly more efficiently. In the third experiment we tested the performance of HER-CA in a 3D path finding task. The space is a cube space with a side length of 45, in which Obstacles and target points were randomly generated in a cubic space with coordinates from 3 to 45. The performance of the HER-CA and HER was tested for the 2 obstacles and 3 obstacles cases, respectively. A decaying learning rate is used, with an initial learning rate of \(5*10^{-6}\) for the action model  
Figure 6: Comparison plot of training success rates for different collision-free experience generation methods. Blue represents obstacle reset red represents obstacle removal, and the shaded area represent one standard deviation. and \(5*10^{-5}\) for the value model, decaying by half every 20,000 epochs.