## 4 Methods  
Building upon the experiments previously mentioned, we introduced the Reptile algorithm [28] as our chosen method for meta-learning. Reptile is a gradient-based meta-learning algorithm aimed at identifying a set of initial parameters that enable rapid adaptation to new tasks with a minimal number of gradient updates. Its main strategy involves applying gradient updates across various tasks, with the goal of achieving a set of initial parameters that ensures good initial performance across a range of tasks. This approach allows the Reptile algorithm to significantly enhance the model's adaptability and flexibility when encountering new tasks. The structure of this methodology is depicted in Figure 5.  
Figure 4: The comparison of three algorithms under the CartPole task. The Reptile algorithm offers a computational advantage over the MAML approach [29], particularly due to its avoidance of second-order derivative computations. MAML seeks to establish a foundational model that, through minor parameter modifications across a spectrum of tasks, can adeptly adapt to new challenges. This methodology necessitates the refinement of model parameters for one task, followed by the application of these tweaked parameters to subsequent tasks, with additional adjustments required for each new task. Such a process demands intricate calculations of second-order derivatives, leading to considerable computational demands and extended processing times. Reptile, on the other hand, adopts a strategy of executing iterative parameter updates within individual tasks, eschewing MAML's approach of multiple parameter fine-tunings across various tasks. Reptile's primary aim is to extract more generalizable features by repetitively adjusting parameters over a collection of tasks, thereby facilitating swifter adaptation to novel tasks. Hence, while MAML's strength lies in crafting an initial model that can be fine-tuned for diverse tasks, Reptile streamlines the adaptation process to new tasks through a restricted series of iterative parameter modifications across tasks. This diagram (Figure 6) illustrates how a model updates its parameters progressively after training on multiple tasks, starting from its initial parameters, to enhance its generalization ability for new tasks. The key elements include:  
* **Initial Model Parameters \(\phi^{0}\)**: Represent the model's parameter settings before training begins. * **Paths in Different Colors**: Illustrate the optimization process of the model on various training tasks. For instance, a blue path indicates training on task n, while a light yellow path signifies training on task m.
* **Arrows**: Indicate the direction of the model parameters' movement in the parameter space during training. * **Model Parameters \(\phi^{1},\phi^{2},\ldots\)**: Denote the model parameters after a series of training steps. * **Final Model Parameters \(\hat{\theta}_{m},\hat{\theta}_{n}\)**: Represent the final state of the model parameters on their respective tasks. Figure 5: This figure depicts the global architecture of the model, demonstrating how to optimize the parameters of the strategy network through a multi-step iterative process, enabling rapid adaptation to various tasks. The main steps include: (1) Outer Loop: This phase is responsible for updating the meta-parameters \(\theta\). Throughout a series of training cycles, these meta-parameters are used to generate initial strategy network parameters adapted to various tasks. (2) Inner Loop: For each subtask, starting from the meta-parameters \(\theta\), data are generated through interaction with the environment, and the parameters \(\Phi\) of the strategy network are iteratively updated using the GAIL algorithm. (3) Parameter Update: After the inner loop training, the task-specific parameters \(\Phi\) obtained are used to update the meta-parameters \(\theta\), thus achieving rapid adaptation to new tasks. The essence of the Reptile algorithm lies in iteratively updating the model parameters to swiftly adapt to new tasks.