## 4 Methods
This involves executing multiple steps of gradient descent on each task and then updating the global parameters. Assume the model parameters are \(\theta\), and for a randomly selected task \(T_{i}\), the model's loss function on this task is \(L_{T_{i}}(\theta)\). The parameter update rule is described as follows:  
**Step 1--In-task Updates:** For each task \(T_{i}\), initialize the temporary parameters \(\theta^{\prime}\) to the global parameters \(\theta\). Then, perform \(K\) iterations of gradient descent updates, expressed as:  
\[\theta^{\prime}=\theta^{\prime}-\beta\
abla_{\theta^{\prime}}L_{T_{i}}(\theta ^{\prime}),\]  
where \(\beta\) is the in-task learning rate. **Step 2--Global Updates:** After the \(K\) gradient updates for task \(T_{i}\), the global parameters \(\theta\) are updated based on the average gradient changes across all tasks:  
\[\theta=\theta+\alpha(\theta^{\prime}-\theta),\]  
where \(\alpha\) is the global learning rate. The specific algorithm is as Algorithm 1:  
```
for each iteration do  Sample task \(T_{i}\) \(\theta^{\prime}=\theta\) for\(k=1,2,\dots,K\)do \(\theta^{\prime}=\theta^{\prime}-\beta\
abla_{\theta^{\prime}}L_{T_{i}}(\theta ^{\prime})\) endfor \(\theta=\theta+\alpha(\theta^{\prime}-\theta)\) endfor
```  
**Algorithm 1** Reptile Algorithm [28]. In the Reptile algorithm, the objective of each iteration is to minimize the loss function \(L_{T_{i}}(\theta)\) of task \(T_{i}\). The algorithm achieves this by performing \(K\) steps of gradient descent on task \(T_{i}\), and then uses these updates to adjust the global parameters \(\theta\). In-task updates aim to quickly adapt to the current task \(T_{i}\), while the global updates seek to find a good starting point for the parameters so that for new tasks, effective learning can be achieved with only a few steps of gradient updates. Based on the above, we combine GAIL with the meta-reinforcement learning Reptile algorithm. The core idea is to use GAIL for imitation learning when training subtasks and then use these training results to update the meta-parameters. This process can be divided into the following key steps:  
Figure 6: Principle of Reptile. **Step 1:** Selection and preparation of subtasks--Choose multiple subtasks from the task distribution and prepare expert demonstration data for each task. **Step 2:** Training subtasks using GAIL--For each subtask, train using the GAIL framework. * Generator (i.e., policy network) training: Train the policy network by attempting to imitate expert demonstrations. * Discriminator training: The discriminator learns to distinguish between the behaviors generated by the policy network and the expert demonstrations. **Step 3:** Policy network update--After completing the GAIL training for each subtask, the policy network is improved. **Step 4:** Updating meta-parameters--After completing training on all selected subtasks, update the meta-parameters using the logic of the Reptile algorithm. This usually involves averaging or integrating the parameter changes obtained across multiple tasks in some form. **Step 5:** Repeat the process--Repeat the above steps until the model performance reaches the expected target or until the predetermined training cycles are completed. Additionally, during the training process, we implemented normalization of the states with the following steps:  
(1) At the start of training, we first store historical training data in the experience replay buffer. (2) Upon completion of training, we calculate the mean and variance of each dimension of the state in the replay buffer, concluding the entire training process. (3) Before commencing the next training session, all states inputted into the network are normalized. Through this methodology, GAIL enhances the policy network's ability to closely replicate expert behaviors across a variety of subtasks. Following this, the Reptile algorithm amalgamates the insights gained to refine the meta-parameters, thus facilitating swift adaptation to new tasks that bear resemblance to those previously encountered.