## 3 Preliminary
### Meta-RL  
Unlike conventional reinforcement learning approaches that are tailored for singular tasks, meta-reinforcement learning (Meta-RL) aspires to craft policies that are capable of rapid adaptation or optimization when faced with a sparse influx of data from new tasks [26]. The prevailing methodologies in Meta-RL typically presuppose a task framework where all constituent subtasks are unified by identical state and action spaces. The distinctions among these tasks often emerge through variations in the reward functions, the probabilities of state transitions, or the distributions of initial states. For instance, in the context of robotic object-grasping experiments, diverse tasks may entail the manipulation of different objects within a consistent physical environment. Despite the variability in tasks, the robot's action space (such as arm movements) and state space (encompassing joint angles and object locations) remain unchanged. Tasks are derived from the distribution \(\mathcal{T}_{i}\sim p(\mathcal{T})\), with each task \(\mathcal{T}_{i}\) characterized by \((S,\mathcal{A},p_{i},r_{i},\gamma)\). Given the shared framework of state and action spaces across tasks, the task distribution \(p(\mathcal{T})\) is delineated by the reward function and the initial state distribution \(p(s_{0})\), thus formulated as \(p(\mathcal{T})=p(R)\times p(s_{0})\). In the meta-training phase, tasks are drawn from the task distribution \(p(\mathcal{T})\) to facilitate the training process. The fundamental goal during this stage is to equip the model with the ability to swiftly acclimate to novel tasks. Following this, in the meta-testing phase, the model is presented with a series of new tasks that, while related, are distinct from those encountered during training. This phase is critical for evaluating the model's capacity for adaptability and its proficiency in assimilating new tasks. Throughout the meta-training period, the agent utilizes specific meta-learning parameters or employs update mechanisms designed to foster this rapid adaptation, which is then put to the test in the subsequent meta-testing phase. The intricacies of the meta-training process are \begin{table}
\begin{tabular}{p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt} p{56.9 pt}} \hline \hline
**[PERSON]** & **[PERSON] et al. [17]** & **[PERSON] et al. [18]** & **[PERSON] et al. [19]** & **[PERSON] et al. [20]** & **[PERSON]** \\ \hline Learning Method & DRL & Q-learning & RL + TEXPLORE & DRL (TD3) & Meta-RL + IL \\ \hline \multirow{3}{*}{Key Contribution} & UAV attitude stability in extreme conditions & Efficient obstacle avoidance & Navigation and obstacle avoidance in unknown environments & Improved navigation in adaptability and generalization \\ \cline{2-6}  & Limited to attitude control & Low sample efficiency, poor generalization & Low sample efficiency & Low sample efficiency, poor generalization & The training time is relatively long \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of related work. illustrated in Figure 2, highlighting the structured approach to achieving adaptability and flexibility in task learning. The process of meta-reinforcement learning can be subdivided into two critical cycles: the inner loop and the outer loop. 1. **Inner Loop:** * During the inner loop phase, the agent leverages meta-knowledge provided by the outer loop, which is crucial for application. * The main goal of this phase is to efficiently use meta-knowledge for quick adaptation to new tasks or environments. * The effectiveness of the meta-knowledge is gauged by the agent's performance in this segment, serving as a vital evaluation criterion. 2. **Outer Loop:** * The outer loop's primary role is to evolve and fine-tune the meta-knowledge itself. * This stage involves a thorough examination of similarities and differences across tasks to enhance the meta-knowledge that facilitates learning. * It also updates the meta-knowledge based on feedback from the inner loop, ensuring its efficacy across diverse tasks.