## 4 Design of Proposed Algorithm
### Algorithm Design
#### 4.3.2 MTD3 CO
The inner-loop learning combines the Seq2 Seq network with the TD3 algorithm, enabling its actor-critic network to adapt to training in environments with different numbers of subtasks. Compared with the VPG algorithm in the literature [25], it has better exploration ability and training stability. For each learning task \(T_{i}\), according to the objective function above we define its value network objective function as \(J_{critic}^{TD3}(\theta_{i}^{T_{i}})\) with the objective of maximizing the expected gain, as shown in Algorithm 1, using the objective function gradient to update \(\theta_{1}\) and \(\theta_{2}\). Each task is updated a certain number of times to obtain the set of value network parameters; similarly, for the strategy network, we define the objective function as \(J_{actor}^{TD3}(\phi^{T_{i}})\), with the objective of minimizing the TD error by updating the parameters \(\phi^{T_{i}}\) with the gradient. In outer-loop learning, following the theory of model-agnostic meta-learning proposed in the literature [5], the computation of the outer-loop learning objective function can be obtained; hence, for the meta critic network of outer-loop learning, its definition can be expressed as follows:  
\[J_{critic}^{\text{MTD3 CO}}(\theta_{1,2})\ =\ \mathbb{E}_{T_{i}\sim\rho(T)}\left[J_{ critic}^{TD3}\left(U\left(\theta_{1,2}^{T_{i}},T_{i}\right)\right)\right], \tag{22}\]  
where \(J_{critic}^{TD3}\) is the objective function of the inner-loop learning critic network for task \(T_{i}\), and \(U(\theta,T_{i})\) is the parameter after the task-set number of inner-loop learning gradient updates,which is defined as \(U\Big{(}\theta_{1,2}^{T_{i}},T_{i}\Big{)}\ =\ \theta_{1,2}^{T_{i}}+\alpha\sum_{t =1}^{k}\
abla_{\theta_{1,2}^{T_{i}}}\int_{crit}^{TD3}\Big{(}\theta_{1,2}^{T_{i} }\Big{)}^{t}\), where k is the number of gradient updates for inner-loop learning and outer-loop learning with the objective of minimizing the objective function with gradient updates of the critic meta-parameters, but this objective function involves gradients of gradients that bring huge computational cost under a complex network such as Seq2 Seq. To solve this problem, we use the first-order approximation method in [26]; the gradient of \(f_{critic}^{\text{MTDSCO}}(\theta_{1,2})\) can be expressed as  
\[\
abla_{\theta_{1,2}}f_{critic}^{\text{MTDSCO}}(\theta_{1,2})\ =\ grad_{critic}^{\text{MTDSCO}}\ =\ \frac{1}{n}\cdot\frac{1}{k}\sum_{i=1t}^{n}\sum_{t=1}^{k}\
abla_{ \theta_{1,2}^{T_{i}}}\int_{crit}^{TD3}\Big{(}\theta_{1,2}^{T_{i}}\Big{)}_{t}, \tag{23}\]  
where \(n\) denotes the number of all tasks, \(k\) denotes the number of inner-loop training gradients, and similarly, for the meta-actor network with outer-loop learning, its objective function and approximate gradient can be obtained as  
\[f_{actor}^{\text{MTDSCO}}(\phi)\ =\ \mathbb{E}_{T_{i}\sim\rho(T)}\big{[} \int_{actor}^{TD3}\big{(}F\big{(}\phi^{T_{i}},T_{i}\big{)}\big{)}\big{]}, \tag{24}\]  
\[F\big{(}\phi^{T_{i}},T_{i}\big{)}\ =\ \phi^{T_{i}}+\alpha\sum_{t\ =1}^{k}\
abla_{\phi^{T_{i}}}\int_{actor}^{TD3}(\phi^{T_{i}})^{t}, \tag{25}\]  
\[\
abla_{\phi}f_{actor}^{\text{MTDSCO}}(\phi)\ =\ grad_{actor}^{\text{MTDSCO}}\ =\ \frac{1}{n}\cdot\frac{1}{k}\sum_{i =1t}^{n}\sum_{t=1}^{k}\
abla_{\phi^{T_{i}}}\int_{actor}^{TD3}\big{(}\phi^{T_{ i}}\big{)}_{t}. \tag{26}\]  
According to the objective function, we describe the overall training procedure in Algorithm 1. ```
Input: Task distribution \(\rho(T)\) Output: DAGs
1. The parameters of the random initialization policy network \(\pi_{\phi}\) and the two value networks \(Q_{\theta_{1}}\) and \(Q_{\theta_{2}}\) are \(\phi\), \(\theta_{1}\), and \(\theta_{2}\)
2. Initialize the parameters of the target network \(\phi^{\prime}\leftarrow\phi\), \(\theta_{1}^{\prime}\leftarrow\theta_{1}\), \(\theta_{2}^{\prime}\leftarrow\theta_{2}\)
3. for iteration \(k\ =\ \{1,2,\dots,K\}\)do
4. According to the task distribution \(\rho(T)\), random sample of n tasks \(\ \{T_{1},T_{2},T_{3},\dots,T_{n}\}\)
5. for task \(T_{i^{\prime}}\), \(i\ =\ \{1,2,\dots,n\}\)do
6.