## 3 Offline Optimization and Adaptation Under Nonlinear Task Constraints
### Problem Formulation  
Given initial ProMPs distribution \(p_{0}(\mathbf{w})=\mathcal{N}(\mathbf{w};\mathbf{\mu}_{\mathbf{w}}^{0},\mathbf{\Sigma}_{\mathbf{w}}^{ 0})\), the main objective is to obtain the optimized ProMPs distribution \(p(\mathbf{w}^{*})=\mathcal{N}(\mathbf{w}^{*};\mathbf{\mu}_{\mathbf{w}}^{*},\mathbf{\Sigma}_{\mathbf{w}}^ {*})\), which is as close as possible to the original distribution while satisfying task constraints. We use the Kullback-Leibler divergence to represent the similarity between optimized and initial distributions owing to its analytical tractability, while expressing the constraints with cumulative distribution functions. However, in this paper, considering ProMPs model every timestep as an independent Gaussian distribution, we optimize the parameters of each timestep separately. Specifically, we convert the initial distribution \(p_{0}(\mathbf{w})=\mathcal{N}(\mathbf{w};\mathbf{\mu}_{\mathbf{w}}^{0},\mathbf{\Sigma}_{\mathbf{w}}^{ 0})\) to the mean and standard deviation \(\big{(}\mathbf{\mu}_{\mathbf{\
u}}^{0},\mathbf{\sigma}_{\mathbf{\
u}}^{0})\) of each timestep \(t\) as follows:\[\begin{split}\mathbf{\mu}_{t}^{0}&=\mathbf{\Psi}_{t}^{T}\mathbf{\mu}_{ \mathbf{w}}^{0}\\ \sigma_{t}^{0}&=\text{sqrt}\Big{(}\mathbf{\Psi}_{t}^{T} \mathbf{\Sigma}_{\mathbf{w}}^{0}\mathbf{\Psi}_{t}\Big{)}\end{split} \tag{15}\]  
For each timestep \(t\), the new mean and standard deviation \((\mathbf{\mu}_{t}^{*},\mathbf{\sigma}_{t}^{*})\) are optimized under task constraints separately. Let \(c_{k,t}=C(\mathbf{\mu}_{t},\mathbf{\sigma}_{t})\) denote the function of the \(k^{th}\) task constraints related to \((\mathbf{\mu}_{t},\mathbf{\sigma}_{t})\), and \(F_{c_{k,t}}(\mathbf{\mu}_{t},\mathbf{\sigma}_{t})\) denote the cumulative distribution function (CDF) of the random variable defined by \(c_{k,t}\), then we can rewrite the optimization problem as a Lagrangian function:  
\[\begin{split} L\big{(}\mathbf{\mu}_{t},\mathbf{\sigma}_{t}\big{\{}\lambda _{k,t}\big{\}}\big{)}=&\text{D}_{\text{KL}}\Big{[}\mathcal{N}( \mathbf{q}_{t};\mathbf{\mu}_{t},\mathbf{\sigma}_{t})||\mathcal{N}\Big{(}\mathbf{q}_{t}^{0};\bm {\mu}_{t}^{0},\mathbf{\sigma}_{t}^{0}\Big{)}\Big{]}\\ &+\sum_{k,t}\lambda_{k,t}\Big{[}a_{k,t}-F_{c_{k,t}}(\mathbf{\mu}_{t},\mathbf{\sigma}_{t})\Big{]}\end{split} \tag{16}\]  
where \(\alpha_{k,t}\) is the confidence level to represent the probability satisfying task constraints \(P_{\mathbf{\mu}_{t},\mathbf{\sigma}_{t}}(H(c_{k,t}))\geqslant\alpha_{k,t}\), and \(H\) represents the one-side or two-side inequality constraints; the optimized mean and standard deviation \((\mathbf{\mu}_{t}^{*},\mathbf{\sigma}_{t}^{*})\) are derived by gradient ascent-descent methods as follows:  
\[\Big{(}\mathbf{\mu}_{t}^{*},\mathbf{\sigma}_{t}^{*},\Big{\{}\lambda_{k,t}^{*}\big{\}} \Big{)}=\max_{\lambda_{k,t}\geqslant 0}\min_{\mathbf{\mu}_{t},\mathbf{\sigma}_{t}}L \big{(}\mathbf{\mu}_{t},\mathbf{\sigma}_{t},\big{\{}\lambda_{k,t}\big{\}}\big{)} \tag{17}\]  
Subsequently, given \((\mathbf{\mu}_{t}^{*},\mathbf{\sigma}_{t}^{*})\), we implement another optimization step to obtain the optimized ProMPs distribution \((\mathbf{\mu}_{\mathbf{w}}^{*},\mathbf{\Sigma}_{\mathbf{w}}^{*})\) with:  
\[\begin{split}\mathbf{\mu}_{\mathbf{w}}^{*}&=\min_{\mathbf{\mu} _{\mathbf{w}}}\Big{(}\mathbf{\Psi}_{t}^{T}\mathbf{\mu}_{\mathbf{w}}-\mathbf{\mu}_{t}^{*}\Big{)}\\ \mathbf{\Sigma}_{\mathbf{w}}^{*}&=\min_{\mathbf{\Sigma}_{\mathbf{w}}} \Big{(}\mathbf{\Psi}_{t}^{T}\mathbf{\Sigma}_{\mathbf{w}}\mathbf{\Psi}_{t}-\mathbf{\sigma}_{t}^{*} \Big{)}\end{split} \tag{18}\]  
In Section 3.2, we formulate every task constraint by defining the function \(c_{k,t}=C(\mathbf{\mu}_{t},\mathbf{\sigma}_{t})\) related to \((\mathbf{\mu}_{t},\mathbf{\sigma}_{t})\), expressing the inequality constraint function \(H\) and approximating the cumulative distribution function \(F_{c_{k,t}}(\mathbf{\mu}_{t},\mathbf{\sigma}_{t})\); the optimized results are then numerically iterated based on automatic differentiation.