## 3 Behavior-Controllable Adversarial Motion Priors
### User Command-Integrated Reward Shaping  
In the learning setup of BCAMP, it is necessary to design a reward function that takes into account both the style reward \(r_{s}\) and the task reward \(r_{t}\), as shown below:  
\[r_{t}=w^{s}r_{s}+w^{s}r_{g} \tag{17}\]  
where \(w^{s}\) and \(w^{g}\) are the weights for the style reward and the task reward, respectively. The style reward comes from the motion prior discriminator learning process. The optimization objective of the motion prior discriminator is designed as  
\[ \begin{split}&\operatorname*{arg\,min}_{\phi}E_{(s(t),s(t+1))\sim D }\Big{[}\big{(}D_{\phi}(s(t),s(t+1))-1\big{)}^{2}\Big{]}\\ &+E_{(s(t),s(t+1))\sim\pi_{(t,\phi)}}\Big{[}\big{(}D_{\phi}(s(t ),s(t+1))+1\big{)}^{2}\Big{]}\\ &+w^{sp}E_{(s(t),s(t+1))\sim D}\Big{[}\big{\|}\
abla_{\phi}D_{ \phi}(s(t),s(t+1))\big{\|}^{2}\Big{]}\end{split}  \tag{18}\]  
where \(D_{\phi}(s(t),s(t+1))\) represents the determination of whether a state transition \((s(t),s(t+1))\) comes from a real sample in the reference trajectory datasets \(D\), or if it is an action generated by the policy network. Here, \(D_{\phi}\) represents the motion prior discriminator parameterized by \(\phi\); \(E_{(s(t),s(t+1))\sim D}\) represents the expectation calculation for a state transition \((s(t),s(t+1))\), with these state transition pairs drawn from the motion trajectory library \(D\); \(E_{(s(t),s(t+1))\sim\pi_{(t,\phi)}}\) is the expectation calculation for samples from the actions generated by the policy network; and \(\
abla_{\phi}\) is the gradient of the discriminator with respect to its parameters \(\phi\). The first two terms in the equation encourage the discriminator to distinguish between real actions and generated actions, while the last term is a gradient penalty to ensure that the discriminator does not produce excessively large gradients during training, which helps with training stability. The gradient penalty is \(w^{sp}\). The style reward is defined as  
\[r_{s}=\max\Big{[}0,1-0.25(D_{\phi}(s(t),s(t+1))-1)^{2}\Big{]} \tag{19}\]  
The objective of the policy network in this paper is to enable the quadruped robot to learn and execute actions based on the standardized reference trajectory datasets \(D_{s}^{i}\). We propose a hierarchical reward structure, where user commands (e.g., discrete task IDs) dynamically activate only the relevant task reward functions during training. For example, tasks such as walking, running, jumping, and bipedal standing are categorized into three types, distinguished by user commands \(U_{cmd}\in\{0,1,2\}\). When \(U_{cmd}=0\), the robot performs walking or running tasks; when \(U_{cmd}=1\), the robot performs jumping tasks; when \(U_{cmd}=2\), the robot performs bipedal standing tasks. Importantly, the corresponding rewards are calculated and accumulated only when these specific \(U_{cmd}\) are issued. This hierarchical reward structure reduces the trial-and-error process, as each task's rewards are independently optimized during their active phases. To achieve these three tasks, the following task reward functions are set for training:  
* When \(U_{cmd}=0\), tracking linear velocity reward \(r_{g}^{lin}\): \[r_{g}^{lin}=\exp\left(-\frac{\|V_{cmd}-V_{actual}\|^{2}}{\sigma}\right)\] (20) where \(V_{cmd}\) represents the commanded desired linear velocity, including the trunk linear velocity in the X and Y directions, \(V_{actual}\) is the actual velocity, \(\sigma\) represents the tracking accuracy, and \(\exp\) denotes the exponential function. * When \(U_{cmd}=0\), tracking angle velocity reward \(r_{g}^{ang}\): \[r_{g}^{ang}=\exp\left(-\frac{\|\theta_{cmd}-\theta_{actual}\|^{2}}{\sigma}\right)\] (21) where \(\theta_{cmd}\) represents the commanded desired angular velocity, and \(\theta_{actual}\) is the actual angular velocity. * When \(U_{cmd}=1\), tracking angle velocity reward \(r_{g}^{a}\): \[r_{g}^{a}=t_{air}\] (22) where \(t_{air}\) represents the duration for which all foot contact forces are zero, i.e., the time the robot is airborne.