## 4 Trajectory Constraints and Reward Design
### Reward Design  
The system obtains rewards and punishments by interacting with the external environment. In this work, a positive value represents a reward and a negative value represents a punishment. First, each step of the whole flight must be rewarded and punished, so the horizontal difference between the remaining flying distance of the system and the distance to the destination should be calculated at each step, and consider the size of the control input. The optimization objective in the whole flight process is to minimize the horizontal error between the target position and the terminal position of the parafoil. Thus, the reward function is required to reflect the final landing result of the parafoil, and the terminal reward guides the parafoil to obtain higher landing accuracy. Then, the reward must consider the terrain avoidance. If the flight trajectory encounters a no-fly zone or exceeds the flyable area, a penalty will be given. First, the setting of reward value should guide the optimization direction of each step in the trajectory planning process. The real-time trajectory constraints are:  
\[D_{t}=\sqrt{\left(x_{t}-x_{f}\right)^{2}+\left(y_{t}-y_{f}\right)^{2}} \tag{15}\]  
\[d=|\frac{z_{t}}{v_{z}}\times v_{xy}-D_{t}| \tag{16}\]  
\[r_{u}=1-|a_{t}|-|a_{t}-{a_{t}}^{\prime}| \tag{17}\]  
\[r_{d}=\left\{ \begin{array}{l}2,if\ d<1\ and\ d^{\prime}<1\\ 0.1\times(d^{\prime}-d),otherwise\end{array} \right. \tag{18}\]  
\[r_{t}=r_{d}+0.5\times r_{u} \tag{19}\]\(D_{t}\) denotes the horizontal distance between the parafoil and the target point under \(S_{t}\). \(d\) and \(d^{\prime}\) are the difference between the remaining flying distance of the system and the distance to the destination under \(S_{t}\) and \(S_{t+1}\), respectively, \(r_{d}\) represents the constraint of real-time distance under \(S_{t}\), the range of \(0.1\times(d^{\prime}-d)\) is \(-1.5\) to \(1.5\), so the maximum reward value when \(d<1\) and \(d^{\prime}<1\) is set to \(2\). \(r_{u}\) represents the constraint of real-time control input at \(S_{t}\). Therefore, \(r_{t}\) represents the reward of real-time constraints. In order to prevent the landing accuracy from decreasing due to excessive constraint control input, \(r_{u}\) should be multiplied by the weight coefficient of \(0.5\) to appropriately reduce the weight. Under these real-time trajectory constraints, the optimization objective is to minimize the landing error in the whole flight process. In addition, the parafoil will obtain a reward value \(r_{f}\) in the terminal state, which is expressed as:  
\[D_{f}=\sqrt{\left(x_{t_{f}}-x_{f}\right)^{2}+\left(y_{t_{f}}-y_{f}\right)^{2}} \tag{20}\]  
\[r_{f}=\left\{ \begin{array}{l}\frac{K^{-D_{f}}}{K^{-D_{f}}},t=t_{f},D_{f} \leq 5\\ \frac{K^{-D_{f}}}{6},t=t_{f},D_{f}\leq 20\\ M,t=t_{f},D_{f}>50\end{array} \right. \tag{21}\]  
where \(\left[x_{t_{f}},y_{t_{f}}\right]\) represents the final landing position of the parafoil, \(D_{f}\) represents the landing error that is meant to be minimized, \(K\) is a constant greater than \(50\), and \(M\) is a negative constant. From (21), the formula shows that the value of \(r_{f}\) increases with the decrease of landing error. The range of \(r_{t}\) is \(-1.5\) to \(2.5\), so it is necessary to avoid diluting the terminal reward value \(r_{f}\) by the \(r_{t}\), the value of \(K\) should be selected in combination with reward decay rate \(\gamma\). When the landing error is different, the weight of the terminal reward value is different. When the landing error is less than \(5\) m, the weight of the terminal reward is the highest, and the parafoil pays more attention to the terminal reward value. When the landing error is large, the weight of the terminal reward value will be reduced, and the parafoil pays more attention to real-time reward values in order to obtain better planning strategies.