# Reinforcement-Learning-Based Trajectory Learning in Frenet Frame for Autonomous Driving
## 4 Methods
### Trajectory Learning
#### 4.1.3 Reward Function  
We have designed the reward function to learn trajectories that avoid obstacles as far as possible while considering the vehicle heading constraints. In the absence of obstacles, the objective is for the agent vehicle to follow the center of the road. The rewards are calculated after following the trajectory entirely. The reward function consists of four components as follows:  
\[r=r_{\text{success}}+r_{\text{dev}}+r_{\text{cte}}+r_{\text{avoid}}\]  
\(r_{\text{success}}\) is a reward that indicates whether the agent vehicle is following a successful trajectory. If the agent successfully follows the trajectory to the end, the \(r_{\text{success}}\) is 400; otherwise, it is \(-500\). \(r_{\text{dev}}=-10\times\frac{1}{p_{\text{end}}-b_{\text{start}}}\sum_{h=0}^ {H}d_{h}\) is a reward that encourages the agent to stay close to the center of the road when there are no obstacles while avoiding veering off the road, where \(b_{end}\) and \(b_{start}\) are the longitudinal position of the agent at the end and start of each episode, respectively. \(H=\{h_{i}|i=1,2,\ldots,H_{end}\}\) is recorded every test time step \(t\) of the GRP, where \(H_{end}\) is when the executed trajectory ends. \(d_{h}\) is \(d_{agent}\) at time \(h\). \(r_{\text{cte}}=-10\times\mathbb{E}\left[\sum_{h=0}^{H}e_{h}(\tau_{\text{any}},p_{agent})_{2}\right]\) evaluates whether the trajectory can accommodate the heading constraint of the vehicle, where \(\tau_{\text{any}}\) is any trajectory and \(e_{h}(\tau_{\text{any}},p_{agent})\) is the cross-track error between \(\tau_{\text{any}}\) and agent position \(p_{agent}\) at time \(h\). A detailed explanation of cross-track error can be found in [39]. \(r_{\text{avoid}}=10\times\sum_{z=1}^{Z}\|p_{obs_{z}}-p_{agent}\|_{2}\) is an obstacle avoidance reward, with higher values rewarded for successfully avoiding obstacles at a farther distance when obstacles are present, where \(Z\) is the total number of obstacle vehicles, \(p_{obs_{z}}\) is the position of the \(z\)-th obstacle, and \(p_{agent}\) is the position of the agent. \(r_{\text{avoid}}\) is computed when the agent vehicle and the obstacle vehicle are at the same \(b\) in Frenet frame. Descriptions and value ranges of each reward term are summarized in the Table 2.  
\begin{table}
\begin{tabular}{c l l l l} \hline \hline  & \multicolumn{2}{c}{**State Space**} & \multicolumn{2}{c}{**Default**} & \multicolumn{2}{c}{**Range**} \\ \hline \multirow{4}{*}{Agent} & Heading error [rad] & \(\psi_{agent}\) & - & \(-\pi/2\sim\pi/2\) \\  & Lateral distance [m] & \(d_{agent}\) & - & \(0\sim\infty\) \\  & Distance with front vehicle [m] & \(b_{fv}\) & \(30\) & \(0\sim 30\) \\  & Velocity [km/h] & \(v_{agent}\) & - & \(0\sim\infty\) \\ \hline \multirow{4}{*}{Obstacle} & Heading error [rad] & \(\psi_{obs}\) & \(0\) & \(-\pi/2\sim\pi/2\) \\  & Difference with agent [m] & \(\Delta x_{obs}\) & \(5.48,-3.16\) & \(5.48\sim 30\) or \(-10\sim-3.16\) \\  & \(\Delta y_{obs}\) & \(5.48,-3.16\) & \(5.48\sim 30\) or \(-10\sim-3.16\) \\  & Lateral distance [m] & \(d_{obs}\) & \(40\) & \(0\sim 40\) \\  & Obstacle length in Frenet frame [m] & \(let_{obs}\) & \(0\) & \(0\sim\infty\) \\ \hline \multirow{2}{*}{Curvature} & Curvature at current position & \(\kappa_{\text{current}}\) & - & \(-\infty\sim\infty\) \\  & Curvature at 15m ahead & \(\kappa_{\text{ahead}}\) & - & \(-\infty\sim\infty\) \\ \hline \hline \end{tabular}
\end{table}
Table 1: State definition of RLTF.