# Reinforcement-Learning-Based Trajectory Learning in Frenet Frame for Autonomous Driving
## 4 Methods
### Reinforcement-Learning-Based Trajectory Learning in Frenet Frame
#### 4.3.3 Policy Update  
We model our policy, which maps a state to the next \(K\) actions, as a multi-layer perceptron (MLP) as shown in Figure 5. We utilize the cross-entropy method [40] to train a \(\theta\) that maximizes \(R(s,\tau_{\pi})\). The cross-entropy method trains the _elite-set_, which consists of actions with the top k% rewards. In RLTF, we will refer to _elite-set_ as \(\mathcal{T}_{top}\). To train in the Frenet frame, \(\mathcal{T}_{top}\) is transformed into the Frenet frame. The transformed \(\tau\) is sliced into \(|\tau|-K\) segments, where \(|\tau|\) is the number of points in \(\tau\). In the current step \(h\), \(a_{h}=\{p_{\tau}(h+1)-p_{\tau}(h),\dots,p_{\tau}(h+k)-p_{\tau}(h+k-1)\}\), where \(p_{\tau}(h+1)-p_{\tau}(h)\) represents the difference between the current step and next step. \(s_{h}\) and \(a_{h}\) are stored in \(D_{z}\), where \(z\) is the number of obstacle vehicles in the current episode.  
The parameter of policy \(\theta\) is updated to minimize \(\ell 2\) loss at intervals when \(|\mathcal{D}|\) exceeds the batch size, where \(|\mathcal{D}|\) is the number of transitions in the \(\mathcal{D}_{z}\) with the smallest number of transitions among all \(\mathcal{D}_{z}\). AdapOptimizer with learning rate \(1e^{-3}\) is employed as our  
Figure 4: Example of hard exploration. Hard exploration uses fully random exploration, where only the starting point and goal point are provided, and all intermediate points are randomly sampled from the GRPs. (**a**) GRP samples in the Frenet frame, (**b**) GRP samples in the Cartesian frame.  
Figure 5: Multi-layer perceptron (MLP) architecture. This figure illustrates the architecture of an MLP used for our algorithm. The input layer consists of 16 neurons, representing the state space of the vehicle. This is followed by two hidden layers with 64 neurons each, using ReLU activation functions. Two curvatures are concatenated to the hidden units before passing through a final linear layer with ReLU activation, resulting in 10 output neurons representing the next \(K=5\) actions.  
optimizer. To consider all scenarios involving obstacle vehicles, samples are taken from all \(\mathcal{D}_{z}\). The proposed algorithm is summarized in Algorithm 1.  
```
1:Initialize \(\theta\), replay buffer \(\mathcal{D}\) and step = 0
2:for every episode do
3: Initialize all vehicles pose.
4:ifrand \(>\) 0.5\(\frac{1}{1+\exp(\frac{-\exp}{10})}\)then
5: Sample \(N\) trajectories \(\tau_{grp}\) from GRPs.
6: Transform \(\tau_{grp}\) to Cartesian frame.
7:for each \(\tau_{grp}\)do
8: Execute \(n\)-th \(\tau_{grp}\) and record \(p_{agent}\) and \(s\).
9:\(\mathcal{T}\leftarrow\mathcal{T}\cup\tau\)
10:endfor
11:else
12: Generate a trajectory \(\tau_{\pi}\) from \(\pi_{\theta}\).
13: Transform \(\tau_{\pi}\) to Cartesian frame.
14: Execute \(\tau_{\pi}\) and record \(p_{agent}\) and \(s\).
15:\(\mathcal{T}\leftarrow\mathcal{T}\cup\tau\)
16:endif
17:for each \(\mathcal{T}\)do
18: Compute the reward \(R_{\tau}=r(s,\tau)\)
19:endfor
20:\(\mathcal{T}_{top}\leftarrow\) top \(k\)% recorded trajectories \(\mathcal{T}\)
21: Transform \(\tau_{top}\) to Frenet frame.
22:for each \(\tau\) in \(\mathcal{T}_{top}\)do
23:for h=0,..., \(|\tau|-K\)do
24:\(a_{h}\leftarrow\{p_{\tau}(h+1)-p_{\tau}(h),\ldots,p_{\tau}(h+k)-p_{\tau}(h+k-1)\}\)
25:\(\mathcal{D}_{i}\leftarrow\mathcal{D}_{i}\cup\{s_{h},a_{h},R_{\tau}\}\)
26:endfor
27:endfor
28:if episode mod interval == 0 and batch \(\leq|\mathcal{D}|\)then
29: step = step + 1
30:for each \(\mathcal{D}_{z}\) in \(\mathcal{D}\)do
31: sample (\(s_{b}\), \(a_{b}\)) from \(\mathcal{D}_{z}\)
32: Update \(\theta\) to minimize \(\ell\)2 loss.
33:endfor
34:endif
35:endfor
```  
**Algorithm 1** Reinforcement-learning-based trajectory learning in the Frenet frame