## 4 Design of Motion Profile Using Reinforcement Learning  
In the previous sections a control strategy was formed with which the safe motion of the autonomous vehicle in an intersection can be guaranteed. Nevertheless, in the design problem of the vehicle motion there are economy aspects that should be considered during the design process; i.e., it is recommended to minimize the control energy of the vehicles. In this section the consideration of the economy performance in the design process is proposed.  
The goal of the design process is to find an agent which is able to generate the control signal \(a_{L}(k)\). The agent is trained through a reinforcement learning process. The structure of the design process is illustrated in Figure 4. The model for the learning process contains the optimization task (13) together with the result of the robust control design (16). The model guarantees the avoidance of the collision in the intersection for every \(a_{L}(k)\) signal. Thus, during the training process of the agent in every episode the safety performances are guaranteed, and similarly, the economy performance is improved. The output of the motion model is reward \(r(k)\), which is composed by \(a(k)\) and \(v_{1}(k)\) as follows:  
\[r(k)=-Q_{1}a_{i}^{2}(k)+Q_{2}(s_{1}(k)-s_{1}(0)), \tag{17}\]  
where \(Q_{1}\) and \(Q_{2}\) positive values are design parameters, which scale the importance of each term in \(r(k)\). The reward contains the control inputs in the vehicles \(a(k)\), which represent the economy performance of the vehicles. If the reward contains only \(a(k)\), it can result in unacceptably slow motion for the autonomous vehicle, because \(a(k)=0\) is the best choice for the maximization of the reward. Thus, in the reward the traveled distance of the autonomous vehicle \((s_{1}(k)-s_{1}(0))\) is also incorporated, where \(s_{1}(0)\) is the initial position of the autonomous vehicle. The observation for the agent includes the position of the autonomous vehicles \(s_{1}(k)\) and of the surrounding vehicles \(s_{j}(k)\)\(j\in n_{s}\), and their velocities \(v_{1}(k),v_{j}(k)\)\(j\in n_{s}\).  
The goal of the reinforcement learning process is to maximize reward (17) during episodes. In this paper the training process through the deep deterministic policy gradient (DDPG) is carried out, which is a model-free, online, off-policy reinforcement learning method [50]. A DDPG agent is an actor-critic reinforcement learning agent that computes an optimal policy that maximizes the long-term reward. In this method, actor and critic approximators are used. Both approximators use the observations, which are represented by \(S\). The purpose of the actor approximator \(\mu(S)\) is to find the action \(A\) with \(a_{L}(k)\), which maximizes the long-term future reward. The role of the critic \(Q(S,A)\) is to find the expected value of the long-term future reward for the task.  
The result of the training process is an agent whose output is \(a_{L}(k)\). In the control process of the autonomous vehicle, the agent works together with the control strategy (13). As a result, the collision avoidance of the vehicles in the intersection is guaranteed, and similarly, the economic performance of the vehicles is improved.