# On the Impact of the Rules on Autonomous Drive Learning
## 4 Experiments  
Our goal was to experimentally assess the impact of the traffic rules on the optimized policies, in terms of overall efficiency and safety. To this aim, we defined 3 tuples, which are, respectively, the reward tuple \(R\), the efficiency tuple \(E\), and the collision tuple \(C\). The _reward_ tuple \(R\in\mathbb{R}^{n_{\text{car}}}\) is the tuple of individual rewards collected by the drivers during an episode, from \(t=0\) to \(t=T\), and is defined as:  
\[R=\left(\sum_{t=0}^{T}r_{1}^{(t)},\ldots,\sum_{t=0}^{T}r_{n_{\text{ cars}}}^{(t)}\right) \tag{15}\]  
The _efficiency_ tuple \(E\in\mathbb{R}^{n_{\text{car}}}\) is the tuple of sums of individual instant _linear speed_\(v_{x}\) for each driver during an episode, from \(t=0\) to \(t=T\), and is defined as:  
\[E=\left(\sum_{t=0}^{T}v_{x_{1}}^{(t)},\ldots,\sum_{t=0}^{T}v_{x_{n_{\text{ cars}}}}^{(t)}\right) \tag{16}\]  
The _collision_ tuple \(C\in\mathbb{N}^{n_{\text{car}}}\) is the tuple of individual collisions for each driver during an episode, from \(t=0\) to \(t=T\), and is defined as:  
\[C=\left(\sum_{t=0}^{T}\mathbbm{1}\left\{s_{1}^{(t-1)}=\text{alive}\wedge s_{1 }^{(t)}=\text{dead}\right\},\ldots,\sum_{t=0}^{T}\mathbbm{1}\left\{s_{n_{ \text{cars}}}^{(t-1)}=\text{alive}\wedge s_{n_{\text{cars}}}^{(t)}=\text{dead }\right\}\right) \tag{17}\]  
Each \(i\)th element \(c_{i}\) of this tuple is defined as the number of times in which the \(i\)th driver change its car status \(s_{i}\) from \(s_{i}=\text{alive}\) to \(s_{i}=\text{dead}\) between 2 consecutive time steps \(t-1\) and \(t\). We considered 2 different driving scenarios in which we aimed at finding optimal policy parameters:y \"no-rules\", in which traffic rules weighting factors are \(w_{1}=w_{1}=w_{3}=0\), such that drivers are not punished for breaking the rules, and \"rules\", in which traffic rules weighting factors are \(w_{1}=w_{2}=w_{3}=1\), such that drivers are punished for breaking the rules, and all the rules have the same relevance. Moreover, we considered 2 different collision scenarios:  
1. cars are kept with status \(s=\text{dead}\) in the road graph for \(t_{\text{dead}}\) time steps, and then are removed; and
2. cars are kept with status \(s=\text{dead}\) in the road graph for \(t_{\text{dead}}\) time steps, and then their status is changed back into \(s=\text{alive}\). The rationale for considering the second option is that the condition in which we remove collided cars after \(t_{\text{dead}}\) time steps may not be good enough for finding the optimal policy. This assumption could ease the task of driving for the non-collided cars, when the number of collided cars grows, and, on the other side, it might provide too few collisions to learn from. We simulated \(n_{\text{cars}}\) cars sharing the same driver policy parameters and moving in the simple road graph in Figure 3 for \(T\) time steps. This road graph has 1 main intersection at the center, and 4 three-way intersections. All road segments \(p\in S\) have the same length \(l(p)\) and same number of lanes \(w(p)\). We used the model parameters shown in Table 1 and performed the simulations using Flow [36], a microscopic discrete-time continuous-space road traffic simulator that allows implementing our scenarios. We repeated \(n_{\text{trials}}\) experiments in which we performed \(n_{\text{train}}\) training iterations in order to optimize the initial random policy parameters \(\theta_{\text{no-rules}}\) and \(\theta_{\text{rules}}\). We collected the values, across the \(n_{\text{trials}}\) repetitions, of \(R\), \(E\), and \(C\) during the training. We employed Proximal Policy Optimization (PPO) [37] as the RL policy optimization algorithm: PPO is a state-of-the-art actor-critic algorithm that is highly effective, while being almost parameters-free.