Some conclusions can be drawn from these three different situations:

I The decision-making horizon needs an appropriate length to anticipate short-term behaviors from other road users. However, the exponential nature of the state space discovery process (subsection 3.2.1) must be avoided.

i One way to do this is to sample the most probable next states given a current state and an action, focusing only on these discovered next states (as some POMDP solvers do). In theory, considering a fixed transition function, if a bigger number of transition were to be used, so as to only eliminate the states inside the decision horizon deemed not useful to the AV, the complexity of the resolution might be decreased. However, the heaviest step to solve this MDP is the evaluation of the reward function, not the iteration in itself, and since it is the reward the indicator of pertinence, this solution would not accomplish what is expected.

ii Another possible solution involves the inclusion of all other road users into the state and to evaluate a transition probability function capable to identify the most pertinent states, given the interaction between all users. This could break the exponential character of the problem if the actions of the AV were subjected to the same treatment as the states of the other users, i.e. not all actions need to be considered, only the ones that somewhat optimize one transition.

II In reality the decision horizon is dependent of the AV's velocity. If it increases too much the capacity to observe sufficiently ahead of time needs to increase, in the case that transition time ($\Delta t_{trans}$) is fixed. One can see the beginning of this problem in the situations commented above, but the real problematic becomes evident if one considers high speed limits. Planning should occur in small steps of trajectory, even more since the trajectory planned is based on piece-wise linear segments. Without a small enough transition time it is clear that the maneuver capability of the AV decreases rapidly.

III Risk evaluation is also necessary to stop the AV from starting a maneuver that cannot be completed in one policy, as it was done in figure 3.14. Defining the state space without a fixed number of transitions and using the end of a maneuver as a sign of policy termination would allow the AV to start and end in low-risk states.

Changing the direction of the first pedestrian from $\frac{\pi}{2}$ rad to $\frac{3\pi}{4}$ rad creates a different situation, but the results obtained are similar. Figure 3.17 shows the trajectory at $x_{veh} = 100$ m and 40 m and figure 3.18 at $x_{veh} = 40$ m for five transitions.

In comparison with figure 3.14, figure 3.17 produces similar behavior, where an additional transition allows the AV to avoid an accident. For $x_{veh} = 40$ and 5 transitions, it makes the AV choose to accelerate to avoid an accident, which in turn causes it to invade the sidewalk. With
<figure><img src="image_29.png" /><figcaption>Figure 3.17 – AV’s behavior for $x_{veh} = 100$, 40m and $\theta_{ped} = \frac{3\pi}{4}$ rad</figcaption></figure>

<figure><img src="image_30.png" /><figcaption>Figure 3.18 – Trajectory for $x_{veh}=40m$, $\theta_{ped}=\frac{3\pi}{4}$ rad for 5 transitions</figcaption></figure>

only four transitions the collision cost was not detected, only the sidewalk invasion cost is, which fix the future outcome to a collision with the environment’s border.

Another critical point in the implementation is the weights used for the reward function. All the considered weights and constants were defined heuristically for the specific use case considered. Even so, the best values obtained still do not reproduce an ideal trajectory, which can be seen in figure 3.14 between [5, 15] at the X axis. The small but perceptible change in AV’s direction, an oscillation in its direction, is not illegal but rather pointless.

## 3.4 Conclusion

This chapter analyzed an proposition of an MDP algorithm for the decision-making of an AV under normal situations. Before its definition, a discussion about which architecture is used in automated vehicles in general was addressed and the question of why a Markov-based algorithm was chosen instead of a deep-learning approach was also answered afterwards. After the MDP model was formally defined, from the necessary components to the existence and uniqueness
of its resulting optimal policy to subsequently define all the particularities of the model for this
instance.

The proposed algorithm was tested in a simulation environment, considering multiple configurations from other road users. It was observed that, for the heuristic weights chosen for the reward function, the AV behaved correctly in most situations, but some problems were flagrant. The exponential nature of the state set discovery did not allowed the use of an appropriate horizon prediction to correctly account the behavior of the other road users with the AV's velocity worsening this problem if the transition time is kept the same. No guarantee is done towards the finalization of a maneuvers before the end of the prediction horizon, which demands at least a risk assessment to avoid terminating in an unsafe state.
# Chapter 4

## Dilemma Deliberation

### Contents

<table><tr><td>4.1</td><td>Dilemma situations</td><td>78</td></tr><tr><td>4.2</td><td>The definition of harm</td><td>79</td></tr><tr><td>4.2.1</td><td>Δv for collision</td><td>82</td></tr><tr><td>4.2.2</td><td>Vulnerability constant</td><td>83</td></tr><tr><td>4.3</td><td>Ethical deliberation process</td><td>86</td></tr><tr><td>4.3.1</td><td>Ethical Valence Theory</td><td>86</td></tr><tr><td>4.3.2</td><td>Ethical Optimization</td><td>90</td></tr><tr><td>4.3.3</td><td>Value iteration for dilemma scenarios</td><td>94</td></tr><tr><td>4.4</td><td>Results and discussion</td><td>98</td></tr><tr><td>4.4.1</td><td>Using EVT as ethical deliberation</td><td>98</td></tr><tr><td>4.4.2</td><td>Using the Ethical Optimization profiles to deliberate</td><td>102</td></tr><tr><td>4.5</td><td>Conclusion</td><td>104</td></tr></table>

Up until now dilemma situations were not treated by the algorithm proposed in chapter 3, even if when the collision was inevitable, notably the case in graphic 3.14. Two methods to treat such situation will be exposed in this chapter: the ethical valence theory (EVT), proposed by Evans et al., 2020 and the ethical optimization, proposed in de Moura et al., 2020. Both methods use to guide its decision the estimated harm that may happen in an accident, while the EVT also uses the ethical valence with a deliberation procedure to choose the most appropriate action. This chapter ends with the explanation about how the value iteration, used to obtain the MDP policy, is modified to account for dilemma scenarios and a discussion about the results of both publications.
**4.1 Dilemma situations**

At each step of its trajectory, the AV should be able to tell whether a situation constitutes a dilemma worthy of moral consideration. This situational classification is necessary to determine if the AV must act according to ethical constraints or simply concrete objectives. Three rules can be defined to represent the AV’s responsibilities towards the other road users in its environment and its own passengers and they must be followed at all times. If one or more of these rules are violated across all possible actions, this indicates that the non-dilemma portion of the AV’s decision-making cannot cope with the consequences of all possible actions, and thus ethical deliberation is required for the AV to act in an acceptable way. Here harm is defined as the negative consequences suffered by a human after some type of collision with another road user.

AV duties towards other road users

* The lives of the passenger(s) must not be put in harm's way.

* The lives of the road users in the environment must not be put in harm's way.

* Traffic regulations must be followed.

Interactions between road users and the AV are covered by the two first rules. If the geometric representation of two road user (including the AV), during a transition, intercept each other, a collision is considered to have occurred. As it was done in the last chapter, a safety frontier around the AV can be defined to discourage the execution of actions which would remove the possibility of breaking without swerving to avoid an accident. But even so the AV could be exposed to dilemma situation, since the main uncertainty source comes from the behavior of the other road users. The general risk of a dilemma can also be minimized with a conservative behavior from the AV, however it can only go so far if it adopts a precautionary approach since it also must produce a performance capable to justify its existence.

There is a certain antagonism between the two first rules and the third one. One can ask himself if the former should take precedence over the other, so it is certainly desirable that the AV should also be sensitive to the interplay between ethical and legal behavior. In this sense, when there is a conflict in dilemma situations between harm to humans on one hand, and adherence to the traffic code on the other, the standard behavior adopted will be that the avoidance of the former should take precedence. As such, the MDP algorithm must be defined so as to express this priority in ways independent from the influence of the temporal discount rate. However, after the transgression, the AV must return to a ‘safe’ state, guaranteeing that another collision does not arise as a direct consequence of its original action choice. If this is the case, then for the AV’s decision-making algorithm all actions, at the first moment, end in collision, considering the proposed policy calculations that will be presented in the next sections.

Make the AV conform to traffic laws by design is not a subject that is widely addressed in
the literature. Generally, legal conformity presents challenges related to the interpretation of laws which can be vague, admit exceptions, or be internally incoherent; the resolution of all of which may demand some degree of common sense reasoning in order to be solved (Prakken, 2017). Additionally, with adherence to traffic laws comes the need to embed relatively abstract norms, used in laws to map concrete behavior, into an AV, and more broadly, into an autonomous system (Leenes and Lucivero, 2014). Some authors have already attempted to implement some portions of various traffic codes — related to circulation and behavior — into an autonomous vehicle, such as Rizaldi et al., 2017 (German legislation). Categorically, these attempts have been made using logic-based approaches to emulate constraints, representing only the procedural demands which usually compose a traffic code.

Given these considerations, when the duty obliging traffic code adherence is adequately defined, the entirety of the given traffic code does not need to be exhaustively implemented. To simplify the implementation of this evaluation algorithm, a set of logical rules will represent the procedural rules present in the specific scenario simulated. This set of rules should almost always allow the AV to cruise in a lawful manner. Exceptions to the code and the resolution of conflict between rules will not be covered here, the latter being treated as an ethical decision (even if ideally the procedures to solve conflicts between rules present in traffic codes should be used where possible).

For example, in a straight line domain, without pedestrian strips or semaphores, and with a solid double line the following logic rules can be used:

### Simplification of traffic rules

* Do not cross over into the opposite lane.

* Do not drive onto the sidewalk.

* Do not surpass the speed limit.

This simplified rules are the same as the ones used in the last chapter to calculate the cost connection to traffic code infringement, in subsection 3.2.3. In a real implementation, where the AV should target full automation (level 4 or more) in specific and generic environments, the actual set of rules will be extended well beyond these three rules and include also the conciliation process between these rules.

## 4.2 The definition of harm

The purpose of considering 'harm' in ethical deliberation is to measure the risk for AV passengers and other road users involved in a hypothetical collision. For decades, the main
measurement variable to estimate this property has been the difference of velocity between the two implicated road users at the moment of collision ($\Delta v$) (Evans, 1994; Jurewicz et al., 2016; Martin and Wu, 2018; Hussain et al., 2019).

Most of the research conducted within the domain of vehicle collisions uses historical accident data to analyze the influence of $\Delta v$ in collisions. To quantify injury, two metrics are popular: risk of fatality and the Abbreviated Injury Scale (AIS) (MacKenzie et al., 1985). The latter is preferred, since it is important to consider not only fatal collisions but those that can inflict severe damage (referred to as MAIS3+, which indicates that at least one injury in some region of the body is above AIS3, a scale going from 0 to 6). In the European Union, this metric is used as standard to measure road accidents (Weijermars et al., 2018).

All $\Delta v$ used as thresholds for severe injuries are indicated in table 4.1, along with their source (typically, an injury is considered as 'severe' if it indicates a MAIS3+ injury probability of 10%<fnref n="1" />). For the pedestrian case, the value was obtained from Kröyer, 2015, which considers severe injury as having an ISS (Injury Severity Score, defined as the squared sum of AIS for the three most severely injured body regions) larger than 9, which is stricter than MAIS3+. Lateral crashes are covered by near side (driver's side) and far side (passenger's side). For single vehicle collisions, the same $\Delta v$ defined for collisions between vehicles is used.

<table><caption>Table 4.1 – $\Delta v$ threshold used for fatality collisions</caption><thead><tr><th>Collision type</th><th>Contact</th><th>Δv value</th><th>Taken from</th></tr></thead><tbody><tr><td>Pedestrian collision</td><td>-</td><td>6.94 m/s</td><td>(Kröyer, 2015)</td></tr><tr><td rowspan="4">Vehicle collision</td><td>Frontal</td><td>7.78 m/s</td><td>(Jurewicz et al., 2016)</td></tr><tr><td>Rear</td><td>10.56 m/s</td><td>(Jurewicz et al., 2016)</td></tr><tr><td>Near side</td><td>5.56 m/s</td><td>(Jurewicz et al., 2016)</td></tr><tr><td>Far side</td><td>6.39 m/s</td><td>(Jurewicz et al., 2016)</td></tr></tbody></table>

The data presented in Jurewicz et al., 2016 was collected by the National Highway Traffic Safety Administration (NHTSA), published in Bahouth et al., 2014, considering injuries in the front seat, with a seat-belt, without rollover, with a passenger age ranging from 16 to 55, involving passenger vehicles and heavy vehicles. In Kröger, 2016 the data comes from the Swedish STRADA database, possessing a complete registry of pedestrian collision between 2004 and 2008.

This retrospective analysis of reported accidents has some drawbacks. According to Rosén et al., 2011, the data may be biased, since it is only collected across a small set of countries. Also, in the pedestrian case, age is an important feature (Kröyer, 2015), therefore the age distribution in the studied population plays a role which is unaccounted for in the resulting curve. Under-reporting of non-dilemma cases (Martin and Wu, 2018), estimation of collision velocities (Rosén et al., 2011), negligence of a vehicle's mass and geometry (Martin and Wu,

<footnote n="1">¹This is the threshold used by some automotive companies to define an unacceptable risk of collision.</footnote>
2018; Mizuno and Kajzer, 1999) and the use of different methodologies to evaluate AIS scores (Weijermars et al., 2018) also reduce the precision of such an approach.

Given that the previous method presents problems when applied to specific situations (despite it generalizing relatively well across a population), accounting for contextual information is necessary. The collision interaction between vehicles can be approximated by a damper-spring-mass system, where the initial velocity of each vehicle is projected onto the axis **n** (normal to contact plane between both vehicles) and **t** (tangential to contact plane). One example of such approach is Pickering et al., 2019, that uses a sprint bi-linear model to estimate the severity of a frontal collision might have if to avoid a pedestrian the AV needs to collide with a static object. The approach that will be considered here should be able to handle any specific situation, being capable to represent multiple collision configurations, from all frontal collisions types to variations of lateral impacts.

The collision can be divided in two phases: the first is the compression, where the deformation of each vehicle happens with the transfer of kinetic energy into the deformation process, followed by the restitution, when the vehicles plastic deformation settles, and part of the energy transferred to deformation returns as kinetic energy (Smit et al., 2019). Given the multiple possible collision configurations and the objective of having an approximated measure of the severity, an impact the point-mass collision approach will be adopted (Brach and Brach, 2011), with some simplifications, to model the interaction between road users and the AV in case of collision. The following simplification assumptions are adopted:

Collision mechanics simplificatory assumptions

1. No slidding during the collision (when the vehicle skid tangentially the other instead of deform in the normal direction)

2. Dissipation forces are negligible.

3. Crash interface between vehicles is a point.

4. Duration of the crash is infinitesimal.

With the collision mechanics defined, one can start to reason what "harm" should represent. This should refer to a single road user, during a collision. Thus, a plausible evaluation would be $h \propto \vec{J}$, $c_{vul}$, $h$ representing the road user's harm, $J$ for the impulse applied during collision and $c_{vul}$ for a measure of physical vulnerability, which will be explained in subsection 4.2.2. Considering the point-mass approach, the expression then becomes $h \propto \Delta\vec{p}$, $c_{vul}$, resulting in the equation 4.1, for a road user k:

$$^k h(s_t, a_t, s'_t) = ^k c_{vul} \cdot (\|\vec{v}_f - \vec{v}_t\|) \qquad (4.1)$$
The collision velocity is calculated using the conservation of linear momentum, that will be explained in subsection 4.2.1. The variable $v_f$, represents the collision velocity for both road users, $k$ and $l$. The velocities $^l\vec{v}_i$ and $^k\vec{v}_i$ are measured at the imminence of impact for $k$ and $l$.

### 4.2.1 $\Delta v$ for collision

To calculate the velocity after collision, $v_f$, used in equation 4.1, both road users ($k$ and $l$) are considered as punctual masses and all dissipation forces are neglected. From the original equations of the momentum, given by 4.2 and 4.3 for each road user, which can be brought together in one single equation, 4.4, which represents the conservation of linear momentum. The masses $m_k$ and $m_l$ correspond to the total mass of each road user (if it is a vehicle, then its mass plus the passengers' mass)

$$m_k^k \vec{v}_f - m_k^k \vec{v}_i = ^k J \qquad (4.2)$$

$$m_l^l \vec{v}_f - m_l^l \vec{v}_i = ^l J = -^k J \qquad (4.3)$$

$$(m_k^k \vec{v}_f - m_k^k \vec{v}_i) + (m_l^l \vec{v}_f - m_l^l \vec{v}_i) = 0 \qquad (4.4)$$

Usually to solve the system of equations presented by equations 4.2 and 4.3 it is necessary to "find" two more equations (six variables and only four equations, two for $\hat{x}$ component and two for the $\hat{y}$ component). Then comes the particular knowledge about the dynamics of collision restitution from the coefficient of restitution $e$ and the impulse ratio $\mu$, that is the relation between the normal and tangential impulse (no relation to the friction coefficient, that frequently is represented by the same letter) (Brach and Brach, 2011).

Given that it is essential for the harm to estimate the danger that a collision might pose and that such collision can be represented roughly by a spring-mass system (dissipation forces are negligible, so the presence of the damper is not necessary), it would be interesting to evaluate the final velocity present at equation 4.1 in the moment of maximal acceleration, which is the most important information to measure injury for passengers/pedestrians. This moment is the end of the compression phase, when the vehicles compose a single structure, right before the restitution takes place and separates them. Thus, this final velocity can be considered equal for both vehicles, giving the following equation:

$$(m_k^k \vec{v}_i + m_l^l \vec{v}_i) = (m_k + m_l) \cdot \vec{v}_f \qquad (4.5)$$
With equation 4.5 it is finally possible to calculate the final velocity for each road user $\vec{v}_f$
used in the harm equation 4.1, without any particular specificity about the vehicle body properties.
For collisions with pedestrians, it is considered that the pedestrian's $\vec{v}_f$ is equal to the AV's $\vec{v}_i$,
which implicitly means that the linear momentum of the pedestrian is negligible in comparison
with a vehicle, given the difference in mass, approximately by a factor of 10 and the difference of
velocity, that can also reach a factor of 10. Thus, the AV's velocity variation due to the collision
can be neglected, and its harm is equal to zero.

This simplification was adopted considering that the most common variables used to predict injury for pedestrians, according to the current research about collisions between pedestrians and vehicles, are the type of vehicle involved (Mizuno and Kajzer, 1999), specially due to the height of bonnet leading edge, parameter that can explain why SUVs are more dangerous to pedestrians than other vehicle types and the mass of the entire vehicle which usually is bigger than the average due to its size (Malczyk et al., 2012; Simms and Wood, 2006), along with the impact velocity. Therefore, this danger that is not captured by the velocity is the focus of another parameter, which accounts for the inherent vulnerability of a road user, together with other parameters. This subject comes into discussion with the definition of the vulnerability constant in equation 4.1.

For collisions with static objects, the same reasoning which was used with vehicle to
vehicle collisions is applied, but in this case $v_f$ is equal to zero. In the last chapter, the borders of
the simulation environment were defined as static walls (subsection 3.2.3); when the AV leaves the
bounds of simulation it considers that the final velocity in a collision (at the end of the respective
transition) is equal to zero to evaluate its harm in a direct collision with this wall.

**4.2.2 Vulnerability constant**

A plethora of studies exist in accidentology addressing collisions between vehicles and pedestrians or only between vehicles. The role of vulnerability constant $c_{vul}$ is to represent the inherent physical vulnerability of each road user during a collision using the information captured by these statistical studies in a way to complete the information given by the velocity variation described in subsection 4.2.1. Lets consider the following scenario: a frontal collision between two vehicles with a $\Delta v$ equal to 30km/h and another between a vehicle and a pedestrian with the same $\Delta v$. Lets first disregard the $c_{vul}$ and define the harm as only being the variation of velocity of a road user due to a collision.

According to equation 4.1 the harm for each vehicle (named *l* and *k*, with $^l\vec{v}_i = (20,0)$ and $^k\vec{v}_i = (-10,0)$ and equal mass) would be the norm of the difference between the final velocity, which is the mass weighted sum of their velocities (equation 4.8), and their initial velocity, at this moment not using $c_{vul}$. This value, expressed by equations 4.6 and 4.7 needs to be compared with the same situation happened with the vehicle *l* but this time with a pedestrian *p*.
$$ ^{k}h = (\|\vec{v}_{f} - \vec{v}_{i}\|) = \|(5,0) - (-10,0)\| = 15 \qquad (4.6) $$

$$ ^{l}h = (\|\vec{v}_{f} - \vec{v}_{i}\|) = \|(5,0) - (20,0)\| = 15 \qquad (4.7) $$

$$ \vec{v}_{f} = \frac{m_{k}^{k} \vec{v}_{i} + m_{l}^{l} \vec{v}_{i}}{(m_{k} + m_{l})} = (5,0) \qquad (4.8) $$

What can be observed from the comparison between the equations 4.6 and 4.7 with 4.9 and 4.10 is that for the pedestrian collision the harm would be 25% higher in comparison with the vehicle case.

$$ ^{p}h = (\|\vec{v}_{f} - \vec{v}_{i}\|) = (20,0) \qquad (4.9) $$

$$ ^{l}h = 0 \qquad (4.10) $$

$$ \vec{v}_{f} = \vec{v}_{i} \qquad (4.11) $$

One can argue that this conclusion can be interpreted as an evidence that in this case the collision with the pedestrian can be more severe. But two important details are not included in this procedure:

* Multiple essential information are left out of the calculation. For example, the type of vehicle is an important information, giving a hint about the mass and geometry of the vehicle; the configuration of collision also is important, since frontal collisions and lateral collision may present different risks for the passengers.

* The graduation between the harm's value and the probability of an MAIS3+ injury is not straightforward either. It cannot be said, for example, that there is a 25% higher change that the collision with the pedestrian result in such injury than in comparison with the vehicle collision.

The objective of the vulnerability constant $c_{vul}$ is to add these information about the road users and also scale the harm measure to be an accurate representation of the danger coming from the collision, considering all different information. Its determination should come from the $\Delta v$ curves available in the literature, that connects a certain value of $\Delta v$ with a probability of an injury MAIS3+. However, given the number of parameters the determination of such constant in rather complex. The following parameters should be considered to define the constant:
Parameters necessary to define the constant of vulnerability

- Type(s) of vehicle(s) involved (minis, SUVs, sedans, etc.).

- Configuration of collision (frontal, oblique, lateral, etc.).

- Homogeneous absolute velocity interval.

Both first and second parameters are self-explanatory. The third refers to the fact that $\Delta v$ is only a difference of velocities, therefore to have a $\Delta v = 30\,km/h$ is different when one of the vehicles have $v_1 = 100\,km/h$ and if it have $v_1 = 30\,km/h$. This parameter guarantees that accident data relative to fast corridors in cities does not pollute the set, pushing up the probability of injury for a certain $\Delta v$.

For each combination of the three parameters presented above, one curve $\Delta v$ versus probability of MAIS3+ injury would be necessary. If it existed, then, assuming some specific curve form for the vulnerability constant, it could be fitted to give the harm the same proportional increase according to the risk of injury. The necessary offer of graphs do not exist today, all the open databases concerning traffic accidents usually does not comport the velocity before collision nor the type of vehicle involved (the french database does not have neither, while the NHTSA has only the first, both also without the injury measurement based on the AIS scale).

The choice made to generalize all information about the correlation between the physiological impact of an accident into humans, together with the parameters connected with the compatibility between vehicles (in V2V collisions) and collision configuration, using the statistical data of accidents, tries to simplify the procedure of estimating the harm considering all these variables. All different modes of collision could be added into the collision mechanics, but would necessitate a 3D dynamic model of the vehicle to calculate all the forces involved, including a precise model of the interaction between road and tires. And to account the inherent vulnerability of the user would be more difficult, maybe with some relation between acceptable levels of short pulses of acceleration for humans. As said before, an accidentology approach also has drawbacks, but all the data, in the format proposed, does not need to be validated since it comes from real accidents.

As the determination of the initial velocity of an vehicle right before a collision and the aforementioned curves themselves are subject of entire projects and because the necessary databases are not open to public, the vulnerability constant was simplified in the implementation presented at the end of this chapter. As such, the $c_{vul}$ was considered equal to one, given that the velocities used in simulation where no higher than $10\,m/s$, originating $\Delta v$s that do not signal a critical difference of risk in the collisions studied later on.
**4.3 Ethical deliberation process**

Given an hypothetical dilemma situation, the AV must be able to act in an ethical manner, using explainable methods and basing its decision in criteria that is clear and well known. This section offers two different methods to achieve such decision procedure: the ethical valence theory and the ethical optimization. This section ends with an explanation concerning the necessary adaptation into the value iteration to allow the AV to switch from the reward-value optimization to one of these ethical deliberation processes.

**4.3.1 Ethical Valence Theory**

The ethical valence theory was proposed in Evans et al., 2020 as a method to mitigate the claims that each road user has about the AV's behavior and also about their right of safety in a public space. It considers two variables to take a decision, the ethical valence, which tries to capture an hierarchy of protection based on a priori criteria, and the harm, already defined in the precedent section. Then an ethical deliberation procedure, based on these variables, define which road user should be safeguarded against a possible collision.

**Valence definition**

The purpose of a valence is to represent the degree of social acceptability that is attached to the claims of the road users in the vehicle's environment. In this sense, the claims of certain road users can be more or less 'acceptable' to satisfy via the vehicle's action selection. The valences, in so far as they are rooted in the phenomenal signature of individuals, then track various physical characteristics which are seen to carry social importance: height, age, gender, helmet-wearing-cyclist, or stroller-pushing-adult, all of which are detectable by the object classification algorithms of the AV. Importantly, the determination of the strength of these valences is accomplished through a type of ranking or hierarchisation, which associates a road user's claim with a certain class or category of valence, as shown by table 4.2. In this way, depending on the amount or detail of the valence features under consideration, there can be more or less valence categories.

In this example for instance, two features are used: age and type of road user. The classification was created considering recent studies which suggest that western societies prefer to spare the young and vulnerable (understood in terms of exposure to injury) in AV collisions (Awad et al., 2018). In the case of multiple people, vehicles or agglomeration of pedestrians, the entity that has the larger number of users with a high classification has the preference. Between an AV with passengers C and F and another with C and D, the latter is considered to have a higher valence.
<table><caption>Table 4.2 – Possible valence hierarchy</caption><thead><tr><th>Feature 1</th><th>Feature 2</th><th>Classification</th></tr></thead><tbody><tr><td>Young (0 – 18 years)</td><td>Pedestrian</td><td>A</td></tr><tr><td>Old (65+ years)</td><td>Pedestrian</td><td>B</td></tr><tr><td>Young</td><td>Vehicle passenger</td><td>C</td></tr><tr><td>Old</td><td>Vehicle passenger</td><td>D</td></tr><tr><td rowspan="2">Adult (18 – 65 years)</td><td>Pedestrian</td><td>E</td></tr><tr><td>Vehicle passenger</td><td>F</td></tr></tbody></table>

In cases where the chosen valence features are minimal or simple (such as in the example above) the likelihood that multiple road users will have the same valence, but differing claims, increases. In this sense, there may be certain situations wherein the harm measurement becomes the decisive factor in action selection. In these cases, the vehicle satisfies the strongest claim in its environment, protecting the person whose welfare is most severely impacted, due either to a dangerous context (high velocity difference) or to an inherent vulnerability (detected by the vulnerability constant). This simple maximization of welfare, however, is complicated by the operational moral profile, which specifies the claim mitigation process between those passengers inside the car, and those road users outside of it. To this end, two possible moral profiles can be seen in table 4.3. Risk is considered severe if $\Delta v$ surpasses the limits defined in table 4.1.

<table><caption>Table 4.3 – Possible moral profiles for an AV</caption><thead><tr><th>Moral profiles</th><th>Definition</th></tr></thead><tbody><tr><td>Risk averse altruism</td><td>Protects the road user with the highest valence as long as AV passenger's risk is not severe.</td></tr><tr><td>Threshold egoism</td><td>Protects AV passengers as long as risk for other road users with higher valence than the AV is not severe.</td></tr></tbody></table>

None of these profiles perfectly resemble any traditional moral theory, or if anything, resemble various positions along the spectrum of egoistic rationality (Parfit, 1984). This is intentional, as these profiles are designed to capture various degrees of compromise between the claims and valences of the AV's passengers and those of the other agents within the AV's environment. These profiles often reinforce the idea that a certain degree of morally admirable partiality is possible, or perhaps even necessary in AV behavior, in order to best align with user expectations, or to garner user trust (Gerdes and Thornton, 2016; Keeling et al., 2019). The profiles listed in table 4.3 are likewise non-exhaustive and represent somewhat factually opaque renditions of the profile types the Ethical Valence Theory can accommodate. In these versions, the role of the harm calculation is important, as it is the principal factor which informs the various consequences of the AV's actions, due to trade-offs between the passenger(s) claims and those of
the other agents in the vehicle's environment.

## Moral deliberation

Once informed by the valences and harms, the AV can deliberate on an action, a step which is crucially guided by the operational moral profile. Each moral profile indicates a unique form of deliberation, as shown in table 4.4. It is perhaps worth restating that the moral profiles—and for that matter, ethical deliberation itself—is only used if the AV detect a possible dilemma situation. Otherwise, concrete, goal-driven planning is operative, using standard decision-making criteria.

<table><caption>Table 4.4 – Optimization procedure based on the moral profile chosen</caption><thead><tr><td>Moral profiles</td><td>Deliberation</td></tr></thead><tbody><tr><td>Risk averse altruism</td><td>Minimize the expected harm of the road user with the highest valence until the AV's collision becomes severe</td></tr><tr><td>Threshold egoism</td><td>Minimize the expected harm of the AV until the risk to a road user with a higher valence becomes severe</td></tr></tbody></table>

Each profile requires a different implementation. Using the risk-averse altruism case as an example, to deliberate, the AV's state (represented with the same variables as in subsection 3.2.1), environment state ($e$, which contains the position and velocity of all agents in the environment), highest road user valence ($\eta$) and maximum $\Delta v$, are the input. The action that should be executed ($a_\eta$), is the output. As a first step, all harm measurements for possible actions and the proceeding states (represented by the state space $S'$, composed by the states reached after one single transition) need to be calculated. Here the decisional horizon is equal to one transition, since the accident will follow immediately afterwards. This is done first by solving equations 4.5 and 4.1. Only one road user is implicated with the AV in a collision. The representation of other road users follow the same procedure used in chapter 3: they have a static behavior and their current velocity is used to predict their movements in the future, with no uncertainty involved.

<caption>**Algorithm 2:** Calculation of all possible harms</caption>

1 **for all** $a_i \in A$ **do**
2 &nbsp;&nbsp;&nbsp;&nbsp;2 **for all** $s'_i \in S'$ **do**
3 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3 $\vec{v}_f \leftarrow$ **calculate final velocities** (equation 4.5)
4 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4 $^k h(s_i, s'_i, a_t) \leftarrow$ **calculate harm for all road users** (including AV, equation 4.1)
5 &nbsp;&nbsp;&nbsp;&nbsp;end
6 end
If all possible outcomes have a velocity difference which is larger than $\Delta v$ (the road user's initial velocity minus the AV's velocity), then the collision is severe and the safety of the AV's passenger is prioritized. In the considered profile, the chosen action minimizes the expected harm for the AV. It should be pointed out that $\Delta v$ changes according to collision type (as can be seen in table 4.1). The transition probability is used to calculate the expected harm ($h_{exp}(s_i, a_j)$, equation 4.12), which represents a mean harm value for a road user k, given that for one state $s_i$ and action $a_j$ different states $s'_i$ can be reached, and therefore different collisions can happen. The position of all road users and the observation of the AV's state is considered to be perfect (no uncertainty in these measures).

$$ ^k h_{exp}(s_i, a_j) = \sum_{s'_i \in S'} p(s'_i|s_i, a_j)^k h(s_i, s'_i, a_t) \quad (4.12) $$

If the set of admissible actions according to $\Delta v$, $A_\eta$, is not empty, the chosen action minimizes the road user's expected harm with the highest valence for the actions $\in A_\eta$. Otherwise the action that minimize the harm for the road user with the highest valence is chosen; if multiple minimal actions still exist, then the one that minimizes the AV's expected harm is chosen. This process is shown by algorithm 3.

<caption>**Algorithm 3: Action selection**</caption>

1. $A_\eta \leftarrow \textbf{all actions in A that } (\|^{\text{RU}}\vec{v}_i - \|^{\text{AV}}\vec{v}_i\| \le \Delta v)$
2. **if** $A_\eta = \emptyset$ **then**
3. $a_\eta = \text{argmin}_{a \in A} h_{exp}(s_i, a_j)$
4. **else**
5. $a_c \leftarrow \text{argmin}_{a \in A_\eta} h_{exp}(s_i, a_j)$
6. **if** $\textbf{Multiple } a_c \textbf{ exists then}$
7. $a_\eta = \text{argmin}_{a_c} h_{exp}(s_i, a_c)$
8. **else**
9. $a_\eta = a_c$
10. **end**
11. **end**

Passing from the AV's harm minimization to the road user's harm minimization may appear to be an extreme position in comparison with other alternatives, such as the possible minimization of both quantities. An infinite number of compromises can be imagined between the AV and road users, however in our examples here both moral profiles oppose each other to maximize the safety of only one road user. For the threshold egoism profile, only the action deliberation process shown by algorithm 3 would change.
### 4.3.2 Ethical Optimization

The original idea behind the optimization procedures discussed in this section is the adaptation of an ethical reasoning into a mathematical procedure given a specific context. Currently and in the close future the existence of some robot capable of capturing all the nuances of human reasoning and behavior, while being able to reason at all levels, from symbolic to practical scenarios, is highly improbable. So the legwork from theory to the application of an ethically-driven form of decision-making must be partly done by the design of the decision-making model itself.

Three ethical principles were chosen to be applied to the collision dilemma situation. The first discussed will be the Rawlsian contractarianism, based on the theory of justice, proposed by Rawls, 1971. Then, the more familiar theory of utilitarianism will be considered, subsequently followed by egalitarianism, an adaptation of the utilitarian idea but also evaluating the scale of disparity between users. The discussion is concentrated on consequentialism methods, with Rawlsian contractarianism and egalitarianism serving as starting points for reflection about the drawbacks of utilitarianism.

#### Contractarian optimization

According to Rawls, 1971, justice is an inviolable right of humans, in which the only compromise acceptable in the case of an unequal distribution of rights guaranteed upon people by justice would be the closest possible to equality. The concept of fairness intervenes when two or more persons are interacting, and fairness should determine which procedures and practices can be used in their competition or cooperation, or the conditions under which one individual should recognize another individual's claim without the feeling of being forced or taken advantaged of. These two properties are highly desirable in the context of an AV decision making that might need to make decision in ethical dilemma situations.

From Rawls, 1958, the two principles of justice proposed are: "... each person participating in a practice, or affected by it, has an equal right to the most extensive liberty compatible with a like liberty for all; ..., inequalities are arbitrary unless it is reasonable to expect that they will work out for everyone's advantage ...". More specifically, the expression "must work out for everyone's advantage" excludes the justification of disadvantages for the worst of given the advantages offered for the most well of.

Translating the generic context used here to the narrow case of justice as is applied to dilemma decision scenarios, one can say that every road user must be awarded the most extensive right to safety as is compatible with the safety of others, and that differences in safety across road users should benefit those road users who are most at risk. In the imminence of a collision, having the same amount of safety means that ideally the harm should be equally distributed for all involved road users (involved here refers to all road users in reach of the AV that might be
struck by it), if no other action is available that protects the most vulnerable road users.

Practically speaking, starting from the most equal state, when the standard deviation is minimal for an action a that is available to the AV in a dilemma situation (equation 4.13; $h_{exp}(a)$ represents the set $^i h_{exp}(a)$, $0 \le i \le n$, n being the number of road users in the scene) an hierarchy can be created, from the road user most protected to the least one. Then the largest expected harm, considering this classification, should be minimized, as long as the harm for all other road users involved does not increase (equation 4.14).

$$a_{int} = \underset{a \in A}{\operatorname{argmin}} \sigma_{h_{exp}}(a) \qquad (4.13)$$

$$\operatorname{Min}_{a} \{^i h_{exp}(a)\}, \text{ subject to } ^j h_{exp}(a) \le ^j h_{exp}(a_{int}) \qquad (4.14)$$

This optimization model is the one used in de Moura et al., 2020, which results are presented in the later subsection 4.4.2. But this formulation can still be improved to represent more faithfully the ideas of justice proposed by Rawls. The author says that justice is a primitive concept, that arises when morality is applied to the negotiation between the involved parties to define practices that would be acceptable for all parties. It is the sentiment that comes from the enforcement of these practices that configures fairness. If one party holds that some claim is not legitimate, then to them the decision if unfair. Each one can have a definition of what is legitimate and what is not.

If one considers the traffic code as the practice that deals with the opposing claims from each road user, which in turn would assume that what is just is defined by this code, one has an authentic picture of the relations between users in modern days. Walking back from this commitment would certainly create an unjust action (without even discussing the legal implications of such a decision), hence from this observation comes the first constraint: **during the deliberation for an action that will inevitably cause harm, the traffic code must continue to be followed.**

So fairness, in a dilemma scenario, would be characterized as the evaluation that a concerned road user would perform concerning the claim which underpinned an AV's action. Let's say that, in such a situation the right for safety is the main parameter to be considered when an action is selected, since intuitively the claims connected to it are, at least in their majority and assuming an acceptable level of pertinence, legitimate. But it would not be legitimate if to respect some claim from one road user the AV should risk the safety of other users not involved in the traffic situation, which means that **road users not involved in the interactions that generated the dilemma situation cannot be put in harms-way.**

For pedestrians this constraint can be described roughly as "the AV must not invade the
sidewalk as a tactic to decrease the harm in a collision" considering all possible options available, since in doing so it inevitably will put other lives in danger that are not necessarily concerned by the interaction on the road. The general idea expressed by this limitation was also proposed by the guideline 9 in Luetge, 2017. The interpretation of what is legitimate used here is minimalist and tries to capture a common sense argument since this legitimacy determination has an important personal component involved.

One last modification should be done in the previously proposed deliberation procedure. Given the emphasis that inequalities should only be accepted if the worst off is favored, a search for the action that minimizes the greatest expected harm while maintaining the same level of harm for everybody else was proposed. But such a minimization can be pushed even further: **the biggest expected harm should be minimized as long as the expected harm for the other road users do not surpass a danger threshold.** However, such threshold can only be defined comparing the harm value before the accident and its consequences; such a comparison is not available today.

Lets assume that this threshold can somehow be defined, for each type of road user. Then, three scenarios might appear, for all considered actions, and are displayed below. The set of actions considered below is formed by the original available actions minus the ones that violate the traffic code and that brings danger to road users not concerned with the road environment activity.

New contractarian deliberation

1. All road users have an expected harm below the threshold: the AV can execute the action that distributes the harm in the best way possible.

2. Some road users' expected harm surpass the limit and some do not: the action to be selected must

(a) Reduce the expected harm of the biggest number of road users possible to below the threshold.

(b) The action should not send road users above the limit, i.e. the one that are below the threshold given the most equal action (determined the same way as before, the action that has the smallest standard deviation) should remain below the limit (but its expected harm can be increased).

(c) Accumulate the biggest reduction of expected harm for the road users above the danger limit.

3. All road users' expected harm surpass the limit: same behavior from the first case.

The proposed deliberation adheres better to the notion of justice expressed by Rawls than the first proposition, given the road environment context. But some barriers still exist for its
implementation, the most notable being the definition of this danger threshold, possibly related to the MAIS3+ injury probability level. Somehow, this limit helps to define which claim has preference in the mitigation process, together with the value of the expected harm itself. One could suggest an implementation without this a priori threshold, calculating the median of all expected harms and setting it to be the limit; in such case every situation would be treated by the second item of the new proposition.

## Utilitarian optimization

This approach is rooted in the Greatest Happiness Principle, which states that: "actions are right in proportion as they tend to promote happiness, wrong as they tend to produce the reverse of happiness. By happiness is intended pleasure, and the absence of pain; by unhappiness, pain, and the privation of pleasure" Mill, 1859. Happiness, in this definition, refers to the happiness of all people, not only for the individual that is deliberating about an action. Differently from the previous theory discussed, which aimed at defining a just framework to arbitrate conflicts and generate agreeable compromises, utilitarianism has its origins in trying to explain the motivation of individuals to take actions and as a base to legal and social reform at the time, using this theory to define what is morally good or right (Driver, 2014); which, according to the theory, is fundamentally connected to the pursuit of pleasure (Brink, 2018).

This idea of the greatest good for everyone embodies the definition of utility, every decision should be taken according to the maximization of utility, for the common good. Using the second formulation of the stated principle, one can minimize the total amount of expected harm to find the "least wrongful action", as equation 4.15 shows, the definition of the deliberation rule for this criteria is rather straightforward, the minimization of the expected harm.

$$a_{eth} = \argmin_{a} \sum_{i=0}^{n} h_{exp}(a) \qquad (4.15)$$

## Egalitarian optimization

One of the biggest drawbacks of the utilitarian maximization approach is that everything rests on general maximization, without any concern towards how the utility is shared among the participants. This poses a real problem for the use case discusses here, because the total expected harm can be minimized for some action while condemning some participants to severe injuries instead of allowing a higher total expected harm for a more safe action for everyone. It is also clear that there is a need to consider the distribution of harm so as to avoid extreme situations like the one used previously as an example.
An approach based on egalitarianism can be used to solve the harm distribution problem. Equality discussion often refers to income distribution, which is an important subject for society, but lets consider that the claim for safety due to each of the road users is more urgent in its strength, demanding a stricter representation of equality in dangerous situations. Thus, in a comparison between the two use cases, the latter should have less "tolerance" with possible inequalities.

One re-interpretation of the utility principle adds how well the utility is distributed as another parameter of the goodness of an action, besides the total amount of good produced by it. Such a view of distribution and total quantity as equal measures of goodness is called communal egalitarianism by Broome, 2017. One clear conclusion of this approach is that the general good can be increased without the need to increase the total amount of good. Such a constraint is translated into the harm optimization process by the minimization of the cost function composed by the total expected harm and the variance of such expected harm for all involved road users, as equation 4.16 shows. The index *i* refers to each individual possibly involved in the accident.

$$a_{eth} = \arg\min_{a} \sum_{i=0}^{n} \text{var}_{h_{exp}}(i, a) \cdot i h_{exp}(a) \quad (4.16)$$

The variance is used as weight for each expected harm so as to modulate the proportionality of the amount of harm expected for a specific road user and its comparison with the quantity of harm estimated for the other. Another important detail of the proposed criteria is the use of variance and not standard deviation, with amplifies the importance of inequalities into the distribution of harm.

### 4.3.3 Value iteration for dilemma scenarios

The value iteration procedure explained in subsection 3.3.1 is a typical implementation of a method to solve an MDP. But, after all the discussions about how the AV should account for collision harm and the ethical consequences an action may present, it is clear that such implementation does not support a way to use both methods to calculate the resulting policy. Therefore, some changes in the value maximization need to be made, so as to, when the operation done by equation 3.27 in algorithm 1 is executed, the action corresponding to the dilemma state be the one chosen by one of the deliberation methods discussed previously, not by the reward calculation proposed in chapter 3.

Some states may have actions that cause collisions and actions that do not. In this case those that cause accidents are discarded during the iteration procedure, since it should be the main prerogative of the AV the avoidance of any collision every time that it is possible. Only the safe actions have their value function calculated and consequently, their action considered as the
optimal solution for a given state. If a dilemma situation arises, then or the ethical value theory is
used to deliberate on an action, in which case the valence of all the road users involved is needed
as an input of the algorithm; or the ethical optimization might be used, to choose an action that
optimized the expected harm according to the three optimization measures defined in subsection
4.3.2. Algorithm 4 shows the resulting modified value iteration to be used to solve the MDP.

During the value iteration, the switch between a normal evaluation and a dilemma
evaluation happens if the set of actions for a state that do not result in accident, $A_{\text{norm}}^{s_i}$, is empty
(line 5). This set is calculated during the estimation of the reward function; it starts having the
same content as A but have actions removed if they end up in accident. If all actions ends up in an
accident, which is the same as saying that $A_{\text{norm}}$ is empty, then the ethical deliberation method
chosen must deliberate on an action (line 6) and the value function correspondent to the current
state $s_i$ and the chosen action $a_{\text{eth}}$ will receive the collision cost. If there is some action that do
not cause a collision, then the value $V^{t+1}(s_i, a_j)$ is calculated normally, only using the actions
inside the set of admissible actions $A_{\text{norm}}$ (line 11).

This abrupt change in the value iteration estimation might raise questions about the
assumptions made to guarantee the existence and convergence of the optimal calculation via value
iteration. Such assumptions are still valid (stationary and bounded rewards, subsection 3.1.2),
since the value adopted is a constant and establishes unequivocally the action to be executed for a
state. Intuitively speaking, the only problem that such approach could induce is a periodical switch
between both calculation methods, but the convergence theorem also remains valid (theorem 4).
**Algorithm 4:** Value iteration including dilemma situations

**Data:** Spaces S and A, functions R and P, constant $\gamma$, environment data e

**Result:** Policy $\pi^*(s)$, Optimal Value $V^t(s)$

1 $t = 0$

2 **do**

3 | **for every** $s_i \in S$ **do**
4 | | $A_{norm}^{s_i}$ determination
5 | **if** $A_{norm}^{s_i} = \emptyset$ **then**
6 | | $a_{eth} = \text{Ethical\_deliberation}(s_i, A, e, P)$
7 | | $V^{t+1}(s_i, a_{eth}) = c_{col}$
8 | | $A_{select}^{s_i} = \{a_{eth}\}$
9 | **else**
10 | | **for every** $a_j \in A_{norm}^{s_i}$ **do**
11 | | | $V^{t+1}(s_i, a_j) = [R(s_i, a_j, s', e) + \gamma \cdot \sum_{s'_j} P(s'_j|s_i, a)V^t(s'_j)]$
12 | | **end**
13 | | $A_{select}^{s_i} = A_{norm}^{s_i}$
14 | **end**
15 | $V^{t+1}(s_i) = \max_{a \in A_{select}^{s_i}} V^{t+1}(s_i, a_i)$
16 | $t = t + 1$
17 | **end**

18 **while** $MSE(V^{t+1} - V^t) > \varepsilon$

19 **for every** $s_i \in S$ **do**

20
| $\pi^*(s_i) = \underset{a \in A}{\operatorname{argmax}} V^t(s_i)$
21 **end**

One should also consider the future consequences of these actions that do not provoke an accident immediately. It is possible that the AV choose an action that currently does not cause an accident, but that in the future ends up in a dilemma situation. In such occasions this action should be discarded. More serious than that, when the only actions available for the AV are certain to provoke dilemma situations in the future the AV needs some capacity to compare both sets of actions from different time epochs. This is done by projecting the parameters (at $t+n$) used to deliberate ethically on which action must be chosen towards the present (at $t$). Figure 4.1 illustrates how the mechanism works.

Red states show that a collision occurred during the transition, and in such a case the AV breaks and stops. State $s_{2,2}$ is critical, since all its actions will cause an accident in all available transitions. When the deliberation is done for $s_{1,0}$, which is not critical *per se*, it realizes that $s_{2,2}$ is. Thus $s_{1,0}$ is considered to be critical and the collision information resulting from the chosen action determined by the ethical deliberation in $s_{2,2}$ is transferred to $s_{1,0}$ to be used in its ethical
<figure><img src="image_31.png" /><figcaption>Figure 4.1 – Projection of future collision into previous states</figcaption></figure>

deliberation. The deliberation at $s_{1,0}$ is done with the collision information (expected harm) for $a_1$ and $a_2$ at $t + \Delta t_{\text{trans}}$ and the information determined by $s_{2,2}$ at $t + 2\Delta t_{\text{trans}}$ for $a_3$. The same procedure is later done for $s_{0,0}$, with its information for $a_1$ being the expected harm for the action chosen in $s_{1,0}$.

## Comments on the effect of the number of people in the ethical deliberation

How the ethical deliberation methods account for the number of persons that participate in the collision while being in the same road user (AV's passengers or many pedestrians close together) is an important detail. In the EVT this is controlled by the moral profile, that uses the expected harm and the valence to choose an action. In the two examples of moral profiles given by table 4.4, the risk averse altruism considers only the road user with the highest valence and each valence inside the AV, with the maximal valence representing the road user and the others having a tiebreak function. The expected harm is calculated for the entire vehicle, no consideration is made concerning differences in passenger placements. This could be done calculating the harm for each of the passengers, but in this case using their placement as another parameter in the calculation of the vulnerability constant.

However, the same is not valid for the ethical optimization approach. In the simulation results that will be presented next, the vehicles (AV and the other vehicle) were assumed to be carrying only one passenger, but lets assume that there are more passengers inside. The correct would be to give each passenger its own harm and that the total harm for the AV in a collision would be the sum of all passengers' harms, which is compatible with the utilitarian philosophy. This could create a disequilibrium in the deliberation because five medium intensity harms could
beat one high harm for a pedestrian for example. Another option would be to keep using only one harm per road user, even if there is multiple persons inside; such hypothesis would allow a better account of the harms when the harm baseline is low, for example when the AV's velocity is low but in high velocities it could favor one individual instead of multiple. Thus, for the contractarian and egalitarian the most coherent choice is to consider one harm per human being in the environment but create the constraint that when the action that concerns multiple road users are evaluated all harms must be evaluated, instead of the one by one basis from the contractarian; the egalitarian would function the same way as the utilitarian, the sum of all expected harms, with the variance as weights.

## 4.4 Results and discussion

### 4.4.1 Using EVT as ethical deliberation

In this publication an example of application of the ethical valence theory was given, starting from a dilemma scenario, to compare the results that each of the deliberation methods (risk-averse altruism and threshold egoism) would produce. The situation is illustrated by figure 4.2, where the AV is represented by the red vehicle, the yellow being a vehicle with driver and the person in the middle of the road a pedestrian. Only three outcomes are possible, depending on the action that the AV takes: collision with the wall if it turns right, collision with the pedestrian if it continues with the same direction and frontal collision with the other vehicle if it turns left. The direction is considered the only variable since the acceleration for this case is set to be negative.

The transition probability used to calculate the expected harm is the same proposed in the last chapter, subsection 3.2.2. Only the changes in direction carry a probability of transition, with the structure 0.1-0.8-0.1 for the action that continues in the same direction and 0.9-0.1 for the two extremity actions. For the vehicle's position prediction the same model used previously was adopted here, a single-track kinematic non-holonomic model without slippage (figure 3.4), with $\Delta t_{trans} = 0.5s$.

To calculate the vulnerability constant, $c_{vul}$, the data available in Kröyer, 2015 and Jurewicz et al., 2016 are used in equation 4.18, with $Prob_{MAIS3+}(\Delta v)$ being the probability of MAIS3+ injury given a $\Delta v$, difference of initial velocities before the collision. Since the interval of velocities considered here is small, this curve was chosen to demonstrate how the idea of $c_{vul}$ should work.

$$c_{vul} = \frac{1}{1 - Prob_{MAIS3+}(\Delta v)} \qquad (4.18)$$
<figure><img src="image_32.png" /><figcaption>Figure 4.2 – Possible dilemma situation</figcaption></figure>

Figure 4.3 shows the collision simulation with all the road users’ initial states in a situation hereby defined as *situation 1*.

<figure><img src="image_33.png" /><figcaption>Figure 4.3 – Initial simulation setup (other road users’ configuration is represented by (x, y, θ, v), not in scale)</figcaption></figure>

Table 4.5 shows the preference order, given the valences for each road user in figure 4.4.

In situation 1, Δv are equal to 23.1 m/s for the AV-vehicle (frontal collision), 14.1 m/s for
<table><caption>Table 4.5 – Valence hierarchy</caption><thead><tr><td>Road User</td><td>Valences</td><td>Classification</td></tr></thead><tbody><tr><td>AV</td><td>C, F, F</td><td>3°</td></tr><tr><td>Vehicle</td><td>C, D</td><td>2°</td></tr><tr><td>Pedestrian</td><td>A</td><td>1°</td></tr></tbody></table>

the AV-pedestrian (pedestrian collision) and 14.2 m/s for AV-wall (frontal collision). Comparing these values with the limits established in table 4.1, one can conclude that all actions pose a serious risk for the AV and all other road users. Following the risk-averse altruism profile would entail choosing to run over the pedestrian, since the AV must be prioritized ($\Delta v$ is above the limit, therefore the AV’s harm is minimized, selecting the red cell in table 4.6; such a procedure was seen in algorithm 3). Table 4.6 shows the harm and expected harm calculated for the AV considering each possible collision.

<figure><img src="image_34.png" /><figcaption>Figure 4.4 – Collision simulation for situation 1</figcaption></figure>

If the AV was configured to have threshold egoism as its operational moral profile, the choice would be to collide with the wall, since the pedestrian’s and vehicle’s valences are higher, according to the table 4.5 (both $\Delta v$ are above the limit, thus the road users with valences higher than the AV have their expected harm minimized, resulting in the blue cell at table 4.7). Table 4.7 presents in its first column the nominal road user’s harm, while in the second and third columns the vehicle’s expected harm and the pedestrian’s expected harm, obtained using the transition probability. Since the wall is a static object, its harm and expected harm is zero (only human safety is considered; historical, cultural or affective value to a static object like a tree or a monument are disregarded).

Figure 4.5 shows *situation 2*, where the initial AV’s state is (10, 3.25, 0, 7.5, 0) and the position and velocity of the other vehicle also are changed. The new consequences of each action are displayed by figure 4.6, changing the collision with the other vehicle from frontal to lateral.
<table><caption>Table 4.6 – AV’s harm for each possible collision for situation 1</caption><thead><tr><th></th><th>AV's harm</th><th>AV's exp. harm</th></tr></thead><tbody><tr><th>Veh. col.</th><td>8.77</td><td>7.02</td></tr><tr><th>Ped. col.</th><td>0</td><td>2.46</td></tr><tr><th>Wall col.</th><td>15.80</td><td>12.64</td></tr></tbody></table>

<table><caption>Table 4.7 – Road users’s harm for each possible collision for situation 1</caption><thead><tr><th></th><th>Road user's h</th><th>Vehicle's h<sub>exp</sub></th><th>Pedestrian's h<sub>exp</sub></th></tr></thead><tbody><tr><th>Veh. col.</th><td>16.80</td><td>15.12</td><td>1.57</td></tr><tr><th>Ped. col.</th><td>15.71</td><td>1.68</td><td>12.57</td></tr><tr><th>Wall col.</th><td>0</td><td>0</td><td>1.57</td></tr></tbody></table>

There is a change in collision severity, given that the velocity differences $\Delta v$ would be 14.87 m/s, 5.63 m/s and 6.07 m/s for vehicle to vehicle, to pedestrian and to wall collision, meaning that the two last ones do not surpass the severe threshold.

<figure><img src="image_35.png" /><figcaption>Figure 4.5 – Second simulation setup (other road users’ configuration is represented by $(x, y, \theta, v)$, not in scale)</figcaption></figure>

Using risk-averse altruism as the operative moral profile results in the wall collision action being executed (the road user that has the highest valence has its expected harm minimized), resulting in the action represented by the red cell in table 4.8 (in here, since two action have the same expected harm the second highest valence had its exp. harm minimized, which is the other vehicle) and for the threshold egoism the chosen action would be collision with the pedestrian (In this case the AV’s expected harm would be minimized), resulting in the blue cell at table 4.9. Tables 4.8 and 4.9 are analogous to tables 4.6 and 4.7, respectively.
<figure><img src="image_36.png" /><figcaption>Figure 4.6 – Collision simulation for situation 2</figcaption></figure>

<table><caption>Table 4.8 – AV’s harm for each possible collision for situation 2</caption><thead><tr><th></th><th>AV's harm</th><th>AV's exp. harm</th></tr></thead><tbody><tr><th>Veh. col.</th><td>5.10</td><td>4.08</td></tr><tr><th>Ped. col.</th><td>0</td><td>1.12</td></tr><tr><th>Wall col.</th><td>6.07</td><td>4.86</td></tr></tbody></table>

<table><caption>Table 4.9 – Road users’s harm for each possible collision for situation 2</caption><thead><tr><th></th><th>Road user's h</th><th>Vehicle's h<sub>exp</sub></th><th>Pedestrian's h<sub>exp</sub></th></tr></thead><tbody><tr><th>Veh. col.</th><td>10.85</td><td>9.76</td><td>0.56</td></tr><tr><th>Ped. col.</th><td>5.63</td><td>1.08</td><td>4.51</td></tr><tr><th>Wall col.</th><td>0</td><td>0</td><td>0.56</td></tr></tbody></table>

### 4.4.2 Using the Ethical Optimization profiles to deliberate

Here the simulation presented in subsection 3.3.3 are slightly modified to produce a dilemma situation, forcing the AV to use the ethical optimization methods proposed previously. The ethical dilemma scenario is created moving the other vehicle abruptly in just one time step (also, the x coordinate of P1 and P2 are changed to 20 and 22.5, respectively). At this point the AV must deliberate about an action. The velocity difference between each road user ($\Delta v$) is not significant enough to be identified in the fatality probability versus $\Delta v$ graphic given by Hussain et al., 2019 and Richards, 2010, thus the constant $c_{vul}$ used for the collisions will be equal to one.
As it can be seen in figure 4.7, the result for all cases is a collision with the pedestrian. But each policy chooses a different action, which imposes different collision consequences. Take the contractarian policy, which has its results displayed in table 4.10: instead of decreasing the velocity right before the collision (actions 1 through 5), it chooses to maintain it ($v_{AV} = 4m/s$) to escape a collision with the other vehicle (action $a_5$). The AV ends up colliding with the first pedestrian (in the next calculated policy) with ($v_{AV} = 4m/s$).

<figure><img src="image_37.png" /><figcaption>Figure 4.7 – Trajectories for contractarian, utilitarian and egalitarian methods</figcaption></figure>

<table><caption>Table 4.10 – Expected harms and $\sigma_{h_{exp}}$ for contractarian policy</caption><thead><tr><th>Action (m/s<sup>2</sup>, ◦/s)</th><th>AV</th><th>P1</th><th>VEH</th><th>P2</th><th>σ<sub>h<sub>exp</sub></sub></th></tr></thead><tbody><tr><td>a<sub>0</sub> (-1, -40)</td><td>3.64</td><td>0</td><td>6.94</td><td>0</td><td>2.89</td></tr><tr><td>a<sub>1</sub> (-1, -20)</td><td>3.64</td><td>0</td><td>6.94</td><td>0</td><td>2.89</td></tr><tr><td>a<sub>2</sub> (-1, 0)</td><td>3.61</td><td>0</td><td>6.90</td><td>0</td><td>2.87</td></tr><tr><td>a<sub>3</sub> (-1, 20)</td><td>3.54</td><td>0</td><td>6.77</td><td>0</td><td>2.82</td></tr><tr><td>a<sub>4</sub> (-1, 40)</td><td>3.49</td><td>0</td><td>6.76</td><td>0</td><td>2.81</td></tr><tr><td>a<sub>5</sub> (0, -40)</td><td>0.383</td><td>4.38</td><td>0.730</td><td>4.00</td><td>1.83</td></tr></tbody></table>

For the utilitarian policy, (second plot of figure 4.7), action $a_0$ is chosen because it minimizes the total amount of harm, as can be seen in table 4.11. It decreases the velocity to $v_{AV} = 3.5m/s$ and then increases to $v_{AV} = 4m/s$ so as to escape the other vehicle. The final collision velocity, in the next calculated policy is $v_{AV} = 3.1m/s$.
<table><caption>Table 4.11 – Expected harms and $\Sigma h_{exp}$ for utilitarian policy</caption><thead><tr><th>Action (m/s<sup>2</sup>, ◦/s)</th><th>AV</th><th>P1</th><th>VEH</th><th>P2</th><th>Σh<sub>exp</sub></th></tr></thead><tbody><tr><td>a<sub>0</sub> (-1, -40)</td><td>0.364</td><td>3.90</td><td>0.694</td><td>0</td><td>4.96</td></tr><tr><td>a<sub>1</sub> (-1, -20)</td><td>3.28</td><td>0.433</td><td>6.25</td><td>0</td><td>9.96</td></tr><tr><td>a<sub>2</sub> (-1, 0)</td><td>3.63</td><td>0</td><td>6.93</td><td>0</td><td>10.57</td></tr><tr><td>a<sub>3</sub> (-1, 20)</td><td>3.58</td><td>0</td><td>6.83</td><td>0</td><td>10.40</td></tr><tr><td>a<sub>4</sub> (-1, 40)</td><td>3.55</td><td>0</td><td>6.81</td><td>0</td><td>10.36</td></tr></tbody></table>

The egalitarian policy, trying to achieve a compromise between total expected harm minimization and dispersion minimization, actually increases the velocity right after the ethical deliberation, and ends up colliding with the pedestrian with $v_{AV} = 4.5m/s$. Action $a_{10}$ predicts that both pedestrians will be struck by the AV. This policy, in this example, produces more negative consequences than the other two, but maybe with another formulation, one that calibrates more the importance of distributing correctly the risks of a collision without allowing the harm be concentrated in one user, while the other share more of the harm, but only slightly. This method can represent a compromise between a contractarian and a utilitarian policy in such formulation.

<table><caption>Table 4.12 – Expected harms and total cost for egalitarian policy</caption><thead><tr><th>Action (m/s<sup>2</sup>, ◦/s)</th><th>AV</th><th>P1</th><th>VEH</th><th>P2</th><th>Total</th></tr></thead><tbody><tr><td>a<sub>0</sub> (-1, -40)</td><td>3.77</td><td>3.34</td><td>7.20</td><td>0</td><td>95.06</td></tr><tr><td>a<sub>1</sub> (-1, -20)</td><td>3.76</td><td>3.05</td><td>7.18</td><td>0</td><td>98.07</td></tr><tr><td>a<sub>5</sub> (0, -40)</td><td>0.385</td><td>4.69</td><td>0.735</td><td>0</td><td>50.00</td></tr><tr><td>a<sub>6</sub> (0, -20)</td><td>3.46</td><td>2.73</td><td>6.61</td><td>0</td><td>77.50</td></tr><tr><td>a<sub>10</sub> (1, -40)</td><td>0.431</td><td>4.91</td><td>0.824</td><td>4.60</td><td>46.05</td></tr></tbody></table>

## 4.5 Conclusion

Two ethical deliberation methods were proposed in this chapter. A formal definition for an ethical dilemma situation in the automated vehicle context was given in the first section, to afterwards proceed to the definition of the harm in a collision, including its main parameters, the difference of velocity due to the collision and the constant of vulnerability. The harm is then used by the EVT or by one of the ethical optimization methods to deliberate on an action. From this point a small modification for the value iteration algorithm that solves the MDP model from last chapter is proposed, allowing the AV to switch between deliberation procedures when an ethical situation is detected.
The two deliberation methods were then tested in two different scenarios. The EVT was used in a hypothetical situation where the AV was to decide if it should collide with the vehicle in the other lane, the pedestrian right in from of with the wall in the right. With a higher AV velocity used in situation 1 the profiles mandated the protection of the AV for the risk-averse altruism, selecting the collision with the pedestrian; and the road user with the highest valence, the pedestrian, for the threshold egoist profile and therefore selecting a collision with the wall. In situation 2, when the AV's velocity was decreased to 7.5m/s the situation inverted concerning the collision with the pedestrian and the wall, both of which was not considered severe anymore.

All of the profiles for the ethical optimization were tested in the same scenario from chapter 3, using the MDP model with the modified value iteration. From the three each one chose the collision with the pedestrians, but with different end velocities, with the utilitarian profile having the lowest one. Given the end velocity from all three instances, around 4m/s this was the correct choice, not generating severe injuries for the pedestrian, due to the rather low velocity, and avoiding a frontal collision with the vehicle in the other lane, that had a speed of 10m/s.
# Chapter 5

# Road User Intent Prediction

## Contents

<table><tr><td>5.1</td><td>Broadening the scope</td><td>108</td></tr><tr><td>5.2</td><td>Intent estimation</td><td>109</td></tr><tr><td>5.2.1</td><td>Simulated Environment</td><td>109</td></tr><tr><td>5.2.2</td><td>Agent's decision model</td><td>112</td></tr><tr><td>5.2.3</td><td>Estimating other agents' intent</td><td>119</td></tr><tr><td>5.3</td><td>Incomplete game model</td><td>125</td></tr><tr><td>5.3.1</td><td>Nash equilibrium</td><td>125</td></tr><tr><td>5.3.2</td><td>Harsanyi's Bayesian game</td><td>126</td></tr><tr><td>5.3.3</td><td>Decision making procedure</td><td>129</td></tr><tr><td>5.4</td><td>Simulation results</td><td>133</td></tr><tr><td>5.4.1</td><td>Multi-agent simulation</td><td>134</td></tr><tr><td>5.5</td><td>Final remarks</td><td>138</td></tr></table>

In the previous chapters the proposed AV decision-making procedure considered the behavior of other road users as constant throughout the prediction, as if no interaction happened inside the environment. This kind of assumption can create a planning procedure that is not capable to capture all the variability present in each situation, ignoring an important, if not essential, source of uncertainty. This chapter focuses on correcting this issue using three algorithms: the first one to determine the behavior presented by other road users from pre-defined profiles, which are then used by the AV to estimate the most probable behaviors; and finally, the AV deliberates on which behavior profile it should adopt while at the same time trying to increase its resilience to errors in the prediction of the other road users' behaviors.
**5.1 Broadening the scope**

In an ideal world the AV would not encounter any dilemma situations. Given the abundant number of sensors one would assume to be embedded in the vehicle and scattered around cities and highways and fast, reliable and low-latency communication with other road users and the cloud combined with enough processing power to execute all necessary operations, a priori passengers can be safely driven anywhere, anytime. However, this is a naive assumption since the quantity of hardware in a vehicle will always be minimized by car manufacturers due to cost concerns, there will always be some imprecision in sensors measurements and also due to sheer number of possible behaviors that might be originated by the inevitable interaction between manual vehicles (or vehicles that do not have any type of communication capability, i.e., no V2X communication capability), automated ones and vulnerable road users, at least in urban areas. Considering the universe of difficulties that the AV can face, three categories of errors can be defined:

Possible errors in automated driving

* Material error: due to failure in sensors, processing unit or actuators of the vehicle.

• Design error: represent mostly errors in the software components of the vehicle.

* Interaction error: unexpected behavior and imprecise measurement from other road users can create situation that are not accounted for.

The first type of error refers to problems occasioned by the failure of hardware during
the automated driving task. This risk can be eliminated, and probably will be, with the adoption
of robustness standards and fail-safe strategies to take the vehicle to a safe configuration in
such occasions. Some inspiration can certainly be drawn from the aeronautical industry, that for
decades deals with this type of errors. The critical situation studied in (de Moura et al., 2020) and
discussed at subsection 4.4.2 was inspired by this type of error since during the simulation an
abrupt change in the position of the other vehicle is considered to be caused by a detection error
due to a sensor failure.

Design errors are a real problem in any software component, and even more in complex
systems. This category encompass the errors that originate from bugs and invalid assumptions
made during the design and implementation of the entire automated system for an AV. Such source
of problems can also be mitigated with the advancement of programming practices, verification
and validation procedures for the developed software and a robust homologation process, which,
as discussed in chapter 1, is not yet set in stone.

This chapter focuses on the third problem: the capacity to predict and correctly account
the behavior of other road users and their interaction with each other. First, the AV must be capable
to predict all possible behaviors of other road users, given the environment constraints that may
exist, which can be different for each road user, and the influence of interactions between road users in the selection of these behaviors; and second, it must be able to estimate the probability that a certain behavior will be adopted for a road user. It is based on this two elements that the AV's decision-making must work to produce a safe behavior or at least be able to plan beforehand its actions in possible dilemma situations that might appear.

Next section will present the new configuration for the simulation, assuming a multi-agent formulation to enable the study of the possible interactions between agents. The decision-making procedure executed independently by each road user will be detailed, followed be the AV's estimation procedure of each decision taken by other road users. Afterwards, an additional procedure to increase the robustness of the AV's decision is proposed.

## 5.2 Intent estimation

Given that the scope of the application was enlarged from the original AV's decision-making proposed in earlier chapters to also include the other road users' behavior prediction and the interaction between all agents<fnref n="1" /> in the environment, a new simulation scenario is necessary, where each road users acts and reacts independently (subsection 5.2.1). To do so, every single one of them (it will be considered only vehicles and pedestrians) need its own behavior selection procedure (subsection 5.2.2). And from the AV's perspective, it should be able to measure the probability that each road user is acting according to the AV's expectation (subsection 5.2.3).

### 5.2.1 Simulated Environment

Figure 5.1 represents the new environment to be simulated. An intersection is one of the most challenging environments for an AV, because if it does not have any semaphores (which is the case here) the negotiation between road users might depend on non-verbal communication or movement cues that are not immediately distinguishable by a machine. Each leg of the T-intersection has 50 meters of length, with an intersection in the form of a square with a 10x10 meters of surface. A manual driver is represented by the blue vehicle, that might turn right<fnref n="2" /> or continue straight; the AV is represented by the red vehicle, that might turn left or right; and finally the pedestrian can cross in any of the two pedestrians strips in the scene.

Each road user behavior is determined by its intent, which is a variable that defines what the road user wants to accomplish in the environment. The intent, $c_i$, is composed by two components ($g_i, f_i$), where $g_i$ represents the goal that the road user has (for example, the vehicle

<footnote n="1">¹From now on agents refers to all the other road users and the AV.</footnote>

<footnote n="2">²When the turn maneuver is mentioned the frontal direction of the vehicle in question defines the left and right sides.</footnote>
might want to continue straight in the intersection) and $f_i$ represents the qualitative behavior that a road user might produce during an interaction with another road user. For the vehicles in figure 5.1 for example, this represents if the intersection priority, as defined by the traffic code, will be respected, while for the pedestrian it signals if, in a situations where it does not have the priority, it will nevertheless cross the road.
<figure><img src="image_38.png" /><figcaption>Figure 5.1 – New simulation environment</figcaption></figure>
### 5.2.2 Agent's decision model

Two different types of agents coexist with the AV in the simulation: vehicles (representing manually driven cars) and pedestrians. To implement an independent decision-making inside each pedestrian and vehicle, a decision tree will be used. This component tries to emulate the behavior of a road user if it follows its intent, according to a trajectory ($g_i$ is connected to a pre-defined trajectory) and to some key behavior connected to a salient situation ($f_i$ indicates generically what will be done). All road users are represented by position, direction and velocity, $s = [x_t, y_t, \theta_t, v_t]$, at some time epoch $t$.

The intent of each agent in the environment is translated into its longitudinal and lateral control. The goal determines the path<fnref n="3" /> to be followed, for the pedestrian, which has a predefined trajectory, and for the vehicle, with always follows the middle-lane line. The reference path determines the lateral control of each road user. As for the longitudinal control, it is determined by the (or absence of) interaction with other road users. According to such interaction, one strategy might be adopted as a way to accomplish the road agent's intent. The box below shows all defined strategies for each of the two possible agent types (the strategies are explained in detail at the appendix A).

Available strategies for vehicle and pedestrian

* Vehicle

- Cruise

- Caution

- Follow

- Stop

- Emergency stop

- Swerve

* Pedestrian

- Walk

- Stop

- Run

Starting with the vehicle's strategies, when cruise is the chosen strategy, the vehicle will maintain its speed or it will continue to accelerate until it reaches the speed limit (with constant acceleration). Caution represents the necessary decrease of speed due to the proximity of some other agent. Follow is reproduced by copying the velocity of another agent in front. Stop is used to reduce the velocity and eventually stop using a reasonable braking, while emergency stop uses the maximum deceleration possible and stops in the shortest distance possible. Finally, swerve reduces the velocity and also updates the trajectory proposed by $g_i$ to avoid a collision.

Walk represented the use of a constant speed for the pedestrian, while stop is the immediate reduction of the velocity to zero. Run simply multiplies the speed of the pedestrian in question

<footnote n="3">³Path is defined in this context as a sequence of waypoints that the agent must pass.</footnote>
by a factor of two.

Each strategy for a road user is determined from a pairwise comparison between the agent in question and all others in the environment, as illustrated by the algorithm 5. Firstly, the trajectory of all agent is determined during a certain prediction horizon, defined by the variable $N_t$ (line 1), assuming a default strategy, which for vehicle is cruise and walk for pedestrians. As for the motion models for vehicles and pedestrians, the former uses same one used in previous chapters, represented by equation 3.10, while the latter uses a simple point-mass model. Considering these predicted states, the road user in question, indexed by $i$, is "compared" (how this comparison is done will be explained shortly) to every other road user, indexed by $j$, depending on the type of each one (lines 6, 8 and 14).

The output of each interaction is the predicted strategy for the agent $i$ in respect to agent $j$. If there are $n$ road users in the scene, then at the end of all comparisons for agent $i$ there will be $n-1$ possible strategies (originated from lines 6, 8 or 14). From this set the final strategy $\rho_i$ is calculated simply by checking which $\rho_{i,j}$ has the higher priority, which grows from top to the bottom, using the previous box as reference. Thus, for the vehicle cruise has the lowest priority and swerve has the highest; for the pedestrian walk has the lowest and run the highest.

All road users have access to some shared information, that are the environment structure (dimensions of roads and localization of each structure in the global map), the position of each road user and the decision procedure in itself, and exclusively for the other road users, the intent of each agent. The AV does not known the others’ intent, only its own. Such choice is justified with the assumption that the driver and the pedestrian can subjectively estimate the intention of each other, and to model this subjective process is not the main objective in this chapter. What the methods presented in this section and in the next ones try to achieve is to reproduce the behavior of real road users in specific situations, to then enable the AV to identify such behaviors, not necessarily to emulate completely this subjective process of intent estimation.

Next, the comparisons between predicted trajectories for pedestrians and vehicles will be explained.
**Algorithm 5: Decision procedure for agent i**

**Input:** Environment state $s^t = \{s_i^t\}, i \in [0, N_t]$ time epoch $t$
**Output:** Strategy set according to i, $\{\rho_j\}, j \in [0, N_t]$ for $t$
1 $s = \{s^t, s^{t+1}, \dots, s^{t+N_t}\} = \text{agents\_prediction}(s^t)$
2 **for** every agent $s_i, i \in [0, N_t]$ **do**
3 &nbsp;&nbsp;&nbsp;&nbsp;**if** agent i is a vehicle **then**
4 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**for** every other agent $j, j \in [0, N_t] - i$ **do**
5 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**if** agent j is a vehicle **then**
6 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $\rho_{i,j} = \text{veh\_decision}(\{s_i^t\}, \{s_j^t\})$
7 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**else**
8 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $\rho_{i,j} = \text{ped\_decision}(\{s_i^t\}, \{s_j^t\})$
9 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**end**
10 &nbsp;&nbsp;&nbsp;&nbsp;**end**
11 &nbsp;&nbsp;&nbsp;&nbsp;**else**
12 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**for** every other agent $j, j \in [0, N_t] - i$ **do**
13 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**if** agent j is a vehicle **then**
14 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; $\rho_{i,j} = \text{veh\_decision}(\{s_i^t\}, \{s_j^t\})$
15 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**end**
16 &nbsp;&nbsp;&nbsp;&nbsp;**end**
17 &nbsp;&nbsp;&nbsp;&nbsp;**end**
18 $\rho_i = \text{strategy\_calculation}(\{\rho_{i,j}\})$
19 **end**

Vehicle model

For a road user that is a vehicle, to compare its trajectory to someone else’s, as done in lines 6 and 14, four logical variables are calculated: if a collision happens, if one vehicle is too close to the other, if there is an intersection ahead and who has priority over the other. Collisions and proximity are checked using the mechanisms proposed in chapter 3, section 3.2, which are the idea of frontiers for the proximity detection and the hyperplane separation theorem for the collision detection. To determine the location of each road user inside the simulated environment, a graph-like structure is used to construct the road environment, which is represented by figure 5.2. According to the position and the geometrical dimensions of the vehicle, it is placed in one of the nodes of the road network graph, which automatically determines also the possible next nodes for vehicles.

With the vehicles placed in one of the nodes, their predicted trajectory can be projected
into the graph. Each straight road has a reference lane and an opposite one; it is adopted that the
reference lane is the one that goes from the extremity not connected to the intersection, which
makes the opposite lane definition straightforward. The intersection has a more complex set of
possible destinations.
<figure><img src="image_39.png" /><figcaption>Figure 5.2 – Road environment graph structure</figcaption></figure>

The priority is calculated according to where the vehicle is. For example, if two vehicles are in str1 and str2, two roads in the graph-like structure in figure 5.2, the priority to access the intersection node is given to the one in the right side, considering that the right is measured as the smallest angle between them. If one of them is already inside the intersection it automatically has the priority. It was also defined that the pedestrian has priority over the vehicle if both are in the same node<fnref n="4" />.

The comparison between agents if *i* is a vehicle is displayed by algorithm 6. For each time epoch predicted (from *t* to *t* + *N*<sub>*t*</sub>) the proximity is verified; if they are close to each other, *j* might be in front of *i*, in which case the latter should follow the former (line 4); or both vehicles might collide without being on the same lane. If so, the collision time and the necessary strategy to avoid the collision are registered inside the variable **colResult** (line 7). Then the priority is determined in two specific cases, if *j* is a vehicle and both are in the imminence of entering a intersection (line 15) or if *j* is a pedestrian (line 17). The final strategy is calculated by the function in line 19, which uses a decision tree, detailed in figure 5.3, to select the correct strategy.

All leafs in the tree have the final strategies according to the four variables calculated in algorithm 6, which is case for cruise, caution and stop when there is no collision; if there is a collision, then two specific labels are used: try to stop and dynamic stop. The first one refers to the adoption of the stop strategy if there is enough space and a reasonable deceleration is sufficient to do it and if not the stop degenerates to the emergency stop strategy. Similarly, the dynamic stop checks by prediction what the vehicle can do to avoid the accident, if stop is enough, which can also degenerates to a emergency stop or if a swerve strategy is necessary. All these tests are made by the *checkCollision* function in line 7.

<footnote n="4">⁴A different approach could have been taken, for example defining that the pedestrian has the priority only inside a pedestrian strip, while the vehicle has the priority if the pedestrian is crossing outside the strip.</footnote>
**Algorithm 6:** Comparison if *i* is a vehicle

**Input:** Agent *i* and *j* trajectories ({sᵢ}, {sⱼ}) in [t, t + Nᵢ], starting time t₀
**Output:** Calculated strategy for *i*, ρᵢ
1 **for** each time epoch in {sᵢ} and {sⱼ} **do**
2 &nbsp;&nbsp;&nbsp;&nbsp;**if** checkProximity(sᵢ, sⱼ) && type(j) != pedestrian **then**
3 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;isClose = true
4 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**if** shouldFollow(sᵢ, sⱼ, road_graph) **then**
5 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;following = true
6 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**end**
7 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**if** checkCollision(sᵢ, sⱼ, colResult) **then**
8 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**if** !following **then**
9 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ρᵢ = colResult
10 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**Break**
11 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**end**
12 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**end**
13 &nbsp;&nbsp;&nbsp;&nbsp;**end**
14 **end**
15 **if** localization(sᵢ) != intersection && type(j) != pedestrian **then**
16 &nbsp;&nbsp;&nbsp;&nbsp;hasPriority = checkV2V_Priority(i, j)
17 **else if** type(j) == pedestrian **then**
18 &nbsp;&nbsp;&nbsp;&nbsp;hasPriority = checkV2P_Priority(sᵢ, sⱼ, road_graph)
19 &nbsp;&nbsp;&nbsp;&nbsp;ρᵢ = decideStrategy(isClose, following, colResult, hasPriority)

The strategy selection is dependent from the four logical variables calculated in the vehicle comparison (algorithm 6): proximity, collision, placement and priority. This last one can be modified by the second element of the intent, fᵢ, to create an unexpected behavior. For example, if one vehicle should yield in an intersection its fᵢ can say that it should enter it anyway, independently from the other user. The variable fᵢ has the function to reproduce irresponsible behavior in the simulation, and the challenge from the AV's point of view is to anticipate such behavior using the cues from the other vehicle's movement. How this variable affects the decision tree of a road user will be explained with an application in subsection 5.4.1.

Longitudinal control is defined by the strategy chosen while the lateral control is guided by the middle-lane line. To calculate the final control for vehicles, given gᵢ and the current position the controller shown in equation 5.1 is used. It is the same controller proposed by (Hoffmann et al., 2007) and used in (Thrun et al., 2006), but here the dynamic terms were neglected. It tries to correct the direction offset and the lateral distance between the desired trajectory and the current one and adjusts the steering angle to make the current direction intersects the tangent line from the desired trajectory at k/v units of distance (Snider, 2009).

$$ \phi_{t+1} = (\theta_{\text{traj}} - \theta_t) + \text{atan}\left(\frac{k \cdot d_{\text{lat}}}{v_{\text{long}}}\right) \quad (5.1) $$
<figure><img src="image_40.png" /><figcaption>Figure 5.3 – Decision tree for vehicles</figcaption></figure>

Pedestrian model

Pedestrians also have a method to compare the vehicle's behavior with its own. It is considered unnecessary to compare the behavior of two pedestrians, thus it is assumed that no interaction exclusively between pedestrians cause a change in behavior, leaving only the comparison with other vehicles to be done. The pedestrian strategy is obtained using the algorithm 7 but this time using the current road location of the pedestrian instead of the logical variable related to the existence of an intersection ahead, as is done for vehicles. All trajectories considered for the pedestrian involves crossing the street according to the environment proposed in figure 5.1.

Given that the agent *i* is a pedestrian, its priority is calculated at line 12 of algorithm 7, in
the same manner that it is calculated when *i* is a vehicle and *j* a pedestrian. The parameter *f*<sub>*i*</sub> also controls this priority decision, where it can overturn the *yield* if necessary.

**Algorithm 7: Comparison is *i* is a pedestrian**

**Input:** Agent *i* and *j* trajectories ({s<sub>*i*</sub>}, {s<sub>*j*</sub>}) in [*t*, *t* + *N*<sub>*i*</sub>], starting time *t*<sub>0</sub>
**Output:** Calculated strategy for *i*, ρ<sub>*i*</sub>
1 **for** each time epoch in {s<sub>*i*</sub>} and {s<sub>*j*</sub>} **do**
2 &nbsp;&nbsp;&nbsp;&nbsp;**if** checkProximity(s<sub>*i*</sub><sup>*t*</sup>, s<sub>*j*</sub><sup>*t*</sup>) &amp; &amp; type(j) != pedestrian **then**
3 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;isClose = true
4 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**if** checkCollision(s<sub>*i*</sub><sup>*t*</sup>, s<sub>*j*</sub><sup>*t*</sup>, colResult) **then**
5 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ρ<sub>*i*</sub> = colResult
6 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**Break**
7 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;**end**
8 &nbsp;&nbsp;&nbsp;&nbsp;**end**
9 **end**
10 **placement** = find_Place(s<sub>*i*</sub><sup>*t*</sup>)
11 **if** placement != sidewalk **then**
12 &nbsp;&nbsp;&nbsp;&nbsp;hasPriority = checkP2V_Priority(*i*, *j*)
13 **end**
14 ρ<sub>*i*</sub> = decideStrategy(isClose, placement, colResult, hasPriority)

<figure><img src="image_41.png" /><figcaption>Figure 5.4 – Decision tree for pedestrians</figcaption></figure>

The lateral direction of the pedestrian, if its trajectory requires change in direction, is
done instantaneously, as is changes in its nominal speed of 1.5m/s.
### 5.2.3 Estimating other agents’ intent

Given the previous decision model, the AV can now make use of it to predict the behavior of all other road users, assuming an *a priori* intent for each one and then find the most probable intents. The following information is available for the AV at the beginning of each iteration: the position of all agents, its own goal $g_i$ and the decision process used by other agents. This last one is considered here as a justified assumption for the AV to have due to the common knowledge about the way a human drives or behaves as a pedestrian, based on the traffic code impositions on the infrastructure and on the behavior of all agents and some introspection.

At the beginning of the simulation the AV faces the question of which intent hypothesis best matches each agent, making it possible to explain why an agent is behaving as observed. The entire scene is explained by the information vector $c$, defined by the union of every agent intent as displayed in the expression 5.2 for N agents. Considering that the AV’s intent is always represented by the subscription $0$ then $\bar{c} = (c_i), i \in [1,N]$ is the vector that needs to be estimated by the AV.

$$c = (c_0, \dots, c_N) = ((g_0, f_0), \dots, (g_N, f_N)) \qquad (5.2)$$

Using all possible values for $c$ the AV tries to predict the next strategy, and consequently, the future trajectories of all other road users. Considering the example given in figure 5.1, where there are three road users and assuming that each one of them has 2 possible $g_i$ values and 2 $f_i$ values, it totals 64 possible $c$ configurations. One might question why the intention of the AV is considered if this estimation about the entire environment is done from the AV’s perspective and as such its intent is known; this is because it is important to understand what the other agents are thinking or expecting as a behavior from the AV, which does not necessary mean to exploit a situation where the others have a misguided impression about the AV’s intent. To profit from a wrong estimation done by another road user is not so straightforward since it is desirable that the AV maintain a defensive stance when driving, although it is a subject that merits more discussion and exploration at a later time.

The proposed road-map of the algorithm consists in the following: from time $t$ to $t+N_t$ the AV predicts the strategies of all road users according to an assumed intent vector $c$ and propagates their positions into the future using the prediction step of a Kalman filter. Then, at $t+N_t$, an observation is made, which is compared with the predicted result to evaluate the probability of $c$ being the right intent vector for the environment. After the calculation of $p(z_{t+1} | c_k, s_{t+1})$, where $z_t$ represents the observation, all the predicted values are modified according to the update step of the Kalman filter. The algorithm 8 shows how this procedure implemented, being executed at each simulation step, $\Delta t$. Given this short period for the agents to develop their strategies and that physical cues to become detectable, the update of the EKF and the probability calculation procedure are only executed after $N_t$ transitions.
Kalman filter for agent prediction

This part will focus in the application of the Kalman filter for a single road user, as such
all the formalism used here refers to a single road user, differently from the variables mentioned
in algorithm 8. Instead, the respective parts of each vector $\mu_t$ and $z_t$ are referenced as $s_{i,t}$ and $z_{i,t}$,
$i$ being the index for the road user. For each agent, one instance of an EKF is used. Following the
prediction equations given previously for the vehicle, shown in 5.3, and for pedestrians, 5.4, the
Kalman matrices $G_t$ and $H_t$ are defined for vehicles in expressions 5.5 and 5.6 and for pedestrians
in 5.7 and 5.8.

$$
s_{t+1}^{\text{veh}} = s_{t+1}^{\text{av}} = \begin{cases} x_{t+1} = x_t + \Delta t \cdot v_t \cos \theta_t \cos \phi_t \\ y_{t+1} = y_t + \Delta t \cdot v_t \sin \theta_t \cos \phi_t \\ \theta_{t+1} = \theta_t + \Delta t \cdot \frac{v_t}{l} \sin \phi_t \\ v_{t+1} = v_t + \Delta t \cdot u_1 \\ \phi_{t+1} = \phi_t + \Delta t \cdot u_2 \end{cases} \quad (5.3)
$$

$$
s_{t+1}^{\text{ped}} = \begin{cases} x_{t+1} = x_t + \Delta t \cdot v_t \cos \theta_t \\ y_{t+1} = y_t + \Delta t \cdot v_t \sin \theta_t \\ v_{t+1} = v_t + \Delta t \cdot u_1 \\ \theta_{t+1} = \theta_t + \Delta t \cdot u_2 \end{cases} \tag{5.4}
$$

The state of each other road user that is a vehicle also contains the steering angle, since,
due to the strategy estimation, the control using the trajectory defined by $g_i$ needs to be calculated.
The pedestrian’s state remains being expressed by the previous four variables $(x, y, \theta, v)$.

$$
G_t^{\text{veh}} = \begin{bmatrix} 1 & 0 & -\Delta t \cdot v_t \sin \theta_t \cos \phi_t & \Delta t \cdot \cos \theta_t \cos \phi_t & -\Delta t \cdot v_t \cos \theta_t \sin \phi_t \\ 0 & 1 & \Delta t \cdot v_t \cos \theta_t \cos \phi_t & \Delta t \cdot \sin \theta_t \cos \phi_t & -\Delta t \cdot v_t \sin \theta_t \sin \phi_t \\ 0 & 0 & 1 & \frac{\Delta t}{l} \sin \phi_t & \frac{\Delta t v_t}{l} \cos \phi_t \\ 0 & 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 0 & 1 \end{bmatrix} \quad (5.5)
$$

$$
H_t^{\text{veh}} = I_5 \tag{5.6}
$$
$$ G_t^{\text{ped}} = \begin{bmatrix} 1 & 0 & -\Delta t \cdot \cos \theta_t & \Delta t \cdot v_t \sin \theta_t \\ 0 & 1 & \Delta t \cdot \sin \theta_t & \Delta t \cdot v_t \cos \theta_t \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix} \qquad (5.7) $$

$$ H_t^{\text{ped}} = I_4 \qquad (5.8) $$

The convergence of an Kalman filter is assured for the systems that obey the pre-requisites given by theorem 5, proved in (Krener, 2003). The system considered is given in the form of 5.9, which is uniformly observable if the initial state can be reconstructed with the output and its $(n-1)$ first derivatives, $n$ being the total number of state variables (Gauthier et al., 1992).

$$ \begin{cases} \dot{x} = f(x, u) \\ y = h(x, u) \end{cases} \qquad (5.9) $$

Continuity of a function $f: X \to Y$ can be represented by the Lipschits condition, given by the expression 5.10. The Euclidean norm is used as the metric for the spaces $X$ and $Y$, with $L \in \mathbb{N}$ and $x_1, x_2 \in X$.

$$ \|f(x_1) - f(x_2)\| \le L \|x_1 - x_2\| \qquad (5.10) $$

Finally, the theorem below says that for the conditions assumed the EKF converges exponentially to the correct result.

**Theorem 5** Convergence of EKFs

Suppose that:

* The system in question is uniformly observable and satisfies the Lipschitz condition.

* The second derivative of $f(x, u)$ is bounded.

* $\mu_t$ and $\Sigma_t$ are the solutions of the extended Kalman filter, where $\Sigma_0$ is positive definite and $s_0 - \mu_0$ is sufficiently small

Then $\|s_t - \mu_t\| \to 0$ exponentially as $t \to \infty$.

When a Kalman filter is used to estimate the evolution of some system through time the output is a Gaussian distribution. This precise point is used to measure the similarity between
what is being assumed as an hypothesis for the behavior of the agents and what is really being observed.

Mahalanobis distance and similarity estimation

Usually when one wants to measure the similarity between two vectors the Euclidean norm is used, if these two vectors are defined as points in an Euclidean space. For example, if one have $u, v \in \mathbb{R}^5$, then the distance between the points is given by expression 5.11, where $u_i$ and $v_i$ are the elements of each one of the respective vectors.

$$d(u, v) = \sqrt{\sum_{i=0}^{4} (u_i - v_i)^2} \qquad (5.11)$$

The same is not valid for the output of the Kalman filter. The distance can be calculated between the observation and the distribution mean, but an importance source of information will be left out of the measurement, the covariance. Therefore one needs to evaluate the similarity between a point, the observation obtained, and a distribution, the prediction of each agent state maintained using the EKF, which can be done using the Mahalanobis distance expressed by equation 5.12. This type of norm measures the distance between a sample from a distribution and its center.

$$M(x) = [(x - \mu)^T \cdot \Sigma^{-1} \cdot (x - \mu)]^{\frac{1}{2}} \qquad (5.12)$$

So, to measure the coherence of an hypothesis $c_k$ used to predict the state of each agent, the observation is considered as a sample of the state distribution.

$$M_t^{[k]} = [(z_t - \mu_t^{[k]})^T \cdot (\Sigma_t^{[k]})^{-1} \cdot (z_t - \mu_t^{[k]})]^{\frac{1}{2}} \qquad (5.13)$$

It is well known that the quadratic form of a Gaussian variable is distributed according to the chi-squared distribution (Bensimhoun, 2009). To be more precise, the cumulative sum of squares of $n$ independently normally distributed Gaussian variables, with mean equal to 0 (or the null vector) and covariance equal to the identity matrix ($I_n$) is called central chi-squared distribution with $n$ degrees of liberty. This distribution will be used to calculated the probability $p(z_{t+1} | c_k, s_{t+1})$. However, as said before, the central chi-squared distribution is only a priori valid for $\mathcal{N}(0, I_n)$ which is not the case for the distributions treated here.
If one considers the expression $(z_t - \mu_t^{[k]})$ a new variable, called $U$, it can be assumed that $U \sim \mathcal{N}(0, \Sigma)$ since $z_t$ is being used as a point of this distribution, thus the difference must be zero. Then the theorem 5.1.3 on page 199 of (Mathai and Provost, 1992) can be stated:

**Theorem 6 Distribution of $M_t^{[k]}$**

If $U \sim \mathcal{N}(\mu_u, \Sigma)$ and $\Sigma$ is positive semi-definite, then a set of necessary and sufficient conditions for $U^T A U \sim \chi_n^2(\delta^2)$ is:

* $\text{trace}(A\Sigma) = n$ and $\mu^T A \mu = \delta^2$

* $\Sigma A \Sigma A \Sigma = \Sigma A \Sigma$

* $\mu^T A \Sigma A \mu = \mu^T A \mu$

* $\mu^T (A\Sigma)^2 = \mu^T A \Sigma$

Given that $A = \Sigma^{-1}$ and $\mu_u = 0$ the non-centered chi-squared distribution from the theorem becomes centered due to $\delta^2 = \mu^T A \mu = 0$ and all the conditions are met. Equation 5.14 defines the opposite value of the cumulative probability of the ellipsoid defined between the observation and the predicted state. Such ellipsoid, if small, shows that the observation is close to the center of the distribution, meaning that the hypothesis $k$ might be pertinent. The probability that the Mahalanobis distance $d_M$ is smaller than the calculated value $M_t^{[k]}$ is the same as the cumulative probability of $\chi_n^2$, which value is commonly known.

$$v_t^{[k]} = 1 - P(d_M \le M_t^{[k]}) \qquad (5.14)$$

Finally, the probability of an observation given an assumed hypothesis and the current state can be defined by equation 5.15.

$$p(z_t | c_k, s_t) = \frac{v_t^{[k]}}{\sum_{i=0}^{N_{\text{ins}}} v_t^{[i]}} \qquad (5.15)$$

Using the algorithm 8, that estimates the probability of each possible intent for the agents being true, the AV can now known which strategy it should use if all road users act according to the decision-making procedure proposed in algorithm 5. One detail was left out of the decision process: the variable $f_k$, which controls the behavior of a road user during the interaction. Such variable is used to emulate instances where an agent do not behaves as it should: for example, if a vehicle does not yield for a pedestrian crossing the road. The AV might not be able to identify such behavior and thus react sub-optimally to it.
As such, the AV needs to choose a strategy considering a deviation from normal behavior coming from other road users. To maximize the utility of the worst possible scenario is an operation embodied by the Nash equilibrium in a game theory approach. But the same detail that created the necessity to establish the estimation process of $c_k$ is also present if this approach is applied: the agents do not know which is the intention of the other ones around them. Everything done up until now compared these two by two to obtain an expected strategy given an assumed intent; it is necessary now to consider the collective set of calculated strategies and the estimated agents' intent and their probabilities of being correct to arrive in a coherent decision for the AV.

Such configuration requires an incomplete game, where part or all the players do not have access to some essential information. For such cases the method proposed in (Harsanyi, 1967) can be used to reach a formulation of the game that has a defined solution.

## 5.3 Incomplete game model

As discussed in chapter 2, in an incomplete game the players do not have access to some information about the other players. Without such information it may be impossible to calculate the utility of each possible action that might be taken (if the utility function form or some other parameter is not known) and therefore to find an equilibrium point of the problem, if it exists. To counteract the lack of information, this incomplete game can be converted into a complete imperfect one in which the Nash equilibrium is applicable.

### 5.3.1 Nash equilibrium

In a non-cooperative *n*-player game each player has at its disposal a set of different strategies<fnref n="5" />, which, according to the idea of a game, produces different utilities for the player, each one also depending on the choices of other road users. Every road user tries to anticipate the others' decision and maximize its utility and it is from this interaction between every player that the notion of equilibrium point emerge.

A pure strategy is defined as a single action that an agent may execute, while a mixed strategy is represented by a probability distribution over different actions (or similarly, over pure strategies). As one may expect the interaction between agents happens through the choice of a strategy; if it is assumed that all agents are rational, i.e. that they search to maximize their preferences (deterministic reward for pure strategies and expected reward for mixed ones), then

<footnote n="5">⁵Used in this sense, strategy here is something more general than the longitudinal control profiles proposed earlier, even though when this theoretical discussion is applied to the problem treated in this chapter, this general sense is compatible with the definition of strategy used in subsection 5.2.3.</footnote>
each agent tries to anticipate the choice of the other agents and counter it with the best strategy
from its point of view. This same procedure is repeated inside other agents.

When, for every agent, there is no other better strategy than the one chosen, i.e. the
current strategy is self-countering, then an equilibrium point was reached (Nash, 1950). It has
been proven in (Nash, 1950) and (Nash, 1951) that every finite game has an equilibrium point,
which configures the best collective reward for all the n-players in a non-cooperation game. The
following definition from (Osborne, 2002) resumes the idea of Nash equilibrium:

A Nash equilibrium is an action profile (or a strategy) $a^*$ with the property that no player i can do better by choosing an action different from $a_i^*$, given that every other player j adheres to $a_j^*$.

However, the existence of an equilibrium point is only assured in the case of mixed
strategies (Osborne, 2002). This concept is essential to define what each agent does in response
to an interaction and it is the main tool to solve a game model. It assumes that each agent is
capable to predict the behavior of all other agents, which entails that the intent of each agent
needs to be perfectly known by the AV, which is not realist. It is at this point that a Bayesian
game formulation is necessary.

**5.3.2 Harsanyi's Bayesian game**

In a *n*-player game with complete information all agents known the entirety of the
necessary information to predict the choices of the other agents, all their possible strategies and
their preferences. However, if these details are not accessible one can argue that each agent
maintain a subjective probability distribution over the possible alternatives. The conversion from
incomplete to complete information proposed in (Harsanyi, 1967) uses this concept of probability
over unknown information to try to find a complete game formulation that has similar properties
to the original one.

As said before, each player tries to anticipate the behavior of all others. In a incomplete game, this means that player 1 has some expectation over the behavior of player 2, which in turn also has some expectations about player 1. Such sequence can go on forever, since now player 1 also has an expectation over the expectation of player 2 has of him and so on. The Bayesian game approach model these expectations as subject<fnref n="6" /> probability distribution over the behavior of all other players (back later on).

<footnote n="6">⁶This probability, differently from the objective definition, refers to the belief of one agent about some phenomena which need not be the same for each agent (Savage, 1954), therefore creating a graded belief towards this phenomena (Hajek, 2019). The objective definition, according to (Harsanyi, 1967), refers to the frequentist interpretation of probability.</footnote>
One can postulate that all the subjective probability distributions considered by each agent are the marginal distributions of some original, base probability distribution that has as parameters the information not known by the players, as indicated in expression 5.16, where R refers to the probability distribution and c to the unknown information (which will be the intent vector for the environment). Then the original game and the complete game formed by the substitution of those subjective probability distributions by the marginal probability from this base distribution, according to the definition 2, are equivalent, which entails that both have the same resulting decision rule and the same Nash equilibrium point(s).

$$R(c) = \begin{cases} R(c_1, c_2, \dots, c_n | c_0) \\ R(c_0, c_2, \dots, c_n | c_1) \\ \vdots \\ R(c_0, c_1, \dots, c_{n-1} | c_n) \end{cases} \quad (5.16)$$

**Definition 2 Bayes-equivalency**

Let G be an incomplete game and G* be a complete one, both in standard form (defined by expression 5.18). Both games are *Bayes-equivalent* if for a player j:

* The two games must have the same strategy ($S_i$) and information ($C_i$) spaces (also called intent).

* Both games must have the same expected value function for the payoffs.

* The subjective probability distribution $R_i$ from player i must satisfy the following relationship:

$$R_i(c_0, c_1, \dots, c_{i-1}, c_{i+1}, \dots | c_i) = R_i^*(c_0, c_1, \dots, c_{i-1}, c_{i+1}, \dots | c_i) \quad (5.17)$$

**Form of a game**

At this point it is necessary to give more details about the standard form of a game and its normal form, which will appear subsequently. The standard form of a particular incomplete game G, defined in (Harsanyi, 1967), has the composition shown in expression 5.18. In this context $S_i$ represents the strategy space, $C_i$ the information space (which space of possible values for the intent vector), $V_i$ represents the expected value for the payoff$^7$ and $R_i$ the base probability

<footnote>$^7$In (Harsanyi, 1967), the function V is defined as the common form of the payoff functions of each player, which is only known by the respective player.</footnote>
distribution.

$$G^* = \{S_0, \dots, S_n; C_0, \dots, C_n; V_0, \dots, V_n; R_i\} \qquad (5.18)$$

The normal form of a game is presented in definition 3 (Shoham and Leyton-Brown, 2008). It is defined directly using actions, but it can also be defined in function of the strategies that are available. If these strategies are pure, i.e. they represent only one action, then nothing changes. But if one needs to consider mixed strategies then the total payoff needs to be calculated as an expected payoff.

**Definition 3 Normal-form game**

A finite n-person game in normal form is defined by the following parameters:

* N, the number of players.

* A = A₁ × A₂ × ... × Aₙ, the finite set of actions.

* u = (u₁, ..., uₙ), where uᵢ: A → ℝ is the payoff function for the player i.

When the game is defined in the normal form, the strategies are also normalized. The difference from the usual definition is that the normalized ones are actually functions from the intent space of the road user in question to the strategy space.

**Solving the equivalent game**

The necessary and sufficient conditions for two games, the original G and the complete one G*, to be equivalent is enunciated by theorem 7, found in (Harsanyi, 1968b). Basically, it says that if two games are equivalent, according to the definition 2, they need to have the same strategy and attribute spaces, the same payoff functions and must respect the relationship described in 5.16. Games where the hypothesis that the original distribution exist can be called consistent games, and those that do not, inconsistent. It will be assumed here that the game involving the interaction between road users is a consistent one, although even if a game is inconsistent, the equivalency can be extended to this case (Harsanyi, 1968a).

In the theorem below the normal form of the game is cited. This game formulation, differently from the extensive game that exposes all possible outcomes for each player, is composed of only three elements: the number of players, the set of available strategies for each player and the combined payoff function, that for each player gives its payoff according to its current strategy, there is not a dependency between players in this calculation.
**Theorem 7 Bayes-equivalency**

Let $G$ be an incomplete game and $G^*$ be a Bayesian equivalent game to $G$. To an $n$-tuple strategy $s^*$ of $G^*$ also be an equilibrium point in $G$ is it necessary and sufficient that in the normal form $\mathfrak{N}(G^*)$ of the game $G^*$, $s^*$ be an equilibrium point.

The estimated probability by algorithm 8 will serve as the base probability to calculate the equilibrium points of the equivalent game. This discrete original distribution, calculated by the Mahalanobis distance and the chi-square distribution, is itself based on the conclusion obtained from the comparison about what each road user should do in the considered situation.

According to the determination of each strategy proposed in the previous section each strategy is a pure one, consisting of a single action possible according to the situation at hand. The Nash equilibrium point is not guaranteed to exist is such situations, but it will be assumed that it does exists. Its determination, given the implementation specifications of this context, is similar to the example presented in (Harsanyi, 1968b).

### 5.3.3 Decision making procedure

The main objective of this procedure is to account for every possible situation in the AV's decision-making, so as to avoid instances where the AV is not prepared for a behavior from some road user. Of course, only the behaviors represented by the considered strategies are involved into the deliberation, any other behavior that cannot be approximated by them is not considered. This is a limitation of the procedure, but if one assumes that such strategies can be extracted from real data, one can find a sufficient set of strategies to represent all possible reactions during road users interactions. From the equivalent game formulation, the Nash equilibrium is calculated the payoff values variations according to the intent vector probabilities and the strategies considered.

Each agent will use a reward function that is similar in form to the MDP implementation displayed in subsection 3.2.3. For each intent vector hypothesis there will be a table of payoffs. These tables are in turn connected to the probability that such situation is actually happening. Considering the three agents in the scene represented by figure 5.1, the intent vector is formed by three components, $c_1$, $c_2$ and $c_3$ and the normalized strategies are represented by the Greek letters $v$, $\gamma$ and $\rho$, respectively for each agent. They are functions that take as input the current value of the respective $c_i$ and output the strategy that should be used in this given situation.

$$c = (c_1, c_2, c_3) \rightarrow (v, \gamma, \rho) \qquad (5.19)$$

Then, for each possible value of the c vector, the payoff is calculated, giving different values for each type of strategy, represented by the Greek letters presented earlier with the
subscription that indicated the strategy index, as it can be seen in table 5.1. It this table agent one's strategy is represented by $v$ in the row direction while the combination of the possible values for $\gamma$ and $\rho$ are in the column direction. For clarity reasons each agent will be considered to have only two possible strategies, even if the vehicles have six possible strategies and pedestrians three. Multiple tables like the one below are calculated for different combinations of the intent vector $c$.

<table><caption>Table 5.1 – Example of reward table for a specific intent vector $c$</caption><thead><tr><th colspan="5">c = (c<sub>1</sub>, c<sub>2</sub>, c<sub>3</sub>)</th></tr><tr><th>v \ &gamma;&rho;</th><th>&gamma;<sup>1</sup>&rho;<sup>1</sup></th><th>&gamma;<sup>1</sup>&rho;<sup>2</sup></th><th>&gamma;<sup>2</sup>&rho;<sup>1</sup></th><th>&gamma;<sup>2</sup>&rho;<sup>2</sup></th></tr></thead><tbody><tr><th>v<sup>1</sup></th><td>r<sub>c</sub><sup>1,11</sup></td><td>r<sub>c</sub><sup>1,12</sup></td><td>r<sub>c</sub><sup>1,21</sup></td><td>r<sub>c</sub><sup>1,22</sup></td></tr><tr><th>v<sup>2</sup></th><td>r<sub>c</sub><sup>2,11</sup></td><td>r<sub>c</sub><sup>2,12</sup></td><td>r<sub>c</sub><sup>2,12</sup></td><td>r<sub>c</sub><sup>2,22</sup></td></tr></tbody></table>

Table 5.1 exemplifies the reward calculated for each strategy given an intent vector. According to the definition of the Nash equilibrium point, no agent can achieve a better outcome, so if the system is solved from the viewpoint of agent one, it is not necessary to calculate all the rewards again and find the equilibrium point for agent two and three.

With the probability distribution $P_{t+1}(c)$ calculated in algorithm 8, the total estimated reward is determined by equation 5.20 for a certain combinations of strategies. The functions $f_1: c \to \sigma$, $f_2$ and $f_3$ represent the normalized strategy being evaluated. For example, $f_1$ can assume four possible values according to the number of intents agent one can have (two in this example) and the number of possible strategies (also two); if, for agent one $f_1 = 21$ this means that for the first intent the agent will choose the strategy 2 and for the other intent, the strategy 1. Using this codification the estimated payoff matrix is formed having as row the possible values for $f_1$ and as column the combination of $f_2$ and $f_3$. Constants $N_{c_1}$, $N_{c_2}$ and $N_{c_3}$ represent the total number of intents for each agent.

$$ \mathbb{E}[r^{v^{f_1}, \gamma^{f_2}, \rho^{f_3}}] = \sum_{i=0}^{N_{c_1}} \sum_{j=0}^{N_{c_2}} \sum_{k=0}^{N_{c_3}} P_{t+1}(c_i, c_j, c_k) \cdot r_{c_i, c_j, c_k}^{v_{f_1}(c_i), \gamma^{f_2}(c_j), \rho^{f_3}(c_k)} \quad (5.20) $$

But if one considers that the intent of agent one is known a priori, only the conditional expected reward is necessary, cutting the number of evaluations needed (equation 5.21). Then, the conditional estimated payoff matrix will have two rows (represented in the equation below by the index $l$), one for each of the possible strategies of the agent one, and the columns are all the possible strategies for agents two and three (($\gamma^{11}\rho^{11}$), ($\gamma^{11}\rho^{12}$), ($\gamma^{11}\rho^{21}$), ..., ($\gamma^{22}\rho^{22}$), a total
of sixteen columns).

$$ \mathbb{E}[r^{\nu^l \gamma^l \rho^{f_3}} | c_1] = \sum_{j=0}^{N_{c_2}} \sum_{k=0}^{N_{c_3}} P_{l+1}(c_j, c_k | c_1) \cdot r_{c_j, c_k}^{\nu^l, (\gamma^{f_2(c_j)} \rho^{f_3(c_k)})} \quad (5.21) $$

The optimal strategy in a Nash sense is calculated by equation 5.22 using a maxmin operation, consisting of a minimum possible reward in the column, represented by the index h, for each possible normalized strategy for the other agents, and then a maximization operation to find the best strategy overall in the minimized row. Such strategy, as expected, will give the security that at the worst possible situation the payoff is maximized but not necessarily that the highest possible payoff if obtained in the best scenario.

$$ v^* = \underset{l}{\operatorname{argmax}} \underset{h}{\operatorname{min}} \mathbb{E}[r^{\nu^l \gamma^l \rho^{f_3}}] \quad (5.22) $$

It is the strategy dictated by $v^*$ that should be executed by the AV, together with the lateral reference given by its trajectory. This normalized strategy, that depends only on the current state of agent one, is a pure strategy since the relationship between current intent and strategy is deterministic. According to the theorem 7 this equilibrium point will also maximize the payoff from the original incomplete game.

## Example of application

To better explain the calculation of the Nash equilibrium point and the use of equations 5.19 to 5.22 an example will be provided below. It will be composed of two players, x and y, that each can have two different intents $i_1$ and $i_2$ and execute the actions $a^1$ and $a^2$. This example is similar to the one found in (Harsanyi, 1968b), but it will serve to explain how to apply the method used to calculate the equilibrium point with more that two players. Table 5.2 gives the reward for each possible situation for player x, while the rewards for player y are the same but multiplied by -1. The circled numbers are the saddle points for each situation, the rows give the x actions and the columns the y actions.

The functions v and γ represent the strategy for agents x and y respectively given their intent. Table 5.3 gives the intent probability, which is the output of the algorithm 8.

So, with these elements it is already possible to calculate the normalized strategies for the road user x. There are two possible ways to proceed to the calculation of the Nash equilibrium point: or the equation 5.20 can be used to calculate the expected payoffs for all possible intents of x or, since its intent is known by itself, equation 5.21 can be applied. The second choice will be
<table><caption>(a) Reward for intentions $c_1 = (i_1, i_1)$</caption><thead><tr><th>x\y</th><th>γ(i₁) = a¹</th><th>γ(i₁) = a²</th></tr></thead><tbody><tr><td>ν(i₁) = a¹</td><td>r<sub>c₁</sub><sup>1,1</sup> = (5)</td><td>r<sub>c₁</sub><sup>1,2</sup> = 10</td></tr><tr><td>ν(i₁) = a²</td><td>r<sub>c₁</sub><sup>2,1</sup> = 3</td><td>r<sub>c₁</sub><sup>2,2</sup> = -5</td></tr></tbody></table>

<table><caption>(c) Reward for intentions $c_3 = (i_2, i_1)$</caption><thead><tr><th>x\y</th><th>γ(i₁) = a¹</th><th>γ(i₁) = a²</th></tr></thead><tbody><tr><td>ν(i₂) = a¹</td><td>r<sub>c₃</sub><sup>1,1</sup> = 9</td><td>r<sub>c₃</sub><sup>1,2</sup> = -31</td></tr><tr><td>ν(i₂) = a²</td><td>r<sub>c₃</sub><sup>2,1</sup> = (15)</td><td>r<sub>c₃</sub><sup>2,2</sup> = 27</td></tr></tbody></table>

<table><caption>(b) Reward for intentions $c_2 = (i_1, i_2)$</caption><thead><tr><th>x\y</th><th>γ(i₂) = a¹</th><th>γ(i₂) = a²</th></tr></thead><tbody><tr><td>ν(i₁) = a¹</td><td>r<sub>c₂</sub><sup>1,1</sup> = 10</td><td>r<sub>c₂</sub><sup>1,2</sup> = -25</td></tr><tr><td>ν(i₁) = a²</td><td>r<sub>c₂</sub><sup>2,1</sup> = -5</td><td>r<sub>c₂</sub><sup>2,2</sup> = (-8)</td></tr></tbody></table>

<table><caption>(d) Reward for intentions $c = (i_2, i_2)$</caption><thead><tr><th>x\y</th><th>γ(i₂) = a¹</th><th>γ(i₂) = a²</th></tr></thead><tbody><tr><td>ν(i₂) = a¹</td><td>r<sub>c₄</sub><sup>1,1</sup> = 1</td><td>r<sub>c₄</sub><sup>1,2</sup> = (-4)</td></tr><tr><td>ν(i₂) = a²</td><td>r<sub>c₄</sub><sup>2,1</sup> = -24</td><td>r<sub>c₄</sub><sup>2,2</sup> = -5</td></tr></tbody></table>

<caption>Table 5.2 – Rewards for player x</caption>

<table><caption>Table 5.3 – Intent probabilities</caption><thead><tr><th>x\y</th><th>c₂ = i₁</th><th>c₂ = i₂</th></tr></thead><tbody><tr><td>c₁ = i₁</td><td>0.5</td><td>0.1</td></tr><tr><td>c₁ = i₂</td><td>0.1</td><td>0.3</td></tr></tbody></table>

made here and the *x* intent will be equal to *i*₁, resulting in the conditional probabilities calculated in table 5.4 (The probabilities $P_{t+1}$ are indicated by *p*).

<table><caption>Table 5.4 – Conditional probabilities for $c_x = i_1$</caption><thead><tr><th>x\y</th><th>c<sub>y</sub> = i<sub>1</sub></th><th>c<sub>y</sub> = i<sub>2</sub></th></tr></thead><tbody><tr><td>p(c<sub>y</sub> | c<sub>x</sub> = i<sub>1</sub>)</td><td>0.833</td><td>0.167</td></tr><tr><td>p(c<sub>y</sub> | c<sub>x</sub> = i<sub>2</sub>)</td><td>0.25</td><td>0.75</td></tr></tbody></table>

Simplifying the equation 5.21 to two players gives:

$$ \mathbb{E}[r^{\nu^l \gamma^2} | c_x = i_1] = \sum_{j=0}^{N_{c_2}} p(c_y = i_j | i_1) \cdot r_c^{\nu^l, (\gamma^{f_2(c_j)})} \quad (5.23) $$

To determine the conditional payoff table, each row is formed by a possible strategy for *x*, $ν(i₁) = a¹$ or $ν(i₁) = a²$, while the columns are determined by the possible strategies for *y*, indicated by $γ$. The values that it can assume are displayed below. It becomes clear then the definition of the function $f_2$ for the general case: if one considers the first item in the box below,
$f_2(i_1) = a^1$ and $f_2(i_2) = a^1$, which defines $\gamma^{11}$ (the numbers indicate the strategy index for each intent of y).

* $\gamma(c_y = i_1) = a^1$ and $\gamma(c_y = i_2) = a^1$ or $\gamma^{11}$

* $\gamma(c_y = i_1) = a^1$ and $\gamma(c_y = i_2) = a^2$ or $\gamma^{12}$

* $\gamma(c_y = i_1) = a^2$ and $\gamma(c_y = i_2) = a^1$ or $\gamma^{21}$

* $\gamma(c_y = i_1) = a^2$ and $\gamma(c_y = i_2) = a^2$ or $\gamma^{22}$

The calculation of the conditional payoff for the cell in line one, column one is:

$$ \begin{aligned} \mathbb{E}[r^{v^1\gamma^{11}} | c_x = i_1] &= p(c_y = i_1 | c_x = i_1) \cdot r_{c_1}^{1,1} + p(c_y = i_2 | c_x = i_1) \cdot r_{c_2}^{1,1} = \\ &= 0.833 \cdot 5 + 0.167 \cdot 10 = 5.833 \end{aligned} $$

And for the cell in line two, column three:

$$ \begin{aligned} \mathbb{E}[r^{v^2\gamma^{21}} | c_x = i_1] &= p(c_y = i_1 | c_x = i_1) \cdot r_{c_1}^{2,2} + p(c_y = i_2 | c_x = i_1) \cdot r_{c_2}^{2,1} = \\ &= 0.833 \cdot (-5) + 0.167 \cdot (-5) = -5 \end{aligned} $$

Finally, the conditional payoff table is given by 5.5.

<table><caption>Table 5.5 – Conditional payoffs for $c_x = i_1$</caption><thead><tr><th>x\y</th><th>γ<sup>11</sup></th><th>γ<sup>12</sup></th><th>γ<sup>21</sup></th><th>γ<sup>22</sup></th></tr></thead><tbody><tr><th>v<sup>1</sup></th><td>5.833</td><td>0</td><td>10</td><td>4.167</td></tr><tr><th>v<sup>2</sup></th><td>1.667</td><td>1.167</td><td>-5</td><td>-5.5</td></tr></tbody></table>

To find the best strategy to be adopted, in a Nash sense, it is only necessary to apply the minimal to each row, giving 0 to $v^1$ and $-5.5$ to $v^2$ and then to maximize this result, finding that the best strategy is $v^1$, given that $c_x = i_1$.

## 5.4 Simulation results

Currently only the multi-agent simulation proposed in subsection 5.2.2 is ready to offer some results; the intent estimation procedure proposed in subsection 5.2.3 has been implemented
but not entirely to offer consistent outputs. Therefore, the results presented here will concern only the interaction between the multiple independent agents in the simulation. The simulated scene is given by the figure 5.1, with the same three road users. Given the limitations cited, the two vehicles are implemented as manual drivers with the decision tree component, and the simulation pertains only to the interaction between these road users.

### 5.4.1 Multi-agent simulation

According to the figure 5.1, there are two possible goals ($g_i$) for each agent. The red vehicle can turn right or left, the blue vehicle can continue straight or turn right and the pedestrian can cross the intersection to the right or continue straight to the other pedestrian strip. These options are combined with the second element, of the intent, $f_i$, which is the reaction during the interaction, creating four possible intents for each road user. For each vehicle and in this context $f_i$ indicates if they will obey an yield imperative, since in an intersection without semaphores the vehicle on the right always has priority; and for the pedestrian it defined which is the reaction of the pedestrian in a dangerous situation: if it starts to run or if it freezes and stop in the middle of the road.

Three different scenes will be displayed as a result:

* **Situation 1:** Red vehicle turns right, blue vehicle turns right and pedestrian crosses to the right.

* **Situation 2:** Red vehicle turns right, blue vehicle continues straight and pedestrian crosses to the right.

* **Situation 3:** Red vehicle turns right, blue vehicle turns right and do not give priority and pedestrian crosses the pedestrian strip on top.

All the details about the vehicle object implementation, the implementation of each strategy and the simulation itself can be found in appendix A.

#### Situation 1

An image from the output of the simulation when all road users are approaching the intersection can be seen in figure 5.5. The same situation is shown (not exactly the same time epoch) in figure 5.6, where it can be seen on the right side the three agents moving and on the left the trajectory colored with the respective strategy in execution at each position.
<figure><img src="image_42.png" /><figcaption>Figure 5.5 – Illustration of the intersection interaction</figcaption></figure>

Considering that the pedestrian starts to move at $t = 7s$, one can clearly see that the red rectangle agent (which represents the red vehicle at figure 5.5) detects that the pedestrian is crossing the road and therefore has the preference. It decreases the velocity to stop in the intersection and avoid a collision, until the point that it can be sure that no collision with this pedestrian can happen, then it resumes the cruise strategy and turns downwards.

<figure><img src="image_43.png" /><figcaption>Figure 5.6 – Strategies used during situation 1</figcaption></figure>
The other vehicle (blue rectangle, blue car) stops first because it detects a collision with the red agent during the turning trajectory, but resumes the cruise soon after. The pedestrian does not detect any danger since the red vehicle waits for it to get to the opposite lane to start moving again.

## Situation 2

In this situation the blue vehicle now will continue straight ahead instead of turning right. As it can be seen in figure 5.7, it detects that there is a pedestrian that might cross the street and that the red vehicle has the priority over him, and because of these reasons it engages the stop strategy very soon into the simulation, stopping right at the intersection. The red vehicle, given that it needs to decelerate to turn and that in that specific instance the acceleration sampled was $2.5m/s^2$ (the discussion about the acceleration sampling can be found in appendix A), did not need to stop and wait for the pedestrian to arrive at the opposite lane.

<figure><img src="image_44.png" /><figcaption>Figure 5.7 – Strategies used during situation 2</figcaption></figure>

After the red vehicle turned, the blue vehicle starts the intersection crossing, it detects that the red vehicle is in front and then start to follow him, as it can be seen by the green trajectory in figure 5.7. Both vehicles continue cruising until the end of the simulation.
## Situation 3

This situation emulates a scenario where one of the vehicles, in this case the blue one, do not abide the priority to the other vehicle and the pedestrian. However, since the idea is to represent a human driver, the decision making is still sensible to collisions. Looking at the figure 5.8 one can see that there is one interval in the blue's trajectory that is an emergency stop strategy, which is exactly due to a possible collision with the pedestrian and the red vehicle. When the pedestrians freeze in reaction to the proximity of the blue car the red vehicle also engages in a controlled stop at the intersection, thus avoiding the collision altogether.

<figure><img src="image_45.png" /><figcaption>Figure 5.8 – Strategies used during situation 3</figcaption></figure>

The parameter $f_i$ from the blue vehicle overrides the common decision tree for vehicles shown in figure 5.3. In the no collision branch, the intermediary nodes related to the priority are eliminated, since the blue vehicle will not yield in any situation; the new branch taken these modifications into account can be seen in figure 5.9. The collision branches continue with the same structure, which is why the blue vehicle, when it perceives a collision, uses the emergency stop strategy.
<figure><img src="image_46.png" /><figcaption>Figure 5.9 – Modified vehicle decision for the no collision sub-tree</figcaption></figure>

**5.5 Final remarks**

In chapter 3, to allow the definition of the MDP state as only the AV's configuration it was assumed that, from the start of the policy execution, after is calculation, the intentions of other road users did not change, which entails that their direction and velocity remains the same. This continuity allows the AV to predict their trajectory only using their current configuration. The main goal of this chapter was to propose a method capable to predict this possible change in intent during the policy execution, so as to relax this assumption made. Three main components form the prediction algorithm: the deterministic decision-making that emulate other vehicles and pedestrians, the probabilistic distance estimation, that calculates the probability of some intent given an observation of the current situation and an incomplete game model to adapt the choice of strategy to the current interaction between agents. Finally, the results of the multi-agent simulation using the decision-making models are shown.

However, some considerations about the proposed intent estimation method needs to be made. During the entire chapter the deterministic decision-making model proposed was suggested to be used by the AV as a way to predict the other road users strategy and as the decision-making process of the other road users themselves. This would work with the estimation of intent by the AV, but the other capability, the game model to account the interactions between road users, is supposed to close the gap between the simulation and real data. Thus, to test the incomplete game formulation in a simulation it would be necessary to find another decision-making procedure to be implemented in the agents, preferably one that mimics real humans using real data.

Another concern related to the game model is that it is based on the assumption that the Nash equilibrium point could be reached using a pure strategy, which entails that for each agent, given its state and intent, the strategy taken is deterministic. These agents are human drivers and
pedestrians, that have personalities and different motivations, thus such assumption that every
single one would act equally is a strong one and must be relaxed. A mixed strategy would be
more suited for the problem, creating an additional degree of liberty according to the specificity
of each component that predicts an agent behavior. To allow such consideration it would be
necessary to update the proposed intent estimation given by algorithm 8 to a nested particle
filter, capable to consider multiple hypothesis of intent and for each intent multiple hypothesis of
strategy. Afterwards the proposed Nash equilibrium point determination would also need to be
modifies to deal with mixed strategies.
# Chapter 6

# Conclusion

## Contents

<table><tr><td>6.1 Final remarks</td><td>141</td></tr><tr><td>6.2 Future research perspectives</td><td>144</td></tr></table>

## 6.1 Final remarks

In this thesis the question concerning how the automated vehicle should deliberate on decisions, given its perception information, mission data and the presence of other road users in its surroundings was treated. Both normal situations for a human driver and the hypothetical case of an ethical dilemma situation were addressed. The starting point of the discussion around automated vehicles, and more generally artificial intelligence and automated mobile robots, was developed in chapter 1, from the early days of artificial intelligence with the definition of the Turing Test, the development of the first automated mobile robots, the paradigms to orient the architecture of such robots and the development of driveless vehicles during this process. Some of the most important breakthroughs in the area were cited, for example the works of Dickmanns and Zapp, 1987 and the two prototypes that won the DARPA challenges, Thrun et al., 2006 and Urmson, Anhalt, Bagnell, et al., 2008.

Having done a review of the history of automated systems, the current state of the art in the domain of automated vehicles was finally described, from most useful advantages that one hopes the AV technology delivers, the current maturity level of the deployment of such robots in the real world and what is currently being commercialized by the automotive industry. The most promising prototypes of level four automated driving were also discussed, with a special attention to the Tesla's implementation of automated driving.
Chapter 2 discusses the state of the art of each domain addressed in the theses: decision-making, ethical decision-making, behavior prediction and game theory. Each subject starts with an explanation about the organization of the concepts that define the field of research to then proceed towards the analysis of a set of representative publications, recent and older, from each domain.

The decision-making algorithm for an AV is proposed and explained in chapter 3. Starting from the discussion about implementation paradigms from chapter 1, the choice of architecture assumed, so as to model the decision-making process with the correct responsibilities and functionalities, was defined to be the 3-tiered architecture Gat, 1998. From the theoretical side, the formulation of a Markov Decision Process has presented next, focusing in the existence and uniqueness of the soon to be calculated policy, given the assumptions made about the system. Finally, the formulation of the MDP is presented, with the hypothesis that the behavior of the other road users does not change throughout the execution of the policy, which enabled the definition of the state to contain only the AV's configuration while the other road users' configuration were accounted for in the reward function. The results obtained from the simulation are discussed next, not before the definition of the simulation environment and the value iteration method. The following conclusions could be drawn from such experimentation:

Conclusions from Chapter 3 experiments

* The necessary horizon to sufficiently account for the behaviors of other road users is in direct conflict with the exponential nature of the defined state space.

* The prediction horizon and the AV's velocity also have an antagonistic relationship, given that the transition time remains fixed.

* Or some risk evaluation is necessary to stop the AV from starting a maneuver that ends in an unsafe state for the current prediction horizon or the idea of prediction horizon should be modified to be dependent to the maneuver completion.

Ethical dilemma situations are treated using the methods proposed in chapter 4. First, one needs to define what are dilemma situations, which is done in the first section of chapter 4 to then step into the definition of the parameters used to calculate the harm of a collision, the difference of velocity due to the physical interaction between road users and the vulnerability constant. Using this concepts two types of ethical decision-making procedures are presented: the Ethical Valence theory (EVT), that uses the ethical valence assigned to each road user, in combination to the predicted harm, to deliberate; and the ethical optimization (EO), three different handcrafted minimization procedures that are inspired by the utilitarian, egalitarian and Rawlsian contractarian ethics. Both deliberation methods were tested, with the EVT in an hypothetical solution and assuming a specific definition of features for the ethical valences while the ethical optimization was tested as an MDP criteria for ethical decision, in the same configuration of the tests executed in the previous chapter. In both implementations the algorithm correctly calculated
that the collision with the opposite vehicle would be too dangerous, and opted or for the collision with the pedestrian or for the collision with the wall in the frontier of the simulated environment.

Conclusions from Chapter 4 experiments

* EVT: According to the moral profile chosen, the deliberation system protected or the AV's passengers or the most vulnerable users.

* EO: The three actions produced the same result, a collision with the pedestrian, but with different intensities; the utilitarian approach gives the smallest collision velocity for the AV with the pedestrian.

The main subject of the chapter 5 revolves around the relaxation of the assumption made in chapter 3, that the behavior of other road users did not change during the MDP policy execution. To eliminate such assumption it is necessary to account the interaction between each road user, which entails the estimation of their motivations, their intent. To entry in the matter at hand, the types of errors that an AV is subject to are discussed, so as to isolate the error that might produce an ethical dilemma situation, when other users act in an unforeseen way from the AV's point of view. From this point an deterministic decision model for vehicles and pedestrians is proposed, making possible the implementation of a multi-agent simulation and allowing the AV to predict what will be the answer of the other road users given some interaction between them. Using this prediction the AV can test which intent describes the behavior of the other users the best, according to this deterministic decision model, and then calculate the probability that some road user has a determined intent given by the distance between the prediction made by the Kalman filter and the observation.

This is only enough to estimate the intent of each road user, not to determine which action it will execute next. For such task it is necessary to apply a game theory model of the interaction, more specifically the interaction at a precise time is modeled as an incomplete game, with the intent of the other agents, that are estimated by the previous algorithm, as the unknown information. Calculating the Nash equilibrium point of the equivalent Bayesian game then gives the adopted strategy for each of the agents in the environment. However, the algorithm described only accounts for pure strategies, which does not guarantee the existence of the Nash equilibrium. To allow mixed strategies as an end result of the procedure, it is necessary to modify the intent estimation process so as to account multiple strategies for a single intent being compared, and then also expand the Nash equilibrium determination procedure as well.

Using this procedure of intent estimation allows the correct evaluation of the transition probability for the MDP proposed in chapter 3, which means that a more precise account of the expected harm and the multiple possible outcomes in a dilemma situation can be determined and deliberated upon. Not only this but the simulation itself is closer to reality due to the independence of other road users behaviors from the AV's controller. All could result in an proposition of a decision-making with ethical deliberation capabilities that accounts for behavior uncertainty, an
implementation yet not studied.

**6.2 Future research perspectives**

Multiple lines of research can be branched out from the work done in this theses. Some of them are presented and commented below, in no specific order:

**New decision-making system using an POMDP** Given that, in the intent estimation for other road users, their positions are predicted using a Kalman filter there is a probability distribution around the state (which are now defined by the configuration of each road user, including the AV), then it would be more appropriate to consider a partially observable Markov decision process to model de decision-making, instead of the model proposed in chapter 3. Of course, this comes with more complications since it is harder to calculate the optimal policy for such cases, often an approximate solution suffices due to the computational cost involved to find the optimal solution. However, the same problems observed in the MDP implementation still need to be solved, notably the antagonistic relationship between the predicted horizon and the capacity to account to other road users behavior (which can be better answered by a state space sampling methods, as the one that Kurniawati and Yadav, 2016 proposes) and the addition of some measure to verify that a maneuver, when started, is at least simulated until its end. In this case the structure of ethical and normal decision making deliberation retains the same functionality as before, if a dilemma situation is detected then the ethical deliberation takes place and a highly negative reward is attached to the situation.

**Definition of other road users' behavior using machine learning** In chapter 5 the decision process of other vehicles and pedestrians were defined in a deterministic fashion by decision trees. One improvement that could be made is to learn a driver or a pedestrian behavior from real data using a machine learning approach, as Kuderer et al., 2015 did. The resulting behavioral model could produce simulated agents that reproduce more realistically the dynamics of driving, for example acceleration, deceleration, respect to the traffic code, etc. Such method would also allows the test of multiple types of driving profiles, from a more cautious to a more aggressive driver, with the same being valid for the pedestrian. Specifically for pedestrians, there is an entire field of research about the prediction of intentions of pedestrians using its pose, head orientation and position. Certainly the capabilities of the intent estimation would be improved with more realistic methods.

**Fusion between the normal reward criteria and the ethical deliberation** The ethical deliberation proposed in chapter 4 are only used if a dilemma situation is detected, which might not necessarily happen for every real dilemma situation. Detection of some phenomena almost always has to deal with false negatives instances of detection and given the gravity of a dilemma situation even the existence of a false negative must be avoided at all times.
Therefore, it would be more suitable to fuse both objective and ethical criteria into one
hybrid deliberation process. Both components would be accounted at every action taken,
with one predominant in normal situations and the other in ethical dilemmas, so as to
minimize the loss of performance given that the other component could influence the
system towards non-optimal decisions (in the context of a non-dilemma situation) and the
same during ethical deliberation. Another motivation for this modification is that up until
this point, outside the dilemma situation, the selection criteria for action is entirely based
on an objective regard of the situation, but an argument can be made that some instances in
normal driving should involve an ethical component for an AV (even if for a normal driver
this is not the case), so as to encompass possible passive risks due to each AV's action. For
example, if there is a cyclist in the same lane as the AV, should it overtake him? From a
performance point of view, yes, but the increase of proximity between both agents would
mean that the risk of something go wrong and this vulnerable road user end up entangled
in a collision increases. Such fact ideally should also be accounted, which is possible using
a hybrid decision process.

**Exploration of a virtue ethics approach for ethical learning** As established in chapter 2, virtue ethics says that morally good actions flows from a good moral character, which is composed by the correct virtues. Implement such an approach in a machine is not as straightforward as is for deontology or consequentialist based methods, since some sort of character needs to be established and than it must evolve through practice, so as to learn to act ethically. The reason why this is proposed here as a possible research perspective is that such system would be interesting to define the weight of each component in a hybrid decision criteria. Using the framework proposed in Arkin and Ulam, 2009 as a model (the emotions considered in the publication is exchanged by the virtues), one could simulate different outcomes for many dilemma situations and assess the consequences of each action to then backtrack to modify the virtue so as to prevent some actions to be execute while allowing others to. The main question is how to represent a virtue, and to be sure that it represents the same as we, humans, think it should. Or even some artificial virtues could be proposed specifically to produce outcomes that could be considered morally correct.

**Verification and validation of stochastic decision-making systems** One of the key problems with stochastic models of decision is that certainty cannot be guaranteed. So, to evaluate the performance of some system and to check its safety operational properties different indexes are needed. The necessity of such tools increase with the critical aspect of the system, which means that decision-making algorithms for automated vehicles should pass through verification and validation (V&V) using strict measures of performance and safety. One example of V&V for AVs is Barbier et al., 2018 using KPI to evaluate a POMDP performance.

**Real-time implementation and test with a real AV** To target a real testing session of the algorithm some important modifications are necessary. The first one is the addition of the perception procedure and the delays involved in the extraction of information before the start of the deliberation process. Also, the measurement uncertainties must be combined with the estimation procedures proposed here. The second question to be treated is the
possible real-time execution of the deliberation. Given the computational effort necessary, the limitations of an embedded system (even if it is a vehicle, it has a far less limitation than a drone for example) and the already observed computation times it is fairly possible that real-time performance is off the table. Nonetheless improvements in calculation effort should be made to increase the responsiveness of the AV. One possible solution is the implementation of parts of the algorithm using GPU, that can massively decrease the processing time of highly parallel tasks. Considering only the evaluation of the reward in chapter 3, it could be parallelized since each calculation is independent from each other. Another algorithm that could take advantage of such operation is the intention estimation nested particle filter structure.
# Appendix A

## Simulation set-up

To test the algorithms proposed in this thesis, the simulator called WeBots was used. It is a 3D simulation environment where dynamic simulations can be done using any type of object, the already available modeled ones or personalized structures created using the constructs available. Most of its applications are in the field of robotics, as for example industrial robots like the UR3, UR5 and UR10 arm robots from Universal Robots, Spot, the dog-like robot from Boston Dynamics and Nao, the humanoid robot from Aldebaran Robotics. It also has pre-modeled vehicles, like the Tesla model 3 and many others. In this appendix it will be presented the main structure of simulation, details about the control of vehicles and pedestrians and the modifications made in the simulator to unlock new capabilities that were necessary for the simulations proposed in the thesis.

### A.1 Webots simulation structure

The simulation structure is similar to other simulators, offering as the main representative entity nodes, that are any type of object in the 3D scene. In the AV context, the road environment in the experiments described at chapters three, four and five was constructed with pieces of straight roads and an intersection, as the figure A.1 shows. For the physical interaction between agents, the Open Dynamics Engine (ODE), an open-source library, is used to calculate rigid bodies dynamics.

The behavior of objects can be controlled externally using a controller, which can be a program written in Python, C++, Matlab or Java. The controllers used, one for the vehicles<fnref n="1" />

<footnote n="1">¹The AV uses the same control functionalities described here combined with the algorithms proposed throughout the thesis.</footnote>
<figure><img src="image_47.png" /><figcaption>Figure A.1 – Main window of the simulator</figcaption></figure>

and another for the pedestrian, were implemented in C++. It is also possible to use perception sensors, cameras, LiDARs, radars and wireless communications, but given that in a simulation it is possible to find the positions and velocities of all moving objects, this was the favored option as a method to determine the inputs of the decision-making. Also, the implementation of perception routines would add a complexity level that is outside the scope of this thesis (although it is an important component that should be considered in a later time).

The controllers for both vehicles and the pedestrians have one component in common: the interface between the control functions and the Webots objects. This concerns uniquely the observation acquisition, since the control reference dispatching is specific according to the controller (and it will be detailed later on). For the observation acquisition, each object being controlled can access the main node tree of the simulation, where every object in the scene have a respective node, and pull the position and velocities for each moving object at each given time.

Every simulation done in this document uses the time step of 10ms, i.e. the frequency in which the simulator calculates all the physical reactions and updates the positions and velocities from the objects accordingly. Each time step happens synchronously; after the dispatch of a control reference each object executes a function called step, that signals to the simulator that it can execute another iteration. If there are other objects that did not yet execute this function, the controller routine is blocked in this function until all other controllers call the same function from their side. Then, after every controller signaled the simulator it calculates another step and unblocks all the controllers.

In the next sections the particularities about each of the controllers used will be explained,
and in the last section the modification of one of the simulator libraries, that consist in the creation
of one special function to control the car-like vehicle and a modification of the pedestrian model,
will be treated.

## A.2 Controlling a car-like vehicle

One of the implemented controllers concerns the car-like vehicles. Apart from the
observation acquisition already described, it builds upon the capabilities of the WeBots libraries
called Driver and Car, the former providing all the functionalities that a human driver usually has
in a vehicle, including functions to steer and to accelerate/brake, and the latter completing the
former library with functions that are not usually available to the common driver, for example to
change the blinking period of the indicator or getting physical parameters of the car.

Two types of longitudinal control are available: speed mode and dynamic mode. The first one emulates a cruising control system, increasing and decreasing the velocity of the vehicle according to a velocity reference, using a constant acceleration value. In this mode the throttle and the brake are not directly used to control the vehicle. This was the function used to change the vehicle's velocity during the simulations for chapters 3, 4 and 5 (this last chapter has some particularities that will be commented in the last section). As for the dynamic mode, it allows to control the vehicle with the throttle, the gears and the braking directly, according to the type of engine of the chosen vehicle. Electric vehicles like the Tesla have only two gears, in a 1+1 configuration (forward and backward), while the combustion vehicles have six or seven gears, in a 5(6)+1 configuration. Given the needs and limitations of the simulations developed throughout the thesis the speed control was enough to model the vehicles' behavior.

Velocity, acceleration and steering selection

As said in chapter 3, the steering and the velocity of the vehicle are defined by the action
selected by the policy being executed. The pair steering angle / velocity is directly sent to the
controller using the respective functions from the Car library with the speed mode activated. This
produces a cruise control-like behavior, with a constant fixed acceleration. Steering in this case is
managed by the MDP, given that the reward of each couple state / action depends on the reference
trajectory.

For the simulations in chapter 5, the Stanley controller, given by equation 5.1, takes care
of defining the steering angle to correct the vehicle's trajectory towards the reference, both for the
AV and for the other vehicle. For the velocity, it is given according to the strategy chosen, and
it is applied to the vehicle using the cruising control from the speed mode, but this time with a
variable acceleration between iterations. In figure A.2, each point of a hypothetical trajectory is
displayed; for each point the controller calculates a velocity limit, that depends on the curvature and on the velocity limit imposed by the traffic code considered, together with an acceleration to be used between the current and the next point, at least until next iteration.

<figure><img src="image_48.png" /><figcaption>Figure A.2 – Longitudinal control for the vehicle</figcaption></figure>

For all trajectories, the curve trajectory was obtained with a 3rd order Bézier curve, which was stitched into the two straight line trajectories at the middle of the road segments. Using this curvature, the maximal velocity was calculated, given the friction limits of the vehicle, by equation A.1, where μ is the tire’s friction coefficient, g is gravity and κ the curvature at the trajectory point being considered. This expression comes from the the equality between the centripetal force and the lateral friction, simplified so as to be equal to the maximum longitudinal friction of only one tire where the correct would be to have a dynamic model capable to determine the lateral force in all four tires given the dynamics of the car’s body and the slippage influence in the friction force. But since the vehicle model used was kinematic, this approximation was applied, which did not resulted in slippage during the turning operation.

$$v_{l,max} = \sqrt{\frac{g\mu}{\kappa}} \qquad (A.1)$$

The maximal velocity to execute a turn operation was determined to be 5.41m/s, close to
<figure><img src="image_49.png" /><figcaption>Figure A.3 – Vehicle’s acceleration sampling distribution</figcaption></figure>

20km/h, which is a reasonable speed to execute a 90° curve. With this velocity determined, the adjustment for all the trajectory points is done modifying the velocity in the points located before the curve start, to arrive in it with the correct velocity. In the straight sectors the maximal velocity is equal to the velocity limit established by the implementation of the traffic rule, 40km/h. As for the longitudinal acceleration, a normal distribution given by the figure A.3 is used to define the factor which multiplies the maximal acceleration from the vehicle used, the Tesla Model 3. Its maximal acceleration is 6.945 m/s², which is then multiplied by the value sampled from a normal distribution defined as N(0.3, 0.15).

But since only positive velocities are used the part of the Gaussian that refers to negative values needs to be disregarded. Adding to this the fact that a minimum acceleration is necessary results in the definition of the constant $k_{acc}$, setting the minimal acceleration to be 1m/s², which has a probability of approximately 15% of being drawn (corresponds to the left area of the dashed line in figure A.3). This sampling is useful to test multiple accelerations profiles, and once defined this acceleration is used throughout the simulation, it is not changed depending on the situations like the deceleration. Next, each of the available strategies for the vehicle at the chapter 5 will be detailed.

## Cruise, caution and follow strategies

The cruise strategy simply adopts the velocity limit given by the traffic code (or the maximal velocity in curves, $v_{l,max}$) as the reference for the cruising control. If the vehicle has a higher velocity before the intersection, it calculates how soon it needs to decelerate to enter the
turning trajectory with the correct velocity. The same value to accelerate is used to decelerate initially, and if it is not enough to arrive at the curve's entry point with the correct velocity, then the necessary deceleration is calculated using the Torricelli's equation (A.2), that determines the resulting velocity $v_{f,x}$ according to the initial one $v_{i,x}$, the acceleration (or deceleration) considered $a_x$ and the distance between initial and final points, $\Delta s_x$, all in one direction.

$$v_{f,x}^2 = v_{i,x}^2 + 2a_x \cdot \Delta s_x \qquad (A.2)$$

The caution strategy cuts in half the velocity of the entire trajectory while it is being used, including in curves. And to execute the follow strategy, a vehicle needs to equate its velocity to the one of the vehicle in front, hence the necessary deceleration is again calculated using equation A.2 to achieve the correct velocity given the distance between both vehicles. This behavior, if applied once and not updated during long stretches of time can cause a collision (which then would result in a stop strategy), but it is refreshed at every iteration so if the vehicle in front reduces its velocity the deceleration is again recalculated, allowing the maintenance of a safe buffer distance.

## Stop, emergency stop and swerve strategies

Two situations can trigger a stop strategy: when a collision is detected and when the vehicle does not have the priority to access the intersection. In the first case, using the position in which the collision was predicted, the deceleration necessary to stop 5 m from this position is calculated, again using equation A.2. If the deceleration necessary is higher than the one available (and since the speed mode is being used the maximal deceleration possible is equal to the max. acceleration, 6.945 m/s<fnref n="2" />) then the emergency stop is activated. It simply consists of applying the maximal deceleration until the vehicle stops or when the collision is no longer predicted.

In the second case, given that the environment is always perfectly observable, i.e. the road graph structure illustrated in 5.2 is always available, so it is always possible to stop before an intersection. If a vehicle perceives that another is driving to the intersection and that it has the priority, then the stop procedure starts, with a calculated deceleration until the entry point of the intersection.

The swerve strategy usually refers to the capacity of the vehicle to accelerate and change direction to avoid a collision that usually cannot be avoided by braking<fnref n="2" />. Given the environment used, the strategy of changing direction is never used, and therefore it was not implemented. On the other hand, if the stop and the emergency stop are not enough, then an increase of velocity is tested; if it works, it is adopted as a swerve, if not then the emergency stop strategy is used.

<footnote n="2">²If one assumes that the increasing the velocity in a dangerous situation is not allowed, so as to mitigate a possible collision severity, the swerve can only happen when braking is not enough to avoid an accident.</footnote>
## A.3 Controlling the pedestrian

Differently from the vehicle case, there is not a special library with control functions for pedestrians, but there is an example of a controller available as an application example preinstalled that makes the pedestrian "walk". The controller implemented then uses some parameters from this application, notably the joint angles. The pedestrian object (figure A.4) in the simulator is defined by thirteen joints:

* Head angle

* Left shoulder angle, left elbow angle, left hand angle.

* Right shoulder angle, right elbow angle, right hand angle.

* Left hip angle, left knee angle, left foot angle.

* Right hip angle, right knee angle, right foot angle.

Each one of the joints has eight angle values that are used to give a sense that the pedestrian is moving, when it is actually being displaced at the same time that its joint angles are being modified. One can even see in figure A.4 that the foot of the pedestrian penetrates the geometric representation of the sidewalk, since the physical interaction between then is not enabled. These joint angles function in a circular fashion, after eight iterations it start again but in a value that is the natural continuation of the last, creating the illusion of a fluid movement.

As for the pedestrian control, it is rather straightforward; its velocity is changed as it was a point of mass and the same is valid for its direction. The only feature implemented was the timer to start, where the controller only starts the movement after certain amount of time.

The trajectory of the pedestrian is defined by five points in simulation (but it could be used many points as needed), that are executed linearly. Its orientation is defined to be the same as the segment from its position towards the next trajectory point. As it was said in chapter 5, danger for a pedestrian is defined as a situation when the vehicle is closer than five meters, triggering a change of strategy from walk to stop or to run. Walk is the behavior mentioned until now, the execution of the trajectory in a linear piece-wise fashion, with a constant velocity of 1.5 m/s, while the stop strategy is characterized simply by the instantaneous change in velocity to zero. And the run strategy multiplies the pedestrian velocity by two, to 3 m/s.
<figure><img src="image_50.png" /><figcaption>Figure A.4 – The pedestrian object</figcaption></figure>

**A.4 Necessary modifications**

During the explanation of the control modes for the vehicle it was said that for the speed mode, which is the one used to control the vehicle in all simulations, the acceleration used by the cruising control is fixed to be the maximal acceleration possible given the car model used. However, the strategies for the vehicle need to change its acceleration to brake under some maximal distance and it is also necessary to change the acceleration used in the cruising control so as to apply the sampled one, for the simulations done in chapter 5. To create this functionality a modification was made to the Driver library.

All libraries for WeBots are implemented in C, to then be referenced by an C++ interface. In the Driver C file, a call to a new function, that changes the acceleration of the angular motor that controls the tires depending on the desired linear acceleration (or deceleration) for the vehicle, makes the acceleration change possible. With the library compiled, the effects of the changes in acceleration tested were correctly predicted by equations 3.10.

Another modification done was related to the capacity of the objects to access the node tree to retrieve positions and velocities from the other objects in the simulated scene. To be able to access such node tree, a robot should have the status of supervisor. For the vehicle this option is already available at the simulator interface (which means that it is listed in the parameters file of this particular car model), but not for the pedestrian. Figure A.5 shows the pedestrian node with the already modified field.
<figure><img src="image_51.png" /><figcaption>Figure A.5 – Pedestrian supervisor field</figcaption></figure>

To enable this option one entry was added in the PROTO file of the pedestrian model, which is the file that lists all existent parameters of an object, including motors’ parameters, colors of the object’s parts, name of the object, initial joint values, etc. In this way the pedestrian can obtain the observation from the environment the same way that the vehicle does, directly from the simulator. Apart from these two modifications, the simulator is used as it is available in the WeBots site.
Bibliography

AAA (2019). *Automatic emergency braking with pedestrian detection*. Report. American Automotive Association. URL: https://newsroom.aaa.com/wp-content/uploads/2020/11/Pedestrian-Detection-Full-Research-Report-Oct-2019.pdf (visited on 03/01/2021).

Ahmed, Sarfraz, M. Nazmul Huda, Sujan Rajbhandari, Chitta Saha, Mark Elshaw, and Stratis Kanarachos (2019). “Pedestrian and Cyclist Detection and Intent Estimation for Autonomous Vehicles: A Survey”. In: vol. 9. 11. MDPI. DOI: 10.3390/app9112335.

Alahi, A., K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese (June 2016). “Social LSTM: Human Trajectory Prediction in Crowded Spaces”. In: *2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 961–971. DOI: 10.1109/CVPR.2016.110.

Alexander, Larry and Michael Moore (2020). *Stanford Encyclopedia of Philosophy: Deontological Ethics*. Stanford University. URL: https://plato.stanford.edu/entries/ethics-deontological/ (visited on 05/08/2021).

Allen, Colin, Gary Varner, and Jason Zinser (2000). “Prolegomena to any future artificial moral agent”. In: *Journal of Experimental & Theoretical Artificial Intelligence* 12.3, pp. 251–261. DOI: 10.1080/09528130050111428. URL: https://doi.org/10.1080/09528130050111428.

Anderson, James M., Nidhi Kalra, Karlyn D. Stanley, Paul Sorensen, Constantine Samaras, and Tobi A. Oluwatola (2016). *Autonomous Vehicle Technology: A Guide for Policymakers*. Santa Monica, CA: RAND Corporation. DOI: 10.7249/RR443-2.

Anderson, Michael and Susan Leigh Anderson (Dec. 2007). “Machine Ethics: Creating an Ethical Intelligent Agent”. In: *AI Magazine* 28.4, p. 15. DOI: 10.1609/aimag.v28i4.2065. URL: https://ojs.aaai.org/index.php/aimagazine/article/view/2065.

Anderson, Michael, Susan Leigh Anderson, and Chris Armen (2005). “MedEthEx: Toward a Medical Ethics Advisor.” In: *AAAI Fall Symposium: Caring Machines*, pp. 9–16.

Arkin, Ronald C, Patrick Ulam, and Brittany Duncan (2009). *An ethical governor for constraining lethal action in an autonomous system*. Tech. rep. Georgia Institute of Technology.
Arkin, Ronald C. (1990). "Integrating behavioral, perceptual, and world knowledge in reactive navigation". In: *Robotics and Autonomous Systems* 6.1. Designing Autonomous Agents, pp. 105-122. ISSN: 0921-8890. DOI: https://doi.org/10.1016/S0921-8890(05)80031-4. URL: https://www.sciencedirect.com/science/article/pii/S0921889005800314.

Arkin, Ronald C. and Patrick Ulam (2009). "An ethical adaptor: Behavioral modification derived from moral emotions". In: *2009 IEEE International Symposium on Computational Intelligence in Robotics and Automation - (CIRA)*, pp. 381-387. DOI: 10.1109/CIRA.2009.5423177.

Arkin, Ronald Craig, Patrick Ulam, and Alan R. Wagner (2012). "Moral Decision Making in Autonomous Systems: Enforcement, Moral Emotions, Dignity, Trust, and Deception". In: *Proceedings of the IEEE* 100.3, pp. 571-589. DOI: 10.1109/JPROC.2011.2173265.

Awad, Edmond, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan (Aug. 2018). "The Moral Machine experiment". In: *Nature* 563.7729, pp. 59-64. ISSN: 1476-4687. DOI: 10.1038/s41586-018-0637-6. URL: https://doi.org/10.1038/s41586-018-0637-6.

Bahouth, George, Jill Graygo, Kennerly Digges, Carl Schulman, and Peter Baur (2014). "The Benefits and Tradeoffs for Varied High-Severity Injury Risk Thresholds for Advanced Automatic Crash Notification Systems". In: *Traffic Injury Prevention* 15.sup1. PMID: 25307378, S134-S140. DOI: 10.1080/15389588.2014.936011. URL: https://doi.org/10.1080/15389588.2014.936011.

Barbier, Mathieu, Christian Laugier, Olivier Simonin, and Javier Ibañez-Guzmán (Nov. 2018). "Probabilistic Decision-Making at Road Intersections: Formulation and Quantitative Evaluation". In: *ICARCV 2018 - 15th International Conference on Control, Automation, Robotics and Vision*. Singapour, Singapore, pp. 1-8. URL: https://hal.inria.fr/hal-01940392.

Barredo Arrieta, Alejandro, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera (2020). "Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI". In: *Information Fusion* 58, pp. 82-115. ISSN: 1566-2535. DOI: https://doi.org/10.1016/j.inffus.2019.12.012. URL: https://www.sciencedirect.com/science/article/pii/S1566253519308103.

Barry, Keith (May 22, 2019). *Tesla's Updated Navigate on Autopilot Requires Significant Driver Intervention*. Consumer Reports. URL: https://www.consumerreports.org/autonomous-driving/tesla-navigate-on-autopilot-automatic-lane-change-requires-significant-driver-intervention/ (visited on 03/04/2021).

Batz, T., K. Watson, and J. Beyerer (June 2009). "Recognition of dangerous situations within a cooperative group of vehicles". In: *2009 IEEE Intelligent Vehicles Symposium*, pp. 907-912. DOI: 10.1109/IVS.2009.5164400.
BBC (Sept. 12, 2017). *Tesla Autopilot 'partly to blame' for crash*. British Broadcast Company.
URL: https://www.bbc.com/news/technology-41242884 (visited on 03/01/2021).

Bellman, Richard Ernst (1978). *An Introduction to Artificial Intelligence: Can Computers Think?*
Boyd & Fraser Publishing Company.

Bensimhoun, Michael (2009). "N-dimensional cumulative function, and other useful facts about gaussians and normal densities". In: *Jerusalem, Israel, Tech. Rep*, pp. 1–8.

Berrada, Jaâfar, Ilyes Mouhoubi, and Zoi Christoforou (2020). “Factors of successful implementation and diffusion of services based on autonomous vehicles: users’ acceptance and operators’ profitability”. In: *Research in Transportation Economics* 83. Thredbo 16 conference, p. 100902. ISSN: 0739-8859. DOI: https://doi.org/10.1016/j.retrec.2020.100902. URL: https://www.sciencedirect.com/science/article/pii/S0739885920301001.

Berreby, Fiona, Gauvain Bourgne, and Jean-Gabriel Ganascia (2015). “Modelling moral reasoning and ethical responsibility with logic programming”. In: *Logic for programming, artificial intelligence, and reasoning*. Springer, pp. 532–548.

Blanco, Myra, Jon Atwood, Sheldon M Russell, Tammy Trimble, Julie A McClafferty, and Miguel A Perez (2016). *Automated vehicle crash rate comparison using naturalistic data*. Tech. rep. Virginia Tech Transportation Institute. URL: https://vtechworks.lib.vt.edu/handle/10919/64420 (visited on 03/08/2021).

Bonnefon, Jean-François, David Černy, John Danaher, Nathalie Devillier, Veronica Johansson, Tatiana Kovacikova, Marieke Martens, Milos Mladenovic, Paula Palade, Nick Reed, et al. (2020). *Ethics of Connected and Automated Vehicles: Recommendations on road safety, privacy, fairness, explainability and responsibility*. Tech. rep. European Commission - Expert Group on Ethics of Connected and Automated Vehicles. DOI: 10.2777/035239. URL: https://ec.europa.eu/info/news/new-recommendations-for-a-safe-and-ethical-transition-towards-driverless-mobility-2020-sep-18_en (visited on 03/08/2021).

Bonnemains, Vincent, Claire Saurel, and Catherine Tessier (2018). “Embedded ethics: some technical and ethical challenges”. In: *Ethics and Information Technology* 20.1, pp. 41–58.

Bonnin, S., F. Kummert, and J. Schmüderich (Sept. 2012). “A Generic Concept of a System for Predicting Driving Behaviors”. In: *2012 15th International IEEE Conference on Intelligent Transportation Systems*, pp. 1803–1808. DOI: 10.1109/ITSC.2012.6338695.

Boudette, Neal E. (July 17, 2019). *Despite High Hopes, Self-Driving Cars Are 'Way in the Future'.*
The New York Times. URL: https://www.nytimes.com/2019/07/17/business/self-driving-autonomous-cars.html (visited on 03/08/2021).

— (Mar. 23, 2021). *Tesla's Autopilot Technology Faces Fresh Scrutiny*. The New York Times.
URL: https://www.nytimes.com/2021/03/23/business/teslas-autopilot-safety-investigations.html?smid themselveshare (visited on 03/23/2021).
Boyd, Stephen and Lieven Vandenberghe (2004). *Convex Optimization*. USA: Cambridge University Press. ISBN: 0521833787.

Brach, Raymond M and R Matthew Brach (2011). *Vehicle accident analysis and reconstruction methods*. 2nd. SAE International.

Brännström, M., E. Coelingh, and J. Sjöberg (2010). “Model-Based Threat Assessment for Avoiding Arbitrary Vehicle Collisions”. In: *IEEE Transactions on Intelligent Transportation Systems* 11.3, pp. 658–669. DOI: 10.1109/TITS.2010.2048314.

Brauckmann, Michael E, Christian Goerick, Jürgen Gross, and Thomas Zielke (1994). “Towards all around automatic visual obstacle sensing for cars”. In: *Proceedings of the Intelligent Vehicles’ 94 Symposium*. IEEE, pp. 79–84.

Brechtel, Sebastian, Tobias Gindele, and Rüdiger Dillmann (2011). “Probabilistic MDP-behavior planning for cars”. In: *2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC)*. IEEE, pp. 1537–1542.

— (2013). “Solving Continuous POMDPs: Value Iteration with Incremental Learning of an Efficient Space Representation”. In: *Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28*. ICML’13. Atlanta, GA, USA: JMLR.org, III-370–III-378.

— (2014). “Probabilistic decision-making under uncertainty for autonomous driving using continuous POMDPs”. In: *17th International IEEE Conference on Intelligent Transportation Systems (ITSC)*. IEEE, pp. 392–399.

Breuer, Joerg J (1998). “Analysis of driver-vehicle-interactions in an evasive manoeuvre-results of moose test’sudies”. In: *Proc. 16th ESV Conf., Paper*. 98-S2. NHTSA.

Brink, David (2018). *Mill’s Moral and Political Philosophy*. Stanford University. URL: https://plato.stanford.edu/entries/mill-moral-political/ (visited on 03/31/2021).

Broadhurst, A., S. Baker, and T. Kanade (2005). “Monte Carlo road safety reasoning”. In: *IEEE Proceedings. Intelligent Vehicles Symposium, 2005*. Pp. 319–324. DOI: 10.1109/IVS.2005.1505122.

Broggi, A., P. Cerri, S. Debattisti, M. C. Laghi, P. Medici, D. Molinari, M. Panciroli, and A. Prioletti (2015). “PROUD—Public Road Urban Driverless-Car Test”. In: *IEEE Transactions on Intelligent Transportation Systems* 16.6, pp. 3508–3519. DOI: 10.1109/TITS.2015.2477556.

Broggi, Alberto, Massimo Bertozzi, Alessandra Fascioli, C Guarino Lo Bianco, and Aurelio Piazzi (1999). “The ARGO autonomous vehicle’s vision and control systems”. In: *International Journal of Intelligent Control and Systems* 3.4, pp. 409–441.

Brooks, Rodney A (1986). “A robust layered control system for a mobile robot”. In: *IEEE journal on robotics and automation* 2.1, pp. 14–23.
— (1991). "Intelligence without reason". In: *Artificial intelligence: critical concepts* 3, pp. 107–63.

— (1989). "A Robot that Walks; Emergent Behaviors from a Carefully Evolved Network". In: *Neural Computation* 1.2, pp. 253–262. DOI: 10.1162/neco.1989.1.2.253. eprint: https://doi.org/10.1162/neco.1989.1.2.253. URL: https://doi.org/10.1162/neco.1989.1.2.253.

Broome, John (2017). "Equality". In: *Weighing Goods*. John Wiley & Sons, Ltd. Chap. 9, pp. 174–201. ISBN: 9781119451266. DOI: https://doi.org/10.1002/9781119451266.ch9. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/9781119451266.ch9.

Cho, K., T. Ha, G. Lee, and S. Oh (2019). "Deep Predictive Autonomous Driving Using Multi-Agent Joint Trajectory Prediction and Traffic Rules". In: *2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)*, pp. 2076–2081. DOI: 10.1109/IROS40897.2019.8967708.

Chokshi, Niraj (Feb. 25, 2020). *Tesla Autopilot System Found Probably at Fault in 2018 Crash*. The New York Times. URL: https://www.nytimes.com/2020/02/25/business/tesla-autopilot-ntsb.html (visited on 03/01/2021).

Christman, John (2008). *Autonomy in moral and political philosophy*. Tech. rep. URL: https://plato.stanford.edu/entries/autonomy-moral/.

Claussmann, Laurene (2019). "Motion planning for autonomous highway driving: a unified architecture for decision-maker and trajectory generator". PhD thesis. Université Paris-Saclay (ComUE).

Claussmann, Lauréne, Marie O'Brien, Sébastien Glaser, Homayoun Najjaran, and Dominique Gruyer (2018). "Multi-Criteria Decision Making for Autonomous Vehicles using Fuzzy Dempster-Shafer Reasoning". In: *2018 IEEE Intelligent Vehicles Symposium (IV)*, pp. 2195–2202. DOI: 10.1109/IVS.2018.8500451.

Commission, European (2018). *How is your country doing? Road safety in 2018*. Tech. rep. European Union. DOI: 10.2832/39063. URL: https://op.europa.eu/en/publication-detail/-/publication/043ee22b-643b-11eb-aeb5-01aa75ed71a1 (visited on 03/20/2021).

Coscia, Pasquale, Francesco Castaldo, Francesco A.N. Palmieri, Alexandre Alahi, Silvio Savarese, and Lamberto Ballan (2018). "Long-term path prediction in urban scenarios using circular distributions". In: *Image and Vision Computing* 69, pp. 81–91. ISSN: 0262-8856. DOI: https://doi.org/10.1016/j.imavis.2017.11.006. URL: https://www.sciencedirect.com/science/article/pii/S0262885617301853.

Daimler (2016). *The PROMETHEUS project launched in 1986: Pioneering autonomous driving*. URL: https://media.daimler.com/marsMediaSite/en/instance/ko.xhtml?oid=13744534&relId=1001&fromOid=13744534&resultInfoId=172&viewType=
*Policy and Practice* 77, pp. 167–181. ISSN: 0965-8564. DOI: https://doi.org/10.1016/j.tra.2015.04.003. URL: https://www.sciencedirect.com/science/article/pii/S0965856415000804.

Fikes, Richard E and Nils J Nilsson (1971). "STRIPS: A new approach to the application of theorem proving to problem solving". In: *Artificial intelligence* 2.3-4, pp. 189–208.

Fildes, B., M. Keall, N. Bos, A. Lie, Y. Page, C. Pastor, L. Pennisi, M. Rizzi, P. Thomas, and C. Tingvall (2015). "Effectiveness of low speed autonomous emergency braking in real-world rear-end crashes". In: *Accident Analysis & Prevention* 81, pp. 24–29. ISSN: 0001-4575. DOI: https://doi.org/10.1016/j.aap.2015.03.029. URL: https://www.sciencedirect.com/science/article/pii/S0001457515001116.

Firby, Robert James (1989). "Adaptive execution in complex dynamic worlds". PhD thesis. Yale University.

Firl, Jonas and Quan Tran (2011). "Probabilistic Maneuver Prediction in Traffic Scenarios". In: *ECMR*, pp. 89–94.

Fisac, J. F., E. Bronstein, E. Stefansson, D. Sadigh, S. S. Sastry, and A. D. Dragan (2019). "Hierarchical Game-Theoretic Planning for Autonomous Vehicles". In: *2019 International Conference on Robotics and Automation (ICRA)*, pp. 9590–9596. DOI: 10.1109/ICRA.2019.8794007.

Galceran, Enric, Alexander G Cunningham, Ryan M Eustice, and Edwin Olson (2017). "Multi-policy decision-making for autonomous driving via changepoint-based behavior prediction: Theory and experiment". In: *Autonomous Robots* 41.6, pp. 1367–1382.

Garcia, Frédérick and Emmanuel Rachelson (2013). "Markov Decision Processes". In: *Markov Decision Processes in Artificial Intelligence*. John Wiley & Sons, Ltd. Chap. 1, pp. 1–38. ISBN: 9781118557426. DOI: https://doi.org/10.1002/9781118557426.ch1. URL: https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118557426.ch1.

Gat, Erann (1991). "Reliable Goal-Directed Reactive Control of Autonomous Mobile Robots". UMI Order No. GAX91-23728. PhD thesis. USA.

— (1998). "Three-Layer Architectures". In: *Artificial Intelligence and Mobile Robots: Case Studies of Successful Robot Systems*. Cambridge, MA, USA: MIT Press, pp. 195–210. ISBN: 0262611376.

Gauthier, J.P., H. Hammouri, and S. Othman (1992). "A simple observer for nonlinear systems applications to bioreactors". In: *IEEE Transactions on Automatic Control* 37.6, pp. 875–880. DOI: 10.1109/9.256352.

Gerdes, J. Christian and Sarah M. Thornton (2016). "Implementable Ethics for Autonomous Vehicles". In: *Autonomous Driving: Technical, Legal and Social Aspects*. Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 87–102. ISBN: 978-3-662-48847-8. DOI: 10.1007/978-3-662-48847-8_5. URL: https://doi.org/10.1007/978-3-662-48847-8_5.
Gindele, T., S. Brechtel, and R. Dillmann (July 2010). "A probabilistic model for estimating driver behaviors and vehicle trajectories in traffic environments". In: *13th International IEEE Conference on Intelligent Transportation Systems*, pp. 1625-1631. DOI: 10.1109/ITSC.2010.5625262.

— (Oct. 2013). "Learning context sensitive behavior models from observations for predicting traffic situations". In: *16th International IEEE Conference on Intelligent Transportation Systems (ITSC 2013)*, pp. 1764-1771. DOI: 10.1109/ITSC.2013.6728484.

Gips, James (1995). "Towards the Ethical Robot". In: *Android Epistemology*. Cambridge, MA, USA: MIT Press, pp. 243-252. ISBN: 0262061848.

Giralt, Georges, Raja Chatila, and Marc Vaisset (1990). "An Integrated Navigation and Motion Control System for Autonomous Multisensory Mobile Robots". In: *Autonomous Robot Vehicles*. New York, NY: Springer New York, pp. 420-443. ISBN: 978-1-4613-8997-2. DOI: 10.1007/978-1-4613-8997-2_31. URL: https://doi.org/10.1007/978-1-4613-8997-2_31.

Godthelp, Hans (1986). "Vehicle Control During Curve Driving". In: *Human Factors* 28.2. PMID: 3733107, pp. 211-221. DOI: 10.1177/001872088602800209. URL: https://doi.org/10.1177/001872088602800209.

González, David, Joshué Pérez, Vicente Milanés, and Fawzi Nashashibi (2016). "A Review of Motion Planning Techniques for Automated Vehicles". In: *IEEE Transactions on Intelligent Transportation Systems* 17.4, pp. 1135-1145. DOI: 10.1109/TITS.2015.2498841.

Goodall, Noah J (2014). "Ethical decision making during automated vehicle crashes". In: *Transportation Research Record* 2424.1, pp. 58-65.

Gottschalk, S., M. C. Lin, and D. Manocha (1996). "OBBTree: A Hierarchical Structure for Rapid Interference Detection". In: *Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques*. SIGGRAPH '96. New York, NY, USA: Association for Computing Machinery, pp. 171-180. ISBN: 0897917464. DOI: 10.1145/237170.237244. URL: https://doi.org/10.1145/237170.237244.

Hajek, Alan (2019). *Interpretations of Probability*. Tech. rep. URL: https://plato.stanford.edu/archives/win2012/entries/probability-interpret/.

Harsanyi, John C and Reinhard Selten (1992). *A general theory of equilibrium selection in games*. 2nd. The MIT Press. ISBN: 978-0-262-08173-3.

Harsanyi, John C. (1967). "Games with Incomplete Information Played by "Bayesian" Players, I-III. Part I. The Basic Model". In: *Management Science* 14.3, pp. 159-182. ISSN: 00251909, 15265501. URL: http://www.jstor.org/stable/2628393.

— (1968a). "Games with Incomplete Information Played by 'Bayesian' Players, Part III. The Basic Probability Distribution of the Game". In: *Management Science* 14.7, pp. 486-502. DOI: 10.1287/mnsc.14.7.486. URL: https://doi.org/10.1287/mnsc.14.7.486.
— (1968b). "Games with Incomplete Information Played by "Bayesian" Players, I-III. Part II. Bayesian Equilibrium Points". In: *Management Science* 14.5, pp. 320–334. ISSN: 00251909, 15265501. URL: http://www.jstor.org/stable/2628673.

— (1976). "Advances in Understanding Rational Behavior". In: *Essays on Ethics, Social Behavior, and Scientific Explanation*. Dordrecht: Springer Netherlands, pp. 89–117. ISBN: 978-94-010-9327-9. DOI: 10.1007/978-94-010-9327-9_6. URL: https://doi.org/10.1007/978-94-010-9327-9_6.

Hartley, R. and F. Pipitone (1991). "Experiments with the subsumption architecture". In: *Proceedings. 1991 IEEE International Conference on Robotics and Automation*, 1652–1658 vol.2. DOI: 10.1109/ROBOT.1991.131856.

Hoffmann, Gabriel M., Claire J. Tomlin, Michael Montemerlo, and Sebastian Thrun (2007). "Autonomous Automobile Trajectory Tracking for Off-Road Driving: Controller Design, Experimental Validation and Racing". In: *2007 American Control Conference*, pp. 2296–2301. DOI: 10.1109/ACC.2007.4282788.

Houenou, A., P. Bonnifait, V. Cherfaoui, and W. Yao (2013). "Vehicle trajectory prediction based on motion model and maneuver recognition". In: *2013 IEEE/RSJ International Conference on Intelligent Robots and Systems*, pp. 4363–4369. DOI: 10.1109/IROS.2013.6696982.

Hubmann, Constantin, Jens Schulz, Marvin Becker, Daniel Althoff, and Christoph Stiller (2018). "Automated driving in uncertain environments: Planning with interaction and uncertain maneuver prediction". In: *IEEE Transactions on Intelligent Vehicles* 3.1, pp. 5–17.

Hummel, Britta, Werner Thiemann, and Irina Lulcheva (2008). "Scene Understanding of Urban Road Intersections with Description Logic". In: *Logic and Probability for Scene Interpretation*. Ed. by Anthony G. Cohn, David C. Hogg, Ralf Möller, and Bernd Neumann. Dagstuhl Seminar Proceedings 08091. Dagstuhl, Germany: Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, Germany. URL: http://drops.dagstuhl.de/opus/volltexte/2008/1616.

Hursthouse, Rosalind and Glen Pettigrove (2016). *Stanford Encyclopedia of Philosophy: Virtue Ethics*. Stanford University. URL: https://plato.stanford.edu/entries/ethics-deontological/ (visited on 05/08/2021).

Hussain, Qinaat, Hanqin Feng, Raphael Grzebieta, Tom Brijs, and Jake Olivier (2019). "The relationship between impact speed and the probability of pedestrian fatality during a vehicle-pedestrian crash: a systematic review and meta-analysis". In: *Accident Analysis & Prevention* 129, pp. 241–249.

Jaritz, Maximilian, Raoul de Charette, Marin Toromanoff, Etienne Perot, and Fawzi Nashashibi (2018). "End-to-End Race Driving with Deep Reinforcement Learning". In: *2018 IEEE International Conference on Robotics and Automation (ICRA)*, pp. 2070–2075. DOI: 10.1109/ICRA.2018.8460934.