# How To Evaluate the Navigation of Autonomous Vehicles Around Pedestrians?

Maria Kabtoul, Manon Prédhumeau, Anne Spalanzani, Julie Dugdale,
Philippe Martinet

► To cite this version:

Maria Kabtoul, Manon Prédhumeau, Anne Spalanzani, Julie Dugdale, Philippe Martinet. How To Evaluate the Navigation of Autonomous Vehicles Around Pedestrians?. IEEE Transactions on Intelligent Transportation Systems, 2023, pp.1-11. 10.1109/TITS.2023.3323662. hal-04255479

HAL Id: hal-04255479

https://inria.hal.science/hal-04255479v1

Submitted on 6 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# How To Evaluate the Navigation of Autonomous Vehicles Around Pedestrians?

Maria Kabtoul<sup>1,3</sup>, Manon Prédhumeau<sup>1,2</sup>, Anne Spalanzani<sup>1</sup>, Julie Dugdale<sup>2</sup> and Philippe Martinet<sup>3</sup>

***Abstract***—The navigation of autonomous vehicles around pedestrians is a key challenge when driving in urban environments. It is essential to test the proposed navigation system using simulation before moving to real-life implementation and testing. Evaluating the performance of the system requires the design of a diverse set of tests which spans the targeted working scenarios and conditions. These tests can then undergo a process of evaluation using a set of adapted performance metrics. This work addresses the problem of performance evaluation for an autonomous vehicle in a shared space with pedestrians. The methodology for designing the test simulations is discussed. Moreover, a group of performance metrics is proposed to evaluate the different aspects of the navigation: the motion safety, the quality of the generated trajectory and the comfort of the pedestrians surrounding the vehicle. Furthermore, the success/fail criterion for each metric is discussed. The implementation of the proposed evaluation method is illustrated by evaluating the performance of a pre-designed proactive navigation system using a shared space crowd simulator under Robot Operating System (ROS).

***Index Terms***—Autonomous vehicles, navigation evaluation, shared spaces, simulation and testing

## I. INTRODUCTION

AUTONOMOUS vehicles (AVs) have the ability to transform urban lifestyle and reduce the carbon footprint of cities by optimizing travel times, reducing congestion and the overall requirement for vehicle ownership [1], [2], [3]. Furthermore, AVs are accessible equally to all demographics (families, elderly, infirm), and they can reduce human losses and injuries caused by traffic accidents [4]. However, these driverless vehicles cannot be fully integrated into our daily lives without the capability to navigate safely and efficiently around vulnerable road users. Developing a fully autonomous vehicle navigation system that can operate around pedestrians is an increasingly critical issue, especially with the growing appearance of the shared space concept in city planning [5]. This growing interest is motivated by the premise of more green and pedestrian-friendly cities where vehicles and pedestrians share public spaces in a safe and efficient way [4], [6]. However, building, testing and evaluating AVs navigation systems in shared spaces with pedestrians are highly challenging tasks, each task being a standalone multidisciplinary problem. These steps of the development process (Build → Test → Evaluate) are equally crucial before letting a potentially harmful system, such as an AV, operate in proximity with humans.

This work discusses the methodology of evaluating the performance of a pre-designed AV navigation system in a simulated environment. The navigation system evaluation process translates into ensuring that the system is functional, efficient and above all safe in all possible working scenarios. However, the evaluation in shared spaces is much more challenging than in any other structured environment. In a more structured environment, such as a highway, the structure of the space itself and the strict driving rules limit the number of possible use cases. On the contrary, the free and open nature of shared spaces creates a wide range of possible working scenarios to test and evaluate. Moreover, the concepts of efficiency, natural and socially compliant behavior are much more challenging to define in such a space.

Evaluating the navigation system in shared spaces with pedestrians requires a set of adapted performance measures. The metrics traditionally used for autonomous robots to validate the navigation are necessary but not sufficient. The validation process should consider the safety and comfort of pedestrians around the navigating system, and of the passengers inside the vehicle. This problem can be summarized in selecting and/or designing a set of performance metrics that, firstly, evaluate the system's motion safety in proximity with pedestrians. Secondly, the proposed set of metrics should evaluate the quality of the resulting AV motion in terms of traveled distance, traveled time and even nature of the generated trajectory. Moreover, the analysis of the AV's trajectory should assess the comfort and safety of the vehicle's passengers. Finally, the proposed metrics should be able to evaluate the AV's social compliance and comfort of the pedestrians around the vehicle. This reduces the evaluation process in shared spaces into three main categories: motion safety, trajectory quality, and pedestrian comfort. In this work, a set of adapted metrics is proposed to evaluate the performance of an AV around pedestrians in a simulated environment using only the spatio-temporal information for both the pedestrian and the AV. The proposed evaluation process and the designed set of metrics are applied to test the performance of a proactive navigation system using a crowd-based simulator under ROS.

The structure of this paper is as follows: Section II gives a background on the methods used in the literature for performance evaluation in autonomous navigation applications. Section III discusses the process of designing the set of test simulations. After establishing the test set, Section IV proceeds to the performance metrics required to evaluate the different aspects of the AV's performance around pedestrians. Furthermore, this section presents how each metric can be implemented and its evaluation criterion. The navigation sys-

<footnote n="1">¹Univ. Grenoble Alpes, Inria, Grenoble, France</footnote>

<footnote n="2">²Univ. Grenoble Alpes, LIG, Grenoble, France</footnote>

<footnote n="3">³Université Cote d'Azur, Inria, Sophia Antipolis, France</footnote>
tem under evaluation along with the simulation environment
are presented in Section V. Finally, the implementation of
the suggested evaluation process and the analysis of the main
simulation results are discussed in Section VI.

## II. RELATED WORK

The problem of AV navigation around pedestrians has only
started getting researchers attention in recent years. Therefore,
not many works in the literature arrived at the performance
validation step of an AV system around pedestrians. However,
works in robotics and transportation engineering have targeted
the issue of evaluating navigation systems around humans. To
conduct a thorough evaluation of the system, a combination of
different metrics is needed, quantifying different aspects of the
system. Therefore, we can classify the navigation performance
metrics into three main categories: motion safety, trajectory
quality and pedestrian comfort.

The Motion Safety of a navigation system is mostly evalu-
ated by reporting the collision rate, i.e. how many pedestrians
who interacted with the vehicle had an accident [7]. The near
missed rate may be used instead of the collision rate, by
counting the occurrences of very close approaches between
pedestrians and the vehicle [8]. A more continuous metric like
the minimum and mean distance between pedestrians and the
AV can be used [9]. Other research qualifies the motion safety
by analyzing the severity of potential collisions. This can be
measured by the distance or the temporal proximity to the
collision when the evasive action starts. For example, the Time
To Collision (TTC) before the vehicle starts reacting [10], [11],
[12] or the Post Encroachment Time (PET) corresponding to
the TTC at the minimal approach distance [11], [12] have been
used. The speed of the vehicle before it starts reacting can also
be used, as proposed by [11].

The **Trajectory Quality** can be defined by the smoothness
of the trajectory produced by the vehicle along the navigation,
as well as its efficiency. Moreover, the trajectory quality
can also translate to measuring the level of comfort of any
passengers along the AV's trajectory.

A smooth trajectory limits abrupt changes in direction and speed. Trajectory smoothness has been measured by the total AV acceleration along its trajectory in [8], by the number of AV accelerations and decelerations in [7], or by the intensity of acceleration and deceleration in [10]. However, these metrics are insufficient to evaluate navigation in an open space, where the AV can adapt its steering in addition to its speed. In order to more thoroughly evaluate the trajectory smoothness, a trajectory stability metric based on the turning angle, the path's length and duration, as well as the speed of the AV was proposed by [13]. A metric used for mobile robots, based on the bending energy and the smoothness of curvature of the robot's trajectory has also been reported in [9].

The most efficient, i.e. optimal, trajectory is a straight line
from the starting point to the goal, of minimal length, covered
in minimal time. To measure a trajectory's efficiency, the
trajectory length or the travel time to reach the destination
can be used [9], as in [7] and [8]. [7] also used a success rate
metric, reporting the percentage of simulations where the AV

reaches its goal within an arbitrary time limit. However, in
order to obtain comparable measures for different trajectories,
it is preferable to use relative distances and times, as in [11].
The AV's travelled trajectory can be compared to its optimal
trajectory, in order to obtain the delay and the excess distance
travelled.

Although passenger comfort is an important factor in the
acceptance and adoption of AVs, only a few studies have
addressed this point. To determine passenger comfort, studies
usually collect data through a survey of the passengers in the
vehicle. In simulation, only data from the vehicle trajectory
can be used to estimate passenger comfort. A metric including
the lateral acceleration of the AV can for example be used as
an indicator of passenger comfort, as shown by [14].

In addition to safety, the **Pedestrian Comfort** around the
vehicle is an important factor in the performance evaluation in
shared spaces. The conflict duration has been used as a comfort
metric by [11], based on the idea that the longer the conflict
with the vehicle, the less comfortable the interaction is for the
pedestrian. The minimal approach distance between the pedes-
trian and the vehicle can also be used, as in [10] and [11]. A
pedestrian discomfort metric has been proposed by [15]. This
metric reflects the average degree and frequency of pedestrian
velocity changes, due to avoidance maneuvers. Similarly, vari-
ations in walking speed and in pedestrian trajectory have been
used by [10], [11] and [12] to measure pedestrian comfort.
These metrics assume that changes in walking behavior to
avoid the car require effort from pedestrians, which may cause
discomfort. Following a similar idea, [10] has proposed using
the severity and complexity of the evasive action performed by
the pedestrian. The severity is the degree of speed change and
the complexity is related to whether the pedestrian performs
a change in speed, direction or a combination of both.

### III. TEST CASES DESIGN FOR PERFORMANCE
VALIDATION

The first step of the performance evaluation is the process
of designing the tests themselves. The interpretation of the
test results is only valid in the context of the considered
test cases. Therefore, the designed test set should be diverse,
including all the possible user cases and scenarios in order to
be sufficient to validate the performance. Firstly, we start by
identifying the set of testing environments $E = \{e_k\}_{1 \le k \le K}$.
Each test environment $e_k$ is defined by the static map of
the testing space that includes the grids occupied by static
obstacles, the free space and the semantics associated with
the occupied space. These semantics (exit/entry points, walls,
etc.) should drive the movement of the agents in the space.
Therefore, they can be used to evaluate the trajectory and
to estimate if it is socially acceptable and if it followed
the rules associated with the specific semantics. Secondly,
a set of test scenarios $S = \{s_j\}_{1 \le j \le J}$ is defined within
each environment. Each test scenario $s_j$ is defined by the
configuration of the interaction between the agents in the space
(pedestrians-vehicle and pedestrians-pedestrians). This is set
by defining the start and goal point of each agent in the space.

In each test scenario, a set of variable test parameters
should be identified. This set of parameters $P = \{p_m\}_{1 \le m \le M}$
represents the different testing conditions that can occur in the identified use cases of the system. This can include the space density or the speed and acceleration limits for example. Each of these parameters is assigned a range of possible values $F_m$ identified based on experience and prior knowledge:

$$ \mathbf{P} = \{p_m : p_m \in F_m, F_m \in \mathbb{R}\}_{1 \le m \le M} \quad (1) $$

Afterwards a discrete set of test values is selected for each parameter within its range of possible values. Let $L_m$ be the number of test values selected for the parameter $p_m$, then the set of test parameters is written as:

$$ \mathbf{P}_T = \{p_m : p_m \in \{x_{m1}, \dots, x_{mL_m}\} \subseteq F_m\}_{1 \le m \le M} \quad (2) $$

For example, if the parameter of interest is the crowd density in the space, the possible range of values can be assigned based on the observations in highly dense crowds, which can reach a limit of 9 *pedestrian/m²* in some dense gatherings [16]. The discrete set of test values can be chosen closer to the range of the expected normal crowd density ([0, 0.5] *pedestrian/m²*) [17]. The set of test configurations **C** is the union of all possible combinations of the test parameters values, which will have a total of $N_C$ configurations:

$$ N_C = \prod_{m=1}^{M} L_m \quad (3) $$

Finally, each test simulation with a parameter configuration $c_i \in \mathbf{C}$ is repeated for *i* times to ensure the convergence of the performance. Therefore, the total number of simulations required to validate the performance in the context of the designed tests is equal to:

$$ N_{tot} = i \times K \times J \times N_C \quad (4) $$

## IV. PERFORMANCE METRICS FOR NAVIGATION EVALUATION IN SHARED SPACES

### A. Motion Safety

The main indices used to validate the motion safety around pedestrians are derived from analyzing the collisions occurring in the simulations. A collision is detected in the simulation when an overlap occurs between the footprints of both the pedestrian and the vehicle. The footprint of a pedestrian in the 2D plane is considered circular with a radius $R_p$ whereas the footprint of the rectangular AV is approximated with the outer Löowner-John ellipse of the rectangle [18].

The **Collision Rate** is an important factor in estimating the safety of the navigation policy. A near-zero collision rate is required to validate the navigation. The collision rate in a set of simulations is calculated as:

$$ \text{Collision Rate} = \frac{\sum_{i} \text{Number Of Collisions in simulation } i}{\text{Number Of Simulations}} \quad (5) $$

However, when looking at the collision rate in a simulated world it is important to evaluate if this collision could actually occur in the real world. A number of non-cooperative purposeful pedestrian collisions can be detected in the simulator that resulted from a behavior on the pedestrian side and did not result from an error on the vehicle's side. Therefore, the rate

of such collisions does not evaluate the vehicle's navigation scheme. An example is shown in Figure 1a where the vehicle is very slow and a pedestrian walks into the side of the vehicle. This case is detected as a collision in the simulator whereas in real-life this is highly unlikely to happen. Such a situation would mean, for example, that a distracted person bumped into a slow vehicle which is not considered as an error in the navigation algorithm. An example of a **Realistic Collision** is shown in Figure 1b. To assess if a collision is realistic or not, a Velocity Obstacle based test is performed.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Two collisions detected at time $t_C$ and the state of the AV and the pedestrian in the previous $t_{CH}$ period: (a) Non-cooperative Purposeful Pedestrian Collision, (b) Realistic Collision</figcaption></figure>

The Velocity Obstacle (VO) is a well-known method in robotics used for motion planning. The method calculates the set of robot velocities that can lead to a collision with another moving obstacle, or what is referred to as the collision cone (Figure 2b). When planning a trajectory, the velocity command is selected outside the collision cone for obstacle avoidance [19]. The idea of the **VO-based Collision Analysis** is to examine the AV's velocity commands in a time period preceding the collision with a pedestrian. If the AV's velocity belongs to the pedestrian's collision cone, then the collision is considered "Realistic". Otherwise, the collision is considered "Non-cooperative Purposeful Pedestrian Collision" as the AV is moving away from the pedestrian and the pedestrian still collided with the AV. One main limitation of the VO method is that it assumes a circular shape in the 2D plane for both of the colliding objects. However, assuming a circular footprint for the rectangular AV would result in large errors. Therefore, the footprint of the vehicle is approximated with a minimum number of enclosing circles (Figure 2a) [20].

Algorithm 1 shows how the approximated footprint $C_{fp}$ of the AV with a pose $(x_o(t), y_o(t), \theta_V(t))$ is obtained in the local frame of the AV then in the global frame using the transformation matrix $R_L^G$:

$$ R_L^G(t) = \begin{bmatrix} \cos(\theta_V(t)) & -\sin(\theta_V(t)) & x_o(t) \\ \sin(\theta_V(t)) & \cos(\theta_V(t)) & y_o(t) \\ 0 & 0 & 1 \end{bmatrix} \quad (6) $$

Once the set of footprint circles is obtained, the VO-based analysis is applied to each circle. The collision is realistic if any of the circles yields a positive result (Algorithm 2).

### B. Trajectory Quality

Let $T_j = \{(x_j, y_j, t_j) : j \in [0, \dots, M]\}$ be the overall trajectory of the AV during the simulation. The quality of the
<figure><img src="image_3.png" /><figcaption>Fig. 2: (a) Circular approximation of the AV footprint (Algorithm 1), (b) VO method applied to one part of the AV footprint with radius $R_o$, and a pedestrian with circular footprint of a center $P$ and radius $R_p$</figcaption></figure>

**Algorithm 1** AV Footprint Circular Approximation

**Input:** AV's pose: $X_V(t) = (x_o(t), y_o(t))$, $\theta_V(t)$, AV's dimensions: Length $L$, Width $W$

**Output:** AV's footprint $C_{fp}(t) = \cup_{i \in \mathbb{N}_+} (c_i(t), R_i) \in \mathbb{R}^2 \times \mathbb{R}_+$

Set: $L_T \leftarrow L$, $i \leftarrow 0$

**while** $L_T > W$ **do**

$c_i^l \leftarrow (\frac{L_T-W}{2}, 0)$ ▷ Frontal circle (AV frame)

$c_{i+1}^l \leftarrow (-\frac{L_T-W}{2}, 0)$ ▷ Rear circle (AV frame)

$i \leftarrow i + 2$

$L_T \leftarrow L_T - W$

**end while**

$c_i \leftarrow (x_o(t), y_o(t))$ ▷ Middle circle (AV frame)

$C_{fp} \leftarrow \{(R_L^G(t)c_j^l(t), \frac{\sqrt{2}}{2}W) : 0 \le j \le i\} \triangleright \text{Transform to global frame}$

AV's trajectory refers to its cost (smoothness), its efficiency, and to the comfort of the AV's passengers along this trajectory.

1) **Cost:** The cost of the trajectory is estimated using two metrics: the **Path Cost** which is based on the maneuvers along a trajectory, and the **Dynamic Cost** which is a speed-based trajectory cost estimate.

The path cost measures the smoothness of the path by evaluating the average tangent of the path during the vehicle's maneuvers. This measure is defined using the formulas in [21], where we used a tangent variation cost rather than the angle variation cost defined in [21]. On the other hand, the dynamic cost measures the average instantaneous variation of the speed from the preferred driving speed $v_{pref}$. This preferred speed is equal to the driving speed in an obstacle-free space, which corresponds to the maximum allowed speed in a shared space $v_{pref} = v_{max}$. The dynamic cost is defined based on the energy consumption impact of this speed change along the trajectory, similar to the cost adapted in [22].

The equivalent path cost $E_P$, and dynamic cost $E_T$ are

**Algorithm 2** VO-Based Collision Analysis

**Input:** Time step: $t_k$, Collision time horizon: $t_{CH}$, Collision time: $t_C$

AV/pedestrian pose and velocity respectively: $\{X_V(t), V_V(t), X_P(t), V_P(t)\}: t \in [t_C - t_{CH}, t_C]$, AV's footprint: $C_{fp}$, Pedestrian's footprint radius: $R_p$

**Output:** Collision Type: $IsReal \in \{0, 1\}$

**for** $(c_i(t), R_i) \in C_{fp}$ **do** ▷ AV footprint circles

$t_s \leftarrow t_C - t_{CH}$

Center $\leftarrow c_i(t_s) - X_P(t_s)$

$D_C \leftarrow D(\text{Center}, R_i + R_p)$ ▷ Collision disk

**while** $t_s \le t_C$ **do**

**if** $t_s * (V_P(t_s) - V_V(t_s)) \in D_C$ **then**

**return** 1

**end if**

$t_s \leftarrow t_s + t_k$

**end while**

**end for**

**return** 0

evaluated as:

$$E_P = \frac{1}{M} \sum_{j=1}^{M} \left( \frac{y_j - y_{j-1}}{x_j - x_{j-1}} \right)^2 \quad (7)$$

$$E_T = \frac{1}{M} \sum_{j=0}^{M} \left( \frac{v_{pref} - v(j)}{v_{pref}} \right)^2 \quad (8)$$

$$v(j) = \sqrt{\dot{x}^2(j) + \dot{y}^2(j)} : v_{pref} = \max_{j \le M} v(j) \quad (9)$$

When evaluated in the path's Frenet frame fixed at the AV's starting point, $E_P \in \mathbb{R}_+$ increases when a path contains more frequent and sharp maneuvers ($E_P = 0$ for a straight path). Figure 3 shows some examples for the path cost corresponding to a U-turn path with a Quadratic function, and a lane change path with a Quintic function [23]. The limit accepted value for $E_P$ is selected based on the working scenario. For example, in the case of uni-directional road driving where the vehicle is not supposed to do any 180° turns, the limit value can be selected around 0.5 to ensure a low-cost lane change. On the other hand, this limit is set higher in scenarios where the vehicle is supposed to go on roundabouts or make u-turns. The dynamic cost $E_T \in [0, 1]$ increases when the trajectory consists of large instantaneous velocity variations and $E_T = 0$ for an optimal constant maximum velocity trajectory ($v = v_{pref}$). Therefore, smaller values of the dynamic cost $E_T \to 0$ is desired from a trajectory quality point of view. However, it is unlikely to operate with the maximum velocity while interacting with the pedestrians.

2) **Efficiency:** The efficiency of the trajectory can be expressed in terms of distance or time [22]. Firstly, the **Relative Traveled Distance** ($C_L$), also called sinuosity, measures the length of the traveled path relative to the shortest distance path
<figure><img src="image_4.png" /><figcaption>Fig. 3: Examples of path cost for U-turn and lane change</figcaption></figure>

between the start and goal points:

$$C_L = \frac{\sum_{i=0}^{M} D(X_{i+1}, X_i)}{D(X_M, X_0)} \qquad (10)$$

$$D(X_2, X_1) = \sqrt{(x_2 - x_1)^2 + (y_2 - y_1)^2} \qquad (11)$$

Secondly, the Relative Traveled Time (TTG) measures the traveled time relative to the time to reach the goal at the average speed:

$$TT_G = \frac{(t_M - t_0) \times v_{avg}}{D(X_M, X_0)} \qquad (12)$$

In efficient trajectories, the length of the traveled distance is close to the shortest path distance. This corresponds to the optimal path between the start point and the goal, in an obstacle free case $C_L \to 1$. Moreover, more efficient trajectories do not introduce large time delays, i.e. $TT_G \le 1$.

3) Passengers' Comfort: The AV's trajectory can further entail information on the comfort of the AV passengers, which is an important part of the trajectory quality evaluation. A recent case study shows that passengers comfort is mainly related to their experience of the centripetal force [14]. Passengers experienced more discomfort when passing small radius curves with high speed. Therefore, the average Centripetal Acceleration is used as a passengers' comfort measure:

$$a_C = \frac{1}{M} \sum_{i=0}^{M} v^2(i) \kappa(i) \qquad (13)$$

where $\kappa(i)$ is the curvature of the path at point $i$. The results of the study in [14] recommends different $a_C$ values depending on the driving speed range. For the case of shared space (speed $\le 20$ km/h), $a_C \le 1$ m/s² is recommended for a very comfortable drive and $a_C \le 1.75$ m/s² for an acceptable comfort level. Finally, similar to measuring the passenger comfort by the centripetal acceleration, the longitudinal acceleration and even the longitudinal jerk can have a role in the passenger comfort as well. Which is a point worth inspecting in future studies to examine the dependency of the comfort index on both the longitudinal and the lateral dynamics.

### C. Pedestrian Comfort

The first metric used to evaluate the pedestrian comfort during the interaction with the AV is the Discomfort Index $\bar{I}_{ucf}$ suggested in [15]. This index reflects the average degree and frequency of the linear velocity change a pedestrian experienced during the navigation. However, this index only considers variations in walking speed, and not variations in direction. The evaluation of pedestrian comfort

was therefore completed with an additional Directional Discomfort Index $\bar{I}_{ucf\theta}$. In a simulation defined at times $\{t_j : j \in [0, \dots, M]\}$, the two discomfort indices are evaluated as:

$$\bar{I}_{ucf} = \frac{\bar{y}_i}{\bar{h}_i} \times 100\% \qquad (14)$$

with:

$$\bar{h}_i = \frac{1}{M} \sum_{j=0}^{M} v_i^2(j) \qquad (16)$$

$$\bar{y}_i = \frac{1}{M} \sum_{j=0}^{M} (v_i(j) - \bar{g}_i)^2 \qquad (18)$$

$$\bar{g}_i = \frac{1}{M} \sum_{j=0}^{M} v_i(j) \qquad (20)$$

where $\bar{h}_i$ and $\bar{l}_i$ are the mean square velocity and the mean square orientation of agent $i$ respectively. $\bar{y}_i$ and $\bar{z}_i$ are the average of the squares of the deviations from the mean speed and the average of the squares of the deviations from the mean orientation of agent $i$ respectively. Finally, $\bar{g}_i$ and $\bar{f}_i$ are the average velocity and average orientation of agent $i$ over the simulation period. When $\bar{I}_{ucf}$ or $\bar{I}_{ucf\theta}$ is equal to 0%, this indicates that the pedestrian walks at his average speed or orientation throughout the simulation, which is very comfortable. On the contrary, higher values of one of these indices indicate higher level of discomfort.

The average $\bar{I}_{ucf}$ and $\bar{I}_{ucf\theta}$ are computed for pedestrians who interacted with the AV and for pedestrians who did not interact with the AV. A pedestrian has interacted with the AV if the AV was perceived by the pedestrian during the simulation, i.e. the AV was within the perception zone of 220° up to 10 metres or 360° up to 3.3 metres around the pedestrian [24]. This distinction helps to overcome differences in pedestrian discomfort due to the context (e.g. crowd density) and to identify discomfort specifically related the interactions with the AV. In more comfortable trajectories, the AV does not cause any change in pedestrians' walking speed or direction, i.e. the part of discomfort related to the AV is close to 0.

## V. THE NAVIGATION SYSTEM UNDER EVALUATION & THE SIMULATION ENVIRONMENT

### A. The proactive navigation system

The proactive navigation system under study is a local navigation system that works in combination with a higher level path planner by locally modifying the global path based on the perceived state of the dynamic space and the local interactions with the nearby pedestrians. The system relies on the concept of cooperation between the vehicle and the pedestrians. The
pedestrians dynamically react to the AV movements.

In order to test the navigation system, SPACiSS can be used to build realistic shared space scenes and simulate a large variety of pedestrian-vehicle interactions. The simulation model is stochastic, so each scenario can be simulated multiple times, to assess the robustness of the navigation system in a given environment. To perform such tests efficiently, the simulator is designed so that several simulations can be executed in parallel. Finally, all simulation data are recorded and used for real-time visualization of the scene. Once the simulation is complete, agents' trajectories can be analyzed to evaluate the performance of the navigation system.

## VI. SIMULATION RESULTS

### A. Designed Test Cases

To test and validate the proactive navigation around pedestrians, we selected one test environment $E = \{e_1\}$ ($K = 1$) that represents a shared space with no specifically assigned pedestrian or vehicle areas. This means that the space does not contain any roads, sidewalks, pedestrian crossings, traffic signals, etc. The space is bounded by two side barriers which can represent a parking lot or a shared business area like the shared space in Exhibition Road in London for example.

The navigation is tested in environment $e_1$ in $J = 7$ interaction scenarios: Frontal Crossing (F), Back Crossing (B), Frontal-Back Crossing (F-B), Lateral Crossing (Lat), Bi-Lateral Crossing (Lat-Bi), 45° Uni-Directional Crossing (45°), 45° Bi-Directional Crossing (45°-Bi). The AV's goal in all scenarios is to cross the shared space safely and efficiently, while ensuring pedestrians' and passengers' comfort. Examples of four interaction scenarios are shown in Figure 5.

<figure><img src="image_5.png" /><figcaption>Fig. 5: Pedestrian-vehicle interaction scenarios</figcaption></figure>

The variable test parameter set includes the space density and the space sparsity (i.e. the distribution of the pedestrians in the space). The density of the space $D$ is measured as the average number of pedestrians per $m^2$. The space sparsity is estimated with a Gini Index that measures the distribution of pedestrians among the interaction space partitions [36].

Let a simulation be defined at times $\{t_j : j \in [0, \dots, M]\}$, $A$ the area of the shared space in $m^2$ and $N(j)$ be the number of pedestrians in the space at time $t_j$. Then, the average pedestrian density and sparsity are measured as follows:

$$D = \frac{1}{M} \sum_{j=0}^{M} \frac{N(j)}{A} \quad (\text{pedestrian/m}^2) \quad (22)$$

$$GI = \frac{1}{M} \sum_{j=0}^{M} \sum_{i=1}^{N(j)} \frac{|1 - N_i(j)|}{N(j) + 1} \times 100\% \quad (23)$$

where the interaction space is first divided into a grid of $N(j)$ equal cells. Then, $N_i$ is the number of pedestrians in a cell $i$ of the grid. For example, if the distribution of the pedestrians in the space is homogeneous, then there is one pedestrian in each of the equal grid cells, yielding $GI = 0\%$. The rest of the parameters that are required to define the scenario are set to a constant value. The parameter values and the range of values for the variable test parameters are as follows:

* **Constants:** AV max speed 5.5 m/s, Pedestrian max speed 6.5 m/s,

* **Variables:** Pedestrian density $D \in [0.003, 0.56]$ ped/m$^2$, Pedestrian distribution (Sparsity) $GI \in [6, 34]\%$.

Every simulation is run 20 times with 8 different values for the pair $(D, GI)$ resulting in 2240 test simulations. Figure 6 shows examples of the resulting trajectories in the XY-Plane for the pedestrians and the AV in some of the testing scenarios.

<figure><img src="image_6.png" /><figcaption>Fig. 6: Examples of the resulting trajectories</figcaption></figure>

### B. Motion Safety

The collision rate and the analysis of the plausibility of each collision is performed in the test simulations set. The realistic collisions caused by the navigation system were analyzed. This was done after discarding the non-cooperative purposeful pedestrian collisions detected in the test simulations set. Table I shows the detected collision rate in each interaction scenario.

<table><caption>TABLE I: Collision rate values across the testing scenarios</caption><thead><tr><th>Type</th><th>F</th><th>B</th><th>F-B</th><th>Lat</th><th>Lat-Bi</th><th>45°</th><th>45°-Bi</th></tr></thead><tbody><tr><td>CR</td><td>1e<sup>-5</sup></td><td>0.16</td><td>0.13</td><td>0.025</td><td>0.2</td><td>0.62</td><td>0.32</td></tr></tbody></table>

A higher rate of realistic collisions was detected in the interactions including back crossing with pedestrians (B and F-B). This can be a result of not accounting for the pedestrian perception range in the navigation system [27]. The behavior of the pedestrians in a back crossing scenario was assumed to be similar to their behavior in frontal crossing. A higher collision rate was also detected in the 45° crossing scenarios (both
uni-directional and bi-directional). This can be a limitation in the pedestrian behavior prediction model used in the system and which did not include such an interaction scenario [26]. Overall, in the interaction scenarios an average of 0.2 realistic collisions per simulation was detected.

As a result of the motion safety test, the system requires further development to avoid the percentage of detected collisions. It is worth noting that the lowest collision rate was detected in the case of frontal and lateral crossing, which are the scenarios used to establish the pedestrian behavioral model used in the system under evaluation [26]. Therefore, the performance might be significantly improved by adapting the pedestrian death cooperation model to account for all the different interaction scenarios. Moreover, the previous system might be implemented in combination with an emergency braking system to take over in critical situations.

### C. Trajectory Quality and Efficiency

The vehicle trajectory in each simulation was qualified by analyzing its path and dynamic costs, the efficiency in terms of the traveled distance and the traveled time, and the passengers comfort using the centripetal acceleration. Table II shows the corresponding performance metric values to the set of testing simulations across all scenarios, density and sparsity levels.

<table><caption>TABLE II: Trajectory quality statistical results</caption><thead><tr><th>Metric</th><th>Mean</th><th>Max</th><th>75%</th><th>Std. Dev.</th><th>Success Criterion</th></tr></thead><tbody><tr><td>E<sub>P</sub></td><td><strong>0.14</strong></td><td>1.10</td><td><strong>0.16</strong></td><td>0.06</td><td>&le; 0.5</td></tr><tr><td>E<sub>T</sub></td><td><strong>0.13</strong></td><td><strong>0.23</strong></td><td><strong>0.17</strong></td><td>0.05</td><td>&rarr; 0</td></tr><tr><td>C<sub>L</sub></td><td><strong>1.12</strong></td><td>1.36</td><td><strong>1.13</strong></td><td>0.02</td><td>&rarr; 1</td></tr><tr><td>TT<sub>G</sub></td><td><strong>0.55</strong></td><td>2.14</td><td><strong>0.78</strong></td><td>0.35</td><td>&le; 1</td></tr><tr><td>a<sub>C</sub></td><td><strong>0.17</strong></td><td><strong>1.29</strong></td><td><strong>0.24</strong></td><td>0.17</td><td>&le; 1.75</td></tr></tbody></table>

Firstly, the navigation system resulted in low cost both in terms of the manoeuvring (path cost) and the linear velocity. The values of the dynamic cost $E_T$ are all within the desired low limit to maintain a smooth linear velocity variation. Although the maximum value of the path cost $E_P$ is a little higher than the desired limit, most of the values fall within a very low acceptable cost range. Secondly, the resulting trajectories are efficient in terms of the traveled distance. We can notice that $C_L$ has values close to 1, which means that the vehicle did not deviate very far from its optimal path. For the efficiency in terms of the traveled time, we can see that most of the test simulations did not have large time delays. Only a small portion had larger time delays with $TT_G$ reaching over double the max desired value. Finally, the values of the centripetal acceleration of the vehicle along its traveled trajectory was overall within the desired range to maintain a comfortable ride for any potential passengers.

### D. Pedestrian Comfort and Social Compliance

Figure 7 shows the average discomfort index $\bar{I}_{ucf}$ for pedestrians in each scenario. Mean values and standard deviations are presented for pedestrians who interacted and for those who did not interact with the AV.

<figure><img src="image_7.png" /><figcaption>Fig. 7: Average discomfort index $\bar{I}_{ucf}$ for interacting (green) and non-interacting (blue) pedestrians.</figcaption></figure>

The pedestrians who did not interact with the AV have an average $\bar{I}_{ucf} \in [1.9; 6.1]\%$ depending on the scenario, as shown in blue in Figure 7. Pedestrians who did interact with the AV have an average $\bar{I}_{ucf} \in [1.6; 7.6]\%$ depending on the scenario. The average discomfort in case of interaction with the AV is very close to the average discomfort without interaction with the AV.

Figure 8 shows the average directional discomfort index $\bar{I}_{ucf_\theta}$ for pedestrians in each scenario, for all parameter combinations. Pedestrians who did not interact with the AV have an average $\bar{I}_{ucf_\theta} \in [5.4; 89.6]\%$ depending on the scenario, while pedestrians who interacted with the AV have an $\bar{I}_{ucf_\theta} \in [8.6; 86.3]\%$. Again, the average discomfort in case of interaction with the AV is very close to the average discomfort without interaction with the AV in all scenarios.

<figure><img src="image_8.png" /><figcaption>Fig. 8: Average directional discomfort index $\bar{I}_{ucf_\theta}$ for interacting (green) and non-interacting (blue) pedestrians.</figcaption></figure>

With both metrics, pedestrian comfort varies according to the interaction scenario. In frontal and back interactions, pedestrians have a low $\bar{I}_{ucf}$, while in 45° and lateral interactions, the average $\bar{I}_{ucf}$ is higher. The $\bar{I}_{ucf}$ index only considers pedestrians' speed adaptations and not direction adaptations. Pedestrians mainly adapt their direction in frontal and back interactions with vehicles, and their speed in lateral interactions. A higher $\bar{I}_{ucf}$ in lateral interactions is therefore expected. Accordingly, in frontal, back and 45° interactions, pedestrians have a higher $\bar{I}_{ucf_\theta}$ than in lateral interactions. Lateral interactions are more comfortable for pedestrians in terms of variations in their walking direction as these interactions require little directional adaptation from the pedestrian. These results confirm the importance of using both metrics to properly assess pedestrian comfort in various scenarios.

The statistical results for the discomfort related to the AV (i.e. the difference of discomfort between interacting pedestrians and non-interacting pedestrians) are shown in Table III.

Note that in some scenarios, especially with $\bar{I}_{ucf_\theta}$, pedestrians who interacted with the AV have even slightly lower
<table><caption>TABLE III: Pedestrian comfort statistical results</caption><thead><tr><th>Metric</th><th>Mean</th><th>Max</th><th>75%</th><th>Std. Dev.</th><th>Success Criterion</th></tr></thead><tbody><tr><td>I<sup>int</sup><sub>ucf</sub> - I<sup>non int</sup><sub>ucf</sub> %</td><td><strong>0.26</strong></td><td>1.60</td><td><strong>0.01</strong></td><td>0.01</td><td>→ 0</td></tr><tr><td>I<sup>int</sup><sub>ucfθ</sub> - I<sup>non int</sup><sub>ucfθ</sub> %</td><td><strong>-0.92</strong></td><td>17.46</td><td><strong>0.01</strong></td><td>0.09</td><td>→ 0</td></tr></tbody></table>

discomfort than those who did not interact. This is due to the fact that sometimes the pedestrians stayed interacting with the AV for a long time, e.g. waiting for the AV to pass. They therefore had less discomfort due to interactions with other pedestrians than those who did not interact with the AV.

Overall, the extra discomfort caused by the AV is very low, with values close to 0. The AV trajectory is then considered as comfortable for pedestrians.

## VII. CONCLUSION

AV navigation in shared spaces introduces close interactions with pedestrians. Therefore, it is essential to test and evaluate the navigation system using simulation before allowing the AV to operate around pedestrians. This leads to an imperative need for an evaluation methodology. In this paper we formalized the evaluation process for an AV navigation system in a simulated shared space. The process starts with the design of the test set to include different targeted scenarios and working conditions. After running a sufficient number of simulated tests, the evaluation of the performance is done based on the spatio-temporal information for the recorded pedestrians and AV trajectories. We divided performance evaluation into three main categories: the AV's motion safety, the quality of the generated AV's trajectory and the comfort of the pedestrians around the AV during an interaction. A set of adapted metrics along with the evaluation criteria was introduced for each category.

Firstly, evaluating the motion safety using the detected collision rate allowed us to identify the challenging scenarios for the navigation system. Secondly, the AV's trajectory was evaluated using the suggested metrics based on the traveled distance, traveled time and the generated path and trajectory costs. While it is convenient to use the trajectory information to evaluate its quality, it is less straightforward to evaluate the comfort of the pedestrians or the passengers in simulation. We suggested evaluating the comfort by using the concept of interacting and non-interacting pedestrians with the AV. The discomfort levels for each type of pedestrians is evaluated. Then, the system was evaluated as comfortable if the interacting and not-interacting pedestrians had similar average levels of discomfort during the simulation. Meaning that the navigation system maintained the comfort of the pedestrians while interacting with them. The passengers' comfort, on the other hand, was evaluated based on the centripetal acceleration as part of the trajectory quality evaluation.

The suggested evaluation process was used to study the performance of a pre-designed proactive navigation system using a crowd simulator under ROS. We found that the proactive system produced good-quality trajectories that maintained an acceptable level of comfort for both pedestrians and

passengers. The system's safety can be improved by including more diverse interaction scenarios in the pedestrian behavioral model used in the navigation framework, or by integrating an emergency braking system in the framework.

Furthermore, the proposed metrics are relevant for the evaluation of a navigation system in normal scenes. However, the simulator and evaluation metrics would need to be fitted if the case study is a mass-panic, emergency evacuation, or stampede situation. In such scenes, pedestrians may have very different behaviour (trying to escape, acting irrationally, ignoring their environment, or being unable to move) and the crowd may reach a very high density.

Moreover, the work was focused on evaluating the performance of the AV around pedestrians and without the presence of other vehicles on the scene. In future works, the evaluation method can be generalized to a multi-vehicle scene. This requires a re-calibration of the metrics and a re-formulation to isolate the effect of the ego AV from the effect of other vehicles on pedestrians.

Finally, future works can also consider using the evaluation method for different simulated scenarios using multiple available simulators (such as [37] and [38]). The results of the evaluation method in simulation can also be compared with the results of real-life scenarios evaluation.

## ACKNOWLEDGMENT

This research is part of the HIANIC project, funded by the French National Research Agency (ANR-17-CE22-0010).

## REFERENCES

[1] F. Steck, V. Kolarova, F. Bahamonde-Birke, S. Trommer, and B. Lenz, "How autonomous driving may affect the value of travel time savings for commuting," *Transp. Res. Record*, vol. 2672, no. 46, pp. 11–20, 2018.

[2] M. Heilig, T. Hilgert, N. Mallig, M. Kagerbauer, and P. Vortisch, "Potentials of autonomous vehicles in a changing private transportation system – a case study in the Stuttgart region," *Transportation Research Procedia*, vol. 26, pp. 13–21, Dec. 2017.

[3] C. Pakusch, G. Stevens, A. Boden, and P. Bossauer, "Unintended effects of autonomous driving: A study on mobility preferences in the future," *Sustainability*, vol. 10, p. 2404, Jul. 2018.

[4] Russ College of Engineering and Technology, "The future of driving," Ohio University, Tech. Rep., Mar. 2021. [Online]. Available: https://onlinemasters.ohio.edu/blog/the-future-of-driving/

[5] S. Moody and S. Melia, "Shared space – research, policy and problems," *Proceedings of the ICE - Transport*, vol. 167, pp. 384–392, Nov. 2014.

[6] R. Skinner and N. Bidwell, "Making better places: Autonomous vehicles and future opportunities," WSP Parsons Brinckerhoff & Farrells, Tech. Rep., 2016.

[7] Y. Luo, P. Cai, A. Bera, D. Hsu, W. S. Lee, and D. Manocha, "PORCA: Modeling and Planning for Autonomous Driving Among Many Pedestrians," *IEEE Robotics and Automation Letters*, vol. 3, no. 4, pp. 3418–3425, Oct. 2018.

[8] H. Bai, S. Cai, N. Ye, D. Hsu, and W. S. Lee, "Intention-aware online POMDP planning for autonomous driving in a crowd," in *Proc. IEEE Int. Conf. Robot. Autom.*, Seattle, WA, USA, May 2015, pp. 454–460.

[9] N. D. Muñoz, J. A. Valencia, and N. Londoño, "Evaluation of navigation of an autonomous mobile robot," in *Proceedings of the 2007 Workshop on Performance Metrics for Intelligent Systems*, ser. PerMIS '07. New York, NY, USA: Association for Computing Machinery, 2007, p. 15–21.

[10] I. Kaparias, M. G. H. Bell, W. Dong, A. Sastriginata, A. Singh, X. Wang, and B. Mount, "Analysis of Pedestrian–Vehicle Traffic Conflicts in Street Designs with Elements of Shared Space," *Transp. Res. Record: J. Transp. Res. Board*, vol. 2393, no. 1, pp. 21–30, Jan. 2013.

[11] F. Pascucci, "A microsimulation based method to evaluate shared space performances," Ph.D. dissertation, Technische Universität Braunschweig, Braunschweig, Germany, 2020.