# Assessment of V2X Communications For Enhanced Vulnerable Road Users Safety

Mouna Karoui, Vincent Berg, Sylvie Mayrargue

► To cite this version:

Mouna Karoui, Vincent Berg, Sylvie Mayrargue. Assessment of V2X Communications For Enhanced Vulnerable Road Users Safety. 2022 IEEE 95th Vehicular Technology Conference (VTC2022-Spring), IEEE, Jun 2022, Helsinki, France. pp.1-5, 10.1109/VTC2022-Spring54318.2022.9860551.cea-04552415

HAL Id: cea-04552415

https://cea.hal.science/cea-04552415

Submitted on 22 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Assessment of V2X Communications For Enhanced
Vulnerable Road Users Safety

Mouna Karoui, Vincent Berg, Sylvie Mayrargue

CEA-Leti, Universite Grenoble Alpes, F-38000 Grenoble, France
{mouna.karoui, vincent.berg, sylvie.mayrargue}@cea.fr

***Abstract***—Vulnerable road users (VRUs) such as pedestrians and cyclists have been the focus of renewed efforts from the car industry on life-saving and crash-reducing solutions. In this paper, we propose an evaluation study with the goal of analyzing the performance of a VRU safety service using the most common V2X ad-hoc technologies (C-V2X (mode 4), IEEE 802.11p and IEEE 802.11bd). To this end, a canonical accident scenario between a vehicle and a cyclist in an urban intersection is modelled, and implemented using a full-stack simulation environment. In this road scene, V2X technologies are then compared and analyzed under Line-of-Sight (LOS) and Non-Line-Of-Sight (NLOS) channel conditions. Our findings illustrate the benefits of V2X technologies in limiting VRU accident risks under different channel conditions. Despite decreased performance in NLOS, V2X communications are still able to warn reliably the driver at least a few precious seconds before it is too late for the driver to take action.

***Index Terms***—C-V2X, IEEE 802.11p, IEEE 802.11bd, VRU safety, system-level simulation.

# I. INTRODUCTION

Pedestrians and cyclists, unprotected by the vehicle frame, are today's most vulnerable road users (VRUs). Since the 1970s, much effort has been devoted to developing vehicles that are not only safer for their occupants but also reduce the likelihood of fatal injuries to VRUs involved in a crash. Modern car safety performance assessment programs, such as Euro NCAP, also test how well cars protect VRUs with whom they might collide. More recently, Advanced Driver Assistance Systems (ADAS) actively detect VRUs thanks to sensors from multiple data sources, including automotive imaging, LiDAR, radar, image processing combined with an on-board processing unit. Additional inputs are possible from other sources separate from the primary vehicle, including from other vehicles (V2V), pedestrians (V2P), and infrastructure (V2I) through wireless communications. V2V, V2I and V2P are referred to as Vehicle-to-everything (V2X).

Cooperative Intelligent Transport Systems (C-ITS) provides such functionality by allowing VRUs to share their location and context information between cars, other road users, and the infrastructure. VRUs may signal their presence by means of a message called VRU Awareness Messages (VAM) as specified in [1]. Decentralized Environmental Notifications Message (DENM) are also used to warn vehicles of a potential collision risk [1]. Multiple standardization efforts have been dedicated to providing a flexible and comprehensive C-ITS network architecture. IEEE 802.11p, part of ITS-G5, is one of the most mature technologies that is already commercialized.

The evolution of this standard is considered by the new IEEE 802.11bd working group to improve its performance. Alternatively, PC5 mode 4 (C-V2X) is derived from the 4G cellular technology proposed by 3GPP. C-V2X does not require any infrastructure and also relies on a distributed scheduler, that autonomously allocates resources.

Many evaluation studies have already been proposed to investigate the performance of V2X technologies. In [2] and [3], the authors have evaluated and compared the packet delivery performance of IEEE 802.11p and C-V2X, but no particular interest in evaluating the effectiveness of V2X technologies in VRU safety has been considered. However, in [3], the authors have shown that C-V2X outperforms ITS-G5 when the traffic density is below 150 users/km². They also concluded that the performance of C-V2X deteriorates more severely than that of ITS-G5 when a high level of network congestion is reached. The authors of [4] studied ITS-G5 performance in the context of VRU safety services. Critical safety scenarios are assessed through V2V communication between two devices equipped with IEEE 802.11p to evaluate the performance of the data link as a function of the transmitted power. The study has been conducted to investigate the communication range. Results show that obstacles significantly affect communication performance, especially with regards to the detection range of the VRU. Similar results have been obtained in [5] where IEEE 802.11p performance is evaluated for urban intersections under non light-of-sight (NLOS) conditions. Results show that the width of the intersecting streets significantly affects communication performance as well as traffic density.

Most of the research contributions reported in recent literature deal with link-level assessment regardless of any concrete application. These studies have focused on a generic performance analysis, comparing different V2X radio access technologies without showing the impact of performance on the V2X service. In this paper, we propose to study different V2X technologies in a crash risk canonical scenario involving a VRU, with the aim of exploring the possible benefits of V2X technologies on VRU safety. More precisely, given different channel conditions in a representative urban scenario, we examine for each V2X technology the probability of packet reception in light of the last moment, when the driver can still avoid the accident.

Indeed, a hazardous situation where a cyclist, equipped with a V2X radio, is analyzed. A simulation environment of this pre-crash scenario is developed in an urban intersection with
Assessment of V2X Communications For Enhanced
Vulnerable Road Users Safety

Mouna Karoui, Vincent Berg, Sylvie Mayrargue

CEA-Leti, Universite Grenoble Alpes, F-38000 Grenoble, France
{mouna.karoui, vincent.berg, sylvie.mayrargue}@cea.fr

***Abstract***—Vulnerable road users (VRUs) such as pedestrians and cyclists have been the focus of renewed efforts from the car industry on life-saving and crash-reducing solutions. In this paper, we propose an evaluation study with the goal of analyzing the performance of a VRU safety service using the most common V2X ad-hoc technologies (C-V2X (mode 4), IEEE 802.11p and IEEE 802.11bd). To this end, a canonical accident scenario between a vehicle and a cyclist in an urban intersection is modelled, and implemented using a full-stack simulation environment. In this road scene, V2X technologies are then compared and analyzed under Line-of-Sight (LOS) and Non-Line-Of-Sight (NLOS) channel conditions. Our findings illustrate the benefits of V2X technologies in limiting VRU accident risks under different channel conditions. Despite decreased performance in NLOS, V2X communications are still able to warn reliably the driver at least a few precious seconds before it is too late for the driver to take action.

***Index Terms***—C-V2X, IEEE 802.11p, IEEE 802.11bd, VRU safety, system-level simulation.

# I. INTRODUCTION

Pedestrians and cyclists, unprotected by the vehicle frame, are today's most vulnerable road users (VRUs). Since the 1970s, much effort has been devoted to developing vehicles that are not only safer for their occupants but also reduce the likelihood of fatal injuries to VRUs involved in a crash. Modern car safety performance assessment programs, such as Euro NCAP, also test how well cars protect VRUs with whom they might collide. More recently, Advanced Driver Assistance Systems (ADAS) actively detect VRUs thanks to sensors from multiple data sources, including automotive imaging, LiDAR, radar, image processing combined with an on-board processing unit. Additional inputs are possible from other sources separate from the primary vehicle, including from other vehicles (V2V), pedestrians (V2P), and infrastructure (V2I) through wireless communications. V2V, V2I and V2P are referred to as Vehicle-to-everything (V2X).

Cooperative Intelligent Transport Systems (C-ITS) provides such functionality by allowing VRUs to share their location and context information between cars, other road users, and the infrastructure. VRUs may signal their presence by means of a message called VRU Awareness Messages (VAM) as specified in [1]. Decentralized Environmental Notifications Message (DENM) are also used to warn vehicles of a potential collision risk [1]. Multiple standardization efforts have been dedicated to providing a flexible and comprehensive C-ITS network architecture. IEEE 802.11p, part of ITS-G5, is one of the most mature technologies that is already commercialized.

The evolution of this standard is considered by the new IEEE 802.11bd working group to improve its performance. Alternatively, PC5 mode 4 (C-V2X) is derived from the 4G cellular technology proposed by 3GPP. C-V2X does not require any infrastructure and also relies on a distributed scheduler, that autonomously allocates resources.

Many evaluation studies have already been proposed to investigate the performance of V2X technologies. In [2] and [3], the authors have evaluated and compared the packet delivery performance of IEEE 802.11p and C-V2X, but no particular interest in evaluating the effectiveness of V2X technologies in VRU safety has been considered. However, in [3], the authors have shown that C-V2X outperforms ITS-G5 when the traffic density is below 150 users/km². They also concluded that the performance of C-V2X deteriorates more severely than that of ITS-G5 when a high level of network congestion is reached. The authors of [4] studied ITS-G5 performance in the context of VRU safety services. Critical safety scenarios are assessed through V2V communication between two devices equipped with IEEE 802.11p to evaluate the performance of the data link as a function of the transmitted power. The study has been conducted to investigate the communication range. Results show that obstacles significantly affect communication performance, especially with regards to the detection range of the VRU. Similar results have been obtained in [5] where IEEE 802.11p performance is evaluated for urban intersections under non light-of-sight (NLOS) conditions. Results show that the width of the intersecting streets significantly affects communication performance as well as traffic density.

Most of the research contributions reported in recent literature deal with link-level assessment regardless of any concrete application. These studies have focused on a generic performance analysis, comparing different V2X radio access technologies without showing the impact of performance on the V2X service. In this paper, we propose to study different V2X technologies in a crash risk canonical scenario involving a VRU, with the aim of exploring the possible benefits of V2X technologies on VRU safety. More precisely, given different channel conditions in a representative urban scenario, we examine for each V2X technology the probability of packet reception in light of the last moment, when the driver can still avoid the accident.

Indeed, a hazardous situation where a cyclist, equipped with a V2X radio, is analyzed. A simulation environment of this pre-crash scenario is developed in an urban intersection with
the aim to evaluate the effectiveness of these V2X technologies on reducing crash risks between VRU and vehicles under different channel propagation conditions. Such an approach has so far not been considered.

The paper is structured as follows. Section II briefly presents the three V2X technologies and highlights the main characteristics of every communication standard. Section III describes the conducted scenarios and the main results. Section IV concludes this paper and opens some perspectives for our future work.

## II. V2X TECHNOLOGY STANDARDS

In this section, we introduce three of the most commonly used V2X communication standards and compare the studied technologies. These technologies operate in the unlicensed frequency band of 5.9 GHz. Using these technologies, notifications message are directly sent from one node to another under a decentralized network architecture.

### A. IEEE 802.11p

IEEE 802.11p is an approved amendment to the IEEE 802.11 standard to add wireless access in vehicular environments. It is derived from the IEEE 802.11a [6]. The MAC layer of IEEE 802.11p is based on the Outside the Context of a Basic Service Set (OCB) operation mode, in which authentication, association, and data confidentiality services are disabled. It uses Enhanced Distributed Channel Access (EDCA), which is an improved version of Distributed Coordination Function (DCF). EDCA is based on Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA). When a node wants to transmit a message, it has to perform a medium sensing step. When the medium is idle for a period greater or equals to Arbitration Inter-Frame Space (AIFS), the node will start transmitting immediately. However, if the channel is busy, the node will defer the transmission by selecting a random back-off time.

### B. IEEE 802.11bd

IEEE 802.11bd is an evolution of IEEE 802.11p inherited from IEEE 802.11ac proposed by a new 802.11 task group. The physical layer of IEEE 802.11bd allows for 10 MHz or 20 MHz bandwidth channel and provides higher spectral efficiency Modulation Coding Scheme (MCS) including 256-QAM. IEEE 802.11bd also uses midambles, to improve performance in rapidly changing channel conditions. Furthermore, low-density parity-check (LDPC) coding is used instead of BCC [7] to improve sensitivity. Regarding the MAC layer, IEEE 802.11bd uses the same MAC as 802.11p described in the previous section.

### C. 3GPP C-V2X (PC5 mode 4 air interface)

3GPP C-V2X or PC5 mode 4 air interface is an evolution of the 3GPP cellular standard designed to enable direct communications between Users Equipment (UEs). Similar to the IEEE 802.11bd, C-V2X supports both 10 MHz and 20

MHz channel bandwidths. The physical layer uses single-carrier frequency-division multiple access (SC-FDMA) similar to the LTE uplink. To support autonomous resource scheduling, Sensing-Based Semi-Persistent Scheduling (SB-SPS) mechanism has been designed [8], i.e. the vehicles sense the medium and identify the best time-frequency resource combination for the transmission of the message. To sense the channel, each vehicle continuously monitors each subchannel to determine the interference level for every subframe. Each vehicle identify then its best resources and randomly selects transmission resources among them. The vehicle can reserve the same resources for a number of consecutive Reselection Counter (RC) packet transmissions. After each transmission, RC is decremented by one. When it is equal to zero, new resources have to be selected and reserved with probability (1 - P), with P ∈ [0, 0.8]. P is the probability for selecting the previous resource again. In our study, we assume that P equals 0 (i.e. all vehicles always select new resources when their Re-selection Counter goes down to 0).

## III. EVALUATION METHODOLOGY AND RESULTS

In this section, we introduce our evaluation methodology and describe the chosen scenario and system-level simulation assumptions. The obtained performance results are then analyzed.

### A. Evaluation assumptions and scenario description

According to Euro NCAP VRU assessment program [9], one of the most frequent accident situations occurs when a vehicle travels forwards towards a bicyclist crossing its path cycling from the farside. For this purpose, we consider the canonical accident scenario illustrated in Figure 1, where a cyclist unexpectedly crosses the path of an incoming vehicle. Even more specifically, we are interested in the way the warning information (upon detection of the cyclist) is distributed to the approaching vehicle, prior to the accident. In our study, we assume that VRUs and vehicles are both equipped with a V2X radio interface. In this paper, we analyze and compare the benefits of providing VRUs with the V2X access radio technologies introduced in Section II.

<figure><img src="image_1.png" /><figcaption>Fig. 1. Simulated canonical (pre-)accident scenario (See also Figure 3).</figcaption></figure>

Performance results are based on a system-level simulation consisting of two components: SUMO and NS3 simulators. A full protocol stack pattern of V2X communication technologies is modelled within NS3 network simulator. A real
the aim to evaluate the effectiveness of these V2X technologies on reducing crash risks between VRU and vehicles under different channel propagation conditions. Such an approach has so far not been considered.

The paper is structured as follows. Section II briefly presents the three V2X technologies and highlights the main characteristics of every communication standard. Section III describes the conducted scenarios and the main results. Section IV concludes this paper and opens some perspectives for our future work.

## II. V2X TECHNOLOGY STANDARDS

In this section, we introduce three of the most commonly used V2X communication standards and compare the studied technologies. These technologies operate in the unlicensed frequency band of 5.9 GHz. Using these technologies, notifications message are directly sent from one node to another under a decentralized network architecture.

### A. IEEE 802.11p

IEEE 802.11p is an approved amendment to the IEEE 802.11 standard to add wireless access in vehicular environments. It is derived from the IEEE 802.11a [6]. The MAC layer of IEEE 802.11p is based on the Outside the Context of a Basic Service Set (OCB) operation mode, in which authentication, association, and data confidentiality services are disabled. It uses Enhanced Distributed Channel Access (EDCA), which is an improved version of Distributed Coordination Function (DCF). EDCA is based on Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA). When a node wants to transmit a message, it has to perform a medium sensing step. When the medium is idle for a period greater or equals to Arbitration Inter-Frame Space (AIFS), the node will start transmitting immediately. However, if the channel is busy, the node will defer the transmission by selecting a random back-off time.

### B. IEEE 802.11bd

IEEE 802.11bd is an evolution of IEEE 802.11p inherited from IEEE 802.11ac proposed by a new 802.11 task group. The physical layer of IEEE 802.11bd allows for 10 MHz or 20 MHz bandwidth channel and provides higher spectral efficiency Modulation Coding Scheme (MCS) including 256-QAM. IEEE 802.11bd also uses midambles, to improve performance in rapidly changing channel conditions. Furthermore, low-density parity-check (LDPC) coding is used instead of BCC [7] to improve sensitivity. Regarding the MAC layer, IEEE 802.11bd uses the same MAC as 802.11p described in the previous section.

### C. 3GPP C-V2X (PC5 mode 4 air interface)

3GPP C-V2X or PC5 mode 4 air interface is an evolution of the 3GPP cellular standard designed to enable direct communications between Users Equipment (UEs). Similar to the IEEE 802.11bd, C-V2X supports both 10 MHz and 20

MHz channel bandwidths. The physical layer uses single-carrier frequency-division multiple access (SC-FDMA) similar to the LTE uplink. To support autonomous resource scheduling, Sensing-Based Semi-Persistent Scheduling (SB-SPS) mechanism has been designed [8], i.e. the vehicles sense the medium and identify the best time-frequency resource combination for the transmission of the message. To sense the channel, each vehicle continuously monitors each subchannel to determine the interference level for every subframe. Each vehicle identify then its best resources and randomly selects transmission resources among them. The vehicle can reserve the same resources for a number of consecutive Reselection Counter (RC) packet transmissions. After each transmission, RC is decremented by one. When it is equal to zero, new resources have to be selected and reserved with probability (1 - P), with P ∈ [0, 0.8]. P is the probability for selecting the previous resource again. In our study, we assume that P equals 0 (i.e. all vehicles always select new resources when their Re-selection Counter goes down to 0).

## III. EVALUATION METHODOLOGY AND RESULTS

In this section, we introduce our evaluation methodology and describe the chosen scenario and system-level simulation assumptions. The obtained performance results are then analyzed.

### A. Evaluation assumptions and scenario description

According to Euro NCAP VRU assessment program [9], one of the most frequent accident situations occurs when a vehicle travels forwards towards a bicyclist crossing its path cycling from the farside. For this purpose, we consider the canonical accident scenario illustrated in Figure 1, where a cyclist unexpectedly crosses the path of an incoming vehicle. Even more specifically, we are interested in the way the warning information (upon detection of the cyclist) is distributed to the approaching vehicle, prior to the accident. In our study, we assume that VRUs and vehicles are both equipped with a V2X radio interface. In this paper, we analyze and compare the benefits of providing VRUs with the V2X access radio technologies introduced in Section II.

<figure><img src="image_2.png" /><figcaption>Fig. 1. Simulated canonical (pre-)accident scenario (See also Figure 3).</figcaption></figure>

Performance results are based on a system-level simulation consisting of two components: SUMO and NS3 simulators. A full protocol stack pattern of V2X communication technologies is modelled within NS3 network simulator. A real
intersection map is imported into SUMO simulator, which is an open source, microscopic and continuous traffic simulation package designed to handle different road topologies [10]. As shown in Figure 1, we considered an urban intersection of two main streets (Quai du Commerce and Pont Robert Schuman) located in the city of Lyon, France, which forms an area of $400m \times 400m$. An urban medium density is considered where 63 nodes are simulated, where 30% of the nodes are cyclists and the rest of the nodes are light-duty vehicles. Vehicles can navigate on two lanes and both directions. As for the cyclists, they can either ride on a cycle lane or on the same lane as the vehicles. The simulated vehicles can reach a maximum speed of 30 km/h, while the cyclists 20 km/h.

In an initial simulation, we used SUMO to extract potential accident risks between simulated vehicles and cyclists based on our road intersection illustrated in Figure 1. Indeed, SUMO keeps track of the full and partial gaps between vehicles and cyclists. When these gaps are reduced below the "minGap" parameter (set at 2.5 m), a risk of accident is detected (See Figure 3). We then log the possible crash moment with the respective identifiers of a potential collider and a victim). Once an accident risk data is identified, the network simulation corresponding to this road scene is then launched and repeated 100 times, so as to ensure the accuracy of our statistical results. As illustrated in the sequence diagram in Figure 2, we suppose

<figure><img src="image_2.png" /><figcaption>Fig. 2. Typical message sequence between vehicles and VRUs in our V2X network architecture.</figcaption></figure>

that it is the vehicle's task to perform the risk assessment (i.e the vehicle predict its crossing with the cyclist and then estimate the level of threat) and therefore take an appropriate action (e.g. reduce speed in case of high risk or maintain his current speed in the opposite case) upon receiving a message from a VRU. The risk assessment process is outside the scope of this work. We also assume that the processing time due to this workflow is negligible.

<figure><img src="image_3.png" /><figcaption>Fig. 3. Timeline of the simulated canonical (pre-)accident.</figcaption></figure>

Following 3GPP specification [11], we assume that a VRU is capable of broadcasting awareness messages (a.k.a., VAM)

including its location and speed. Vehicles are also broadcasting awareness messages (a.k.a., CAM) between each others. Vehicles and cyclists can receive and decode these messages. The message transmission period, T, is assumed to be equal to 100 ms. The goal is for the vehicle to detect VRUs early enough before an accident occurs. To this end, we extended NS3 module introduced in [12] to evaluate C-V2X mode 4, 802.11p and 802.11bd under this canonical accident scenario. The three V2X technologies are configured in equivalent way. All technologies uses a dedicated channel of 10 MHz. A transmission power of 23 dBm is considered following 3GPP guidelines [13]. C-V2X is configured to support an MCS of 7 with 50 resource blocks configuration ensures a data rate of 6.2 Mbps including signalling overhead [14]. Similarly, 802.11p and 802.11bd are configured to support a data rate of 6 Mbps. This is to ensure a fair comparison between different technologies.

As for channel modelling, two propagation conditions are thus evaluated: Line-Of-Sight (LOS) and Non-line-of-sight (NLOS). LOS occurs when vehicles and VRUs drive in very open environments without large buildings and obstacles. LOS conditions are optimistic in an urban environment and give a best case performance scenario. NLOS occurs when communications are obstructed and provide more realistic performance estimations. For the first use case, we considered LOS channel conditions based on equation in [15]. For the second case, we used the WINNER+B1 channel model [15], which allows transitions between LOS and NLOS propagation conditions based on the calculation of the LOS probability as illustrated in [15].

<table><caption>TABLE I
SIMULATION PARAMETERS</caption><thead><tr><th colspan="2">Traffic mobility settings</th></tr></thead><tbody><tr><td>Number of V2X equipped nodes</td><td>63</td></tr><tr><td>Maximum vehicles speed (km/h)</td><td>30</td></tr><tr><td>Nominal deceleration (m/s<sup>2</sup>)</td><td>4.5</td></tr><tr><td>Maximum VRU speed (km/h)</td><td>20</td></tr><tr><td>Simulation length (s)</td><td>26</td></tr><tr><th colspan="2">Network communication settings</th></tr><tr><td>Tx power (dBm)</td><td>23</td></tr><tr><td>Message size (Bytes)</td><td>300</td></tr><tr><td>Channel bandwidth (MHz)</td><td>10</td></tr><tr><td>Gross data rate (Mbps)</td><td>≈ 6</td></tr><tr><td>Transmission period T (ms)</td><td>100</td></tr><tr><td>Channel model</td><td>LOS|WINNER+B1</td></tr><tr><td>Number of simulation trials</td><td>100</td></tr></tbody></table>

Table I summarizes the simulation scenario parameters, which we classify into two categories: mobility-related and network communication-related.

## B. Results and discussions

The intersection scenario is then used to evaluate the V2X communication performance considering two main angles: first, the overall statistical performance as a function of the propagation model is measured, then the benefits of the technology for VRU crash risks are analyzed. For the overall
intersection map is imported into SUMO simulator, which is an open source, microscopic and continuous traffic simulation package designed to handle different road topologies [10]. As shown in Figure 1, we considered an urban intersection of two main streets (Quai du Commerce and Pont Robert Schuman) located in the city of Lyon, France, which forms an area of $400m \times 400m$. An urban medium density is considered where 63 nodes are simulated, where 30% of the nodes are cyclists and the rest of the nodes are light-duty vehicles. Vehicles can navigate on two lanes and both directions. As for the cyclists, they can either ride on a cycle lane or on the same lane as the vehicles. The simulated vehicles can reach a maximum speed of 30 km/h, while the cyclists 20 km/h.

In an initial simulation, we used SUMO to extract potential accident risks between simulated vehicles and cyclists based on our road intersection illustrated in Figure 1. Indeed, SUMO keeps track of the full and partial gaps between vehicles and cyclists. When these gaps are reduced below the "minGap" parameter (set at 2.5 m), a risk of accident is detected (See Figure 3). We then log the possible crash moment with the respective identifiers of a potential collider and a victim). Once an accident risk data is identified, the network simulation corresponding to this road scene is then launched and repeated 100 times, so as to ensure the accuracy of our statistical results. As illustrated in the sequence diagram in Figure 2, we suppose

<figure><img src="image_3.png" /><figcaption>Fig. 2. Typical message sequence between vehicles and VRUs in our V2X network architecture.</figcaption></figure>

that it is the vehicle's task to perform the risk assessment (i.e the vehicle predict its crossing with the cyclist and then estimate the level of threat) and therefore take an appropriate action (e.g. reduce speed in case of high risk or maintain his current speed in the opposite case) upon receiving a message from a VRU. The risk assessment process is outside the scope of this work. We also assume that the processing time due to this workflow is negligible.

<figure><img src="image_4.png" /><figcaption>Fig. 3. Timeline of the simulated canonical (pre-)accident.</figcaption></figure>

Following 3GPP specification [11], we assume that a VRU is capable of broadcasting awareness messages (a.k.a., VAM)

including its location and speed. Vehicles are also broadcasting awareness messages (a.k.a., CAM) between each others. Vehicles and cyclists can receive and decode these messages. The message transmission period, T, is assumed to be equal to 100 ms. The goal is for the vehicle to detect VRUs early enough before an accident occurs. To this end, we extended NS3 module introduced in [12] to evaluate C-V2X mode 4, 802.11p and 802.11bd under this canonical accident scenario. The three V2X technologies are configured in equivalent way. All technologies uses a dedicated channel of 10 MHz. A transmission power of 23 dBm is considered following 3GPP guidelines [13]. C-V2X is configured to support an MCS of 7 with 50 resource blocks configuration ensures a data rate of 6.2 Mbps including signalling overhead [14]. Similarly, 802.11p and 802.11bd are configured to support a data rate of 6 Mbps. This is to ensure a fair comparison between different technologies.

As for channel modelling, two propagation conditions are thus evaluated: Line-Of-Sight (LOS) and Non-line-of-sight (NLOS). LOS occurs when vehicles and VRUs drive in very open environments without large buildings and obstacles. LOS conditions are optimistic in an urban environment and give a best case performance scenario. NLOS occurs when communications are obstructed and provide more realistic performance estimations. For the first use case, we considered LOS channel conditions based on equation in [15]. For the second case, we used the WINNER+B1 channel model [15], which allows transitions between LOS and NLOS propagation conditions based on the calculation of the LOS probability as illustrated in [15].

<table><caption>TABLE I
SIMULATION PARAMETERS</caption><tr><th colspan="2">Traffic mobility settings</th></tr><tr><td>Number of V2X equipped nodes</td><td>63</td></tr><tr><td>Maximum vehicles speed (km/h)</td><td>30</td></tr><tr><td>Nominal deceleration (m/s<sup>2</sup>)</td><td>4.5</td></tr><tr><td>Maximum VRU speed (km/h)</td><td>20</td></tr><tr><td>Simulation length (s)</td><td>26</td></tr><tr><th colspan="2">Network communication settings</th></tr><tr><td>Tx power (dBm)</td><td>23</td></tr><tr><td>Message size (Bytes)</td><td>300</td></tr><tr><td>Channel bandwidth (MHz)</td><td>10</td></tr><tr><td>Gross data rate (Mbps)</td><td>≈ 6</td></tr><tr><td>Transmission period T (ms)</td><td>100</td></tr><tr><td>Channel model</td><td>LOS|WINNER+B1</td></tr><tr><td>Number of simulation trials</td><td>100</td></tr></table>

Table I summarizes the simulation scenario parameters, which we classify into two categories: mobility-related and network communication-related.

## B. Results and discussions

The intersection scenario is then used to evaluate the V2X communication performance considering two main angles: first, the overall statistical performance as a function of the propagation model is measured, then the benefits of the technology for VRU crash risks are analyzed. For the overall
statistical performance, we computed the Packet Reception Ratio (PRR). It is calculated as the ratio between the number of packets received from neighbors in a specific geographical area and the total number of packets transmitted in the same geographical area. We used the baseline radius distance of 150 m as recommended by 3GPP in [11] for urban intersections. In Figure 4, we plotted the Cumulative Distribution Function

<figure><img src="image_4.png" /><figcaption>Fig. 4. CDF of PRR in presence of LOS and WINNER+B1 conditions.</figcaption></figure>

(CDF) of the PRR. Statistical results have been obtained by repeating the scene 100 times for each studied scenario. We notice that under LOS channel propagation conditions, 802.11bd outperforms 802.11p. For C-V2X, the performance is significantly lower. This is due to the SPS scheduling mechanism which, unlike the 802.11p/bd contention-based MAC scheme, is less responsive in terms of channel sensing. In C-V2X, a sensing window of 1 s is used for the selection and reservation of resources. Therefore, two nodes may choose the same time-frequency resource for a transmission, the collision cannot be avoided and cannot be detected by either party. Even worse, the collision episode may persist for several messaging intervals without the colliding node being aware of it.

However, all V2X technologies achieve an average PRR above 95% of reliability and meet 3GPP release 14 requirement [16].

However, when NLOS conditions are taken into account, the performance order is inverted and C-V2X outperforms 802.11bd and 802.11p. To better understand these results, we also plotted the Signal-to-Interference-and-Noise Ratio (SINR) probability distribution for both propagation conditions in Figure 5 along with SINR thresholds for which the physical layer of each technology is starting to perform for the selected set of parameters used in the simulated environment. SINR distribution is only dependent on the behavior of the V2X communication MAC layer as it rely on energy sensing to determine the appropriate time of transmission and all technologies have been simulated for the same bandwidth of

10 MHz. In Figure 5, we notice that the SINR for IEEE

<figure><img src="image_5.png" /><figcaption>Fig. 5. SINR distribution in presence of LOS and WINNER+B1 conditions.</figcaption></figure>

802.11p and IEEE 802.11bd are identical as expected since their MAC layers are identical. However, for C-V2X, the SINR spreads over much lower values than for the simulated IEEE 802.11 standards. This is a consequence of the Semi-Persistent Scheduling algorithm used by C-V2X. It reserves the selected resource for consecutive packets and is therefore less reactive to hidden nodes phenomenon. The interference that is generated is not compensated by the superior physical layer when LOS propagation is considered in this sparsely populated scenario. The SINR threshold is about 4 dB (resp. 7 dB) lower for the C-V2X physical layer than for IEEE 802.11bd (resp. IEEE 802.11p). When NLOS is considered, the SINR distribution is shifted towards lower levels for all standards. However, the shift has a larger impact for IEEE 802.11p and 802.11bd than for C-V2X.

To better understand a key element of our VRU scenario, we finally analyzed performance in a crash risk situation as shown in Figure 3. To this end, we processed the accident risk data previously logged by SUMO and then analyzed the packet reception performance over time from the beginning of the simulation to the time of the recorded crash risk between the vehicle (i.e. collider) and the cyclist (i.e. victim).

Figure 6 gives the packet reception probability over time of the VAM transmitted by the cyclist and received by the vehicle in a crash risk situation for LOS and WINNER+B1 channel propagation conditions. Two important event instants are identified in Figure 6: the crash time (i.e. red dashed line) that occurs at time 24s and the crash avoidance time (CAT) threshold (i.e. the black dashed line). The CAT threshold represents the moment after which the driver cannot take late actions to avoid the accident. According to our particular simulation timeline, CAT is set to time t = 20.65 seconds. This value is calculated from the crash time, which is supposed
statistical performance, we computed the Packet Reception Ratio (PRR). It is calculated as the ratio between the number of packets received from neighbors in a specific geographical area and the total number of packets transmitted in the same geographical area. We used the baseline radius distance of 150 m as recommended by 3GPP in [11] for urban intersections. In Figure 4, we plotted the Cumulative Distribution Function

<figure><img src="image_5.png" /><figcaption>Fig. 4. CDF of PRR in presence of LOS and WINNER+B1 conditions.</figcaption></figure>

(CDF) of the PRR. Statistical results have been obtained by repeating the scene 100 times for each studied scenario. We notice that under LOS channel propagation conditions, 802.11bd outperforms 802.11p. For C-V2X, the performance is significantly lower. This is due to the SPS scheduling mechanism which, unlike the 802.11p/bd contention-based MAC scheme, is less responsive in terms of channel sensing. In C-V2X, a sensing window of 1 s is used for the selection and reservation of resources. Therefore, two nodes may choose the same time-frequency resource for a transmission, the collision cannot be avoided and cannot be detected by either party. Even worse, the collision episode may persist for several messaging intervals without the colliding node being aware of it.

However, all V2X technologies achieve an average PRR above 95% of reliability and meet 3GPP release 14 requirement [16].

However, when NLOS conditions are taken into account, the performance order is inverted and C-V2X outperforms 802.11bd and 802.11p. To better understand these results, we also plotted the Signal-to-Interference-and-Noise Ratio (SINR) probability distribution for both propagation conditions in Figure 5 along with SINR thresholds for which the physical layer of each technology is starting to perform for the selected set of parameters used in the simulated environment. SINR distribution is only dependent on the behavior of the V2X communication MAC layer as it rely on energy sensing to determine the appropriate time of transmission and all technologies have been simulated for the same bandwidth of

10 MHz. In Figure 5, we notice that the SINR for IEEE

<figure><img src="image_6.png" /><figcaption>Fig. 5. SINR distribution in presence of LOS and WINNER+B1 conditions.</figcaption></figure>

802.11p and IEEE 802.11bd are identical as expected since their MAC layers are identical. However, for C-V2X, the SINR spreads over much lower values than for the simulated IEEE 802.11 standards. This is a consequence of the Semi-Persistent Scheduling algorithm used by C-V2X. It reserves the selected resource for consecutive packets and is therefore less reactive to hidden nodes phenomenon. The interference that is generated is not compensated by the superior physical layer when LOS propagation is considered in this sparsely populated scenario. The SINR threshold is about 4 dB (resp. 7 dB) lower for the C-V2X physical layer than for IEEE 802.11bd (resp. IEEE 802.11p). When NLOS is considered, the SINR distribution is shifted towards lower levels for all standards. However, the shift has a larger impact for IEEE 802.11p and 802.11bd than for C-V2X.

To better understand a key element of our VRU scenario, we finally analyzed performance in a crash risk situation as shown in Figure 3. To this end, we processed the accident risk data previously logged by SUMO and then analyzed the packet reception performance over time from the beginning of the simulation to the time of the recorded crash risk between the vehicle (i.e. collider) and the cyclist (i.e. victim).

Figure 6 gives the packet reception probability over time of the VAM transmitted by the cyclist and received by the vehicle in a crash risk situation for LOS and WINNER+B1 channel propagation conditions. Two important event instants are identified in Figure 6: the crash time (i.e. red dashed line) that occurs at time 24s and the crash avoidance time (CAT) threshold (i.e. the black dashed line). The CAT threshold represents the moment after which the driver cannot take late actions to avoid the accident. According to our particular simulation timeline, CAT is set to time $t = 20.65$ seconds. This value is calculated from the crash time, which is supposed
to occur at time $t = 24s$ (i.e. in accordance with the SUMO accident risk log data) minus the reaction time and the braking time of the car. Reaction time is set to 1.5s and stopping time to 1.85s as the results from the division of the car maximum speed (8.3 m/s) by its nominal deceleration (4.5 m/s²). Good performance of packet reception is therefore crucial before the CAT reference time.

<figure><img src="image_6.png" /><figcaption>Fig. 6. Packet reception probability for vehicles in a crash risk situation.</figcaption></figure>

Under LOS channel conditions, the probability of receiving the packet 15 seconds before the CAT is very close to 1.0 for both IEEE 802.11p and 802.11bd. For C-V2X, performance is consistently increasing as the distance between vehicle and cyclist is reduced. However, it seems to exhibit a pseudo-periodic behavior, certainly induced by the periodicity of the SPS algorithm. When considering the 5s before the CAT threshold, the probability of packet reception is close to, but not equal to 1.0. When more severe NLOS conditions are considered, the behavior of the reception probability as a function of time is very different between both families of standards. For C-V2X, a degradation of the probability is consistently present along the time window of observation. An average reception probability of around 0.93 in the last 5s before CAT is measured. On the other hand, for IEEE 802.11bd and 802.11p, the probability of reception is significantly degraded when vehicle and cyclist are further apart, while when considering the last 5s before CAT, it is consistently above 0.95 and equal on averaged to 0.97. This behavior, is particularly relevant to the VRU scenario and suggests that although a significant amount of packet are lost when propagation conditions are severe, the packet reception probability is very high as the risk is increased. This confirms the safety benefits of V2X equipped VRUs.

## IV. CONCLUSION AND FUTURE WORK

In this paper, we have investigated the performance of V2X technologies in an urban environment and tried to estimate

their role in preventing and reducing the risk of accidents between VRUs and vehicles. To this end, we set up a simulation framework, in which we analyzed a canonical accident scene, evaluating the effectiveness of the most common V2X technologies in warning the driver to avoid a crash involving a cyclist. The main result is that a cyclist equipped with V2X technology can effectively warn the driver in LOS conditions, regardless of the V2X technology being used. Our findings show that in LOS conditions, IEEE 802.11bd and IEEE 802.11p slightly outperform C-V2X due to the contention-based MAC scheme, which better handles the collision mitigation and interference in a non congested environment. However, in a more holistic use case, taking into account the NLOS conditions, reception performance is more severely deteriorated when considering well before CAT for IEEE 802.11bd and 802.11p, as the physical layer performance of C-V2X provides a better link budget. Furthermore, when crash risks are imminent, we have shown that, for our intersection scenario, all V2X technologies are significantly beneficial for the drivers to detect the presence of a V2X equipped VRU and therefore avoid the dangerous situation.

In this evaluation study, we assumed that VRUs are all equipped with V2X technologies. However, privacy issues are some examples that leave pedestrians suspicious about the acceptance of V2X technology. Hence, further work will extend this study by evaluating the probability of VRU detection assuming vehicle on-board sensors and V2X cooperation among vehicles or infrastructure. Centralized architecture based on 5G New Radio (5G-NR) technology for VRU safety services will be also evaluated in our future work.

## ACKNOWLEDGMENT

This research was supported by the SAFE-UP project (proactive SAFEty systems and tools for a constantly UPgrading road environment). SAFE-UP has received funding from the European Union's Horizon 2020 research and innovation programme under Grant Agreement 861570.

## REFERENCES

[1] ETSI, ETSI TS 103 300-2 V2.1.1 Intelligent Transport System (ITS); Vulnerable Road Users (VRU) awareness; Part 2: Functional Architecture and Requirements definition; Release 2. 2020.

[2] Nagar Anwar, Norman Tranchi, and Gerhard Fettweis. "Physical layer evaluation of V2X communications technologies: 5G NR-V2X, LTE-V2X, IEEE 802.11 bd, and IEEE 802.11 p". In: 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall). IEEE, 2019, pp. 1-7.

[3] Valerian Mannoni et al. "A comparison of the V2X communication systems: ITS-G5 and C-V2X". In: 2019 IEEE 89th Vehicular Technology Conference (VTC2019-Spring). IEEE, 2019, pp. 1-5.

[4] Mirjami Jutila et al. "ITS-G5 performance improvement and evaluation for vulnerable road user safety services". In: JET Intelligent Transport Systems 11.3 (2017), pp. 126-133.

[5] Henrik Schumacher et al. "Vehicle-to-Vehicle IEEE 802.11 p performance measurements at urban intersections". In: 2012 IEEE International Conference on Communications (ICC). IEEE, 2012, pp. 7131-7135.

[6] Andreas Festag. "Cooperative intelligent transport systems standards in Europe". In: IEEE communications magazine 52.12 (2014), pp. 166-172.

[7] Andy Trivinaro, Iyad Dayoub, and Soumaya Chertaoui. "PHY layer enhancements for next generation V2X communication". In: Vehicular Communications 32 (2021), p. 100385.

[8] 3GPP. Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access (EUTRA); Medium Access Control (MAC) protocol specification, "Tech. Rep., Release 15, 3GPP TS 36.321 V15.2.0. July 2018.

[9] European New Car Assessment Programme (Euro NCAP) Assessment Protocol - Vulnerable Road User Protection Version 10.0.5. December 2021. URL: https://cdn.euroncap.com/media/67553/euro-ncap-assessment-protocol-vru-v1005.pdf.

[10] Pablo Alvarez Lopez et al. "Microscopic Traffic Simulation using SUMO". In: The 21st IEEE International Conference on Intelligent Transportation Systems. IEEE, 2018. URL: https://elib.dlr.de/124092/.

[11] 3GPP TR 37.885; Study on evaluation methodology of new vehicle-to-Everything (V2X) use cases for LTE and NR. 2018.

[12] Fabian Eckermann, Moritz Kahler, and Christian Wietfeld. "Performance analysis of C-V2X mode 4 communication introducing an open-source C-V2X simulator". In: 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall). IEEE, 2019, pp. 1-5.

[13] 3GPP. Study on LTE-based V2X services, TR 36.885 V14.0.0 July 2016.

[14] 3GPP TS 36.213; Evolved Universal Terrestrial Radio Access (E-UTRA); Physical layer procedures (v14.3.0, Release 14). June 2017.

[15] Yvo de Jong Bultitude and Terhi Rautiainen. "IST-4-027756 WINNER II D1. 1.2 V1. 2 WINNER II Channel Models". In: EBITG, TUI, UOULU, CU/CRC, NOKIA, Tech. Rep (2007).

[16] 3GPP TR 22.885 V14.0.0, 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Study on LTE support for Vehicle to Everything (V2X) services (Release 14), 2015-12.
to occur at time $t = 24s$ (i.e. in accordance with the SUMO accident risk log data) minus the reaction time and the braking time of the car. Reaction time is set to 1.5s and stopping time to 1.85s as the results from the division of the car maximum speed (8.3 m/s) by its nominal deceleration (4.5 m/s²). Good performance of packet reception is therefore crucial before the CAT reference time.

<figure><img src="image_7.png" /><figcaption>Fig. 6. Packet reception probability for vehicles in a crash risk situation.</figcaption></figure>

Under LOS channel conditions, the probability of receiving the packet 15 seconds before the CAT is very close to 1.0 for both IEEE 802.11p and 802.11bd. For C-V2X, performance is consistently increasing as the distance between vehicle and cyclist is reduced. However, it seems to exhibit a pseudo-periodic behavior, certainly induced by the periodicity of the SPS algorithm. When considering the 5s before the CAT threshold, the probability of packet reception is close to, but not equal to 1.0. When more severe NLOS conditions are considered, the behavior of the reception probability as a function of time is very different between both families of standards. For C-V2X, a degradation of the probability is consistently present along the time window of observation. An average reception probability of around 0.93 in the last 5s before CAT is measured. On the other hand, for IEEE 802.11bd and 802.11p, the probability of reception is significantly degraded when vehicle and cyclist are further apart, while when considering the last 5s before CAT, it is consistently above 0.95 and equal on averaged to 0.97. This behavior, is particularly relevant to the VRU scenario and suggests that although a significant amount of packet are lost when propagation conditions are severe, the packet reception probability is very high as the risk is increased. This confirms the safety benefits of V2X equipped VRUs.

## IV. CONCLUSION AND FUTURE WORK

In this paper, we have investigated the performance of V2X technologies in an urban environment and tried to estimate

their role in preventing and reducing the risk of accidents between VRUs and vehicles. To this end, we set up a simulation framework, in which we analyzed a canonical accident scene, evaluating the effectiveness of the most common V2X technologies in warning the driver to avoid a crash involving a cyclist. The main result is that a cyclist equipped with V2X technology can effectively warn the driver in LOS conditions, regardless of the V2X technology being used. Our findings show that in LOS conditions, IEEE 802.11bd and IEEE 802.11p slightly outperform C-V2X due to the contention-based MAC scheme, which better handles the collision mitigation and interference in a non congested environment. However, in a more holistic use case, taking into account the NLOS conditions, reception performance is more severely deteriorated when considering well before CAT for IEEE 802.11bd and 802.11p, as the physical layer performance of C-V2X provides a better link budget. Furthermore, when crash risks are imminent, we have shown that, for our intersection scenario, all V2X technologies are significantly beneficial for the drivers to detect the presence of a V2X equipped VRU and therefore avoid the dangerous situation.

In this evaluation study, we assumed that VRUs are all equipped with V2X technologies. However, privacy issues are some examples that leave pedestrians suspicious about the acceptance of V2X technology. Hence, further work will extend this study by evaluating the probability of VRU detection assuming vehicle on-board sensors and V2X cooperation among vehicles or infrastructure. Centralized architecture based on 5G New Radio (5G-NR) technology for VRU safety services will be also evaluated in our future work.

## ACKNOWLEDGMENT

This research was supported by the SAFE-UP project (proactive SAFETY systems and tools for a constantly UPgrading road environment). SAFE-UP has received funding from the European Union's Horizon 2020 research and innovation programme under Grant Agreement 861570.

## REFERENCES

[1] ETSI, ETSI TS 103 300-2 V2.1.1 Intelligent Transport System (ITS); Vulnerable Road Users (VRU) awareness; Part 2: Functional Architecture and Requirements definition; Release 2. 2020.

[2] Nagar Anwar, Norman Franchi, and Gerhard Fettweis. "Physical layer evaluation of V2X communications technologies: 5G NR-V2X, LTE-V2X, IEEE 802.11 bd, and IEEE 802.11 p". In: 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall). IEEE, 2019, pp. 1-7.

[3] Valerian Mannoni et al. "A comparison of the V2X communication systems: ITS-G5 and C-V2X". In: 2019 IEEE 89th Vehicular Technology Conference (VTC2019-Spring). IEEE, 2019, pp. 1-5.

[4] Mirjami Jutila et al. "ITS-G5 performance improvement and evaluation for vulnerable road user safety services". In: JET Intelligent Transport Systems 11.3 (2017), pp. 126-133.

[5] Henrik Schumacher et al. "Vehicle-to-Vehicle IEEE 802.11 p performance measurements at urban intersections". In: 2012 IEEE International Conference on Communications (ICC). IEEE, 2012, pp. 7131-7135.

[6] Andreas Festag. "Cooperative intelligent transport systems standards in Europe". In: IEEE communications magazine 52.12 (2014), pp. 166-172.

[7] Andy Triwinarko, Iyad Dayoub, and Soumaya Chertaoui. "PHY layer enhancements for next generation V2X communication". In: Vehicular Communications 32 (2021), p. 100385.

[8] 3GPP. Technical Specification Group Radio Access Network; Evolved Universal Terrestrial Radio Access (EUTRA); Medium Access Control (MAC) protocol specification, "Tech. Rep., Release 15, 3GPP TS 36.321 V15.2.0. July 2018.

[9] European New Car Assessment Programme (Euro NCAP) Assessment Protocol - Vulnerable Road User Protection Version 10.0.5. December 2021. URL: https://cdn.euroncap.com/media/67553/euro-ncap-assessment-protocol-vru-v1005.pdf.

[10] Pablo Alvarez Lopez et al. "Microscopic Traffic Simulation using SUMO". In: The 21st IEEE International Conference on Intelligent Transportation Systems. IEEE, 2018. URL: https://elib.dlr.de/124092/.

[11] 3GPP TR 37.885; Study on evaluation methodology of new vehicle-to-Everything (V2X) use cases for LTE and NR. 2018.

[12] Fabian Eckermann, Moritz Kahler, and Christian Wietfeld. "Performance analysis of C-V2X mode 4 communication introducing an open-source C-V2X simulator". In: 2019 IEEE 90th Vehicular Technology Conference (VTC2019-Fall). IEEE, 2019, pp. 1-5.

[13] 3GPP. Study on LTE-based V2X services, TR 36.885 V14.0.0 July 2016.

[14] 3GPP TS 36.213; Evolved Universal Terrestrial Radio Access (E-UTRA); Physical layer procedures (v14.3.0, Release 14). June 2017.

[15] Yvo de Jong Bultitude and Terhi Rautiainen. "IST-4-027756 WINNER II D1. 1.2 V1. 2 WINNER II Channel Models". In: EBITG, TUI, UOULU, CU/CRC, NOKIA, Tech. Rep (2007).

[16] 3GPP TR 22.885 V14.0.0, 3rd Generation Partnership Project; Technical Specification Group Services and System Aspects; Study on LTE support for Vehicle to Everything (V2X) services (Release 14), 2015-12.