All actors wanted to be part of this and to take advantage. The AV technology, with the help of the advancement of artificial intelligence, built the ambition that it could be done fast and revenues can be available within a few years’ time frame. This race to the tech market pushed actors to overcommunicate on their potentials, ambitions, and vision of the future, which lead to overinvestments and overexpectations. In its 2019 report, McKinsey and Company (2019) identified that “Since 2010, investors have poured $220 billion into more than 1,100 companies across ten technology clusters. Investors invested the first $100 billion of these funds by mid-2016 and the rest thereafter.” Within a total of 120 billion investments in mobility, in the 24-month time frame (2016–2018), automated vehicle technologies were the most rapidly increasing technology sector. Today, mid-2023, the picture is much different. Expectations are even higher, but within longer time frames and with a more realistic approach, as clearly reported in the 2023 report of Deichmann et al. (2023). It is anticipated that automated driving will trigger even more revenues and will also benefit to ADAS market. Both ADS and ADAS are correlated and can benefit one to the other, and many technologies and features of ADS can be used for ADAS.

### 3.2.2 Automated Driving

Just as a reminder, when talking about driving autonomy, the standard SAE J3016 of the SAE International and ISO is the reference, with five levels of driving automation. Full automated driving System (ADS) is seen at Levels 4 and 5. Vehicles do not have driving wheels nor pedals, there is no driver, and the ADS system will fully drive the vehicle.

The advanced driver assistance system (ADAS) encompasses vehicles operating at Level 3 and below. In these levels, cars assist the drivers during the driving process. Even in Level 3, where the software can handle the vehicle in various scenarios, there will come a point when the system cannot handle the situation anymore and will request that the driver take control of the vehicle. At that moment, the system disengages technically passing full control to the driver. Level 3 automation introduces ambiguity by potentially misleading the driver about onboard responsibilities—whether the driver or the system is primarily responsible. This raises questions about legal liability in case of accidents during the transition from system control to driver control. A notable instance highlighting this issue is the well-known 2018 car accident involving an Uber self-driving car, which struck and killed a pedestrian (Said, 2018).

### 3.2.3 The Landscape of Automated Mobility

Significant technological players initiated an exceptional disruption within the traditional original equipment manufacturer (OEM) car market. While traditional OEMs were gradually incorporating ADAS features, advancing from level to level,
tech giants such as Google/WAYMO, UBER, LIFT, BAIDU, and others recognized that technology was the true game changer. Instead of incrementally moving from L0 to L5, they took a more direct approach to working on SAE L5, even if, in practice, it resembled more of an L3 or L4. Their perspective envisions the future of automobiles as robustly connected computers on wheels, embodying the tech giants' vision, rather than mere vehicles with software components, as seen in the OEM's perspective.

The tech giants harnessed cutting-edge sensor technologies, including lidars, radars, and other advanced sensor systems. Additionally, they leveraged state-of-the-art AI technologies, whether in the realm of computer vision or AI prediction, to enhance the development of perception, trajectory planning, and driving capabilities within their advanced driver assistance systems (ADS). These companies were also well equipped with high-performance computing power, which they employed to a substantial extent. New emerging comers also followed this trend of top-down computer on wheels' approach.

In addition to those concentrating on vehicle or driving system development, a comprehensive ecosystem was also established, featuring numerous participants operating across all facets of the automated driving market. This includes players involved in hardware, electronic control units (ECUs), sensors, feature provisioning, AI solutions, fleet management, localization technologies, mapping, networked transport of RTCM via internet protocol (NTRIP) data, and more.

### 3.2.4 NAVYA before 2018

In 2015, a small French startup, NAVYA, took the pioneering step of introducing commercial automated shuttles for public transportation. At a time when most market competitors were engaged in the race for driverless personal cars, NAVYA shifted its focus to public transport. They recognized the numerous advantages that automated driving can offer to cities, rather than to individual drivers, such as reduced traffic congestion. Their goal was to develop automated shuttles capable of accommodating up to 15 passengers, offering a shared and public transport service that encourages people to leave their personal vehicles at home for short rides. By adopting this innovative approach, NAVYA emerged as one of the pioneers that significantly influenced the global landscape of automated vehicles.

#### 3.2.4.1 Hardware

As NAVYA investments were not at the same level as OEM or tech giants, NAVYA focused on solving the risk step by step, developing one feature after the other, solving ODD incrementally, and leveraging speed accordingly. No OEM, nor any industrial provider, was ready to build a vehicle platform able to be automotive ready to receive the PCs and automated driving system. Thus, NAVYA built its own factory and its own mechanical platform called the Autonom® Shuttle (Fig. 3.1, NAVYA
<figure><img src="image_37.png" /><figcaption>Fig. 3.1 NAVYA ARMA© specifications</figcaption></figure>

<figure><img src="image_38.png" /><figcaption>Fig. 3.2 NAVYA automation positioning in the market, based on SAE J3016© levels of driving automation (Illustration by the author based on SAE J3016©, 2021)</figcaption></figure>

ARMA specifications). The conception was targeted to be SAE L4 ready by design without steering wheel, without pedals, without car mirrors, and able to scale to full SAE L4 (Fig. 3.2, NAVYA ARMA sensor overview).

NAVYA successfully integrated lidars, sensors, and various components (Fig. 3.3, NAVYA ARMA sensor overview) that were readily available from the market, even though these parts had not yet reached mass production levels with the highest automotive industry standards, mainly because such standards were nonexistent in the market at that time. The majority of industry players utilized identical
<figure><img src="image_39.png" /><figcaption>Fig. 3.3 NAVYA ARMA© sensor overview</figcaption></figure>

components, including the same lidars and sensor sets, for their prototypes and small-scale production runs (Fig. 3.3).

### 3.2.4.2 Software

During that period, NAVYA demonstrated that their shuttle could be completely driven and operated by embedded software connected to a central supervision center. They employed localization technologies that relied on pre-mapped environments, utilizing global navigation satellite system real-time kinematic (GNSS RTK) technologies, among others. The maps used were basic 2D representations, and the commissioning and mapping technologies were relatively straightforward, typically involving simple round trips and linear routes. These systems effectively operated within a limited operational design domain (ODD).

The shuttle’s primary routes consisted of straightforward journeys, such as round trips, covering short distances, typically about 1 or 2 km. There were minimal interactions, and if traffic lights were encountered, they were typically managed by the onboard operator.

### 3.2.4.3 Services

To put shuttles in operation, a common work with the PTO is done to define the trajectory and the route. A radio GNSS base is installed on the site, open to the sky and covering the geofenced area. This base will get several satellite signals and will use GNSS RTK technologies to send correction to the shuttle giving precise 10-cm positioning. Subsequently, NAVYA implemented an in-house mapping hardware
and software solution called mapping and localization measurement system (MMS). This system had the capability to be installed on any car, allowing the digitalization and creation of highly accurate route maps with precise positioning. Furthermore, it collected all the necessary data to digitally recreate the surrounding environment. The map was sent to the R&D for cleaning and taking out parked vehicles, pedestrians, dynamic elements, etc., to keep only the road and structural elements of the environment. A clean map was built by the R&D, and the commissioning teams can start working with the Autonom Shuttle® and a dedicated in-house tool on site to create and build the virtual rail, called the path. The path is what the shuttle will use to follow. It is built block by block, and each has its own parameters related to the real road (speed, intersection, curves, passenger crossing, etc.). Once the path is built, it is inserted in the vehicle system and put on the server. The shuttle can then use it to start the blank testing without passengers, driving and following the path and of course handling and reacting to all dynamic environments on the road. This will allow to fine-tune the path and to adjust also all the real driving parameters to have a smooth ride. NAVYA accomplished all these tasks using internally developed tools. Each of these steps was essential in establishing the mobility service they aimed to provide.

### 3.2.5 NAVYA Ecosystem

The success of NAVYA was based not only on the driving technology alone but is the whole system that makes it happen. NAVYA delivers to cities a full solution and service to put in place automated shuttles, to train the PTO, and to allow daily operation. The service value chain is public transport authority (PTA), to public transport operator (PTO), and to NAVYA AV system shuttle and supervision. NAVYA provides the technology to the PTO that will help him to operate the daily service for the PTA or the city.

From NAVYA shuttles, the value chain extends to the automotive industry, encompassing the suppliers of each hardware element of the shuttle. These hardware suppliers provide various elements, including fundamental components such as static or mechanical parts, batteries, battery management systems, sensors, and low-level command control systems, among others. Many of these suppliers lack prior experience in the field of automated vehicles, and the requirement for achieving a high level of safety in each component has not yet become common practice in their factories or production processes. Notably, basic elements like sensors remain relatively fragile, and their maturity is not yet at a stage where failure rates and false positives are optimal. Nonetheless, they are considered safe enough for deployment in vehicles under specific controlled conditions.

The entire ecosystem is collectively learning about its needs. Sensor suppliers require real data from manufacturers to enhance their products, while manufacturers seek greater reliability from the sensors. Public transport operators (PTOs) demand high-quality service and uptime from shuttle suppliers. These demands cascade
down to the suppliers of various systems and subsystems. However, it is challenging to achieve a high level of safety as defined by Automotive Safety Integrity Level (ASIL) based on the regulatory standards outlined in automotive ISO 26262 and still make it a mass-market product. Everyone involved is on a learning curve as they work together to address these challenges and refine the ecosystem. Therefor even if the ADS of the shuttle would be the best of bread technology on the market, if the hardware subsystem is not reliable, the driving experience is not smooth. If sensors are providing false positive, stating there is an obstacle where there is none or just a leaf falling, the ADS will trigger harsh brakes to avoid accidents. That is why perfect correlation between hardware and software should always be harmonized. Moreover, to sell these shuttles on the market, it should be balanced in an accessible eco-financial equation.

### 3.2.6 Legal Boundaries

In 2015, when NAVYA released its first commercial automated shuttles called The Autonom Shuttle® ARMA, no legal institution in any country had a good knowledge of such vehicles and technology. None was ready to deliver the authorization to put them on open road. Each vehicle on open road should have a registration plate, and the vehicle should be homologated by the authorities based on the related legal framework of its category (M1, M2, etc.). As the NAVYA shuttle does not have steering wheels nor pedals, it is seen as an unclassified vehicle. Authorities took into account that it is an innovative vehicle, an experimental vehicle, and they delivered partial homologation with a registration plate limited in the time of testing, e.g., 1 or 2 years, if the vehicle can prove that it is safe enough to operate on open road under certain conditions.

NAVYA had to operate on a country-by-country basis, collaborating closely with government authorities or entities representing the government. Their goal was to assist these authorities in comprehending the technology, the vehicle, and how to address the challenges posed by the absence of traditional elements such as a steering wheel, pedals, left and right rearview mirrors, bidirectional driving, and the associated lighting systems. This cooperative effort aimed to ensure the smooth integration of their automated shuttle technology into each region's regulatory framework and infrastructure.

In the absence of a standardized legal framework, the response to automated driving initiatives varied from one country to another. In nearly 20 countries, NAVYA took the pioneering step of being the first to introduce automated driving on open roads, where regulations and acceptance of this innovative technology were largely uncharted territory. In some, it took more than 1 year and half to get the registration plate, convincing the authorities that the AV shuttle is safe and secured for its innovation purpose on the dedicated road. Once the authorities gained the knowledge and established the procedures for partial homologation in collaboration with NAVYA, it paved the way for competitors to enter the country
more swiftly. This scenario played out in countries like Denmark, as exemplified by
the AVENUE project. Through the joint efforts of NAVYA, the local partner,
AMobility (trademarked Holo), and AVENUE, we became the first to introduce an
automated vehicle (AV) shuttle on open roads in Denmark, creating a precedent for
others to follow.

## 3.3 Technology Improvements Through AVENUE

### 3.3.1 A Global View

One specificity of the AVENUE project was that the AV operations started from day one. Four different cities in Europe put in operation a total of ten automated shuttles and connected them to their public transport system. AV shuttles were based on the technology available in 2018 and the operational design domain that NAVYA shuttles could manage at that time. The PTOs used these deployments to advance their process, build the IT, learn from operational deployment, and prepare to scale and reach the target built by AVENUE to have a smooth and seamless on-demand door-to-door service. Although some regions had prior experience with AV shuttle experimentation, embracing this technology for regular use represented a significant advancement and a substantial request for progress in terms of regulatory and operational readiness. The other project partners were able to get a deep dive into the technology, to understand the current limits and potential, and to ideate to support the delivery of all targeted services of the project.

To put in place automated shuttles services, a global view of the technology was needed to understand the enhancements at all levels. The AV shuttle operational capabilities were the starting point. If the ODD of the site is above these capabilities, the service is due to fail. For example, the AV shuttle cannot go faster than 25 km per hour due to safety issues. If in the route defined by the operator there is a part at 50 km/h, even a short one, the shuttle will not be able to follow, and it is better to decide not to build the service.

### 3.3.2 NAVYA Software

Even though NAVYA is known as a manufacturer of automated shuttles, its core competencies are the ADS software and related technologies. NAVYA does conceive and build the vehicle and architect the whole set of sensors. The ADS, along with its related technologies, comprises the software suite that governs the automated shuttle, encompassing its full range of functionalities. It supports all aspects of the aforementioned system, from deploying to operating a mobility service using such automated shuttles. The software of NAVYA can be seen as three major parts: NavyaDrive®, NavyaOperate®, and public API (Fig. 3.4).
<figure><img src="image_40.png" /><figcaption>**Fig. 3.4** NAVYA ADS overview</figcaption></figure>

**NavyaDrive®** is an embedded software installed on the computing system in each vehicle and allows to operate the vehicle. Internally, every new iteration of the **NavyaDrive®** software is referred to as a “Setup,” which is linked to a specific version number.

**NavyaOperate®** is an off-board software installed on the cloud and allows to supervise and manage vehicles. NAVYA supervision center team uses this software to supervise sites, as well as to take actions on the vehicle like for maintenance, analysis, and other non-driving actions.

**Navya API** is a software installed on a cloud system that allows a third party to receive and send information to one or several vehicles. No direct connectivity to vehicles is allowed to third parties due to security and cybersecurity issues.

### 3.3.3 Automotive New Release Process

The **NavyaDrive®** software is the main intelligence embedded in the shuttle. It is installed in the computing systems and connected to sensors from one side and to mechanical hardware from the other. The connection to mechanical hardware manages low-level command control system. It sends and receives all information to fully control vehicle dynamics. As AV shuttles are not yet full in industrial production with volumes, the providers made some evolution of existing hardware and mechanical parts to allow the use in AV. New software updates of these parts are made regularly, correcting bugs or adding some features. For this purpose, shuttles, even if they are all based on the same ARMA model, do have differences between each other.

For each new software version of **NavyaDrive®**, a comprehensive series of testing and validation procedures is essential to ensure compatibility and compliance with the various versions of the ARMA shuttles that have received authorization from regulatory authorities. This rigorous process is crucial to enable **NavyaDrive®** to fully utilize the capabilities of any deployed ARMA shuttle, offering a safe and reliable automated driving solution. With each new release of the **NavyaDrive®** software and setup, it is imperative to adhere to automotive standards during the testing and validation process. This ensures that no unintended regressions are introduced from previous versions, that all new features function as intended, and that the ARMA shuttles maintain the requisite safety standards to be deployed on the road. This meticulous process guarantees the reliability and safety of each software
update. Adding a new feature or correcting even a small bug should go through this long process.

The AVENUE project played a crucial role in enhancing the standardization of this process. It ensured that all AVENUE shuttles adhered to the same service standards set for the project. Prior to this, NAVYA had been focused on promptly addressing client issues to maintain the continuity of their mobility services. In response to client-specific concerns, they often corrected bugs and released software updates after limited local testing, ensuring compatibility with the client's hardware version. However, this approach led to significant segmentation of the NavyaDrive software and introduced some unintended issues for other clients. Standardizing the process through projects like AVENUE helped streamline software development and ensure consistency in service across all deployments.

Due to the growing diversity in hardware among different clients, the approach of addressing client-specific issues had led to an accumulation of software discrepancies. With the AVENUE project involving ten vehicles and four public transport operators (PTOs), each facing unique scenarios and operational design domains (ODDs), the testing and validation process had to be meticulously fine-tuned and adapted to ensure that each software setup could run seamlessly on the various ARMA shuttles under different conditions.

While this approach improved overall software consistency, it occasionally resulted in frustration for the PTOs. The time required to go through this thorough process and reintroduce the service with new features could lead to operational challenges and delays. However, the aim was to strike a balance between software consistency and efficient service delivery. Each software setup iteration is a heavy specification, design, and validation process that takes several months to make sure that no safety issues are released to final customers. Fig. 3.5 (NAVYA software update and validation process) describes the software specification tasks and the test and validation process.

AVENUE operations commenced right from the project's inception, with both software and hardware versions already at an advanced stage. Over the course of the 4.5 years of the AVENUE project and due to the continuous evolution of hardware providers, several updates to both hardware and software became

<figure><img src="image_41.png" /><figcaption>**Fig. 3.5** NAVYA software update and validation process</figcaption></figure>
<figure><img src="image_42.png" /><figcaption>**Fig. 3.6** NAVYA software version setups</figcaption></figure>

necessary. The AVENUE shuttles initially utilized version 3.X of the software and eventually progressed to the final release at version 7.X, reflecting the dynamic nature of the technology and the ongoing development efforts within NAVYA's research and development, driven by Agile development methodology. Above is a timeline of different major setups of NavyaDrive® during the AVENUE project (Fig. 3.6).

### 3.3.4 NavyaDrive® Evolutions

#### 3.3.4.1 The Operating System

In order to have a powerful and highly reactive driving system, NAVYA proceeded to refactoring the software architecture and migrate all from Windows to Linux operating system.

This migration was a major step to scale the NavyaDrive® system to higher level, bring more features, provide higher stability and performance, and structure the upcoming setups. A better homogeneity was also established with other connected hardware parts of shuttles like routers and other elements.

Enhancements at all levels were made to allow agility and resilience to cybersecurity attacks. The cybersecurity was not simple to manage when you are system dependent like on Windows. With Linux NAVYA did enhance the signature of PC images, the control of the access, the governance of the profiles management, and the cyphering and certificate management.

Several PTOs and authority assessors quickly noticed these changes by comparing not only the driving but also by comparing the cybersecurity audits done before and after this update.

This phase was difficult at several levels starting from the refactoring of the software till the installation on the ARMA shuttles. It must be done with the presence of a NAVYA engineer on site for each shuttle. It needs to travel to the site with the computing system, replace the previous ones with the new ones, update the software parameters depending on each site specificities, and run blank test to ensure non-regression, and only then can it go back to normal service.
### 3.3.4.2 Over-the-Air Update

With the introduction of the over-the-air (OTA) feature in the latest software setup, the previously time-consuming and on-site installations for updates have become obsolete. This innovation enables software updates to be delivered through 3G or 4G networks, simplifying the process and enhancing safety compliance with EU regulations. Furthermore, OTA updates have accelerated the management capabilities of NAVYA's remote supervision center. It allows for remote diagnostics, investigation of events, retrieval of logs, assistance to onboard operators in various situations, vehicle maintenance diagnostics, and more, making the operation and maintenance of AV shuttles significantly more efficient and streamlined.

Now for each new setup, NAVYA can activate this feature remotely, transfer the new software, and check the integrity and health of software before putting back the AV shuttle in the hands of the PTO. It can additionally customize the display screens to include personalized elements such as sounds, animated images, icons, and more for PTOs.

### 3.3.4.3 On-Demand Service

In AVENUE objectives, integrating on-demand/door-to-door system within the public transport service would be a game changer for urban mobility. This was a major technical challenge. A transformation of how the AV shuttle should conceive its route. The state of art of all AV shuttles at that time was to follow a virtual rail with a round trip. The shuttle starts in point A to finish at point D and then do the reverse from D to A. It should pass by bus station B and bus station C. This track is followed in a regular way according to a timetable, passing always by the same stops at the same time. Everything is done without any alterations or interaction with the passengers, except for stopping at the bus station.

In the AVENUE project, the on-demand system operates differently from traditional public transport services with fixed routes and timetables. Instead, any passenger can utilize a mobile application or an interactive bus stop to select any destination within a geofenced area. They can then wait until the AV shuttle arrives and pick them up. During the journey, the AV shuttle might make slight adjustments to its route to pick up another passenger heading in the same direction, even if they have a different final destination. This approach provides a more flexible and personalized transportation service.

This service requests several elements:

* An IT architecture where the IT system is connected to a fleet of vehicles covering the geofenced area

* An application, mobile, or interactive outdoor screen, connected to the system that a passenger can use to request a travel

* A fleet management system to receive the travel orders, detect the nearest AV shuttle, and send the mission

* AV shuttles that can receive and operate the travel missions, sending back real time information for tracking
NAVYA supported building the system architecture with technical partners and focused on developing the parts related to AV shuttle and external connectivity. Many steps were needed to build this flexible innovative system, described below.

**The driving route structure** was previously built with a fixed virtual rail (Fig. 3.7, fixed route/rail approach), called path, which was prepared by the commissioning team and with a dedicated map. The path is constructed with fixed stops. Once ready, the path is put inside the software of the AV shuttle. The driving system follows this path, guiding the shuttle accordingly. It executes the relevant driving instructions while also considering all new dynamic and static surrounding elements, such as pedestrians, traffic conditions, and other vehicles. If the shuttle gets out of its path, the embedded security system of the NavyaDrive® will stop the shuttle. So if there is any change to hail another passenger out of the path, it will immediately stop the shuttle. The idea of changing dynamically the defined path during the operation was at the time inconceivable (Fig. 3.8).

**The structure of the driving system** was refactored to take into account step by step these changes. The first step was to allow the onboard operator to define the pickup/destination points of the travel within a fixed path by using the onboard tactical screen, dashboard user interface (DUI).

<figure><img src="image_43.png" /><figcaption>**Fig. 3.7 Fixed route/rail approach (Navya®)**</figcaption></figure>

<figure><img src="image_44.png" /><figcaption>**Fig. 3.8 Fixed route as seen by the vehicle (Navya®)**</figcaption></figure>
**A mission management system** was built to prepare and control the way the driving system should treat pickup/destination requests. Tracking the mission is taken in account to follow, send information, and acknowledge the status of the missions. A mission can be modified, canceled, or replaced with a new one by the system. However, the system does not permit changes to the current driving if they introduce any safety risks. Additional features were added to allow the management of the events for the ride. **The event management system** is described later. It allows to receive missions with additional actions like opening the door and rolling the ramp once arrived. Such flexibility will enhance the service to be used easily by disabled passenger or elderly.

The next step in the development process involved creating a comprehensive **off-board system**, which was hosted on a cloud-based platform. This system is designed to receive mission instructions from external sources, verify them, and then transmit them to the shuttles. This ensures better control over external connections and prevents any compromise to driving safety. It will facilitate the creation of new functionalities for end users from external systems. Therefore, all security, cybersecurity features, access control, and stack management should be done to prevent hacking the driving system.

A secured API was created for the partners so they can connect from external IT system. The connectivity structure involves third parties interfacing with the NAVYA cloud, which then acts as an intermediary to communicate with the AV shuttles. This configuration ensures that direct connections from third parties to the shuttles are not permitted, enhancing security and control over the communication channels.

The software architecture described above enables partners to efficiently manage the fleet of shuttles on one side and oversee passenger applications on the other. This dual functionality streamlines the operation and coordination of the AV shuttle service, providing control over both the vehicles and the passenger experience. This step allowed the **first usage of on-demand service** in a round trip and provided a first connectivity between the passenger application and the shuttle connected to the fleet management system for the AVENUE sites.

To extend the coverage across the entire geofenced area, enabling the shuttle to deviate from round trips and pick up passengers anywhere within this zone, it is essential to have the capability to dynamically construct the driving path. This necessitates an adaptation in NavyaDrive®'s approach, shifting from predefined static paths to the construction of dynamic routes in real time. A representative site can serve as a testing ground to develop and refine this new path structure, allowing for greater flexibility in shuttle operations.

**The site of Belle Idée in Switzerland** (Fig. 3.9, the Belle-Idee site in Geneva) is the best use case to do so. It covers 38 acres, 9.2 km of roads, 24 buildings, 6 parking, and several points of interest, served by an extensive road mesh.

The ambition was that a passenger can request a pickup on his mobile app from any place to any destination within this area.

**The whole area was mapped**, and a clean and structural HD map was built for this purpose. NAVYA has always been innovative and active in developing its own
<figure><img src="image_45.png" /><figcaption>**Fig. 3.9** The Belle-Idee site in Geneva</figcaption></figure>

technology of mapping and building a bench of advanced tools to do so. AVENUE helped enhancing such tools that allowed better knowledge and perception of the environment and brought a higher-level of localization.

Instead of building a path on a round trip, all the roads of the area were covered with a network of paths with different specificities as shown in Fig. 3.10 (Belle-idee routes and virtual stops).

**The NavyaDrive® navigation** system underwent a significant transformation. Instead of relying on a single static path to follow, it now possesses the capability to dynamically construct the path by assembling pieces of pre-mapped and pre-pared routes. This approach ensures that the navigation system can adapt and construct the optimal route for driving by piecing together predefined path segments as needed.

**For the door-to-door service**, instead of fixed stops, virtual stops were used. Almost 73 virtual stops (Fig. 3.10) were decided with the PTO to allow safe boarding and dropping of the passengers in the most probable points of interest that are usually used. No physical bus stop was built. When a user selects a pickoff, there is always a virtual stop next to his/her position, and he/she was advised to go to this spot for safety issues.

The mission management system also was enhanced to allow:

* Add/cancel missions dynamically

* Dynamic rerouting

* Refuse any mission that could lead to dangerous maneuvers like changing at the last minute and then turning left and right when already engaged

* Status and event management

**A specific driving "Mode" in NavyaDrive® system** was built to allow AVENUE partners to connect and start engaging mission sending for the on-demand service.
<figure><img src="image_46.png" /><figcaption>**Fig. 3.10 Belle-Idee routes and virtual stops**</figcaption></figure>

<figure><img src="image_47.png" /><figcaption>**Fig. 3.11 The TPG control center, passenger app, and vehicle on the site**</figcaption></figure>

The system was first tested, trained, and fine-tuned by starting the operations without passengers on board. Connection to local supervision center was installed. Once validated, it was open to all passengers (Fig. 3.11).

The Belle-Idée site was at the forefront of implementing the on-demand system and incorporating all the latest features with the most recent setup, marking a significant milestone during the final year of the AVENUE project. Other sites
activated the on-demand service and put it in operation, each differently, and with different application or fleet management provider. All were using the same innovation brought by NAVYA, and all demonstrated that such integration is a game changer for public transport. In Lyon, the deployment site was at Parc Olympic de Lyon (Parc OL) (Fig. 3.12, Lyon Parc OL deployment site) and in Copenhagen at the Slagelse site (Fig. 3.13, Copenhagen Slagelse site).

<figure><img src="image_48.png" /><figcaption>Fig. 3.12 Lyon Parc OL deployment site (Keolis ©)</figcaption></figure>

<figure><img src="image_49.png" /><figcaption>Fig. 3.13 Copenhagen Slagelse site (Holo ©)</figcaption></figure>
#### 3.3.4.4 V2X Traffic Light Management

The AVENUE project played a pivotal role in enhancing the driving experience by integrating vehicle-to-everything (V2X) technology. V2X technology was employed consistently, particularly at traffic light intersections, and occasionally in managing challenging turn situations. This technological integration improved overall traffic flow and maneuverability. As a reminder, in Europe traffic light goes from *green* to *orange* to *red* where only *green* allows to go on. In Europe the *orange* light appears just before the *red* light. Decisions are sometimes not easy to take when the vehicle is about to pass a *green* light and at the last minute the light changes to *orange*. Even normal drivers brakes aggressively when it happens, without stating those who accelerate instead. Previously NAVYA shuttles impulse harsh braking in such situations leading to uncomfortable situation for passenger and cars behind.

**NAVYA built a decision diagram** handling every phase of the traffic light, *green* to *orange* to *red* to *green*, taking into account the *time to change* to the next light, the traffic conditions, and the shuttle speed. All these elements together are considered to handle the right deceleration/acceleration profiles.

NAVYA's experience with V2X technology revealed that errors could occur, and at times, V2X might not transmit the correct signals. Although such instances were rare, they were a reality. To address the critical issue of avoiding running a red light, NAVYA developed a redundant solution that leverages camera-based recognition to assess the status of traffic lights in various lighting conditions. This approach is built on artificial intelligence models and was tested during the AVENUE project but subsequently integrated into production to enhance safety and reliability.

To ensure a smooth and safe driving experience, the vehicle employs an estimation system to determine the time remaining before a traffic light changes to *green* or *red*. Depending on the situation, the shuttle can take one of the following two actions: Frist is anticipatory deceleration. If the light is changing from *green* to *orange* to *red*, the shuttle applies the appropriate deceleration profile in advance to optimize passenger comfort. It also informs the onboard operator about the approaching change in signal. Second is speed regulation. In cases where the light is transitioning from *red* to *green*, the vehicle regulates its speed to ensure a safe and controlled crossing of the intersection when the signal turns *green*. These actions are designed to enhance the overall driving experience, ensuring smooth and efficient intersection navigation.

In specific situation like flashing *orange* light on all the traffic lights of the intersection, meaning the driver should be careful and follow the normal priority rules if there is no sign, the ARMA shuttle stops, disengages, and informs the operator to take over: "traffic light is not manageable." In this case, the operator can then either take the lead in manual driving to cross the intersection or wait until the shuttle receives readable information to cross safely. NAVYA added new diagnostics and errors code, on flashing *orange* lights and on missing V2X signals.
### 3.3.4.5 V2X Solution for Complex Situations

The site in France, near Lyon, NAVYA and the PTO (Keolis) faced two major driving difficulties. First the shuttle must go through an important roundabout where more than 40,000 vehicles drive in a day. And second the shuttle must do an important turn left where traffic is difficult. The solution facing these two situations was to use an interactive V2X system with external partners.

For the roundabout, active traffic lights with V2X are installed on each of its entry. Before arriving to the roundabout, the ARMA shuttle sends information to the traffic light controller and requests the priority on the traffic. The first traffic light in front of the shuttles will take control and will allow the entry of the shuttle in the roundabout. The other traffic lights on the roundabout will turn *red* progressively one after the other on each entry of the roundabout and will block other vehicles from getting in the path of the shuttle. Once the shuttle passes through, the traffic lights behind get back to *green* to unleash the traffic allowing other vehicles to get in. In this way the shuttle will have a fluid traffic and will ensure a safe and smooth ride.

As for the left turn difficulty, the ARMA shuttle will send information to the traffic light controller when it is near the left turn. The shuttle will take the left lane, and the traffic light will turn to *green* smoothly and to *red* for the vehicles on the other side of the road. Therefore, the road will be clear, and the shuttle can engage safely the left turn without stopping. This V2X management was the world's first demonstration of how infrastructure and automated shuttles can actively interact to ensure secure traffic. Major developments were made on the NavyaDrive® software and the low level stack of V2X to ensure compatibility and non-regression.

### 3.3.4.6 Driving Enhancement

Safe and smooth driving is the essence of driving activity, especially when it comes to automated driving. Many elements do interact and are associated to deliver such results including all hardware and software enhancements. As there are many done during AVENUE, it is difficult to go through all of them here, but we can get a peak on some.

**The acceleration and the deceleration profiles** are the major ones. Sudden and aggressive braking or acceleration can lead to accidents, cause discomfort for passengers due to jerky movements, and also lead to the disapproval from other road users and vehicles. As a result, the service can be stopped by the PTO if it does not deliver customer satisfaction at all levels, including comfort. The acceleration and deceleration profiles of the NavyaDrive® were enhanced from a setup to another to provide smooth driving. The purpose of this algorithm was to ensure the right trade-off between safety and comfort for both passengers inside and other road users outside. Acceleration was boosted upon departure, and intersections were efficiently navigated with increased acceleration. As for the deceleration and braking, they are
softer, and the harsh brakes and jerks were minimized. The braking distances were modified, depending on the deceleration profile criteria.

"Adaptive cruise control" was also added to the NavyaDrive®. The ADS detects and evaluates the speed of the vehicle in front and adapts its acceleration and deceleration profiles accordingly. Adjustment of the speed of the shuttle on the speed of the vehicle in front was done automatically. This prevents the shuttle to always deploy different acceleration and deceleration profile or to do several braking. This is also valuable in traffic jams, a kind of traffic jam feature for ADAS.

**Obstacle tracking system** allows to anticipate the obstacle trajectory and to anticipate the entry of an obstacle in the path of the ARMA shuttle. This results in better behavior in several situations like on intersection and passenger crossing. For example, when the shuttle stops at a passenger crossing, without a traffic light, the shuttle tracks the movement of the passenger. It detects when the passenger is out of its path and when the passenger trajectory is getting farther away. So when the passenger passes the shuttle front lane, the shuttle gets back in movement carefully even if the passenger is always on the crossing. The same thing happens when the shuttle is driving and detects a moving obstacle on the left that is drifting and comes in the alert zone in front or around the shuttle.

**Overtaking** is a major difficulty for automated driving. Overtaking in all situations, expected or unexpected ones, and at any speed is not yet delivered by any ADS shuttle system by the time this document is written. Nevertheless, in some situations several OEMs have incorporated it in ADAS.

NAVYA during the AVENUE project made enhancement on this topic, and some overtaking was achieved. Overtaking of small objects (Fig. 3.14), objects parked half on the pavement and half on the road, and not well parked vehicles, were possible to avoid to stop or disengage. A maximal lateral distance of possible overtaking was set to keep safety first.

**Steering control** was enhanced, and a new software was implemented to improve responsiveness of the steering.

**Driving alignment (trajectory tracking):** Maintaining precise trajectory and adhering to correct lines or curves are a fundamental aspect of manual driving. In the realm of automated driving, this aspect becomes even more critical and requires constant monitoring. It is important to recognize that deviations from the intended path can occur not only due to vehicle performance but also as a result of road conditions, such as potholes, uneven road surfaces, curves, road grip, and overall stability.

A trajectory tracker is of a high importance to keep safe driving, and NAVYA enhanced the trajectory tracker within the AVENUE project. This module improved accuracy of trajectory with higher control of the jerk. The jerk became a major measurement element of the comfort of the passenger and the smooth driving. Curve management is easy to build in software, but when hardware is involved, a 100% alignment is not simple. The hardware reactivity, curve angle, and speed while driving can lead to some misalignment. All drivers notice this curve's lateral deviation upon inertia while driving.
<figure><img src="image_50.png" /><figcaption>**Fig. 3.14** Overtaking a static obstacle</figcaption></figure>

Shuttle deviates from its initial trajectory to overtake a static obstacle

NAVYA developed a more precise trajectory tracking allowing a smooth driving on curves and better radius alignment on the road, reducing lateral deviation. This enhanced the safety in curves and prevented incidents due to out of path alerts (Fig. 3.15).

**Driving alignment (line/lane check):** Transitions from “MANUAL DRIVING” mode to “AUTO DRIVING” are sometimes difficult to manage depending on its position toward the road. While starting a journey, the shuttle should be correctly aligned within its path, with a centimeter-wise precision. A human-machine interface (HMI) has been developed to assist the operator in this peculiar situation (Fig. 3.16).

The perception viewer (Fig. 3.17) provides the operator with the exact position of the vehicle: lateral alignment and angular alignment error (offset angle and centimeter gap with the path).

**Vehicle behavior improvements:** As per the legal framework in effect during the AVENUE project, an onboard operator had to be present inside each AV shuttle when it was on open roads. The onboard operator must keep hands on the steering controls and remain standing to maintain an unobstructed view of the road ahead. This requirement was in accordance with the Vienna Convention, which mandates that every driving vehicle must have a driver on board. These regulations were implemented to
<figure><img src="image_51.png" /><figcaption>Fig. 3.15 NAVYA trajectory tracker</figcaption></figure>

<figure><img src="image_52.png" /><figcaption>Fig. 3.16 Median detection for confident departure feature</figcaption></figure>
<figure><img src="image_53.png" /><figcaption>Fig. 3.17 Perception viewer</figcaption></figure>

ensure safety and align with established international driving standards. Until the change of this convention, which was done in 2022 as described in the chapter "Governance Impact Assessment" from Lionel Binz later in this book, in AV shuttle the driver is the onboard operator even if he does not do the driving operations.

The shuttle occasionally needs to disengage the ADS and allow the onboard safety operator to take over manual driving. This is necessary for situations where the ADS faces technological constraints, such as overtaking or reversing, or when the AV shuttle must exit the parking area and be positioned at the starting point of its automated route. Similarly, at the end of service, the shuttle may need to be manually driven back to the parking area or to a maintenance center. These maneuvers are to be done manually by the onboard operator with the steering controls. During manual driving, the ADS software and safety control system are deactivated. The onboard operator controls totally all the driving. Most of the time, incidents and minor accidents occur during such situations.

Based on the requests of several PTOs, a new driving mode was introduced to allow the onboard operator to drive safely the AV shuttle. The assisted manual driving (AMD) feature assists the operator during all manual driving. In a later iteration activation became optional, but once activated, speed limitation, obstacle detection, and Autonomous Emergency Braking were enabled. With this mode also, a buzzer inside the vehicle alerts the onboard operator and the passengers on the presence of an obstacle (Fig. 3.18).

In manual driving mode, certain onboard operators provided feedback regarding the risk of passengers falling within the shuttle due to the detection of nearby obstacles, prompting the shuttle to initiate abrupt stops. As a result, NAVYA ceased to restrict the vehicle's speed in AMD mode, and braking procedures were adjusted accordingly. The decision to stop or continue driving remains with the onboard operator. Additionally, a new tone signal alerts the onboard operator to nearby obstacles, with frequency indicating proximity: the more frequent the tone, the closer the obstacle.
<figure><img src="image_54.png" /><figcaption>**Fig. 3.18** Information process to the onboard operator</figcaption></figure>

<figure><img src="image_55.png" /><figcaption>**Fig. 3.19** NAVYA blinking management</figcaption></figure>

* Obstacle in the “close” zone = continuous tone signal

* Obstacle in the “middle” zone = very regular tone signal

* Obstacle in the “far” zone = irregular tone signal

**Blinker management:** A more operational blinker management was set up to allow better interaction with other vehicles and road users, especially during departure and arrival to stations. They stay activated on stations. Such feedback from the onboard operators and from AVENUE users allowed NAVYA to enhance such elements that cannot be easily seen in development or test and validation phases (Fig. 3.19).

**Blackbox improvements:** ARMA shuttles have been recording all their activities for a long time. Still enhancements were made within AVENUE by reducing significantly the recording time of the data and adding more data to allow a clearer view and to replay certain situations afterward. Reducing time of recording reduces the use of the CPU power and keeps the compute power to the NavyaDrive® system.
### 3.3.5 Supervision Improvements and NavyaOperate®

Automated public transport services, like automated metro, are connected to a control tower and a supervision center. All the vehicles are connected, facilitating the exchange of various types of information with the supervision center:

* Telematic information sent from the vehicles to the supervision center. This is the upstream where all details of the shuttles are viewed on a dashboard. It includes video stream inside and outside the vehicles, the position, speed, events, bus stops, unexpected stops, alerts, etc. All these information are needed to operate a daily service and are provided to the PTO either via the NavyaOperate® system or via a partner API. It allows access to the shuttle states and events, current and passed, with time stamps and associated status codes.

* Technical information exchange in both upstream and downstream. This level of exchange is under the control of NAVYA supervision and maintenance center. It allows to access the NavyaDrive® software of the shuttle, only under maintenance access and in stopped situation. The supervision center can get logs, diagnosis of the vehicle's vital information, and hardware details. It also allows the maintenance and the supervision teams to update over-the-air the software of the shuttles with a downstream channel.

* Mission management. This two-way stream is the one developed for the AVENUE on-demand system that allows a third-party connection to the shuttles and to send missions or event triggering. The upstream provides acknowledgment and tracking of the missions.

Numerous enhancements where done during the AVENUE project to allow a complete integration of the shuttles in the AVENUE technical architecture and to provide the complete on-demand service.

The AVENUE project emphasized the importance of having a richer dataset to facilitate more straightforward analysis. This dataset encompassed various components, including historical diagrams, diagnostic information, detailed event logs; highly technical data such as velocity, obstacle events, as well as comprehensive descriptions; and maintenance-related details like temperature and heat data. Additionally, it involved real-time streaming, complete mission information, and the ability to remotely trigger various actions such as honking, buzzing, controlling lights, stops, doors, and ramps, and monitoring interior vehicle information. The dataset also encompassed GNSS signals, hit ratios, and other signals, together with detailed site-specific information and synchronization with onboard data like bus stops, stop names, and more. They were made available, making easier the analysis. Some of this information was also put in the record management system.

There were three supervision modes developed:

* "PARTNERS" supervision mode: this mode is reserved only for PTOs that have access to the Navya API. PTO can send missions to the ARMA shuttle.
* "STAND-ALONE" supervision mode: through this feature, some other supervision solutions can send commands to the vehicle: STOP, GO, and OPEN AND CLOSE DOORS.

* "METRO" supervision mode: the supervision can now ask the vehicle to leave a station remotely in "METRO" mode.

### 3.3.6 Navya API

The connectivity between the shuttles and remote applications of third parties is due to the standardized public API put in place and enhanced during AVENUE. It is a major outcome of AVENUE. The architecture developed goes through the NAVYA cloud system before connecting to the shuttles as described earlier. This is not only for security reasons but also for functional and technical performance across all sites, all connected partners, and all vehicles in operation or not, avoiding to impede the main task of the shuttle, which is driving smoothly and safely. No partner can connect directly to a shuttle. All must pass through the NAVYA cloud system.

The API evolved a lot during the AVENUE project, and the connectivity protocols were simplified and standardized across the 4.5 years of the project. At the beginning only the partner Bestmile was able to connect, send, and receive information with a dedicated closed protocol. To add a new connection, it would take several months with this architecture. NAVYA changed the protocol to RESTful HTTP API with WebSockets, and with this, any new partner was able to connect its application within days. That is how three new AVENUE partners were able to quickly connect their fleet management system (FMS) and mobile application applications at all sites.

Also a training and support program was put in place to allow the new partners to onboard quickly. Before any connection to the real AV fleet, a simulation program with the digital twin site was developed and installed to train the FMS and to be sure that all sending and receiving are working. Once the FMS is working correctly with the simulator, a real testing phase with people on site is applied to be sure of real integration, and continuity is done between simulation and the real site.

With the AVENUE project, this public API got through different versions bringing many features like:

* Change of the exchange protocol to have a standard secured HTTP and a new architecture of WebSocket and associated low-level services

* Security and certificate management

* Access per vehicle per several vehicles per site

* Access mode management

* Enrich data that should be collected by the NavyaDrive® and then shared to the NAVYA cloud

* Video live streaming

* In-vehicle monitoring and information sharing

* Share event data
* Share control non-driving action on the shuttle (buzz, light, doors, ramp, sound, screens, etc.)

* Share diagnosis data (engine temperature, GNSS signals, lights and blinkers, batterie status, etc.)

* Mission management create, cancel, change, and track.

* Event triggering with mission management

Opening a new feature in the public API is more than just providing data through the connectivity and putting the API interface on the server. For each and every new feature or data delivered, development is needed on the NavyaDrive® software in the shuttle to collect, clean, secure, and send the data. And for each development on the NavyaDrive®, a new setup must be made, and the full test and validation process must be applied to be sure that there is no regression and no impact on the safety in operation.

The AVENUE project fed NAVYA with new features and ideas to enhance the API for fleet management systems, the supervision, and the shepherding features.

### 3.3.7 HMI and Experience Enhancement

#### 3.3.7.1 Operator User Interface

Inside each shuttle, a screen is installed to replace the vehicle dashboard of a standard vehicle. This screen must respect legal regulations in place like for normal vehicle. and several mandatory elements are to be followed. This screen is called the dashboard user interface (DUI) (Fig. 3.20).

Several functions are to be support by the DUI:

* Provide and display of mandatory visual and control elements to the safety driver

* Provide tools to the safety driver to perceive, analyze, ad control the shuttle

<figure><img src="image_56.png" /><figcaption>Fig. 3.20 NAVYA© DUI interface</figcaption></figure>
* Provide passengers tools to define which bus stops are requested and to follow the status of the trip

As described earlier, in some cases the onboard operator needs to manually drive the shuttle. The DUI is also to help him in these maneuvers and to provide a clear vision. It helps to show the exact perception view as the NavyaDrive® is seeing it. Many advancements were done on the DUI based on the feedbacks provided by the PTOs of AVENUE during the operations. These AVENUE outcomes were important elements that could not be seen without feedback from the ground, from the people handling the real daily operation of the shuttles.

Below are some evolutions made on the DUI (example in Fig. 3.25):

* Authentication process.

* System status overview.

* Service mode control.

* Support-assisted manual driving mode.

* Diagnosis and analysis.

* Track and acknowledge any system event.

* Display help page.

* Alignment with regulation related to vehicle dashboard.

* A clear perception viewer indicating how NavyaDrive® is perceiving the environment.

* Several elements were added to the perception viewer like: positioning of the environment and all objects seen by the vehicles sensors, obstacles that are blocking the vehicle advancement clearly highlighted when needed, and dynamic point visibility.

* Visual color enhancement to clearly indicate elements like lidar hit ratio and % level.

* Better positioning of some buttons used by the onboard operator like “Navigator,” “States,” “GO,” and “STOP” buttons.

* Door states displayed inside the GO button.

Asked by the PTO and for the comfort of the safety operators, a night mode of the DUI was added (Fig. 3.26). Three main possible choices are available: day mode, automatic mode (display of the night mode according to the time of the sunrise and sunset in the geographical area), and night mode.

Adding to the above, a new event management system was developed within NavyaDrive® during the AVENUE project. This allows to track events for the supervision center. For a better management between the onboard operator on site and the supervision center, an event viewer was added to the DUI allowing him/her to better diagnose the vehicle status (Fig. 3.21). An interactive interface helps to deep dive in the events, and solutions are directly proposed delivering more independency of the onboard operator.

Software and display changes in the “EVENTS” section (Fig. 3.22):

1. Icon indicating the severity of the event.

2. A short description of the event is automatically proposed.

3. A padlock indicates the impact of the event.
<figure><img src="image_57.png" /><figcaption>Fig. 3.21 NAVTYA© DUI day mode selection and vehicle status</figcaption></figure>

4. Detailed information by clicking on the event.

5. Procedure to be followed by the operator to solve the detected event.

### 3.3.7.2 Event Triggering System

A real-time information with audio announcement was another outcome of AVENUE project. The AVENUE PTOS from different countries and with different languages are used to have inside their buses an information system for the passenger: visual indication of next stops, full itinerary display, audio announcements, etc.

NAVYA designed this software to have the capability to automatically initiate actions in response to event statuses. The event manager functions by alerting other systems when a particular event occurs, enabling those systems to execute corresponding actions. For instance, when the vehicle reaches a bus stop, the system can trigger the opening of doors and the deployment of the ramp. This real-time information display provides all the necessary passenger information, ensuring a smooth and well-informed experience.

Additionally, the system can also make adjustments to various supplementary elements within the vehicle and its interior. This includes modifications such as regulating ambient lighting, controlling low beams, making adjustments to the dashboard user interface (DUI), and deploying the ramp, among other features.
<figure><img src="image_58.png" /><figcaption>Fig. 3.22 NAVYA© DUI event reporting</figcaption></figure>

### 3.3.7.3 In-Vehicle Audio Announcements (UI)

The event triggering system is used also to automatically trigger audio announcement upon events, combining prerecorded messages as well as text-to-speech system. Audio announcements aim of course to assist disabled people. This can be done in multilanguage.
This system was first tested during the Dubai World Challenge for Self-Driving Transport for its first edition in 2019. Arabic and English languages were used to announce the bus stops and other information.

### 3.3.7.4 Interactive Interface for Passengers (UI)

Based on the advancements made on the DUI stated previously for the onboard operator and the real-time information system, the internal screen was enhanced for the passengers. It was designed to assist users along their journey:

* Visualize the top-view map of the operation site

* Visualize the real-time position of the vehicle

* Visualize the list of available stations

* Visualize the current itinerary

* Visualize the estimated time of arrival

* Select a desired station when on-demand service is enabled

* Select the desired interface language

* Trigger the departure/stop of the vehicle

* Display help page

In AVENUE, all shuttles were ARMA shuttles, and all visual enhancements stated were done for DUI screen that is used for the onboard operator and the passengers. One outcome of AVENUE is to add to the next generation of NAVYA shuttles, called EVO, and a screen dedicated to passenger information. EVO will have a large lateral display facing the entry of the shuttle above the lateral window. This display will be used for all internal and real-time information. Such system was conceived, produced, and tested in a prototype during the Dubai World Challenge stated above. All visual and sound information were displayed and triggered in Arabic and in English.

### 3.3.7.5 External Sound (UI)

Usually, NAVYA shuttles are operating in mixed traffic in cities and peri-urban areas where other vehicles are driving. Horns, sounds, and other external elements are of normal usage for these environments.

In AVENUE, the site of Belle-Idée is a hospital. Three ARMA shuttles are covering the whole area of the hospital. Cars, buses, and pedestrians are encountered, but all respect this quiet environment. NAVYA worked with the PTO to get all recommendations and adjustments needed to adapt the shuttles accordingly to this quiet environment. The features that can, in a way or another, disturb or make noise were deactivated, like busing when an obstacle is detected. These adjustments can be turned on and off easily on site.
### 3.3.7.6 External Screen and Human-Machine Interface (HMI)

The interaction with external road users is necessary. The ARMA shuttle can include two dedicated screens. Each displays information through the front and back window. The event management described above allows to trigger displays on these screens upon the perception of the environment or to interact with the road users. For example, the system can display animations when pedestrians are crossing ahead, indicating to them that the vehicle has detected them and is stopping. Alternatively, it can show information to vehicles behind, informing them of various actions of the ARMA shuttle vehicle, such as boarding or disembarking passengers, turning left or right, displaying a stop or yield sign in front of the shuttle, indicating proximity to a traffic light, signaling manual driving, or alerting to the detection of an obstacle just ahead, etc. (Fig. 3.23).

## 3.3.8 Other Enhancements

### 3.3.8.1 Hardware Enhancement

NAVYA is enhancing the hardware constantly through commercial projects, upon customer feedback, and of course the needs of the AVENUE sites. AVENUE brought added value to this process as within AVENUE acceleration was requested in areas that were not explored before. The shuttle itself, mechanical parts, brakes and steering box, ECU and electronics, electrical network, sensor architecture, the safety concept, low-level control, etc., all were in constant enhancement, and AVENUE contributed to the assessment of needs. Several versions of ARMA shuttles went through this constant enhancement for AVENUE PTOS. It is difficult to highlight hardware enhancement here. Sometimes, small hardware details block the authorization to deploy a service on open road, and hardware enhancements must be made.

One specific issue was raised for the Copenhagen deployments. One mandatory point was requested and hardware changes in the structure of the vehicle so that the vehicles should handle the safety of wheelchairs according to the Danish law. In order to fix the wheelchairs, Q-straint has been installed in the shuttles. Q-straint works simply by having four mounted points in the floor, with seatbelts that can be hooked to the wheelchair. The seatbelts retract automatically and lock when necessary. When a wheelchair has to board the shuttle, the onboard operator mounts the four seatbelt heads on the floor. This means that the floor is empty when riding without a wheelchair. When carrying a wheelchair user, the three foldable seats cannot be used, and there is room for eight additional passengers (Fig. 3.24).
<figure><img src="image_59.png" /><figcaption>Fig. 3.23 Images and animated figures can be personalized for the PTO</figcaption></figure>
<figure><img src="image_60.png" /><figcaption>Fig. 3.24 Attaching a wheelchair (Holo©)</figcaption></figure>

### 3.3.8.2 Mapping, Commissioning, and Tools

A whole set of tools, not directly related to driving the shuttles or supervising the operation, are needed and contribute a lot to deliver and AV service. The different levels of complexity of AVENUE sites contributed a lot in the enhancements of all these tools.

**The MMS tool**, related to the acquisition of the lidar cloud point and all environmental data of the road, was enhanced and got higher performance of lidars, sensors, and localization parts. A dedicated hardware and software tool was built to easily operate the system on site by the commissioning engineers on site.

**HD mapping** was developed to provide a higher level of precision in the perception of the environment and its knowledge. It brought higher localization precision while driving, and a notable enhancement was seen once the sites were mapped in HD mapping and updated. This enhancement is not only related to the acquisition of data. A full set of mapping tools and features were built by NAVYA engineers to manage this map, to rebuild this digital twin, to clean it, and to identify structural elements like driving zone, the road, etc.

Related to the commissioning, a dedicated **path construction tool** was built and enhanced to get the most of the HD map and to build the path with the easiest and the fastest way. Commissioning engineers can then quickly put a site in operation.
Also **diagnosis tools** accelerated the way the maintenance people can understand, correct, and put back shuttles in operation.

All these tools are fundamental to the AV ecosystem to deploy mobility service.

### 3.3.8.3 Additional Tool Enhancements

A complete set of additional tools was required for other tasks, such as creating the path and shuttle maintenance. Having more than 13 shuttles in the project across 6 cities greatly contributed to the enhancement of these tools.

**Enhanced perception viewer** was developed for commission engineers and maintenance team to help them in their task. It provides a 360° view of the surroundings, the same view as the shuttle sees it. Such feature reduced the time to detect and solve issues from within the shuttle. Several information were delivered like the lidar map, perception, wheel angles, obstacles display in different angles, etc.

It also allowed to calibrate sensors with a specific process to follow and be sure that sensors are perfectly aligned (Fig. 3.25).

**Event registry**, as stated previously, is logged, aggregated, and managed through this software feature.

**Log management** is made simpler to allow the supervision center to isolate and retrieve information faster. The supervision center and R&D had also easier tools to analyze the different diagrams retrieved.

<figure><img src="image_61.png" /><figcaption>Fig. 3.25 NAVYA© DUI 360 perception</figcaption></figure>
<figure><img src="image_62.png" /><figcaption>**Fig. 3.26** NAVYA© booting status</figcaption></figure>

**Deploy, sync, and store the configuration files** for faster commissioning and maintenance updates.

**Tool mode was created** for maintenance and deployment training purposes, with richer information (site name, vehicle technical details, hardware id, temperature of elements, etc.).

**NavyaDrive® boot** indication guides users through various startup stages, conducting autochecks and diagnostics, and displaying relevant information on the proper functioning of each step (Fig. 3.26).

## 3.4 Beyond Avenue

Developing profitable business models for automated shuttles requires substantial investment in research and development. Operational design domains (ODDs) vary, and with each step toward enhancing new features, there is a significant increase in the level of required resources and workload. Initially, many AVENUE public transport operators (PTOs) prioritized higher speeds, but as operational services were implemented, it became evident that speed was secondary to reliability and safety. Increasing speed affects all aspects of the software, sensors, and hardware components. It necessitates a more extensive capability to detect obstacles at greater distances, improve analysis and prediction, react more quickly, and ensure the reliability and safety of hardware at all levels.

Therefore, the hardware driving platform, an SAE L4 drive-by-wire system, is a crucial component. At present, no provider offers a fully homologated platform that closely aligns with automotive standards such as ISO 26262, and includes hardware redundancy features like brakes and steering. Additionally, this platform must incorporate a low-level security concept to ensure the vehicle can safely stop when
the software system detects dangerous maneuvers or encounters issues that cause a loss of control.

Several universities, startups, and engineers demonstrated pieces of software that can control longitudinal and latitudinal maneuvers of a vehicle. However, all require a vehicle, and none can bring such software to market without having safe, homologated hardware. Producing such a vehicle cannot be done for very small series. It should be done in an industrial way and will require very high financial and resource investment that can be justified by high volumes (price/volume). All experts and OEM agree on this, and that is why, today, historical OEMs are working on incrementing features of the ADAS to go toward SAE L3 before getting to SAE L4.

Nevertheless, some are taking the challenge and are starting to invest in such drive-by-wire platform, and in CES 2022 some announced that they will have such vehicle by 2026. NAVYA has already started this process during the AVENUE project by building partnership with a bus OEM, Bluebus. This collaboration is made to build the first automated minbus SAE L4 ready and aligned with legal framework that is to be put in place. Since 2023 the hardware was out of Bluebus factory, and NAVYA connected the NavyaDrive® software in a few weeks and made first automated L4 tests without a driver on board. This L4 bus will officially hit the market in 2025 and will scale the automated vehicles for public transport to another level.

## 3.5 Conclusion

AVENUE was a game changer in the AV landscape in Europe and toward all the AV community. It changed the way the community and decision-makers are seeing the automated vehicles.

I remember when, in 2019, NAVYA launched the Autonom® Shuttle® in Brussels, right under the European Commission's headquarters, the Berlaymont. As the community experienced it, everyone saw the potential of this technology for public transport. It became clear that it is not just software, but an entire ecosystem capable of supporting the local socio-economic systems of EU countries.

This change paradigm from vehicle centric to ecosystem was needed, and the EU started to work on the legal framework. NAVYA was integrated in this legal work from the beginning, and the ADS act was released in 2022.

The work done in this area by the French government, where NAVYA was active since 2018, inspired and supported the ADS Act. The French government was the first to release in 2022 a full set of regulation on AV SAE L4, sowing the ecosystem dynamics. Just 1 month before, Germany released a part of its legal framework.

All are based on a double authorization as stated in other chapters: A type approval of the automated vehicle to be put on the open road on one hand and an authorization of the AV service for a specific area with a specific ODD associated to the vehicle on the other. One delivered at a global level, the other at local level.

Before AVENUE all PTOs were waiting to receive their automated shuttles to put in place a service. But once they receive it, none was ready to put in place, and
all stumbled on two elements: the legal authorization and how to manage the daily operations with their non-experienced team.

The role of public transport operators (PTOs) traditionally involves managing buses and drivers, overseeing driver competencies, handling client complaints, and maintaining schedules and frequencies. When traditional buses experience mechanical failures, their repair centers can manage these issues, as their personnel are trained for such maintenance. Similarly, transitioning from conventional fuel-powered buses to electric ones often requires minimal changes, as the drivers remain the same, with some necessary training.

However, the situation is markedly different when it comes to automated shuttles. Managing and maintaining automated shuttles present unique challenges that differ from traditional vehicles, necessitating specialized expertise and considerations.

Automated shuttles are digital objects, controlled by computers. At that time, they were experimental objects and were not industrialized with a high level of after-sales processes, local after-sales ecosystems in all countries, or trained repair centers. No driving competencies are needed to handle them but different expertise. Many public transport operators (PTOs) are not yet fully familiar with the digitalization of their profession and the related processes. Transitioning from traditional diesel-powered buses with fixed timetables to automated buses that don't require traditional driving skills poses a significant challenge. The move towards fully digitalized and connected buses, which operate without fixed timetables and instead offer an on-demand service, represents a disruptive shift.

In this context, when an automated shuttle encounters issues, the onboard operator from the PTO must, rather than calling his maintenance technician, engage in a digital process with the NAVYA supervision center. This process involves providing a detailed description and obtaining logs. Unlike conventional buses that rely on mechanical maintenance, solutions for automated shuttles are predominantly digital in nature. They may entail immediate digital troubleshooting or involve software updates that follow a dedicated process.

For instance, there were cases where certain onboard operators believed they could address incidents by attempting to manage or access the shuttle's computers. However, their efforts often resulted in exacerbating the issues rather than resolving them. In some instances, after attempting to independently rectify shuttle incidents, they resorted to manual driving to maneuver the shuttle. Yet, instead of moving forward, the bidirectional shuttle unexpectedly moved backward within the garage, ultimately colliding with a wall and causing damage to its lidars and sensors.

Also the authorities were not ready. They do not know what is this technological vehicle and in which legal "box" should they put it. They knew that it is game changer in the industry, but they wanted to take more time to understand it, to control all aspects of it to avoid any human or material damage, and to put all safety elements in a well-defined regulation.

The choice of the site, where the AV vehicle should drive, is an important part and role of the PTO. If the site's ODD is not fully aligned with the automated shuttles' capabilities, daily operations will struggle, and operational issues will
start. It was seen in AVENUE as some of the sites that were decided on paper were not the most successful ones and brought many legal and daily complexity.

A shift in mindset was required among all partners, from public transport operators (PTO) to technical partners. AVENUE demonstrated that their “vehicle-centric” vision, solely focused on the level of driving autonomy of the shuttle, is not the optimal approach. Merely possessing an automated shuttle does not suffice to establish a safe and high-quality daily mobility service. The PTO must undertake several tasks: selecting suitable sites with the appropriate operational design domains (ODD) according to the shuttle capabilities, preparing and training staff, digitizing all systems, establishing connections with the supervision center, and developing operational processes. These processes, such as pre-operation checklists, procedures for starting and ending the service, incident management protocols, and passenger handling guidelines, are crucial components of a robust and sustainable AV service.

AVENUE allowed to put the focus on this ecosystem and all actions needed for that. A comprehensive vision was established, urging all partners to consider the “big picture” within urban planning and the public transport network, rather than focusing solely on AV technology. This encompassed questions on how AV shuttles should be integrated into Mobility as a Service (MaaS), identifying the external technologies required for integration, and evaluating in which use cases AV shuttles were more beneficial to citizens compared to traditional buses. AV shuttle should be, as any other transport vehicle, part of the whole mobility service of cities, bringing added value to all citizens.

Based on this knowledge and this “big picture” vision, the AVENUE team built a very ambitious follower project, ULTIMO project, to integrate AV in MaaS, in 3 major EU cities with at least 15 AV shuttles per site. This project of around 55 million Euros was the winning proposal of a major EU call and received funding from the European Commission in Horizon Europe program.

NAVYA significantly influenced all partners, the European community, cooperative connected and automated mobility (CCAM), competitors, researchers, and notably the European Commission. It demonstrated that even a smaller entity can make a difference and contribute to the establishment of an ecosystem that challenges industry giants. Many startups were created following NAVYA models. Big PTOs created dedicated departments, and today some actors of car giants created a strategic plan based on AV shuttles to produce in mass production around 2026.

We all hope that AV shuttles can become a regular part of our daily transportation, and every partner in AVENUE, along with NAVYA teams, hold strong confidence and take great pride in being part of this transformative disruption.

**Acknowledgements** This project has received funding by the European Union's Horizon 2020 research and innovation program under grant agreement No. 769033.
## References

Deichmann, J., Ebel, E., Heineke, K., Heuss, R., Kellner, M., & Stei-ner, F. (2023). Autonomous driving's future. Convenient and connected. McKinsey & Company. Available online at https://www.mckinsey.com/industries/automotive-and-assembly/our-insights/autonomous-drivings-future-convenient-and-connected. Accessed 30 Nov 2023.

Frost & Sullivan. (2018). Global Autonomous Driving Market Outlook, 2018. With assistance of Global Automotive & Transportation Research Team at Frost & Sullivan (K24A-18).

McKinsey & Company. (2019). The future of mobility is at our doorstep. Compendium 2019/2020. With assistance of McKinsey Center for Future Mobility.

Panetta, K. (2017). Top trends in the gartner hype cycle for emerging technologies 2017. Gartner. Available online at https://www.gartner.com/smarterwithgartner/top-trends-in-the-gartner-hype-cycle-for-emerging-technologies-2017. Accessed 30 Nov 2023.

SAE J3016. (2021). Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. Available online at https://www.sae.org/blog/sae-j3016-update. Accessed 30 Nov 2023.

Said, C. (2018, March 20). Uber halts self-driving operations after car kills Arizona pedestrian. San Francisco Chronicle. Available online at https://www.sfchronicle.com/business/article/Uber-self-driving-car-strikes-kills-pedestrian-12764464.php. Accessed 30 Nov 2023.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
<figure><img src="image_63.png" /></figure>

Chapter 4

Safety, Security and Service Quality
for Automated Minibuses: State of the Art,
Technical Requirements and Data Privacy
in Case of Incident

Philippe Chrétien, Antonios Lalas, Konstantinos Votis, and Dimitrios Tzovaras

**Abstract** The AVENUE project confirmed that the automated minibus technology is attractive to improve urban mobility services with contribution to the ecologic transition but at risk in terms of safety and service quality. Maintaining the public transportation level of injuries, based on heavy bus experience, is challenging and implies (i) an intermediate target for market introduction, gathering active safety and passive safety requirements, and (ii) an improvement process, based on anonymised and shared lesson learnt, under public power control. With resulting demands concerning reliability and maintainability, the safety improvement will contribute to the service quality and to the building of a sustainable business.

**4.1 Introduction**

Automated minibuses constitute an emerging technology that will transform public transportation in various ways. During the journey of this transformation from the conventional buses to the automated approach, a variety of safety, security and service quality needs are required to be addressed for achieving a successful transition. Sustainability and durability of the autonomous vision are also required for all stakeholders to engage acceptability. Traffic management, energy consumption, customer satisfaction are additional aspects have to be taken into account to accomplish sustainability. Mobility as a service (MaaS), which denotes mobility provided as a service by combining transportation services from several transportation

P. Chrétien (✉)

Centre Européen d'Etudes de Sécurité et d'Analyse des Risques, CEESAR, Nanterre, France
e-mail: philippe.chretien@ceesar.fr

A. Lalas · K. Votis · D. Tzovaras

Centre for Research and Technology Hellas, Information Technologies Institute,
Thessaloniki, Greece
providers, is deemed critical to change the future of transportation by enabling more
personalised transportation services. Data privacy and relevant regulations have to
be taken into account to facilitate this procedure.

In this chapter, a shared sustainability and durability target for the society and the related companies is described, along with the conditions to achieve these objectives. Next, the critical path for market introduction of safe automated minibuses is presented, while the quality and safety state of the art for automated minibuses is discussed. A self-learning automated transport system at the European level is also revealed, whereas the data privacy of incident analysis and lesson learned sharing is explained. Finally, the automated minibus safety and service quality levers are summarised.

## 4.2 A Shared Sustainability and Durability Target for the Society and for Companies

With the different functionalities tested on the demonstration sites, the AVENUE project confirmed that automated minibus solution is attractive in terms of urban mobility services and now technically accessible, as presented in the previous chapter.

This is very important for the public power, as this technology can also contrib-
ute to the urgently required urban ecologic transition, if carefully integrated into the
existing urban mobility systems, as a green and flexible complement to the conven-
tional public transportation infrastructure. The best example of a conventional infra-
structure is railway, which is adapted to massive transportation but not flexible: this
introduction of automated minibuses is now a clear target for the European Union.

The following chapters will discuss the environmental and social impacts, which are positive when replacing private cars by such solutions, thanks to new attractive services: no doubt about that, but preliminary conditions have to be fulfilled to avoid any future regression.

This introduction of automated minibuses has to be organised carefully by the
European Union. If not, the transition towards a more flexible and ecologic trans-
portation system will not be sustainable in terms of societal acceptance, and the
created business will not be durable for private actors, investing in this technology,
which has to be financially amortised in the future.

## 4.3 The Conditions to Make it a Sustainable and Durable Solution

With these two goals in mind, the new automated minibus services will have to be
not only attractive but also to confirm this good feeling with efficiency, reliability
and safety, to establish year after year the customer satisfaction.
These new services will also have to get confidence from the public authorities, including other concerns like traffic management, energy consumption and health protection, with the associated mediatic and cost impacts.

### 4.3.1 Traffic Management and Energy Consumption

On-demand and door-to-door services are crucial to get a flexible transportation service, which has to be inclusive for the elderly and persons with reduced mobility: these functionalities imply that automated minibuses will drive and park among other vehicles, including private cars, without any remarkable disturbance to the traffic of other vehicles, which does not mean any impact. If a conventional driver is autonomous in such situation, the vehicle has to be autonomous in the existing infrastructure and traffic: "a vehicle equipped with ADS aims to perform the entire dynamic driving task on a sustained basis within a defined operational design domain without driver involvement" (Standing General Order on Crash Reporting) (NHTSA, 2021, 2023).

In the future, the traffic performance will not be measured by the maximum speed available on a specific route. The average speed is more representative, but it will not be sufficient to measure the traffic performance: the target of traffic management will be to guarantee an acceptable travel time from A to B and so to limit the variability.

The responsibility associated to public transportation with automated driving will lead to a careful vehicle control, compared to private cars, with the hope of a quieter driving, without traffic jams, avoiding energy consumption and time wasting, with working loss, etc.

This quieter traffic could be slower, but the careful introduction of these automated minibuses has to avoid traffic disturbance and congestion, so the automated minibus will have to manage blocking situations, even when coming from driving rules' strict application.

A simple example is the necessity to cross the white line to pass a private car which is not correctly parked, which implies a human solution to be used: in such a case, the US strategy for Automated Driving (NHTSA, 2020) is a driver mimicry based on vehicle artificial intelligence, which will be sometimes necessary, maybe thanks to the quick arbitration of a distant operator.

### 4.3.2 "Customer" Durable Satisfaction, Including Safety

The "customers" are service users but also other citizens, public transport authorities and more generally the different European states, with public concerns on emissions, energy economy, public health and cost.
Concerning the citizens who will pay in the future for this service (project demonstrators were free), they have explicit requirements in terms of comfort, punctuality, trustworthiness and resilience, which will have obviously to be satisfied.

In comparison, some other requirements are more difficult to manage as they are hidden, because they are implicit: it is generally quality and safety, with the associated conditions of technical reliability, cybersecurity, regulation compliance, insurance and juridical responsibility, which are behind quality and safety.

For these implicit requirements, the public power involvement is necessary: a roadmap has to be established from the realised demonstrations to the future durable customer-paid services.

To avoid future regressions, the roadmap has to anticipate the probable risks concerning traffic impact, service quality and road user safety, especially vulnerable users, and to organise the global improvement of these new services.

### 4.3.3 Safety Measurable Targets and Steps

In addition to the fact that it is an implicit requirement, there is a second difficulty to manage safety, which is the difficulty to measure it: the fatality target has to be close to zero and not measurable, as it is clearly unacceptable that a city transportation vehicle kills a citizen!

The minimal requirement level is to maintain the global safety level of existing transport services, with coaches or buses (EU, 2023a). This fatality level is not easily measurable but can be estimated globally, with the help of massive data collected in terms of person travelled kilometres, which is the principal function of public transportation, but also of the traffic system, where comparison can be made between different transport modes, including individual vehicles.

In fact, this minimal requirement is already very ambitious, as buses and coaches are very heavy in comparison with the planned minibuses and with the existing vehicles, which is a mechanical advantage to protect passengers in case of an accident.

As a consequence, it can be discerned that the passive safety performance of buses and coaches is much higher than cars, even with lower passive safety technology.

The future automated minibuses will be smaller, as the absence of a paid driver allows us to reduce the number of passenger places, with benefit to transportation flexibility: the targeted weight for the urban automated minibuses is close to the weight of the coming electric SUVs, i.e. 2500 kg.

Based on that, two recommendations have been made to the European Community:

1. The safety measurement has to be done at a vehicle level and based on injuries compared to running hours, associated later to travelled kilometres. One condition is that public transport operators will be obliged to declare any accident and more generally any incident putting at risk safety or quality, based on a regulatory list to be defined (door blocked, vandalism, aggression, forgotten luggage, etc.).
2. A realistic target should be officialised for market introduction, which is $10^{-5}$ injury/hour, for example, less than 1 accident during 1000 running hours in terms of active safety performance and less than 1 injury among 100 accidents for passive safety. After the continuous improvement process, at the end of the roadmap, the target should be at the estimated level of buses and coaches: 1 passenger injury +1 vulnerable injury for one million hours: not easy!

Required in the USA as identical to conventional vehicles which is demanding, the passive safety level should be high, as citizens will consider they have to be protected from external vehicles when installed in the minibus, and as pedestrians will refuse that the minibus front face would be less protective, or more aggressive, than a conventional car.

Today, conventional cars are equipped with a front bonnet over the thermal engine, which is very useful in case of pedestrian crashes. Without this bonnet constraint, these front faces will be vertical, to limit the occupied space and maximise place number, with high risk of pedestrian projection and crossing, even at low speed.

At low speed, the projection is not far: the pedestrian can fall just in front of the running automated minibus, limited in terms of braking deceleration because of standing passengers inside the minibus, which is also called the “tramway or trolley dilemma” (Exeter, 2020).

Passive safety innovations will have to be financed and developed for door-to-door transportation in pedestrian streets, and new principles have been explored, as an inflatable structure integrated in the vertical front face, giving a better kinematic for pedestrian crash with energy absorption.

For years, researchers have been waiting for regulation demand and industry financing, to protect pedestrians but also cyclists and scooterists in such situations, against a front face, relatively high, which is already the case for a child in front of an SUV.

## 4.4 The Critical Path for Market Introduction of Safe Automated Minibuses

More than active safety systems, the passive safety requirements are structuring for the vehicle architecture and have to be introduced at the early beginning of development planning.

Two different usages will have to be dissociated, as one of them is compatible with a conventional architecture:

1. Downtown usage with derogative access to pedestrian streets requires vertical faces with a specific architecture to protect vulnerable street users and to facilitate vehicle stop and go with persons with reduced mobility (PRM) and standing passengers. To allow such vehicle developments (low production volume, limited investments), passive safety requirements have to be adapted: hopefully, in that case, performances can be limited (speed, acceleration, braking, etc.).
2. On the contrary, the liaison usage between suburbs and cities at higher speed (or the rural usages) must reuse industrial vehicle platforms, with conventional passive safety requirements. For technical protection in case of crash, the best battery position is under the cockpit, with impact to the platform height and to the quick access, especially for elderly or for parents of young children: this reuse will reduce the critical path, with applications available in the near future.

To develop correctly and safely the door-to-door services in the town centres, we will need a specific architecture, a low robotised platform adapted to elderly and PRM access, accepting standing passengers, represented by our H2020 AVENUE demonstrators, but they did not have to be homologated in terms of vehicle safety.

The critical path is longer for such application, which is the most important societal target: the first commercial application will probably use new electric robotised platforms, already under development for duty applications, but these vehicles will not be so optimised as H2020 AVENUE demonstrators in terms of access and vulnerable protection.

Their lesson learned will be useful for the later introduction of specific vehicles for downtown transportation, everywhere at low speed, respecting new passive safety requirements, to be urgently defined and integrated in the vehicle certification process.

## 4.5 Quality and Safety State of the Art for Automated Minibuses

The AVENUE experimentations were too short to measure anything in terms of quality and safety, and only attractivity measurement and characterisation were planned and done.

In any case, these experimentations showed that four preliminary conditions were not met, where the state of the art has to be improved before market introduction:

1. The technical reliability is not able to guarantee a quasi-nominal behaviour, which has to respect the ISO 26262 (ISO, 2018) automotive standard, which is available and manageable.

2. This level of reliability has to be obtained with an acceptable maintenance level, which is not the case today for sensor cleaning, as an example. Another risk to be taken into account is the consequence of light material accidents if sensors are not well protected.

3. The robustness towards meteorology and environment factor components is not sufficient, especially with the lack of reliability of the connected infrastructure in the urban and complex environment.
4. The protection against cybersecurity attacks of connected vehicles, which has to be built very early in the software and hardware architectures, is distributed among many different actors: standards have to be established and their application controlled, since a connected automated vehicle can be turned into a weapon, inside the city.

When these four basics will be mastered or better satisfied in service, the guaranteed nominal behaviour of the minibus will have to be confronted to risky scenarios, coming from real situations and human behaviours.

These risky scenarios have to be identified as they are leading to incidents and sometimes to accidents: so in addition to the ISO26262 (ISO, 2018) standard, the safety of the intended functionality (SOTIF, ISO21448) (ISO, 2022) will have to be checked in front of real driving situations and real human behaviours.

To guarantee a positive result at the end of the development, these risky scenarios will have to be specified at the beginning, using a scenario library as built by the MOSAR project in France (SystemX, 2023b) or automated driving scenarios (ADScene) (SystemX, 2023a), so they will be taken into account in the design, which implies numeric simulation of these scenarios.

## 4.6 A Self-Learning Automated Transport System at European Level

As explained concerning safety targets, a continuous improvement process will be necessary, based on systematic declarations of incidents and scientific analysis at transportation system level (including human management), with anonymised feedback to all actors, at the European level.

The proposed improvement process will need the implication of the public transport authorities to demand a systematic incident declaration when giving agreements, to organise the confidential analysis, to share the lessons learned, to get the corrective actions and to adapt norms and regulations. This field experience will help to establish future norms or to improve regulations, in cooperation with technical representatives, ensuring technical feasibility.

The establishment of this improvement process under the control of a public authority will allow us to accept an intermediate level of safety for market introduction. The market introduction with paid services will begin the first quality measurements: for sure, corrective actions will be necessary and will benefit from the same improvement process, sharing lessons learned and best practices, to the benefit of all actors.
## 4.7 Data Privacy of Incident Analysis and Lesson Learned Sharing

The absence of a human driver introduces new risks in terms of vehicle safety and passenger behaviour: to manage these new risks and to avoid bad incidents which could lead to a rejection of the solution, it is necessary to add a specific process when an incident occurs, as depicted in Fig. 4.1.

An incident will be defined as a situation that could compromise the safety or the service quality and lead later to a rejection and most probably to expensive corrective actions. It can be an accident but also a harsh braking, a difficulty to close the door, a vehicle blocked, vandalism facts and an aggression.

In case of an incident, the data collection includes personal data concerning passengers and other road users but also private information coming from technical products, the automated vehicle and its environment.

To establish robust lessons learned, we need all available data and relevant technical skills, which is common with the road accident analysis, for road safety improvement: the data privacy has to be protected, taking into account relevant regulations, such as General Data Protection Regulation (GDPR) (EU, 2023b; EUR-Lex, 2016).

Road safety methodologies can be reused to conciliate data availability with privacy, as presented in Fig. 4.2, but also to satisfy commonly legal treatment needs with data needs for scientific analysis:

* Beyond the system daily management, incidents will be collected, and data will be protected and encrypted with restrained access.

* Data will be analysed by trusted experts in no-profit organisations: scenario coding, simulation and statistical analysis.

* The technical feedback will be shared, as anonymised, without any personal data and technical secret.

<figure><img src="image_64.png" /><figcaption>Fig. 4.1 Data collection process in case of incident</figcaption></figure>
<figure><img src="image_65.png" /><figcaption>Fig. 4.2 Lesson learnt process for anonymisation and sharing</figcaption></figure>

Justified by safety and included in the AVENUE vision recommended to the European Community, this continuous improvement process will contribute to both safety and service quality improvement.

## 4.8 Automated Minibus Safety and Service Quality Levers

In addition to the societal benefits, the general target of the introduction of automated minibuses in the urban transportation system is to offer the same level of safety and service quality as conventional transportation, which is very ambitious for the explained technical reasons and dependent on many human factors not well known today.

This is very ambitious, and it cannot be reached without the anticipated implication of the public power to prepare our countries and towns to their future challenges.

Six levers will have to be used by the public transportation authority at the European level, but not only, as many agreements are delegated at the local level:

1. Continuous and general improvement process, based on known methodologies.

2. Systematic and limited incident data collection, feeding risky scenario database.

3. Functional safety and SOTIF ISO norms (26262 & 21448) (ISO, 2018, 2022), using this database.

4. Appropriate effort repartition between active and passive safety, with trade-offs.

5. Dissociation of urban needs from liaisons at higher speed and different vehicles.

6. Innovation for urban shuttle architecture, especially for vulnerable protection.
From now, the regulation work has to include active but also adequate passive safety for two different usage modes.

Secured by the public control on the applications, this coming “automated minibus step” is a valuable investment for general interest: it will not only contribute to the climatic and ecologic transition, but it will contribute to the future life in our towns.

It will also prepare the introduction of private automated cars in towns: it can be later with benefits from this experience or never because of unacceptable risks.

## 4.9 Conclusion

In the advancing of automated minibuses, sustainability and durability are required to enable wider adoption within Europe. Several safety, security and service quality needs are deemed also mandatory to be addressed in order to attain broad acceptability of the new services. In this chapter, the conditions to achieve these objectives are presented. The critical path for market introduction of safe automated minibuses is also discussed, along with quality and safety state of the art for automated minibuses. The data privacy of incident analysis and lesson learned sharing is introduced to enable further discussions about the evolution of the public transport system in Europe. Finally, the automated minibus safety and service quality levers are presented. This analysis envisions to adequately facilitate the adoption of automated minibuses by providing specific guidelines.

**Acknowledgements** This project has received funding by the European Union's Horizon 2020 research and innovation program under grant agreement No. 769033.

## References

EU, E. C. (2023a, December). Buses and coaches. Retrieved from Mobility & Transport - Road Safety: https://road-safety.transport.ec.europa.eu/eu-road-safety-policy/priorities/safe-vehicles/archive/buses-and-coaches_en

EU, E. C.. (2023b, December). GDPR European regulation. Retrieved from General Data Protection Regulation (GDPR)—Official Legal Text: https://gdpr-info.eu/

EUR-Lex. (2016, April 27). Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Da. Retrieved from Document 32016R0679: https://eur-lex.europa.eu/eli/reg/2016/679/oj

Exeter, U. O. (2020, January 21). Trolley dilemma: When it's acceptable to sacrifice one person to save others is informed by culture. Retrieved from phys.org: https://phys.org/news/2020-01-trolley-dilemma-sacrifice-person-culture.html

ISO. (2018, December). Road vehicles—Functional safety. Retrieved from Standard Number: ISO 26262-1:2018: https://www.iso.org/standard/68383.html
ISO. (2022, June). Road vehicles—Safety of the intended functionality. Retrieved from Standard Number: ISO 21448:2022: https://www.iso.org/standard/77490.html

NHTSA, N. H. (2020, January). Automated driving systems. Retrieved from https://www.nhtsa.gov/vehicle-manufacturers/automated-driving-systems

NHTSA, N. H. (2021, June). Standing general order (National Highway Traffic Safety Administration (NHTSA)). Retrieved December 2023, from https://www.nhtsa.gov/laws-regulations/standing-general-order-crash-reporting

NHTSA, N. H. (2023, April). Standing general order 2021-01. Retrieved from https://www.nhtsa.gov/sites/nhtsa.gov/files/2023-04/Second-Amended-SGO-2021-01_2023-04-05_2.pdf

SystemX. (2023a, December 14). ADScene, a scenario database for the validation of automated driving and driving-assistance systems via digital simulation. Retrieved from https://www.irt-systemx.fr/en/adscene-a-scenario-database-for-the-validation-of-automated-driving-and-driving-assistance-systems-via-digital-simulation/

SystemX. (2023b). MOSAR platform (Methods and tools for operational safety assessment and robustness analysis of autonomous vehicles). Retrieved from Meeting the challenge of validating the autonomous vehicle: https://cahiers-transformation-numerique.irt-systemx.fr/accueil/designing-the-digital-world/meeting-the-challenge-of-validating-the-autonomous-vehicle-2/?lang=en

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

<figure><img src="image_66.png" /></figure>
<figure><img src="image_67.png" /></figure>

**Chapter 5**

# In-Vehicle Services to Improve the User Experience and Security when Traveling with Automated Minibuses

Dimitrios Tsiktsiris, Anastasios Vafeiadis, Antonios Lalas, Minas Dasygenis, Konstantinos Votis, Dimitrios Tzovaras, Christian Zinckernagel, and Kevin Salvi

**Abstract** This chapter looks at the innovative development and deployment of in-vehicle services to improve both the user experience and safety in automated minibuses, a critical component of future mobility-as-a-service (MaaS) ecosystems. With the rise of driverless public transport, ensuring the safety and comfort of passengers becomes a primary concern. The chapter discusses the integration of cutting-edge artificial intelligence (AI) and deep learning technologies to address these challenges, focusing on services such as enhanced safety and confidence, automated passenger presence monitoring, and intelligent feedback systems. It outlines the AVENUE project's initiatives to create an AI-powered framework that not only promotes the widespread adoption of automated minibuses but also addresses social and personal safety concerns in the absence of a human driver. The services discussed aim to replace the driver's monitoring and interaction functions with technology-enabled solutions, thereby maintaining service quality and promoting

<footnote>D. Tsiktsiris
Centre for Research and Technology Hellas, Information Technologies Institute,
Thessaloniki, Greece</footnote>

<footnote>Department of Electrical and Computer Engineering, University of Western Macedonia,
Kozani, Greece</footnote>

<footnote>A. Vafeiadis · A. Lalas (✉) · K. Votis · D. Tzovaras
Centre for Research and Technology Hellas, Information Technologies Institute,
Thessaloniki, Greece
e-mail: lalas@iti.gr</footnote>

<footnote>M. Dasygenis
Department of Electrical and Computer Engineering, University of Western Macedonia,
Kozani, Greece</footnote>

<footnote>C. Zinckernagel
Amobility, Copenhagen, Denmark</footnote>

<footnote>K. Salvi
Mobile Thinking, Geneva, Switzerland</footnote>
passenger confidence. Using real-world deployments and pilot studies, the chapter illustrates how these technological interventions contribute to the overall efficiency, safety, and acceptance of automated vehicles in urban environments and provide valuable insights into the future of automated public transport systems.

## 5.1 Introduction

Automated vehicles offer significant assistance to drivers across various levels through multiple means. During the last decade, public transportation systems are progressively integrating different levels of automated vehicles into mobility as a service (MaaS) within their services. During the last few years, fully automated vehicles started appearing in the market, able to operate without a driver in complex road conditions. In this context, as there is no driver, no assistance can be provided to the passengers, and routes and destinations will be automatically determined by the associated fleet management system, based on the passenger trip requests. The ultimate goal of mobility as a service (MaaS) is to provide comprehensive, efficient, and seamless transportation solutions to users. MaaS aims to integrate various forms of transportation services into a single, accessible platform that allows users to plan, book, and pay for their journeys using different modes of transport, such as buses, trains, taxis, rideshares, bikes, and even micromobility options like scooters.

In our vision in the AVENUE project, one of the most promising mobility modes for MaaS services will be the use of fully automated vehicles (e.g., minibuses), providing the glue for all other transport modes of a successful MaaS in an urban environment.

However, the transition to fully automated public transportation vehicles is not seamless, and several obstacles arise in a real-world scenario.

Several concerns among end users are associated with the safety and reliability of fully automated vehicles (AVs), directly impacting the acceptance of this new technology. Two major concerns for potential passengers involve automated shuttle sharing with other passengers and trust in the technology itself. The absence of a driver in the bus raises various possible issues: for instance, no immediate personnel would be present to administer first aid in case of emergencies, and passengers might feel uneasy being alone, especially at night or in certain neighborhoods.

Additionally, the lack of an authoritative figure onboard could be a concern, especially when transporting groups such as schoolchildren, potentially leading to incidents like vandalism, theft, or altercations among passengers. With the removal of the driver and the introduction of external intervention teams, questions arise about how the formal and informal services provided by the driver could still be upheld while maintaining the current high-quality service standards.

Addressing these social and personal safety and security concerns within automated vehicles necessitates the implementation of specific services to replace the driver's presence. Within the AVENUE project, various IT-based solutions and
services have been identified and developed. These solutions aim to provide pas-
sengers with similar services to those offered by a driver onboard. This includes the
creation of a novel artificial intelligence (AI)-supported framework that facilitates
the widespread adoption of these services. This approach shows great promise in
significantly enhancing safety and security levels in automated public transportation
and bolstering passenger trust.

The services that will be presented in this chapter are the following:

* Enhance the sense of security and trust

* Automated passenger presence

* Follow your kid/grandparents

* Shuttle environment assessment

* Smart feedback system

The reason for choosing these services is twofold: (a) They are all essential services necessary to remove the safety driver from the shuttle, as safety and efficient operation are critical factors. For example, ensuring safety for the travelers have been identified from end users as the single most important factor for choosing to ride with the automated shuttle. (b) They are all based on the same foundation of technology with cameras, sensors, and algorithms, meaning that testing multiple types of services is feasible, once the necessary equipment has been installed. The analysis is also similar, and therefore the results can be compiled together.

## 5.2 Service: Enhance the Sense of Security and Trust

The service “enhance the sense of security and trust” aims to address the new reality that is formed in automated shuttles’ mobility infrastructures with the absence of a bus driver and the threat from criminal activities in European cities. Typically, drivers are trained to handle incidents of passengers’ abnormal behavior, such as incidents of petty crimes, according to standard procedures adopted by the transport operator. Surveillance using sensors such as cameras (cameras of different technologies can be used so that passengers’ privacy is protected) and microphones, as well as smart software in the bus, will maximize the feeling of security and the actual level of security. However, several concerns of the end users regarding the safety and robustness of the AVs, where a driver is not present, which are directly linked to the final user acceptance of the new technology, can be identified. Prospective passengers harbor apprehensions about potential scenarios in the absence of a driver in the bus, including:

* No one will be in the bus to perform first aid if required

* Feeling of discomfort being all alone in the bus at night, especially in certain neighborhoods

* No authority figure present to keep passengers calm (e.g., schoolkids)

* Vandalism, bag snatching, indoor fighting, and unaccompanied luggage
To address the aforementioned concerns on social and personal safety and security into the vehicle, certain measures need to be implemented. For example, the detection of unaccompanied luggage and of other personal belongings may raise a notification or an alert to the supervisor and/or the suitable authorities. This may be followed by appropriate notifications and/or instructions to the passengers, while the vehicle may also implement respective actions. Moreover, implementing a solution for enhancing the safety and security inside the automated buses will support safekeeping not only the users of the automated public bus but also the vehicle itself.

This section details the implementation of a video and audio analytics software module designed for an embedded security subsystem or for the cloud-based services within the system. Additionally, it covers the deployment and testing of this service at the pilot sites of the AVENUE project.

The service addresses the timely, accurate, robust, and automatic detection of various petty crime types or misdemeanors as well as the assistance of authorized end users toward the reidentification of any offenders. A misdemeanor is any “lesser” criminal act in some “common law” legal systems. Misdemeanors are generally punished less severely than felonies but theoretically more than administrative infractions (also known as minor, petty, or summary offences) and regulatory offences. Many misdemeanors are punished with monetary fines. The petty crimes that are targeted for identification by the sensors include petty theft like bag snatching and pickpocketing, vandalism, aggression, illegal consumption of cigarettes, public intoxication, simple assault, and disorderly conduct. These are explained in more detail:

*   **Petty theft:** Theft is the taking of another person's property or services without that person's permission or consent with the intent to deprive the rightful owner of it.

*   **Vandalism:** Vandalism is the action involving deliberate destruction of or damage to public or private property.

*   **Aggression:** Aggression is overt or covert, often harmful, social interaction with the intention of inflicting damage or other unpleasantness upon another individual.

*   **Public intoxication:** Public intoxication, also known as “drunk and disorderly” and drunk in public, is a summary offense in some countries rated to public cases or displays of drunkenness.

*   **Simple assault:** An assault is the act of inflicting physical harm or unwanted physical contact upon a person or, in some specific legal definitions, a threat or attempt to commit such an action.

*   **Disorderly conduct:** Disorderly conduct makes it a crime to be drunk in public, to “disturb the peace,” or to loiter in certain areas.

In the “enhance the sense of security and trust” service, two distinct petty crime detection approaches are implemented: video analytics and audio analytics. The video analytics approach supports end-to-end detection of abnormal events; achieves real-time inference on modern hardware; offers flexibility with supervised, unsupervised, and semi-supervised learning to compensate the scarcity of data in the security domain; supports multiple camera types, positions, and angles; and is able to
operate in embedded setup with limited power requirements. On the other hand, the
audio approach uses information from the acoustic sensors of the shuttle for abnor-
mal event detection by comparing different spectrogram representations and focus-
ing on the effect of signal-to-noise ratio (SNR) to audio recognition and the potential
of the generalization of a model in different SNR settings and datasets collected
under different environments. More specifically, for the video analytics approach, a
pose classification approach is developed that classifies the extracted skeleton key
points from each dataset (Tsiktsiris et al., 2020). For training the proposed model,
five distinct datasets were used, including data simulated in lab in CERTH/ITI facil-
ities, data captured from Geneva Public Transport (TPG) shuttles in Geneva and
HOLO shuttles in Copenhagen, data from the P-REACT project, the NTU-RGB-D
(NTU-RGB-D is the name of the dataset from Nanyang Technological University
(NTU) (https://www.ntu.edu.sg/rose) featuring RGB (Red-Green-Blue/color
images) and D (Depth)) dataset by ROSE lab, and the UCSD Anomaly Detection
Dataset. Three different models were tested for pose classification, namely, a
stacked bidirectional long short-term memory (LSTM) network classifier, a spatio-
temporal autoencoder, and a spatiotemporal LSTM classifier. The first approach
consists of a stacked LSTM model as a classifier. An overview of the pipeline is
depicted in Fig. 5.1. Overall, the classification is performed in four stages: (a) In the
first stage, pose estimation techniques are applied to obtain skeleton key points. The
generated pose proposals are refined by parametric pose non-maximum suppression
to obtain the estimated human poses. (b) In the second stage, tracking is performed
to match cross-frame poses and form pose flows. (c) In the third stage, features are
generated from the detected and tracked human body key points and are forwarded
into the network (d), which classifies the action into normal or abnormal.

<figure><img src="image_68.png" /><figcaption>**Fig. 5.1** Pipeline of the pose classification (Tsiktsiris et al., 2020)</figcaption></figure>
For the pose estimation stage, the regional multi-person pose estimation (RMPE) by Fang et al. (2017) was adopted with a pretrained VGG19 backend, a convolutional neural network model proposed by Simonyan and Zisserman (2014). In the tracking stage, a matching of cross-frame poses and form pose flows is performed, using a real-time algorithm that is based in a distance matrix. In addition, a pose flow non-maximum suppression is applied, in order to reduce unnecessary pose flows and relink temporal disjoint ones. This is an important step that associates poses indicating the same person across multiple frames. A skeleton tracking algorithm was implemented, in order to meet the performance requirements of a real-time service. The algorithm is sorting the skeletons based on the distance between neck and image center, from small to large. Certain heuristics are taken into consideration, such as the position of the joints, the average height of the person, and the height difference between frames. Height variation improves the ability of the algorithm to understand depth, since the analysis is based on two-dimensional input. Such parameters of the algorithms are optimized for in-shuttle space and fine-tuned to specified weights based on the camera calibration. A skeleton near the center will be processed first and be given a smaller human ID. Later on, each skeleton's features are matched based on its previous and current frame. The distance matrix (or cost) between the skeleton joints is the main criterion for the matching function. Skeletons with the less distance are paired between the frames and are given the same ID (Fig. 5.2).

<figure><img src="image_69.png" /><figcaption>**Fig. 5.2** Skeleton matching across two subsequent frames (blended) (Tsiktsiris et al., 2020). Notice that the passenger ID, highlighted in green at the left of each bounding box, is the same across the frames</figcaption></figure>
For the spatiotemporal autoencoder, there are two stages to be formed: encoding and decoding. Autoencoders set the number of encoder input units to be less than the input; thus, they were first used to reduce dimensionality. Usually, unsupervised backpropagation is used for training, helping the reconstruction error of the decoding results from the original inputs to decrease. Generally, an autoencoder can extract more useful features when the activation function is nonlinear rather than some common linear transformation methods, such as principal component analysis (PCA).

In order to learn the regular events in training data, a spatiotemporal autoencoder was introduced. In particular, the spatial autoencoder consists of an encoder and a decoder that are composed of two convolutional and transpose convolutional layers, respectively, whereas the temporal encoder is comprised of three convolutional LSTM layers, as depicted in Fig. 5.3.

Finally, it was observed that even if the model is trained on thousands of data, some false positives will still be observed in certain occasions. As a result, it is possible to manually shift through the anomaly outputs and flag some of them as false positives, in order to let the previous autoencoder neural network model act as a high recall. Semi-supervised learning is employed by decreasing the threshold, so that the majority of true anomalies (high recall) can be detected, as well as other false positives (low precision). To achieve the semi-supervised approach, a new model has been designed, which includes the previous encoder and an LSTM, which acts as a classifier as depicted in Fig. 5.4.

<figure><img src="image_70.png" /><figcaption>**Fig. 5.3** Model architecture of the autoencoder. The first two convolutional layers are spatial encoders, followed by temporal encoder and decoder. Between them, a ConvLSTM with reduced filters is used as a bottleneck to eliminate non-useful information. At the last two layers, spatial decoding is performed, reconstructing the input image to the same format (Tsiktsiris et al., 2020)</figcaption></figure>
<figure><img src="image_71.png" /><figcaption>**Fig. 5.4** Model architecture of the hybrid model. The red container contains components of the previous autoencoder approach. The green components indicate the new hybrid model which acts as a classifier (Tsiktsiris et al., 2020)</figcaption></figure>

The pose classification approach was tested on the NTU RGB + D dataset (Fig. 5.5a-c) and on the TPG dataset captured by CERTH inside the AV's shuttle (Fig. 5.5d, e). In the images, there are various debugging layers enabled, such as skeleton points, lines, tracker ID, and bounding boxes of each detection. The predicted result is marked as green when the classifier indicates it as “normal” and red when “abnormal,” correspondingly. So far, NTU dataset samples were not included in the training set, so it is safe to assume that the derived model can generalize across different people, view angles, and events. Figures 5.6, 5.7, and 5.8 depict the aforementioned conditions and use cases.

For the audio analysis (Papadimitriou et al., 2020), the different procedures implemented include spectrograms, single-channel representation, multichannel representation, and transfer learning, and the dataset used for sound events classification (glass breaking, gunshot, screaming) is the MIVIA Audio Events dataset. The three spectrogram representations used were namely the single-channel short-time Fourier transform (STFT) (Salamon & Bello, 2017), the mel scale, and the mel-frequency coefficients (MFCCs) (Zhang et al., 2015).

The experimental results in Fig. 5.9 showed that the MFCC is able to generalize better than the STFT spectrogram and the mel spectrogram, when comparing single-channel representations. This most probably owes to the fact that this representation includes all the important information of the audio signal in the lowest MFCC features (e.g., first ten features) with regard to the concentrated energies and has minimum changes in the highest ones. Hence, it is suggested that it has its place in a
<figure><img src="image_72.png" /><figcaption>**Fig. 5.5** Evaluation on test data: (a–c). Abnormal event detection (violence/passengers are fighting) using different camera angles from the NTU-RGB dataset (d, e). Detection of fighting/bag snatching real-world scenarios inside the shuttle (Tsiktsiris et al., 2020)</figcaption></figure>

<figure><img src="image_73.png" /><figcaption>**Fig. 5.6** Evaluation on multiple camera angles, excessive occlusion, and partial presence (Tsiktsiris et al., 2020)</figcaption></figure>

feature representation combination, and for that reason, it was indeed used in both methods of multichannel representation. With regard to the multichannel representation (Fig. 5.10), the stacked features method proved to be more generalizable compared to the concatenated features method, especially when training was
<figure><img src="image_74.png" /><figcaption>**Fig. 5.7** Evaluation across various scenarios (left to right): Bag snatching, fighting, vandalism, and unaccompanied luggage (Tsiktsiris et al., 2020)</figcaption></figure>

<figure><img src="image_75.png" /><figcaption>**Fig. 5.8** Additional evaluation on the NTU-RGB dataset. Metrics at the top left depict the prediction scores for the P1 (Tsiktsiris et al., 2020)</figcaption></figure>

carried out on higher signal-to-noise ratios (SNRs) and testing was carried out on lower ones. Neither the concatenated features method nor the separate single-channel spectrogram representations (STFT, mel, and MFCC) performed as well.

Finally, in Fig. 5.11, the generalization capabilities of the two multichannel methods are shown in terms of event-based recognition (GB, G, and S). As one moves along the sequence of the ten models, it is evident that the generalization capabilities of the stacked multichannel method are significantly better than the corresponding concatenated multichannel method. In both cases, the model that was trained in -5 dB and tested in 15 dB showed the best performance, with a recognition score of 91.51% for the concatenated method and 90.23% for the stacked method, with the lowest standard deviation, namely, 0.034 and 0.019, respectively.

Moving up in terms of SNR training (and model number), it became more difficult to generalize, especially in the case of zero and below SNRs. This is due to the fact that the lower SNR audio contains higher levels of noise and thus is more challenging, leading to more robust and generalizable classification.
<figure><img src="image_76.png" /><figcaption>**Fig. 5.9** Frame-by-frame recognition rate (RR) for all models using concatenated features from STFT, mel, and MFCC spectrograms (single channel) validated for each signal-to-noise ratio (SNR): each column group refers to a specific model (Papadimitriou et al., 2020)</figcaption></figure>

<figure><img src="image_77.png" /><figcaption>**Fig. 5.10** Frame-by-frame RR for all nodes using stacked features from STFT, mel, and MFCC spectrograms (multichannel) validated for each SNR: each column group refers to a specific model (Papadimitriou et al., 2020)</figcaption></figure>
<figure><img src="image_78.png" /><figcaption>**Fig. 5.11** A comparison of the models trained with the concatenated (top) and stacked (bottom) features method with regard to event-based RR: the sequence of the models increases from 1 to 10 (Papadimitriou et al., 2020)</figcaption></figure>

## 5.3 Service: Automated Passenger Presence

The service “automated passenger presence” aims to address a basic problem of operators’ services, which is related to the occupation of their vehicles, as well as the awareness of the number of people onboard in order to schedule the routes. Furthermore, the passengers would like to know in advance if there is an available seat or enough space on a shuttle to plan their boarding. Traditionally, but also
nowadays, passenger counting is conducted manually via passenger surveys or human ride checkers. Typically, the driver or inspectors are responsible for performing enumeration of the onboard passengers, something not feasible in an automated shuttle. Automatic passenger counting has been rapidly emerging in recent years to address similar needs. An automated system is introduced capable to detect passenger presence in real-time with high accuracy, count onboard passengers, and calculate vehicle occupancy. Surveillance using sensors such as cameras (cameras of different technologies can be used so that passengers' privacy is protected) and smart software in the bus will automate the detection of passenger presence.

Several concerns of the end users regarding the safety and robustness of the automated vehicles that are directly linked to the final user acceptance of the new technology can be identified. The prospective passengers may deal with several possible instances that could arise in case there is no staff inside the shuttle. Indicatively:

* No one will be in the bus to count the number of passengers with regard to the shuttle's capacity.

* There are continuous stops throughout the entire route, even in cases where the shuttle is fully occupied.

* No authority figure is present to alert passengers of their designated bus stop.

To address the aforementioned concerns on social and personal safety and quality of service into the vehicle, certain measures need to be implemented. For example, counting the number of passengers being inside the automated vehicle could help avoid overcrowding in the shuttle, as well as meaningless stops in cases where the bus is in full capacity. This may be followed by appropriate notifications and/or instructions to the passengers, while the vehicle may also implement respective actions.

The service provides a video analysis of the vehicle internals, using the onboard camera, in order to identify the vehicle occupation, vehicle free space, as well as counting people onboard. Automatic assessment of space occupation using the onboard cameras is enabled. Capacity is set as an absolute number of space units. For example, each space unit is associated with one standing passenger. Occupancy is set as an absolute number of space units currently in the shuttle. For the operation manager, occupancy is visible on the dashboard of the AVENUE platform, whereas occupancy is displayed as real-time information via the AVENUE mobile app, wherever the traveler is. Each passenger (normal, big size, wheelchair user, seated) can determine whether he/she can fit in or not. Assessment for different cases can be provided to assist the passengers on determining whether to request onboarding or not. Automatic counting of people using the onboard cameras is also provided. Moreover, occupancy marked with information for the different user cases is displayed as real-time information, wherever the traveler is; however it does not guarantee them a free spot by the time the shuttle reaches the station of their choice.

The implementation of a video analytics software module for an automated passenger presence counting subsystem or for cloud-based services of the system is described along with deployment and test of the service into the pilot sites of the AVENUE project, where the following use cases have been identified to be further examined and addressed:
*   **Passenger counting:** The automated shuttle has a fixed capacity regarding the number of passengers it can carry. The video cameras installed in the automated shuttle acquire the color depth images, and the data are fed into the system's video analytics algorithms for further analysis. In case the algorithms identify that the total number of passengers is reached, the shuttle stops receiving any others, and appropriate notifications are sent to the AVENUE mobile app for the passengers that would like to board.

*   **Route optimization:** Even though the shuttle is in full capacity, there may still be people waiting on a bus stop to go aboard. The bus only makes a stop when a passenger needs to get off, while the route is modified to save time and cost. The number of onboard passengers is always monitored, so that the new passengers could get on, in case of availability.

*   **Passenger awareness:** Even though the automated vehicle has reached its terminal, there could still be passengers onboard. The shuttle counts the number of passengers to make sure there is no one left. If there are passengers, the bus alerts them to get off.

For the “automated passenger presence” service, a deep learning-based distance assessment service is proposed that uses an overhead perspective, which is able to function with high accuracy and low-power consumption in confined spaces, such as the inside of the automated shuttle (Tsiktsiris et al., 2022). For this purpose, a fisheye wide-angle camera with a top-down perspective is used. In order to timely and accurately detect human items, a pretrained RAPiD model was implemented that outputs bounding box coordinates used for computing their centroids.

More specifically, as already mentioned, the network architecture is inspired by RAPiD (Duan et al., 2020) and therefore consists of three stages: the backbone network, the Feature Pyramid Network (FPN), and the bounding box regression network. The backbone network works as a feature extractor that takes an image as input and outputs a list of features from different parts of the network. In the next stage, these features are passed into the FPN, in order to extract features related to object detection. Finally, at the last stage, a convolutional neural network (CNN) is applied to each feature extractor in order to produce a transformed version of the bounding box predictions, as depicted in Fig. 5.12.

<figure><img src="image_79.png" /><figcaption>**Fig. 5.12** An illustration of multiple convolutional layers and multidimensional matrices, such as the feature maps with 1024 × 1024 input resolution (Tsiktsiris et al., 2022)</figcaption></figure>
<figure><img src="image_80.png" /><figcaption>**Fig. 5.13** Results on unseen scenarios from a real pilot site and the BOSS dataset. Green lines represent a safe distance, while red lines an unsafe one (Tsiktsiris et al., 2022) (Color figure online)</figcaption></figure>

Furthermore, the Euclidean distance formula is also implemented to compute the pairwise distances for each bounding box. This distance is later transformed into the real distance between passengers by multiplying it with a weight value that is defined via the camera calibration.

Experimental results indicated that the service efficiently identifies passengers with unsafe proximity, according to COVID-19 regulations (passengers must keep a distance of 1 m apart (Olivera-La Rosa et al., 2020)), as depicted in Fig. 5.13.
However, handling reflections is still a challenge on certain scenarios, as also happens in similar approaches. Passenger figures might appear to the windows of the shuttle as reflections, especially when the lighting is low. To mitigate this issue, a custom mask is applied on large reflective surfaces.

## 5.4 Service: Follow My Kid/Grandparents

The service “follow my kid/grandparents” is designed to increase autonomy of partly autonomous people (kids, grandparent(s), disabled people). It will allow carriers or family members to be sure that their beloved family members are safe while moving around the city using public transport. On the other hand, it will increase confidence to the non-fully autonomous people to use public transports knowing that their family can “be with them.” Surveillance using sensors such as cameras (different technologies can be used so that passengers’ privacy is protected) and microphones, as well as smart software in the shuttle, will maximize the feeling of security and the actual level of security. The prospective passengers fear several instances that could arise if there is no driver in the bus:

* Passengers feeling discomfort travelling alone during nighttime

* Parents not being able to know if their kids have reached their destination safely

* Caregivers not being able to track passengers with dementia or other health issues

To address those concerns on social and personal safety and security, certain measures need to be implemented. For example, third parties monitoring the route of minors or passengers with health issues could make their route much easier and less frightening. This may be followed by appropriate notifications and/or instructions to the third party, while the vehicle may also implement respective actions. Moreover, implementing a solution for monitoring the routes of kids and patients will support safekeeping not only the users of the automated public shuttle but also the vehicle itself. The implementation of a video and audio analytics software module for an embedded security subsystem or for cloud-based services of the system is described along with appropriate the deployment and test of the service into the pilot sites of the AVENUE project.

The service and scenario propose a full-fledged solution that allows designated “guardians” to follow the automated public transport journeys of more vulnerable people, since the guardians can check the trip via a dashboard or mobile app, receive notifications, add people to their “guarded” list, and share trips/positions and estimated time of arrival (ETA) with others. In the context of AVENUE project, the following use cases have been identified to be further examined and addressed:

* Passenger monitoring: Travelling without a guardian during nighttime can be unsettling for a vulnerable person.

* Kids monitoring: Parents need to be able to track their kids.

* Patients monitoring: Caregivers need to track their patients, especially when they are not able to commute on their own.
The video cameras installed in the automated shuttle record the color images, and the data are fed into the system’s video analytics algorithms for further analysis. When the automated bus’s system identifies the passenger/kid/patient, the tracking begins, and the parents/caregivers can monitor their route.

For the “follow my kid/grandparents” service, an end-to-end service is developed based on deep learning models, for automated facial recognition inside the automated shuttle (Tsiktsiris et al., 2021). The techniques introduced in this service are based on attention to mitigate the occlusion issues introduced by face masks during the COVID-19 pandemic. More specifically, the first layer of the sensors connects to the hardware abstraction layer (HAL) that implements the IP and the USB cameras, respectively, but also requests raw data by the API end points to perform face recognition. The input data are then converted and transformed in a compatible format and passed into the analytics algorithms. The result is transferred via the API end points into the cloud. The user has access to the data and acts accordingly. A new passenger can be enrolled to the service using a single image of his/her face, which will be stored in a database, and by using it as a reference, the network will calculate the similarity of any new instances presented to it. An overview of the service is presented in Fig. 5.14.

As for the video analysis, facial recognition techniques identify human faces in images or videos by measuring specific facial characteristics. The extracted information is later combined to create a facial signature or a profile. When used for facial verification, a camera frame is compared to the recorded profile. More specifically, a multi-task cascaded convolutional network (MTCNN) (Zhang et al., 2016) receives the input frame to extract and align facial images. The facial images are then preprocessed and passed into a feature extractor (CNN backbone) linked with the explainable cosine (xCos) module that features an explainable cosine metric.

As current face verification models use fully connected layers, spatial information is lost along with the ability to understand the convolution features in a human sense. To address this obstacle, the plug-in xCos module is integrated as described below and depicted in Fig. 5.15, while experimental results are illustrated in Fig. 5.16.

<figure><img src="image_81.png" /><figcaption>**Fig. 5.14** Overview of the “follow my kid” service (Tsiktsiris et al., 2021)</figcaption></figure>
<figure><img src="image_82.png" /><figcaption>**Fig. 5.15** Network pipeline: the two input images are preprocessed and passed into the backbone CNN for feature extraction along with the plugin xCos module (Tsiktsiris et al., 2021)</figcaption></figure>

<figure><img src="image_83.png" /><figcaption>**Fig. 5.16** The Follow-my-kid service detecting passengers in real time inside the shuttle</figcaption></figure>

* **Input:** The two input images are preprocessed and passed into the feature extractor. Input A is the database image, while Input B is the image cropped from the video stream.
* **Backbone:** The same CNN feature extractor is implemented as in ArcFace (Deng et al., 2019). However, to employ the xCos module, the last fully connected layer and the previous flatten layer are replaced with a 1 × 1 convolutional layer.
* **Lcos calculation (xCos):** Patch-wise cosine similarity is multiplied by the attention maps and then summed to calculate the Lcos.
## 5.5 Service: Shuttle Environment Assessment

The service “shuttle environment assessment” aims to maintain the environmental conditions in the automated vehicle that may not be adequately controlled due to the absence of the shuttle driver at acceptable levels. Minimum acceptable conditions and comfort, such as good air quality, acceptable odors, and absence of smoke, are necessary for the safe transport of passengers, as well as the viability of the whole automated service, since lack of these conditions within the vehicle could significantly discourage potential users. After all, monitoring the environmental conditions could enable passengers’ alert and warning services via notifications, thus enhancing the user experience and safety during their trips. Under these circumstances, there are several instances that must be considered in order for the prospective passengers to feel content and safe. While there would be no driver inside the vehicle, various problems might come up, such as the following:

* There will be no staff inside the bus to prevent someone from lighting a cigarette.

* In quite high or low temperatures, there will be no driver in order to regulate the air conditioning or heating system.

* In emergency situations, there will be no one in charge of informing the operators and the competent authorities.

* If the air concentration of CO₂ inside the bus is high, and someone might get dizzy or exhibit breathing difficulties, there will be no driver so as to either open the windows or stop the shuttle.

As far as it is considered, the buses should create a comfortable environment for all the passengers. This feeling could undoubtedly be strengthened by controlling the temperature inside the vehicle. Besides, heating, ventilation, and air conditioning control now belong to the standard equipment of city buses. As a result, it is crucial for temperature sensors to be positioned in specific locations inside the vehicle. Simultaneously, these sensors will be connected to the air conditioning system, and if the humidity/temperature exceeds a suitable limit, the air condition will be put into operation. In that way, the existing humidity/temperature will be automatically adjusted, to provide the appropriate indoor climate, neither too hot nor too cold for the passengers. In addition, detection of certain pollutants, such as CO₂, NO₂, or dust particles in the indoor environment, along with critical temperature variations, is critical for the condition of certain passengers, especially ill people, such as asthma patients. Smoke in the vehicle, i.e., from a person that lights a cigarette, will deteriorate the passenger experience but may also put the whole vehicle in danger (fire hazard). Detection of certain events (air quality deterioration, smoke) may raise a notification or an alert to the passengers along with instructions on how to handle this situation, while the vehicle may also implement respective actions. Moreover, smoke in the vehicle may also result in cancelling the automated transport service. Detection of certain events (air quality deterioration, smoke) may raise a notification or an alert to the supervisor and/or the suitable authorities (i.e., police, fire department). This may be followed by appropriate notifications and/or instructions to the passengers, while the vehicle may also implement respective actions.
The service is responsible for the timely, accurate, robust, and automatic detection of any change in the air quality and the presence of smoke or fire, inside the vehicle. In cases where there will be an alert on the system, notifications and instructions will be sent to the passengers, to the operators, and/or to the suitable authorities. Considering this, several possible situations could take place in counting from high-level CO₂ and NO₂ concentrations at the indoors air composition to presence of humidity, smoke, or even fire. Especially, exposure to carbon dioxide can produce a variety of health effects (Azuma, 2018). These may include headaches, dizziness, restlessness, a tingling or pin or needle prick feeling, difficulty breathing, sweating, tiredness, and increased heart rate. Furthermore, detection of certain air quality indexes and pollutants in the indoor environment, along with critical temperature variations, is necessary for providing a secure service to the passengers. A gas composition sensor could be used for checking the inside air quality. This sensor monitors the air intake of the heating, ventilation, and air conditioning system of the vehicle, while it detects undesirable gases and adjusts the system accordingly by shutting off the intake and recirculating the indoor air back to the outside. Despite that, another possible scenario, mostly observed during the rainy days of the year, is the fogging of the windows due to the increased humidity. This might have a negative impact on the passengers' attitude while reinforcing feeling of confinement. Consequently, including a fogging prevention sensor inside the vehicle might be an efficient solution. More specifically, fogging prevention sensors are used to prevent fogging of the windshield glass. These sensors consist of three sensing elements for sensing indoor temperature, windshield glass temperature, and cabin humidity. The fogging sensor feedback is used for adjusting the heating, ventilation, and air conditioning system to maintain the interior temperature higher than the windshield glass temperature. Hence, it prevents the windshield from fogging up.

To summarize, it is passengers' wish to travel in a clean and comfortable environment and be notified if the conditions deteriorate. Moreover, operators would like to be notified when environmental conditions are considered harmful for the passengers. Representative use cases are indicatively displayed as follows:

* Lighting a cigarette: Inside the shuttle, a passenger lights a cigarette. The smoke detection sensors detect the smoke coming out of the cigarette. The real-time sensor data is sent to a central PC, installed in the vehicle and in which the data processing takes place. With the real-time data processing, the PC decides that it is an emergency case and sends the message of smoke detection to the operators. The operator evaluates the criticality of the situation and decides how to intervene (with an announcement from the loudspeakers or by taking more drastically measures, i.e., stopping the bus).

* Exposure to carbon dioxide: While the bus is on its route, high levels of CO₂ are detected from the relevant sensor. The sensor data is sent in real time to the central PC and is processed. CO₂ in unusual levels of air concentrations might have an adverse effect on passengers' health. For instance, high levels of CO₂ are related to dizziness, restlessness or breathing difficulties, and increased heart rate. To prevent an event concerning these health issues, such as a fainting, the PC sends
<figure><img src="image_84.png" /><figcaption>Fig. 5.17 Environmental assessment process</figcaption></figure>

the vehicle's central system the command to open the windows, so that the air is refreshed and can come back to its normal composition. Also, the passengers are informed of the air composition through the mobile application in real time.

* High temperature on the automated vehicle: The sensor measures the temperature and sends the data to the central PC. When the temperature exceeds a pre-defined level, the PC sends the command to start the cooling system. At the same time, passengers can be informed for the temperature inside the vehicle through the mobile application.

For the “shuttle environment assessment” service, a set of sensors was used to determine the air quality inside the automated shuttle environment, as well as to detect any passengers smoking and to prevent any fogging, as depicted in Fig. 5.17. After the sensors were deployed, they measured several metrics, such as CO₂, NO₂, humidity, temperature, fog, dust, and smoke, for a sufficiently long period of time. The collected values are used to predict the indoor conditions regarding the air quality for the next couple of hours. These measurements, as well as the conclusions of the real-time assessment, are sent to both the passengers and the operators of the shuttle. In case of an alarm, such as possible fire, they are sent to the suitable authorities. Moreover, this may be followed by appropriate notifications and/or instructions to the passengers, while the vehicle may also implement respective actions.

## 5.6 Service: Smart Feedback System

The service “smart feedback system” aims to allow the travelers inside the shuttle to give easy and effortless feedback to the operators when the safety driver is no longer inside the shuttle. It is important for the operators to know if people are satisfied with the services and transportation. Currently, the safety driver talks to the travelers and with his/her presence also becomes the conversation channel between the operators and the travelers. When the safety driver is removed, knowing whether they are satisfied or disappointed can be even more important, as the safety driver is not there to support, hence assisting the travelers.
When removing the operator, automated services must perform the same level of service and interaction as he/she did while being in the shuttle. This service aims to allow travelers to give their feedback about the service experience as easily as possible. This will be done by instructing the travelers to give a hand gesture to one of the cameras inside the shuttle. This will allow the travelers to effortlessly say “I like” or “I don’t like” the experience with a thumbs up or a thumbs down. The concept will be communicated to the travelers via stickers inside the shuttle. Camera technology is used to capture the thumbs up or thumbs down, but, if possible, sound sensors will also be tested to capture the experience/feedback from the travelers. The service will be communicated to the travelers as follows:

* Giving a thumbs up/down in light settings: Midday with sunlight. Good visibility for the cameras.

* Giving a thumbs up/down in dark settings: Early morning or night with no sunlight. Low visibility for the cameras.

* Giving a thumbs up/down in crowded settings: Many passengers inside the shuttle, both standing and seating. Low visibility for the cameras due to people standing close to the cameras.

* Giving thumbs up/down in empty settings (or few passengers): Little or no passengers inside the shuttle. Good visibility for the cameras, easy to see the hand gesture.

For the “smart feedback system” service, the model is trained end to end and regularized so that it distills the most compact profile of the normal patterns of training data and effectively detects the “thumbs up” and “thumbs down” gestures. The original images of the hand gestures are acquired through the USB camera that is inside the shuttle and then passed through a single-shot detector (SSD) for the detection of the bounding box of where the hand is and the corresponding cropped frame. The cropped frame of the hand is then passed to the CNN that predicts a class vector output of values between 0 and 1, as illustrated in Fig. 5.18. These values correspond to the probability of the frame to be one of the classes. Real-time results are depicted in Fig. 5.19.

The results were validated during a twofold live session. Staff from HOLO performed fighting, bag snatching, falling, and vandalism scenarios in vehicle P109

<figure><img src="image_85.png" /><figcaption>Fig. 5.18 CNN architecture for thumb orientation detection</figcaption></figure>
<figure><img src="image_86.png" /><figcaption>**Fig. 5.19** The smart feedback service operating in real time inside the shuttle</figcaption></figure>

<figure><img src="image_87.png" /><figcaption>**Fig. 5.20** Maintenance front-end interface regarding the service “enhance the sense of security and trust”</figcaption></figure>

that serves the route Slagelse in Denmark. The algorithms were able to correctly identify the performed scenarios with 89% accuracy, and the appropriate notifications were also captured in the operator's dashboard. The two dashboards were correctly synchronized regarding real-time event detection. Figures 5.20 and 5.21 show the maintenance and operator dashboards regarding the results for the service “enhance the sense of security and trust.” The validation of the automated passenger counting has been completed through manual comparison with data received from the operator phone's data stream. The automated count is considered precise in
<figure><img src="image_88.png" /><figcaption>**Fig. 5.21** Operator dashboard regarding the service “enhance the sense of security and trust”</figcaption></figure>

terms of timestamp and accurate in terms of the passenger count. Follow my kid, shuttle environment assessment, and smart feedback services were also validated successfully during the live session.

## 5.7 Conclusion

The transport service quality and passengers' safety and comfort are the major pre-occupation of the public transportation operators. Thus, the automated AI-supported in-vehicle services will eventually substitute the role of the safety driver and the essential functions he/she currently provides inside the shuttle while enhancing the adherence of the passengers to the novel services and accelerating the adoption of automated mobility.

**Acknowledgements** This project has received funding by the European Union's Horizon 2020 research and innovation program under grant agreement No. 769033.

## References

Azuma, K. E. (2018). Effects of low-level inhalation exposure to carbon dioxide in indoor environments: A short review on human health and psychomotor performance. *Environment International, 121*, 51–56.

Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019). Arcface: Additive angular margin loss for deep face recognition. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition* (pp. 4690–4699).

Duan, Z., Tezcan, O., Nakamura, H., Ishwar, P., & Konrad, J. (2020). RAPiD: Rotation-aware people detection in overhead fisheye images. In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops* (pp. 636–637).

Fang, H.-S., Xie, S., Tai, Y.-W., & Lu, C. (2017). Rmpe: Regional multi-person pose estimation. In *Proceedings of the IEEE international conference on computer vision* (pp. 2334–2343).

Olivera-La Rosa, A., Chuquichambi, E. G., & Ingram, G. P. (2020). Keep your (social) distance: Pathogen concerns and social perception in the time of COVID-19. *Personality and Individual Differences, 166*, 110200.

Papadimitriou, I., Vafeiadis, A., Lalas, A., Votis, K., & Tzovaras, D. (2020). Audio-based event detection at different SNR settings using two-dimensional spectrogram magnitude representations. *Electronics, 9*, 1593.

Salamon, J., & Bello, J. P. (2017). Deep convolutional neural networks and data augmentation for environmental sound classification. *IEEE Signal Processing Letters, 24*, 279–283.
Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

Tsiktsiris, D., Dimitriou, N., Lalas, A., Dasygenis, M., Votis, K., & Tzovaras, D. (2020). Real-time abnormal event detection for enhanced security in autonomous shuttles mobility infrastructures. *Sensors, 20*, 4943.

Tsiktsiris, D., Lalas, A., Dasygenis, M., Votis, K., & Tzovaras, D. (2021). Enhanced security framework for enabling facial recognition in autonomous shuttles public transportation during COVID-19. In *IFIP international conference on artificial intelligence applications and innovations* (pp. 145–154).

Tsiktsiris, D., Lalas, A., Dasygenis, M., Votis, K., & Tzovaras, D. (2022). An efficient method for addressing COVID-19 proximity related issues in autonomous shuttles public transportation. In *IFIP international conference on artificial intelligence applications and innovations* (pp. 170–179).

Zhang, H., McLoughlin, I., & Song, Y. (2015). Robust sound event recognition using convolutional neural networks. In *2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)* (pp. 559–563).

Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016). Joint face detection and alignment using multitask cascaded convolutional networks. *IEEE Signal Processing Letters, 23*, 1499–1503.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

<figure><img src="image_89.png" /></figure>
<figure><img src="image_90.png" /></figure>

**Chapter 6**
# Cybersecurity and Data Privacy: Stakeholders' Stand on Regulations and Standards

**Niels A. Nijdam, Meriem Benyahya, and Anastasija Collen**

**Abstract** AVENUE's technological interest lies in those state-of-the-art technologies and solutions that either are already commercial or close to the market and are expected to reach commercialisation within the following years. For the successful implementation of the project activities, a wide range of technological as well as non-technological elements will be employed, adapted, integrated, and, where required, partially developed. The present chapter focuses on those connected automated vehicle (CAV) technologies from the perspective of cybersecurity, delving into questions on in-vehicle, back-end, and infrastructure, including the communications between vehicle to vehicle (V2V), vehicle to infrastructure (V2I), vehicle to cloud (V2C), vehicle to everything (V2X), software safety, as well as security and privacy by design principles for the development of connected devices. Furthermore, non-technological issues cover stakeholder and user acceptance, regulatory and legislative requirements, a new standardisation progress, ethical considerations, and vehicle and technology certifications and licensing. The purpose of this chapter is to present the project context and relating it to the potential cyber assaults and data privacy threats. It further delineates the conducted assessment and the provided recommendations which were built based on the key standards and regulations wrapping together CAVs, cybersecurity, and personal data protection pursuits.

**6.1 Introduction**

More than ever, cybersecurity and data privacy are crucial to the introduction of new technologies, as everything tends to be *automated, autonomous*, and, most of all, *connected*. In this chapter, connected automated vehicle (CAV) technologies related

<footnote>N. A. Nijdam (✉) · M. Benyahya · A. Collen
Centre Universitaire d'Informatique, Geneva School of Economics and Management,
University of Geneva, Carouge, Switzerland
e-mail: niels.nijdam@unige.ch</footnote>
<figure><img src="image_91.png" /><figcaption>Fig. 6.1 Attack vectors within the CAV ecosystem</figcaption></figure>

to new digital services will be closer inspected to elaborate on a plurality of new threat vectors.

## 6.1.1 CAVs’ Threats

AVENUE involves a wide spectrum of technologies for a broad coverage and analysis of the existing advancements in a full ecosystem of the CAVs as depicted in Fig. 6.1:

* In-vehicle equipment: categorised in the present work as (1) sensors, representing the elementary devices through which the vehicle builds its perception and awareness model; (2) in-vehicle communication, the subsystems within the vehicle that serve for communication; and (3) ports, granting physical access to various parts of the vehicle systems.

* External communication technologies: from vehicle to vehicle (V2V) for fleet coordination to vehicle to infrastructure (V2I) and vehicle to everything (V2X) for infrastructure deployment in the cities and public transport operators (PTOs).

### 6.1.1.1 In-Vehicle Equipment

To assure the CAV’s highly automated navigation of levels 4 and 5 as defined by the Society of Automotive Engineering (SAE) (SAE, 2021), the vehicle intelligently compiles inputs collected from its internal equipment (including cameras, Global Positioning System (GPS), radio detection and ranging (RADAR), light detection and ranging (LiDAR), tire pressure monitoring system (TPMS), odometry, and ultrasound sensors). The processing and fusion of the collected data from such sensors support on providing accurate data on positioning, behaviour predictions, collision avoidance, pedestrian detection, and object recognition (Parkinson et al., 2017). Despite the sensors’ crucial role to safe self-driving, they remain victims to
attacks like spoofing, jamming, relay, and tampering, leading the vehicle to either
hit non-perceived objects or consider non-existent obstacles (Benyahya, Collen,
et al., 2022).

Furthermore, electronic control units (ECUs) are part of the essential in-vehicle
components which control the CAV's systems by receiving and processing broad-
cast signals from sensors (Dibaei et al., 2020). On the same note, ECUs are con-
nected to in-vehicle networks like controller area network (CAN), Local Interconnect
Network (LIN), Media-Oriented System Transport (MOST), FlexRay, and Ethernet
(Wu et al., 2020). Each protocol supports different communication within the in-
vehicular network. However, they embed many security concerns and risks of
potential attacks such as denial of service (DoS), packet injection, sniffing, eaves-
dropping, spoofing, relaying, and bus-off (El-Rewini et al., 2020).

Physical ports can broaden the points of entrance to an attack over the in-vehicle
network. The on-board diagnostic (OBD) is a port that is normally used for the
vehicle diagnosis or for ECU firmware upgrade (Elliott et al., 2019). Though if false
messages are injected through it, the control over the automotive functions (like
braking) can be granted to the attacker (Parkinson et al., 2017). Similarly, falsified
data, malware, or a virus can be injected through Universal Serial Bus (USB) ports
or the electric charging port (Bhusal et al., 2020).

6.1.1.2 External Communication Technologies

External vehicular communications come to complement the in-vehicle equipment on the automated driving functions. CAVs communicate to their surroundings using V2V to broadcast and receive signals from other vehicles and V2I to exchange data with the infrastructure (Elliott et al., 2019). Besides, V2X embeds both V2V and V2I in addition to communication with any external devices like smartphones (El-Rewini et al., 2020). Such communication modes are taking place thanks to vehicular ad hoc networks (VANETs) where vehicles and infrastructure are referred to as nodes which can exchange traffic-related messages.

The VANET architecture is composed of on-board units (OBUs), which are
located at the level of the CAV, roadside units (RSUs), which are placed within the
infrastructure, and a trusted authority (TA), which is the base station connecting
OBUs and RSUs to the core network and which distributes public/private keys and
certificates among nodes (Noh et al., 2020). VANETs are supported by wireless
technologies like IEEE 802.11p, which is the basis of dedicated short-range com-
munication (DSRC) (Ali & Li, 2020). Maple et al. (2019) added that further Internet
of Things (IoT) technologies such as IEEE 802.15.4 or ZigBee might be used to
facilitate information exchange. However, if the VANETs' signals are eavesdropped
by an attacker, the CAV can be compromised or maliciously tracked (Veitas &
Delaere, 2018). Further attacks were identified and asserted to dangerously impact
the traffic including sybil attack, distributed denial of service (DDOS), and man in
the middle (MitM) (Dibaei et al., 2020).
<figure><img src="image_92.png" /><figcaption>**Fig. 6.2** Investigation methodology</figcaption></figure>

CAVs use a plethora of technologies to come to fruition of safe automated driving and high connectivity with their surroundings. Nevertheless, such minibuses end up inheriting security weaknesses and accumulating additional threat vectors that can be real showstoppers to the CAV's prevalence.

**6.1.2 Motivation**

Bearing in mind the rampant CAV's threats, the different automation levels, and the multiple stakeholders within the AVENUE ecosystem, a cybersecurity and data privacy assessment and a guide of references regarding standards and regulations will be provided. Figure 6.2 depicts the adopted approach fusing thorough understanding of CAV's challenges, analysis of existing regulations and standards chasms, and an evaluation of the AVENUE landscape, resulting to what is presented as the standards coverage map (SCM).

This article addresses the following questions:

* Within the AVENUE ecosystem, how can cybersecurity and data privacy threats be efficiently mitigated?

* What are the key technical tools recommended by legal policies and standardisation bodies to countermeasure those threats?

* How are risk assessment, vulnerability analysis, and penetration testing considered by the partners to shield the AVENUE minibuses?

* By being compliant to the existing standards, how anchored would the CAV be from both cyber assaults and data leakages?

The remainder of this chapter is structured as follows. Section 6.2 sheds light on the most recent (up to 2021 fourth quarter) regulations and standards to be considered within the CAV's landscape. Moreover, it discusses global efforts which can infer some lessons learned. Section 6.3 outlines the cybersecurity assessment and data collection tools adopted within the AVENUE scope. Section 6.4 presents the means and input collection from the different PTOs and software providers. Section 6.5 depicts the key recommendations upon the identified shortcomings from the cybersecurity and data privacy perspectives. Finally, Sect. 6.6 provides concluding statements and future work orientations.
## 6.2 Regulations and Standards

Being crucial for the evolution, development, and deployment of CAVs, cybersecurity and data privacy challenges have attracted many stakeholders including automotive manufacturers, legal and regulatory bodies, information technology (IT) and telecommunication suppliers, operators of intelligent transport system (ITS), and mobility service providers to collaborate and come up with new laws, strategies, and guidelines. Figure 6.3 reflects an in-depth overview of the existing and forthcoming efforts from the key players in providing both mandatory and nice to have requirements.

The 4 years of the AVENUE project witnessed a major progress in regulating CAVs' deployment. In August 2022, the European Commission (EC) published the regulation (EU) 2022/1426 (Regulation (EU) 2022/1426, 2022) where technical specifications for the type approval of fully automated vehicles were defined, built upon the generic vehicular regulation 2019/2144 (Regulation (EU) 2019/2144, 2019). Furthermore, Network and Information Security (NIS) 1 and 2 (EU 2016/1148; EU 2020/1148) directives call the operators of IT service providers to take the appropriate measures to manage cyber risks posed to the security in a general scope.

While the EC regulations laid down on either generic type of approval or IT cybersecurity, the United Nations Economic Commission for Europe (UNECE) published acts joining cybersecurity measures to the CAVs' environment. The UNECE R155 (UNECE R155, 2020) and R156 (UNECE R155, 2020) came with the purpose to unify the automotive standards by requiring cybersecurity management system (CSMS) and software update management system (SUMS) certifications accordingly for the SAE level 3 onward. The two certifications cover the cybersecurity risk management, security by design, and security incident detection and mitigation and secure software updates over the CAV's life cycle including development, production, and post-production (Suh, 2020). The involvement of

<figure><img src="image_93.png" /><figcaption>Fig. 6.3 Taxonomy of regulation and standardisation of stakeholders</figcaption></figure>
certification third parties, such as TÜV SÜD and CertX, can be compulsory to generate CSMS and SUMS either for the first time or for their renewal occurring every 3 years.

Under the auspices of standard development organisations (SDOs), efforts were made to shield the entire CAV's environment. The International Organisation for Standardisation (ISO) and SAE working groups claim to provide a complete cybersecurity management for the driverless landscape with a focus on the in-vehicle components (Schoitsch & Schmittner, 2020). The ISO/SAE 21434 (ISO/SAE 21434, 2021) and ISO/PAS 5112 (ISO/PAS 5112, 2021) represent the salient standards providing a high-level guidance on cybersecurity governance and auditing for the CAV's ecosystem. The European Telecommunication Standards Institute (ETSI), Publicly Available Specification (PAS), Automotive Open System Architecture (AUTOSAR), European Committee for Standardisation (CEN), World Wide Web Consortium (W3C), IEEE Standards Association (IEEE SA), and 5G Automotive Association (5GAA) institutions provide standards for securing vehicular communication (Kim & Shrestha, 2020).

As depicted in Fig. 6.3, further recommendations were provided by the International Telecommunication Union (ITU) outlining security threats definition, security guidelines for V2X, specification of secure software update procedure for ITS devices, and guidelines for intrusion and misbehaviour detection (ITU-T, 2020). Additionally, the European Union Agency for Cybersecurity (ENISA) published several reports spotlighting the CAVs' cybersecurity risks once deployed within smart cities. Besides, automotive associations such as the European Automobile Manufacturers Association (ACEA) (ACEA, 2019) and the Automotive Information Sharing and Analysis Centre (Auto-ISAC) (Auto-ISAC, n.d.) are orienting the original equipment manufacturers (OEMs) towards self-audit, testing, and deploying incident response plans.

### 6.2.1 CAVs Privacy Initiatives

The General Data Protection Regulation (GDPR) is the fundamental privacy data law in the European Union (EU). It sets strict obligations related to personal data (PD) processing, rights for concerned individuals (data subjects), technical requirements to employ privacy preserving techniques (anonymisation and pseudonymisation), and Data Protection Impact Assessment (DPIA) as an assessment for any new technologies with privacy risks (Regulation (EU) 2016/679, 2016). Nevertheless, within the CAV's complex environment, GDPR implementation remains convoluted, as the stakeholder can accumulate multiple roles (e.g. being data processor and data controller simultaneously) (Benyahya, Kechagia, et al., 2022). Another limitation of the GDPR within the CAV context is considering anonymisation as a permanent solution. Though with minor reverse engineering efforts, the PD can be de-anonymised with no compliance violation to the GDPR (ENISA, 2022). Hence, a full compliance does not rhyme with an absolute PD protection.
To limit the GDPR pitfalls, the European Data Protection Board (EDPB) provided guidance on the processing of PD in the context of ITS, which highlighted privacy risks and provided recommendations on data protection by design and by default within the CAV's environment (European Data Protection Board, 2020). Moreover, the guidelines focused on consent as the legal basis for processing PD inside the vehicle and through V2X communications (European Data Protection Board, 2021).

Further European initiatives, such as GAIA-X, DATEX-II, and Data for Roads, aim to overcome the current privacy hurdles within the vehicular environment. GAIA-X provided proposals on data protection rules and architecture standards in many areas including location-based services (LBS) and mobility as a service (MaaS) in smart mobility (BMW, 2020). DATEX II has been addressing traffic data sharing and transmission including transmitted data in cooperative and connected mobility (DATEX-II, n.d.). Data for Road Safety is another EU initiative who has been discussing connected vehicles of all automation levels and gathers partners from the European Commission, industries, and governments to reach cooperative, trustworthy, and free of charge vehicle data exchange with respect to the protection of PD (Data for Road Safety, 2021).

Within the AVENUE landscape, data should be processed with the consideration of the essential factors summarised in Fig. 6.4. It wraps up the GDPR and other aforementioned privacy initiatives recommendations. In other words, it reflects the data controllers' obligations regarding the data subjects' rights including the processing restrictions, privacy by design deployment, and consent implementation. From the privacy preserving perspectives, the figure depicts the anonymisation and pseudonymisation challenges. Finally, it calls back the high privacy risks linked to LBS and MaaS which are more likely to be deployed within the CAV's environment.

<figure><img src="image_94.png" /><figcaption>**Fig. 6.4** Data privacy challenges within the AVENUE environment</figcaption></figure>
## 6.3 Methodology

An overall evaluation and recommendations are built based on the final in-vehicle and out-of-vehicle services implemented in AVENUE. A questionnaire was conducted to collect updated inputs from the different key stakeholders. Based on the collected data, the cybersecurity and data privacy mechanisms were assessed per deployment site and service provider. The questionnaire findings were evaluated further by matching the identified surface attacks to required standards from Sect. 6.2. Such mapping supported on providing the recommendations through an SCM defining the required standards to adopt per layer and sub-component of the minibuses' ecosystem.

To conduct the intended cybersecurity and data privacy assessment, it was crucial to identify the involved service providers and the final AVENUE services. Multiple data collection tools were deployed through several iterations, where questionnaires represented the most relevant source of extensive findings, summarised in Fig. 6.5. The orange entries were substituted by the respondent organisation name and related services. Questions related to hardware attributes were targeting the OEM, and questions on software patterns were designated to software service providers. The questionnaire was shared with several key stakeholders (henceforth providers):

(1) Three on-demand service providers, each providing services in one or more deployment sites (SP1, SP2, and SP3); (2) three software developers, developing in-vehicle and out-of-vehicle services (SD1, SD2, SD3); and (3) one OEM responsible for manufacturing the automated minibuses and vehicle operational software. Furthermore, the four public transport operators (TP1, TP2, TP3, TP4) for the deployment sites have been included as well.

The questionnaire consists of 19 questions that are split into 4 main parts. The first section spotlights the applications that are in the AVENUE scope and which are intended to be delivered by the end of the project. With the multiple types of respondents, the first section of the questionnaire aims to identify the site, the application that is or will be deployed, and by whom. Then multiple questions are raised regarding cybersecurity in the second part of the questionnaire with the purpose to highlight the cybersecurity tools, standards, and regulations adopted by the respondent to identify if the mandatory UNECE regulations (R155 and R156) and the key vehicular cybersecurity ISO standards are taken into consideration. The third part of the questionnaire collects information related to data privacy and to the GDPR compliance. Finally, the fourth part gathers any sketched architecture or design that the respondent aims to deploy within the AVENUE project.
Partner name

Final Applications

Q1 What are the applications that *Partner* is using and/or developing for AVENUE?

Q2 Please specify in which site your platform will be deployed within the AVENUE scope?

Cybersecurity

Q3 What are the security measures you have in place to prevent source code vulnerabilities?

Q4 Do you conduct internal pentests regularly or just if a risk is raised? Do you conduct hardware or software based pentests?

Q5 What is the minimum CVSS accepted?

Q6 How are Spoofing and Jamming attacks prevented by the GNSS receiver at the level of the vehicle GPS?

Q7 Do you have any vulnerability report? How often are security audits conducted over the implemented application? Would you like to share with us your latest audit report?

Q8 Do you coordinate with other partners' cybersecurity teams to assess risks on connecting your platform to the other servers/systems?

Q9 Do you provide training to vehicle operators/supervisors to face cybersecurity attacks?

Q10 Are you working on the CSMS certification required by the UN 155 regulation?

Q11 How is security handled while doing a software update? Is there any vulnerability scanning before?

Q12 Do you refer to any security standards/guidelines such as ISO9001, ISO27000, ISO/SAE21434...?

Data Privacy

Q13 Does your application use any personal data? Does it aggregate data from other service providers?

Q14 Is the location data, collected through *your applications* encrypted/anonymized before it is saved on the database? Do you process any IMEI/MEID ID? If yes, is it anonimised/pseudonymised?

Q15 How do you secure the connection and the data exchange between *your applications* and data providers services (eg Navya/TPG,...)?

Q16 How data processing and storage are handled with regard to the GDPR recommendations as well as the FDPIC acts? How long is personal data stored? and how often it is destroyed? Do you ask for the user's consent before storing the data?

Q17 Do you share any data with any LBS platforms?

Q18 Which procedure do you deploy in case of a data breach? Do you have a clearly elaborated procedure?

Architecture

Q19 Could you provide a view of your architecture as it is used within AVENUE?

<caption>**Fig. 6.5 Applications' service provider questionnaire**</caption>
## 6.4 Findings

From the qualitative analysis perspective, it is noteworthy to highlight that the synthesised outputs from the questionnaire provided valuable findings regarding the final applications and the cybersecurity and privacy governance within AVENUE.

Table 6.1 gives an overview of the various application types and their usage by the operators. During the project's lifetime, the operators faced numerous challenges, which affected the selection and development of the services that were ultimately realised and utilised thereafter. The presented table abstracts this view and shows the mapping of the applications provided by the various service providers and software developers with the operators. It can be noted that only one service provider (SP1) was utilised by two PTOs. In most cases the service operators provided the client application (a mobile application) and also an application for the safety operator to receive new on-demand trip orders from the client. Meaning, the service provider may send an order through the OEM interface and act as the fleet manager or utilise a semimanual approach by informing the safety operator (from the PTO) to select the destination through the in-vehicle control service (using a touch screen).

In addition to the applications' mapping, the questionnaire inputs supported in assessing the cybersecurity governance generally and the code security measures specifically. Prevention, monitoring, and audit tools were gathered from the collected answers. Table 6.2 wraps up the key security measures adopted by every service provider. All respondents are highly concerned with the code vulnerabilities prevention. However, more efforts are still required to conduct continuous assessments and internal and external audits. From the standardisation perspective, valuable inspirations are retrieved from ISO and SAE key standards, while the mandatory UNECE R155 (UNECE R155, 2020) and R156 (UNECE R156, 2020) are still envisioned and planned before end of 2023. Though no accomplished certification or concrete compliance has been recorded yet.

<table><caption>**Table 6.1** Application type and usage by transport operators</caption><thead><tr><th rowspan="2">Provider</th><th rowspan="2">Applications</th><th colspan="4">Operators</th></tr><tr><th>TP1</th><th>TP2</th><th>TP3</th><th>TP4</th></tr></thead><tbody><tr><td>SP1</td><td>On-demand service (passenger and safety operator app), control centre</td><td>✓</td><td>✓</td><td></td><td></td></tr><tr><td>SP2</td><td>On-demand service (passenger and safety operator app)</td><td></td><td></td><td>✓</td><td></td></tr><tr><td>SP3</td><td>On-demand service (passenger and safety operator app), control centre</td><td></td><td></td><td></td><td>✓</td></tr><tr><td>SD1</td><td>Data statistics dashboard</td><td>✓</td><td>✓</td><td></td><td>✓</td></tr><tr><td>SD2</td><td>In-vehicle dashboard (interface)</td><td>✓</td><td></td><td></td><td>✓</td></tr><tr><td>SD3</td><td>In-vehicle dashboard (back-end)</td><td>✓</td><td></td><td></td><td>✓</td></tr><tr><td>OEM</td><td>Vehicle control, in-vehicle control service, fleet manager, data statistics (API)</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr></tbody></table>
<table><caption>**Table 6.2** Code security measures indicated by providers</caption><thead><tr><th rowspan="2">Provider</th><th colspan="6">Security measures</th></tr><tr><th>Prevention</th><th>Assessment</th><th>Penetration testing</th><th>Audit</th><th>Standards</th><th>Certification</th></tr></thead><tbody><tr><td>SP1</td><td>✓</td><td>SCA (Ferrara et al., 2021)</td><td>✓</td><td>✓</td><td>ISO 27001<sup>a</sup></td><td>N/A</td></tr><tr><td>SP2</td><td>✓</td><td>✗</td><td>✓</td><td>✗</td><td>✗</td><td>N/A</td></tr><tr><td>SP3</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>N/A</td></tr><tr><td>SD1</td><td>✓</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td><td>N/A</td></tr><tr><td>SD2</td><td>✓</td><td>SAST (Synopsys, n.d.)</td><td>✗</td><td>✗</td><td>✗</td><td>N/A</td></tr><tr><td>SD3</td><td>✓</td><td>CVSS V3.1 (NIST, n.d.)</td><td>✓</td><td>✓</td><td>✗</td><td>N/A</td></tr><tr><td>OEM</td><td>✓</td><td>MOABI (Moabi, n.d.)<br/>Nessus (Tenable, n.d.)</td><td>✓</td><td>✓</td><td>SAE J3061<br/>ISO/SAE 21,434<br/>ISO 2700 IEC 62443</td><td>R155 (UNECE R155, 2020)<br/>R156 (UNECE R156, 2020)</td></tr></tbody></table>

<table><caption>**Table 6.3** Data privacy measures indicated by providers</caption><thead><tr><th rowspan="2">Provider</th><th colspan="4">Privacy measures</th><th rowspan="2">LBS sharing</th></tr><tr><th>Processed PD</th><th>Connectivity</th><th>GDPR</th><th></th></tr></thead><tbody><tr><td>SP1</td><td>Location data</td><td>Encrypted HTTPS</td><td>User consent<br>Data storage<br>Data destruction</td><td>✓</td></tr><tr><td>SP2</td><td>Name and phone number</td><td>Encrypted HTTPS</td><td>User consent<br>Data storage</td><td>✗</td></tr><tr><td>SP3</td><td>Not collected</td><td>Tokenised</td><td>Not applicable</td><td>✗</td></tr><tr><td>SD1</td><td>Not collected</td><td>Tokenised</td><td>Not applicable</td><td>✗</td></tr><tr><td>SD2</td><td>Not collected</td><td>Encrypted HTTPS</td><td>User consent</td><td>✗</td></tr><tr><td>SD3</td><td>Video and audio + facial biometrics</td><td>VPN + TLS certificates (encryption)</td><td>User consent<br>In-vehicle data storage</td><td>✗</td></tr><tr><td>OEM</td><td>Location data<br>Video streaming</td><td>Cyphered VPN</td><td>Data storage<br>Data destruction</td><td>✗<sup>a</sup></td></tr></tbody></table>

From the data privacy angle, Table 6.3 showcases how the providers are careful about the GDPR compliance in terms of respect to data processing principles. For key stakeholders who are collecting sensitive data such as personal and location data, their efforts are reflected through the user consent implementation, data storage limitation, and data destruction procedures. Additionally, worthy pseudonymisation techniques are deployed like encryption and tokenisation for authentication.

<footnote><sup>a</sup>Certified</footnote>

<footnote><sup>a</sup>Likely to occur</footnote>
Moreover, at that stage of the mini-shuttle deployment, sharing data with LBS plat-
forms remains limited without claiming any PD. Albeit laudable efforts are adopted
by the service providers, data privacy assessments such as DPIA are omitted from
the data protection procedures set in the AVENUE project.

6.5 Discussion and Recommendations

Per the conducted assessment, the cybersecurity and data privacy governance were
evaluated including risk prevention, vulnerability management, events monitoring,
and data leakage plans. From the aforementioned discussion, the governance
strengths can be summarised as follows:

* The AVENUE partners are highly aware of the cybersecurity risks.

* Several “security by design” principles such as risk prevention are considered while implementing AVENUE services.

* The GDPR compliance is a high-priority obligation for all partners.

* The mandatory certifications (CSMS and SUMS) related to the UNECE R155 (UNECE R155, 2020) and R156 (UNECE R156, 2020) are envisioned and planned before end of 2023.

* The V2X immaturity represents an instinctive mitigation solution to the connec-
tivity threats where the minibuses have limited V2I and V2V.

On the other hand, it is noteworthy to spotlight the following weaknesses:

* Albeit the high awareness about the cyber assaults, further thorough implemen-
tation of a cybersecurity culture is still required through all the system layers and
among all the partners.

* For a flawless cybersecurity governance, the partners’ efforts should not be lim-
ited to prevention tools, but it should be broadened to implement monitoring,
continuous assessments, and risk mitigation strategies through the overall life
cycle including the decommissioning stage.

* Despite the GDPR considerations within the project, an in-depth implementation of the law is still recommended as several obligations are still missing such data privacy assessments.

* There is a scarcity on deploying cybersecurity validation process as no penetra-
tion testing is conducted while testing the vehicles on the AVENUE sites.

6.5.1 Standards Coverage Map

To overcome the asserted hurdles, the CAV’s ecosystem SCM is suggested as a
roadmap to be followed by the AVENUE partners upon their provided services and
scope. The SCM is foreseen as a suitable approach on building a robust
cybersecurity and data privacy governance based on the CAV's standards and regulation discussed in Sect. 6.2. The map combines both the technical and organisational audit avenues applied to the automated minibuses' landscape as depicted in Fig. 6.6. The map is classified into four layers, in vehicle, out of vehicle, applications, and organisation, where every layer groups the respective technical standards.

As a parent node of the four layers, ISO/SAE 21434(ISO/SAE 21434, 2021) is set as the core standard; however it is very generic to be followed per se. The combination of both generic and technical standards on the SCM is foreseen to overcome the broadening of the ISO/SAE 21434 leading to a more thorough assessment.

The in-vehicle layer addresses the potential attacks at the vehicle level (as discussed in Sect. 6.1.1) which were classified into six sub-layers. First, the sensors category defines the guidance on standardising the interfaces between the different sensors and the fusion unit leading to the automation navigation decisions. Second, the network buses category where standards propose guidelines on detecting intrusions and authentication measures within the in-vehicle communication networks. Third, the ECU standard aims to prevent non-authorised access to the vehicular software modules. Fourth, the software update outlines the directives on how to conduct secure software updates during the vehicle life cycle. Fifth, the artificial intelligence (AI) components standard provides guidance on secure usage of AI-based functions involved on the automation decision-making. Finally, the physical access specifies countermeasures against threats from plugged-in external devices.

Even with a limited V2X implementation within AVENUE, the communication-related recommendations for future development are anticipated. In Fig. 6.6, the out-of-vehicle layer relies on two main categories wrapping standards related to countermeasure CAV's Internet and V2X threats. To secure the vehicle's Internet access using DSRC, long-term evolution (LTE), and 5G, standards provided a set of secure channel models and through several use cases. Besides, the multiple V2X communications have been standardised by ISO, ETSI, and SAE. The security confidential management standards, which sets V2X certificates security and privacy requirements, define the precise structure, format, and authentication schemes supporting the minibuses' communication to peer instances. It is noteworthy to mention that other V2X communications such as V2I and vehicle to grid (V2G) have been supported by dedicated standards, while others as per the vehicle to cloud (V2C) is still considered under the umbrella of broad standards like SAE J2735.

Moreover, the application layer consists of two sub-layers reflecting two types of applications: users and ITS. The minibuses' deployment is associated to the means of several services provided to the end user, such as the on-demand application, and to the smart city. The user application standards focus on data access and cryptography best practices to consider while building interfaces to the CAV's hardware or software. Likewise, the ITS application standards recommend mechanisms to determine permitted actions among the peer ITS applications to achieve security properties such as authorisation, integrity, and confidentiality. Nevertheless, standards such as SAE J2735 and ISO/TS 2177 have a larger scope covering the V2X communication in general and, hence, other subcomponents from the second layer too.
<figure><img src="image_95.png" /><figcaption>**Fig. 6.6** Standards coverage map</figcaption></figure>

Finally, the organisation layer in Fig. 6.6 incorporates four procedural sub-layers supporting on mitigating the assessed weaknesses in the previous section. The risk assessment reflects evaluation procedures on quantifying the likelihood and impact of cybersecurity threats. The privacy impact assessment wraps standardised procedures, sample reports, and checklists to fulfil. The regulatory obligations sets the mandatory laws that the minibuses' environment has to comply with, which are summarised into the GDPR, the UNECE R155, and the UNECE R156. Finally, the associate's authorisation group points out the managerial methods that need to be conducted by the minibuses' trained associates while facing a cyber threat and processing any PD. The combination of both generic and technical standards on Fig. 6.6 is foreseen to provide a clear roadmap overcoming the cybersecurity and data privacy challenges discussed in the present work.