ANNEX III

Compliance assessment

The overall compliance assessment of the ADS is based on:

— Part 1: The traffic scenarios to consider

— Part 2: The assessment of the ADS safety concept and the audit of the manufacturer safety management system.

— Part 3: The tests for the most relevant traffic scenarios.

— Part 4: The principles to be used for the credibility assessment for using virtual toolchain to ADS validation

— Part 5: The system established by the manufacturer to ensure in-service reporting.

Any requirement in Annex II may be checked by means of tests performed by the type-approval authority (or its technical service).

PART 1

TRAFFIC SCENARIOS TO CONSIDER

1. Minimum set of traffic scenarios

1.1. Scenarios and parameters listed in point 1 shall be used, when these scenarios are relevant for the ODD of the ADS. If the manufacturer deviates from the parameters proposed in point 1, the safety performance metrics and inherent assumptions used by the manufacturer shall be documented in the documentation package. The safety performance metrics and inherent assumptions chosen shall demonstrate that the fully automated vehicle is free of unreasonable safety risks. The validity of such safety performance metrics and inherent assumptions shall be supported by in-service monitoring data.

1.2. Parameters to be used for the lane change scenarios by the fully automated vehicle

1.2.1. The scenarios and parameters, with regard to lane change, shall be applied as specified in UN Regulation No 157 ( 1 )

1.3. Parameters to be used for the turning and crossing scenario by the fully automated vehicle.

1.3.1. In the absence of more specific traffic rules, the following requirements shall be taken into account with regard to interaction with other road users involved in the movement when turning and crossing (see Figure 1) in dry and proper road pavement conditions.

1.3.2. In the case of merging with privileged traffic during turning with and without crossing the opposite traffic direction, privileged traffic in the target lane should not have to decelerate. However, it must be ensured that the TTC of the approaching privileged traffic in the target road (case (a) in Figure 1) never falls below the threshold TTC dyn defined as: With: v e equal to the speed of the fully automated vehicle v a equal to the speed of the privileged approaching traffic β equal to 3 m/s2 being the maximum admissible deceleration for the privileged approaching traffic. ρ equal to 1,5 s being the reaction time of the privileged approaching

1.3.3. In the case of a turning manoeuvre crossing the opposite traffic direction, when considering oncoming traffic, privileged traffic in the target lane should not have to decelerate. However, if justified by the traffic density, it must be ensured – in addition to the distance from the approaching privileged traffic in the target road – that the TTC of the privileged crossing traffic to the fictitious collision point (point of intersection of the trajectories, case (b) in Figure 1) never falls below the threshold TTC int defined as: With: v c equal to the speed of the privileged conflicting traffic β equal to 3 m/s2 being the maximum admissible deceleration for the privileged crossing traffic. ρ equal to 1,5 s being the reaction time of the privileged crossing traffic The same applies to crossing with privileged traffic (case (c) in Figure 1): The TTC of privileged traffic to the imaginary collision point (point of intersection of the trajectories) shall never fall below the threshold TTC int defined in the present point. Figure 1: Visualisation of the distances during turning and crossings. Case (a): Distance to the approaching privileged traffic in the target lane to be observed during turning-in and merging with privileged traffic. Case (b): Distance to the oncoming privileged traffic to be observed when turning by crossing the opposite traffic direction. Case (c): distance to the privileged crossing traffic to be respected when crossing.

1.4. Parameters to be used for the emergency manoeuvre scenarios by the fully automated vehicle (DDT under critical scenarios)

1.4.1. The ADS shall avoid a collision with a leading vehicle which decelerates up to its full braking performance provided that there was no cut-in by another vehicle.

1.4.2. Collisions with cutting in vehicles, pedestrians and cyclists travelling in the same direction, as well as with pedestrians who can start to cross the street, shall be avoided at least within the conditions determined by the following equation. With: being the time to-collision at the moment of the cut-in of the vehicle or cyclist by more than 30 cm in the lane of the fully automated vehicle. v rel being the relative speed in metres per second [m/s] between the fully automated vehicle and the cutting-in vehicle (positive if the ADS is faster than the cutting-in vehicle). β being the maximum deceleration of the fully automated vehicle and assumed to be equal to: 2,4 m/s2 if it is transporting standing or not fastened vehicle occupants and there is a cutting in vehicle scenario; 6 m/s2 if it is transporting standing or not fastened vehicle occupants for other scenarios with pedestrians or cyclists. 6 m/s2 for other fully automated vehicles. ρ being the time required by the fully automated vehicle to initiate an emergency braking and assumed to be equal to 0,1 s τ being the time to reach the maximum deceleration β and assumed to be equal to 0,12 s for fully automated vehicles transporting standing or not fastened vehicle occupants; 0,3 s for other fully automated vehicles The compliance with this equation is required only for road users cutting in, and only if the inserting road users were visible at least 0,72 seconds before cut-in: This results in a required collision avoidance when another road user enters ego lane above the following TTC values (for example shown for speeds in 10 km/h steps). These requirements shall be met independently of environmental conditions. v rel [km/h] [s] for vehicles with standing or unfastened vehicle occupants [s] for other vehicles 10 0,74 0,48 20 1,32 0,71 30 1,9 0,94 40 2,47 1,18 50 3,05 1,41 60 3,63 1,64 If a lane change with a lower TTC is carried out to the lane of the fully automated vehicle, it can no longer be assumed that there will be no collision avoidance. The control strategy of the ADS may change between collision avoidance and mitigation only if the manufacturer can demonstrate that this increases the safety of the vehicle occupants and the other road users (e.g. by prioritising braking over an alternative manoeuvre).

1.4.3. The ADS shall avoid a collision with a crossing pedestrian or a cyclist in front of the vehicle.

1.4.3.1. Urban and rural driving conditions

1.4.3.1.1. The ADS shall avoid a collision, up to a speed of 60 km/h, with an unobstructed pedestrian crossing with a lateral speed component of not more than 5 km/h or an unobstructed cyclist crossing with a lateral speed component of not more than 15 km/h in front of the vehicle. This shall be ensured independently from the specific manoeuvre the ADS is undertaking.

1.4.3.1.2. In the case the pedestrian or the cyclist proceed with higher speed than the aforementioned values and the ADS can no longer avoid collision, the control strategy of the ADS may change between collision avoidance and mitigation only if the manufacturer can demonstrate that this increases the safety of the vehicle occupants and the other road users (e.g. by prioritizing braking over an alternative manoeuvre).

1.4.3.1.3. The ADS shall mitigate a collision with an obstructed pedestrian or cyclist crossing in front of the vehicle by reducing its speed at impact by at least 20 km/h. This shall be ensured independently from the specific manoeuvre the ADS is undertaking.

1.4.3.1.4. For the purpose of demonstrating fulfilment of the previous requirements related to crossing of pedestrians and cyclists in front of the vehicle, test and assessment scenarios developed under the European New Car Assessment Programme (Euro NCAP) may be taken as guidance.

1.4.3.2. Motorway driving conditions

1.4.3.2.1. The relevant scenarios, with regard to pedestrian crossing, shall be applied as specified in UN Regulation No 157.

1.4.3.2.2. In the case the pedestrian crosses with parameter values outside the boundaries specified in UN Regulation 157 and the ADS can no longer avoid collision, the control strategy of the ADS may change between collision avoidance and mitigation only if the manufacturer can demonstrate that this increases the safety of the vehicle occupants and the other road users (e.g. by prioritizing braking over an alternative manoeuvre).

1.5. Motorway entry The fully automated vehicle shall be able to safely enter the motorway by adapting the speed to the traffic flow, and activate the relevant direction indicator according to the traffic rules. The direction indicator shall be deactivated once the vehicle has performed the lane change manoeuvre (LCM). The parameters used in the lane change scenario shall be applied.

1.6. Motorway exit The fully automated vehicle shall be able to anticipate the targeted motorway exit by driving on the adjacent lane to the exit lane and shall not unnecessarily decelerate before the LCM into the exit lane starts. The fully automated vehicle shall apply the direction indicator in accordance with the traffic rules, and perform the LCM into the exit lane without undue delay. The direction indicator shall be deactivated once the LCM has been completed in compliance with the traffic rules in the country of operation.

1.7. Passing a toll station Depending on the ODD, the fully automated vehicle shall be able to select the proper passing gate, and adapt its speed to that permitted limits within the toll area while considering the traffic flow.

1.8. Operation on other road types than motorways Depending on the ODD, the relevant scenario defined in points 1.2. to 1.4. above shall be applied.

1.9. Parameters to be used for Automated valet parking

1.9.1. Depending on the ODD, the relevant scenarios defined in points 1.3. to 1.5 above shall be applied. The parameters to be used for these scenarios may need to be adapted to take into account of the limited driving speed and the general lack of visibility that may occur in a parking facility. Special attention shall be given to avoiding of collision with pedestrians and in particular with children and prams.

2. Scenarios not covered by point 1.

2.1. Scenarios that are not listed in point 1 shall be generated to cover reasonably foreseeable critical situations, including failures and traffic hazards within the operational design domain.

2.2. When ADS capabilities depend on remote capabilities, scenarios shall include failures and traffic hazards stemming from the corresponding remote capabilities.

2.3. The method to generate scenarios that are not listed in Section 1, shall follow the principles set in Appendix 1 to Part 1 of this Annex.

2.4. The method used by the manufacturer to generate scenarios that are not listed in point 1 shall be documented in the documentation package to be provided for the ADS assessment. Appendix 1 Principles to be followed to derive scenarios relevant for the ODD of the ADS 1. Generation and classification of scenarios From a qualitative perspective, scenarios can be classified into Nominal/Critical/Failure and correspond to normal or emergency operation. For each of these categories, a data-based approach and a knowledge-based approach can be used to generate corresponding traffic scenarios. A knowledge-based approach utilizes expert knowledge to identify hazardous events systematically and create scenarios. A data-based approach utilizes the available data to identify and classify occurring scenarios. Scenarios shall be derived from the ODD of the fully automated vehicle. 2. Nominal scenarios A series of analytical frameworks can help the manufacturer to derive additional nominal scenarios to ensure coverage for the specific application. These frameworks are divided into: 2.1. ODD analysis An ODD consists of scenery elements (e.g., physical infrastructure), environmental conditions, dynamic elements (e.g., traffic, vulnerable road users) and operational constraints to the specific ADS application. The aim of this analysis is to identify the characteristics of the ODD, allocate properties and define interactions between the objects. Here the effect of ODD on the behaviour competencies of the ADS is explored. An example of the analysis is provided in Table 1. Table 1 Dynamic elements and their properties Objects Events/Interactions Vehicles (e.g. cars, light trucks, heavy trucks, buses, motorcycles) Lead vehicle decelerating (frontal) Lead vehicle stopped (frontal) Lead vehicle accelerating (frontal) Changing lanes (frontal/side) Cutting in (adjacent) Turning (frontal) Encroaching opposing vehicle (frontal/side) Encroaching adjacent vehicle (frontal/side) Entering roadway (frontal/side) Cutting out (frontal) Pedestrians Crossing road – inside crosswalk (frontal) Crossing road – outside crosswalk (frontal) Walking on side walk/shoulder Cyclists Riding in lane (frontal) Riding in adjacent lane (frontal/side) Riding in dedicated lane (frontal/side) Riding on sidewalk/shoulder Crossing road – inside crosswalk (frontal/side) Crossing road – outside crosswalk (frontal/side) Animals Static in lane (frontal) Moving into/out of lane (frontal/side) Static/moving in adjacent lane (frontal) Static/moving on shoulder Debris Static in lane (frontal) Other dynamic objects (e.g. shopping carts) Static in lane (frontal/side) Moving into/out of lane (frontal/side) Traffic signs Stop, yield, speed limit, crosswalk, railroad, crossing, school zone Traffic signals Intersection, railroad crossing, school zone Vehicle signals Turn signals (direction indicator) 2.2. OEDR Analysis: Behaviour competency identification Once the objects and relevant properties have been identified, it is possible to map the appropriate ADS response. The ADS response is modelled on applicable functional requirements and by applying the performance requirements of this regulation and the traffic rules of the country of operation. The outcome of the OEDR analysis is also a set of competences that can be mapped to the behavioural competences applicable to the ODD, to ensure compliance with the relevant regulatory and legal requirements. Table 2 provides a qualitative example of a matching event – response. The combination of objects, events, and their potential interaction, as a function within the ODD, constitute the set of nominal scenarios pertinent to the ADS under analysis. The identification of nominal scenarios can benefit from an enhanced combination of scenario descriptors covering, within the ODD, e.g. infrastructure attributes, objects and events characteristics, hazards affecting responses (e.g. weather, visibility). The identification of nominal scenarios is not limited to traffic conditions but also covers environmental conditions, human factors, connectivity and miscommunication. As parameters (assumptions) for the events are yet to be defined, the nominal scenarios derived from the application of the analysis are to be considered in their functional and logical abstraction layer. Table 2 Behaviour competences for given events Event Response Lead vehicle decelerating Follow vehicle, decelerate, stop Lead vehicle stopped Decelerate, stop Lead vehicle accelerating Accelerate, follow vehicle Lead vehicle turning Decelerate, stop Other vehicle changing lanes Yield, decelerate, follow vehicle Other vehicle cutting-in Yield, decelerate, stop, follow vehicle Vehicle entering roadway Follow vehicle, decelerate, stop Opposing vehicle encroaching Decelerate, stop, shift within lane, shift outside lane Adjacent vehicle encroaching Yield, decelerate, stop Lead vehicle cutting out Accelerate, decelerate, stop Pedestrian crossing road – inside crosswalk Yield, decelerate, stop Pedestrian crossing road – outside crosswalk Yield, decelerate, stop Cyclists riding in lane Yield, follow Cyclists riding in dedicated lane Shift within lane Cyclists crossing road – inside crosswalk Yield, decelerate, stop Cyclists crossing road – outside crosswalk Yield, decelerate, stop 3. Critical scenarios Critical scenarios can be derived by either considering edge-case assumptions on nominal traffic scenarios (data-based) or applying standardised methods (knowledge-based) for the evaluation of operational insufficiencies (see example of methods in point 3.5.5. of Part 2). The identification of critical scenarios can benefit from an enhanced combination of scenario descriptors and edge values covering, within the ODD, e.g. infrastructure attributes, objects and events characteristics, hazards affecting responses (e.g. weather, visibility masks, interactions with other road users than the triggered object or event). The identification of critical scenarios is not limited to traffic conditions but also covers environmental conditions, human factors, connectivity and miscommunication. Critical scenarios correspond to emergency operation of the ADS. 4. Failure scenarios These scenarios aim to assess how the ADS responds to a failure. Different methods are available in literature (see example of methods in point 3.5.5. of Part 2). For each of the behaviour failures and consequential effects identified, the manufacturer shall put in place relevant strategies when developing the ADS (i.e., fail-safe). When applying the failure scenarios, the objective is to assess the ability of the ADS to comply with requirements for safety-critical situations, including for example ‘The ADS shall manage safety-critical driving situations’ and ‘The ADS shall safely manage failure modes’ and their respective sub-requirements. 5. Assumptions: Logical to concrete scenarios To ensure that the scenarios identified in the previous points are ready to be assessed through simulation or physical testing, the manufacturer may need to coherently parametrise them by applying assumptions. The manufacturer shall provide evidence supporting the assumptions made such as data collection campaigns performed during the development phase, real-world accidentology and realistic driving behaviour evaluations. Parameters used to characterise critical scenarios should cover reasonably foreseeable values in scenario descriptors, but shouldn’t be limited to values already observed in documented data bases.

PART 2

ASSESSMENT OF THE ADS SAFETY CONCEPT AND AUDIT OF THE MANUFACTURER SAFETY MANAGEMENT SYSTEM

1. General

1.1. The type-approval authority granting the type-approval or the technical service acting on its behalf shall verify through targeted spot checks and tests, in particular as specified in point 4 of this annex, that the safety argumentation provided by the documentation complies with the requirements of Annex II and that the design and processes described in documentation are actually implemented by the manufacturer.

1.2. While based on the provided documentation, evidences provided for the audit of the safety management system and the assessment of the ADS safety concept carried out to the satisfaction of the type-approval authority in accordance with this regulation, the residual level of safety risk of the type-approved ADS is deemed to be acceptable for the entry into service of the vehicle type, the overall ADS safety during the ADS lifetime in accordance with the requirements of this regulation remains the responsibility of the manufacturer requesting the type-approval.

2. Definitions

For the purposes of this annex,

2.1. ‘safety concept’ means a description of the measures designed into the ADS, so that the fully automated vehicle operates for the scenarios and events relevant to the ODD in such a way that it is free of unreasonable safety risks to the vehicle occupants and other road users under fault (functional safety) and non-fault conditions (operational safety). The possibility of a fall-back to partial operation or even to a back-up system for vital ADS functions shall be a part of the safety concept.

2.2. ‘units’ means the smallest divisions of system components which will be considered in this annex, since these combinations of components will be treated as single entities for purposes of identification, analysis or replacement.

2.3. ‘transmission links’ means the means used for inter-connecting distributed units for the purpose of conveying signals, operating data or an energy supply. This equipment is generally electrical but may, in some part, be mechanical, pneumatic or hydraulic.

2.4. ‘range of control’ means an output variable and defines the range over which the system is likely to exercise control.

2.5. ‘boundary of functional operation’ means the boundaries of the external physical limits within which the ADS is able to perform the dynamic driving tasks.

3. Documentation on the ADS

3.1. Requirements

The manufacturer shall provide a documentation package which gives access to the basic design of the ADS and the means by which it is linked to other vehicle systems or by which it directly controls output variables as well as off-board hardware/software and remote capabilities.

The function(s) of the ADS, including the control strategies, and the safety concept, as laid down by the manufacturer, shall be explained.

Documentation shall be brief, yet provide evidence that the design and development has had the benefit of expertise from all the ADS fields which are involved.

For periodic road worthiness tests, the documentation shall describe how the current operational status of the ADS and the functionality and software integrity can be checked.

The type-approval authority shall assess the documentation package which shall show that the ADS:

(a) is designed and was developed to operate in such a way that it is free from unreasonable risks for a vehicle occupants and other road users within the declared ODD and boundaries;

(b) fulfils the performance requirements of Annex II to this Regulation;

(c) was developed according to the development process/method declared by the manufacturer.

3.1.1. Documentation shall be made available in three parts: (a) Application for type-approval: the information document submitted to the type-approval authority at the time of the type-approval application shall contain brief information on the items listed in Annex I. It will become part of the type-approval. (b) The formal documentation package for the type-approval, containing the material listed in this Section 3. (with the exception of that of point 3.5.5.) which shall be supplied to the type-approval authority for the purpose of conducting the ADS type-approval. This documentation package shall be used by the type-approval authority as the basic reference for the verification process set out in point 4. of this annex. The type-approval authority shall ensure that this documentation package remains available for a period of at least 10 years counted from the time when production of the vehicle type is definitely discontinued. (c) Additional confidential material and analysis data (intellectual property) of point 3.5.5. which shall be retained by the manufacturer, but made open for inspection (e.g. on-site in the engineering facilities of the manufacturer) at the time of the ADS type-approval. The manufacturer shall ensure that this material and analysis data remains available for a period of 10 years counted from the time when production of the vehicle type is definitely discontinued.

3.2. General description of the ADS

3.2.1. A description shall be provided giving a simple explanation of the operational characteristics of the ADS and ADS features.

3.2.2. The description shall include:

3.2.2.1 the operational design domain such as maximum speed of operation, road type (e.g. dedicated lane), country(ies)/areas of operation, road conditions and environmental conditions required (e.g. no snow), etc.)/Boundary conditions

3.2.2.2 basic performance (e.g. object and event detection and response, off-board infrastructure needed during operation)

3.2.2.3. Interaction with other road users

3.2.2.4. main conditions for minimal risk manoeuvres.

3.2.2.5. interaction concept with vehicle occupants, the on board operator (if applicable) and the remote intervention operator (if applicable).

3.2.2.6. the means to activate or deactivate the ADS by the on-board operator (if relevant) or the remote intervention operator (if relevant), vehicle occupants (if relevant) or other road users (if relevant).

3.2.2.7. operational measures (e.g. on-board operator or remote intervention operator needed) to be met to ensure safety during the fully automated vehicle operation.

3.2.2.8. backend, off-board infrastructure needed to ensure safety during the fully automated vehicle operation.

3.3. Description of the functions of the ADS

A description shall be provided giving an explanation of all the functions including control strategies to ensure the robust and safe operation of the ADS and the methods used to perform the dynamic driving tasks within the ODD, and the boundaries under which the automated driving system is designed to operate, including a description on how this is ensured.

Any enabled or disabled automated driving functions for which the hardware and software are present in the vehicle at the time of production, shall be declared and are subject to the requirements of this annex as well as Annex II to this Regulation, prior to their use in the vehicle. The manufacturer shall also document the data processing if continuous learning algorithms are implemented.

3.3.1. A list of all input and sensed variables shall be provided and the working range of these defined, along with a description of how each variable affects the ADS behaviour.

3.3.2. A list of all output variables that are controlled by the ADS shall be provided and an explanation given, in each case, of whether the control is direct or via another vehicle system. The range over which the ADS is likely to exercise control on each such variable shall be defined.

3.3.3. Limits defining the boundaries of functional operation including ODD-limits shall be stated where appropriate to ADS performance.

3.3.4. The human machine interaction (HMI) concept with the vehicle occupants/on-board operator/remote intervention operator (if any) when ODD limits are approached and then reached shall be explained. The explanation shall include the list of types of situations in which the ADS will generate a support request to the on board operator/remote intervention operator (if applicable), the way the request is performed, the procedure that handles a failed request and the minimal risk manoeuvre. Signals and information given to the on-board operator/remote intervention operator, vehicle occupants and other road users in each of the above aspects shall also be described.

3.4. ADS layout and schematics

3.4.1. Inventory of components. A list shall be provided, collating all the units of the ADS and mentioning the other vehicle systems as well as off-board hardware/software and remote capabilities that are needed to achieve specified performance of the ADS to be approved according to its ODD. An outline schematic showing these units in combination, shall be provided with both the equipment distribution and the interconnections made clear. This outline shall include: (a) perception and objects/events detection including mapping and positioning (b) Characterisation of Decision-making (c) The ADS data elements. (d) links and interface with other vehicle systems, off-board hardware/software and remote capabilities

3.4.2. Functions of the units The function of each unit of the ADS shall be outlined and the signals linking it with other units or with other vehicle systems shall be shown. It shall include off-board systems supporting the ADS and other vehicle systems. This may be provided by a labelled block diagram or other schematic, or by a description aided by such a diagram.

3.4.3. Interconnections within the ADS shall be shown by a circuit diagram for the electric transmission links, by a piping diagram for pneumatic or hydraulic transmission equipment and by a simplified diagrammatic layout for mechanical linkages. The transmission links both to and from other systems shall also be shown.

3.4.4. There shall be a clear correspondence between transmission links and the signals carried between units. Priorities of signals on multiplexed data paths shall be stated wherever priority may be an issue affecting performance or safety.

3.4.5. Identification of units

3.4.5.1. Each unit shall be clearly and unambiguously identifiable (e.g. by marking for hardware, and by marking or software output for software content) to provide corresponding hardware and documentation association. Where a software version can be changed without requiring the replacement of the marking or component, the software identification must be by software output only.

3.4.5.2. Where functions are combined within a single unit or indeed within a single computer, but shown in multiple blocks in the block diagram for clarity and ease of explanation, only a single hardware identification marking shall be used. The manufacturer shall, by the use of this identification, affirm that the equipment supplied conforms to the corresponding document.

3.4.5.3. The identification defines the hardware and software version and, where the latter changes such as to alter the function of the unit as far as this Regulation is concerned, this identification shall also be changed.

3.4.6. Installation of sensing system components The manufacturer shall provide information on the installation options for the individual components that comprise the sensing system. These options shall include, but are not limited to, the location of the component in/on the vehicle, the material(s) surrounding the component, the dimensioning and geometry of the material surrounding the component, and the surface finish of the materials surrounding the component, once installed in the vehicle. The information shall also include installation specifications that are critical to the ADS’s performance, e.g. tolerances on installation angle. Changes to the individual components of the sensing system, or the installation options, shall be notified to the type-approval authority and be subject to further assessment.

3.5. Safety concept of the manufacturer and validation of the safety concept by the manufacturer

3.5.1. The manufacturer shall provide a statement which affirms that the ADS is free from unreasonable risks for the vehicle occupants and other road users.

3.5.2. In respect of software employed in the ADS, the outline architecture shall be explained and the design methods and tools used shall be identified (see 3.5.1). The manufacturer shall show evidence of the means by which they determined the realisation of the ADS logic, during the design and development process.

3.5.3. The manufacturer shall provide the type-approval authority with an explanation of the design provisions built into the ADS so as to ensure functional and operational safety. Possible design provisions in the ADS are for example: (a) fall-back to operation using a partial system. (b) redundancy with a separate system. (c) diversity of systems performing the same function. (d) removal or limitation of the automated driving function(s).

3.5.3.1. If the chosen provision selects a partial performance mode of operation under certain fault conditions (e.g. in case of severe failures), then these conditions shall be stated (e.g. type of failure) and the resulting limits of effectiveness defined (e.g. immediate initiation of a minimal risk manoeuvre) as well as the warning strategy to the operator/remote operator, occupants and other road users (when applicable).

3.5.3.2. If the chosen design provision selects a second (back-up) or diverse means to realise the performance affected by the fault, the principles of the change-over mechanism, the logic and level of redundancy and any built-in checking features shall be explained and the resulting limits of effectiveness defined.

3.5.3.3. If the chosen design provision selects the removal of the automated driving function(s), this shall be done in compliance with the relevant provisions of this regulation. All the corresponding output control signals associated with this function shall be inhibited.

3.5.4. The manufacturer shall also provide the type-approval authority with an explanation of the operational safety measures to be put in place for the safe operation of the ADS such as an on-board operator or a remote intervention operator, supporting off-board infrastructure, transport and physical infrastructure requirements, maintenance measures, etc.

3.5.5. The documentation shall be supported, by an analysis that shows, how the ADS will behave to mitigate or avoid hazards that can have a bearing on the safety of vehicle occupants and other road users.

3.5.5.1. The chosen analytical approach(es) shall be established and maintained by the manufacturer and shall be made open for inspection by the type-approval authority at the time of the type-approval and afterwards.

3.5.5.2. The type-approval authority shall assess the application of the analytical approach(es): (a) Inspection of the safety approach at the concept level. This approach shall be based on a Hazard analysis/Risk assessment appropriate to system safety. (b) Inspection of the safety approach at the ADS level including a top down (from possible hazard to design) and bottom-up approach (from design to possible hazards). The safety approach may be based on a failure mode and effect analysis (FMEA), a fault tree analysis (FTA) and a System-theoretic process analysis (STPA) or any similar process appropriate to system functional and operational safety. (c) Inspection of the validation/verification plans and results including appropriate acceptance criteria. This shall include testing appropriate for validation, for example, hardware in the Loop (HIL) testing, vehicle on-road operational testing, testing with real end users, or any other testing appropriate for validation/verification. Results of validation and verification may be assessed by analysing coverage of the different tests and setting minimum coverage thresholds for various metrics.

3.5.5.3. The analytical approach under 3.5.5.2. shall confirm that at least each of the following items is covered: (i) Issues linked to interactions with other vehicle systems (e.g. braking, steering); (ii) Failures of the automated driving system and system risk mitigation reactions; (iii) Situations within the ODD where the ADS may create unreasonable safety risks for the vehicle occupants and other road users due to operational disturbances (e.g. lack of or wrong comprehension of the vehicle environment, lack of understanding of the reaction from the operator/remote operator, vehicle occupants or other road users, inadequate control, challenging scenarios) (iv) Identification of the relevant scenarios within the boundary conditions and management method used to select scenarios and validation tool chosen. (v) Decision making process resulting in the performance of the dynamic driving tasks (e.g. emergency manoeuvres), for the interaction with other road users and in compliance with national traffic rules (vi) Reasonably foreseeable misuse by the vehicle occupants/other road users, mistakes or misunderstanding by the operator/remote operator/occupants/other road users (e.g. unintentional override) and intentional tampering of the ADS. (vii) Cybersecurity threats on the safety of the ADS (to be covered by the analysis done in accordance with UN Regulation No 155 on Cyber Security and Cyber Security Management System). (viii) Operational safety issues: problems with the supporting off-board infrastructure, problem with the remote intervention operator, loss of connectivity, lack of maintenance, etc.

3.5.5.4. The assessment by the type-approval authority shall consist of spot checks to establish that argumentation supporting the safety concept is understandable and logical and implemented in the different functions of the ADS. The assessment shall also check that validation plans are robust enough to demonstrate safety (e.g. reasonable coverage of chosen scenarios testing by the validation tool chosen) and have been properly completed.

3.5.5.4.1. It shall demonstrate that the operation of fully automated vehicle is free from unreasonable risks for the vehicle occupants and other road users in the operational design domain, i.e. through: (a) an overall validation target (i.e. overall validation acceptance criteria) supported by validation results, demonstrating that the entry into service of the ADS will overall not increase the level of risk for the vehicle occupants and other road users compared to a manually driven vehicles; and (b) a scenario specific approach (i.e. scenario based validation acceptance criteria) showing that the ADS will overall not increase the level of risk for the vehicle occupants and other road users compared to a manually driven vehicles for each of the safety relevant scenarios;

3.5.5.5. The type-approval authority shall perform or shall require to perform tests as specified in point 4. of this Annex to verify the safety concept.

3.5.5.6. This documentation shall itemize the parameters being monitored and shall set out, for each failure condition of the type defined in point 3.5.4. of this annex, the warning signal to be given to the operator/remote operator/vehicle occupants/other road users and/or to service/technical inspection personnel.

3.5.5.7. This documentation shall also describe the measures in place to ensure the ADS is free from unreasonable risks to vehicle occupants, and other road users when the performance of the ADS is affected by environmental conditions e.g. climatic, temperature, dust ingress, water ingress, ice packing, inclement weather.

4. Verification and tests

Taking into account the results of the analysis of the manufacturer’s documentation package, the type-approval authority shall request the tests to be performed or witnessed by the Technical Service to check specific points arising from the assessment.

4.1. The functional operation of the ADS, as laid out in the documents required in point 3., shall be tested as follows:

4.1.1. Verification of the function of the ADS The type-approval authority shall verify the ADS under non-failure conditions by testing on a track a number of selected functions, as deemed necessary by the type-approval authority, from those described by the manufacturer, and by checking the overall behaviour of the ADS in real driving conditions including compliance with traffic rules. These tests shall include scenarios whereby the ADS is overridden by the remote intervention operator (if applicable). These tests can be based on test scenarios listed in Part 3 of this Annex and/or on additional scenarios not covered by Part 3.

4.1.1.1. The test results shall correspond with the description, including the control strategies, provided by the manufacturer in point 3.2. and shall comply with the performance requirements of this regulation.

4.1.2. Verification of the ADS safety concept The reaction of the ADS shall be checked under the influence of a fault in any individual unit by applying corresponding output signals to electrical units or mechanical elements in order to simulate the effects of internal failure within the unit. The type-approval authority shall verify that these tests include aspects that may have an impact on vehicle controllability and user information (HMI aspects e.g. interaction with the operator/remote operator).

4.1.2.1. The type-approval authorities shall also check a number of scenarios that are critical for the Object and Event Detection and Response (OEDR) and Characterisation of the decision-making and HMI functions of the ADS (e.g. object difficult to detect, when the ADS reaches the ODD boundaries, traffic disturbance scenarios, connectivity issue, problem with off-board systems, remote capabilities issues e.g. the absence of the remote intervention operator) as defined in this regulation.

4.1.2.2. The verification results shall correspond with the documented summary of the hazard analysis, to a level of overall effect such that the safety concept and execution are confirmed as being adequate and in compliance with the requirements of this regulation.

4.2. Simulation tool and mathematical models to verify of the safety concept may be used in accordance with Annex VIII to Regulation (EU) 2018/858, in particular for scenarios that are difficult on a test track or in real driving conditions. Manufacturers shall demonstrate the scope of the simulation tool, its validity for the scenario concerned as well as the validation performed for the simulation tool chain (correlation of the outcome with physical tests). To demonstrate the validity of the simulation toolchain, the principles of Part 4 of this Annex shall apply. Simulation shall not be a substitute for physical tests in Part 3 of this Annex.

4.3 The manufacturer shall have a valid certificate of compliance for the safety management system (SMS) relevant to the vehicle type being approved.

5. Safety management system (SMS)

5.1. In respect of the ADS, the manufacturer shall demonstrate to the type-approval authority in terms of a safety management system (SMS) that effective processes, methodologies, training and tools are in place, up to date and being followed within the organization to manage the safety and continued compliance throughout the ADS lifecycle.

5.2. The design and development process shall be established and documented including safety management system, requirements management, requirements’ implementation, testing, failure tracking, remedy and release.

5.3. The manufacturer shall ensure effective communication channels between manufacturer departments responsible for functional/operational safety, cybersecurity and any other relevant disciplines related to the achievement of vehicle safety.

5.4. The manufacturer shall have processes aimed at collecting vehicle data, and data from other sources to monitor and analyse safety-relevant incidents/accidents caused by the engaged automated driving system. The manufacturer shall report to type-approval authorities, market surveillance authorities and the Commission the relevant occurrences in accordance with Part 5 of this Annex.

5.4.1. The manufacturer must enable the transport service operator to provide the type-approval authorities, market surveillance authorities or other authorities designated by the Member States with the vehicle data in accordance with paragraph 5.4 above, as well as with the ADS data and the specific data elements for event data recorder collected in accordance with Section 9 of Annex II.

5.5. The manufacturer shall have processes to manage potential safety-relevant gaps post-registration and to update the vehicles if necessary.

5.6. The manufacturer shall demonstrate that periodic independent internal process audits (e.g. every 2 years) are carried out to ensure that the processes established in accordance with points 5.1 to 5.5. are implemented consistently.

5.7. Manufacturers shall put in place suitable arrangements (e.g. contractual arrangements, clear interfaces, quality management system) with suppliers to ensure that the supplier safety management system comply with the requirements of points 5.1. (except for vehicle related aspects like ‘operation’ and ‘decommissioning’), 5.2, 5.3 and 5.6.

5.8. Certificate of compliance for safety management system

5.8.1. An application for a Certificate of Compliance for Safety Management System shall be submitted by the manufacturer or by their duly accredited representative to the type-approval authority.

5.8.2. It shall be accompanied by the undermentioned documents in triplicate, and by the following particular: (a) Documents describing the Safety Management System. (b) A signed Declaration of Compliance of the SMS with all the requirements for safety management according to this Regulation, using the model as defined in Appendix 3 to this Annex.

5.8.3. When this audit of the SMS has been satisfactorily completed and in receipt of a signed declaration from the manufacturer according to the model as defined in Appendix 3, a certificate named Certificate of Compliance for SMS as described in Appendix 4 (hereinafter the Certificate of Compliance for SMS) shall be granted to the manufacturer.

5.8.4. The Certificate of Compliance for SMS shall remain valid for a maximum of three years from the date of deliverance of the certificate unless it is withdrawn.

5.8.5. The type-approval authority may at any time verify that the requirements for the Certificate of Compliance for SMS continue to be met. The type-approval authority shall withdraw the Certificate of Compliance for SMS if major non-conformities in the compliance with the requirements laid down in this Regulation are discovered and not immediately addressed.

5.8.6. The manufacturer shall inform the type-approval authority or its technical service of any change that will affect the relevance of the certificate of compliance for SMS. After consultation with the manufacturer, the type-approval authority or its Technical Service shall decide whether new checks are necessary.

5.8.7. In due time, the manufacturer shall apply for a new or for the extension of the existing Certificate of Compliance for SMS. The type-approval authority shall, subject to a positive audit, issue a new Certificate of Compliance for SMS or extend its validity for a further period of three years. The type-approval authority shall verify that the SMS continue to comply with the requirements of this Regulation. The type-approval authority shall issue a new certificate in cases where changes have been brought to the attention of the type-approval authority or its Technical Service and the changes have been positively re-assessed.

5.8.8. The expiry or withdrawal of the manufacturer’s Certificate of Compliance for SMS shall be considered, with regard to the vehicle types to which the SMS concerned was relevant, as modification of approval, which may include the withdrawal of the approval if the conditions for granting the approval are not met anymore.

6. Reporting provision

6.1. The reporting of the safety assessment of the ADS safety concept as well as the audit of the safety management system of the manufacturer shall be performed in such a manner that allows traceability, e.g. versions of documents inspected are coded and listed in the records of the Technical Service.

6.2. An example of layout for the report on the assessment of the ADS safety concept from the Technical Service to the type-approval authority is provided in Appendix 1 to this part. The listed items in this Appendix are outlined as minimum set of items that need to be covered.

6.3. The granting type-approval authority shall issue the safety assessment results to be annexed to the type-approval certificate based on the documentation provided by the manufacturer, the report of the assessment of the ADS safety concept by the technical service and on the outcomes of the verification and test campaigns performed in accordance with Part 3 of this Annex. An example of a possible layout for the safety assessment results is given in Appendix 4.

7. Competence of the auditors/assessors

7.1. The assessment of the ADS safety concept and the audit of the safety management system under this part shall only be conducted by assessors/auditors with the technical and administrative knowledge necessary for such purposes. They shall in particular be competent as auditor/assessor for ISO 26262-2018 (Functional Safety – Road Vehicles), and ISO/PAS 21448 (Safety of the Intended Functionality of road vehicles); and shall be able to make the necessary link with cybersecurity aspects in accordance with UN Regulation No 155 and ISO/SAE 21434). This competence shall be demonstrated by appropriate qualifications or other equivalent training records. Appendix 1 Model for the assessment report of the ADS safety concept Safety assessment report No: 1. Identification 1.1. Vehicle make 1.2. Vehicle type 1.3. Means of identification of vehicle type if marked on the vehicle 1.4. Location of that marking 1.5. Manufacturer’s name and address 1.6. If applicable, name and address of manufacturer’s representative 1.7. Manufacturer’s formal documentation package Documentation reference No: Date of original issue: Date of latest update: 2. Assessment method 2.1. Description of the assessment processes and methodologies 2.2. Acceptability criteria 3. Results of the review of the documentation package 3.1. Review of the ADS description 3.2. Review of Manufacturer’s safety concept and the manufacturer safety analysis 3.3. Review of the Verification and Validation performed by the manufacturer in particular coverage of the different tests and setting minimum coverage thresholds for various metrics 3.4. Review of the methods and tools (software, laboratory, others) and the credibility assessment 3.5. Review of ADS data requirements and specific data elements for event data recorder for fully automated vehicles 3.6. Checks of the Cyber Security and Software Updates certificates are covering he ADS 3.7. Review of the information provided in the Operating Manual 3.8. Review of the provisions for the periodic roadworthiness tests of the ADS 3.9. Review of additional information not included in the Information Document 4. Verification of ADS functions under non-failure conditions (referred to in point 4.1.1. of Annex III Part 2 to Commission Implementing Regulation (EU) 2022/1426 of 5 August 2022 laying down rules for the application of Regulation (EU) 2019/2144 of the European Parliament and of the Council as regards uniform procedures and technical specifications for the type-approval of the automated driving system (ADS) of fully automated vehicles (2) 4.1. Rationale for the selection of test scenarios 4.2. Selected test scenarios 4.3. Test reports 4.3.1. Test No (add as many as the tests performed) 4.3.1.1. Objectives of the test 4.3.1.2. Test conditions 4.3.1.3. Measured quantities and measuring devices 4.3.1.4. Acceptability criteria 4.3.1.5. Test results 4.3.1.6. Comparison with the manufacturer’s supplied documentation 5. Verification of ADS safety concept under failure (referred to in point 4.1.2. of Annex III Part 2 to Implementing Regulation (EU) 2022/1426 5.1. Rationale for the selection of test scenarios 5.2. Selected test scenarios 5.3. Test reports 5.3.1. Test No (add as many as the tests performed) 5.3.1.1. Objectives of the test 5.3.1.2. Test conditions 5.3.1.3. Measured quantities and measuring devices 5.3.1.4. Acceptability criteria 5.3.1.5. Test results 5.3.1.6. Comparison with the manufacturer’s supplied documentation 6. Safety management system certificate (shall be appended to this test report) 7. Date of the assessment 8. Final judgement on the safety assessment outcome 9. This assessment has been carried out and the results reported in accordance with Implementing Regulation (EU) 2022/1426 Technical Service carrying out the assessment Signed: ... Date: ... 10. Comments: Appendix 2 Model of the ADS assessment results to be annexed to the type-approval certificate 1. Identification 1.1. Vehicle make 1.2. Vehicle Type 1.3. Means of identification of vehicle type if marked on the vehicle 1.4. Location of that marking 1.5. Manufacturer’s name and address 1.6. If applicable, name and address of manufacturer’s representative 1.7. Manufacturer’s formal documentation package Documentation reference No: Date of original issue: Date of latest update: 2. Assessment method 2.1. Description of the assessment processes and methodologies 2.2. Acceptability criteria 3. Verification of ADS functions under non-failure conditions (referred to in point 4.1.1. of Annex III Part 2 to Implementing Regulation (EU) 2022/1426 3.1. Rationale for the selection of test scenarios 3.2. Selected test scenarios 4. Verification of ADS safety concept under single failure (referred to in point 4.1.2. of Annex III Part 2 to Implementing Regulation (EU) 2022/1426 4.1. Rationale for the selection of test scenarios 4.2. Selected test scenarios 5. Assessment results 5.1. Results of the review of the Information Document 5.2. Results of the verification of ADS functions under non-failure conditions 5.3. Results of the verification of ADS safety concept under single failure 5.4. Results of the assessment of the Safety Management System 5.5. Results of the verification of provisions for the periodic roadworthiness tests 6. Final judgement on the safety assessment outcome Appendix 3 Model of Manufacturer’s Declaration of Compliance for SMS Manufacturer’s declaration of compliance with the requirements for the Safety Management System Manufacturer’s Name: Manufacturer’s Address: …(Manufacturer’s Name) attests that the necessary processes to comply with the requirements for the Safety Management System laid down in Implementing Regulation (EU) 2022/1426 are installed and will be maintained. Done at: … (place) Date: Name of the signatory: Function of the signatory: (Stamp and signature of the manufacturer’s representative) Appendix 4 Model of Certificate of Compliance for SMS Certificate of Compliance for Safety Management System With Implementing Regulation (EU) 2022/1426 Certificate number [Reference number] [… Type-approval authority] Certifies that Manufacturer: ... Address of the manufacturer: complies with the provisions of Implementing Regulation (EU) 2022/1426 Checks have been performed on: by (name and address of the type-approval authority or Technical Service): Number of report:... The certificate is valid until [...Date] Done at [...Place] On [...Date] [...Signature] Attachments: description of the Safety Management System by the manufacturer.

PART 3

TESTS

1. General provisions

Pass- and fail-criteria to assess ADS safety shall be based on the requirements set out in Annex II and the scenario described in Part 1 of this annex. The requirements are defined in such a way that the pass/fail criteria can be derived not only for a specific set of test parameters, but also for all safety-relevant combinations of parameters that may occur in the operating conditions covered by the type approval and the specified operating range (e.g., speed range, longitudinal and transverse acceleration range, radii of curvature, brightness, number of lanes). For conditions not tested but that may occur within the defined ODD of the system, the manufacturer shall demonstrate as part of the assessment described in Part 2 to the satisfaction of the type-approval authority, that the vehicle is safely controlled.

These tests shall confirm the minimum performance requirements described in Annex II and the functionality of the ADS and the safety concept of the manufacturer as described in Part 2 of this Annex. Test results shall be documented and reported in accordance with point 6 of Part 2 of this annex.

These tests shall also confirm, that the ADS complies with the traffic rules, adapts its operations to environmental conditions, avoids disruption to the flow of traffic (e.g. blocking the lane because of too many MRMs), does not show unpredictable behaviour and shows reasonable cooperative and anticipatory behaviour in relevant situations (i.e. merging in dense traffic or in vicinity of vulnerable road users).

2. Test site

The test site shall comprise characteristics (example: friction value) that correspond to the specified ODD of the ADS. As necessary to apply the specific conditions of the ODD of the ADS, physical tests will be performed within the actual ODD (on-road) or at any test facility that replicates the ODD conditions and shall be determined by the manufacturer and the type approval authority. The ADS shall be tested on-road in accordance with the applicable law of the Member States and provided that tests can be carried out safely and without any risk to other road users.

3. Environmental conditions

Tests shall be carried out under different environmental conditions, within the limits of the defined ODD for the ADS. For environmental conditions not tested that may occur within the defined ODD, the manufacturer shall demonstrate as part of the assessment to the satisfaction of the type-approval authority that the vehicle is safely controlled.

To test the requirements for failure of functions, self-testing of the ADS and initiation and implementation of a minimal risk manoeuvre, errors may be artificially induced and the vehicle may be artificially brought into situations where it reaches the limits of the defined operating range (e.g., environmental conditions).

4. System modifications for testing purposes

If ADS modifications are required in order to allow testing, e.g. road type assessment criteria or road type information (map data), it shall be ensured that these modifications do not affect the test results. These modifications shall in principle be documented and annexed to the test report. The description and the evidence of influence (if any) of these modifications shall be documented and annexed to the test report.

5. Vehicle conditions

5.1. Test mass The subject vehicle shall be tested with any permissible vehicle load. No load alteration shall be made once the test procedure has begun. The manufacturer shall demonstrate, through the use of documentation, that the ADS works at all load conditions.

5.2. The subject vehicle shall be tested at the tyre pressure recommended by the manufacturer.

5.3. It shall be verified, that the condition of the system is according to the intended testing purpose (e.g. in a fault-free condition or with the specific faults to be tested).

6. Test tools

In addition to real vehicles, state-of-the-art test tools may be used to carry out the tests, replacing real vehicles and other road users (e.g., soft targets, mobile platforms, etc.). The replacement test tools shall comply with the characteristics relevant for sensory performance assessment, real vehicles and other traffic participants. Tests shall not be carried out in a way that would endanger the personnel involved, and significant damage of the vehicle being tested must be avoided where other means of validation are available.

7. Test parameter variation

The manufacturer shall declare the system boundaries to the type-approval authority. The type-approval authority shall define different combinations of test parameters (e.g. present speed of the vehicle, type and offset of target, curvature of lane, etc.) in order to test the ADS. The selected test cases shall provide sufficient test coverage for all scenarios, test parameters and environmental influences. Adequate robustness of the perceptions systems for the ADS against input/sensor data malfunction and adverse environmental conditions shall be demonstrated.

Test parameters selected by the type-approval authority shall be recorded in a test report in a manner that allows traceability and repeatability of the test setup.

8. Tests scenarios to assess the performance of the ADS on a test track (points 8.1., 8.2., 8.5, 8.6, 8.7, 8.8, 8.9.) and on-road (8.3., 8.4., 8.10.)

The scenarios included in the following points have to be considered a minimum set of tests. At the request of the type-approval authority, additional scenarios that are part of the ODD can be executed. If a scenario described in point 8 of this annex does not belong to the ODD of the vehicle, it shall not be taken into consideration.

Depending on the ODD, test scenarios shall be selected as part of the type-approval test. The test scenarios shall be selected in accordance with Part 1 of this annex. Type-approval testing may be carried out on the basis of simulations, manoeuvres on the test track and driving tests on real road traffic. However, it may not be based solely on computer simulations and at the time of type-approval, the type-approval authority shall conduct or shall witness at least the following tests to assess the behaviour of the ADS.

8.1. Lane keeping

The test shall demonstrate that the fully automated vehicle does not leave its lane and maintains a stable motion inside its lane across the speed range and different curvatures within its system boundaries.

8.1.1. The test shall be based on the ODD of the ADS and shall be executed at least: a) with a minimum test duration of 5 minutes; b) with a passenger car target as well as a power-two-wheeler (PTW) target as the other vehicle; c) with a lead vehicle swerving in the lane; and d) with another vehicle driving close beside in the adjacent lane.

8.2. Lane changing manoeuvre (LCM)

The tests shall demonstrate that the fully automated vehicle does not cause an unreasonable risk to safety of the vehicle occupants and other road users during a lane change procedure, and that the ADS is able to assess the criticality of the situation before starting the lane change manoeuvre (LCM) throughout the entire operational speed range. These tests are only required if the fully automated vehicle is capable of performing lane changes either during a Minimal risk manoeuvre or during regular operation.

8.2.1. The following tests shall be executed: a) with the fully automated vehicle performing lane change to the adjacent (target) lane; b) merging at lane end; c) merging into an occupied lane.

8.2.2. The tests shall be executed at least: a) with different vehicles, including a power two-wheeler (PTW) approaching from the rear; b) in a scenario where it is possible to execute a lane changing manoeuvre in regular operation; c) in a scenario where a lane changing manoeuvre in regular operation is not possible due to a vehicle approaching from the rear; d) with an equally fast vehicle following behind in the adjacent lane, preventing a lane change; e) with a vehicle driving beside in the adjacent lane preventing a lane change; f) in a scenario where a LCM during a minimal risk manoeuvre is possible and executed; g) in a scenario where the fully automated vehicle reacts to another vehicle that starts changing into the same space within the target lane, to avoid a potential risk of collision.

8.3. Response to different road geometries

These tests shall ensure, that the fully automated vehicle detects and adapts to a variation of different road geometries which can occur within the intended ODD across its whole speed range.

8.3.1. The test shall be executed with at least the list of scenarios below based on the ODD of the ADS: a) T-junctions (3-way intersections) with and without traffic lights, with different rights of way; b) crossroads (4 or more way intersections) with and without traffic lights, with different rights of way; c) roundabouts.

8.3.2. Each test shall be executed at least: a) without a lead vehicle; b) with a passenger car target as well as a PTW target as the lead vehicle/other vehicle; c) with and without approaching or passing vehicles.

8.4. Response to national traffic rules and road infrastructure

These tests shall ensure that the fully automated vehicle complies with national traffic rules and that it adapts to a various permanent and temporary changes of the road infrastructure (e.g. road construction sites) in the entire speed range.

8.4.1. The tests shall be executed with at least with the list of scenarios below that are relevant for the ODD of the ADS: a) different speed limit signs, so that the ADS has to change its speed according to the indicated values; b) signal lights and/or stop instructed by a road safety officer/enforcement agents with situations of going straight, turning left and right; c) pedestrian and cyclist crossings with and without pedestrians/cyclist approaching/on the road. d) temporary modifications: e.g., road maintenance operations indicated by traffic signs, cones and other signalisation, access restrictions. e) motorway entry, exit and toll stations.

8.4.2. Each test shall be executed at least: a) without a lead vehicle; b) with a passenger car target as well as a PTW target as the lead vehicle/other vehicle.

8.5. Collision avoidance: Avoid a collision with road users or objects blocking the lane

The test shall demonstrate that the fully automated vehicle avoids a collision with a stationary vehicle, road user or fully or partially blocked lane up to the maximum specified speed of the ADS.

8.5.1. This test shall be executed at least with the following scenarios, where relevant in the ODD: a) with a stationary passenger car target; b) with a stationary PTW target; c) with a stationary pedestrian target; d) with a pedestrian target crossing the lane with a speed of 5 km/h, also in the presence of other objects relevant in the ODD (e.g. a ball, a shopping bag, etc.); e) with a pedestrian target moving a speed of up to 5 km/h within and partially occupying the lane of the ADS and following the same or the opposite direction of the fully automated vehicle; f) with a pedestrian target swerving in the same lane of the fully automated vehicle; g) with a cyclist target crossing the lane with a speed of 15 km/h; h) with a cyclist target which is travelling in the same direction with a speed of 15 km/h; i) with the fully automated vehicle turning right and crossing the path of the cyclist travelling in the same direction with a speed of 15 km/h; j) with a target representing a blocked lane; k) with a target partially within the lane; l) with one or more different types of unpassable objects relevant in the ODD (e.g., a dustbin, a fallen bicycle or scooter, a fallen traffic sign, a stationary or moving ball, etc.); m) with multiple consecutive obstacles blocking the lane relevant in the ODD (e.g., in the following order: ego-vehicle -motorcycle – car); n) on a curved section of road.

8.6. Avoid emergency braking before a passable object in the lane. A ‘passable object’ is such an object, that may be rolled over without causing an unreasonable risk to the vehicle occupants or other road users.

The test shall demonstrate that the fully automated vehicle is not initiating an Emergency Braking with a deceleration demand greater than 5 m/s2 due to a passable object in the lane relevant for the ODD (e.g., a manhole lid or a small branch) up to the maximum specified speed of the ADS.

8.6.1. This test shall be executed at least with the following scenarios, where relevant in the ODD: a) without a lead vehicle; b) with a passenger car target as well as a PTW target as the lead vehicle/other vehicle.

8.7. Following a lead vehicle

The test shall demonstrate that the fully automated vehicle is able to maintain and restore a stable motion and a safety distance to a vehicle in front and is able to avoid a collision with a lead vehicle which decelerates up to its maximum deceleration.

8.7.1. This test shall be executed at least with the following scenarios, where relevant in the ODD: a) across the entire speed range of the fully automated vehicle; b) using a passenger car target a PTW target as well as a bicycle target as lead vehicle, provided standardised PTW targets suitable to safely perform the test are available; c) for constant and varying lead vehicle velocities (realistic speed profile); d) for straight and curved sections of road; e) for different lateral positions of lead vehicle in the lane; f) with a deceleration of the lead vehicle of at least 6 m/s2 mean fully developed deceleration until standstill.

8.8. Lane change of another vehicle into lane (cut-in)

The test shall demonstrate that the fully automated vehicle is capable of avoiding a collision with a vehicle or other road user cutting into the lane of the fully automated vehicle up to a certain criticality of the cut-in manoeuvre.

8.8.1. The criticality of the cut-in manoeuvre shall be determined according to the provisions introduced Part 1 of this annex and depending on the distance between the rear-most point of the cutting-in vehicle and front-most point of the fully automated vehicle.

8.8.2. The test shall be executed at least with the following scenarios, where relevant for the ODD: a) with different TTC, distance and relative velocity values of the cut-in manoeuvre, covering types of cut-in scenarios in which a collision can be avoided and those in which a collision cannot be avoided; b) with cutting-in vehicles travelling at constant longitudinal speed, accelerating and decelerating; c) with different lateral velocities, lateral accelerations of the cut-in vehicle; d) with a passenger car, PTW as well as bicycle targets as the cutting-in vehicle, provided standardised PTW targets suitable to safely perform the test are available.

8.9. Stationary obstacle after lane change of the lead vehicle (cut-out)

The test shall demonstrate that the fully automated vehicle is capable of avoiding a collision with a stationary vehicle, road user or blocked lane that becomes visible after a preceding vehicle avoided a collision by an evasive manoeuvre. The test shall be based on the requirements set out in Annex II and scenario parameters in Part 1 of this annex. For conditions not tested that may occur within the defined operating range of the vehicle, the manufacturer shall demonstrate as part of the assessment described in Annex III, Part 2 to the satisfaction of the relevant authorities that the vehicle is safely controlled.

8.9.1. The test shall be executed at least with the following scenarios, where relevant for the ODD: a) with a stationary passenger car target centred in lane; b) with a PTW target centred in lane; c) with a stationary pedestrian target centred in lane; d) with a target representing a blocked lane centred in lane; e) with multiple consecutive obstacles blocking the lane (e.g. in the following order: ego-vehicle – lane change vehicle – motorcycle – car).

8.10. Parking

The test shall demonstrate that the ADS is able to park in different parking spaces and parking layouts under different conditions; and that during the parking manoeuvre it is not causing damage to the surrounding objects, road users and itself.

8.10.1. The test shall be executed at least with the following scenarios, where relevant for the ODD: (a) with parking spaces parallel and perpendicular to the road; (b) on even and slant surfaces; (c) with other vehicles in the surrounding parking spaces, including PTWs and bicycles; (d) parking to parking spaces with different geometrical dimensions; (e) on different road slope angles; (f) with another vehicle cutting in to the parking space during the parking manoeuvre.

8.11. Navigating in a parking facility

The test shall demonstrate that the ADS is able to handle the low driving speed and the general lack of visibility that may occur in a parking.

8.11.1. The test shall be executed at least with the following scenarios, where relevant for the ODD: a) with an initially obstructed pedestrian target crossing the path of the fully automated vehicle with a speed of 5 km/h. b) with a vehicle driving out of a parking place in front of the fully automated vehicle. c) with a stationary obstacle in the path of the fully automated vehicle. d) with different paths, where the infrastructure is obstructing the field of view. e) with a small obstacle on the floor after a ramp obstructed by other objects in the path of the fully automated vehicle.

8.12. Specific scenarios for motorway

8.12.1. Motorway entry The test shall demonstrate that the ADS is able to safely enter the motorway.

8.12.1.1. The test shall be executed at least with the following scenarios, where relevant for the ODD: a) with different vehicles, including a PTW approaching from the rear; b) with vehicles approaching with different speed from the rear; c) with a platoon of vehicles driving beside in the adjacent lane.

8.12.2. Motorway exit The test shall demonstrate that the ADS is able to safely exit the motorway.

8.12.2.1. The test shall be executed at least with the following scenarios, where relevant for the ODD: a) without a lead vehicle; b) with a passenger car target as well as a PTW target as the lead vehicle/other vehicle. c) with other vehicle(s) or obstacle(s) blocking the motorway exit.

8.12.3. Toll station The test shall demonstrate that the ADS is able to select the proper passing gate, and adapt its speed to that permitted within the toll area.

8.12.3.1. The test shall be executed at least with the following scenarios, where relevant for the ODD: a) with and without a lead vehicle; b) with other vehicles blocking the passing gate(s); c) with closed and opened passing gates. d) with different permitted speeds in the toll area.

8.13. For dual mode vehicles, transition between the manual driving mode and the fully automated mode.

The test shall demonstrate that the ADS takes over the DDT in a safe manner and only when the vehicle is standstill.

8.13.1. The test shall be executed at least with the following scenarios, where relevant for the ODD: a) with and without human driver present in the vehicle; b) with open and closed vehicle doors; c) with and without obstacles around the vehicle, d) inside and outside of the dedicated parking area, if applicable

8.13.2. The test shall be executed at least with the following scenarios, where relevant for the ODD: a) in a situation where transition is possible and executed; b) in a situation where transition is not possible to be executed.

PART 4

PRINCIPLES FOR CREDIBILITY ASSESSMENT FOR USING VIRTUAL TOOLCHAIN IN ADS VALIDATION

1. General

1.1. The credibility can be achieved by investigating and assessing five properties of Modelling and Simulation (M&S): (a) capability – what can the M&S do, and what the risks are associated with it; (b) accuracy – how well does M&S reproduce the target data; (c) correctness – how sound & robust are M&S data and algorithms; (d) usability – what training and experience is needed. (e) fit for purpose – how suitable is the M&S for the ODD and ADS assessment.

1.2. At the same time, the credibility assessment framework shall be general enough to be used for different M&S types and applications. However, the goal is complicated by the broad differences between ADS features and the variety of M&S types and applications. These considerations require a (risk-based/informed) credibility assessment framework relevant and appropriate to all M&S applications.

1.3. The credibility assessment framework provides a general description of the main aspects considered for assessing the credibility of an M&S solution together with principles on the role of third parties assessors in the validation process with respect to credibility. Concerning the latter point, the type-approval authority shall investigate the produced documentation supporting credibility at the assessment phase, whereas the actual validation tests occur once the manufacturer has developed the integrated simulation systems.

1.4. Ultimately, the outcome of the current credibility assessment shall define the envelope in which the virtual tool can be used to support the ADS assessment.

1.5. The requirements of this part are therefore intended to demonstrate the credibility of any simulation model or virtual toolchain for its use in ADS validation.

2. Definitions

For the purpose of this annex

2.1. ‘abstraction’ means the process of selecting the essential aspects of a source system or referent system to be represented in a model or simulation, while ignoring non-relevant aspects. Any modelling abstraction carries with it the assumption that shall not significantly affect the intended uses of the simulation tool.

2.2. ‘closed loop testing’ means a virtual environment that takes the actions of the element-in-the loop into account. Simulated objects respond to the actions of the system (e.g. system interacting with a traffic model).

2.3. ‘deterministic’ means a term describing a system whose evolution over time can be predicted exactly and a given set of input stimuli will always produce the same output.

2.4. ‘driver-in-the-loop (DIL)’ is typically conducted in a driving simulator used for testing the human–automation interaction design. DIL has components for the driver to operate and communicate with the virtual environment.

2.5. ‘Hardware-In-the-Loop (HIL)’ involves the final hardware of a specific vehicle sub-system running the final software with input and output connected to a simulation environment to perform virtual testing. HIL testing provides a way of replicating sensors, actuators and mechanical components in a way that connects all the I/O of the electronic control units (ECU) being tested, long before the final system is integrated.

2.6. ‘model’ is a description or representation of a system, entity, phenomenon, or process.

2.7. ‘model calibration’ is the process of adjusting numerical or modelling parameters in the model to improve agreement with a referent.

2.8. ‘model Parameter’ are numerical values used to support characterizing a system functionality. A model parameter has a value that cannot be observed directly in the real world but that must be inferred from data collected in the real world (in the model calibration phase).

2.9. ‘model-in-the-loop (MIL)’ is an approach which allows quick algorithmic development without involving dedicated hardware. This level of development usually involves high-level abstraction software frameworks running on general-purpose computing systems.

2.10. ‘open loop testing’ means a virtual environment that does not take the actions of the element-in-the loop into account (e.g. system interacting with a recorded traffic situation).

2.11. ‘probabilistic’ is a term pertaining to non-deterministic events, the outcomes of which are described by a measure of likelihood.

2.12. ‘proving ground or test-track’ is a physical testing facility closed to the traffic where the performance of an ADS can be investigated on the real vehicle. Traffic agents can be introduced via sensor stimulation or via dummy devices positioned on the track.

2.13. ‘sensor stimulation’ is a technique whereby artificially generated signals are provided to the element under testing in order to trigger it to produce the result required for verification of the real world, training, maintenance, or for research and development.

2.14. ‘simulation’ is the imitation of the operation of a real-world process or system over time.

2.15. ‘simulation model’ is a model whose input variables vary over time.

2.16. ‘simulation toolchain’ is a combination of simulation tools that are used to support the validation of an ADS.

2.17. ‘software-in-the-loop (SIL)’ is where the implementation of the developed model will be evaluated on general-purpose computing systems. This step can use a complete software implementation very close to the final one. SIL testing is used to describe a test methodology, where executable code such as algorithms (or even an entire controller strategy), is tested within a modelling environment that can help prove or test the software.

2.18. ‘stochastic’ means a process involving or containing a random variable or variables. Pertaining to chance or probability.

2.19. ‘validation of the simulation model’ is the process of determining the degree to which a simulation model is an accurate representation of the real world from the perspective of the intended uses of the tool.

2.20. ‘vehicle-in-the-loop (VIL)’ is a fusion environment of a real testing vehicle in the real-world and a virtual environment. It can reflect vehicle dynamics at the same level as the real-world and it can be operated on a vehicle test bed or on a test track.

2.21. ‘verification of the simulation model’ is the process of determining the extent to which a simulation model or a virtual testing tool is compliant with its requirements and specifications as detailed in its conceptual models, mathematical models, or other constructs.

2.22. ‘virtual testing’ is the process of testing a system using one or more simulation models.

3. Components of the credibility assessment framework and related documentation requirements

3.1. The credibility assessment framework introduces a way to assess and report the credibility of M&S based on quality assurance criteria where the levels of confidence in the results can be indicated. In other words, the credibility is established by evaluating the following M&S influencing factors that are considered as main contributors for M&S properties and therefore for the overall M&S credibility: (a) M&S management; (b) team’s experience and expertise; (c) M&S analysis and description; (d) data/input pedigree and (e) verification; validation, uncertainty Characterisation. Each of these factors indicates the level of quality achieved by M&S, and the comparison between the obtained levels and the required levels shall determine whether the M&S is credible and fit to use for virtual testing. A graphical representation of the relationship between the components of the credibility assessment framework is shown below.

3.2. Models and simulation management.

3.2.1. The M&S lifecycle is a dynamic process with frequent releases that shall be monitored and documented. Management activities shall be established to support the M&S in a work product management fashion. Relevant information on the following aspects shall be provided.

3.2.2. The M&S management process shall: (a) Describe the modifications within the releases; (b) Designate the corresponding software (e.g., specific software product and version) and hardware arrangement (e.g., XiL configuration); (c) Record the internal review processes that accepted the new releases; (d) Be supported throughout the full duration of the virtual model utilization.

3.2.3. Release management.

3.2.3.1. Any M&S toolchain’s version used to release data for certification purposes shall be stored. The virtual models constituting the testing toolchain shall be documented in terms of the corresponding validation methods and acceptance thresholds to support the overall credibility of the toolchain. The developer shall enforce a method to trace generated data to the corresponding M&S version.

3.2.3.2. Quality check of virtual data. Data completeness, accuracy, and consistency shall be ensured throughout the releases and lifetime of an M&S toolchain to support the verification and validation procedures.

3.2.4. Team’s experience and expertise.

3.2.4.1. Even though experience and expertise (E&E) are already covered in a general sense within the organization, it is important to establish the basis for confidence in the specific E&E for M&S activities.

3.2.4.2. The credibility of M&S depends not only on the quality of the simulation models but also on the E&E of the personnel involved in the validation and usage of the M&S. For instance, a proper understanding of the limitations and validation domain will prevent the possible misuse of M&S or misinterpretation of its results.

3.2.4.3. Therefore, it is important to establish the basis for the manufacturer’s confidence on the E&E of: (a) The Teams that will validate the simulation toolchain and, (b) The Teams that will use the validated simulation for the execution of virtual testing with the purpose of validating the ADS.

3.2.4.4. A proper management of the team’s E&E increases the level of confidence on the credibility of M&S and its outcomes by ensuring that the human factors behind the M&S are taken into consideration and any possible human component risk is controlled, as is expected in any suitable Management System

3.2.4.5. If the manufacturer’s tool chain incorporates or relies upon inputs from organisations or products outside of the manufacturer’s own team, the manufacturer will provide an explanation of measures it has taken to support its confidence in the quality and integrity of those inputs.

3.2.4.6. Team’s E&E consists of two levels.

3.2.4.6.1. Organizational level The credibility is established by setting up processes and procedures to identify and maintain skills, knowledge, and experience to perform M&S activities. The following processes shall be established, maintained and documented: (i) Process to identify and evaluate the individual’s competence and skills; (ii) Process for training competent personnel to perform M&S-related duties

3.2.4.6.2. Team level Once a M&S has been finalised, its credibility is mainly dictated by the skills and knowledge of the individual/team that will validate the M&S toolchain and use the M&S for the validation of the ADS. Credibility is established by documenting that these teams have received adequate training to fulfil their duties. The manufacturer shall then: (i) provide the basis for the manufacturer’s confidence in the E&E of the individual/team that validates the M&S Toolchain. (ii) provide the basis for the manufacturer’s confidence in the E&E of the individual/team that uses the simulation to carry out virtual testing in order to validate the ADS. The manufacturer’s demonstration of how it applies the principles of ISO 9001 or a similar best practice or standard to ensure the competence of its M&S organization and the individuals in that organisation will be the basis for this determination. The type-approval authority may not substitute its judgment on the E&E of the organisation or its members with that of the manufacturer.

3.2.5. Data/input pedigree

3.2.5.1. The data/input pedigree contains a record of traceability from the manufacturer’s data used in the validation of the M&S.

3.2.5.2. Description of the data used for the M&S (a) The manufacturer shall document the data used to validate the model and note important quality characteristics; (b) The manufacturer shall provide documentation showing that the data used to validate the models covers the intended functionalities that the toolchain aims at virtualising; (c) The manufacturer shall document the calibration procedures used to fit the virtual models’ parameters on the collected input data.

3.2.5.3. Effect of the data quality (e.g. data coverage, signal to noise ratio, and sensors’ uncertainty/bias/sampling rate) on model parameters uncertainty. The quality of the data used to develop the model will affect the estimation and calibration of the model parameters. Uncertainty in model parameters will be another important aspect in the final uncertainty analysis.

3.2.6. Data/output pedigree

3.2.6.1. The data/output pedigree contains a record of the M&S outputs used for the ADS validation.

3.2.6.2. Description of the data generated by the M&S (a) The manufacturer shall provide information on any data and scenarios used for virtual testing toolchain validation. (b) The manufacturer shall document the exported data and note important quality characteristics. (c) The manufacturer shall trace a M&S output to the corresponding simulation setup

3.2.6.3. Effect of data quality on M&S credibility (a) The M&S output data shall be sufficiently wide to ensure the correct execution of the validation computation. The data shall sufficiently reflect the ODD relevant to the virtual assessment of the ADS. (b) The output data shall allow for a consistency/sanity check of the virtual models via possibly exploiting redundant information

3.2.6.4. Managing stochastic models (a) Stochastic models shall be characterised in terms of their variance (b) Stochastic models shall be ensured the possibility of deterministic re-execution

3.3. M&S analysis and description

3.3.1. The M&S analysis and description aim to define the whole M&S and identify the parameter space that can be assessed via virtual testing. It defines the scope and limitations of the models and toolchain and the uncertainty sources that can affect its results.

3.3.2. General description

3.3.2.1. The manufacturer shall provide a description of the complete toolchain along with how the simulation data will be used to support the ADS validation strategy

3.3.2.2. The manufacturer shall provide a clear description of the test objective

3.3.3. Assumptions, known limitations and uncertainty sources

3.3.3.1. The manufacturer shall motivate the modelling assumptions that guided the design of the M&S toolchain

3.3.3.2. The manufacturer shall provide evidence on: (i) How the manufacturer-defined assumptions play a role in defining the limitations of the toolchain; (ii) The level of fidelity required for the simulation models

3.3.3.3. The manufacturer shall provide justification that the tolerance for sim-real correlation is acceptable for the test objective

3.3.3.4. Finally, this section shall include information on the sources of uncertainty in the model. This will represent an important input to final uncertainty analysis, which will define how the model outputs can be affected by the different sources of uncertainty of the model used.

3.3.4. Scope (how the M&S is used in the ADS validation)

3.3.4.1. The credibility of the virtual tool shall be enforced by a clearly-defined scope of utilization the developed models.

3.3.4.2. The matured M&S shall allow a virtualisation of the physical phenomena to a degree of accuracy which matches the fidelity level required for certification. Thus, the M&S will act as a ‘virtual proving ground’ for ADS testing.

3.3.4.3. Simulation models need dedicated scenarios and metrics for validation. The scenario selection used for validation shall be sufficient so that the toolchain will perform in the same manner in scenarios outside of the validation scope.

3.3.4.4. The Manufacturer shall provide a list of validation scenarios together with the limitations of the corresponding parameters.

3.3.4.5. The ODD analysis is a crucial input to derive requirements, scope and effects that the M&S must consider in order to support ADS validation.

3.3.4.6. Parameters generated for the scenarios will define extrinsic and intrinsic data for the toolchain and the simulation models.

3.3.5. Criticality assessment

3.3.5.1. The simulation models and the simulation tools used in the overall tool-chain shall be investigated in terms of their responsibility in case of a safety error in the final product. The proposed approach for criticality analysis is derived from ISO 26262, which requires qualification for some of the tools used in the development process.

3.3.5.2. In order to derive how critical the simulated data are, the criticality assessment shall consider the following parameters: (a) The consequences for human safety e.g. severity classes in ISO 26262. (b) The degree to which the simulated results influence’s the ADS

3.3.5.3. From the perspective of the criticality assessment, the three possible cases for assessment are: (a) Those models or tools that are clear candidates for following a full credibility assessment; (b) Those models or tools that may or may not be candidates for following the full credibility assessment at the discretion of the assessor; (c) Those models or tools that are not required to follow the credibility assessment.

3.4. Verification

3.4.1. The verification of an M&S involves analysing the correct implementation of the conceptual/mathematical models building up the M&S toolchain. The verification contributes to the M&S’s credibility by providing assurance that the M&S will not exhibit unrealistic behaviour for a set of inputs that cannot be tested. The procedure is based on a multi-step approach including code verification, calculation verification and sensitivity analysis.

3.4.2. Code verification

3.4.2.1. Code verification involves tests demonstrating that no numerical/logical flaws affect the virtual models

3.4.2.2. The manufacturer shall document the execution of proper code verification techniques, e.g. static/dynamic code verification, convergence analysis and comparison with exact solutions if applicable

3.4.2.3. The manufacturer shall provide documentation showing that the exploration in the domain of the input parameters was sufficiently wide to identify parameters’ combination for which the M&S shows unstable or unrealistic behaviour. Coverage metrics of combinations of parameters may be used to demonstrate the required exploration of the models behaviours.

3.4.2.4. The manufacturer shall adopt sanity/consistency checking procedures whenever data allows.

3.4.3. Calculation verification

3.4.3.1. Calculation verification deals with the estimation of numerical errors affecting the M&S

3.4.3.2. The manufacturer shall document numerical error estimates (e.g. discretization error, rounding error, iterative procedures convergence);

3.4.3.3. The numerical errors shall be kept sufficiently bounded to not affect validation.

3.4.4. Sensitivity analysis

3.4.4.1. Sensitivity analysis aims to quantify how model output values are affected by changes in the model input values and thus to identify the parameters having the greatest impact on the simulation model results. The sensitivity study also helps to determine the extent to which the simulation model satisfies the validation thresholds when it is subjected to small variations of the parameters. It is therefore fundamental to support the credibility of the simulation results.

3.4.4.2. The manufacturer shall provide supporting documentation demonstrating that the most critical parameters influencing the simulation output have been identified by means of sensitivity analysis techniques such as by applying a perturbation of the model’s parameters;

3.4.4.3. The manufacturer shall demonstrate that robust calibration procedures have been adopted when identifying and calibrating the most critical parameters in order to increase the credibility of the developed toolchain.

3.4.4.4. Ultimately, the sensitivity analysis results will also help to define the inputs and parameters whose uncertainty Characterisation needs particular attention in order to properly define the uncertainty of the simulation results.

3.4.5. Validation

3.4.5.1. The quantitative process of determining the degree to which a model or a simulation is an accurate representation of the real world from the perspective of the intended uses of the M&S requires the selection and definition of several elements.

3.4.5.2. Measures of performance (metrics)

3.4.5.2.1. The measures of performance are the metrics used to compare the simulation model with the real world. Measures of performance are defined during the M&S analysis.

3.4.5.2.2. Metrics for validation may include: (i) discrete value analysis e.g. detection rate, firing rate; (ii) time evolution e.g. positions, speeds, acceleration; (iii) flow of actions-based analysis e.g. distance/speed calculations, TTC calculation, brake initiation.

3.4.5.3. Goodness of fit measures

3.4.5.3.1. The analytical frameworks are used to compare real world and simulation metrics. They are generally key performance indicators (KPIs) indicating the statistical comparability between two sets of data.

3.4.5.3.2. The validation shall show that these KPIs are met.

3.4.5.4. Validation methodology

3.4.5.4.1. The manufacturer shall define the logical scenarios used for virtual testing toolchain validation. They shall be able to cover to the maximum possible extent the ODD of virtual testing for ADS validation

3.4.5.4.2. The exact methodology depends on the structure and purpose of the toolchain. The validation may consist of one or more of the following: (i) validate subsystem models e.g. environment model (road network, weather conditions, road user interaction), sensor models (radio detection and ranging (RADAR), light detection and ranging (LiDARs), camera), vehicle model (steering, braking, powertrain); (ii) validate vehicle system (vehicle dynamics model together with the environment model); (iii) validate sensor system (sensor model together with the environment model); (iv) validate integrated system (sensor model + environment model with influences from vehicle model).

3.4.5.5. Accuracy requirement

3.4.5.5.1. The requirement for the correlation threshold is defined during the M&S analysis. The validation shall show that the KPIs identified in 3.4.5.3.1. of the present part are met.

3.4.5.6. Validation scope (the part of the toolchain to be validated)

3.4.5.6.1. A toolchain consists of multiple tools, and each tool will use a number of models. The validation scope includes all tools and the relevant models subject to validation.

3.4.5.7. Internal validation results

3.4.5.7.1. The documentation shall not only provide evidence of the simulation model validation but shall also be used to obtain sufficient information on the processes and products that provide overall credibility of the toolchain used.

3.4.5.7.2. Documentation/results may be carried over from previous credibility assessments.

3.4.5.8. Independent validation results

3.4.5.8.1. The type-approval authority shall assess the documentation provided by the manufacturer and may carry out physical tests of the complete integrated tool

3.4.5.9. Uncertainty characterisation

3.4.5.9.1. This section is concerned with characterising the expected variability of the virtual toolchain results. The assessment shall be consist of two phases. In a first phase the information collected in the M&S analysis and description and the data/input pedigree sections are used to characterise the uncertainty in the input data, in the model parameters and in the modelling structure. Then, by propagating all the uncertainties through the virtual toolchain, the uncertainty in the model results is quantified. Depending on the uncertainty in the model results, proper safety margins will need to be introduced by the manufacturer in the use of virtual testing for ADS validation.

3.4.5.9.2. Characterisation of the uncertainty in the input data The manufacturer shall demonstrate to have appropriately estimated the critical model’s inputs by means of robust techniques such as multiple repetitions for the assessment of the quantity;

3.4.5.9.3. Characterisation of the uncertainty in the model parameters (following calibration) The manufacturer shall demonstrate that the critical model’s parameters that cannot be estimated identically are characterised by means of a distribution and/or confidence intervals;

3.4.5.9.4. Characterisation of the uncertainty in the M&S structure The manufacturer shall provide evidence that the modelling assumptions are given a quantitative characterisation of the generated uncertainty (e.g. comparing the output of different modelling approaches whenever possible).

3.4.5.9.5. Characterisation of aleatory vs. epistemic uncertainty: The manufacturer shall aim to distinguish between the aleatory component of the uncertainty (which can only be estimated but not reduced) and the epistemic uncertainty deriving from the lack of knowledge in the virtualisation of the process (that can instead be reduced).

4. Documentation structure

4.1. This section sets out how the above information will be collected and organised in the documentation provided by the manufacturer to the relevant authority.

4.2. The manufacturer shall produce a document (a ‘simulation handbook’) structured in line with the present outline to provide evidence for the topics presented.

4.3. The documentation shall be delivered together with the corresponding release of the M&S and related produced data.

4.4. The manufacturer shall provide clear reference