Safe and certiﬁed software
for autonomous mobility
How Apex.AI Certiﬁed ROS 2 According to ISO 26262 ASIL-D
Dejan Pangercic (dejan@apex.ai)© 2021 Apex.AI, Inc.

Relevance to Autoware Community
1. Autoware.Auto uses ROS2. If AWF members want tocertify whole or parts of Autoware.Auto and then sellthat as part of the fully certiﬁed complete SW AD
stack - they need a certiﬁed framework
2. ROS2 is a C++14 product. This talk is largely about
how to certify any C++ code base (independent ofROS2, Autoware.Auto, ...)
3. The process of certiﬁcation improved the quality ofApex.AI fork of ROS2. Consequently this talk is also
about the code quality
4. ISO 26262 is at the center of standardization in theautomotive industry. There is no more important
standard and one can not avoid it if you build
production systems for automotive. Consequently
this talk is about sharing an experience of going
through the process of ISO 26262 certiﬁcation© 2021 Apex.AI, Inc.

But the entire AD Stack is Huge
Focus of this talk© 2021 Apex.AI, Inc.

AVP

© 2021 Apex.AI, Inc.

We have a Base Functional AD System - What Next?
1. System Safety:
a. ISO 26262 Certiﬁcationi. Code2. AV Technology:
a. Object and event detection and response
b. Fallback systems

ii. HWb. System speciﬁcation and operating environment(ODD)
c. HARAd. Design for redundancye. Validation plan

3. AV Operation:
a. ODDb. AV Operators
c. Incident response and management

i. System validation (ISO 15288)
f. SOTIF
i. Scenario-based testing with statistical samplingin simulation (NCAP, NHTSA scenarios)
g. Closed course testing

4. Interfaces:a. Passenger and road user interface
b. Cybersecurityc. Data management

h. Public road testingi. Simulation
i. SIL and HIL Testing

Source: Motional, VSSA, 2021
© 2021 Apex.AI, Inc.

We have a Base Functional System - What Next?
1. System Safety:
a. ISO 26262 Certiﬁcation

Focus of the rest ofthe talki. Code2. AV Technology:
a. Object and event detection and response
b. Fallback systems

ii. HWb. System speciﬁcation and operating environment(ODD)
c. HARAd. Design for redundancye. Validation plan

3. AV Operation:
a. ODDb. AV Operators
c. Incident response and management

i. System validation (ISO 15288)
f. SOTIF
i. Scenario-based testing with statistical samplingin simulation (NCAP, NHTSA scenarios)
g. Closed course testing

4. Interfaces:
a. Passenger and road user interface
b. Cybersecurityc. Data management

h. Public road testingi. Simulation
i. SIL and HIL Testing

Source: Motional, VSSA, 2021
© 2021 Apex.AI, Inc.

Apex.OS Development Lifecycle
ISO 26262/SEooC/part3,part6…. processes

Automotive
Stakeholder
Require-ments (ASR)

Requirements

Architecture

Unit Design

V&VConf. Reviews

builtin_interfaces_cert

Elicitation,
SafetyConcept, SW
SafetyRequirements

UML (uniﬁed
modelinglanguage),
FMEA

SCA (Static CodeAnalysis), SW
practices outline,
coverage, FMEA

Req, arch, unit,
integration,system,performance,
fault injection
tests

Safety manual,
Restrictions,
Traceability

connext_micro_support_cert
allocator_cert
logging_cert
rclcpp_cert
threading_cert

builtin_interfaces

Apply real-timeand determinism
constraints

connext_micro_support
allocator
logging
rclcppFeature set
reduction

1. Memory static
2. Removeblocking calls andrecursionsApex_ecu_monitor (native)
Apex_utils (native)

Apex.OS Certthreading© 2021 Apex.AI, Inc.

Apex.OS Certiﬁcation Activities per package
1. Reduce the feature set of a package and extensions
2. Investigation to make APIs memory static & ensure
no blocking calls
3. Static Code Analysis (SCA)

Total 24 pkgs selected for ﬁrst release of Apex.OS

4. Structural Coverage (statement, branch and MC/DC)
5. Notations of Designs (modelling diagrams)

Some activities such as Tool classiﬁcation and qualiﬁcation,

6. Principles of SW architecture and design

integration testing done at Apex.OS level.

7. Control and data ﬂow analysis
8. Integration and Specialized tests
9. Requirements and traceability

Close to 100 safety artifacts had to be generated to provide
evidence of ASIL D compliance to our certiﬁcation agency.

10. Safety Analysis (FMEA)
11. Generate Safety Artifacts (TUV submission)
12. Testing on Target platform/hardware
13. Tool Classiﬁcation and Qualiﬁcation© 2021 Apex.AI, Inc.

Real-Time Gaps in ROS 2ROS 2 exhibits the following gaps to enable real-time performance.

Real-Time GapsNon static mem operations

Standard containers

Runtime mem
allocation

non-real-time middlewareStandard exceptions

Mem
Exception throw Handler lookup
fragmentation causes mem non-deterministic
allocation
due to inheritanceThread priorities,
scheduling, pinning

No control(std::thread)Standard threading

Blocking calls/deadlocks

Higher risk ofdead locks since
no tooling

Scheduling based onreadiness of data (executor)

Increasedthrashing© 2021 Apex.AI, Inc.

Apex.OS Solution
Apex.OS addresses the following gaps to achieve real-time performance.

Real-Time

Real-Time DDS

static mem operations

apex::containers

apex::string

apex::
map/setapex::vector

apex::exceptions

apex::
mallocProcess
deﬁned to
makecatching
exceptions
deterministic

apex::threading

apex::thread

Blocking calls/deadlocks

Better control
over threadpriorities,
scheduling andpinning

EliminatedReliance on OS scheduler
vs ROS executor

Greatly
reducedthrashing© 2021 Apex.AI, Inc.

Apex.OS Certiﬁcation Activities per package
1. Reduce the feature set of a package and extensions
2. Investigation to make APIs memory static & ensure
no blocking calls
3. Static Code Analysis (SCA)

Total 24 pkgs selected for ﬁrst release of Apex.OS

4. Structural Coverage (statement, branch and MC/DC)
5. Notations of Designs (modelling diagrams)

Some activities such as Tool classiﬁcation and qualiﬁcation,

6. Principles of SW architecture and design

integration testing done at Apex.OS level.

7. Control and data ﬂow analysis
8. Integration and Specialized tests
9. Requirements and traceability

Close to 100 safety artifacts had to be generated to provide
evidence of ASIL D compliance to our certiﬁcation agency.

10. Safety Analysis (FMEA)
11. Generate Safety Artifacts (TUV submission)
12. Testing on Target platform/hardware
13. Tool Classiﬁcation and Qualiﬁcation© 2021 Apex.AI, Inc.

Branch coverage vs. MC/DC coverage
Branch coverage

MC/DC coverage

Each branch (True and False) should be tested atleast once

For example (A && B),

A && B
False
True

Every condition in a decision (True and False)
should be tested independently

1. Create the truth table

ROWA

B

Res

1

TT

T

2

TFF

3

FTF

4

FFF

2. Find pairs for which only one condition
independently aﬀect the outcome
a = {1,3}, b= {1,2} -> 1, 2, 3 condition should betested.
3. For n conditions we only require n+1 tests© 2020 Apex.AI, Inc. Conﬁdential.

Structural Coverage
To get to the 100% of line (statement), branch andMC/DC (pairs) test coverage we had to add 3000 tests(on top of the 1500 existing tests).

©2021 Apex.AI, Inc. Confidential

What was tedious?
• Getting 100% MC/DC coverage for heavily templated modern C++ code is tedious.
◦ Commercial coverage tool has issues, while it parses modern C++ codes such as a lambda
function and template code. (e.g. on the next slide)• The code base has a lot of hard to reach defensive type coding.
◦ Required signiﬁcant stubbing/mocking of C++ standard library, middleware, and externalfunctions that are implemented in Apex.OS. (e.g. on the next slide)© 2021 Apex.AI, Inc.

Issues Parsing Certain Modern C++ Constructs• A method with multiple lambda functions◦ The commercial coverage tool could not parse amethod that contained multiple lambda functions.
Solution: ﬁxing the bug of the commercialcoverage tool.

• Template code with locally deﬁned class◦ The commercial coverage tool cannot parse a classinternally deﬁned in a function or the class that isused for the parameter of the template class orfunction.

class Sub{};class Pub{};class C{
public:Sub* create_sub();
Pub* create_pub();};

void testme(C* ptr){
C* node = new C;
auto get_sub = [&node] { return node->create_sub(); };auto get_pub = [&node] { return node->create_pub(); };}int main(){class A {};f<A>();}

Solution: deﬁning the class with the global scope© 2021 Apex.AI, Inc.

Mocking GNU C Lib Functions: Example: clock_gettime( )rcutils_system_time_now(rcutils_time_point_value_t * now){
RCUTILS_CHECK_ARGUMENT_FOR_NULL(now,RCUTILS_RET_INVALID_ARGUMENT);
struct timespec timespec_now;
int32_t posix_error;
posix_error = clock_gettime(CLOCK_REALTIME, &timespec_now);
if (posix_error != 0) {RCUTILS_SET_ERROR_MSG("clock_gettime error");return RCUTILS_RET_ERROR;}
if (RCUTILS_WOULD_BE_NEGATIVE(timespec_now.tv_sec,timespec_now.tv_nsec)) {RCUTILS_SET_ERROR_MSG("unexpected negative time");return RCUTILS_RET_ERROR;}

int clock_gettime(clockid_t clk_id, struct timespec * tp) __THROW__{
int ret = 0;
if (nullptr != timeUnixPtr) {ret = timeUnixPtr->clock_gettime(clk_id, tp);}
return ret;}
TEST_F(time_gmock, rcutils_system_time_now) {rcutils_time_point_value_t now = 0;
rcutils_ret_t ret;
EXPECT_CALL(*timeUnixPtr, clock_gettime(_,_)).WillRepeatedly(Return(-1));
ret = rcutils_system_time_now(&now);
EXPECT_EQ(ret, RCUTILS_RET_ERROR);
rcutils_reset_error();}

Apex.OS source code

Apex.OS test code

clock_gettime(CLOCK_REALTIME, &timespec_now)

clock_gettime(CLOCK_REALTIME, &timespec_now)

GNU C LibraryMock© 2021 Apex.AI, Inc.

Apex.OS Certiﬁcation Activities per package
1. Reduce the feature set of a package and extensions
2. Investigation to make APIs memory static & ensure
no blocking calls
3. Static Code Analysis (SCA)

Total 24 pkgs selected for ﬁrst release of Apex.OS

4. Structural Coverage (statement, branch and MC/DC)
5. Notations of Designs (modelling diagrams)

Some activities such as Tool classiﬁcation and qualiﬁcation,

6. Principles of SW architecture and design

integration testing done at Apex.OS level.

7. Control and data ﬂow analysis
8. Integration and Specialized tests
9. Requirements and traceability

Close to 100 safety artifacts had to be generated to provide
evidence of ASIL D compliance to our certiﬁcation agency.

10. Safety Analysis (FMEA)
11. Generate Safety Artifacts (TUV submission)
12. Testing on Target platform/hardware
13. Tool Classiﬁcation and Qualiﬁcation© 2021 Apex.AI, Inc.

What was technically challenging?
• There are no good commercial tools for identifying runtime memory allocations and blocking calls.
We created new internal tool (apex_tracing_check) that uses LTTng framework to ﬂag infractions.

• Making exceptions handling memory static is complex (and still a research topic)
We solved it by patching system malloc() and a special (exception handling) memory pool.
(see next slide)© 2021 Apex.AI, Inc.

Elimination of Memory Allocation and Blocking Calls (MA/BC) - Approach• We implemented apex_tracing_check tool which is based on LTTng• The code is instrumented by adding the macro on top of the function• It requires to be build with some extra compilation ﬂags to enable themacro• After this, test cases are executed to ﬁnd infractions• Example on how apex_tracing_check will detect and report infraction/s

Example of mutex infraction in sleep_for() function© 2021 Apex.AI, Inc.

Elimination of Memory Allocation and Blocking Calls (MA/BC) - Findings• Using apex_tracing_check and having 100% MC/DC coverage, it’s possible to verify that there is no MA/BC in runtime.

Replacing std::string and std::exception to
avoid memory allocations in runtime

Replacing std::mutex with apex::time_limit_mutex
to avoid blocking system call in runtime© 2021 Apex.AI, Inc.

Allocation Handling during Exception (apex_malloc pkg)

Allocation requestfrom an exception
during runtime?

Back to runtime__cxa_allocate_exception malloc()

Apex::exception_allocator()

Exception routine uses the ptr to service.
Stack unwinding will release the blockback to “pre-deﬁned mem pool”

Search for a free block in“pre-deﬁned mem pool” andreturn ptr to this free block

If search return null ptr (unlikely): c++ > qnx
emergency pool. If emergency pool return nullthen process terminates

Not from heap,
no fragmentation,
deterministic memory allocation© 2021 Apex.AI, Inc.

Apex.OS Certiﬁcation Activities per package
1. Reduce the feature set of a package and extensions
2. Investigation to make APIs memory static & ensure
no blocking calls
3. Static Code Analysis (SCA)

Total 24 pkgs selected for ﬁrst release of Apex.OS

4. Structural Coverage (statement, branch and MC/DC)
5. Notations of Designs (modelling diagrams)

Some activities such as Tool classiﬁcation and qualiﬁcation,

6. Principles of SW architecture and design

integration testing done at Apex.OS level.

7. Control and data ﬂow analysis
8. Integration and Specialized tests
9. Requirements and traceability

Close to 100 safety artifacts had to be generated to provide
evidence of ASIL D compliance to our certiﬁcation agency.

10. Safety Analysis (FMEA)
11. Generate Safety Artifacts (TUV submission)
12. Testing on Target platform/hardware
13. Tool Classiﬁcation and Qualiﬁcation© 2021 Apex.AI, Inc.

FMEA
FMEA (Failure Mode and Eﬀects) Analysis wasperformed on each public API in every Cert
package.
New tests were added as a result of FMEA=>
add example bug from cpputils
33 Software Safety Requirements were added
as a result of FMEA Activity.
40 Restrictions and 25 Recommendations
added to Safety Manual because of FMEA.

Activity# Real Issues Found# Files Changed# Merge Requests

Git Commits# Changed code lines

FMEA Analysis

17551736223© 2021 Apex.AI, Inc.

Examples of Safety Related Changes from ROS 2 to Apex.OS
Example of requirement tracing“The function rclcpp::Context::sleep_for() shall timeout immediately when zero and negative values are given
for the nanoseconds argument”
Solution: The function was not working as described. During the requirement tracing the function and its
corresponding tests were ﬁxed
“rclcpp shall provide functionality to assert the liveliness of a publisher”
Solution: There were no tests verifying these requirements. As a result of the analysis a test was added
Example of issue detected as a result from FMEA
“If rclcpp publisher takes more time than is expected to publish a message the application could malfunction”
Solution: If used along with the ApexNode, function calls that exceed the expected time may cause
max_cycle_time to be exceeded, which will then notify the user of the failure.

© 2021 Apex.AI, Inc.

Certiﬁcation in Numbers● First round of Apex.OS Cert contained ~65K lines of code
● 14 person years of eﬀort (1 full time for 2 years, 12 full time for a year)● 24 ROS 2 + native Apex.OS packages certiﬁed● > $5M cost in tool licenses, infrastructure, and engineering resources● 100% statement, branch, and MC/DC coverage● ~3000 new tests added to fulﬁll safety/certiﬁcation compliance● ~300 safety requirements generated from FMEA, TSC, and Tools C&Q● ~100 artifacts submitted to third party auditor (TÜV NORD) for ISO 26262 ASIL D compliance assessment(~2000 A4 pages if printed)○ Total of 5 iterations of audits were conducted by TÜV NORD

© 2021 Apex.AI, Inc.

Summary of Safety Related Changes from ROS 2 to Apex.OS
ROS 2Apex.OS• Not real-time/deterministic• No formal requirements compliant to ISO 26262• No safety analysis• No Static Code Analysis (SCA) or code coverage• Several changes to improve real-time/determinism.
Removed all runtime memory allocations and blocking calls.
• Formal requirements written and traced to design and test.
• SW FMEA carried on every package to derive additional
requirements and/or restrictions.
• Full compliance to AUTOSAR cpp14 V3.19 coding guidelines.
• Full MC/DC coverage.

Getting full MC/DC coverage and removing
runtime memory allocations was challenging
and took most of the time!

© 2021 Apex.AI, Inc.

ThanksApex.AI
Autoware.AutoAutoware Foundation
Contact: dejan@apex.ai© 2021 Apex.AI, Inc.