OSAKA NDS EMBEDDED LINUX CROSS FORUM #8
2019- 2- 5
http://www.nds-osk.co.jp/forum/onlcf8.html

Embedded Linux Conference Europe 2018

三菱電機におけるOSS自動試験活用のご紹介
三菱電機(株) 情報技術総合研究所
リアルタイムプラットフォーム技術部
茂田井寛隆© Mitsubishi Electric Corporation

OUTLINE• Who I am
• Overview• Related Tools

本資料はEmbedded Linux Conference Europe 2018にて発表された資料です– Automated Test Framework / Fuego– Real-time latency tool / cyclictest– Tracing kernel feature / ftrace• Issue• Approach– Our Use Cases• Conclusion and Future work
© Mitsubishi Electric Corporation

1WHO I AM• Hirotaka MOTAI– Software Researcher for embedded systemsof Mitsubishi Electric Corp.

• We have collaborated with LF projects.
– LTSI: Long Term Support Initiative– AGL : Automotive Grade Linux– FUEGO: Test framework• Specifically designed for testing Embedded Linux© Mitsubishi Electric Corporation2• Who I am
• Overview• Related Tools– Automated Test Framework / Fuego– Real-time latency tool / cyclictest– Tracing kernel feature / ftrace• Issue• Approach– Our Use Cases• Conclusion and Future work
© Mitsubishi Electric Corporation

3

OVERVIEW• Linux can be adapted to various embedded
devices, even though they need a hard real-timeresponse.
• We need tons of time to ensure adequate real-timeperformance.
– Real-time applications need to satisfy timing constraints.
– We have to avoid kernel changes which might cause long
delays.
© Mitsubishi Electric Corporation

4OVERVIEW• I have developed a new test script on Fuego.
– measure the real-time performance, plus get tracing.
– get clues to isolate the problem whether it was caused
by our changes or not.
FUEGO
Test Script

stressprogram

Test Target Kernel

test logs &trace data
ourreposcyclictest

Build scriptPC

ftraceourchanges

H/W© Mitsubishi Electric Corporation5• Who I am
• Overview• Related Tools– Automated Test Framework / Fuego– Real-time latency tool / cyclictest– Tracing kernel feature / ftrace• Issue• Approach– Our Use Cases• Conclusion and Future work
© Mitsubishi Electric Corporation

6FUEGO• Fuego is the automated test framework– created by LTSI project, based on Jenkins.
– OSS: anyone can use and contribute!
– AGL-JTA: AGL chose Fuego as standard test
environment.
FUEGO
Test Script

stressprogram

Test Target Kernel

test logs &trace data
ourrepos

cyclictest

Build scriptPC

ftraceourchanges

H/W© Mitsubishi Electric Corporation7FUEGO• Fuego = "test distribution + Jenkins + host scripts +pre-packaged tests" on container• Fuego can do specific tests automatically that istriggered by software update.
FUEGO
Host scripts• build• deploy• test run• processing

Jenkins

test programs
on host(Fuego Linux Distribution)
Docker Container

repos

Host PC

test programs ontarget(compiled by Fuego)

Test Target Linux
Target device© Mitsubishi Electric Corporation

8FUEGO• You can click to start manually and monitor tests on Jenkins.
click to startresultmonitor© Mitsubishi Electric Corporation

9FUEGO• You can also check test results on Jenkins.

10© Mitsubishi Electric CorporationCYCLICTEST• What is Cyclictest?
– Benchmark tool for interval timer latency.
– Refer to: https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/cyclictest

FUEGO
Test Script

stressprogram

Test Target Kernel

test logs &trace data
ourrepos

cyclictest

Build scriptPC

ftraceourchanges

H/W© Mitsubishi Electric Corporation

11

CYCLICTEST• What is Cyclictest?
– Benchmark tool for interval timer latency.
– Refer to: https://wiki.linuxfoundation.org/realtime/documentation/howto/tools/cyclictest
Cyclictest Histgram

Cyclictest
100000

50000
45000

10000

35000
30000

Count

Latency[usec]

40000

25000
20000

1000
100

15000
10000

105000010102030

4050

60

70

80

90010000

Event number
CPU#0

20000

30000

40000

50000

Latency[usec]

CPU#1

CPU#0

CPU#1

12

© Mitsubishi Electric CorporationFTRACE• Ftrace have been in the kernel since Linux v2.6.27.
• Traces kernel without recompiling.
• Useful for data gathering, debugging, and performancetuning.
• Detailed in Documentation/trace/index.rst
FUEGO
Test Script

stressprogram

Test Target Kernel

test logs &trace data
ourrepos

cyclictest

Build scriptPC

ftraceH/Wourchanges

13© Mitsubishi Electric CorporationFTRACE• The Function Tracer# tracer: function
##
###
#
#
TASK‐PID#| |stress‐ng‐shm‐s‐1257
stress‐ng_1h_pl‐1194
stress‐ng‐shm‐s‐1257_‐‐‐‐‐=> irqs‐off/ _‐‐‐‐=> need‐resched| / _‐‐‐=> hardirq/softirq|| / _‐‐=> preempt‐depth||| /delay
CPU# ||||TIMESTAMP FUNCTION|||||||[000] .... 7523.267555: vmacache_find <‐find_vma[001] .... 7523.267556: mutex_unlock <‐rb_simple_write[000] .... 7523.267559: handle_mm_fault <‐__do_page_fault[snip]
14© Mitsubishi Electric Corporation• Who I am
• Overview• Related Tools– Automated Test Framework / Fuego– Real-time latency tool / cyclictest– Tracing kernel feature / ftrace• Issue• Approach– Our Use Cases• Conclusion and Future work15© Mitsubishi Electric Corporation

ISSUE
What changes have occurred the delay ?
• Our changes?
• Potential performance issues?

It is necessary to save a trace log
to isolate the problem.
16© Mitsubishi Electric Corporation• Who I am
• Overview• Related Tools– Automated Test Framework / Fuego– Real-time latency tool / cyclictest– Tracing kernel feature / ftrace• Issue• Approach– Our Use Cases• Conclusion and Future work17

© Mitsubishi Electric Corporation

Our Approach• Do in a simple way.
– Script-based Test driver program– 1: setup ftrace configurations– 2: run stress program
Fuego– 3: run cyclictest– 4: save a log with a ftrace data
Test

newly introduced

stressprogram

cyclictest

script• Cyclictest option– "--breaktrace breaktime[us]"can stop tracing
when latency > breaktime

test logs

ftracetrace logs

Test Target Kernel

PC

H/W18

© Mitsubishi Electric Corporation

STOP TRACING• But, --breaktrace option will stop not only tracing
but also testing, when diff > breaktime.
stop873
if (!stopped && tracelimit&& (diff > tracelimit)) {tracing874stopped++;875
tracemark("hit latency
threshold (%llu > %d)",stop876(unsigned longtestinglong) diff, tracelimit);877
shutdown++;878
pthread_mutex_lock(&break_thread_id_lock);879
if (break_thread_id == 0)880
break_thread_id = stat‐>tid;881
break_thread_value = diff;882 recordpthread_mutex_unlock(&break_thread_id_lock);The
may not be the worst case…
883}
19© Mitsubishi Electric Corporation

Improved Approach• Using a ftrace feature "Snapshot"– Cyclictest is modified to take a snapshot when maximumlatency is updated.

newly introduced

Cyclictest Histgram

Fuego

100000Count10000save the snapshot log
in the worst case

1000
100

Test
scripttest logs

snapshot

10snapshot logs

1010000

20000

30000

Latency[usec]
CPU#040000

stressprogramcyclictest

ftraceTest Target Kernel

50000

PC

H/W

20

CPU#1

© Mitsubishi Electric Corporation

OUR ACTUAL CASE
WITH AN IMPROVED APPROACH…

21© Mitsubishi Electric Corporation

Detail of Our Evaluation Environment• Our test case
– Latency of real-time priority process
under some kinds of stress by non-realtime process.
FUEGO
Test
scriptcyclictest

test logs

Debianstress-ng

snapshot logs

LTSI KernelPC

Minnowboard• Cyclictest with realtimepriority• Stress-ng with nonrealtime priority– making stressful22© Mitsubishi Electric Corporation

Hardware and OS• Target board: Minnowboard Turbot Dual-core– Intel Atom E3826• #Cores / Threads: 2/2• Freq / Cache : 1.46GHz / 1MB– 2GB DDR3L 1067MT/s– Storage: microSD– Ethernet: Intel i211

• Debian Gnu/Linux 9.5.0– Linux 4.14 LTSI

FUEGO
Test
scriptcyclictest

test logs

Debiansnapshot logsPCstressng

LTSI Kernel
Minnowboard

23© Mitsubishi Electric Corporation

Stress-ng• stress-ng has stressors for a lot of components– cpu, fork, timer, sync, dentry, flock, udp, pipe,
semaphore…
• Beware, extreme scenarios seldom happen in real-life.

• $ ./stress-ng –stressors | wc –l
207

FUEGO
Test
scriptcyclictest

test logs

Debiansnapshot logs
PC

stressng

LTSI Kernel
Minnowboard

24© Mitsubishi Electric Corporation

Result• Stress-ng– Create&delete directory entry (dentry), 8 instances without
CPUSET, with non-realtime priority.
– 1 hour
Cyclictest histgram

100000Count10000

FUEGO

1000

76.463msec

100
101020,000 40,000 60,000
Latency [usec]
CPU#0
CPU#180,000

Test
scriptcyclictest

test logs

Debiansnapshot logs
PCstressng

LTSI Kernel
Minnowboard

25© Mitsubishi Electric Corporation

Result• Snapshot log in the worst case.
cyclictest‐17361 [001] .... 55024.413733: mutex_lock <‐__fdget_pos
cyclictest‐17361 [001] .... 55024.413733: vfs_write <‐SyS_write
cyclictest‐17361 [001] .... 55024.413734: rw_verify_area <‐vfs_write
cyclictest‐17361 [001] .... 55024.413735: security_file_permission <‐rw_verify_area
cyclictest‐17361 [001] .... 55024.413736: __sb_start_write <‐vfs_write
cyclictest‐17361 [001] .... 55024.413737: preempt_count_add <‐__sb_start_write
76.463msec
cyclictest‐17361 [001] ...1 55024.413738: preempt_count_sub <‐__sb_start_write
cyclictest‐17361 [001] .... 55024.413739: __vfs_write <‐vfs_write
cyclictest‐17361 [001] .... 55024.413741: tracing_mark_write: hit latency
snapshot threshold (76463 > 1000)
cyclictest‐17361 [001] .... 55024.413745: __fsnotify_parent <‐vfs_write
cyclictest‐17361 [001] .... 55024.413746: fsnotify <‐vfs_write

26© Mitsubishi Electric CorporationEvaluation• We got clues to detect the factor by doing a test andtracing at the same time.
• Fuego helped repeat execution of both.
• Therefore, we can effectively
Fuego
figure out the reason of thestresscyclictest
delay with using the snapshot
Testprogramscriptlog.
test logs

snapshotsnapshot log

Test Target KernelPC

ftraceH/W27© Mitsubishi Electric Corporation• Who I am• Overview• Related Tools– Automated Test Framework / Fuego– Real-time latency tool / cyclictest– Tracing kernel feature / ftrace• Issue• Approach– Our Use Cases• Conclusion and Future work28© Mitsubishi Electric Corporation

CONCLUSION• Summary– It is important to ensure adequate performance before
releasing products.
– It is necessary to repeat tests for reproducing the rare casewhich does not meet real-time performance requirements.
– Fuego is useful to us for not only measuring but also tracing atthe same time.

• Future Works– Discussion with Fuego community for adding our test script
which I have shown.
29© Mitsubishi Electric Corporation

Resources• Test Framework: FUEGO– http://fuegotest.org/– https://elinux.org/Fuego• LTSI Project– https://ltsi.linuxfoundation.org/• AGL Test framework: AGL-JTA– https://wiki.automotivelinux.org/agl-jta• rt-tests– https://wiki.linuxfoundation.org/realtime/documentation/howto/t
ools/rt-tests• stress-ng– http://kernel.ubuntu.com/~cking/stress-ng/
30© Mitsubishi Electric Corporation

APPENDIX

33© Mitsubishi Electric Corporation

Use Case: Cyclictest Options• 1 msec latency, 10 msec interval, in 1 hour.
– $ cyclictest• --histogram=10000 --interval=10000 --duration=3600s• --smp --quiet --mlockall --priority=50• --snapshot=1000 (instead of --breaktime)34© Mitsubishi Electric Corporation

Use Case: Test Script• Test script runs on target.
– 1: chrt itself– 2: run stress-ng program as non-rt process– 3: maximize /proc/sys/kernel/sched_rt_runtime_us– 4: setup ftrace configurations• echo 0 > $ftracedir/tracing_on• echo 0 > $ftracedir/snapshot• echo function > $ftracedir/current_tracer• echo 1 > $ftracedir/tracing_on– 5: run cyclictest– 6: save the log and a snapshot data (if recorded) to Fuego– 7: normalize /proc/sys/kernel/sched_rt_runtime_us
35© Mitsubishi Electric Corporation