Project: OpenEBS (Runtime / Cloud Native Storage)
File: systemtap-and-go.pdf

Dynamic 
Instrumentation
@JeffryMolanus @openebsGolang Meetup Bangalore XXIV
15 July 2017Tracing
•To record information about a programs execution
•Useful for understanding code, in particular a very large code base
•Used during debugging, statistics, so on and so forth
•Dynamic tracing is the ability to ad -hoc add or remove certain instrumentation without 
making changes to the code that is subject to tracing or restarting the program or system
•In general, tracing should not effect the stability of the program that is being traced in 
production, during development its less of importance
•When no tracing is enabled there should be no overhead; when enabled the overhead 
depends on what is traced and how 
•User land tracing requires abilities in kernel (which is the focus of this talk)
•user space tracing has a little more overhead due to the induced context switchTracers on other platforms
•Illumos/Solaris and FreeBSD
•Dtrace, very powerful and production safe used for many years
•Compressed Type Format (CTF) data is available in binaries and libraries, no 
need for debug symbols to work with the types
•Solaris uses the same CTF data for type information for debugging
•Event Tracing for Windows (EWT)
•Linux
•Requires debug symbols to be downloaded depending on what you trace and how 
specific you want to trace
•With DWARF data more can be done then with plain CTF howeverBasic architecture of tracing
•There are generally, two parts of tracing in Linux
•Frontend tools to work/consume with/the in kernel tracing 
facilities
•We will look briefly in ftrace, systemtap and BCC 
•Backend subsystems
•Kernel code that executes what ever code you want to be 
executed on entering the probes function or address
•kprobes, probes, tracepoints, sysdigftrace
•Tracepoints ; static probes defined in the kernel that can be enabled at 
run time
•ABI is kept stable by kernel 
•static implies you have to know what you want to trace while 
developing the code
•Makes use of sysfs interface to interact with it 
•Several wrappers exist to make things a little easier
•tracecmd and kernelshark (UI)
•Also check the excellent stuff from Brendan GreggAdding a tracepoint
Trace points in sysfs
kernelshark
kprobes
•kprobes is defined in multiple sub categories
•jprobes : trace function entry (optimised for function entry, copy stack)
•kretprobes : trace function return 
•kprobes : trace at any arbitrary instruction in the kernel
•To use it one has to write a kernel module which needs to be loaded at run 
time
•this is not guaranteed to be safe
•A kprobe replaces the traced instruction with a break point instruction
•On entry, the pre_handler is called after instrumenting, the post handlerkprobes
Kprobe example
Kprobe example
jprobes
•Note: function 
prototype needs to 
match the actual 
syscall
utrace/uprobes
•Roughly the the same as the kprobe facility in the kernel but focused 
on user land tracing
•current ptrace() in linux is implemented using the utrace frame work
•tools like strace and GDB use ptrace()
•Allows for more sophisticated tooling, one of which is uprobes
•Trace points are placed on the an inode:offset tuple
•All binaries that map that address will have a SW breakpoint 
injected at that addressftrace & user space
•The same ftrace interface is available for working with uprobes
•Behind the scene the kernel does the right thing (e.g use kprobe, 
tracepoints, or uprobes)
•The same sysfs interface is used, general work flow:
•Find address to place the probe on
•Enable probing
•Disable probing
•View results (flight recorder)eBPF
•Pretty sure everyone here has used 
BPF likely with out knowing
•tcpdump uses BPF
•eBPF is enhanced BPF
•sandboxed byte code executed by 
kernel which is safe and user 
defined
•attach eBPF to kprobes and 
uprobes
•certain restrictions in abilities
BCC
•BPF Compiler Collection, 
compiles  code for the in kernel 
VM to be executed
•Several high level wrappers for 
Python, lua and GO
•Code is still written in C 
however
Recap
•Several back -end tracing capabilities in the kernel
•Tracepoints, kprobes, jprobes, kretprobes and uprobes
•eBPF allows attachment to kprobe, uprobes and tracepoints for 
safe execution 
•Linux tracing world can use better generic frontends for adhoc 
tracing
•Best today are perf and systemtap (IMHO)
•Who wants to write C when you want to print a member of a 
complex struct? (ply)Systemtap
•High level scripting language to work with the aforementioned tracing 
capabilities of Linux
•Flexible as it allows for writing scripts that can trace specific lines 
within a file (debug symbols)
•Next to tracing, it can also make changes to running programs when 
run in “guru mode”
•Resulting scripts from systemtap are kernel modules that are loaded 
in to the kernel (kprobe and uprobes)
•Adding a eBPF target is in the works as currently, systemtap may 
result in unremovable modules or sudden death of traced processesstp files
•Example script oneliner: 
•stap -e ‘probe syscall.open { printf(“exec %s, file%s, execname(), 
filename) }’
•stap -L ‘syscall.open'
•syscall.open: __nr:long name:string filename:string flags:long 
flags_str:string mode:long argstr:string
•List user space functions in process “trace”
•stap -L ‘process(“./trace").function("*")'
•.call and .return probes for each function List probes
Tracing line numbers
•What's the value of ret after 
line 35?
•Could be done by tracing ret 
values, but that is not the 
purpose of this exercise
•gcc -g -O0 
•full debug info
Tracing line number
•.statement(“main@code/talk/trace.c:36”) { … }
Understanding code flow
Understanding code flow
Downstack
•All functions 
being called by 
a function
Tracing go
Cant trace return values
Calling convention
•AMD64 calling conventions
•RDI, RSI, RDX, RCX, R8 and R9
•Go is based on PLAN9 which uses a different approach  therefore tracing does not work as 
well as one would like it to be (yet)
•This also goes for debuggers
•Perhaps Go will start using the X86_64 ABI as it moves forward or all tools and debuggers 
will add specific PLAN9 support
•https://go -review.googlesource.com/#/c/28832/ (ABI change?)
•GO bindings to the BCC tool chain
•Allows for creating eBPF tracing tools written in go
•but still requires writing the actual trace logic in CSummary
•Dynamic tracing is an invaluable tool for 
understanding code flow
•To verify hypotheses around software bugs or 
understanding
•Ability to make changes to code on the fly with out 
recompiling (guru mode)
•Under constant development most noticeable the 
eBPF/BCC work