# Linux Applications Debugging Techniques/Print Version

Current, editable version of this book is available in Wikibooks, collection of open-content textbooks at URL: http://en.wikibooks.org/wiki/Linux_Applications_Debugging_Techniques
Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".

## Preamble

A collection of notes about debugging applications under Linux and related topics. Probably useful for debugging dogs. Partly applicable to other Unices. Work in progress, code is now on github.

## Authors

Aurelian Melinte

## Table of Contents

* The debugger
* The dynamic linker
* Core files
* The call stack
* The interposition library
* Memory issues
** Leaks
** Heap corruption
** Stack corruption
* Signals
* Deadlocks
* Race conditions
* Resource leaks
* Transactional memory
* The compiler
* Aiming for and measuring performance
* Hardware emulators
* Building the toolkit
* Appendices
* References and further reading
Please add {{alphabetical}} only to book title pages.

# The debugger
### Preparations

Someday some hard to reproduce issue will be found on a production machine. Typically, such a machine is difficult to access, has no development environment and nothing can be installed on it. At most it will have gdb, but very likely not. Typically also, the heaviest user will be the one to find the issue and typically still, the heaviest user is the one being the bigger -- money-wise. And that issue has to be root cause diagnosed and fixed.
Unless the application has been prepared for this forensic gathering moment, not much can be done to diagnose where the issue is. Thus, preparations should start with compilation:
* Have a "symbol server" and make sure it is reacheable. Compile on the symbol server.
* Compile the release with debugging symbols. Strip them if you do not want to ship them but keep them.
* Ship gdbserver with the application for remote debugging.
These preparations will allow one to:
* Debug the application running on any machine, including machines where there is no gdb installed.
* Debug from any machine that has network visibility to the symbol server.
Also, think beforehand how would you debug on the machine:
* Embed a breakpoint in the code, at hard of reach places of interest, then
** Start the application
** Attach to it with the debugger
** Wait until the breakpoint is hit

### The "symbol server"

One way to easily reach the right code from within the debugger is to build the binaries within an auto-mounted folder, each build in its own sub-folder. The same auto-mount share should be accessible from the machine you are debugging on.

##### Debian

* Install autofs
* In /etc/auto.master uncomment the line:
* In /etc/exports export the folder:
* As root: restart autofs & nfs and export the build share:

##### Redhat

* Export the folder: edit /etc/exports
* As root (RedHat): service autofs start
Finally, build the binaries within the automounted directory on the build machine (here the build machine is bear):
Notice the filename path that is resolved with the symbol information:
If the symbols have been stripped from the binaries, point gdb to the folders where the symbols are with the debug-file-directory directive.

##### References

* Debugging Information in Separate Files
* RedHat debuginfod symbol server gdb 10.1
* See also debuginfod

### Source Code

To point the debugger to the source files:

### Remote debugging

* On the machine where the application runs (appmachine):
** If gdbserver is not present, copy it over.
** Start the application.
** Start gdbserver: gdbserver gdbmachine:2345 --attach program
* On gdbmachine:
** At the gdb prompt, enter: target remote appmachine:2345
Sometimes you may have to tunnel over ssh:
* On gdbmachine:
** ssh -L 5432:appmachine:2345 user@appmachine
** At the gdb prompt: target remote localhost:5432

##### References

* GDB Tunneling

### Attaching to a process

Find out the PID of the process, then:

### Debugging programs that spawn multiple children

* set detach-on-fork off
* see "all-stop" vs "non-stop" modes in the GDB documentation and their related settings

### Embedding breakpoints in the source

On x86 platforms:
Or a more elaborate one:
This will break into the debugger on hard to reach conditions:

##### References

* http://mainisusuallyafunction.blogspot.com/2012/01/embedding-gdb-breakpoints-in-c-source.html

### Data breakpoints (watchpoints)

Watchpoints can be implemented either in software either in hardware if the CPU supports it. Typically on an Intel processor there are eight debug registers out of which only four can be used for hardware breakpoints and this limits the number of watchpoints system wide.
Install a condition on a hardware watchpoint so that only accesses are stopped that increase the value of the variable:

### Breakpoints conditional on caller

This requires gdb 7.9 or later, configured with python support:

### The text user interface

GDB features a text user interface for code, disassembler and registers. For instance:
* Ctrl-x 1 will show the code pane
* Ctrl-x a will hide the TUI panes
None of the GUI interfaces to gdb (Qt Creator stands out for being intuitive and easy to use) can offer access to all of the gdb functionality.
curses gdb offers an improved TUI. A comprehensive list of debugger GUIs is available here.

##### Reverse debugging

As an example, reverse debugging is a functionality no GUI offers access to:
See also
* rr & gdbgui

##### Register watch

You can watch registers. Note this will force the debugger to single step the debugged program and it will run very slowly:

##### References

* http://sourceware.org/gdb/onlinedocs/gdb/TUI.html
* http://sourceware.org/gdb/news/reversible.html

### .gdbinit

As a note, in upcoming gdb releases, .gdbinit will be replaced by gdb-gdb.gdb:

### C++ support
##### Canned gdb macros

* gdb STL support
* STL macros (and more)

##### Mangling

gdb might need a bit of guidance with C++11 binaries:

##### Templates

One can eventually use templight to debug and profile templates.

##### Getting the subclass from the base class pointer
### References

* gdb & lib debugging support
* https://sourceware.org/gdb/onlinedocs/gdb/JIT-Interface.html
* https://sourceware.org/gdb/current/onlinedocs/gdb/Unwinding-Frames-in-Python.html
* drgn debugger
* Python-based frontend

# The dynamic linker
### Dependencies

* ldd -d -r /path/to/binary
* A script to visualize libraries and their dependencies.
* Binaries Analysis Tools
* tldd

### Resolved symbols

To find out which dynamic library is a symbol coming from:

##### References

* man 8 ld.so
* libc symbols visibility & linking

### Notes

* It is run under whatever credentials would be accorded to the application program; thus, for example, if a set-user-ID-root program is being executed, the dynamic linker will run with an effective user ID of root.

### Varia

* A look at dynamic linking

# Core files

A core dump is a snapshot of the memory of the program, processor registers including program counter and stack pointer and other OS and memory management information, taken at a certain point in time. As such, they are invaluable for capturing the state of rare occurring races and abnormal conditions.
What is more, such rarities will be found usually on heavily used production or QA machines where gdb is not available, nor is access easy to the machine. Worse, the heaviest users are usually the biggest clients (moneywise...). As such, it is important to get as much forensic data as available, and plan for it.
One can force a core dump from within the program or from outside at chosen moments. What a core cannot tell is how the application ended up in that state: the core is no replacement for a good log. Verbose logs and core files go hand in glove.

### Prerequisites

For a process to be able to dump core, a few prerequisites have to be met:
* the set core size limit should permit it (see the man page for ulimit). E.g.: ulimit -c unlimited. It can also be set from within the program.
* the process to dump core should have write permissions to the folder where the core is to be dumped to (usually the current working directory of the process)

### Where is my core?

Usually the core is dumped in the current working directory of the process. But the OS can be configured otherwise:

### Dumping core from outside the program

One possibility is with gdb, if available. This will let the program running:
Another possibility is to signal the process. This will terminate it, assuming the signal is not caught by a custom signal handler:

### Dumping core from within the program

Again, there are two possibilities: dump core and terminate the program or dump and continue:
Note: use dump_core_and_continue with care: in a multi-threaded program, the forked child will have only a clone of the parent thread that called fork [Butenhof Ch5; re: threads & fork]. This has number of implications, in particular with respect to mutexes, but the particular point here is that the core that the child will dump will contain information only for one thread. If you need to dump a core with all threads without aborting the process, try to use the google core dumper library, even if it has not been maintained for years.

### Shared libraries

To obtain a good call stack, it is important that the gdb loads the same libraries that were loaded by the program that generated the core dump. If the machine we are analyzing the core has different libraries (or has them in different places) from the machine the core was dumped, then copy over the libraries to the analyzing machine, in a way that mirrors the dump machine. For instance:
At the gdb prompt:

### Source Code

To point the debugger to the source files:

### analyze-cores

Here is a script that will generate a basic report per core file. Useful the days when cores are raining on you:
An alternative worth exploring is btparser.

### Canned user-defined commands

Same reporting functionality can be canned for gdb:

### analyze-pid

A script that will generate a basic report and a core file for a running process:

### Thread Local Storage

TLS data is rather difficult to access with gdb in the core files, and __tls_get_addr cannot be called.

##### References

* __thread variables

### Links

* crash tool
* Kernel dumps

# The call stack
### The API

Sometimes we need the call stack at a certain point in the program. These are the API functions to get basic stack information:
Notes:
* C++ symbols are still mangled. Use abi::__cxa_demangle or something similar.
* Some of the these functions do allocate memory - either temporarily either explicitly - and this might be a problem if the program is unstable already.
* Some of the these functions do acquire locks (e.g. dladdr).
* backtrace* are expensive calls on some platforms (x86_64 for instance), the deeper the call stack is the more expensive the call is.
* dladdr will fail to rsolve any symbol that is not exported. Hence, to get the most out of it:
** Link with -rdynamic and -ldl
To extract more information (file, line number), use libbfd. Possible alternatives: libbacktrace; http://libcsdbg.sourceforge.net/

### A C++ wrapper

With this simple class, getting the stack is one line away:
This class can be used with assertions, logging or exceptions.
The full code can be found at the LPT site and it offers three symbol resolver varieties. This code will require a C++11 compiler.
* A "basic" address-only resolver, without memory side-effects
* A libc resolver
* A libbfd resolver
A boost-based version of the call stack utilities collection can be found at https://github.com/melintea/Boost-Call_stack.
A stack resolved with the libc resolver, embedded in an exception, with debug-compiled code would look like:
Note how dladdr fails to resolve some of the frames. Whereas a stack resolved with the bfd resolver would show file and line numbers and would resolve all frames:

##### Caveats

* Do NOT use in asynchronous interrupts such as signal handlers.
* It might or it might not work when the program is unstable - prefer a core dump if possible. Same for memory issues handlers .
* Performance hit varies greatly with platform and compiler version.

### From within gdb

A canned command to resolve a stack address from within gdb:

### Links

* Calling conventions

# The interposition library

The dynamic liker allows for interception of any function call an application makes to any shared library it uses. As such, interposition is a powerful technique allowing to tune performance, collect runtime statistics, or debug the application without having to instrument the code of that application.
As an example, we can use an interposition library to trace calls, with arguments' values and return codes.

### Call tracing

Note that part of code below is 32-bit x86 and gcc 4.1/4.2 specific.

##### Code instrumentation

In the library, we want to address the following points:
* when a function/method is entered and exited.
* what were the call arguments when the function is entered.
* what was the return code when the function is exited.
* optionally, where was the function called from.
The first one is easy: if requested, the compiler will instrument functions and methods so that when a function/method is entered, a call to an instrumentation function is made and when the function is exited, a similar instrumentation call is made:
This is achieved by compiling the code with the -finstrument-functions flag. The above two functions can be used for instance to collect data for coverage; or for profiling. We will use them to print a trace of function calls. Furthermore, we can isolate these two functions and the supporting code in an interposition library of our own. This library can be loaded when and if needed, thus leaving the application code basically unchanged.
Now when the function is entered we can get the arguments of the call:
And when the function is exited, we get the return value:
Since these two instrumentation functions are aware of addresses and we actually want the trace to be readable by humans, we need also a way to resolve symbol addresses to symbol names: this is what libtrace_resolve does.

##### Binutils and libbfd

First, we have to have the symbols information handy. To achieve this, we compile our application with the -g flag. Then, we can map addresses to symbol names and this would normally require writing some code knowledgeable of the ELF format.
Luckily, the there is the binutils package which comes with a library that does just that: libbfd; and with a tool: addr2line. addr2line is a good example on how to use libbfd and I have simply used it to wrap around libbfd. The result is the libtrace_resolve function.
Since the instrumentation functions are isolated in a stand-alone module, we tell this module the name of the instrumented executable through an environment variable (CTRACE_PROGRAM) that we set before running the program. This is needed to properly init libbfd to search for symbols.

##### Stack layout

To address the first point the work has been architecture-agnostic (actually libbfd is aware of the architecture, but things are hidden behind its API). However, to retrieve function arguments and return values we have to look at the stack, write a bit of architecture-specific code and exploit some gcc quirks. Again, the compilers I have used were gcc 4.1 and 4.2; later or previous versions might work differently. In short:
* x86 dictates that stack grows down.
* GCC dictates how the stack is used - a "typical" stack is depicted below.
* each function has a stack frame marked by the ebp (base pointer) and esp (stack pointer) registers.
* normally, we expect the eax register to contain the return code
In an ideal world, the code the compiler generates would make sure that upon instrumenting the exit of a function: the return value is set, then CPU registers pushed on the stack (to ensure the instrumentation function does not affects them), then call the instrumentation function and then pop the registers. This sequence of code would ensure we always get access to the
Also, in practice, many of gcc's flags affect the stack layout and registers usage. The most obvious ones are:
* -fomit-frame-pointer. This flag affects the stack offset where the arguments are to be found.
* The optimization flags: -Ox; each of these flags aggregates a number of optimizations. These flags did not affected the stack, and, quite amazingly, arguments were always passed to functions through the stack, regardless of the optimization level. One would have expected that some arguments would pe passed through registers - in which case getting these arguments would have proven to be difficult to impossible. However, these flags did complicated recovering the return code. However, on some architectures, these flags will suck in the -fomit-frame-pointer optimization.
* In any case, be wary: other flags you use to compile your application may reserve surprises.

##### Function arguments

In my tests with the compilers, all arguments were invariably passed through the stack. Hence this is trivial business, affected to a small extent by the -fomit-frame-pointer flag - this flag will change the offset at which arguments start.
How many arguments a function has, how many arguments are on the stack? One way to infer somehow the number of arguments is based on its signature (for C++, beware of the 'this' hidden argument) and this is the technique used in __cyg_profile_func_enter.
Once we know the offset where arguments start on the stack and how many of them there are, we just walk the stack to retrieve their values:

##### Function return values

Obtaining the return value proved to be possible only when using the -O0 flag.
Let's look what happens when this method
is instrumented with -O0:
Note how the return code is moved into the ebx register - a bit unexpected, since, traditionally, the eax register is used for return codes - and then the instrumentation function is called. Good to retrieve the return value but to avoid that the ebx register gets clobbered in the instrumentation function, we save it upon entering the function and we restore it upon exit.
When the compilation is done with some degree of optimization (-O1...3; shown here is -O2), the code changes:
Note how the instrumentation function gets called first and only then the eax register is set with the return value. Thus, if we absolutely want the return code, we are forced to compile with -O0.

##### Sample output

Finally, below are the results. At at shell prompt type:
Note how libbfd fails to resolve some addresses when the function gets inlined.

##### Resources

* Code
* Overview of GCC on x86 platforms
* The Intel stack
* etrace: a tool to generate the run-time function call tree with gcc
* ELF details

# Memory issues

Linux Applications Debugging Techniques/Memory issues

# Leaks
### What to look for

Memory can be allocated through many API calls:
* malloc
* calloc
* realloc
* memalign
* posix_memalign
* valloc
* mmap
* brk / sbrk
To return memory to the OS:
* free
* munmap

### Valgrind

Valgrind should be the first stop for any memory related issue. However:
* it slows down the program by at least one order of magnitude. In particular server C++ programs can be slowed down 15-20 times.
* from experience, some versions might have difficulties tracking mmap allocated memory.
* on amd64, the vex disassembler is likely to fail (v3.7) rather sooner than later so valgrind is of no use for any medium or intensive usage.
* you need to write suppressions to filter down the issues reported.
If these are real drawbacks, lighter solutions are available.

### mudflap

* http://gcc.gnu.org/wiki/Mudflap_Pointer_Debugging
Note: Mudflap has been removed from GCC 4.9 and later

### memleax

* memleax
Memleax debugs memory leak of a running process by attaching it, without recompiling program or restarting target process!!!
It's very convenient and suitable for production environment.
It works on GNU/Linux-x86_64 and FreeBSD-amd64.

### gcc/clang flags

See https://lemire.me/blog/2016/04/20/no-more-leaks-with-sanitize-flags-in-gcc-and-clang/

### bcc

See https://github.com/iovisor/bcc/blob/master/tools/memleak.py in https://github.com/iovisor/bcc.

### libmemleak

* libmemleak
Libmemleak finds memory leaks that cause a process to slowly increase the amount of memory it uses, also without the need to recompile the program as it can be LD_PRELOAD-ed when starting the program under test. Unlike valgrind it hardly slows down the process under test. Leaks are reported on a per backtrace basis. This is sometimes very important because often the caller, deeper into the backtrace, is responsible for the leak (not freeing it) while the actual place where the allocation happens wouldn't tell you anything.
It's been tested on GNU/Linux-x86_64.

##### Sample report
### DIY:libmtrace

The GNU C library comes with a built-in functionality to help detecting memory issues: mtrace. One of the shortcomings: it does not log the call stacks of the memory allocations it tracks. We can build an interposition library to augment mtrace.

##### The basics

The malloc implementation in the GNU C library provides a simple but powerful way to detect memory leaks and obtain some information to find the location where the leaks occurs, and this, with rather minimal speed penalties for the program.
Getting started is as simple as it can be:
* #include mcheck.h in your code.
* Call mtrace to install hooks for malloc, realloc, free and memalign. From this point on, all memory manipulations by these functions will be tracked. Note there are other untracked ways to allocate memory.
* Call muntrace to uninstall the tracking handlers.
* Recompile.
Under the hood, mtrace installs the four hooks mentioned above. The information collected through the hooks is written to a log file.
Note: there are other ways to allocate memory, notably mmap. These allocations will not be reported, unfortunately.
Next:
* Set the MALLOC_TRACE environment variable with the memory log name.
* Run the program.
* Run the memory log through mtrace.
One of the leaks (the malloc call) was precisely traced to the exact file and line number. However, the other leaks at line 25, while detected, we do not know where they occur. The two memory allocations for the std::string are buried deep inside the C++ library. We would need the stack trace for these two leaks to pinpoint the place in our code.
We can use GDB (or the trace_call macro) to get the allocations' stacks:

##### A couple of improvements

It would be good to have mtrace itself dump the allocation stack and dispense with gdb. The modified mtrace would have to supplement the information with:
* The stack trace for each allocation.
* Demangled function names.
* File name and line number.
Additionally, we can put the code in a library, to free the program from being instrumented with mtrace. In this case, all we have to do is interpose the library when we want to trace memory allocations (and pay the performance price).
Note: getting all this information at runtime, particularly in a human-readable form will have a performance impact on the program, unlike the plain vanilla mtrace supplied with glibc.

##### The stack trace

A good start would be to use another API function: backtrace_symbols_fd. This would print the stack directly to the log file. Perfect for a C program but C++ symbols are mangled:
For C++ we would have to get the stack (backtrace_symbols), resolve each address (dladdr) and demangle each symbol name (abi::__cxa_demangle).

##### Caveats

* Memory allocation is one of these basic operation everything else builds on. One needs to allocate memory to load libraries and executables; needs to allocate memory to track memory allocations; and we hook onto it very early in the life of a process: the first pre-loaded library is the memory tracking library. Thus, any API call we make inside this interposition library can reserve surprises, especially in multi-threaded environments.
* The API functions we use to trace the stack can allocate memory. These allocations are also going through the hooks we installed. As we trace the new allocation, the hooks are activated again and another allocation is made as we trace this new allocation. We will run out of stack in this infinite loop. We break out of this pitfall by using a per-thread flag.
* The API functions we use to trace the stack can deadlock. Suppose we would use a lock while in our trace. We lock the trace lock and we call dladdr which in turn tries to lock a dynamic linker internal lock. If on another thread dlopen is called while we trace, dlopen locks the same linker lock, then allocates memory: this will trigger the memory hooks and we now have the dlopen thread wait on the trace lock with the linker lock taken. Deadlock.
* On some platforms (gcc 4.7.2 amd64) TLS calls would trip the memalign hook. This could result in an infinite recursion if the memalign hook in its turn, accesses a TLS variable.

##### What we got

Let's try again with our new library:
Apparently, not much of an improvement: the summary still does not get us back to line 25 in main.cpp. However, if we search for address 8bf89b0 in the trace log, we find this:
This is good, but having file and line information would be better.

##### File and line

Here we have a few possibilities:
* Run the address (e.g. 0x40178450 above) through the addr2line tool. If the address is in a shared object that the program loaded, it might not resolve properly.
* If we have a core dump of the program, we can ask gdb to resolve the address. Or we can attach to the running program and resolve the address.
* Use the API described here. The downside is that it takes a quite heavy toll on the performance of the program.

##### Resources

* The GNU C library manual
* Using libbfd
* Linux Programming Toolkit

### DIY:libmemleak

Building on libmtrace, we can go one step further and have an interposition library track the memory allocations made by the program. The library generates a report on demand, much like Valgrind does.
Libmemleak is significantly faster than valgrind but also has limited functionality (only leak detection).

##### Operation

libmemtrace has two drawbacks:
* The log file will quickly reach gigs
* You are left grepping the log to figure out what leaks when
A better solution would be to have an interposition library to collect memory operations information and to generate a report on-demand.
For mmap/munmap we have no choice but hook these directly. Thus, an call from within the application would first hit the hooks in libmemleak, then go to libc. For mallocrealloc/memalign/free we have two options:
* Use mtrace/muntrace as before, to install hooks that will be called from within libc. Thus, a malloc call would first go through libc which will then call the hooks in libmemtrace. This leave us at the mercy of libc.
* The second solution is to hook these like m(un)map.
The second solution also frees mtrace/muntrace for on-demand report generation:
* A first call to mtrace will kick in data collection.
* Subsequent calls to mtrace will generate reports.
* muntrace will stop data collection and will generate a final report.
* MALLOC_TRACE is not needed.
The application can then sprinkle its code with mtrace calls at strategic places to avoid reporting much noise. These calls will do nothing in a normal operation as long as MALLOC_TRACE is not set. Or, the application can be completely ignorant of the ongoing data collection (no mtrace calls within the application code) and libmemleak can start collecting as early as being loaded and generate one report upon being unloaded.
To control the libmemleak functionality, an environment variable - MEMLEAK_CONFIG - has to be set before loading the library:
* mtraceinit will instruct the library to start collecting data upon being loaded. Default is off and the application has to be instrumented with m(un)trace calls.
Thus, all the hooks have to do is to call into the reporting:

##### Sample report
### mallinfo

The mallinfo API is rumored to be deprecated. But, if available, it is very useful:

##### References

* mallinfo
* mallinfo deprecated

### /proc

Coarse grained information can be obtained from /proc:

##### References

* Memory usage script

### Various tools

* gdb-heap extension
* Dr. Memory
* Pin tool
* heaptrack
* Data-type profiling for perf
* Type-preserving heap profiler
* See https://stackoverflow.com/questions/18455698/lightweight-memory-leak-debugging-on-linux for a list of tools. Out of which, memleax deserves a special mention for its out-of-process approach.

# Heap corruption
### Electric Fence

Electric Fence is still the reference for dealing with heap corruption, even if not maintined for a while. RedHat ships a version that can be used as an interposition library.
Drawback: might not work with code that uses mmap to allocate memory.

### Duma

Duma is a fork of Electric Fence.

### glibc builtin

man (3) malloc: Recent versions of Linux libc (later than 5.4.23) and GNU libc (2.x) include a malloc implementation which is tunable via environment variables. When MALLOC_CHECK_ is set, a special (less efficient) implementation is used which is designed to be tolerant against simple errors, such as double calls of free with the same argument, or overruns of a single byte (off-by-one bugs). Not all such errors can be protected against, however, and memory leaks can result. If MALLOC_CHECK_ is set to 0, any detected heap corruption is silently ignored and an error message is not generated; if set to 1, the error message is printed on stderr, but the program is not aborted; if set to 2, abort is called immediately, but the error message is not generated; if set to 3, the error message is printed on stderr and program is aborted. This can be useful because otherwise a crash may happen much later, and the true cause for the problem is then very hard to track down.

### C++ stdlib builtin

Compile with -D_GLIBCXX_DEBUG - it turns on debugging checks in the C++ standard library.

### AddressSanitizer

A fast memory error detector via -fsanitize=address , with a recent gcc (since 4.8) or clang (since 3.1). Memory access instructions will be instrumented to detect heap-, stack-, and global-buffer overflow as well as use-after-free bugs. To get nicer stacktraces, use -fno-omit-frame-pointer. The AddressSanitizer is available on IA-32/x86-64/x32/PowerPC/PowerPC64 GNU/Linux and on x86-64 Darwin.
* http://clang.llvm.org/docs/AddressSanitizer.html
* https://code.google.com/p/address-sanitizer/

### ubsan

Starting with gcc 4.9 and later, you can use the ubsan sanitizer for bounds checking.

### gdb heisenbugs

Memory-related bugs that don't happen when using GDB but do without GDB are often caused by GDB disabling ASLR. This is usually useful for reproducibility of debugging sessions, but may indeed make address space layout too predictable and fail to reproduce a bug. Try using the GDB command "set disable-randomization off" (without quotes) to undo this behavior and thus start your program in a normal, randomized, environment.

### Links

* Intel support for memory issues

# Stack corruption

Stack corruption is rather hard to diagnose. Luckily, gcc 4.x can instrument the code to check for stack corruption:
* -fstack-protector Add stack protection to functions that have “alloca” or have a (signed or unsigned) char array with size > 8 (SSP_BUFFER_SIZE)
* -fstack-protector-strong To more functions, see below
* -fstack-protector-all To ALL functions
gcc will add guard variables and code to check for buffer overflows upon exiting a function. A quick example:
When run, the program will dump core:

### -fstack-protector-strong

Added by google to gcc.
Benefit - gain big performance while sacrificing little security (for scenarios using -fstack-protector-all)
Background - some times stack-protector is too-simple while stack-protector-all over-kills, for example, to build one of our core systems, we forcibly add "-fstack-protector-all" to all compile commands, which brings big performance penalty (due to extra stack guard/check insns on function prologue and epilogue) on both atom and arm. To use "-fstack-protector" is just regarded as not secure enough (only "protects" <2% functions) by the system secure team. "-fstack-protector-strong" hits the balance between "-fstack-protector" and "-fstack-protector-all".
Adds the check to a function:
* if any of its local variable’s address is taken, as part of the RHS of an assignment
* or if any of its local variable’s address is taken as part of a function argument.
* or if it has an array, regardless of array type or length
* or if it has a struct/union which contains an array, regardless of array type or length.
* or if function has register local variables
See http://gcc.gnu.org/ml/gcc-patches/2012-06/msg00974.html

##### Further reading

* http://www.drdobbs.com/cpp/240001832
* New -strong stack protector option for gcc
Starting with gcc 4.9 and later, you can use the ubsan sanitizer for bounds checking.

# Deadlocks
### Analysis

Searching for a deadlock means reconstructing the graph of dependencies between threads and resources (mutexes, semaphores, condition variables, etc.) - who owns what and who wants to acquire what. A typical deadlock would look like a loop in that graph. The task is tedious, as some of the parameters we are looking for have been optimized by the compiler into registers.
Below is an analysis of an x86_64 deadlock. On this platform, register r8 is the one containing the first argument: the address of the mutex:
Threads 3 and 4 are deadlocking over two mutexes.
Note: If gdb is unable to find the symbol pthread_mutex_t because it has not loaded the symbol table for pthreadtypes.h, you can still print the individual members of the struct as follows:
An std::mutex has a similar structure where __owner is the LWP:

### Automation

An interposition library can be built to automate deadlock analysis. A significant number of APIs have to be interposed and even then there are cases that would go unnoticed by the library. For instance, one creative way to deadlock two threads without involving any userland locking mechanism is to have each thread join the other. Thus, interposition tools have limited diagnostic functionality.
There are a number of other tools available:
* gdb-automatic-deadlock-detector - script adds new command 'blocked' to GDB. This command analyze all threads and display which threads are waiting for other threads. It also shows deadlocks between threads.
* Userspace lockdep. Incipient work.
* Locksmith. Basic.
* Valgrind Helgrind. Does not suffer from the terrible slowdown other Valgrind tools have (in particular the memory analysis ones) but does not survive long on an amd64 platform.

### pthreads

pthreads has some built-in support for certain synchronization mechanisms (e.g. PTHREAD_MUTEX_ERRORCHECK mutexes).
Also, there's a POSIX mutex construction attribute for robust mutexes (a recoverable mutex that was held by a thread whose process has died): an error return code indicates the earlier owner died; and lock acquisition implies acquired responsibility for dealing with any cleanup.

### lockdep

On lockdep-enabled kernels, pressing Alt+SysRq+d will dump information about all the locks known to the kernel.

# Race conditions
### Valgrind Helgrind

* [v 3.7] On amd64 platforms it does not survive for long because of the vex disassembler.

### Valgrind Drd

* Same.

### ThreadSanitizer (tsan)

* Usually better than Helgrind, as it does not depend on valgrind anymore.

### Relacy

* C++0x/11 synchronization modeler/unit tests tool.

### References

* gdb & lib debugging support
* Lockless Algorithms

# Resource leaks
### Zombie threads

Any thread that has terminated but has not been joined or detached will leak OS resources until the process terminates. Unfortunately, neither /proc nor gdb will show you these zombie threads, at least not on some kernels.
One way to get them is with a gdb canned command:
Another way is to use (again) an interposition library:
The output is a bit rough but it can be refined down to file and line by replacing backtrace_symbols_fd with appropriate code:

### File descriptors

As just about anything is a file (folders, sockets, pipes, etc.), just about anything can result in a file descriptor that needs to be closed. /proc can help:
The trace_call command for gdb can help with the call stack.
If gdb is not available on the machine, an interposition library hooking open, pipe, socket, etc. can be built.
Other tools that can be used:
* lsof
* fuser

### Ports

Which process is using a port? As root:
Other tools:
* fuser

### IPC

For semaphores, shared memory and message queues.
* ipcs
* ipcrm

### DYI: an interposition resource counter

libmemleak can be easily modified to keep track of whatever resources are leaking. Hook the right API (e.g. open/close).

# Transactional memory
### Notes

* If writing alters a cache line used in a transaction that transaction will be aborted.

##### References

* Lock elision in the GNU C library
* Intel extensions

# The compiler

When requested to optimize (-O2 and above), the compiler is granted a "total license" to modify it: treats undefined behavior (by the standard) as it cannot happen. Thus, the resulting code behaves differently in release-optimized mode than debug mode. Such differences in behavior cannot be found by static analyzers or code reviews.

### Signed Integer Overflow

1. This is C99 undefined behavior and the compiler assumes overflow cannot occur. Thus, any checks for overflow are discarded and the following is an infinite loop in optimized code:
2. The following:
is optimized into:
* Use -Wstrict-overflow=N with N>=3 to signal cases where code is optimized away.
* Use -fwrapv
If lucky enough to use gcc 4.9 and later, you can use the ubsan sanitizer:
* compile and link the program with -fsanitize=undefined option
* use other available flags as needed.

### Unsigned Wrap Around

The compiler assumes unsigned integers do not wrap. Keep the unsigned variables in the range [INT_MIN/2, INT_MAX/2] to avoid surprises.

### "Dead Code" Removed

The memset call could be removed because the compiler deems buf unused at that point in the code and after:
* Use #pragma optimize directives to force the code in.
* Use volatile.

### volatile Pitfalls

Code can be moved around:
could be optimized into:
* Use compiler intrinsic barriers to prevent the flag moving.
Loops could be optimized into one read call only:

### Pointers

* Use restrict
* Use -fno-delete-null-pointer-checks. Null pointer checks are deleted if placed after the first use of the pointer.

### References

* Dangerous Optimizations
* http://blog.regehr.org/archives/918
* LWN about optimizations
* LWN about optimizations (bis)
* cdecl
* Sources of the first C compiler
* gcc internals

# Aiming for and measuring performance
### gprof & -pg

To profile the application with gprof:
* Compile the code with -pg
* Link with -pg
* Run the application. This creates a file gmon.out in the current folder of the application.
* At the prompt, in the folder where gmon.out lives: gprof path-to-application

### PAPI

The Performance Application Programming Interface (PAPI) offers the programmer access to the performance counter hardware found in most major microprocessors. With a decent C++ wrapper, measuring branch mispredictions and cache misses (and much more) is literally one line of code away.
By default, these are the events that papi::counters<Print_policy> is looking for:
The counters class is parameterized with a Print_policy instructing what to do with the counters once they get out of scope.
As an example, lets look a bit at these lines:
The code just loops over an array but in the wrong order: the innermost loop iterates on the outer index. While the result is the same whether we loop over the first index first or over the last one, theorically, to preserve cache locality, the innermost loop should iterate over the innermost index. This should make a big difference for the time it takes to iterate over the array:
papi::counters is a class wrapping around PAPI functionality. It will take a snaphost of some performance counters (in our case, we are interested in cache misses and in branch mispredictions) when a counters object is instantiated and another snapshot when the object is destroyed. Then it will print out the differences.
A first measure, with non-optimized code (-O0), shows the following:
While the cache misses have indeed improved, branch mispredictions exploded. Not exactly a good tradeoff. Down in the pipeline of the processor, a comparison operation translates into a branch operation. Something is funny with the unoptimized code the compiler generated.
Typically, branch machine code is generated directly by if/else and ternary operators; and indirectly by virtual calls and by calls though pointers
Maybe the optimized code (-O2) is behaving better? Or maybe not:
This time the compiler optimized the loops out! It figured we do not really use the data in the array, so it got rid of. Completely!
Let's see how this code behaves:
Both cache misses and branch mispredictions improved by at least an order of magnitude. A run with unoptimized code will show the same order of improvement.

##### References

* Locality of reference

### OProfile

OProfile offers access to the same hardware counters as PAPI but without having to instrument the code:
* It is coarser grained than PAPI - at function level.
* Some out of the box kernels (RedHat) are not OProfile-friendly.
* You need root access.

##### References

* OP Manual
* IBM OP Intro
* System Profiling

### perf

perf is a kernel-based subsystem that provide a framework for performance analysis of the impact of the progams being run on the kernel. It covers: hardware (CPU/PMU, Performance Monitoring Unit) features; and software features (software counters, tracepoints).
perf tutorial

##### perf list

Lists the events available on a particular machine. These events vary based on the performance monitoring hardware and the software configuration of the system.
Note: Running this as root will give out an extended list of events; some of the events (tracepoints?) require root privileges.

##### perf stat <options> <cmd>

Gathers overall statistics for common performance events, including instructions executed and clock cycles consumed. There are option flags to gather statistics on events other than the default measurement events.

##### perf record

Records performance data into a file which can be later analyzed using perf report.

##### perf report

Reads the performance data from a file and analyzes the recorded data.

##### perf annotate

Reads the input file and displays an annotated version of the code. If the object file has debug symbols then the source code will be displayed alongside assembly code. If there is no debug info in the object, then annotated assembly is displayed. Broken!?

##### perf top

Performance top tool look-alike. It generates and displays a performance counter profile in realtime.

##### perf timechart
### Valgrind: cachegrind

Cachegrind simulates a machine with two level ([I1 & D1] and L2) cache and branch (mis)prediction. Useful in that it annotates the code down to line level. Can significantly differ from machine's real CPU. Will not go very far on an AMD64 CPU (vex disassembler issues). Extremely slow, typically slows down the application 12-15 times.

### DIY: libhitcount

libmemleak can be easily modified to keep track of what calls into a particular point in the code. Just insert an mtrace call in that place.

### How things scale up

* Source: https://gist.github.com/2843375
* Source: Paul E. McKenney

### Notes on Hyper Threading

* Presumed loss of performance for intensive floating point calculations (only one FPU and one ALU (two pipelines) per core).
* http://ayumilove.net/hyperthreading-faq/
* http://en.wikipedia.org/wiki/Hyper-threading

### Other tools

* bpftrace
* SchedViz
* eBPF & friends
* Intel VTune
* List of performance analysis tools

### Varia

* Microbenchmarks
* Macrobenchmarks
* Atomics, fences

# Hardware emulators
### umockdev

umockdev mocks hardware devices for creating unit tests for libraries and programs that handle Linux hardware devices. It also provides tools to record the properties and behaviour of particular devices. Useful for reproducing failure scenarios encountered in the wild.

# Building the toolkit

The code detailed herein is slowly melted into a toolkit and is available at the LPT site.

### Prerequisites
##### A C++11/C++0x compiler

If your distribution does not have such a compiler, compile one from scratch. First compile the gcc prerequsites, available here:
* cloog
* ppl
* mpc
* mpfr
* gmp
Then compile gcc itself:

###### References

* Compiling gcc

##### PAPI

This one is a straightforward ./configure && make. No need to install.

##### -fPIC libbfd

The call stack code has a dependency on libbfd. If you wish to link libbfd statically to cut short a dependency on the deployment machine - which may or may not have a dynamic libbfd - you need an libbfd.a that is -fPIC compiled. This flag is needed because the static library will be linked into a shared object. The plain libbfd.a coming with the platforms might not be an -fPIC one, in which case you have to compile your own.
No need to install it either.

### Compilation

Finally, we need to adjust the LPT Makefile:

# Appendices
### Tools

* Overview: http://www.brendangregg.com/Perf/linux_observability_tools.png
* bpftrace

### Things to watch for
##### Interrupted calls

A number of API functions return an error code if the call was interrupted by a signal. Usually this is not an error by itself and the call should be restarted. For instance:
The list of interruptible function differs from Unix-like platform to platform. For Linux see signal(7). Note that sem_wait & co. are on list.

##### Spurious wake-ups

Threads waiting on a pthreads condition variable can be waken up even if the condition hs not been met. Upon waking up, the condition should be explicitly checked and return waiting if it is not met.

### Code review checklist

* Functional impact
** Which modules are affected; which functionality is impacted.
** Which modules & functionality is consumed by the changed code.
** Which modules/functionality is using it.
** Side effects.
* Architecture
** Scalability
** Stability
** Functionality
** Robustness
* Technicalities
** Generic
*** First pass: logic. Correctness. What does it (tries to) do.
**** Side effects.
**** Why is code there?
*** Second pass: error handling & odd conditions.
**** Side effects.
**** Are returned codes checked. Can the code handle them.
**** Arguments:
***** Border/out of bounds/strange values handled.
***** Arg type: convenient?
***** Where is the data coming from? Three levels upstream.
***** Where are the outputs going? Three levels downstream.
****** What are the possible/permitted outputs? Are all handled downstream?
**** Variables: initialized?
**** (over/under)flows:
***** buffers.
***** integral types.
**** Asserted invariants & assumptions.
**** Loops: off-by-one? infinite?
**** Recursion: base case?
*** Third pass: thread safety.
*** Localization?
*** Duplicate information?
*** Math:
**** Small values processed first?
**** NaN checks/std::isnan & friends.
**** Division by zero.
** C
*** return: resource handling.
*** Use the N functions (ex. s'n'printf vs. sprintf).
*** printf & friends: do formatters match type and number of arguments?
*** Non re-entrant/non thread-safe API
*** Unsafe/race condition afflicted API ( mktemp )
*** enums:
**** all cases handled?
**** default returns/logs/throws error
*** Unspecified/undefined/implementation-defined/non-standard behavior or extensions
** C++
*** Fourth pass: exception handling:
**** In destructors.
**** Exception catching boundaries?
**** Are resources RAII guarded.
*** Fifth pass: performance.
**** C++11: perfect forwarding & temporaries.
**** How does it scale?
*** Initialized data members.
*** The big 6 (default constructor, destructor, copy/move constructors/operators).
*** Virtual destructor?
*** Resource handling: RAII.
*** Where is created & destructed? Lifetime management?
*** Thorough const & volatile.
*** Are predicates stateless.
*** Use size_t for std::string positions.
*** Why casts?
*** Do not trace under lock
*** Do not destroy objects under lock
* Platform specific
** POSIX/SUS
*** Interrupted calls.
*** Spurious wake-ups.
*** Calls known to have race conditions.

##### Tools

* Source Code Audit
* CERT C Secure Coding Standard

##### Links

* Undefined Behavior

### /proc

A pseudo-filesystem exposing information about running processes:

###### References

* Man page
* Kernel doc

##### sysstat, sar

* Site

### Other tools
##### addr2line

Given an address in an executable or an offset in a section of a relocatable object, addr2line translates it into file name and line number.

##### c++filt

A tool to demangle symbol names.

##### objdump

* Disassemble binary, with source code: objdump -C -S -r -R -l <binary>

##### Varia

* Source Code Audit

##### Links

* Bunch of tools
* dstat

### Abbreviations

* HLE: Hardware Lock Elision
* HT: Hyper Threading
* RTM: Restricted Transactional Memory
* TSX: Intel Transactional Synchronization Extension (, )

# References and further reading
### Books, articles, presentations

* Agar, Eric; Writing Reliable AIX Daemons
* Alexandrescu, Andrei; volatile: The Multithreaded Programmer's Best Friend
* Butenhof, David; Programming with POSIX Threads
* Drepper, Ulrich; What Every Programmer Should Know About Memory
* McKenney, Paul; Is Parallel Programming Hard (git, blog, other papers)
* https://www.youtube.com/watch?v=aIHAEYyoTUc&list=PLHxtyCq_WDLXryyw91lahwdtpZsmo4BGD
* https://www.youtube.com/watch?v=zULU6Hhp42w
* https://www.agner.org/optimize/

### Software

* Linux Programming Tools

### Links

* http://www.brendangregg.com/ebpf.html
* https://github.com/iovisor/bcc
* https://lwn.net/Articles/742082/
* https://jvns.ca/blog/2017/09/05/finding-out-where-packets-are-being-dropped/
* https://blog.cloudflare.com/path-mtu-discovery-in-practice/
* BPF
* bpftrace
* List of C++ resources

# GNU Free Documentation License

Version 1.3, 3 November 2008 Copyright (C) 2000, 2001, 2002, 2007, 2008 Free Software Foundation, Inc. <http://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed.

## 0. PREAMBLE

The purpose of this License is to make a manual, textbook, or other functional and useful document "free" in the sense of freedom: to assure everyone the effective freedom to copy and redistribute it, with or without modifying it, either commercially or noncommercially. Secondarily, this License preserves for the author and publisher a way to get credit for their work, while not being considered responsible for modifications made by others.
This License is a kind of "copyleft", which means that derivative works of the document must themselves be free in the same sense. It complements the GNU General Public License, which is a copyleft license designed for free software.
We have designed this License in order to use it for manuals for free software, because free software needs free documentation: a free program should come with manuals providing the same freedoms that the software does. But this License is not limited to software manuals; it can be used for any textual work, regardless of subject matter or whether it is published as a printed book. We recommend this License principally for works whose purpose is instruction or reference.

## 1. APPLICABILITY AND DEFINITIONS

This License applies to any manual or other work, in any medium, that contains a notice placed by the copyright holder saying it can be distributed under the terms of this License. Such a notice grants a world-wide, royalty-free license, unlimited in duration, to use that work under the conditions stated herein. The "Document", below, refers to any such manual or work. Any member of the public is a licensee, and is addressed as "you". You accept the license if you copy, modify or distribute the work in a way requiring permission under copyright law.
A "Modified Version" of the Document means any work containing the Document or a portion of it, either copied verbatim, or with modifications and/or translated into another language.
A "Secondary Section" is a named appendix or a front-matter section of the Document that deals exclusively with the relationship of the publishers or authors of the Document to the Document's overall subject (or to related matters) and contains nothing that could fall directly within that overall subject. (Thus, if the Document is in part a textbook of mathematics, a Secondary Section may not explain any mathematics.) The relationship could be a matter of historical connection with the subject or with related matters, or of legal, commercial, philosophical, ethical or political position regarding them.
The "Invariant Sections" are certain Secondary Sections whose titles are designated, as being those of Invariant Sections, in the notice that says that the Document is released under this License. If a section does not fit the above definition of Secondary then it is not allowed to be designated as Invariant. The Document may contain zero Invariant Sections. If the Document does not identify any Invariant Sections then there are none.
The "Cover Texts" are certain short passages of text that are listed, as Front-Cover Texts or Back-Cover Texts, in the notice that says that the Document is released under this License. A Front-Cover Text may be at most 5 words, and a Back-Cover Text may be at most 25 words.
A "Transparent" copy of the Document means a machine-readable copy, represented in a format whose specification is available to the general public, that is suitable for revising the document straightforwardly with generic text editors or (for images composed of pixels) generic paint programs or (for drawings) some widely available drawing editor, and that is suitable for input to text formatters or for automatic translation to a variety of formats suitable for input to text formatters. A copy made in an otherwise Transparent file format whose markup, or absence of markup, has been arranged to thwart or discourage subsequent modification by readers is not Transparent. An image format is not Transparent if used for any substantial amount of text. A copy that is not "Transparent" is called "Opaque".
Examples of suitable formats for Transparent copies include plain ASCII without markup, Texinfo input format, LaTeX input format, SGML or XML using a publicly available DTD, and standard-conforming simple HTML, PostScript or PDF designed for human modification. Examples of transparent image formats include PNG, XCF and JPG. Opaque formats include proprietary formats that can be read and edited only by proprietary word processors, SGML or XML for which the DTD and/or processing tools are not generally available, and the machine-generated HTML, PostScript or PDF produced by some word processors for output purposes only.
The "Title Page" means, for a printed book, the title page itself, plus such following pages as are needed to hold, legibly, the material this License requires to appear in the title page. For works in formats which do not have any title page as such, "Title Page" means the text near the most prominent appearance of the work's title, preceding the beginning of the body of the text.
The "publisher" means any person or entity that distributes copies of the Document to the public.
A section "Entitled XYZ" means a named subunit of the Document whose title either is precisely XYZ or contains XYZ in parentheses following text that translates XYZ in another language. (Here XYZ stands for a specific section name mentioned below, such as "Acknowledgements", "Dedications", "Endorsements", or "History".) To "Preserve the Title" of such a section when you modify the Document means that it remains a section "Entitled XYZ" according to this definition.
The Document may include Warranty Disclaimers next to the notice which states that this License applies to the Document. These Warranty Disclaimers are considered to be included by reference in this License, but only as regards disclaiming warranties: any other implication that these Warranty Disclaimers may have is void and has no effect on the meaning of this License.

## 2. VERBATIM COPYING

You may copy and distribute the Document in any medium, either commercially or noncommercially, provided that this License, the copyright notices, and the license notice saying this License applies to the Document are reproduced in all copies, and that you add no other conditions whatsoever to those of this License. You may not use technical measures to obstruct or control the reading or further copying of the copies you make or distribute. However, you may accept compensation in exchange for copies. If you distribute a large enough number of copies you must also follow the conditions in section 3.
You may also lend copies, under the same conditions stated above, and you may publicly display copies.

## 3. COPYING IN QUANTITY

If you publish printed copies (or copies in media that commonly have printed covers) of the Document, numbering more than 100, and the Document's license notice requires Cover Texts, you must enclose the copies in covers that carry, clearly and legibly, all these Cover Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on the back cover. Both covers must also clearly and legibly identify you as the publisher of these copies. The front cover must present the full title with all words of the title equally prominent and visible. You may add other material on the covers in addition. Copying with changes limited to the covers, as long as they preserve the title of the Document and satisfy these conditions, can be treated as verbatim copying in other respects.
If the required texts for either cover are too voluminous to fit legibly, you should put the first ones listed (as many as fit reasonably) on the actual cover, and continue the rest onto adjacent pages.
If you publish or distribute Opaque copies of the Document numbering more than 100, you must either include a machine-readable Transparent copy along with each Opaque copy, or state in or with each Opaque copy a computer-network location from which the general network-using public has access to download using public-standard network protocols a complete Transparent copy of the Document, free of added material. If you use the latter option, you must take reasonably prudent steps, when you begin distribution of Opaque copies in quantity, to ensure that this Transparent copy will remain thus accessible at the stated location until at least one year after the last time you distribute an Opaque copy (directly or through your agents or retailers) of that edition to the public.
It is requested, but not required, that you contact the authors of the Document well before redistributing any large number of copies, to give them a chance to provide you with an updated version of the Document.

## 4. MODIFICATIONS

You may copy and distribute a Modified Version of the Document under the conditions of sections 2 and 3 above, provided that you release the Modified Version under precisely this License, with the Modified Version filling the role of the Document, thus licensing distribution and modification of the Modified Version to whoever possesses a copy of it. In addition, you must do these things in the Modified Version:
* Use in the Title Page (and on the covers, if any) a title distinct from that of the Document, and from those of previous versions (which should, if there were any, be listed in the History section of the Document). You may use the same title as a previous version if the original publisher of that version gives permission.
* List on the Title Page, as authors, one or more persons or entities responsible for authorship of the modifications in the Modified Version, together with at least five of the principal authors of the Document (all of its principal authors, if it has fewer than five), unless they release you from this requirement.
* State on the Title page the name of the publisher of the Modified Version, as the publisher.
* Preserve all the copyright notices of the Document.
* Add an appropriate copyright notice for your modifications adjacent to the other copyright notices.
* Include, immediately after the copyright notices, a license notice giving the public permission to use the Modified Version under the terms of this License, in the form shown in the Addendum below.
* Preserve in that license notice the full lists of Invariant Sections and required Cover Texts given in the Document's license notice.
* Include an unaltered copy of this License.
* Preserve the section Entitled "History", Preserve its Title, and add to it an item stating at least the title, year, new authors, and publisher of the Modified Version as given on the Title Page. If there is no section Entitled "History" in the Document, create one stating the title, year, authors, and publisher of the Document as given on its Title Page, then add an item describing the Modified Version as stated in the previous sentence.
* Preserve the network location, if any, given in the Document for public access to a Transparent copy of the Document, and likewise the network locations given in the Document for previous versions it was based on. These may be placed in the "History" section. You may omit a network location for a work that was published at least four years before the Document itself, or if the original publisher of the version it refers to gives permission.
* For any section Entitled "Acknowledgements" or "Dedications", Preserve the Title of the section, and preserve in the section all the substance and tone of each of the contributor acknowledgements and/or dedications given therein.
* Preserve all the Invariant Sections of the Document, unaltered in their text and in their titles. Section numbers or the equivalent are not considered part of the section titles.
* Delete any section Entitled "Endorsements". Such a section may not be included in the Modified version.
* Do not retitle any existing section to be Entitled "Endorsements" or to conflict in title with any Invariant Section.
* Preserve any Warranty Disclaimers.
If the Modified Version includes new front-matter sections or appendices that qualify as Secondary Sections and contain no material copied from the Document, you may at your option designate some or all of these sections as invariant. To do this, add their titles to the list of Invariant Sections in the Modified Version's license notice. These titles must be distinct from any other section titles.
You may add a section Entitled "Endorsements", provided it contains nothing but endorsements of your Modified Version by various parties—for example, statements of peer review or that the text has been approved by an organization as the authoritative definition of a standard.
You may add a passage of up to five words as a Front-Cover Text, and a passage of up to 25 words as a Back-Cover Text, to the end of the list of Cover Texts in the Modified Version. Only one passage of Front-Cover Text and one of Back-Cover Text may be added by (or through arrangements made by) any one entity. If the Document already includes a cover text for the same cover, previously added by you or by arrangement made by the same entity you are acting on behalf of, you may not add another; but you may replace the old one, on explicit permission from the previous publisher that added the old one.
The author(s) and publisher(s) of the Document do not by this License give permission to use their names for publicity for or to assert or imply endorsement of any Modified Version.

## 5. COMBINING DOCUMENTS

You may combine the Document with other documents released under this License, under the terms defined in section 4 above for modified versions, provided that you include in the combination all of the Invariant Sections of all of the original documents, unmodified, and list them all as Invariant Sections of your combined work in its license notice, and that you preserve all their Warranty Disclaimers.
The combined work need only contain one copy of this License, and multiple identical Invariant Sections may be replaced with a single copy. If there are multiple Invariant Sections with the same name but different contents, make the title of each such section unique by adding at the end of it, in parentheses, the name of the original author or publisher of that section if known, or else a unique number. Make the same adjustment to the section titles in the list of Invariant Sections in the license notice of the combined work.
In the combination, you must combine any sections Entitled "History" in the various original documents, forming one section Entitled "History"; likewise combine any sections Entitled "Acknowledgements", and any sections Entitled "Dedications". You must delete all sections Entitled "Endorsements".

## 6. COLLECTIONS OF DOCUMENTS

You may make a collection consisting of the Document and other documents released under this License, and replace the individual copies of this License in the various documents with a single copy that is included in the collection, provided that you follow the rules of this License for verbatim copying of each of the documents in all other respects.
You may extract a single document from such a collection, and distribute it individually under this License, provided you insert a copy of this License into the extracted document, and follow this License in all other respects regarding verbatim copying of that document.

## 7. AGGREGATION WITH INDEPENDENT WORKS

A compilation of the Document or its derivatives with other separate and independent documents or works, in or on a volume of a storage or distribution medium, is called an "aggregate" if the copyright resulting from the compilation is not used to limit the legal rights of the compilation's users beyond what the individual works permit. When the Document is included in an aggregate, this License does not apply to the other works in the aggregate which are not themselves derivative works of the Document.
If the Cover Text requirement of section 3 is applicable to these copies of the Document, then if the Document is less than one half of the entire aggregate, the Document's Cover Texts may be placed on covers that bracket the Document within the aggregate, or the electronic equivalent of covers if the Document is in electronic form. Otherwise they must appear on printed covers that bracket the whole aggregate.

## 8. TRANSLATION

Translation is considered a kind of modification, so you may distribute translations of the Document under the terms of section 4. Replacing Invariant Sections with translations requires special permission from their copyright holders, but you may include translations of some or all Invariant Sections in addition to the original versions of these Invariant Sections. You may include a translation of this License, and all the license notices in the Document, and any Warranty Disclaimers, provided that you also include the original English version of this License and the original versions of those notices and disclaimers. In case of a disagreement between the translation and the original version of this License or a notice or disclaimer, the original version will prevail.
If a section in the Document is Entitled "Acknowledgements", "Dedications", or "History", the requirement (section 4) to Preserve its Title (section 1) will typically require changing the actual title.

## 9. TERMINATION

You may not copy, modify, sublicense, or distribute the Document except as expressly provided under this License. Any attempt otherwise to copy, modify, sublicense, or distribute it is void, and will automatically terminate your rights under this License.
However, if you cease all violation of this License, then your license from a particular copyright holder is reinstated (a) provisionally, unless and until the copyright holder explicitly and finally terminates your license, and (b) permanently, if the copyright holder fails to notify you of the violation by some reasonable means prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is reinstated permanently if the copyright holder notifies you of the violation by some reasonable means, this is the first time you have received notice of violation of this License (for any work) from that copyright holder, and you cure the violation prior to 30 days after your receipt of the notice.
Termination of your rights under this section does not terminate the licenses of parties who have received copies or rights from you under this License. If your rights have been terminated and not permanently reinstated, receipt of a copy of some or all of the same material does not give you any rights to use it.

## 10. FUTURE REVISIONS OF THIS LICENSE

The Free Software Foundation may publish new, revised versions of the GNU Free Documentation License from time to time. Such new versions will be similar in spirit to the present version, but may differ in detail to address new problems or concerns. See http://www.gnu.org/copyleft/.
Each version of the License is given a distinguishing version number. If the Document specifies that a particular numbered version of this License "or any later version" applies to it, you have the option of following the terms and conditions either of that specified version or of any later version that has been published (not as a draft) by the Free Software Foundation. If the Document does not specify a version number of this License, you may choose any version ever published (not as a draft) by the Free Software Foundation. If the Document specifies that a proxy can decide which future versions of this License can be used, that proxy's public statement of acceptance of a version permanently authorizes you to choose that version for the Document.

## 11. RELICENSING

"Massive Multiauthor Collaboration Site" (or "MMC Site") means any World Wide Web server that publishes copyrightable works and also provides prominent facilities for anybody to edit those works. A public wiki that anybody can edit is an example of such a server. A "Massive Multiauthor Collaboration" (or "MMC") contained in the site means any set of copyrightable works thus published on the MMC site.
"CC-BY-SA" means the Creative Commons Attribution-Share Alike 3.0 license published by Creative Commons Corporation, a not-for-profit corporation with a principal place of business in San Francisco, California, as well as future copyleft versions of that license published by that same organization.
"Incorporate" means to publish or republish a Document, in whole or in part, as part of another Document.
An MMC is "eligible for relicensing" if it is licensed under this License, and if all works that were first published under this License somewhere other than this MMC, and subsequently incorporated in whole or in part into the MMC, (1) had no cover texts or invariant sections, and (2) were thus incorporated prior to November 1, 2008.
The operator of an MMC Site may republish an MMC contained in the site under CC-BY-SA on the same site at any time before August 1, 2009, provided the MMC is eligible for relicensing.

# How to use this License for your documents

To use this License in a document you have written, include a copy of the License in the document and put the following copyright and license notices just after the title page:
    Copyright (c) YEAR YOUR NAME.
    Permission is granted to copy, distribute and/or modify this document
    under the terms of the GNU Free Documentation License, Version 1.3
    or any later version published by the Free Software Foundation;
    with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
    A copy of the license is included in the section entitled "GNU
    Free Documentation License".
If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts, replace the "with...Texts." line with this:
    with the Invariant Sections being LIST THEIR TITLES, with the
    Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.
If you have Invariant Sections without Cover Texts, or some other combination of the three, merge those two alternatives to suit the situation.
If your document contains nontrivial examples of program code, we recommend releasing these examples in parallel under your choice of free software license, such as the GNU General Public License, to permit their use in free software.
* ↑ {https://gcc.gnu.org/gcc-4.9/changes.html}