# Linux Applications Debugging Techniques/Leaks
### What to look for

Memory can be allocated through many API calls:
* malloc
* calloc
* realloc
* memalign
* posix_memalign
* valloc
* mmap
* brk / sbrk
To return memory to the OS:
* free
* munmap

### Valgrind

Valgrind should be the first stop for any memory related issue. However:
* it slows down the program by at least one order of magnitude. In particular server C++ programs can be slowed down 15-20 times.
* from experience, some versions might have difficulties tracking mmap allocated memory.
* on amd64, the vex disassembler is likely to fail (v3.7) rather sooner than later so valgrind is of no use for any medium or intensive usage.
* you need to write suppressions to filter down the issues reported.
If these are real drawbacks, lighter solutions are available.

### mudflap

* http://gcc.gnu.org/wiki/Mudflap_Pointer_Debugging
Note: Mudflap has been removed from GCC 4.9 and later

### memleax

* memleax
Memleax debugs memory leak of a running process by attaching it, without recompiling program or restarting target process!!!
It's very convenient and suitable for production environment.
It works on GNU/Linux-x86_64 and FreeBSD-amd64.

### gcc/clang flags

See https://lemire.me/blog/2016/04/20/no-more-leaks-with-sanitize-flags-in-gcc-and-clang/

### bcc

See https://github.com/iovisor/bcc/blob/master/tools/memleak.py in https://github.com/iovisor/bcc.

### libmemleak

* libmemleak
Libmemleak finds memory leaks that cause a process to slowly increase the amount of memory it uses, also without the need to recompile the program as it can be LD_PRELOAD-ed when starting the program under test. Unlike valgrind it hardly slows down the process under test. Leaks are reported on a per backtrace basis. This is sometimes very important because often the caller, deeper into the backtrace, is responsible for the leak (not freeing it) while the actual place where the allocation happens wouldn't tell you anything.
It's been tested on GNU/Linux-x86_64.

##### Sample report
### DIY:libmtrace

The GNU C library comes with a built-in functionality to help detecting memory issues: mtrace. One of the shortcomings: it does not log the call stacks of the memory allocations it tracks. We can build an interposition library to augment mtrace.

##### The basics

The malloc implementation in the GNU C library provides a simple but powerful way to detect memory leaks and obtain some information to find the location where the leaks occurs, and this, with rather minimal speed penalties for the program.
Getting started is as simple as it can be:
* #include mcheck.h in your code.
* Call mtrace to install hooks for malloc, realloc, free and memalign. From this point on, all memory manipulations by these functions will be tracked. Note there are other untracked ways to allocate memory.
* Call muntrace to uninstall the tracking handlers.
* Recompile.
Under the hood, mtrace installs the four hooks mentioned above. The information collected through the hooks is written to a log file.
Note: there are other ways to allocate memory, notably mmap. These allocations will not be reported, unfortunately.
Next:
* Set the MALLOC_TRACE environment variable with the memory log name.
* Run the program.
* Run the memory log through mtrace.
One of the leaks (the malloc call) was precisely traced to the exact file and line number. However, the other leaks at line 25, while detected, we do not know where they occur. The two memory allocations for the std::string are buried deep inside the C++ library. We would need the stack trace for these two leaks to pinpoint the place in our code.
We can use GDB (or the trace_call macro) to get the allocations' stacks:

##### A couple of improvements

It would be good to have mtrace itself dump the allocation stack and dispense with gdb. The modified mtrace would have to supplement the information with:
* The stack trace for each allocation.
* Demangled function names.
* File name and line number.
Additionally, we can put the code in a library, to free the program from being instrumented with mtrace. In this case, all we have to do is interpose the library when we want to trace memory allocations (and pay the performance price).
Note: getting all this information at runtime, particularly in a human-readable form will have a performance impact on the program, unlike the plain vanilla mtrace supplied with glibc.

##### The stack trace

A good start would be to use another API function: backtrace_symbols_fd. This would print the stack directly to the log file. Perfect for a C program but C++ symbols are mangled:
For C++ we would have to get the stack (backtrace_symbols), resolve each address (dladdr) and demangle each symbol name (abi::__cxa_demangle).

##### Caveats

* Memory allocation is one of these basic operation everything else builds on. One needs to allocate memory to load libraries and executables; needs to allocate memory to track memory allocations; and we hook onto it very early in the life of a process: the first pre-loaded library is the memory tracking library. Thus, any API call we make inside this interposition library can reserve surprises, especially in multi-threaded environments.
* The API functions we use to trace the stack can allocate memory. These allocations are also going through the hooks we installed. As we trace the new allocation, the hooks are activated again and another allocation is made as we trace this new allocation. We will run out of stack in this infinite loop. We break out of this pitfall by using a per-thread flag.
* The API functions we use to trace the stack can deadlock. Suppose we would use a lock while in our trace. We lock the trace lock and we call dladdr which in turn tries to lock a dynamic linker internal lock. If on another thread dlopen is called while we trace, dlopen locks the same linker lock, then allocates memory: this will trigger the memory hooks and we now have the dlopen thread wait on the trace lock with the linker lock taken. Deadlock.
* On some platforms (gcc 4.7.2 amd64) TLS calls would trip the memalign hook. This could result in an infinite recursion if the memalign hook in its turn, accesses a TLS variable.

##### What we got

Let's try again with our new library:
Apparently, not much of an improvement: the summary still does not get us back to line 25 in main.cpp. However, if we search for address 8bf89b0 in the trace log, we find this:
This is good, but having file and line information would be better.

##### File and line

Here we have a few possibilities:
* Run the address (e.g. 0x40178450 above) through the addr2line tool. If the address is in a shared object that the program loaded, it might not resolve properly.
* If we have a core dump of the program, we can ask gdb to resolve the address. Or we can attach to the running program and resolve the address.
* Use the API described here. The downside is that it takes a quite heavy toll on the performance of the program.

##### Resources

* The GNU C library manual
* Using libbfd
* Linux Programming Toolkit

### DIY:libmemleak

Building on libmtrace, we can go one step further and have an interposition library track the memory allocations made by the program. The library generates a report on demand, much like Valgrind does.
Libmemleak is significantly faster than valgrind but also has limited functionality (only leak detection).

##### Operation

libmemtrace has two drawbacks:
* The log file will quickly reach gigs
* You are left grepping the log to figure out what leaks when
A better solution would be to have an interposition library to collect memory operations information and to generate a report on-demand.
For mmap/munmap we have no choice but hook these directly. Thus, an call from within the application would first hit the hooks in libmemleak, then go to libc. For mallocrealloc/memalign/free we have two options:
* Use mtrace/muntrace as before, to install hooks that will be called from within libc. Thus, a malloc call would first go through libc which will then call the hooks in libmemtrace. This leave us at the mercy of libc.
* The second solution is to hook these like m(un)map.
The second solution also frees mtrace/muntrace for on-demand report generation:
* A first call to mtrace will kick in data collection.
* Subsequent calls to mtrace will generate reports.
* muntrace will stop data collection and will generate a final report.
* MALLOC_TRACE is not needed.
The application can then sprinkle its code with mtrace calls at strategic places to avoid reporting much noise. These calls will do nothing in a normal operation as long as MALLOC_TRACE is not set. Or, the application can be completely ignorant of the ongoing data collection (no mtrace calls within the application code) and libmemleak can start collecting as early as being loaded and generate one report upon being unloaded.
To control the libmemleak functionality, an environment variable - MEMLEAK_CONFIG - has to be set before loading the library:
* mtraceinit will instruct the library to start collecting data upon being loaded. Default is off and the application has to be instrumented with m(un)trace calls.
Thus, all the hooks have to do is to call into the reporting:

##### Sample report
### mallinfo

The mallinfo API is rumored to be deprecated. But, if available, it is very useful:

##### References

* mallinfo
* mallinfo deprecated

### /proc

Coarse grained information can be obtained from /proc:

##### References

* Memory usage script

### Various tools

* gdb-heap extension
* Dr. Memory
* Pin tool
* heaptrack
* Data-type profiling for perf
* Type-preserving heap profiler
* See https://stackoverflow.com/questions/18455698/lightweight-memory-leak-debugging-on-linux for a list of tools. Out of which, memleax deserves a special mention for its out-of-process approach.
* ↑ {https://gcc.gnu.org/gcc-4.9/changes.html}