Nobody Can Program Correctly
Lessons from 20 Years of Debugging C++ CodeSebastian Theophil, think-cell Software, Berlin
stheophil@think-cell.com

1 / 70What is a bug?
Your program does not conform to your specification.
Specification may be explicit or implicit.
A bug report describes, at best, a symptom of a bug.

1 / 70First, you have to notice the bug

2 / 70First, you have to notice the bug
Better:A crash, a core dump, an error message, a line in a log file, a misbehavior
Even Better:
Systematic testing by your QA engineers
Best:
Unit testing, automated testing of any kind
These detect symptoms of your bug

3 / 70First, you have to notice the bug
Better:A crash, a core dump, an error message, a line in a log file, a misbehavior
Even Better:
Systematic testing by your QA engineers
Best:
Unit testing, automated testing of any kind
These detect symptoms of your bug
How many to you notice?
How many when they occur on your client's computer?

4 / 70First, you have to notice the bug

5 / 70First, you have to notice the bug
Compile time
Type checking
static_assert
constexpr evaluation

6 / 70First, you have to notice the bug
Compile time
Type checking
static_assert
constexpr evaluation

CppCon 2020: “Constexpr Everything” - The Standard Library, Microkernel, Apps, and Unit Tests - Rian Quinn

7 / 70First, you have to notice the bug
Compile time
Type checking
static_assert
constexpr evaluation

CppCon 2020: “Constexpr Everything” - The Standard Library, Microkernel, Apps, and Unit Tests - Rian Quinn
Build time & QA
Unit testing
Automated testing

8 / 70First, you have to notice the bug
RuntimeStrict error checking
Check all API return values and report them
Assert pre-conditions and post-conditions
Report them
Enforce invariants, notice unexpected behavior sooner

9 / 70First, you have to notice the bug

10 / 70

Learning from a single occurrence?
"One in a million is always next Tuesday."
Gordon Letwin, architect for MS-DOS 4
https://docs.microsoft.com/en-us/archive/blogs/larryosterman/one-in-a-million-is-next-tuesday
May be a rare chance to analyze a problem.
Hard to reproduce in the lab, yet with an obvious fix.
Will occur 1000s of times once product is rolled out!

11 / 70

Learning from a single occurrence?
Techniques
Attach debugger and look at stack trace
Get full memory dump, e.g., with ProcDump
Don't turn it off and on again!
Results
Form a hypothesis on the cause of the symptom
Better error reporting, stricter invariants?

12 / 70

Example
Shared temporary data between processes in a single file

13 / 70

Example
Shared temporary data between processes in a single file

14 / 70

Example
Shared temporary data between processes in a single file

15 / 70

Example
Shared temporary data between processes in a single file
int append(std::vector<std::byte> vecbyte) {std::scoped_lock lock(lock_on_temp_file());
CompactSharedTempFileIfNecessary();
int handleNew = AppendDataToTempFile(vecbyte);
return handleNew;}

16 / 70

Example
Shared temporary data between processes in a single file
int append(std::vector<std::byte> vecbyte) {std::scoped_lock lock(lock_on_temp_file());
CompactSharedTempFileIfNecessary();
int handleNew = AppendDataToTempFile(vecbyte);
return handleNew;}
void read(int handle, void* pv, std::size_t offset, std::size_t count) {std::shared_lock lock(lock_on_temp_file());
CopyDataAtHandleOffset(handle, offset, count, pv);}

17 / 70

Example
Shared temporary data between processes in a single file
int append(std::vector<std::byte> vecbyte) {std::scoped_lock lock(lock_on_temp_file());
CompactSharedTempFileIfNecessary();
int handleNew = AppendDataToTempFile(vecbyte);
return handleNew;}
void read(int handle, void* pv, std::size_t offset, std::size_t count) {std::shared_lock lock(lock_on_temp_file());
CopyDataAtHandleOffset(handle, offset, count, pv);}
void delete(int handle) noexcept {
std::shared_lock lock(lock_on_temp_file());
LookupHandle(handle).m_bDeleted = true;}
18 / 70

Example
Shared temporary data between processes in a single file
int append(std::vector<std::byte> vecbyte) {std::scoped_lock lock(lock_on_temp_file());
CompactSharedTempFileIfNecessary();
int handleNew = AppendDataToTempFile(vecbyte);
return handleNew;}
void read(int handle, void* pv, std::size_t offset, std::size_t count) {std::shared_lock lock(lock_on_temp_file());
CopyDataAtHandleOffset(handle, offset, count, pv);}
void delete(int handle) noexcept {
std::shared_lock lock(lock_on_temp_file());
LookupHandle(handle).m_bDeleted = true;}
19 / 70

Closing in on the problem

20 / 70

Closing in on the problem
Debugging is an iterative process
Don't do it fast, do it right.

21 / 70

Reproduction
1. Always reproducible in debug builds, on any machine
2. Sometimes reproducible, in debug, on any machine
3. Always reproducible, in debug, on specific machines
4. Sometimes reproducible, in debug, on specific machines
5. ...
6. ...
7. Sometimes reproducible, only in release builds, only on specific machines
We want to move up!

22 / 70

Reproduction"Only sometimes reproducible, ..."
Use tools that detect hard-to-reproduce issues:
AddressSanitizer, ThreadSanitizer, UndefinedBehaviorSanitizer
Is it a timing issue? Debugging may make the issue disappear because now the code is running too slow.
Can you write a stress test to force the issue?
Can you write to a log file and still reproduce the bug?
Write code to diagnose system when the problem occurs.

23 / 70

Reproduction"Only reproducible on some machines ..."
Gather info about environment:
OS version, CPU, version of your software, etc
Anything that could interfere with your program? Desktop environments are the worst!
Virus scanner blocking files
System tools hooking file access
System management tools disabling parts of your software
Non-standard user rights management
DRM software that hooks into your software
Can you reproduce this environment in a VM? On a client machine? Can the client ship an identical machine?
Could also be a timing issue. Very slow machine? Very fast? Very busy?
24 / 70

Reproduction
If all else fails:
Automatically report errors to catch more similar issues (Google CrashPad)
Write and ship analysis code that tries to nail down the issue
Try out fixes if you have great reporting
Can you look at your program state after the problem?
A file written by your program showing the wrong output?

25 / 70

Example
Customer report: Starting our software fails. According to log file with std::bad_alloc .

26 / 70

Example
Customer report: Starting our software fails. According to log file with std::bad_alloc .
Telemetry tells us, loading settings file fails.
No reproduction but we get the settings file and can look at it.
Contains bogus data in number formatted string.
2^18 times format string "%a" instead of once.

27 / 70

Example
Customer report: Starting our software fails. According to log file with std::bad_alloc .
Telemetry tells us, loading settings file fails.
No reproduction but we get the settings file and can look at it.
Contains bogus data in number formatted string.
2^18 times format string "%a" instead of once.
Format string doubled 18 times?

28 / 70

Example
Customer report: Starting our software fails. According to log file with std::bad_alloc .
Telemetry tells us, loading settings file fails.
No reproduction but we get the settings file and can look at it.
Contains bogus data in number formatted string.
2^18 times format string "%a" instead of once.
Format string doubled 18 times?
std::vector<std::pair<std::size_t, format>>

29 / 70

Example
Customer report: Starting our software fails. According to log file with std::bad_alloc .
Telemetry tells us, loading settings file fails.
No reproduction but we get the settings file and can look at it.
Contains bogus data in number formatted string.
2^18 times format string "%a" instead of once.
Format string doubled 18 times?
std::vector<std::pair<std::size_t, format>>Stored in shared memory, shared between different processes

30 / 70

Example
Customer report: Starting our software fails. According to log file with std::bad_alloc .
Telemetry tells us, loading settings file fails.
No reproduction but we get the settings file and can look at it.
Contains bogus data in number formatted string.
2^18 times format string "%a" instead of once.
Format string doubled 18 times?
std::vector<std::pair<std::size_t, format>>Stored in shared memory, shared between different processes
Vector size (end - begin) / sizeof(std::pair<std::size_t, format>)

31 / 70

Example
Customer report: Starting our software fails. According to log file with std::bad_alloc .
Telemetry tells us, loading settings file fails.
No reproduction but we get the settings file and can look at it.
Contains bogus data in number formatted string.
2^18 times format string "%a" instead of once.
Format string doubled 18 times?
std::vector<std::pair<std::size_t, format>>Stored in shared memory, shared between different processes
Vector size (end - begin) / sizeof(std::pair<std::size_t, format>)How big is sizeof(std::size_t) ?

32 / 70

Example
Customer report: Starting our software fails. According to log file with std::bad_alloc .
Telemetry tells us, loading settings file fails.
No reproduction but we get the settings file and can look at it.
Contains bogus data in number formatted string.
2^18 times format string "%a" instead of once.
Format string doubled 18 times?
std::vector<std::pair<std::size_t, format>>Stored in shared memory, shared between different processes
Vector size (end - begin) / sizeof(std::pair<std::size_t, format>)How big is sizeof(std::size_t) ?
Not the same size in 32 bit and 64 bit processes

33 / 70

Finding the problem
From Reproduction to Problem Description
Sometimes symptom does not lead directly to cause
Problem may have happened earlier & somewhere else in your code
How to find the real problem?
Tools: Again, use all sanitizers!

34 / 70

Finding the problem
Gain understanding of larger system by tracing
Get an understanding of code being called/order of calls
Good old printf debugging
Downside: Requires recompilation
Better: Use gdb/lldb/Visual Studio tracing breakpoints
No recompilation necessary
Tracing breakpoints can be added to OS functions, binary code
gdb and lldb are powerful
Can print stack traces
Can be scripted to execute commands automatically, add a large number of breakpoints automatically
gdb/lldb even have Python API
Careful: Tracing may make timing-dependent bugs disappear

35 / 70

Finding the problem
Gain understanding of OS interaction by tracing
Know your OS specific logging tools, e.g.,
Windows: ProcessMonitor (https://docs.microsoft.com/en-us/sysinternals/downloads/procmon)
macOS: dtrace (http://dtrace.org/blogs/about/)
Linux: strace etc (https://linux-audit.com/monitor-file-access-by-linux-processes/)
Know your Operating System!
Semantics of OS primitives
File locks, shared memory, virtual memory, file system, I/O, User Interface, Rendering

36 / 70

Finding the problem
Isolate the wrong part of your code
Do you have a state that still worked?
Can you find breaking change by binary searching your commits?
git bisect

Understand change to prevent error cycles
Step through working/broken versions in parallel, where does behavior differ?
Do the reverse:
disable code until bug disappears
Again, use "binary search" to track down the problem in least number of steps
Both require knowledge of code base. Can be very time consuming.

37 / 70

Finding the problem
Improve code to find the problem
Document invariants: Use asserts a lot
For complex checks, use temporary asserts to narrow down problems
Legacy code?
Introduce safe programming techniques e.g. smart pointers, RAII etc.
May fix bugs you haven't even reproduced yet

38 / 70

Finding the problem
Reverse debugging tools
let you step backwards through program
WinDbg https://www.youtube.com/watch?v=l1YJTg_A914
Undo (Linux) https://undo.iorr (Linux) https://rr-project.org
Know your debugger itself
Do you use data breakpoints/watchpoints?
Do you put frequently used functionality into debugger scripts?
Write debug visualizers for your data types!
Get at least passive assembly skills

39 / 70

Example
std::array<int, 4> an = {1, 5, 7, 8};
auto rng = GetItemFromIndices(&an[0], 4);assert(rng.size()==4); // Fails with rng.size()==2

40 / 70

Example
std::array<int, 4> an = {1, 5, 7, 8};
auto rng = GetItemFromIndices(&an[0], 4);assert(rng.size()==4); // Fails with rng.size()==2
lldb> watch set expression -s 4 -- &an[0]41 / 70

Example
std::array<int, 4> an = {1, 5, 7, 8};
auto rng = GetItemFromIndices(&an[0], 4);assert(rng.size()==4); // Fails with rng.size()==2
lldb> watch set expression -s 4 -- &an[0]std::byte* p = &an[0];for(int v=0; v<4; ++v) {// do something with p + v * sizeof(int)}42 / 70

Example
std::array<int, 4> an = {1, 5, 7, 8};
auto rng = GetItemFromIndices(&an[0], 4);assert(rng.size()==4); // Fails with rng.size()==2
lldb> watch set expression -s 4 -- &an[0]std::byte* p = &an[0];for(int v=0; v<4; ++v) {// do something with p + v * sizeof(int)
}

->

ldr
ldrsw
add

x8, [x9, #0x100]
x10, [x9, #0xf8]
x8, x8, x10, lsl #3

43 / 70

Example
std::array<int, 4> an = {1, 5, 7, 8};
auto rng = GetItemFromIndices(&an[0], 4);assert(rng.size()==4); // Fails
lldb> watch set expression -s 4 -- &an[0]std::byte* p = &an[0];for(int v=0; v<4; ++v) {// do something with p + v * sizeof(int)
}

->

ldr
ldrsw
add

x8, [x9, #0x100]
x10, [x9, #0xf8]
x8, x8, x10, lsl #3; get pointer p to an; get loop variable v; access p[v<<3] or p[v*8]

44 / 70

Example
std::array<int, 4> an = {1, 5, 7, 8};
auto rng = GetItemFromIndices(&an[0], 4);assert(rng.size()==4); // Fails
lldb> watch set expression -s 4 -- &an[0]std::byte* p = &an[0];for(int v=0; v<4; ++v) {// do something with p + v * sizeof(int)
}

->

ldr
ldrsw
add

x8, [x9, #0x100]
x10, [x9, #0xf8]
x8, x8, x10, lsl #3; get pointer p to an; get loop variable v; access p[v<<3] or p[v*8]

The buggy code was ported from Windows to macOS.

45 / 70

Example
std::array<int, 4> an = {1, 5, 7, 8};
auto rng = GetItemFromIndices(&an[0], 4);assert(rng.size()==4); // Fails
lldb> watch set expression -s 4 -- &an[0]std::byte* p = &an[0];for(int v=0; v<4; ++v) {// do something with p + v * sizeof(int)
}

->

ldr
ldrsw
add

x8, [x9, #0x100]
x10, [x9, #0xf8]
x8, x8, x10, lsl #3; get pointer p to an; get loop variable v; access p[v<<3] or p[v*8]

The buggy code was ported from Windows to macOS.
int* is cast to long* somewhere. On Windows, sizeof(long)==4 . On macOS, sizeof(long)==8 .

46 / 70

Finding the problem
From reproduction to cause of the bug
Iterative Process: Analysis - Hypothesis - Test - Repeat
Use all tools at your disposal and learn to use themdebuggers, sanitizers
reverse debuggers
OS facilities to capture process traces
Get to know the operating system

47 / 70

Finding the problem
From reproduction to cause of the bug
Iterative Process: Analysis - Hypothesis - Test - Repeat
Use all tools at your disposal and learn to use themdebuggers, sanitizers
reverse debuggers
OS facilities to capture process traces
Get to know the operating system
But most importantly

48 / 70

Finding the problem
From reproduction to cause of the bug
Iterative Process: Analysis - Hypothesis - Test - Repeat
Use all tools at your disposal and learn to use themdebuggers, sanitizers
reverse debuggers
OS facilities to capture process traces
Get to know the operating system
But most importantly

Question your assumptions.

49 / 70

Classify bug
You didn't write what you meant
Uninitialised data (e.g. indices?)
Memory management problem
use after free,or rather reference counting bug?
use of out-of-scope temporary
Stack corruption
Data corruption through missing locks
Often, fix is a local change or use of better programming practices

50 / 70

Classify bug
You didn't write what you meant
Uninitialised data (e.g. indices?)
Memory management problem
use after free,or rather reference counting bug?
use of out-of-scope temporary
Stack corruption
Data corruption through missing locks
Often, fix is a local change or use of better programming practices
But: Always check the rest of code base!

51 / 70

Classify bug
You wrote what you meant, but meant wrong
i.e. your mental model was wrong
You need a new one. A local fix will not be enough.
You didn't understand the spec of somebody else's code correctlyWrong use of internal and external API
Use of OS facilities that don't work like you thought they did
You didn't understand your own requirements correctly
Is your algorithm correct at all?
Is the algorithm the best choice?

52 / 70

Classify bug
How to tell those two cases apartThat is not easy.

53 / 70

Classify bug
How to tell those two cases apartThat is not easy.
void do_something_interesting(std::vector<std::unique_ptr<T>> const& vecp) noexcept {
tc::for_each(vecp, [](auto const& p) noexcept {foo(*p);});}54 / 70

Classify bug
How to tell those two cases apartThat is not easy.
void do_something_interesting(std::vector<std::unique_ptr<T>> const& vecp) noexcept {
tc::for_each(vecp, [](auto const& p) noexcept {foo(*p); // crashes here with dereferencing null pointer});}

55 / 70

Classify bug
How to tell those two cases apartThat is not easy.
void do_something_interesting(std::vector<std::unique_ptr<T>> const& vecp) noexcept {
tc::for_each(vecp, [](auto const& p) noexcept {foo(*p); // crashes here with dereferencing null pointer});}void do_something_interesting(std::vector<std::unique_ptr<T>> const& vecp) noexcept {
tc::for_each(vecp, [](auto const& p) noexcept {if(p) {
foo(*p);}});}56 / 70

Classify bug
How to tell those two cases apartThat is not easy.
void do_something_interesting(std::vector<std::unique_ptr<T>> const& vecp) noexcept {
tc::for_each(vecp, [](auto const& p) noexcept {foo(*p); // crashes here with dereferencing null pointer});}void do_something_interesting(std::vector<std::unique_ptr<T>> const& vecp) noexcept {
tc::for_each(vecp, [](auto const& p) noexcept {if(p) {
foo(*p);}});}Wrong! Or at least, possibly wrong.

57 / 70

Classify bug
How to tell those two cases apart
1. Look at the bigger picture
2. What are you trying to achieve?
3. How are you trying to achieve that?
4. Is that the correct approach?
5. Do you understand all the invariants?
6. Is the bug violating one of those invariants?
Rethink your assumptions, your mental model!

58 / 70

Fixing bugs
Smallest possible fix
Solves the problem
Does not introduce new bugs

59 / 70

Fixing bugs
Smallest possible fix
Solves the problem
Does not introduce new bugs
But
May not fix the root of the problem
Or worse, it only hides one instance of the problem.
May, over time, reduce code quality and make code harder to understand60 / 70

Fixing bugs
What should the ideal solution look like in an ideal world?
Given everything you know now, how would you solve the problem?
You need to move towards this solution!
Given my constraints, what should I change now?
Do you need to ship a fix quickly?
Do you work in an especially secure environment?
In a very regulated environment?
1. Deliver small fix in the stable build, ship it fast.
2. Attempt thorough fix in development branch.

61 / 70

Fixing bugs
Why did the bug happen in the first place?
Was it too hard to program correctly, easy to program incorrectly?
What was missing to program correctly?
A library feature or helper function?
An algorithm?
A standard programming practice?
How can we prevent it from happening again?
Can you make your fix elsewhere in the code?
e.g. introducing smart pointersreplacing self-written for-loops with the correct standard algorithm
Look through your codebase for that pattern!
Can you introduce the missing abstraction?

62 / 70

Fixing bugs
Missing abstractions
std::vector<int> vecn;
std::ranges::sort(vecn, std::ranges::less());
auto rng2 = std::ranges::unique(rng1, std::ranges::equal());

Note the two different predicates!
They must be compatible! This was done 74 times in our code base, in different ways, some were wrong!

63 / 70

Fixing bugs
Missing abstractions
std::vector<int> vecn;
std::ranges::sort(vecn, std::ranges::less());
auto rng2 = std::ranges::unique(rng1, std::ranges::equal());

Note the two different predicates!
They must be compatible! This was done 74 times in our code base, in different ways, some were wrong!
Replaced bytemplate<typename Rng, typename Less = tc::fn_less>auto sort_inplace_unique_range(Rng&& rng, Less&& less)

64 / 70

Fixing bugs
Missing abstractions
Make correct error handling easy, convenient and mandatory!
if (auto const ohfile = ERRNORETRYIGNORE(
open(file, ...)
tc::err::returned_nonnegative_value(), // success
tc::err::returned(invalid_filehandle, EINTR), // retry
tc::err::returned(invalid_filehandle, {EPERM, ENOENT, EACCES}) // allowed errors)){}

65 / 70

Fixing bugs
Missing abstractions
Make correct error handling easy, convenient and mandatory!
if (auto const ohfile = ERRNORETRYIGNORE(
open(file, ...)
tc::err::returned_nonnegative_value(), // success
tc::err::returned(invalid_filehandle, EINTR), // retry
tc::err::returned(invalid_filehandle, {EPERM, ENOENT, EACCES}) // allowed errors)){}
RECT rect;
APIERR(GetClientRect(wnd, &rect));_ASSERTEQUAL(rect.top, 0);_ASSERTEQUAL(rect.left, 0);

66 / 70

Delivering Fix
Documentation in the code and outside the code
Write high-level documentation to explain concepts
Documentation in source files goes less out of date
Document what, when, who and why you did something
Can you reference an issue in your bug tracker?
The next developer may ask "Do we still need this?"
Code Reviews
or other collaborative practices like pair programming
explain what you did to others
often, errors in your thinking become obvious, once you have to spell it out

67 / 70

Delivering Fix
Good version control practices
Split changes into self-contained chunks
Separate refactors from changes to functionality
Do refactor during debugging!
Tests
May also prevent a regression from ever happening again
Tests also need to be well-written
Run automatically ideally
Trivial test cases are uselessCan you test random input?

68 / 70

Summary
1. Maximize number of bug reports
2. Analyze them quantitatively & qualitatively
3. Get best possible reproduction
Use sanitizers and other tools to improve and minimize reproduction
4. Analyze the problem to find underlying cause
Gain enough understanding of your system to do so!
5. Classify the bug
6. Decide the scope: Small fix, large fix, or both?
7. Fix8. Prevent bug from ever happening again!
9. Document, test, and review

69 / 70

Thank you!
And yes, we are recruiting: hr@think-cell.com

70 / 70