mozilla/rr — pull request #1357 by froydnj (2014-10-23 18:38:08+00:00)
Title: generate monkeypatching templates from Python

While investigating monkeypatching libc for x86-64, I found that doing a proper job requires a number of assembly language templates for verifying we're patching the right thing and for the monkeypatches themselves:

* separate templates to verify pthread cancellation point syscalls vs. syscalls that are not cancellation points;
* separate templates for *each* of the above depending on whether 3 or fewer syscall arguments are required (4 or more arguments requires argument shuffling to place things into the registers the kernel expects, and therefore extra instruction matches are needed);
* ...and separate templates for monkeypatching each of the above.

And that only counts syscalls that are auto-generated assembly in glibc's build process; if we want to support written-in-C syscalls, we need more. Keeping track of all these, writing code for manually matching them (along with extracting any necessary constants we're interested in), and writing code for hand-patching things (especially because some of the x86-64 monkeypatches will require multiple constants to be subbed into the code) sounded like it would get very tedious.

This patch is the proposed alternative: write the assembly templates in Python and then auto-generate code for matching sequences of bytes or for substituting particular values into them. I think it turned out pretty well...and if you don't like this version, imagine what the first iteration with recursive C++ templates must have looked like!