Package: codetransformer
Bytecode transformers for CPython inspired by theastmodule’sNodeTransformer.What iscodetransformer?codetransformeris a library that allows us to work with CPython’s bytecode
representation at runtime.codetransformerprovides a level of abstraction
between the programmer and the raw bytes read by the eval loop so that we can
more easily inspect and modify bytecode.codetransformeris motivated by the need to override parts of the python
language that are not already hooked into through data model methods. For example:Override theisandnotoperators.Custom data structure literals.Syntax features that cannot be represented with valid python AST or source.Run without a modified CPython interpreter.codetransformerwas originally developed as part oflazyto implement
the transformations needed to override the code objects at runtime.Example UsesOverloading LiteralsWhile this can be done as an AST transformation, we will often need to execute
the constructor for the literal multiple times. Also, we need to be sure that
any additional names required to run our code are provided when we run. Withcodetransformer, we can pre compute our new literals and emit code that is
as fast as loading our unmodified literals without requiring any additional
names be available implicitly.In the following block we demonstrate overloading dictionary syntax to result incollections.OrderedDictobjects.OrderedDictis like adict;
however, the order of the keys is preserved.>>>fromcodetransformer.transformers.literalsimportordereddict_literals>>>@ordereddict_literals...deff():...return{'a':1,'b':2,'c':3}>>>f()OrderedDict([('a',1),('b',2),('c',3)])This also supports dictionary comprehensions:>>>@ordereddict_literals...deff():...return{k:vfork,vinzip('abc',(1,2,3))}>>>f()OrderedDict([('a',1),('b',2),('c',3)])The next block overridesfloatliterals withdecimal.Decimalobjects. These objects support arbitrary precision arithmetic.>>>fromcodetransformer.transformers.literalsimportdecimal_literals>>>@decimal_literals...deff():...return1.5>>>f()Decimal('1.5')Pattern Matched ExceptionsPattern matched exceptions are a good example of aCodeTransformerthat
would be very complicated to implement at the AST level. This transformation
extends thetry/exceptsyntax to accept instances ofBaseExceptionas
well subclasses ofBaseException. When excepting an instance, theargsof the exception will be compared for equality to determine which exception
handler should be invoked. For example:>>>@pattern_matched_exceptions()...deffoo():...try:...raiseValueError('bar')...exceptValueError('buzz'):...return'buzz'...exceptValueError('bar'):...return'bar'>>>foo()'bar'This function raises an instance ofValueErrorand attempts to catch it. The
first check looks for instances ofValueErrorthat were constructed with an
argument of'buzz'. Because our custom exception is raised with'bar',
these are not equal and we do not enter this handler. The next handler looks forValueError('bar')which does match the exception we raised. We then enter
this block and normal python rules take over.We may also pass their own exception matching function:>>>defmatch_greater(match_expr,exc_type,exc_value,exc_traceback):...returnmath_expr>exc_value.args[0]>>>@pattern_matched_exceptions(match_greater)...deffoo():...try:...raiseValueError(5)...except4:...return4...except5:...return5...except6:...return6>>>foo()6This matches on when the match expression is greater in value than the first
argument of any exception type that is raised. This particular behavior would be
very hard to mimic through AST level transformations.Core AbstractionsThe three core abstractions ofcodetransformerare:TheInstructionobject which represents anopcodewhich may be paired
with some argument.TheCodeobject which represents a collection ofInstructions.TheCodeTransformerobject which represents a set of rules for
manipulatingCodeobjects.InstructionsTheInstructionobject represents an atomic operation that can be performed
by the CPython virtual machine. These are things likeLOAD_NAMEwhich loads
a name onto the stack, orROT_TWOwhich rotates the top two stack elements.Some instructions accept an argument, for exampleLOAD_NAME, which modifies
the behavior of the instruction. This is much like a function call where some
functions accept arguments. Because the bytecode is always packed as raw bytes,
the argument must be some integer (CPython stores all arguments two in bytes).
This means that things that need a more rich argument system (likeLOAD_NAMEwhich needs the actual name to look up) must carry around the actual arguments
in some table and use the integer as an offset into this array. One of the key
abstractions of theInstructionobject is that the argument is always some
python object that represents the actual argument. Any lookup table management
is handled for the user. This is helpful because some arguments share this table
so we don’t want to add extra entries or forget to add them at all.Another annoyance is that the instructions that handle control flow use their
argument to say what bytecode offset to jump to. Some jumps use the absolute
index, others use a relative index. This also makes it hard if you want to add
or remove instructions because all of the offsets must be recomputed. Incodetransformer, the jump instructions all accept anotherInstructionas
the argument so that the assembler can manage this for the user. We also provide
an easy way for new instructions to “steal” jumps that targeted another
instruction so that can manage altering the bytecode around jump targets.CodeCodeobjects are a nice abstraction over python’stypes.CodeType. Quoting theCodeTypeconstructor docstring:code(argcount, kwonlyargcount, nlocals, stacksize, flags, codestring,
      constants, names, varnames, filename, name, firstlineno,
      lnotab[, freevars[, cellvars]])

Create a code object.  Not for the faint of heart.Thecodetransformerabstraction is designed to make it easy to dynamically
construct and inspect these objects. This allows us to easy set things like the
argument names, and manipulate the line number mappings.TheCodeobject provides methods for converting to and from Python’s code
representation:from_pycodeto_pycode.This allows us to take an existing function, parse the meaning from it, modify
it, and then assemble this back into a new python code object.NoteCodeobjects are immutable. When we say “modify”, we mean create a copy
with different values.CodeTransformersThis is the set of rules that are used to actually modify theCodeobjects. These rules are defined as a set ofpatternswhich are a DSL used
to define a DFA for matching against sequences ofInstructionobjects. Once
we have matched a segment, we yield new instructions to replace what we have
matched. A simple codetransformer looks like:fromcodetransformerimportCodeTransformer,instructionsclassFoldNames(CodeTransformer):@pattern(instructions.LOAD_GLOBAL,instructions.LOAD_GLOBAL,instructions.BINARY_ADD,)def_load_fast(self,a,b,add):yieldinstructions.LOAD_FAST(a.arg+b.arg).steal(a)ThisCodeTransformeruses the+operator to implement something likeCPPs token pasting for local variables. We read this pattern as a sequence
of twoLOAD_GLOBAL(global name lookups) followed by aBINARY_ADDinstruction (+operator call). This will then call the function with the
three instructions passed positionally. This handler replaces this sequence with
a single instruction that emits aLOAD_FAST(local name lookup) that is the
result of adding the two names together. We then steal any jumps that used to
target the firstLOAD_GLOBAL.We can execute this transformer by calling an instance of it on a
function object, or using it like a decorator. For example:>>>@FoldNames()...deff():...ab=3...returna+b>>>f()3Licensecodetransformeris free software, licensed under the GNU General Public
License, version 2. For more information see theLICENSEfile.SourceSource code is hosted on github athttps://github.com/llllllllll/codetransformer.