Package: brm
Bicycle Repair ManBRM is a python source modification library to perform lossless modifications with the guarantee of full-roundtripability. It is
generally used for unstructured source parts, where the modification can be done directly on the tokens.A simple example would be theTokenTransformer, where we change each+(plus) operator
to a-(minus) operator.classDestoryAllOfThem(TokenTransformer):# Replace each PLUS token with a MINUSdefvisit_plus(self,token):returntoken._replace(string="-")transformer=DestoryAllOfThem()asserttransformer.transform("(2p) + 2 # with my precious comment")=="(2p) - 2 # with my precious comment"One advantage of token based refactoring over any form of structured tree representation is that, you are much more
liberal about what you can do. Do you want to prototype a new syntax idea, for example a√operator; here you go:classSquareRoot(TokenTransformer):# Register a new token called `squareroot`defregister_squareroot(self):return"√"# Match a squareroot followed by a number@pattern("squareroot","number")defremove_varprefix(self,operator,token):returnself.quick_tokenize(f"int({token.string}** 0.5)")sqr=SquareRoot()asserteval(sqr.transform("√9"))==3Why BRMBRM is an extremely simple, dependency-free, pure-python tool with 500 LoC that you can easily vendor.BRM supports each new Python syntax out of the box, no need to wait changes on our upstream.BRM supports incomplete files (and files that contain invalid python syntax).BRM supports introducing new syntax and making it permanent for prototypes.If you need any of these, BRM might be the right fit. But I would warn against using it for complex
refactoring tasks, since that is not a problem we intend to tackle. If you need such a tool, take a look
atrefactororparso.PermanencyIf you loved the concept of transformers and use them in real world code, BRM exposes a custom
encoding that will run your transformers automatically when specified.Write a transformerCopy it to the~/.brmfolder, or simply usecp <file>.py $(python -m brm)Specify# coding: brmon each fileExample:frombrmimportTokenTransformer,patternclassAlwaysTrue(TokenTransformer):STRICT=False# Make every if/elif statement `True`@pattern("name","*any","colon")defalways_true_if(self,*tokens):statement,*_,colon=tokensifstatement.stringnotin{"if","elif"}:returntrue,=self.quick_tokenize("True")return(statement,true,colon)Let's put our transformer to the BRM's transformer folder, and run our example.(.venv) [  9:12ÖS ]  [ isidentical@x200:~ ]
 $ cat -n r.py
     1  # coding: brm
     2
     3  a = 2
     4  if a > 2:
     5      print("LOL")
(.venv) [  9:12ÖS ]  [ isidentical@x200:~ ]
 $ cp test.py $(python -m brm)
(.venv) [  9:12ÖS ]  [ isidentical@x200:~ ]
 $ python r.py
LOLTA-DA!BRM Pattern SyntaxFor BRM, a python source code is just a sequence of tokens. It doesn't create any relationships between them,
or even verify the file is syntactically correct. For example take a look at the following file:ifa==x:2+2# lolFor BRM, in an abstract fashion, the file is just the following text:NAME NAME EQEQUAL NAME COLON NEWLINE INDENT NUMBER PLUS NUMBER COMMENT NEWLINE DEDENT ENDMARKERAnd internally it is processed like this:If you want to match binary plus operation here (2 + 2), you can create pattern withnumber, plus, name.Note: If you want to visualize your patterns and see what they match, giveexamples/visualize.pya shot.ExtrasIf you are using theTokenTransformer, there are a few handy functions that you might check out:FunctionReturnsDescriptionquick_tokenize(source: str, *, strip: bool = True)List[TokenInfo]Break the givensourcetext into a list of tokens. IfstripisTrue, then the last 2 tokens (NEWLINE,EOF) will be omitted.quick_untokenize(tokens: List[TokenInfo])strConvert the given sequence oftokensback to a representation which would yield the same tokens back when tokenized (a lossy conversion). If you want a full round-trip / lossless conversion, usetokenize.untokenize.directional_length(tokens: List[TokenInfo])intCalculate the linear distance between the first and the last token of the sequence.shift_all(tokens: List[TokenInfo], x_offset: int, y_offset: int)List[TokenInfo]Shift each token in the given sequence byx_offsetin the column offsets, and byy_offsetin the line numbers. Return the new list of tokens.until(toktype: int, stream: List[TokenInfo])Iterator[TokenInfo]Yield all tokens until a token oftoktypeis seen. If there are no such tokens seen, it will raise aValueError_get_type(token: TokenInfo)intReturn the type of the given token. Useful withuntil(). (internal)