Package: baron
IntroductionBaron is a Full Syntax Tree (FST) library for Python. By opposition to anASTwhich
drops some syntax information in the process of its creation (like empty lines,
comments, formatting), a FST keeps everything and guarantees the operationfst_to_code(code_to_fst(source_code)) == source_code.RoadmapCurrent roadmap is as boring as needed:bug fixsnew small features (walker pattern, maybe code generation) and performance improvement.Installationpip install baronBasic Usagefrombaronimportparse,dumpsfst=parse(source_code_string)source_code_string==dumps(fst)Unless you want to do low level things,useRedBaroninstead of using Baron
directly. Think of Baron as the "bytecode of python source code" and RedBaron
as some sort of usable layer on top of it.If you don't know what Baron is or don't understand yet why it might be
useful for you, read the« Why is this important? » section.DocumentationBaron documentation is available onRead The Docs.ContributingIf you want to implement new grammar elements for newer python versions, here
are the documented steps for that:https://github.com/PyCQA/baron/blob/master/add_new_grammar.mdAlso note that reviewing most grammar modifications takes several hours of
advanced focusing (we can't really afford bugs here) so don't despair if you PR
seems to be hanging around, sorry for that :/And thanks in advance for your work!Financial supportBaron and RedBaron are a very advanced piece of engineering that requires a lot
of time of concentration to work on. Until the end of 2018, the development
has been a full volunteer work mostly done byBram,
but now, to reach the next level and bring those projects to the stability and
quality you expect, we need your support.You can join our contributors and sponsors on our transparentOpenCollective, every contribution will
count and will be mainly used to work on the projects stability and quality but
also on continuing, on the side, the R&D side of those projects.Our supportersWhy is this important?The usage of a FST might not be obvious at first sight so let's consider a
series of problems to illustrate it. Let's say that you want to write a program that will:rename a variable in a source file... without clashing with things that are not a variable (example: stuff inside a string)inline a function/methodextract a function/method from a series of line of codesplit a class into several classessplit a file into several modulesconvert your whole code base from one ORM to anotherdo custom refactoring operation not implemented by IDE/ropeimplement the class browser of smalltalk for python (the whole one where you can edit the code of the methods, not just showing code)It is very likely that you will end up with the awkward feeling of writing
clumpsy weak code that is very likely to break because you didn't thought about
all the annoying special cases and the formatting keeps bothering you. You may
end up playing withast.pyuntil
you realize that it removes too much information to be suitable for those
situations. You will probably ditch this task as simply too complicated and
really not worth the effort. You are missing a good abstraction that will take
care of all of the code structure and formatting for you so you can concentrate
on your task.The FST tries to be this abstraction. With it you can now work on a tree which
represents your code with its formatting. Moreover, since it is the exact
representation of your code, modifying it and converting it back to a string
will give you back your code only modified where you have modified the tree.Said in another way, what I'm trying to achieve with Baron is a paradigm change in
which writing code that will modify code is now a realist task that is worth
the price (I'm not saying a simple task, but a realistic one: it's still a
complex task).OtherHaving a FST (or at least a good abstraction build on it) also makes it easier
to do code generation and code analysis while those two operations are already
quite feasible (usingast.pyand a templating engine for example).Some technical detailsBaron produces a FST in the form of JSON (and by JSON I mean Python lists
and dicts that can be dumped into JSON) for maximum interoperability.Baron FST is quite similar to Python AST with some modifications to be more
intuitive to humans, since Python AST has been made for CPython interpreter.Since playing directly with JSON is a bit raw I'm going to build an abstraction
on top of it that will looks like BeautifulSoup/jQuery.State of the projectCurrently, Baron has been tested on the top 100 projects and the FST converts
back exactly into the original source code. So, it can be considered quite
stable, but it is far away from having been battle tested.Since the project is very young and no one is already using it except my
project, I'm open to changes of the FST nodes but I will quickly become
conservative once it gets some adoption and will probably accept to
modify it only once or twice in the future with clear indications on how to
migrate.Baron is supporting python 2 grammar and up to python 3.7 grammar.TestsRun eitherpy.test tests/ornosetestsin the baron directory.CommunityYou can reach us onirc.freenode.net#baronorirc.freenode.net##python-code-quality.Code of ConductAs a member ofPyCQA, Baron follows itsCode of Conduct.MiscOld blog post announcing the project.Not that much up to date.Changelog0.10.1 (2021-12-08)bug fix: in "a." the "." part was incorrectly recognized as a float, by bram0.10 (2021-12-08)bug fix: baron is now able to parse "class A(b, c=d): pass" by bramsome project cleaned and integration of tox with good pratices like flake8 and check-manifestbug fix for missing edge case in inner formatting by EhsanKiahttps://github.com/PyCQA/baron/pull/156complet support for float with underscores in them by tamentishttps://github.com/PyCQA/baron/pull/157bug fix for failure of parsing of "{**a}" by wavenatorhttps://github.com/PyCQA/baron/pull/1610.9 (2019-02-01)First version of full python 3.7 grammar support.BREAKING CHANGE: annotations are now member of {def,list,dict}_argument to flatten the data structureadd support for ... in from import by bramadd support for return annotation by bramadd support for exec function by bramadd support for variable annotationhttps://github.com/PyCQA/baron/pull/145by scottbelden and additional work by bramadd support for *var expressions in tuple assignment by bramadd support for raise fromhttps://github.com/PyCQA/baron/pull/120by odcinek with additional work by bramadd support for arglist usage in class definition inheritence by brambug fix byhttps://github.com/PyCQA/baron/pull/126/commits/91e839a228293698cc755a7f28afeca2669cb66ekyleatmakrs0.8 (2018-10-29)add typed parameters supporthttps://github.com/PyCQA/baron/pull/140by Scott Belden and and additional work by bram0.7 (2018-08-21)fix line continuationhttps://github.com/PyCQA/baron/pull/92by ibizamanhandle corrupt cache file situationhttps://github.com/PyCQA/baron/pull/76by ryu2fix special crashing edge case in indentation markerhttps://github.com/PyCQA/barby Ahugefixed incorrect tokenization case "d*e-1". Fixes #85https://github.com/PyCQA/baron/pull/107by boxedfix endl handling inside groupings by kyleatmakrs (extracted fromhttps://github.com/PyCQA/baron/pull/126)Python 3:python 3 parsing extracted fromhttps://github.com/PyCQA/baron/pull/126support ellipsishttps://github.com/PyCQA/baron/pull/121by odcineksupport matrix operatorhttps://github.com/PyCQA/baron/pull/117by odcineksupport f-stringshttps://github.com/PyCQA/baron/pull/110by odcineksupport numeric literalshttps://github.com/PyCQA/baron/pull/111by odcineksupport nonlocal statementhttps://github.com/PyCQA/baron/pull/112by odcineksupport keyword only markershttps://github.com/PyCQA/baron/pull/108by boxedsupport yield from statementhttps://github.com/PyCQA/baron/pull/113by odcinek and additional work by bramsupport async/await statementshttps://github.com/PyCQA/baron/pull/114by odcinek and additional work by bram0.6.6 (2017-06-12)fix situation where a deindented comment between a if and elif/else broken
parsing, seehttps://github.com/PyCQA/baron/issues/87around 35-40% to 75% parsing speed improvment on big files by duncfhttps://github.com/PyCQA/baron/pull/990.6.5 (2017-01-26)fix previous regression fix was broken0.6.4 (2017-01-14)fix regression in case a comment follow the ":" of a if/def/other0.6.3 (2017-01-02)group formatting at start of file or preceded by space with comment0.6.2 (2016-03-18)fix race condition when generating parser cache filemake all user-facing errors inherit from the same BaronError classfix: dotted_name and float_exponant_complex were missing from
nodes_rendering_order0.6.1 (2015-01-31)fix: the string was having a greedy behavior on grouping the string tokens
surrounding it (for string chains), this ends up creating an inconsistancy in
the way string was grouped in generalfix: better number parsing handling, everything isn't fixed yetmake all (expected) errors inherit from the same BaronError classfix: parsing fails correctly if a quoted string is not closed0.6 (2014-12-11)FST structure modification: def_argument_tuple is no more and all arguments
now have a coherent structure:def_argument node name attribute has been renamed to target, like in assigntarget attribute now points to a dict, not to a stringold name -> string are now target -> name_nodedef_argument_tuple is now a def_argument where target points to a tuplethis specific tuple will only has name and comma and tuple members (no more
def_argument for name)new node: long, before int and long where merged but that was causing problems0.5 (2014-11-10)rename "funcdef" node to "def" node to be way more intuitive.0.4 (2014-09-29)new rendering type in the nodes_rendering_order dictionary: string. This
remove an ambiguity where a key could be pointing to a dict or a string, thus
forcing third party tools to do guessing.0.3.1 (2014-09-04)setup.py wasn't working if wheel wasn't used because the CHANGELOG file
wasn't included in the MANIFEST.in0.3 (2014-08-21)path becomes a simple list and is easier to deal withbounding box allows you to know the left most and right most position
of a node seehttps://baron.readthedocs.io/en/latest/#bounding-boxredbaron is classified as supporting python3https://github.com/PyCQA/baron/pull/51ensure than when a key is a string, it's empty value is an empty string and
not None to avoid breaking libs that use introspection to guess the type of
the keykey renaming in the FST: "delimiteur" -> "delimiter"name_as_name and dotted_as_name node don't have the "as" key anymore as it
was useless (it can be deduce from the state of the "target" key)dotted_name node doesn't exist anymore, its existance was unjustified. In
import, from_import and decorator node, it has been replaced from a key to a
dict (with only a list inside of it) to a simple list.dumps now accept a strict boolean argument to check the validity of the FST
on dumping, but this isn't that much a public feature and should probably be
changed of API in the futurname_as_name and dotted_as_name empty value for target is now an empty string
and not None since this is a string type keyboundingbox now includes the newlines at the end of a nodeall raised exceptions inherit from a common base exception to ease try/catch
constructionsPosition's left and right functions become properties and thus
attributesPosition objects can be compared to other Position objects or any
iterablesmake_position and make_bounding_box functions are deleted in favor of
always using the corresponding class' constructor0.2 (2014-06-11)Baron now provides documentation onhttps://baron.readthedocs.iofeature: baron now run in python3 (butdoesn't implement the full python3
grammar yet) by Pierre Penninckxhttps://github.com/ibizamanfeature: drop the usage of ast.py to find print_function, this allow any
version of python to parse any other version of python also by Pierre
Penninckxfix: rare bug where a comment end up being confused as an indentation level2 new helpers: show_file and show_node, seehttps://baron.readthedocs.io/en/latest/#show-fileandhttps://baron.readthedocs.io/en/latest/#show-nodenew dictionary that provides the informations on how to render a FST node:
nodes_rendering_order seehttps://baron.readthedocs.io/en/latest/#rendering-the-fstnew utilities to find a node, seehttps://baron.readthedocs.io/en/latest/#locate-a-nodenew generic class that provide templates to work on the FST seehttps://baron.readthedocs.io/en/latest/#rendering-the-fst0.1.3 (2014-04-13)set sugar syntaxic notation wasn't handled by the dumper (apparently no one
use this on pypi top 100)0.1.2 (2014-04-08)baron.dumps now accept a single FST node, it was only working with a list of
FST nodesdon't add a endl node at the end if not present in the input stringde-uniformise call_arguments and function_arguments node, this is just
creating more problems that anything elsefixhttps://github.com/PyCQA/redbaron/issues/4fix the fact that baron can't parse "{1,}" (but "{1}" is working)0.1.1 (2014-03-23)It appears that I don't know how to write MANIFEST.in correctly0.1 (2014-03-22)Init