Package: horast
Human-oriented abstract syntax tree (AST) parser/unparser for Python 3 that doesn’t discard comments.This package provides new AST node types (Comment and Directive) which inherit from nodes in typed_ast.ast3 module.
Additionally, it provides implementation of parser and unparser for the extended AST allowing
straightforward readable code generation.Simple example of how to use this package:fromhorastimportparse,unparsetree=parse("""a = 1  # a equals one after this""")print(unparse(tree))# this will print the code with original commentMore examples inexamples.ipynb.technical detailsParser is based on built-in tokenize module, as well as community packages asttokens and typed_ast.Unparser is essentially an extension of Unparser class from static_typing package.Nodes provided and handled by horast are listed below.CommentFull line as well as end-of-line comments are parsed/unparsed correctly when they are outside
of multi-line expressions.Currently, handling of comments within multi-line expressions is implemented only partially.BlockCommentNot implemented yet.This node type is intended to store consecutive full-line comments in a
single AST node, therefore simplifying handling of large blocks of comments.DirectiveSince Python doesn’t offer a direct way to convey directives in code,
comments beginning with any of the following prefixes will not be classified
as typical comments, but as directives:ifelseendifdefundefifdefifndefHowever in Python code they still remain as usual comments.For example, the comments in the following code will all be classified as directives.#ifdef DEBUGdebugging=True#elsedebugging=False#fiHowever, when executed as Python,debuggingwill always end upFalsebecause directives are preserved as usual comments in Python and therefore
they are ignored.Therefore, theDirectivenode is not meant to enable preprocessing of
Python, at least for now.Note: the prefix is checked exactly. See the following example:#if something# if some other thingThe comment in the first line will becomeDirectiveobject, while the one
on the second like will becomeCommentobject.Currently, this node is meant to work towards AST compatibility between
Python and other languages, to aide code generation from Python AST into code
in other languages – one such use case is*transpyle* project.PragmaPragmanodes follow the same general rules asDirectivenodes, i.e. they are
stored in Python code as comments, but a comment will be classified as a pragma
when it’s prefixed with a predefined prefix:" pragma:"Additionally, two subclasses of Pragma are defined in horast, each with its
own prefix that builds upon the generic pragma prefix:OpenMpPragmaclass defines prefix" pragma: omp"and stores OpenMP pragmas.OpenAccPragmaclass defines prefix" pragma: acc"and stores OpenACC pragmas.A code snippet below contains all 3 pragma types.# pragma: onceuse_openmp=Trueuse_openacc=True...a,b=np.ndarray(...)c=np.zeros(...)# pragma: acc parallel copyin(a,b) copyout(c)# pragma: acc loop gangforyinrange(ymax):# type: np.int32# pragma: acc loop workerforiinrange(imax):# type: np.int32# pragma: acc loop vector reduction(+: c[y][x])forxinrange(xmax):# type: np.int32c[y,x]+=a[y,i]*b[i,x]# pragma: acc end parallel...# pragma: omp parallel doforiinrange(input_data.size):# type: int# here we compute spam spam spamheavy_compute(input_data[i])...And thus, in the example above:all comments starting with" pragma: omp"becomeOpenMpPragmaobjects,all comments starting with" pragma: acc"becomeOpenAccPragmaobjects,all other comments starting with" pragma:"becomePragmaobjects,type comments are ignored, andall other comments becomeCommentobjects.Additionally, horast module provides an extensible infrastructure to define
customPragmasubclasses, enabling user to define their own pragmas for
experimentation. The provided OpenMP and OpenACC pragma definitions serve
as examples of how to use this feature.IncludeSimilarly to how pragmas are handled, if a comment begins with" include:"prefix, it will be classified as a special kind of include directive.Again, this will be preserved as comment in Python code, but it’s useful
for enhancing syntactic compatibility between Python and other, especially
statically compiled languages.requirementsCPython 3.5 or later.Python libraries as specified inrequirements.txt.Building and running tests additionally requires packages listed intest_requirements.txt.