Package: code-ast
Code ASTFast structural analysis of any programming language in PythonProgramming Language Processing (PLP) brings the capabilities of modern NLP systems to the world of programming languages.
To achieve high performance PLP systems, existing methods often take advantage of the fully defined nature of programming languages. Especially the syntactical structure can be exploited to gain knowledge about programs.code.astprovides easy access to the syntactic structure of a program. By relying ontree-sitteras the back end, the parser supports fast parsing of variety of programming languages.The goal of code.ast is to combine the efficiency and variety of languages supported by tree-sitter with the convenience of more native parsers (likelibcst).To achieve this, code.ast adds the features:Auto-loading:Compile of source code parsers for any language supported by tree-sitter with a single keyword,Visitors:Search the AST quickly,Transformers:Transform source code easily by transforming the AST structureInstallationThe package is tested under Python 3. It can be installed via:pipinstallcode-astQuick startcode.ast can parse nearly any program code in a few lines of code:importcode_ast# Pythoncode_ast.ast('''def my_func():print("Hello World")''',lang="python")# Output:# PythonCodeAST [0, 0] - [4, 4]#    module [1, 8] - [3, 4]#        function_definition [1, 8] - [2, 32]#            identifier [1, 12] - [1, 19]#            parameters [1, 19] - [1, 21]#            block [2, 12] - [2, 32]#                expression_statement [2, 12] - [2, 32]#                    call [2, 12] - [2, 32]#                        identifier [2, 12] - [2, 17]#                        argument_list [2, 17] - [2, 32]#                            string [2, 18] - [2, 31]# Javacode_ast.ast('''public class HelloWorld {public static void main(String[] args){System.out.println("Hello World");}}''',lang="java")# Output:# JavaCodeAST [0, 0] - [7, 4]#    program [1, 0] - [6, 4]#        class_declaration [1, 0] - [5, 1]#            modifiers [1, 0] - [1, 6]#            identifier [1, 13] - [1, 23]#            class_body [1, 24] - [5, 1]#                method_declaration [2, 8] - [4, 9]#                    ...Visitorscode.ast implements the visitor pattern to quickly traverse the AST structure:importcode_astfromcode_astimportASTVisitorcode='''def f(x, y):return x + y'''# Count the number of identifiersclassIdentifierCounter(ASTVisitor):def__init__(self):self.count=0defvisit_identifier(self,node):self.count+=1# Parse the AST and then visit it with our visitorsource_ast=code_ast.ast(code,lang="python")count_visitor=IdentifierCounter()source_ast.visit(count_visitor)count_visitor.count# Output: 5TransformersTransformers provide an easy way to transform source code. For example, in the following, we want to mirror each binary addition:importcode_astfromcode_astimportASTTransformer,FormattedUpdate,TreeUpdatecode='''def f(x, y):return x + y + 0.5'''# Mirror binary operator on leaveclassMirrorAddTransformer(ASTTransformer):defleave_binary_operator(self,node):ifnode.children[1].type=="+":returnFormattedUpdate("%s+%s",[TreeUpdate(node.children[2]),TreeUpdate(node.children[0])])# Parse the AST and then visit it with our visitorsource_ast=code_ast.ast(code,lang="python")mirror_transformer=MirrorAddTransformer()# Mirror transformer are initialized by running them as visitorssource_ast.visit(mirror_transformer)# Transformer provide a minimal AST editmirror_transformer.edit()# Output:# module [2, 0] - [5, 0]#    function_definition [2, 0] - [3, 22]#        block [3, 4] - [3, 22]#            return_statement [3, 4] - [3, 22]#                binary_operator -> FormattedUpdate [3, 11] - [3, 22]#                    binary_operator -> FormattedUpdate [3, 11] - [3, 16]# And it can be used to directly transform the codemirror_transformer.code()# Output:# def f(x, y):#    return 0.5 + y + xProject InfoThe goal of this project is to provide developer in the
programming language processing community with easy
access to AST parsing. This is currently developed as a helper library for internal research projects. Therefore, it will only be updated
as needed.Feel free to open an issue if anything unexpected
happens.Distributed under the MIT license. SeeLICENSEfor more information.We thank the developer oftree-sitterlibrary. Without tree-sitter this project would not be possible.