Package: codetext
______________________________________________________________________BranchBuildUnittestReleaseLicensemainCode-Text parseris a customtree-sitter's grammar parser for extract raw source code into class and function level. We support 10 common programming languages:PythonJavaJavaScriptPHPRubyRustCC++C#GoInstallationcodetextpackage require python 3.7 or above and tree-sitter. Setup environment and install dependencies manually from source:githttps://github.com/FSoft-AI4Code/CodeText-parser.git;cdCodeText-parser
pipinstall-rrequirement.txt
pipinstall-e.Or install viapypipackage:pipinstallcodetextGetting startedcodetextCLI Usagecodetext[options][PATHorFILE]...For example extract any python file insrc/folder:codetextsrc/--languagePythonIf you want to store extracted class and function, use flag--jsonand give a path to destination file:codetextsrc/--languagePython--output_file./python_report.json--jsonOptionspositionalarguments:pathslistofthefilename/paths.

optionalarguments:-h,--helpshowthishelpmessageandexit--versionshowprogram'sversionnumberandexit-lLANGUAGE,--languageLANGUAGETargettheprogramminglanguagesyouwanttoanalyze.-oOUTPUT_FILE,--output_fileOUTPUT_FILEOutputfile(e.greport.json).--jsonGeneratejsonoutputasatransformofthedefaultoutput--verbosePrintprogressbarExampleFile circle_linkedlist.py analyzed:
==================================================
Number of class    : 1
Number of function : 2
--------------------------------------------------

Class summary:
+-----+---------+-------------+
|   # | Class   | Arguments   |
+=====+=========+=============+
|   0 | Node    |             |
+-----+---------+-------------+

Class analyse: Node
+-----+---------------+-------------+--------+---------------+
| #   | Method name   | Paramters   | Type   | Return type   |
+=====+===============+=============+========+===============+
| 0   | __init__      | self        |        |               |
|     |               | data        |        |               |
+-----+---------------+-------------+--------+---------------+

Function analyse:
+-----+-----------------+-------------+--------+---------------+
| #   | Function name   | Paramters   | Type   | Return type   |
+=====+=================+=============+========+===============+
| 0   | push            | head_ref    |        | Node          |
|     |                 | data        | Any    | Node          |
| 1   | countNodes      | head        | Node   |               |
+-----+-----------------+-------------+--------+---------------+Usingcodetextas Python moduleBuild your languagecodetextneed tree-sitter language file (i.e.sofile) to work properly. You can manually compile language (see more) or automatically build use our pre-defined function (the<language>.sowill saved in a folder name/tree-sitter/):fromcodetext.utilsimportbuild_languagelanguage='rust'build_language(language)# INFO:utils:Not found tree-sitter-rust, attempt clone from github# Cloning into 'tree-sitter-rust'...# remote: Enumerating objects: 2835, done. ...# INFO:utils:Attempt to build Tree-sitter Language for rust and store in .../tree-sitter/rust.soUsing Language ParserEach programming language we supported are correspond to a customelanguage_parser. (e.g Python isPythonParser()).language_parsertake input as raw source code and use breadth-first search to traveser through all syntax node. The class, method or stand-alone function will then be collected:fromcodetext.utilsimportparse_coderaw_code="""/*** Sum of 2 number* @param a int number* @param b int number*/double sum2num(int a, int b) {return a + b;}"""# Auto parse code into tree-sitter.Treeroot=parse_code(raw_code,'cpp')root_node=root.root_nodeGet all function nodes inside a specific node:fromcodetext.utils.parserimportCppParserfunction_list=CppParser.get_function_list(root_node)print(function_list)# [<Node type=function_definition, start_point=(6, 0), end_point=(8, 1)>]Get function metadata (e.g. function's name, parameters, (optional) return type)function=function_list[0]metadata=CppParser.get_function_metadata(function,raw_code)# {'identifier': 'sum2num', 'parameters': {'a': 'int', 'b': 'int'}, 'type': 'double'}Get docstring (documentation) of a functiondocstring=CppParser.get_docstring(function,code_sample)# ['Sum of 2 number \n@param a int number \n@param b int number']We also provide 2 command for extract class objectclass_list=CppParser.get_class_list(root_node)# andmetadata=CppParser.get_metadata_list(root_node)Limitationscodetextheavly depends on tree-sitter syntax:Since we use tree-sitter grammar to extract desire node like function, class, function's name (identifier) or class's argument list, etc.codetextis easily vulnerable by tree-sitter update patch or syntax change in future.While we try our best to capture all possiblity, there are still plenty out there. We open for community to contribute into this project.