Package: tree-hugger
For People in a Hurry :)tree-huggerMine source code repositories at scale. Easily. Tree-hugger is a light-weight, high level library which provides Pythonic APIs  to mine trough Git repositories (it works on any collection of supported code files, actually).Tree-hugger is built on top oftree-sitter.Covered languages:PythonPHPJavaJavaScriptC++System Requirement: Python 3.6ContentsTable of contentsInstallationSetupHello world exampleAPI referenceExtending tree-huggerAdding languagesAdding queriesRoadmapInstallationFrom pip:pip install -U tree-hugger PyYAMLFrom Source:git clone https://github.com/autosoft-dev/tree-hugger.git

cd tree-hugger

pip install -e .The installation process is tested in macOS Mojave, we have aseparate docker bindingfor compiling the libraries for Linux and soon this library will be integrated in that as wellYou may need to install libgit2. In case you are in mac just usebrew install libgit2SetupGetting your .so filesFrom onwards tree-hugger 0.9 we ship a new commanddownload_libs.If you are working on Debian based Linux or Newer version of MacOS then you should probably just use this command to get the library. At any point of time we will maintain a .so file for both those OSs with all the supported languages in it.To get the .so file for your platform you can simply do the followingdownload_libsHere is the full usage guide of the commandusage: download_libs [-h] [--local_file_name LOCAL_FILE_NAME]

optional arguments:
  -h, --help            show this help message and exit
  --local_file_name LOCAL_FILE_NAME
                        The local file where you want to save the library.
                        Default - my-languages.soBuilding the .so filesPlease note that building the libraries has been tested under a macOS Mojave with Apple LLVM version 10.0.1 (clang-1001.0.46.4)Please check out our Linux specific instructionshereOnce this library is installed it gives you a command line utility to download and compile tree-sitter .so files with ease. As an example -create_libs pythonHere is the full usage guide of the commandusage: create_libs [-h] [-c] [-l LIB_NAME] langs [langs ...]

positional arguments:
  langs                 Give the name of languages for tree-sitter (php,
                        python, go ...)

optional arguments:
  -h, --help            show this help message and exit
  -c, --copy-to-workspace
                        Shall we copy the created libs to the present dir?
                        (default: False)
  -l LIB_NAME, --lib-name LIB_NAME
                        The name of the generated .so fileEnvironment variablesYou can set upTS_LIB_PATHenvironment variable for the tree-sitter lib path and then the libary will use them automatically. Otherwise, as an alternative, you can pass it when creating anyParserobject.Hello world exampleGenerate the librairies: run the above command to generate the libraries.In our settings we use the-cflag to copy the generatedtree-sitterlibrary's.sofile to our workspace. Once copied, we place it under a directory calledtslibs(It is in the .gitignore).⚠ If you are using linux,you will need to use ourtree-sitter-dockerimage and manually copy the final .so file. Unless you are in a debian based distro and in that case you should probably use our pre-compiled version viadownload_libscommand as described aboveSetup environment variable(optional)
Assuming that you have the necessary environment variable setup. The following line of code will create aParserobject according to the language you want to analyse:Python# Pythonfromtree_hugger.coreimportPythonParserpp=PythonParser()pp.parse_file("tests/assets/file_with_different_functions.py")pp.get_all_function_names()Out[4]:['first_child','second_child','say_whee','wrapper','my_decorator','parent']PHP# PHPfromtree_hugger.coreimportPHPParserphpp=PHPParser()phpp.parse_file("tests/assets/file_with_different_functions.php")phpp.get_all_function_names()Out[5]:['foo','test','simple_params','variadic_param']Java# Javafromtree_hugger.coreimportJavaParserjp=JavaParser()jp.parse_file("tests/assets/file_with_different_methods.java")jp.get_all_class_names()Out[6]:['HelloWorld','Animal','Dog']JavaScript# JavaScriptfromtree_hugger.coreimportJavascriptParserjsp=JavascriptParser()jsp.parse_file("tests/assets/file_with_different_functions.js")jsp.get_all_function_names()Out[7]:['test','utf8_to_b64','sum','multiply']C++fromtree_hugger.coreimportCPPParsercp=CPPParser()cp.parse_file("tests/assets/file_with_different_functions.cpp")cp.get_all_function_names()Out[8]:['foo','test','simple_params','variadic_param']API referenceLanguageFunctionsMethodsClassesPythonget_all_function_names get_all_function_doctrings  get_all_function_names_and_params  get_all_function_bodiesget_all_class_methods  get_all_class_method_docstrings  get_all_class_method_bodiesget_all_class_names  get_all_class_docstringsPHPget_all_function_names  get_all_function_names_and_params   get_all_function_bodiesget_all_class_methodsget_all_class_namesJavaget_all_class_methods   get_all_method_names_and_params  get_all_method_bodiesget_all_class_namesJavaScriptget_all_function_names  get_all_function_names_and_params  get_all_function_bodiesget_all_class_methodsget_all_class_namesC++get_all_function_names  get_all_function_names_and_params  get_all_function_bodiesget_all_class_methodsget_all_class_namesExtending tree-huggerExtending tree-hugger for other languages and/or more functionalities for the already provided ones, is easy.Adding languages:Parsed languages can be extended through adding a parser class from the BaseParser class. The only mandatory argument that a Parser class should pass to the parent is thelanguage. This is a string. Such aspython(lower case). Each parser class must have the options to take in the path of the tree-sitter library (.so file that we are using to parse the code) and the path to the queries yaml file, in their constructor.The BaseParser class can do few things:Loading and preparing the .so file with respect to the language you just mentioned.Loading, preparing and parsing the query yaml file. (for the queries, we internally use an extended UserDict class)Providing an API to parse a file and prepare it for query.BaseParser.parse_fileIt also gives you another (most likely not to be exposed outside) API_run_query_and_get_captureswhich lets you run any queries and return back the matched results (if any) from the parsed tree.We use those APIs once we have calledparse_fileand parsed the file.Adding queries:Queries processed on source code are s-expressions, they are listed in aqueries.ymlfile for each parser class. Tree-hugger gives you a way to write your queries in yaml file for each language parsed.Query structure: A name of a query followed by the query itself. Written as an s-expression.Example:all_function_docstrings:
        "
        (
            function_definition
            name: (identifier) @function.def
            body: (block(expression_statement(string))) @function.docstring
        )
        "You have to follow yaml grammar while writing these queries. You can see a bit more about writng these queries in thedocumentation of tree-sitter.Some example queries, that you will find in the yaml file (and their corresponding API from the PythonParser class) -* all_function_names => get_all_function_names()

* all_function_docstrings => get_all_function_documentations()

* all_class_methods => get_all_class_method_names()RoadmapDocumentation: tutorial on queries writingWrite *Parser class for other languagesLanguagesStatus-FinishedAuthorPython✅ShubhadeepPHP✅ClémentJava✅ClémentJavaScript✅ClémentC++✅ClémentIf you are using tree-hugger in your project, please consider puttingin your prject :)