Package: code-tokenize
Fast tokenization and structural analysis of
any programming language in PythonProgramming Language Processing (PLP) brings the capabilities of modern NLP systems to the world of programming languages.
To achieve high performance PLP systems, existing methods often take advantage of the fully defined nature of programming languages. Especially the syntactical structure can be exploited to gain knowledge about programs.code.tokenizeprovides easy access to the syntactic structure of a program. The tokenizer converts a program into a sequence of program tokens ready for further end-to-end processing.
By relating each token to an AST node, it is possible to extend the program representation easily with further syntactic information.InstallationThe package is tested under Python 3. It can be installed via:pip install code-tokenizeUsagecode.tokenize can tokenize nearly any program code in a few lines of code:importcode_tokenizeasctok# Pythonctok.tokenize('''def my_func():print("Hello World")''',lang="python")# Output: [def, my_func, (, ), :, #NEWLINE#, ...]# Javactok.tokenize('''public static void main(String[] args){System.out.println("Hello World");}''',lang="java",syntax_error="ignore")# Output: [public, static, void, main, (, String, [, ], args), {, System, ...]# JavaScriptctok.tokenize('''alert("Hello World");''',lang="javascript",syntax_error="ignore")# Output: [alert, (, "Hello World", ), ;]Supported languagescode.tokenize employstree-sitteras a backend. Therefore, in principal, any language supported by tree-sitter is also
supported by a tokenizer in code.tokenize.For some languages, this library supports additional
features that are not directly supported by tree-sitter.
Therefore, we distinguish between three language classes
and support the following language identifier:native: pythonadvanced: javabasic: javascript, go, ruby, cpp, c, swift, rust, ...Languages in thenativeclass support all features
of this library and are extensively tested.advancedlanguages are tested but do not support the full feature set. Languages of thebasicclass are not tested and
only support the feature set of the backend. They can still be used for tokenization and AST parsing.How to contributeYour language is not natively supported by code.tokenize or the tokenization seems to be incorrect?Then change it!While code.tokenize is developed mainly as an helper library for internal research projects, we welcome pull requests of any sorts (if it is a new feature or a bug fix).Want to help to test more languages?Our goal is to support as many languages as possible at anativelevel. However, languages onbasiclevel are completly untested. You can help by testingbasiclanguages and reporting issues in the tokenization process!Release history0.2.0Major API redesign!CHANGE: AST parsing is now done by an external library:code_astCHANGE: Visitor pattern instead of custom tokenizerCHANGE: Custom visitors for language dependent tokenization0.1.0The first proper releaseCHANGE: Language specific tokenizer configurationCHANGE: Basic analyses of the program structure and token roleCHANGE: Documentation0.0.1Work in progressProject InfoThe goal of this project is to provide developer in the
programming language processing community with easy
access to program tokenization and AST parsing. This is currently developed as a helper library for internal research projects. Therefore, it will only be updated
as needed.Feel free to open an issue if anything unexpected
happens.Distributed under the MIT license. SeeLICENSEfor more information.This project was developed as part of our research related to:@inproceedings{richter2022tssb,title={TSSB-3M: Mining single statement bugs at massive scale},author={Cedric Richter, Heike Wehrheim},booktitle={MSR},year={2022}}We thank the developer oftree-sitterlibrary. Without tree-sitter this project would not be possible.