Package: codeprep
CodeprepThis is a tool for preprocessing source code corpora according to a specified vocabulary modeling choice.Supported modeling choices are:Splitting algorithm (no identifier splitting, camel-case splitting, snake-case splitting, BPE (byte-pair-encoding),
number-splitting, ronin:http://joss.theoj.org/papers/10.21105/joss.00653);Number of merges if using BPE;Ignoring/preserving string literals;Ignoring/preserving comments;Preserving case/lowercasing;Preserving/ignoring newlines and tabs.applying/not applying stemming after basic splittingGetting startedMake sure you have python >= 3.6 installed in your system; pip, setuptools and wheel are up to date.python--version
python-mpipinstall--upgradepipsetuptoolswheelInstallcodepreplib:pipinstallcodeprepIn order to run theroninalgorithm, you will have to additionally install Spiral module (https://github.com/casics/spiral/):pipinstallgit+https://github.com/casics/spiral.gitThe tool can be usedas a python libraryas well as a standalone module runnable with aCLI.
You can pass the path to the dataset or the text itself to be preprocessed. When using Python API for the former option
you need to import methods fromcodeprep.api.textmodule, for the latter - fromcodeprep.api.corpus.
Below you can see the general patterns of usage.Python API>>>importcodeprep.api.textascp>>>cp.<commmand>('Some code to be split')>>>importcodeprep.api.corpusascp>>>cp.<commmand>('/path/to/the/dataset')CLIcodeprep<commmand>"Some code to be split"codeprep<commmand>--path/path/to/the/datasetHereafter we will demonstrate the usage as a python library. The CLI is analogous to the python API. You can find the documentation about how to use ithere.Usage examplesBasic splittingTokenization + CamelCase- and snake_case- splitting:>>>importcodeprep.api.textascp>>>input_code='''void test_WordUeberraschungPrinter() {...     if (eps >= 0.345e+4) { // FIXME...         printWord("     ...     Überraschung");...     }... }'''>>>cp.basic(input_code)['void','<w>','test','_','Word','Ueberraschung','Printer','</w>','(',')','{','\n','\t','if','(','eps','>','=','0','.','<w>','345','e','</w>','+','4',')','{','/','/','FIXME','\n','\t','\t','<w>','print','Word','</w>','(','"','\t','.','.','.','\t','Überraschung','"',')',';','\n','\t','}','\n','}']Tokenize but don't split identifiers>>>importcodeprep.api.textascp>>>input_code='''void test_WordUeberraschungPrinter() {...     if (eps >= 0.345e+4) { // FIXME...         printWord("     ...     Überraschung");...     }... }'''>>>cp.nosplit(input_code)['void','test_WordUeberraschungPrinter','(',')','{','\n','\t','if','(','eps','>','=','0','.','345e','+','4',')','{','/','/','FIXME','\n','\t','\t','printWord','(','"','\t','.','.','.','\t','Überraschung','"',')',';','\n','\t','}','\n','}']BPE (Byte-Pair encoding)The following code doescamelCase-andsnake_case-splitting and appliesbpe with 10k mergeson top:>>>importcodeprep.api.textascp>>>input_code='''void test_WordUeberraschungPrinter() {...     if (eps >= 0.345e+4) { // FIXME...         printWord("     ...     Überraschung");...     }... }'''>>>cp.bpe(input_code,bpe_codes_id='10k')['v','oid</t>','test_','Word','U','eb','err','as','ch','un','g','Print','er</t>','(</t>',')</t>','{</t>','\n','\t','i','f</t>','(</t>','e','ps</t>','></t>','=</t>','0</t>','.</t>','34','5','e</t>','+</t>','4</t>',')</t>','{</t>','/</t>','/</t>','FIX','M','E</t>','\n','\t','\t','print','Word</t>','(</t>','"</t>','\t','.</t>','.</t>','.</t>','\t','Ü','b','err','as','ch','un','g</t>','"</t>',')</t>',';</t>','\n','\t','}</t>','\n','}</t>']codeprepby default does BPE using bpe codes leaned onthe Github Java Corpus. The argumentbpe_codes_id='10k'tells thecodepreptool to use 10,000 bpe merges.
Other possible values are1kand5k(1,000 and 5,000 merges respectively). Please refer to sectionLearning custom BPE codesto train custom bpe codes.For other commands and options likechars,--split-numbers,--ronin,--stem, please refer to thedocs.Calculate vocabularySetcalc_vocabparam toTruewhen calling a preprocessing method to calculate the vocabulary of the preprocessed corpus, e.g.:>>>importcodeprep.api.corpusascp>>>cp.basic('/path/to/train/on',calc_vocab=True)...Vocabisavailableat/path/to/vocabLearning custom BPE codesIf you don't want to use, pre-trained BPE codes, it's possible to train custom ones. For example, to train 10,000 merges on the corpus located at the path/path/to/train/on, the following command should be run (only CLI):codepreplearn-bpe10000-p/path/to/train/on--idcustom-bpe-codesNow it is possible to do bpe splitting by running the bpe command with the number of merges from 0 to 10,000 (for example with 3500 merges):codeprepbpecustom-bpe-codes-3500-p/path/to/preprocessBefore bpe codes are trained, thebasic preprocessingis done, which can also be tuned with arguments described in sectionTweaking preprocessing.Additional optionsTweaking preprocessingYou can pass the following parameters with aTruevalue (default values for all of them are False), to tweak the way the imput is preprocessed:no_str- replace strings with  placeholders.no_com- replace comments with  placeholders.no_spaces- remove newlines and tabs.no_unicode- replace words containing non-ascii characters with  placeholders.no_case- lowercase words and encode information about case in   tokens.>>>importcodeprep.api.textascp>>>input_code='''void test_WordUeberraschungPrinter() {...     if (eps >= 0.345e+4) { // FIXME...         printWord("     ...     Überraschung");...     }... }'''>>>cp.basic(input_code,no_spaces=True,no_unicode=True,no_case=True,no_com=True,no_str=True)['void','<w>','test','_','<Cap>','word','<Cap>','ueberraschung','<Cap>','printer','</w>','(',')','{','if','(','eps','>','=','0','.','<w>','345','e','</w>','+','4',')','{','/','/','<CAPS>','fixme','<w>','print','<Cap>','word','</w>','(','"','.','.','.','<Cap>','<non-en>','"',')',';','}','}']Similar params can be specified as switches--no-str,--no-com,--no-spaces,--no-unicode,--no-casein CLI commands.Specifying the languageUnless explicitely specified,codeprepwill assume the language is java. To make sure the input is preprocessed as intended, it is alwayshighly recommendedto specify it:importcodeprep.api.textascp>>>cp.bpe("volatile",'1k')['volatile']>>>cp.bpe("volatile",'1k',extension="py")['v','ol','a','ti','le</t>']# Since 'volatile' is a keyword in java, it is represented as one token unlike in python# where it is pretty rare when used as an identifier and therefore represented as multiple subtokens.When preprocessing a corpus,codeprepidentifies the language based on the file extension. If you want only files with (a) certain extension(s) to be preprocessed, you can specify --ext paramcodeprepbasic--path/path/to/be/preprocessed--ext"java"# or if you want to pre-process multiple types of files:codeprepbasic--path/path/to/be/preprocessed--ext"java|c|py|js"MiscellaneousYou can specify the path to where the preprocessed corpus will be written:codeprepbasic--path/path/to/preprocess--output-path/path/to/outputTo print logs with log level DEBUG and higher to stdout:codeprepbasic--path/path/to/preprocess--verboseGetting HelpTo get help on commands and options:codeprep--helpPaperThis library was build to run experiments for our paper accepted at ICSE 2020:Big Code != Big Vocabulary: Open-Vocabulary Models for Source CodeIf you you the library or the results, please cite the paper:@article{karampatsis2020big,
 title={Big Code!= Big Vocabulary: Open-Vocabulary Models for Source Code},
 author={Karampatsis, Rafael-Michael and Babii, Hlib and Robbes, Romain and Sutton, Charles and Janes, Andrea},
 journal={arXiv preprint arXiv:2003.07914},
 year={2020}
}AdvancedCachingWhen preprocessing a dataset,codeprepfirst parses source code and converts it into internal representation,
which is after that converted to a preprocessed dataset depending on provided parameters. The intermediate
representation is cached, so that when the same dataset is pre-processed again with different parameters,codeprep(providing no changes have been made to the dataset) would use the cache rather than parsing
the source code again.To store the cache,codeprepuses a directory speecified by$XDG_CACHE_HOME/codeprep/<codeprep_version>variable if its value is set,$HOME/.cache/codeprep/<codeprep_version>otherwise.Removing the cache will not change the final result, however, will result in slower pre-processing.Releases1.0.3Add more flixibility with versions of dependencies1.0.1Fix training custom bpe codes (Thanks to @mir-am)Fix corpus pre-processing on Windows1.0.0DOI assigned1.0.0-alpha.12Bugfixes and minor improvements1.0.0-alpha.11 (NOT backward compatible with 1.0.0-alpha.10)Include token types in the metadataExpand on token type hierarchyMake possible to return full token index in the iterator1.0.0-alpha.10 (NOT backward compatible with 1.0.0-alpha.9)Add boundaries of comments to pre-processing metadataAdd Windows and OSx supportSwitch from unittest to pytest+doctestBugfixes related to literal presentation of tokens on the diskBugfixes related to adding  to mark the end of a full token1.0.0-alpha.9 (NOT backward compatible with 1.0.0-alpha.7)Addget_corpus_size()method toPreprocessedCorpusclassDo BPE splitting without splitting by convention firstUse  to mark the last sub-token of a tokenReplacing non-ascii sequences with a special charFollow symlinks when reading a datasetmake possible to preserve case when doing stemmingBugfixes1.0.0-alpha.7 (NOT backward compatible with 1.0.0-alpha.6)Store version incodeprep.__version__implement--full-stringsand--max-str-lengthoptionsreplaceroninmethod/command wit--roninoption and apply ronin algorithm on word level instead of full identifier levelifsplit_numbersoption is set toTrue, split numbers not only in code but also in strings and commentschange placeholder values to more human-readableimprove logging displayingBugfixes1.0.0-alpha.6Initial PyPI release