Package: giganticode-langmodels
langmodelsThis is a repository forneural language models (LMs)trained on a large corpus of source code
and a toolkit to work with such models.You could be interested in using this library if you want to:Use existing pre-trained models for tasks such as autocompletion and bug prediction;Use the pre-trained models for transfer transfer learning or further fine-tuning;Train a model from scratch by choosing one of the wide range of corpus preprocessing choices,
neural network (NN) architectures, and training options.This project usesfastaiandpytorchlibraries for NN training/inference.
For corpus preprocessinggiganticode-dataprepis used.Quick startPrerequisitesPython version >= 3.6 required!Installationpipinstallgiganticode-langmodelsOR to build from source:git clone https://github.com//giganticode/langmodels
cd langmodels
python -m venv langmodels-venv
source langmodels-venv/bin/activate
pip install -r requirements.txtUsing existing pre-trained modelsLoading a default pre-trained model>>>importlangmodels.repositoryasrepo>>>trained_model=repo.load_default_model()[langmodels.repository]INFO:Modelisnotfoundincache.Downloadingfromhttps://www.inf.unibz.it/~hbabii/pretrained_models/langmodel-large-split_10k_2_1024_191007.112241_-_langmodel-large-split_10k_2_1024_191022.141344...[langmodels.model]DEBUG:Loadingmodelfrom:/home/hlib/.local/share/langmodels/0.0.1/modelzoo/langmodel-large-split_10k_2_1024_191007.112241_-_langmodel-large-split_10k_2_1024_191022.141344/best.pth...[langmodels.model]DEBUG:UsingGPUforinferenceOther model loading optionsTo see which models are available, you can calllist_pretrained_modelsfunction.Setcachedparameter toTrue(default isFalse) to display only cached LMs (e.g. if offline).>>>importlangmodels.repositoryasrepo>>>repo.list_pretrained_models(cached=False)IDBPE_MERGESLAYERS_CONFIGARCHBIN_ENTROPYTRAINING_TIME_MINUTES_PER_EPOCHN_EPOCHSBEST_EPOCHTAGSlangmodel-large-split_10k_2_1024_191007.112241_-_langmodel-large-spl10k1024/2/1024AWD_LSTM2.1455788479142965['BEST','DEFAULT']it_10k_2_1024_191022.141344langmodel-large-split_10k_3_1024_191007.112257_-_langmodel-large-spl10k512/3/1024AWD_LSTM2.14730056622143265[]it_10k_3_1024_191022.134822langmodel-large-split_10k_2_2048_191007.112249_-_langmodel-large-spl10k512/2/2048GRU2.19923468325142965[]it_10k_2_2048_191022.141335langmodel-large-split_10k_1_512_190926.12014610k512/1/512AWD_LSTM2.6901949325347998['MEDIUM']langmodel-small-split_10k_1_512_190906.15494310k512/1/512AWD_LSTM4.7376814117241918['TINY']dev_10k_1_10_190923.13232810k10/1/10AWD_LSTM9.1568819109200-1['RANDOM']Usequery_all_modelsmethod to get a list ofModelDescriptionobjects>>>importlangmodels.repositoryasrepo>>>repo.query_all_models()[0]ModelDescription(id='langmodel-large-split_10k_2_1024_191007.112241_-_langmodel-large-split_10k_2_1024_191022.141344',bpe_merges='10k',layers_config='1024/2/1024',arch='AWD_LSTM',bin_entropy=2.1455788479,training_time_minutes_per_epoch=1429,n_epochs=6,best_epoch=5,tags=['BEST','DEFAULT'])A model can be loaded by tag or by id.You can specify if you want to load a model to CPU despite having cuda-supported GPU withforce_use_cpuparameter
(defaults toFalse). If cuda-supported GPU is not available, this parameter is disregarded.>>>trained_model=repo.load_model_with_tag('BEST')>>>trained_model=repo.load_model_by_id('dev_10k_1_10_190923.132328_new',force_use_cpu=True)Also, you can use a lower-level API to load a model by path :trained_model=repo.load_from_path('/home/hlib/.local/share/langmodels/0.0.1/modelzoo/dev_10k_1_10_190923.132328_new')InferenceAutocompletionExample>>>importlangmodels.repositoryasrepo>>>trained_model=repo.load_default_model()>>>trained_model.feed_text('public static main() { if',extension='java')# this does not change the state of the model:>>>predictions=trained_model.predict_next_full_token(n_suggestions=5)[('(',0.9334765834402862),('.',0.01540983953864937),('=',0.008939018331858162),(',',0.005372771784601065),('the',0.00309070517292041)]# adding more context:>>>trained_model.feed_text('(',extension='java')>>>trained_model.predict_next_full_token(n_suggestions=3)[('(',0.14554535082422237),('c',0.018005003646104294),('!',0.01614662429123089)]# resetting the state of the model (make it forget the context)>>>trained_model.reset()>>>trained_model.predict_next_full_token(n_suggestions=5)[('/',0.7209196484717589),('package',0.27093282656897594),('import',0.0007366385365522241),('.',0.0005714365190590807),('public',0.0003926736567296)]Bug prediction based on per-line entropies evaluationAn LM can be used to calculate cross-entropies for each line of a file. High values can give an idea about
unusual/suspicious chunks of code[1].Check sectionLM Evaluationsection to learn how to calculate
cross-entropy for a project/file/string,Check ourvsc pluginfor highlighting suspicious code.Fine-tuning and Transfer learningTBDTraining from scratch (Not supported on OSx)Python API>>>fromlangmodels.training.trainingimporttrain>>>fromlangmodels.lmconfig.datamodelimport*>>>train(LMTrainingConfig(corpus=Corpus(path='/path/to/the/dataset')))More parameters to customize corpus pre-processing, NN architecture, and the training process can be specified:>>>fromlangmodels.training.trainingimporttrain>>>fromlangmodels.lmconfig.datamodelimport*>>>train(LMTrainingConfig(corpus=Corpus(path='/path/to/the/dataset'),prep_function=PrepFunction(options=PrepFunctionOptions(no_com=False,no_unicode=True)),arch=GRUArchj(n_layers=2),training=Training(weight_decay=5e-6)))Below you can see all the default parameters specified explicitly:>>>fromlangmodels.lmconfig.datamodelimport*>>>fromlangmodels.training.trainingimporttrain>>>train(LMTrainingConfig(base_model=None,bs=32,corpus=Corpus(path=os.path.join(HOME,'dataset'),extensions="java"),prep_function=PrepFunction(corpus_api.bpe,['10k'],PrepFunctionOptions(no_com=False,no_unicode=True,no_spaces=True,max_str_length=sys.maxsize)),arch=LstmArch(bidir=False,qrnn=False,emb_sz=1024,n_hid=1024,n_layers=3,drop=Dropouts(multiplier=0.5,oute=0.02,outi=0.25,outh=0.15,w=0.2,out=0.1),tie_weights=True,out_bias=True),bptt=200,training=Training(optimizer=Adam(betas=(0.9,0.99)),files_per_epoch=50000,gradient_clip=0.3,activation_regularization=ActivationRegularization(alpha=2.,beta=1.),schedule=RafaelsTrainingSchedule(init_lr=1e-4,mult_coeff=0.5,patience=0,max_epochs=50,max_lr_reduction_times=6),weight_decay=1e-6)))CLI APITraining can be run from command line as simple as runningtraincommand passing path to the config in json format
as--configparam. To override values in the json file (or default values if--configparam is not specified),
you can use--patchparam.>>langmodelstrain--config="/path/to/json/config.json"--patch="bs=64,arch.drop.multiplier=3.0"If neither--confignor--patchparams are specified, the training will be running with the default parameters.
The json with the default parameters would look like follows:{'arch':{'bidir':False,'drop':{'multiplier':0.5,'out':0.1,'oute':0.02,'outh':0.15,'outi':0.25,'w':0.2},'emb_sz':1024,'n_hid':1024,'n_layers':3,'name':'lstm','out_bias':True,'qrnn':False,'tie_weights':True},'base_model':None,'bptt':200,'bs':32,'config_version':'0.0.3-alpha.0','corpus':{'extensions':'java','path':'/Users/hlib/dataset'},'prep_function':{'callable':'bpe','options':{'max_str_length':9223372036854775807,'no_com':False,'no_spaces':True,'no_str':False,'no_unicode':True},'params':['10k']},'training':{'activation_regularization':{'alpha':2.0,'beta':1.0},'files_per_epoch':50000,'gradient_clip':0.3,'optimizer':{'betas':[0.9,0.99],'name':'Adam'},'schedule':{'init_lr':0.0001,'max_epochs':50,'max_lr_reduction_times':6,'mult_coeff':0.5,'name':'rafael','patience':0},'weight_decay':1e-06}}Most probably, you would have to override at least thecorpus.pathvalue.For more options, run:>>langmodelstrain--helpLM EvaluationWhen training a language model, it is important to be able to evaluate LM's performance.
In this section we describe different ways to do this usinglangmodelslibrary.
You can also use ourtoolto visualize the evaluation.Evaluation on a string / fileFirst, a model can be evaluate on a string withevaluate_model_on_stringmethod. Note that the result may differ a lot depending
on the state of the model. Use methodsresetandfeed_textto reset the model
to initial state and change the context of the model respectively.>>>importlangmodels.repositoryasrepo>>>fromlangmodels.evaluationimportevaluate_model_on_string>>>model=repo.load_default_model()>>>evaluate_model_on_string(model,'public class MyClass {'){full_token_entropy/ParsedToken:EvaluationResult(tokens=['public</t>','class</t>','MyClass</t>','{</t>'],token_types=['KeyWord','KeyWord','SplitContainer','OpeningCurlyBracket'],values=[1.8144783973693848,3.668722629547119,0.5620064437389374,0.2571456730365753],aggregated_value=1.5755882859230042)}Similarly,evaluate_model_on_filewill return a list ofEvaluationobject (1 per each line)Evaluation on a corpusEvaluation can be run on a set of files withevaluate_model_on_pathmethod>>>importlangmodels.repositoryasrepo>>>fromlangmodels.evaluationimportevaluate_model_on_path>>>model=repo.load_default_model()>>>evaluate_model_on_path(model,'/path/to/file')100%|████████████████████████████████████████████████████████████████████████████|28/28[00:11<00:00,2.35it/s]{full_token_entropy/ParsedToken:(5.859160765187885,5745)}Infull_token_entropy/ParsedToken:full_token_entropyis a metric used to evaluate the performance;ParsedTokenmeans that all the tokens were considered when evaluating (See the next section for more details).
Thus, the average full-token-entropy is ~ 5.85 evaluated on 5.7k tokens.Specifying metricsYou can specify based on which metrics the model is to be evaluated.>>>importlangmodels.repositoryasrepo>>>fromlangmodels.evaluationimportevaluate_model_on_path>>>model=repo.load_default_model()>>>evaluate_model_on_path(model,'/path/to/file',metrics={'full_token_entropy','mrr'})Possible metric values arefull_token_entropy,subtoken_entropy,mrr. Default metric set is{full_token_entropy}Release Notes0.0.4-alpha.0 (NOT backward-compatible with 0.0.1-alpha.2)Config datamodel improvements:Add possibility to specify SGD optimizer;Add patience param to training scedule;Add converters between versions of configs;Training:Report binary entropy instead of log-base-e entropy;Save more model metrics (size on disk, trainable params, training time per epoch);Do not save model after every epoch by default;Evaluation improvements:Return token types inEvaluationResult;Add possibility to specify token types to be considered when running evaluation;Trained_model.predict_next_token(): return 1 suggestion by default;Add script for new models upload.0.0.1-alpha.2 (NOT backward-compatible with 0.0.1-alpha.1)Make downloading model from the repository thread-safeForce to specify the extension which corresponds to the type of the code fed into
theTrainedModel.API change:trained_model.feed_text(text: str)->trained_model.feed_text(text: str, extension: str)0.0.1-alpha.1Make methods ofTrainedModelthat change underlying PyTorch model thread-safe0.0.1-alpha.0Initial PyPI releaseReferences[1]Ray, B., Hellendoorn, V., Godhane, S., Tu, Z., Bacchelli, A., & Devanbu, P. (2016, May).
On the" naturalness" of buggy code. In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)
(pp. 428-439). IEEE.