Package: critical-path
critical-path-nlpTools for adapting universal language models to specifc tasksPlease note these are tools for rapid prototyping - not brute force hyperparameter tuning.Adapted from:Google's BERTInstallationClone repo (just for now - will be on pip soon)Download a pretrainedBERT model- start withBERT-Base Uncasedif you're not sure where to beginUnzip the model and make note of the pathExamplesFull implementation examples can be found here:SQuAD exampleMulti-Label Classification exampleSingle-Label Classification exampleCurrent CapabilitiesBERT for Question AnsweringTrain and evaluate the SQuAD datasetSQuAD 2.0 - Stanford Question Answering DatasetBERT for Multi-Label ClassificationTrain and evaluate custom datasets for multi-label classification tasks (multiple labels possible)Kaggle - Google Toxic Comment Classification ChallangeBERT for Single-Label ClassificationTrain and evaluate custom datasets for single-label classification tasks (one label possible)CoLA - Corpus of Linguistic AcceptabilityMPRC - Microsoft Research Paraphrase CorpusMNLI - Multi-Genre Natural Language Inferenceetc.Future CapabilitiesGPT-2 Training and GenerationUsage + Core ComponentsConfiguring BERTFirst, define the model pathsbase_model_folder_path="../models/uncased_L-12_H-768_A-12/"# Folder containing downloaded Base Modelname_of_config_json_file="bert_config.json"# Located inside the Base Model foldername_of_vocab_file="vocab.txt"# Located inside the Base Model folderoutput_directory="../models/trained_BERT/"# Trained model and results landing folder# Multi-Label and Single-Label Specificdata_dir=None# Directory .tsv data is stored in - typically for CoLA/MPRC or other datasets with known structureSecond, define the model run parameters"""Settable parameters and their default valuesNote: Most default values are perfectly fine"""# Administrativeinit_checkpoint=Nonesave_checkpoints_steps=1000iterations_per_loop=1000do_lower_case=True# Technicalbatch_size_train=32batch_size_eval=8batch_size_predict=8num_train_epochs=3.0max_seq_length=128warmup_proportion=0.1learning_rate=3e-5# SQuAD Specificdoc_stride=128max_query_length=64n_best_size=20max_answer_length=30is_squad_v2=False# SQuAD 2.0 has examples with no answer, aka "impossible", SQuAD 1.0 does notverbose_logging=Falsenull_score_diff_threshold=0.0Initialize the configuration handlerfromcritical_path.BERT.configsimportConfigClassifierFlags=ConfigClassifier()Flags.set_model_paths(bert_config_file=base_model_folder_path+name_of_config_json_file,bert_vocab_file=base_model_folder_path+name_of_vocab_file,bert_output_dir=output_folder_path,data_dir=data_dir)Flags.set_model_params(batch_size_train=8,max_seq_length=256,num_train_epochs=3)# Retrieve a handle for the configsFLAGS=Flags.get_handle()A single 1070GTX using BERT-Base Uncased can handleModelmax_seq_lenbatch_sizeBERT-Base Uncased2566...3844For full batch size and sequence length guidelines see Google'srecommendationsUsing Configured ModelFirst, create a new model with the configured parameters"""For Multi-Label Classification"""fromcritical_path.BERT.model_multilabel_classimportMultiLabelClassifiermodel=MultiLabelClassifier(FLAGS)Second, load your data sourceSQuAD has dedicated dataloadersread_squad_examples(), write_squad_predictions()in/BERT/model_squadMulti-Label Classification has a generic dataloaderDataProcessorin/BERT/model_multilabel_classNote:This requires data labels to be in string formatlabels=[["label_1","label_2","label_3"],["label_2"]]Single-Label Classification dataloadersColaProcessoris implemented in/BERT/model_classifierMore dataloader formats have been done bypytorch-pretrained-BERT"""For Multi-Label Classification with a custom .csv reading function"""fromcritical_path.BERT.model_multilabel_classimportDataProcessor# read_data is dataset specifc - see /bert_multilabel_example.pyinput_ids,input_text,input_labels,label_list=read_toxic_data(randomize=True)processor=DataProcessor(label_list=label_list)train_examples=processor.get_samples(input_ids=input_ids,input_text=input_text,input_labels=input_labels,set_type='train')Third, run your task"""Train and predict a Multi-Label Classifier"""ifdo_train:model.train(train_examples,label_list)ifdo_predict:model.predict(predict_examples,label_list)For full examples please see:Full implementations:SQuAD exampleMulti-Label Classification exampleSingle-Label Classification example