Package: genienlp
Genie NLP libraryThis library contains the NLP models for theGenietoolkit for
virtual assistants. It is derived from thedecaNLPlibrary by Salesforce,
but has diverged significantly.The library is suitable for all NLP tasks that can be framed as Contextual Question Answering, that is, with 3 inputs:text or structured input ascontexttext input asquestiontext or structured output asanswerAs the work byMcCann et al.shows, many NLP tasks can be framed in this way.
Genie primarily uses the library for paraphrasing, translation, semantic parsing, and dialogue state tracking, and this is
what the models work best for.Installationgenienlp is available on PyPi. You can install it with:pip3installgenienlpAfter installation,genienlpcommand becomes available.UsageTraining a semantic parserThe general form is:genienlptrain--tasksalmond--train_iterations50000--data<datadir>--save<model_dir><flags>The<datadir>should contain a single folder called "almond" (the name of the task). That folder should
contain the files "train.tsv" and "eval.tsv" for train and dev set respectively.To train a BERT-LSTM (or other MLM-based model) use:genienlptrain--tasksalmond--train_iterations50000--data<datadir>--save<model_dir>\--modelTransformerLSTM--pretrained_modelbert-base-cased--trainable_decoder_embedding50To train a BART or other Seq2Seq model, use:genienlptrain--tasksalmond--train_iterations50000--data<datadir>--save<model_dir>\--modelTransformerSeq2Seq--pretrained_modelfacebook/bart-large--gradient_accumulation_steps20The default batch sizes are tuned for training on a single V100 GPU. Use--train_batch_tokensand--val_batch_sizeto control the batch sizes. Seegenienlp train --helpfor the full list of options.NOTE: the BERT-LSTM model used by the current version of the library is not comparable with the
one used in our published paper (cited below), because the input preprocessing is different. If you
wish to compare with published results you should use genienlp <= 0.5.0.Inference on a semantic parserIn batch mode:genienlppredict--tasksalmond--data<datadir>--path<model_dir>--eval_dir<output>The<datadir>should contain a single folder called "almond" (the name of the task). That folder should
contain the files "train.tsv" and "eval.tsv" for train and dev set respectively. The result of batch prediction
will be saved in<output>/almond/valid.tsv, as a TSV file containing ID and prediction.In interactive mode:genienlpserver--path<model_dir>Opens a TCP server that listens to requests, formatted as JSON objects containingid(the ID of the request),task(the name of the task),contextandquestion. The server writes out JSON objects containingidandanswer. The server listens to port 8401 by default, use--portto specify a different port or--stdinto
use standard input/output instead of TCP.Calibrating a trained modelCalibrate the confidence scores of a trained model:Calcualate and save confidence features of the evaluation set in a pickle file:genienlppredict--taskalmond--data<datadir>--path<model_dir>--save_confidence_features--confidence_feature_path<confidence_feature_file>Train a boosted tree to map confidence features to a score between 0 and 1:genienlpcalibrate--confidence_path<confidence_feature_file>--save<calibrator_directory>--name_prefix<calibrator_name>Now if you provide--calibrator_pathsduring prediction, it will output confidence scores for each output:genienlppredict--tasksalmond--data<datadir>--path<model_dir>--calibrator_paths<calibrator_directory>/<calibrator_name>.calibParaphrasingTrain a paraphrasing model:genienlptrain-paraphrase--train_data_file<train_data_file>--eval_data_file<dev_data_file>--output_dir<model_dir>--model_typegpt2--do_train--do_eval--evaluate_during_training--logging_steps1000--save_steps1000--max_steps40000--save_total_limit2--gradient_accumulation_steps16--per_gpu_eval_batch_size4--per_gpu_train_batch_size4--num_train_epochs1--model_name_or_path<gpt2/gpt2-medium/gpt2-large/gpt2-xlarge>Generate paraphrases:genienlprun-paraphrase--model_name_or_path<model_dir>--temperature0.3--repetition_penalty1.0--num_samples4--batch_size32--input_file<input_tsv_file>--input_column1Named Entity DisambiguationFirst run a bootleg model to extract mentions, entity candidates, and contextual embeddings for the mentions.genienlpbootleg-dump-features--train_tasks<train_task_names>--save<savedir>--preserve_case--data<dataset_dir>--train_batch_tokens400--val_batch_size400--database_typejson--database_dir<database_dir>--ned_featurestype_idtype_prob--ned_features_size11--ned_features_default_val01.0--num_workers0--min_entity_len1--max_entity_len4--bootleg_model<bootleg_model>This command generates several output files. In<dataset_dir>you should see aprepdir which contains preprocessed data (e.g. data converted to memory-mapped format, several array to facilitate embedding lookup etc.) If your dataset doesn't change you can reuse the same files.
It will also generate several files in <results_temp> folder. Ineval_bootleg/[train|eval]/<bootleg_model>/bootleg_lables.jsonlyou can see the examples, mentions, predicted candidates and their probabilities according to bootleg.Now you can use the extracted features from bootleg in downstream tasks such as semantic parsing to improve named entity understanding and consequently generation:genienlptrain--train_tasks<train_task_names>--train_iterations60000--preserve_case--save<savedir>--data<dataset_dir>--modelTransformerLSTM--pretrained_modelbert-base-uncased--trainable_decoder_embeddings50--train_batch_tokens1000--val_batch_size1000--do_ned--database_typejson--database_dir<database_dir>--ned_retrieve_methodbootleg--ned_featurestype_idtype_prob--ned_features_size11--ned_features_default_val01.0--num_workers0--min_entity_len1--max_entity_len4--bootleg_model<bootleg_model>Seegenienlp --helpandgenienlp <command> --helpfor more details about each argument.CitationIf you use the MultiTask Question Answering model in your work, please citeThe Natural Language Decathlon: Multitask Learning as Question Answering.@article{McCann2018decaNLP,title={The Natural Language Decathlon: Multitask Learning as Question Answering},author={Bryan McCann and Nitish Shirish Keskar and Caiming Xiong and Richard Socher},journal={arXiv preprint arXiv:1806.08730},year={2018}}If you use the BERT-LSTM model (Identity encoder + MQAN decoder), please citeSchema2QA: High-Quality and Low-Cost Q&A Agents for the Structured Web@InProceedings{xu2020schema2qa,title={{Schema2QA}: High-Quality and Low-Cost {Q\&A} Agents for the Structured Web},author={Silei Xu and Giovanni Campagna and Jian Li and Monica S. Lam},booktitle={Proceedings of the 29th ACM International Conference on Information and Knowledge Management},year={2020},doi={https://doi.org/10.1145/3340531.3411974}}If you use the paraphrasing model (BART or GPT-2 fine-tuned on a paraphrasing dataset), please citeAutoQA: From Databases to QA Semantic Parsers with Only Synthetic Training Data@inproceedings{xu-etal-2020-autoqa,title="{A}uto{QA}: From Databases to {QA} Semantic Parsers with Only Synthetic Training Data",author="Xu, Silei  and Semnani, Sina  and Campagna, Giovanni  and Lam, Monica",booktitle="Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",month=nov,year="2020",address="Online",publisher="Association for Computational Linguistics",url="https://www.aclweb.org/anthology/2020.emnlp-main.31",pages="422--434",}If you use MarianMT/ mBART/ T5 for translation task, or XLMR-LSTM model for Seq2Seq tasks, please citeLocalizing Open-Ontology QA Semantic Parsers in a Day Using Machine Translationand the original paper that introduced the model.@inproceedings{moradshahi-etal-2020-localizing,title="Localizing Open-Ontology {QA} Semantic Parsers in a Day Using Machine Translation",author="Moradshahi, Mehrad and Campagna, Giovanni and Semnani, Sina and Xu, Silei and Lam, Monica",booktitle="Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)",month=November,year="2020",address="Online",publisher="Association for Computational Linguistics",url="https://www.aclweb.org/anthology/2020.emnlp-main.481",pages="5970--5983",}