Package: ylab-yato
YATO: Yet Another deep learning based Text analysis Open toolkitQuick LinksIntroductionGetting StartedData FormatConfiguration PreparationTraining ConfigurationDataloaderModelHyperparametersDecode ConfigurationPerformanceResults for sequence labeling tasks.Results for sequence classification tasks.Add Handcrafted FeaturesSpeedN best DecodingText Attention Heatmap VisualizationReproduce Paper Results and Hyperparameter TuningReport Issue or ProblemCiteFuture PlanUpdatesIntroductionYATO, an open-source Python library for text analysis. In particular,YATOfocuses on sequence labeling and sequence classification tasks, including extensive fundamental NLP tasks such as part-of-speech tagging, chunking, NER, CCG super tagging, sentiment analysis, and sentence classification.YATOcan design both specific RNN-based and Transformer-based through user-friendly configuration and integrating the SOTA pre-trained language models, such as BERT.YATOis a PyTorch-based framework with flexible choices of input features and output structures. The design of neural sequence models withYATOis fully configurable through a configuration file, which does not require any code work.Its previous version calledNCRF++has been accepted as a demo paper by ACL 2018. The in-depth experimental report based onNCRF++was accepted as the best paper by COLING 2018.Welcome to star this repository!Getting StartedWe provide an easy way to use the toolkitYATOfrom PyPIpipinstallyatoOr directly install it from the source  codegit clone https://github.com/jiesutd/YATO.gitThe code to train a ModelfromyatoimportYATOmodel=YATO(configurationfile)model.train()The code to decode prediction files:from yato import YATO
decode_model = YATO(configuration file)
result_dict = decode_model.decode()return dictionary contents following value:speed: decoding speedaccuracy: If the decoded file contains annotation results, accuracy means verifying the accuracyprecision:  If the decoded file contains annotation results, precision means verifying the precisionrecall:  If the decoded file contains annotation results, recall means verifying the recallpredict_result: predicted resultnbest_predict_score: nbest scores of decoded predictionlabel: Mapping between labels and indexesData FormatRefersample_datafor the detailed data format.YATOsupports both BIO and BIOES(BMES) tag schemes.Notice that IOB format (differentfrom BIO) is currently not supported, because this tag scheme is old and works worse than other schemesReimers and Gurevych, 2017.The differences among these three tag schemes are explained in thispaper.We provided ascriptwhich supports convertation of tag scheme among IOB/BIO/BIOES. Welcome to have a try.Configuration PreparationYou can specify the model, optimizer, and decoding through the configuration file:Training ConfigurationDataloadertrain_dir=the path of the train filedev_dir=the path of the validation filetest_dir=the path of the test filemodel_dir=the path to save model weightsdset_dir=the path of configuration encode fileModeluse_crf=True/Falseuse_char=True/Falsechar_seq_feature=GRU/LSTM/CNN/Falseuse_word_seq=True/Falseuse_word_emb=True/Falseword_emb_dir=The path of word embedding fileword_seq_feature=GRU/LSTM/CNN/FeedFowrd/Falselow_level_transformer=pretrain language model from huggingfacelow_level_transformer_finetune=True/Falsehigh_level_transformer=pretrain language model from huggingfacehigh_level_transformer_finetune=True/Falsecnn_layer=layer numberchar_hidden_dim=dimension numberhidden_dim=dimension numberlstm_layer=layer numberbilstm=True/FalseHyperparameterssentence_classification=True/Falsestatus=train/decodedropout=Dropout Rateoptimizer=SGD/Adagrad/adadelta/rmsprop/adam/adamwiteration=epoch numberbatch_size=batch sizelearning_rate=learning rategpu=True/Falsedevice=cuda:0scheduler=get_linear_schedule_with_warmup/get_cosine_schedule_with_warmupwarmup_step_rate=warmup steo rateDecode Configurationstatus=decoderaw_dir=The path of decode filenbest=0 (NER)/1 (sentence classification)decode_dir=The path of decode result fileload_model_dir=The path of model weightssentence_classification=True/FalsePerformanceFor multiple sequence labeling and sequence classification tasks, YATO has reproduced or outperformed the reported SOTA results on majority datasets.By default, theLSTMis a bidirectional LSTM. TheBERT-baseis huggingface's bert-base-uncased. TheRoBERTa-baseis huggingface's roberta-base. TheELECTRA-baseis huggingface's google/electra-base-discriminator.Results for sequence labeling tasks.IDModelCoNLL2003OntoNotes 5.0MSRAOntonotes 4.0CCG1CCNN+WLSTM+CRF91.0081.5392.8374.5593.802BERT-base91.6184.6895.8180.5796.143RoBERTa-base90.2386.2896.0280.9496.164ELECTRA-base91.5985.2596.0390.4796.29Results for sequence classification tasks.IDModelSST2SST5ChnSentiCorp1CCNN+WLSTM87.6143.4888.222BERT-base93.0053.4895.863RoBERTa-base92.5551.9996.044ELECTRA-base94.7255.1195.96For more details, you can refer to our papers mentioned below.We have compared twelve neural sequence labeling models ({charLSTM, charCNN, None} x {wordLSTM, wordCNN} x {softmax, CRF}) on three benchmarks (POS, Chunking, NER) under statistical experiments, detailed results and comparisons can be found in our COLING 2018 paperDesign Challenges and Misconceptions in Neural Sequence Labeling.The results based on Pretrain Language Model were recorded inYATO: Yet Another deep learning based\ Text analysis Open toolkitAdd Handcrafted FeaturesYATOhas integrated several SOTA neural character sequence feature extractors: CNN (Ma .etc, ACL16), LSTM (Lample .etc, NAACL16) and GRU (Yang .etc, ICLR17). In addition, hand-crafted features have been proven to be important in sequence labeling tasks.YATOsupports users designing their features such as Capitalization, POS tag, or any other features (grey circles in the above figure). Users can configure the self-defined features through a configuration file (feature embedding size, pretrained feature embeddings .etc). The sample of input format is given attrain.cappos.bmes, which includes two hand-crafted features[POS]and[Cap]. ([POS]and[Cap]are two examples, you can set your feature any name you want, just follow the format[xx]and configure the feature with the same name in the configuration file.)
Users can configure each feature in configuration file by usingfeature=[POS]emb_size=20emb_dir=%your_pretrained_POS_embeddingfeature=[Cap]emb_size=20emb_dir=%your_pretrained_Cap_embeddingThe feature without pretrained embedding will be randomly initialized.SpeedYATOis implemented using a fully batch computing approach, making it quite efficient in both model training and decoding.With the help of GPU (Nvidia RTX 2080ti) and large batches, models built withYATOcan be decoded efficiently.N best DecodingThe traditional CRF structure decodes only one label sequence with the largest probabilities (i.e. 1-best output). In contrast,YATOcan decodenlabel sequences with the topnprobabilities (i.e. n-best output). The nbest decoding has been supported by several popularstatisticalCRF frameworks. To the best of our knowledge,YATOis the only and the first toolkit which supports nbest decoding inneuralCRF models.In our implementation, the model built inYATOcan improve the F1-score by 5.7%-6.8% in the CoNLL 2003 NER task when nbest=10.Text Attention Heatmap VisualizationYATOtakes the list of words and the corresponding weights as input to generate Latex code for visualizing the attention-based result.The Latex code will generate a separate .pdf visualization file.For example,fromyatoimportYATOfromutilsimporttext_attentionmodel=YATO(decodeconfigurationfile)sample=["a fairly by-the-books blend of action and romance with sprinklings of intentional and unintentional comedy . ||| 1"]probsutils,weights_ls=model.attention(input_text=sample)sentece="a fairly by-the-books blend of action and romance with sprinklings of intentional and unintentional comedy . "atten=weights_ls[0].tolist()text_attention.visualization(sentece,atten[0],tex='sample.tex',color='red')Reproduce Paper Results and Hyperparameter TuningTo reproduce the results in our COLING 2018 paper, you only need to set theiteration=1asiteration=100in the configuration filedemo.train.configand then configure your file directory.The default configuration file adopts theChar CNN + Word LSTM + CRFmodel, and you can develop your model by modifying the configuration accordingly. The parameters in this demo configuration file are the same as our paper. (Notice theWord CNNrelated models need slightly adjust parameters, details can be found in our COLING paper.)If you want to use this framework in new tasks or datasets, here are some tuningtipsby @Victor0118.Report Issue or ProblemIf you want to report an issue or ask a problem, please attach the following materials if necessary. With these information, we can provide a fast and accurate discussion and the corrsponding suggestions.log fileconfig filesample dataCiteIf you useNCRF++in your paper, please cite ourACL demo paper:@inproceedings{yang2018ncrf,
title={**YATO**: An Open-source Neural Sequence Labeling Toolkit},
author={Yang, Jie and Zhang, Yue},
booktitle={Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics},
 Url = {http://aclweb.org/anthology/P18-4013},
 year={2018}
}If you use experiment results and analysis ofNCRF++, please cite ourCOLING paper:@inproceedings{yang2018design,
title={Design Challenges and Misconceptions in Neural Sequence Labeling},
author={Yang, Jie and Liang, Shuailong and Zhang, Yue},
booktitle={Proceedings of the 27th International Conference on Computational Linguistics (COLING)},
 Url = {http://aclweb.org/anthology/C18-1327},
 year={2018}
}Future PlanDocument classification (working)Support API usageUpload trained model on Word Segmentation / POS tagging / NERUpdates2022-May-14  YATO, init version2020-Mar-06, dev version, sentence classification, framework change, model saved in one file.2018-Dec-17,YATOv0.2, support PyTorch 1.02018-Mar-30,YATOv0.1, initial version2018-Jan-06, add result comparison.2018-Jan-02, support character feature selection.2017-Dec-06, init version