# LIMIT-BERT Code and model for the *EMNLP 2020 Findings* paper: [LIMIT-BERT: Linguistic Informed Multi-task BERT](https://arxiv.org/abs/1910.14296)) ## Contents 1. [Requirements](#Requirements) 2. [Training](#Training) ## Requirements * Python 3.6 or higher. * Cython 0.25.2 or any compatible version. * [PyTorch](http://pytorch.org/) 1.0.0+. * [EVALB](http://nlp.cs.nyu.edu/evalb/). Before starting, run `make` inside the `EVALB/` directory to compile an `evalb` executable. This will be called from Python for evaluation. * [pytorch-transformers](https://github.com/huggingface/pytorch-transformers) PyTorch 1.0.0+ or any compatible version. #### Pre-trained Models (PyTorch) The following pre-trained models are available for download from Google Drive: * [`LIMIT-BERT`](https://drive.google.com/open?id=1fm0cK2A91iLG3lCpwowCCQSALnWS2X4i): PyTorch version, same setting with BERT-Large-WWM,loading model with [pytorch-transformers](https://github.com/huggingface/pytorch-transformers). ## How to use ``` from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("cooelf/limitbert") model = AutoModel.from_pretrained("cooelf/limitbert") ``` Please see our original repo for the training scripts. https://github.com/cooelf/LIMIT-BERT ## Training To train LIMIT-BERT, simply run: ``` sh run_limitbert.sh ``` ### Evaluation Instructions To test after setting model path: ``` sh test_bert.sh ``` ## Citation ``` @article{zhou2019limit, title={{LIMIT-BERT}: Linguistic informed multi-task {BERT}}, author={Zhou, Junru and Zhang, Zhuosheng and Zhao, Hai}, journal={arXiv preprint arXiv:1910.14296}, year={2019} } ```