Package: belt-nlp
BELT(BERT ForLongerTexts)Project description and motivationThe BELT approachThe BERT model can process texts of the maximal length of 512 tokens (roughly speaking tokens are equivalent to words). It is a consequence of the model architecture and cannot be directly adjusted. Discussion of this issue can be foundhere. Method to overcome this issue was proposed by Devlin (one of the authors of BERT) in the previously mentioned discussion:comment. The main goal of our project is to implement this method and allow the BERT model to process longer texts during prediction and fine-tuning. We dub this approach BELT (BERT ForLongerTexts).More technical details are described in thedocumentation. We also prepared the comprehensive blog post:part 1,part 2.Attention is all you need, but 512 words is all you haveThe limitations of the BERT model to the 512 tokens come from the very beginning of the transformers models. Indeed, the attention mechanism, invented in the groundbreaking 2017 paperAttention is all you need, scales quadratically with the sequence length. Unlike RNN or CNN models, which can process sequences of arbitrary length, transformers with the full attention (like BERT) are infeasible (or very expensive) to process long sequences.
To overcome the issue, alternative approaches with sparse attention mechanisms were proposed in 2020:BigBirdandLongformer.BELT vs. BigBird vs. LongFormerLet us now clarify the key differences between the BELT approach to fine-tuning and the sparse attention models BigBird and Longformer:The main difference is that BigBird and Longformers are not modified BERTs. They are models with different architectures. Hence, they need to be pre-trained from scratch or downloaded.This leads to the main advantage of the BELT approach - it uses any pre-trained BERT or RoBERTa models. A quick look at the HuggingFace Hub confirms that there are about 100 times more resources forBERTthan forLongformer. It might be easier to find the one appropriate for the specific task or language.On the other hand, we have not done any benchmark tests yet. We believe that the comparison of the BELT approach with the models with sparse attention might be very instructive. Some work in this direction was done in the 2022 paperExtend and Explain: Interpreting Very Long Language Models. The authors cited our implementation under the former nameroberta_for_longer_texts. We encourage more research in this direction.Installation and dependenciesThe project requires Python 3.9+ to run. We recommend training the models on the GPU. Hence, it is necessary to installtorchversion compatible with the machine. The version of the driver depends on the machine - first, check the version of GPU drivers by the commandnvidia-smiand choose the newest version compatible with these drivers according tothis table(e.g.: 11.1). Then we installtorchto get the compatible build.Here, we find which torch version is compatible with the CUDA version on our machine.Another option is to use the CPU-only version of torch:pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpuNext, we recommend installing via pip:pip3 install belt-nlpIf you want to clone the repo in order to run tests or notebooks, you can use therequirements.txtfile.Model classesTwo main classes are implemented:BertClassifierTruncated- base binary classification model, longer texts are truncated to 512 tokensBertClassifierWithPooling- extended model for longer texts (more details in thedocumentation)InterfaceThe main methods are:fit- fine-tune the model to the training set, use the list of raw texts and labelspredict_classes- calculate the list of classifications for the given list of raw texts. The model must be fine-tuned before that.predict_scores- calculate the list of probabilities for the given list of raw texts. The model must be fine-tuned before that.Loading the pre-trained modelAs a default, the standard Englishbert-base-uncasedmodel is used as a pre-trained model. However, it is possible to use any Bert or Roberta model. To do this, use the parameterpretrained_model_name_or_path.
It can be either:a string with the name of a pre-trained model configuration to download from huggingface library, e.g.:roberta-base.a path to a directory with the downloaded model, e.g.:./my_model_directory/.TestsTo make sure everything works properly, run the commandpytest tests -rA. As a default, during tests, models are trained on small samples on the CPU.Examplesfit and predict method for base modelfit and predict method for model with poolingContributorsThe project was created atMIM AIby:Michał BrzozowskiMarek WachnickiIf you want to contribute to the library, see thecontributing info.Version historySeeCHANGELOG.md.LicenseSee theLICENSEfile for license rights and limitations (MIT).For MaintainersFilerequirements.txtcan be updated using the command:bash pip-freeze-without-torch.sh > requirements.txtThis script saves all dependencies of the current active environment excepttorch.In order to add the next version of the package to pypi, do the following steps:First, increment the package version inpyproject.toml.Then build the new version: runpython3.9 -m buildfrom the main folder.Finally, upload to pypi:twine upload dist/*(two newly created files).