SqueezeBERT
Source: transformers

# SqueezeBERT## OverviewThe SqueezeBERT model was proposed in [SqueezeBERT: What can computer vision teach NLP about efficient neural networks?](https://arxiv.org/abs/2006.11316) by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, Kurt W. Keutzer. It's abidirectional transformer similar to the BERT model. The key difference between the BERT architecture and theSqueezeBERT architecture is that SqueezeBERT uses [grouped convolutions](https://blog.yani.io/filter-group-tutorial)instead of fully-connected layers for the Q, K, V and FFN layers.The abstract from the paper is the following:*Humans read and write hundreds of billions of messages every day. Further, due to the availability of large datasets,large computing systems, and better neural network models, natural language processing (NLP) technology has madesignificant strides in understanding, proofreading, and organizing these messages. Thus, there is a significantopportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. In particular, weconsider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today'shighly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, withBERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methodssuch as grouped convolutions have yielded significant speedups for computer vision networks, but many of thesetechniques have not been adopted by NLP neural network designers. We demonstrate how to replace several operations inself-attention layers with grouped convolutions, and we use this technique in a novel network architecture calledSqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE testset. The SqueezeBERT code will be released.*This model was contributed by [forresti](https://huggingface.co/forresti).## Usage tips- SqueezeBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right  rather than the left.- SqueezeBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore  efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained  with a causal language modeling (CLM) objective are better in that regard.- For best results when finetuning on sequence classification tasks, it is recommended to start with the  *squeezebert/squeezebert-mnli-headless* checkpoint.## Resources- [Text classification task guide](../tasks/sequence_classification)- [Token classification task guide](../tasks/token_classification)- [Question answering task guide](../tasks/question_answering)- [Masked language modeling task guide](../tasks/masked_language_modeling)- [Multiple choice task guide](../tasks/multiple_choice)## SqueezeBertConfig[[autodoc]] SqueezeBertConfig## SqueezeBertTokenizer[[autodoc]] SqueezeBertTokenizer    - build_inputs_with_special_tokens    - get_special_tokens_mask    - create_token_type_ids_from_sequences    - save_vocabulary## SqueezeBertTokenizerFast[[autodoc]] SqueezeBertTokenizerFast## SqueezeBertModel[[autodoc]] SqueezeBertModel## SqueezeBertForMaskedLM[[autodoc]] SqueezeBertForMaskedLM## SqueezeBertForSequenceClassification[[autodoc]] SqueezeBertForSequenceClassification## SqueezeBertForMultipleChoice[[autodoc]] SqueezeBertForMultipleChoice## SqueezeBertForTokenClassification[[autodoc]] SqueezeBertForTokenClassification## SqueezeBertForQuestionAnswering[[autodoc]] SqueezeBertForQuestionAnswering