Longformer inference speed is slower than bert of the same length
Author: henrycCoder
Pull request: no
Labels: name: Migration

I used bert-base as the basic model and retrained my long-bert with a length of 1536. Then I compared the difference in inference speed of the original bert-base-1536. After a lot of testing, I found that long-bert-1536 and bert-base-1536 are basically the same in inference speed. I see a similar problem [https://github.com/allenai/longformer/issues/106] , but the length of my test data is all greater than 1000. I think window attention should be faster than self-attention because the amount of calculation is smaller, but why does this problem occur? Here are some settings:
attention windows (each layer is the same): 512
Global attention: only used for cls token
Inference device: cpu
task: text classification

By the way, does the size of the attention window affect the speed of inference? I tested different window sizes, but the speed is basically the same.

Comments:
Hey @chenlin038,

Can you copy paste the configs of `bert-base-1536` and `long-bert-1536` below? This way I can see exactly which configs you are using. Otherwise, I also can only refer to the answers in https://github.com/allenai/longformer/issues/106 .

> Hey @chenlin038,
> 
> Can you copy paste the configs of `bert-base-1536` and `long-bert-1536` below? This way I can see exactly which configs you are using. Otherwise, I also can only refer to the answers in [allenai/longformer#106](https://github.com/allenai/longformer/issues/106) .

Sorry for the late reply！This is the config of ### bert-base-1536:
{
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "directionality": "bidi",
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 1536,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "type_vocab_size": 2,
  "vocab_size": 21128
}

The following is the config of ### long-bert-1536:
{
  "architectures": [
    "BertLongForSequenceClassification"
  ],
  "attention_probs_dropout_prob": 0.1,
  "attention_window": [
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512,
    512
  ],
  "directionality": "bidi",
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 1536,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "type_vocab_size": 2,
  "vocab_size": 21128
}


Ok, yeah I'm not very surprised that `bert-base-1536` is faster in your case. Longformer should mainly be used to prevent out-of-memory problems for long sequences. To do so it uses a complex attention mechanism than BERT which makes it a bit slower (especially for shorter sequences). So, in your case I would expect the Longformer model to use less memory than the BERT model, but not necessarily to be faster. If `bert-base-1536` fits in memory, then I think it's a good idea to use the model

> Ok, yeah I'm not very surprised that `bert-base-1536` is faster in your case. Longformer should mainly be used to prevent out-of-memory problems for long sequences. To do so it uses a complex attention mechanism than BERT which makes it a bit slower (especially for shorter sequences). So, in your case I would expect the Longformer model to use less memory than the BERT model, but not necessarily to be faster. If `bert-base-1536` fits in memory, then I think it's a good idea to use the model

Does the longformer use less memory only have an effect on the GPU, or even specific GPU types, such as Nvidia Ampere? If I use CPU as a inference device, can I save more memory? Is there currently any optimization for the storage or calculation of sparse matrices by the CPU?

You should definitely see an improvement in CPU memory usage when using longformer! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hi, It seems I have a similar issue. However, the time difference between Longformer and Roberta is nearly a factor 10. Does it seem normal? After checking it is in the longformer.encoder step that this is quite slow. Here is the config I use (I have similar factor whatever the config):
LongFormer_TOY_MODEL_HPARAMS = {
    "vocab_size": len(LongFormer_VOCAB),
    "hidden_size": 64,
    "num_hidden_layers": 3,
    "num_attention_heads": 8,
    "intermediate_size": 32,
    "hidden_act": "gelu",
    "hidden_dropout_prob": 0.1,
    "attention_probs_dropout_prob": 0.1,
    "max_position_embeddings": 512
    + 2,  # tokenizer's model_max_length + 2 (
 / 
 tokens of sequence)
    "initializer_range": 0.02,
    "layer_norm_eps": 1e-12,
    "attention_window": 512
}



Thanks!