No speed diference when doing prediction between BERT and ALBERT
Author: ayrtondenner
Pull request: no
Labels: name: Discussion

name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

I'm comparing two trained models, one using BERT and the other using ALBERT. I'm using the following code to do a prediction in both models: it tokenizes a list of phrases, padding them according to biggest max length per tokenized phrase, and then it applies a prediction:

```
def padding(phrase_list):
    """
    Add padding to phrases in phrase list
    """
    max_size = 0
    for phrase in phrase_list:
        max_size = max(max_size, len(tokenizer_phrase.encode(phrase)))
    print(f"Max_size: {max_size}")
    padded_list = []
    for phrase in phrase_list:
        phrase_encoded = tokenizer_phrase.encode_plus(phrase, max_length=max_size, pad_to_max_length=True)
        padded_list.append(phrase_encoded)
    return padded_list
```

```
def predict_batch_outputs(phrase_list):
    """
    Receive list of phrases and return model prediction WITHOUT softmax
    """
    with torch.no_grad():
        phrases_padded = padding(phrase_list)
        input_ids = torch.tensor([pad['input_ids'] for pad in phrases_padded])
        token_type_ids = torch.tensor([pad['token_type_ids'] for pad in phrases_padded])
        attention_mask = torch.tensor([pad['attention_mask'] for pad in phrases_padded])
        labels = torch.tensor([[1] for i in range(0, len(input_ids))])
        outputs = model_phrase(input_ids.to('cuda'), token_type_ids = token_type_ids.to('cuda'), attention_mask = attention_mask.to('cuda'), labels = labels.to('cuda'))
    return outputs[1].tolist()
```

My question is: I'm not seeing any difference in speed between two models. For instance, I have a script that read a csv dataset, break every row into a list of phrases using nltk, and then send such frases to a model (either Bert or Albert) and print it prediction. Using the same script, with the same methods, same dataset, and changing only which model is doing the prediction, I have the opposite result than expected: Bert can predict 11.89 docs per second, while Albert can predict 9.13 docs per second. I've done other tests and they also showed Albert being SLOWER than Bert.

Can someone share their experiences between Bert and Albert in matters of speed? Thanks.

Comments:
Hi, there's no reason ALBERT would be faster than BERT. They have the same number of layers, ALBERT just uses repeating layers (its `n` layers are a just a single one) instead of different layers (BERT's `n` layers are `n` different layers).

Hi @LysandreJik !
If ALBERT is a lot smaller than BERT (in terms of parameters), wouldn't it be faster? Taking up less memory, allowing bigger batch sizes, etc. Isn't this "scalability" one of the advantages of ALBERT over BERT?
Thanks!

If training speeds "proportionality" is similar to inferencing, then  Section 4.3 and Table 2, page 7 of the latest Albert paper, [https://arxiv.org/pdf/1909.11942.pdf](https://arxiv.org/pdf/1909.11942.pdf) compares  "Speedup" of BERT & ALBERT models.  For example, Albert_xxlarge "speed of data throughput" is 0.3x of BERT_large the baseline, so only 30% the throughput.

> If training speeds "proportionality" is similar to inferencing, then Section 4.3 and Table 2, page 7 of the latest Albert paper, https://arxiv.org/pdf/1909.11942.pdf compares "Speedup" of BERT & ALBERT models.

Interesting. Section 4.3 also says:

>ALBERT models have higher data throughput compared to their corresponding BERT models. If we
use BERT-large as the baseline, we observe that ALBERT-large is about 1.7 times faster in iterating
through the data while ALBERT-xxlarge is about 3 times slower because of the larger structure.

Judging from this Section and from Table 2, some versions of ALBERT are indeed faster than BERT.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.