Inference is slow with 
Author: simonefrancia
Pull request: no
Labels: name: wontfix

Hi @julien-c,
I know that @loretoparisi already talked with you guys about some problems we have with inference using BertTokenizer and BertForSequenceClassification Classes.
So for starting to discuss about that, we would like to show you the most important lines of our inference and training code, that probably can have some bugs and slowdowns.

Briefly, we are developing a language classifier trained on 93 languages, so we started from multilingual bert ( both tokenizer and model ) and then we want to finetune it on our dataset that has about 8M lines.

# INFERENCE CODE
```
# tokenize entire text
    tokens = tokenizer.encode(original_text.strip())
    # remove bos and eos tokens 
    tokens = tokens[1:-1]

    # get number of slices we have to insert in DL model
    number_of_slices = len(tokens) // (MAX_SEQUENCE_LENGTH - 2)
    if len(tokens) % (MAX_SEQUENCE_LENGTH - 2) != 0: number_of_slices +=1

    # create slices to be inserted
    slices = []
    for index in range(number_of_slices):
        slice_ = tokens[ index*(MAX_SEQUENCE_LENGTH - 2) : (index+1)*(MAX_SEQUENCE_LENGTH - 2)]
        slice_ = [tokenizer.bos_token_id] + slice_ + [tokenizer.eos_token_id]
        slices.append(slice_)


    # for every slice, preprocess data creating mask and padding
    texts = []
    masks = []

    for text in slices:
        padding = [tokenizer.pad_token_id] * (MAX_SEQUENCE_LENGTH - len(text))
        mask = torch.zeros(MAX_SEQUENCE_LENGTH, dtype=torch.int32).tolist()
        mask[:len(text)] = [1]*len(text)
        text = text + padding
        texts.append(text)
        masks.append(mask)
    
    # texts to tensor pytorch
    texts = torch.tensor(texts)
    # masks to tensor pytorch
    masks = torch.tensor(masks)

    #inference from DL model
    logits = model(texts, attention_mask=masks)

    # stack logits
    logits = torch.stack(logits).mean(dim=0)
    #sum logits in order to have a single logits
    logits = torch.sum(logits, dim=0)
```

# TRAINING CODE
```
tokenizer = BertTokenizer.from_pretrained( os.path.join(data_path, 'model') )

    special_tokens_dict = {'eos_token': '[CLS]', 
                            'unk_token' : '[UNK]',
                            'eos_token' : '[SEP]', 
                            'bos_token' : '[CLS]', 
                            'pad_token' : '[PAD]' }

    num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)

    train_loader, validation_loader = load_datasets(data_path,
                                                    tokenizer, 
                                                    batch_size,
                                                    max_sequence_length, 
                                                    random_sequence_length, 
                                                    epoch_size,
                                                    token_dropout, 
                                                    seed)

    model = BertForSequenceClassification.from_pretrained(os.path.join(data_path, 'model'))

    print("Num_labels:")
    print(model.num_labels)
    
    if torch.cuda.is_available():
        model.cuda()

    if rank == 0:
        summary(model)
        if distributed():
            dist.barrier()

    if world_size > 1:
        model = DistributedDataParallel(model, [rank], output_device=rank, find_unused_parameters=True)

    optimizer = Adam(model.parameters(), lr=learning_rate, weight_decay=weight_decay)
    epoch_loop = count(1) if max_epochs is None else range(1, max_epochs + 1)

    logdir = os.environ.get("LOGDIR", "logs")
    os.makedirs(logdir, exist_ok=True)

    from torch.utils.tensorboard import SummaryWriter
    writer = SummaryWriter(logdir) if rank == 0 else None
    best_validation_accuracy = 0

    for epoch in epoch_loop:

        try:

            if world_size > 1:
                train_loader.sampler.set_epoch(epoch)
                validation_loader.sampler.set_epoch(epoch)

            train_metrics = train(model, optimizer, device, train_loader, f'Epoch {epoch}')
            validation_metrics = validate(model, device, validation_loader)
```

Do you have some suggestion about that our inference is slow and loading model is very long?
We do not use hub actually, we pre download model on local disk and we load with local path.
Thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.