Translation takes too long - from fine-tuned mbart-large-50 model
Author: aloka2209
Pull request: no

I have fine-tuned from, facebook mbart-large-50 for Si-En language pairs. When I try to translate 1950 sentences as (1) full batch (2) batch size=16 etc. still process crashes.

I then passed 16-lines per batch, ie. as **src_lines** and it takes considerable time.

Could you help on how I can reduce the translation time? My code is as follows.

Highly appreciate your help.

**However from the fairseq fine-tuned checkpoint the entire file can be translated in 2 mints in the same machine.**

tokenizer = MBart50TokenizerFast.from_pretrained("mbart50-ft-si-en-run4", src_lang="si_LK", tgt_lang="en_XX")
model_inputs = tokenizer(src_lines, padding=True, truncation=True, max_length=100, return_tensors="pt")

generated_tokens = model.generate(
**model_inputs,
forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])

trans_lines=tokenizer.batch_decode(generated_tokens, skip_special_tokens=True) #crashes

Comments:
Hello! Could you provide a reproducible code example so that we may take a look?

from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

#mbart50-ft-si-en-run4 the fine-tuned model directory
tokenizer = MBart50TokenizerFast.from_pretrained("mbart50-ft-si-en-run4", src_lang="si_LK", tgt_lang="en_XX")
model = MBartForConditionalGeneration.from_pretrained("mbart50-ft-si-en-run4/checkpoint-21500")

#The file has 1950 lines
src_lines=[line.strip() for line in open('data/parallel-27.04.2021-tu.un.si-en-ta.si', 'r', encoding='utf8')]

model_inputs = tokenizer(src_lines, padding=True, truncation=True, max_length=100, return_tensors="pt")
generated_tokens = model.generate(
                                            **model_inputs,
                                            forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])

trans_lines=tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)

Please refer the code I am trying as above.
The above code works when a subset of about 10 examples are given. ie. len(src_lines)=10.
I am having trouble translating the full batch. 


Hi @aloka2209 

Thank you for opening the issue. Could you please provide more details? 
- Are you using GPU ? fairseq would use GPU if it's available, we need to do this manually with Transformers models.
- Is the fairseq model in fp16 ? By default Transformers models are in `fp32`
- what are the generation params, i.e num_beams, `max_length` etc ? both for fairseq model and for Transformers models.
 For transformer models, you could find these values in `config`

Hi @patil-suraj 

Thank you for your attention regarding this issue.

For the same training, tuning and testing sets I have conducted a fine-tuning from mbart-large-50 pre-trained model for Si-En languages, using fairseq and huggingface.

### Fairseq fine-tuning
The preprocessing, fine-tuning & generation parameter settings have been followed according to example in [github](https://github.com/pytorch/fairseq/tree/579a48f4be3876082ea646880061a98c94357af1/examples/multilingual).

I have used the gpu for fine-tuning with  fp16, max-tokens 1024  as parameters.
Generation command as follows:
cat data-spm/test_si_en.bpe.si_LK \
      | fairseq-interactive $path_2_data \
      --path $model \
      --task translation_multi_simple_epoch \
      --lang-dict  $lang_list \
      --lang-pairs $lang_pairs \
      --source-lang $source_lang \
      --target-lang $target_lang \
      --batch-size 32 \
      --remove-bpe 'sentencepiece' \
      --buffer-size 32 \
      --encoder-langtok 'src' \
      --decoder-langtok \
    > 'data/test_si_en.stdout.si_LK_en_XX'

To translate 1950 Si lines the following times are taken.
gpu - 3 mints

### Huggingface fine-tuning

From the model hub I obtained the facebook/mbart-large-50 model and used the following code to fine tune for Si-En languages. The above training, validation, testing sets were used.

> 
> model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50")
> tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50", src_lang="si_LK", tgt_lang="en_XX")
> 
> #initialized model parameters
> args = Seq2SeqTrainingArguments(output_dir='mbart50-ft-si-en-run4',
>                         do_train=True,
>                         do_eval=True,
>                         evaluation_strategy="epoch",
>                         per_device_train_batch_size=16,
>                         per_device_eval_batch_size=16,
>                         learning_rate=2e-5,
>                         weight_decay=0.01,
>                         save_total_limit=12,                        
>                         num_train_epochs=120,
>                         predict_with_generate=True,
>                         save_steps=500)
> 
> 
> trainer = Seq2SeqTrainer(model=model, 
>                 args=args, 
>                 data_collator=data_collator, 
>                 train_dataset=train_data, 
>                 eval_dataset=valid_data,
>                 compute_metrics=compute_metrics,
>                 tokenizer=tokenizer)
> 
> 
> trainer.train()

Few things I need to confirm with fine-tuning
- Are the training hyper parameters I have used are correct?
- I have used the fp16 during hf fine-tuning
- Although I have given the max_length=100 in config.json file its recorded as "max_length": 200. Is 200 a default value?
- I have set the num_train_epochs=120 as the fairseq best checkpoint obtained was at epoch 120
- gradient_accumulation_steps=16. My training set is 56K lines. Since my batch size is 16, should this value be 3500 (56K/16)?

My generation command as follows. 

> from transformers import MBartForConditionalGeneration, MBart50TokenizerFast
> 
> #mbart50-ft-si-en-run4 the fine-tuned model directory
> tokenizer = MBart50TokenizerFast.from_pretrained("mbart50-ft-si-en-run4", src_lang="si_LK", tgt_lang="en_XX")
> model = MBartForConditionalGeneration.from_pretrained("mbart50-ft-si-en-run4/checkpoint-21500")
> 
> 
> model_inputs = tokenizer(src_lines, padding=True, truncation=True, max_length=100, return_tensors="pt")
> generated_tokens = model.generate(
> **model_inputs,
> forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])
> 
> trans_lines=tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)

I have tried to pass the entire batch as above. Then the task crashes

Afterwards I tried with trans_lines=tokenizer.batch_decode(generated_tokens, 
 batch_size=32, skip_special_tokens=True). But this takes a long time. So I have stopped it.

Highly appreciate your help on improving the fine-tuning and/or generation. Many Thanks.







Hi, 

Please report one issue at a time, as the original issue is about `generation`, let's talk about that :)

Also it would be nice if you could please format code using [markdown code syntax ](https://docs.github.com/en/github/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks), otherwise the code is hard to read.

Looking at the generation code snippet it seems that the model is not on GPU, to put the model on GPU which could explain the slowdown.

```python
model = MBartForConditionalGeneration.from_pretrained("mbart50-ft-si-en-run4/checkpoint-21500").to("cuda")
```

Also, you are passing all 2000 examples at once to `tokenizer` and `model`, which could again explain why tokenization is low. If you look at the `fairseq` command, it accepts a `batch_size` argument and does generation for one batch at a time.

If you pass all 2000 examples at once it might OOM on GPU. You could instead create a `Dataset` and `DataLoader` do generation for each batch. 

Hope this helps :)

I have added the fine-tuning related questions to the forum and would appreciate your answers.  Sorry for getting everything here.

I have used the gpu command as suggested and now it translates fast. Thank you!

One of my intentions is to get this translated in cpu setting. So earlier I tried by feeding the inputs in 16-line batches and then by changing the batch_size parameter during generation. Still, it was taking a long time.
 
ie. 
```python
trans_lines=tokenizer.batch_decode(generated_tokens, batch_size=32, skip_special_tokens=True) 
```
Are there any suggestions where I can still cut-down the translation time in cpu? 




This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.