Finetuned Donut model taking too much time on local machine for inference , around 5 minutes.
Author: shubh1608
Pull request: no

Finetuned Donut model is taking **4 minutes 37 seconds** for inference on my local windows laptop which has **16GB RAM and 4 cores**. However, inference time is under **5 seconds on a google colab CPU machine**, it has 32GB RAM. On Colab GPU, the inference time is under a second.

**_Why it's taking too much time on my local Windows machine?_** seems like it's not a normal behavior. Could someone help and guide me on what could be wrong here?

I am using **Transformers Version: 4.28.1**, it's the same on my windows machine as well.

Also, below is the prediction function which I am using and it's the model.generate method which it taking time. 

```
def run_prediction(image):
    pixel_values = processor(image, return_tensors="pt").pixel_values
    outputs = model.generate(
        pixel_values.to(device),
        decoder_input_ids=decoder_input_ids.to(device),
        max_length=model.decoder.config.max_position_embeddings,
        early_stopping=True,
        pad_token_id=processor.tokenizer.pad_token_id,
        eos_token_id=processor.tokenizer.eos_token_id,
        use_cache=True,
        num_beams=1,
        bad_words_ids=[[processor.tokenizer.unk_token_id]],
        return_dict_in_generate=True)
    
    sequence = processor.batch_decode(outputs.sequences)[0]
    sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
    sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
    return processor.token2json(sequence)
```

Comments:
Hi @shubh1608, thanks for raising this issue. 

Could you share the following information, so that we can best help you: 
* Checkpoint of the Donut model being used
* Environment being run (locally and on Colab). Copy-paste the output of `transformers-cli env` run in the terminal
* Expanded snippet to allow for full reproduction. In particular showing how the model and processor are loaded and how to code is being timed. 
 

Hi @amyeroberts, please find below the requested details:

* model checkpoint - shubh1608/donut_pdf_ocr
* Colab CPU environment 

        - `transformers` version: 4.28.1
        - Platform: Linux-5.10.147+-x86_64-with-glibc2.31
        - Python version: 3.9.16
        - Huggingface_hub version: 0.13.4
        - Safetensors version: not installed
        - PyTorch version (GPU?): 2.0.0+cu118 (False)
        - Tensorflow version (GPU?): 2.12.0 (False)
        - Flax version (CPU?/GPU?/TPU?): 0.6.8 (cpu)
        - Jax version: 0.4.8
        - JaxLib version: 0.4.7
        - Using GPU in script?: no
        - Using distributed or parallel set-up in script?: no

* Local Windows environment

        - `transformers` version: 4.28.1
        - Platform: Windows-10-10.0.19045-SP0
        - Python version: 3.9.16
        - Huggingface_hub version: 0.13.4
        - Safetensors version: not installed
        - PyTorch version (GPU?): 1.12.1 (False)
        - Tensorflow version (GPU?): not installed (NA)
        - Flax version (CPU?/GPU?/TPU?): not installed (NA)
        - Jax version: not installed
        - JaxLib version: not installed
        - Using GPU in script?: no
        - Using distributed or parallel set-up in script?: no

* Expanded code snippet. NOTE: I have cloned the model repo locally and loaded weights from there.

        model_processor_path = '../model-weights/donut/donut_pdf_ocr'
        processor = DonutProcessor.from_pretrained(model_processor_path)
        model = VisionEncoderDecoderModel.from_pretrained(model_processor_path)
        device = "cuda" if torch.cuda.is_available() else "cpu"
        model.to(device)
        # prepare decoder inputs
        task_prompt = "
"
        decoder_input_ids = processor.tokenizer(task_prompt, add_special_tokens=False, return_tensors="pt").input_ids
        
        def run_prediction(file):
          image = Image.open(file).convert('RGB')
          pixel_values = processor(image, return_tensors="pt").pixel_values
          outputs = model.generate(
              pixel_values.to(device),
              decoder_input_ids=decoder_input_ids.to(device),
              max_length=model.decoder.config.max_position_embeddings,
              early_stopping=True,
              pad_token_id=processor.tokenizer.pad_token_id,
              eos_token_id=processor.tokenizer.eos_token_id,
              use_cache=True,
              num_beams=1,
              bad_words_ids=[[processor.tokenizer.unk_token_id]],
              return_dict_in_generate=True)
          
          sequence = processor.batch_decode(outputs.sequences)[0]
          sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
          sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
          return processor.token2json(sequence)

Let me know if you need any more information for debugging. 

Thanks.

Guys, any update on this?

Hey @shubh1608 I guess there has not been any updates for quite a while, sorry about that! 
It's a bit hard for us to debug this as we need to find why your computer is basically slow. Would be great if you can tell us more about the specs of your compute. You mentioned that: 
> windows laptop which has 16GB RAM and 4 cores

You might have a lot of RAM but super slow CPU. What comes to me is also your torch version, which is `1.12` compared to `2.0`, a LOT of improvements are thus missing on your local setup. Could you try updating torch? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.