[Whisper] TFWhisperFromPretrained : Can we run transcription by using the call method instead of generate from the transformers class
Author: monowaranjum
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-6.5.0-14-generic-x86_64-with-glibc2.35
- Python version: 3.11.7
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.2+cu121 (False)
- Tensorflow version (GPU?): 2.14.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help?

@sanchit-gandhi @gante @Rocketknight1 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Here is a short script to reproduce the issue.

```
import tensorflow as tf
import numpy as np
from transformers import AutoProcessor, TFWhisperForConditionalGeneration
from datasets import load_dataset
import soundfile as sf
import librosa

processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en")
base_asr_model = TFWhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en")

# Read some inputs 
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
inputs = processor(ds[0]["audio"]["array"], return_tensors="tf")
input_features = inputs.input_features

# Generate some predictions

generated_ids = base_asr_model.generate(input_features=input_features)
transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(transcription)

# Save the model

base_asr_model.save('./whisper-saved-unsigned')

# Load the model from saved state

loaded_base_asr_model = tf.keras.models.load_model('./whisper-saved-unsigned')

# Try running inference on the loaded model
new_generated_ids = loaded_base_asr_model.generate(input_features = input_features) # <-- This won't work
transcription = processor.batch_decode(new_generated_ids, skip_special_tokens=True)[0]
print(transcription)

```

The script fails for the second call of ```generate()``` function with the following error:

```
Traceback (most recent call last):
  File "/home/rashik/Documents/reproduction/reproduction.py", line 31, in <module>
    new_generated_ids = loaded_base_asr_model.generate(input_features = input_features) # <-- This won't work
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'TFWhisperForConditionalGeneration' object has no attribute 'generate'


```

### Expected behavior

I expected the loaded model to behave exactly like the model originally behaved. I listed all the attributes of the loaded model using ```dir(loaded_base_asr_model)```. Here is a screenshot of the output: 

![image](https://github.com/huggingface/transformers/assets/19803082/a16d49a4-1c23-4e07-8cde-1a0fa9a15224)

On the other hand, I did the same for the original model. Here is the screenshot of that output: 

![image](https://github.com/huggingface/transformers/assets/19803082/707710d3-fdbb-461f-9d63-044b0afa95dc)

Clearly, I am missing something about how the model is saved and how it is loaded later. 


Comments:
Hi @monowaranjum, thanks for raising an issue! 

The supported way to save and load models is using the `model.save_pretrained(checkpoint)` and `ModelClass.from_pretrained(checkpoint)` methods. Is there a reason you're using the keras methods instead? 

Hi @monowaranjum, just to expand on @amyeroberts's comment - all `transformers` TF models are Keras models, but not all Keras models are `transformers` models. When you use Keras save methods, what is saved is a pure Keras model, not a `transformers` model. This is okay when you just want to export a model for inference, but you will lose access to methods like `generate()`, which are part of `transformers`, not Keras.

If you want to save and load a `transformers` model, you should use `save_pretrained` and `TFWhisperForConditionalGeneration.from_pretrained()`. We recommend saving pure TF/Keras models only when you want to export the model for inference with e.g. TFLite or something like that.

HI, Thank you so much for the responses. It clears up a few confusions I had. Before I proceed any farther: I would like to give some context as to why I am trying to do so. 

I am trying to compile the whisper model for a specific GPU. For the compilation process to complete, I am using tensorflow and OpenXLA (IREE) compilation toolchain. In order to lower the model in MLIR and then lowering it all the way to LLVM bitcode that can be processed for the specific GPU backend, I need the input model to be in tensorflow saved_model format. So, the process look like this: 

TF Saved model -> Import into MLIR -> Convert to LLVM -> GPU specific Kernel Generation -> GPU executable for inference. 

Now in order to do that, I tried to save the model using keras's save function. 

Q1: I see when I use keras's save function and load it there are still ```call``` methods in it. Does that perform inference on the model?
Q2: If Q1 is answered no, is there a way I can run inference on the keras saved model ? Which function to call for that purpose and where can I find a documentation/worked out example for that. 
Q3: If Q1 is yes, can you please point me to a direction where I can find a worked out example/ documentation on how to use the call function? I am specifically looking for how to set the following parameters: ```decoder_attention_mask, decoder_input_ids```. I tried calling the call function, but could not understand the following error message. 


```
Positional arguments (16 total):
    * <tf.Tensor 'input_features:0' shape=(1, 80, 3000) dtype=float64>
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * False
  Keyword arguments: {}

 Expected these arguments to match one of the following 2 option(s):

Option 1:
  Positional arguments (16 total):
    * {'decoder_attention_mask': TensorSpec(shape=(None, None), dtype=tf.int32, name='decoder_attention_mask'),
 'decoder_input_ids': TensorSpec(shape=(None, None), dtype=tf.int32, name='decoder_input_ids'),
 'input_features': TensorSpec(shape=(None, 80, None), dtype=tf.float32, name='input_features_input_features')}
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * True
  Keyword arguments: {}

Option 2:
  Positional arguments (16 total):
    * {'decoder_attention_mask': TensorSpec(shape=(None, None), dtype=tf.int32, name='decoder_attention_mask'),
 'decoder_input_ids': TensorSpec(shape=(None, None), dtype=tf.int32, name='decoder_input_ids'),
 'input_features': TensorSpec(shape=(None, 80, None), dtype=tf.float32, name='input_features_input_features')}
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * None
    * False
  Keyword arguments: {}
```

Thank you so much again for the wonderful work you are doing for the community.