Not able to run evaluate on whisper.tflite that got generated from TFWhisper model
Author: nyadla-sys
Pull request: no

### Model description

@gante
Generated HF TFwhisper model into whisper.tfllite model. However, I'm not sure how to evaluate the created whisper tflite model.

https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/tflite_from_huggingface_whisper.ipynb

I would appreciate your assistance in evaluating whisper.tflite. The notebook mentioned above produces a whisper.tflite file.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hi @nyadla-sys 👋 

That is a great question! The problem here is that generation is much more than a forward pass of the model. Fortunately, our generation code is compatible with TF Graph mode, which means you can compile the entire generation procedure into a graph, which you can directly compare to our examples.

Here is a continuation of your notebook, which creates a TF Lite model for generation with Whisper: https://colab.research.google.com/drive/1tGL73xRs9mFUY5R03im0R6NNcvJriHun?usp=sharing

@gante is it possible to add representative_dataset and generate tflite(int8) model.
converter.representative_dataset = representative_dataset
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/tinynn_pytorch_to_tflite_int8.ipynb


> Hi @nyadla-sys wave
> 
> That is a great question! The problem here is that generation is much more than a forward pass of the model. Fortunately, our generation code is compatible with TF Graph mode, which means you can compile the entire generation procedure into a graph, which you can directly compare to our examples.
> 
> Here is a continuation of your notebook, which creates a TF Lite model for generation with Whisper: https://colab.research.google.com/drive/1tGL73xRs9mFUY5R03im0R6NNcvJriHun?usp=sharing

@gante Great work and appreciate for your efforts to make it open 

@nyadla-sys I don't know how to answer your latest question. 

Gently pinging @hollance, who might have better pointers for Whisper + TF Lite + int8

@gante Is it feasible to include Conv2d and avoid getting FlexConv2D as part of the model?
 TFLite interpreter needs to link Flex delegate in order to run the model since it contains the following Select TFop(s):
Flex ops: FlexConv2D
Details:
	tf.Conv2D(tensor<1x1x?x?xf32>, tensor<1x3x80x384xf32>) -> (tensor<1x1x?x384xf32>) : {data_format = "NHWC", device = "", dilations = [1, 1, 1, 1], explicit_paddings = [], padding = "VALID", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true}


@gante 
When I run generated tflite file with the minimal example from tensorflow/lite/example and it fails with below error msg

Execution plan as the list of 568 nodes invoked in-order: [0-567]
--------------Subgraph-8 dump has completed--------------

--------------Memory Arena Status Start--------------
Total memory usage: 396 bytes (0.000 MB)
- Total arena memory usage: 396 bytes (0.000 MB)
- Total dynamic memory usage: 0 bytes (0.000 MB)

Subgraph#0   Arena (Normal)            268 (67.68%)
Subgraph#0   Arena (Persistent)        128 (32.32%)
--------------Memory Arena Status End--------------

2022-10-20 16:55:50.791845: W tensorflow/core/framework/op_kernel.cc:1830] OP_REQUIRES failed at conv_ops.cc:688 : INVALID_ARGUMENT: input depth must be evenly divisible by filter depth: 1 vs 80
ERROR: input depth must be evenly divisible by filter depth: 1 vs 80
ERROR: Node number 696 (TfLiteFlexDelegate) failed to invoke.
Error at /home/niranjanyadla/useful_sensors/download_tools/openai-work/tflite_linux/tflite_build/tensorflow/tensorflow/lite/examples/minimal/minimal.cc:71

@gante I modified generation code as below and it works fine

  @tf.function(
    # shouldn't need static batch size, but throws exception without it (needs to be fixed)
    input_signature=[
      tf.TensorSpec((1, 80, 3000), tf.float32, name="input_features"), 
    ],
  )

@gante 
I found that my 30-second audio has more generated ids than the 21 produced by the whisper TFlite model.
Is there anything from the tflite model that I am missing?
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/tflite_from_huggingface_whisper.ipynb
and also it does not produce english transcript for total of 30 seconds audio


increased the max_tokens to 200 and now I could generate whole audio text 

@nyadla-sys two questions to help pinpoint the problem:
1. Does the standard TF model (i.e. non-TFLite) work correctly for that audio file?
2. If the answer to 1 is yes: can you share a code example of the problem? (the link above doesn't work for me)

@gante now I modified the [colab](https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/tflite_from_huggingface_whisper.ipynb) notebook to generate more tokens as per below line from HF colab

predicted_ids = model.generate(inputs, max_length=480_000)
Referred this snippet from HF colab
https://colab.research.google.com/drive/191WGH59ZZ-xyu8d6GWbuqZHa_MQJmQpA?usp=sharing#scrollTo=yENhy_7Qq5nU



@gante @hollance

Have added something like below and it is giving segmentation fault.
Could you please help me on this ?
"converter.representative_dataset = representative_dataset" and 
def representative_dataset():
    for x in range(1):
      inputs = feature_extractor(
      ds[x]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="tf")
      input_features = inputs.input_features
     # print(input_features)
      yield [input_features]


Please see the below code for detailed information:
class GenerateModel(tf.Module):
  def __init__(self, model):
    super(GenerateModel, self).__init__()
    self.model = model

  @tf.function(
    # shouldn't need static batch size, but throws exception without it (needs to be fixed)
    input_signature=[
      tf.TensorSpec((1, 80, 3000), tf.float32, name="input_features"), 
    ],
  )
  def serving(self, input_features):
    outputs = self.model.generate(
      input_features,
      max_new_tokens=223, #change as needed
      return_dict_in_generate=True,
    )
    return {"sequences": outputs["sequences"]}

def representative_dataset():
    for x in range(1):
      inputs = feature_extractor(
      ds[x]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="tf")
      input_features = inputs.input_features
     # print(input_features)
      yield [input_features]

import tensorflow as tf

saved_model_dir = '/content/tf_whisper_saved'
tflite_model_path = 'whisper.tflite'

generate_model = GenerateModel(model=model)
tf.saved_model.save(generate_model, saved_model_dir, signatures={"serving_default": generate_model.serving})

# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.target_spec.supported_ops = [
  tf.lite.OpsSet.TFLITE_BUILTINS, # enable TensorFlow Lite ops.
  tf.lite.OpsSet.SELECT_TF_OPS # enable TensorFlow ops.
]
converter.representative_dataset = representative_dataset
#converter.inference_input_type = tf.int8  # or tf.uint8
#converter.inference_output_type = tf.int8  # or tf.uint8
converter.optimizations = [tf.lite.Optimize.DEFAULT]
tflite_model = converter.convert()

# Save the model
with open(tflite_model_path, 'wb') as f:
    f.write(tflite_model)

@hollance @gante 
I was able to convert from Hugging face whisper onnx to tflite(int8) model,however am not sure how to run the inference on this model 
Could you please review and let me know if there is anything i am nissing in onnx to tflite conversion
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/whisper_to_onnx_tflite_int8.ipynb


Hey @nyadla-sys -- model quantization with TFLite is beyond what we support at the moment here in `transformers`, I am afraid I won't dig into your issue at the moment.

You can, however, try asking that question in our [forum](https://discuss.huggingface.co/) 🤗, you might find support from other users there.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

keep it open .!

@gante
Is it possible to modify the input audio spectrograms from 30s to 10 seconds in order to use them as input for a Hugging Face Whisper TensorFlow model?

on other note 
if you have any clue to generate int8 model ,please share your thoughts?

@nyadla-sys 

> Is it possible to modify the input audio spectrograms from 30s to 10 seconds in order to use them as input for a Hugging Face Whisper TensorFlow model?

Not directly -- the model expects a fixed size input, corresponding to 30s. 

> if you have any clue to generate int8 model ,please share your thoughts?

I'm not an int8 expert, so I have minimal pointers: see our [Optimum](https://github.com/huggingface/optimum) library, which has support for int8 quantization

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@gante I separated encoder and decoder tflite models, however while running inference of decoder I only get single output .
Could you please review the notebook and let me know if you have any input for me.


Hi @nyadla-sys 👋 TF Lite is not (yet) a priority for us, as we don't have enough bandwidth to support it. I won't look at your notebook.


I was able to successfully separate the encoder and decoder whisper tflite models in the following notebook and working correctly .
https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/whisper_encoder_decoder_tflite.ipynb

Posting here to help some of HF users who are interested in whisper tflite models



@sanchit-gandhi 
how do i get transcript from the below script
```
import torch
from transformers import AutoFeatureExtractor, WhisperModel
from datasets import load_dataset

model = WhisperModel.from_pretrained("openai/whisper-base")
feature_extractor = AutoFeatureExtractor.from_pretrained("openai/whisper-base")
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt")
input_features = inputs.input_features
decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id
last_hidden_state = model(input_features, decoder_input_ids=decoder_input_ids).last_hidden_state
list(last_hidden_state.shape)
```

@gante I am attempting to divide the TFWhisperModel into an encoder and a decoder, but the code I have is producing an error. Can you assist me in resolving this issue?

```
import tensorflow as tf
from transformers import TFWhisperModel
class WhisperEncoder(TFWhisperModel):
    def call(self, inputs, **kwargs):
        return self.encoder(inputs, **kwargs)

class WhisperDecoder(TFWhisperModel):
    def call(self, inputs, **kwargs):
        return self.decoder(inputs, **kwargs)


model = TFWhisperModel.from_pretrained("openai/whisper-tiny")
encoder_model = WhisperEncoder.from_pretrained("openai/whisper-tiny")
decoder_model = WhisperDecoder.from_pretrained("openai/whisper-tiny")


tf.saved_model.save(encoder_model, "whisper_encoder_model_dir")
tf.saved_model.save(decoder_model, "whisper_decoder_model_dir")

```
here is the error message
TypeError: Exception encountered when calling layer "whisper_encoder" (type WhisperEncoder).

encoder() got an unexpected keyword argument 'training'

Call arguments received by layer "whisper_encoder" (type WhisperEncoder):
  • inputs={'input_features': 'tf.Tensor(shape=(2, 80, 2999), dtype=float32)', 'decoder_input_ids': 'tf.Tensor(shape=(1, 2), dtype=int32)'}
  • kwargs={'training': 'None'}

Hey @nyadla-sys 👋 The encoder and decoder components of Whisper, when isolated, are not compatible with `from_pretrained`. However, you can still serialize them separately, from different sources:

```py
import tensorflow as tf
from transformers import TFWhisperModel

model_1 = TFWhisperModel.from_pretrained("openai/whisper-tiny")
model_2 = TFWhisperModel.from_pretrained("openai/whisper-tiny")

tf.saved_model.save(model_1.get_encoder(), "/tmp/whisper/encoder")
tf.saved_model.save(model_2.get_decoder(), "/tmp/whisper/decoder")
```

Hey @nyadla-sys! For inference, we can use the `.generate()` method to auto-regressively generate using the Whisper model:

```python
import torch
from transformers import AutoProcessor, WhisperForConditionalGeneration
from datasets import load_dataset

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-base")
processor = AutoProcessor.from_pretrained("openai/whisper-base")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
inputs = processor(ds[0]["audio"]["array"], return_tensors="pt")

input_features = inputs.input_features

with torch.no_grad():
    predicted_ids = model.generate(input_features)

transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
print(transcription)
```
**Print Output:**
```
[' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.']
```

@sanchit-gandhi is it possible to generate transcript using TFWhisperModel ? instead of WhisperForConditionGeneragion

Hey @nyadla-sys! 

TFWhisperModel is just the base encoder-decoder model that outputs decoder hidden-states: https://huggingface.co/docs/transformers/model_doc/whisper#transformers.TFWhisperModel

TFWhisperForConditionalGeneration adds a language modelling head on top of TFWhisperModel, mapping the decoder hidden-states to logits over the vocabulary: https://huggingface.co/docs/transformers/model_doc/whisper#transformers.TFWhisperForConditionalGeneration

So you'll need TFWhisperForConditionalGeneration in order to get logits over the vocab (and hence generate text)

Hope that makes sense!

@sanchit-gandhi 
Is it possible to directly map the decoder hidden states to logits without using the language modeling head? I am focusing on using only TFWhisperModel because it can be fully converted into an int8 model. I'm curious if there is any way to generate text using the decoder hidden states without adding the language modeling head.

Hey @nyadla-sys, it's precisely the job of the language modelling head to **directly** map the decoder hidden-states to logits. The language modelling head is a single linear layer that maps from $\mathcal{R}^{d} -> \mathcal{R}^{v}$, where $d$ is the dimensionality of the hidden-states, and $v$ is the dimensionality of the vocabulary, so for Whisper small this is a mapping from 768 -> 52000

So if you need to map to the vocabulary, you're best off using `TFWhisperForConditionalGeneration`!

@sanchit-gandhi Thanks for your detailed information

@gante @younesbelkada @sanchit-gandhi, I encountered an issue while generating a tflite file from the TFWhisperForConditionalGeneration model for transcribing Spanish. The output is only producing zeros. For further reference, please refer to the attached [notebook](https://colab.research.google.com/drive/1cS3jorRn1kcVbakCcd_VF_zcZOcpQ6LC?usp=sharing). Can you please let me know if I made any mistakes in the conversion process?

@nyadla-sys have you confirmed that the model works as expected before converting to TFLite? :)

> @nyadla-sys have you confirmed that the model works as expected before converting to TFLite? :)

Yes, it worked as expected before converting 

@gante any update on this ?


@sayakpaul do you have any tips for debugging TF to TFLite mismatches?

@nyadla-sys, this is better off in the TensorFlow repository in the TFLite category. Here are some things that we can try out: 

* I would first see if removing the dynamic-range quantization from conversion (TFLite) helps with the predictive performance of the model. If the outputs are as expected with dynamic-range quantization, then probably that's the bug. 
* I noticed you're not using the processor that comes with the TF Whisper model from `transformers`. I would use that to prepare the inputs for the TFLite model. Just like you did for the `tf.Module`. 

I am afraid, beyond this point, we won't be able to help you out with anything related to TFLite. 

@sayakpaul thanks for your response
I have already raised a couple of issues in the TFLite category on the TF repository, but I have found the support from Google to be lacking when it comes to resolving these kinds of issues.
https://github.com/tensorflow/tensorflow/issues/58451
https://github.com/tensorflow/tensorflow/issues/59506

I also removed the dynamic quantization and issue still persists..




If I simply convert the Whisper PyTorch model to an ONNX model and then use a TFLite encoder and decoder, it performs well. You can refer to the notebook available at [Link](https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/whisper_encoder_decoder_tflite.ipynb) for more details.

However, I suspect that the HF TF Whisper model may be the root cause of the error I am encountering.


Hey @nyadla-sys! I had a quick look at the code snippet. Could you double check that you're setting the correct prompt tokens? E.g. that we force the model to **translate** (not transcribe). Since our source audio (English) is different from our target text (Spanish), we need to force the model to do speech translation (rather than speech transcription).

This is most easily done by using the `processor`'s `get_decoder_prompt_ids` method:
https://github.com/huggingface/transformers/blob/a8eb4f79f946c5785f0e91b356ce328248916a05/src/transformers/models/whisper/processing_whisper.py#L44

Which enables you to specify the target language and task:
```python
forced_ids = processor. get_decoder_prompt_ids(language="Spanish", task="translate")
```

And then pass them to the generate config as required.

The first two tokens generated by the model look off, which suggests that the forced decoder ids are not being set correctly.

More details here: https://huggingface.co/openai/whisper-tiny#usage

@sanchit-gandhi 

Created simple notbook to reproduce the issue
https://colab.research.google.com/drive/1jvXy4SVqVocOYgAMGDKGcBxMN5u97oet?usp=sharing

Please review and let me know your comments 


Hey @nyadla-sys

I reviewed the Colab up until the second code snippet (where you run `TFWhisperForConditionalGeneration`). I don't see any abnormalities in how the TF model performs - it correctly transcribes the audio in French:
```
[' Un vrai travail intéressant va enfin être mené sur ce sujet.']
```

So we can verify that the HF TF Whisper implementation is indeed correct ✅

I converted HF TF to TFLite model then i see the issue.
I am unsure why the TFLite model is causing this issue.


Hey @nyadla-sys, since the problem is with the TFLite model, the issue is better suited for the TensorFlow repository (see https://github.com/huggingface/transformers/issues/19691#issuecomment-1432418643). Feel free to open an issue there for dedicated TFLite support.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@gante @sanchit-gandhi 

Please see the below information to solve converting from HF TF to tflite 

Here is the monkey patch that got it working for me. I can set forced_decoder_ids in the generate call now.

The issue was that in the huggingface transformers code, to force a token they set the scores for all tokens to -tf.float(inf) and then update the score for the forced token to 0, so it gets chosen. But in tflite the -inf value sometimes gets rounded and ends up as a NaN. Its the closest value to a overflow error that you can get. So I just replaced it with -1 as it is still less than 0.

Include the cell below before your generate call.

import tensorflow as tf
import numpy as np
from transformers import TFForceTokensLogitsProcessor, TFLogitsProcessor
from typing import List, Optional, Union, Any

# Patching methods of class TFForceTokensLogitsProcessor(TFLogitsProcessor):

def my__init__(self, force_token_map: List[List[int]]):
    force_token_map = dict(force_token_map)
    # Converts the dictionary of format {index: token} containing the tokens to be forced to an array, where the
    # index of the array corresponds to the index of the token to be forced, for XLA compatibility.
    # Indexes without forced tokens will have an negative value.
    force_token_array = np.ones((max(force_token_map.keys()) + 1), dtype=np.int32) * -1
    for index, token in force_token_map.items():
        if token is not None:
            force_token_array[index] = token
    self.force_token_array = tf.convert_to_tensor(force_token_array, dtype=tf.int32)

def my__call__(self, input_ids: tf.Tensor, scores: tf.Tensor, cur_len: int) -> tf.Tensor:
    def _force_token(generation_idx):
        batch_size = scores.shape[0]
        current_token = self.force_token_array[generation_idx]

        # Original code below generates NaN values when the model is exported to tflite
        # it just needs to be a negative number so that the forced token's value of 0 is the largest
        # so it will get chosen
        #new_scores = tf.ones_like(scores, dtype=scores.dtype) * -float("inf")
        new_scores = tf.ones_like(scores, dtype=scores.dtype) * -float(1)
        indices = tf.stack((tf.range(batch_size), tf.tile([current_token], [batch_size])), axis=1)
        updates = tf.zeros((batch_size,), dtype=scores.dtype)
        new_scores = tf.tensor_scatter_nd_update(new_scores, indices, updates)
        return new_scores

    scores = tf.cond(
        tf.greater_equal(cur_len, tf.shape(self.force_token_array)[0]),
        # If the current length is geq than the length of force_token_array, the processor does nothing.
        lambda: tf.identity(scores),
        # Otherwise, it may force a certain token.
        lambda: tf.cond(
            tf.greater_equal(self.force_token_array[cur_len], 0),
            # Only valid (positive) tokens are forced
            lambda: _force_token(cur_len),
            # Otherwise, the processor does nothing.
            lambda: scores,
        ),
    )
    return scores

TFForceTokensLogitsProcessor.__init__ = my__init__
TFForceTokensLogitsProcessor.__call__ = my__call__

Hey @nyadla-sys - glad to hear you were able to monkey patch a fix! Since we need to maintain equivalence with PyTorch Transformers, we need to set the logits to -inf in the logits processor in TF Transformers to numerically match the implementations. Hopefully your comment serves as a pointer for those facing issues with TF Lite exports.

Hi friends, I want to use this whisper model for language identification, but the language I want it to recognize is one of the languages with few sources, so I have to fine-tune it on the custom data source, can you help me? I look forward to hearing from you and thank you.@sanchit-gandhi