Repository: huggingface/transformers
Pull request: Granite speech - minor fixes to support training with the HF trainer

# What does this PR do?

Minor updates to `granite_speech` to enable finetuning it with HF trainers.
- avoids a crash when trainers pass `padding=True` to the processor
- ensure all trainable parameters have gradients (bugfix) - remove `.data` from a forward call
- rename `melspec` to `mel_filters` to leverage [this](https://github.com/huggingface/transformers/blob/d5d007a1a0f0c11a726a54c8f00bd71825f84d02/src/transformers/feature_extraction_utils.py#L595), which avoids a crash on `save_pretrained`


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

CC: @ArthurZucker @eustlb @alex-jw-brooks @avishaiElmakies @gsaon


Changed files:
filename: src/transformers/models/granite_speech/feature_extraction_granite_speech.py
patch: @@ -50,15 +50,16 @@ def __init__(
         **kwargs,
     ):
         super().__init__(**kwargs)
+        self.sampling_rate = sampling_rate
         self.melspec_kwargs = {
             "sample_rate": sampling_rate,
             "n_fft": n_fft,
             "win_length": win_length,
             "hop_length": hop_length,
             "n_mels": n_mels,
         }
-        # Currently lazily initialized
-        self.melspec = None
+        requires_backends(self, ["torchaudio"])
+        self.mel_filters = torchaudio.transforms.MelSpectrogram(**self.melspec_kwargs)
         self.projector_window_size = projector_window_size
         self.projector_downsample_rate = projector_downsample_rate
 
@@ -91,34 +92,16 @@ def __call__(
         ).view(-1, 1)
         return BatchFeature(data=speech_inputs)
 
-    def _ensure_melspec_transform_is_initialized(self):
-        """
-        Ensures the mel spectrogram transform on this instance is initialized.
-
-        We do this for now since some logging explodes since the mel spectrogram
-        transform is not JSON serializable.
-        """
-        requires_backends(self, ["torchaudio"])
-
-        if self.melspec is None:
-            # TODO (@alex-jw-brooks / @eustlb) move this to common batch
-            # feature extraction in audio utils once they are written!
-            self.melspec = torchaudio.transforms.MelSpectrogram(**self.melspec_kwargs)
-
     def _extract_mel_spectrograms(self, audio: "torch.Tensor", device="cpu"):
         """
         Compute the Mel features to be passed to the conformer encoder.
         """
         requires_backends(self, ["torchaudio"])
-
-        # Initialize the mel spectrogram if isn't not already and
-        # move the melspec / audio to the computation device.
-        self._ensure_melspec_transform_is_initialized()
         if device is not None:
-            melspec = self.melspec.to(device)
+            melspec = self.mel_filters.to(device)
             audio = audio.to(device)
         else:
-            melspec = self.melspec
+            melspec = self.mel_filters
 
         bsz = audio.shape[0]
         with torch.no_grad():

filename: src/transformers/models/granite_speech/modeling_granite_speech.py
patch: @@ -83,7 +83,7 @@ def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
         hidden_states = hidden_states.view(batch_size * nblocks, self.window_size, dim)
 
         query_output = self.qformer(
-            query_embeds=self.query.data,
+            query_embeds=self.query,
             encoder_hidden_states=hidden_states,
             encoder_attention_mask=None,
             return_dict=True,

filename: src/transformers/models/granite_speech/processing_granite_speech.py
patch: @@ -88,7 +88,9 @@ def __call__(
         else:
             audio_inputs = {}
 
-        text_inputs = self.tokenizer(prompt_strings, padding=True, **kwargs)
+        if "padding" not in kwargs:
+            kwargs["padding"] = True
+        text_inputs = self.tokenizer(prompt_strings, **kwargs)
         return BatchFeature(data={**text_inputs, **audio_inputs})
 
     def _get_validated_text(self, text: Union[str, list]) -> list[str]: