Mimi model gives different outputs when using batch encode vs single encode
Author: avishaiElmakies
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.46.3
- Platform: Linux-6.6.20-aufs-1-x86_64-with-glibc2.36
- Python version: 3.11.2
- Huggingface_hub version: 0.26.1
- Safetensors version: 0.4.5
- Accelerate version: 1.0.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA A10

### Who can help?

@ylacombe @eustlb 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
import torchaudio
from transformers import MimiModel, AutoFeatureExtractor
model = MimiModel.from_pretrained("kyutai/mimi", num_quantizers=8)
feature_extractor = AutoFeatureExtractor.from_pretrained("kyutai/mimi")
model.cuda()
# load some audio file
inputs = feature_extractor(raw_audio=[audio.squeeze(0).numpy(),audio.squeeze(0).numpy()] , sampling_rate=feature_extractor.sampling_rate, return_tensors="pt")
inputs = {key:value.cuda() for key,value in inputs.items()}
out_batch = model.encode(**inputs).audio_codes
inputs = feature_extractor(raw_audio=audio.squeeze(0).numpy(), sampling_rate=feature_extractor.sampling_rate, return_tensors="pt")
inputs = {key:value.cuda() for key,value in inputs.items()}
out = model.encode(**inputs).audio_codes
(out_batch[0] == out[0]).all() # prints tensor(False, device='cuda:0')
```

### Expected behavior

the output for out_batch[0] and out[0] should be the same.

Comments:
Hey @avishaiElmakies, thanks for opening this issue!

I wasn't able to reproduce your error. I had to make a few changes:
```python
import torchaudio
from datasets import load_dataset, Audio
from transformers import MimiModel, AutoFeatureExtractor
librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")

device = "cuda"

model = MimiModel.from_pretrained("kyutai/mimi").to(device)
feature_extractor = AutoFeatureExtractor.from_pretrained("kyutai/mimi")

# load audio sample
librispeech_dummy = librispeech_dummy.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate))
audio = librispeech_dummy[-1]["audio"]["array"]


# load some audio file
inputs = feature_extractor(raw_audio=[audio, audio] , sampling_rate=feature_extractor.sampling_rate, return_tensors="pt").to(device)
out_batch = model.encode(**inputs, num_quantizers=8).audio_codes

inputs = feature_extractor(raw_audio=audio, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt").to(device)
out = model.encode(**inputs, num_quantizers=8).audio_codes
(out_batch[0] == out[0]).all()
``` 

As you can see, and apart from the audio itself, I've changed how to place tensors and models to the GPU, but used `num_quantizers=8` differently. The way you did it dropped some RVQ layers, which might change the way the model behave.

That said, I'm not sure these changes explain your issue. Would you be able to send a complete self-contained reproducer? Or maybe try with my code snippet if it solves your issue?  


@ylacombe 

Thanks for the reply. your example seems to work fine.

```python
import torchaudio
from transformers import MimiModel, AutoFeatureExtractor

device = "cuda"

model = MimiModel.from_pretrained("kyutai/mimi").to(device)
feature_extractor = AutoFeatureExtractor.from_pretrained("kyutai/mimi")

audio, sr = torchaudio.load("ll_167_1220_kublakhan_coleridge_py_64kb_0000.flac")
audio = torchaudio.functional.resample(audio,orig_freq=sr,new_freq=feature_extractor.sampling_rate)

# load some audio file
inputs = feature_extractor(raw_audio=[audio.squeeze(0).numpy(), audio.squeeze(0).numpy()] , sampling_rate=feature_extractor.sampling_rate, return_tensors="pt").to(device)
out_batch = model.encode(**inputs, num_quantizers=8).audio_codes

inputs = feature_extractor(raw_audio=audio.squeeze(0).numpy(), sampling_rate=feature_extractor.sampling_rate, return_tensors="pt").to(device)
out = model.encode(**inputs, num_quantizers=8).audio_codes
(out_batch[0] == out[0]).all()
```

The above code returns False for me. this is very weird.
I added the file I used as well in a zip.

[ll_167_1220_kublakhan_coleridge_py_64kb_0000.zip](https://github.com/user-attachments/files/18062618/ll_167_1220_kublakhan_coleridge_py_64kb_0000.zip)




This returns True for me, what versions of torch and transformers do you use? What device are you using as well?
Thanks for the audio btw

torch - 2.5.1
transformers - 4.63.3
torchaudio - 2.5.1

I am using an Nvidia A10
I am also using a jupyter notebook, I have no idea if that's relevant. but this is one of the weirdest bugs I've seen so maybe it is :)

If you want me to check other things then let me know


It's really weird indeed. Could you send the first timestamps of each output as well?

What do you mean by timesteps? timesteps of what? 

Sorry I wasn't clear, could you send the audio codes you get?

OK, i will send them tomorrow. I am no longer near my computer.

Thanks for the help! 

@ylacombe 

here are my outputs

[out.txt](https://github.com/user-attachments/files/18076481/out.txt)
[out_batch.txt](https://github.com/user-attachments/files/18076482/out_batch.txt)


Thanks @avishaiElmakies, it helps a lot.

As you can see [here](https://www.diffchecker.com/kxgBEFK7/), there's only 27 differences, out of many many tokens. I think I remember there might be tiny differences in matrix computation with regards to the batch size, this might explain what you observe here.

so IMO, it's not at all worrying and it shouldn't change the output quality. Would you like to investigate a bit more where does the difference comes from?


@ylacombe Thanks for looking into this. Maybe it is some matrix multiplication changes. I will try to look into this more if i have time. for now 27 tokens who are different is probably fine. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.