`MoshiIntegrationTests` started to fail after #34464
Author: ydshieh
Pull request: no
Labels: name: bug

name: Audio

`MoshiIntegrationTests` started to fail after #34464.

### Reproduction

For example, 

> RUN_SLOW=1 python3 -m pytest -v tests/models/moshi/test_modeling_moshi.py::MoshiIntegrationTests::test_moshika_greedy_unconditional_fp16

Since then, during 8 months, there are several periods where those tests are failing with different errors (not even to run the forward/generate). But when they could run, the output values are consistent with the output values given by #34464. Only before/after #34464, the output values are different.

 @gante Could you check if the changed outputs are expected?

Your commit : 8a734ea2

Parent commit: 913330ca

(You will need to run `pip install -e .` when checking out to old commits like these)


Comments:
Hey @ydshieh, I'm new contributor to this repository. Wanna help with this issue, as I don't have that much context of this repo. it might take extra time.
Let me know, if need to look at other issue.

FYI, @gante 

Hi @sandeepyadav1478 

For this issue, it might be better for @gante to take a look.

In the issue page, you can find several issues with label `Good First Issue` :-)

https://github.com/huggingface/transformers/labels/Good%20First%20Issue



Hey @ydshieh 

I have gone through issues with label `Good First Issue`. Their are less issues, as every issue have active people working on them. And had long conversations already. :-)

For sure @gante, Can have look into it. I just wanna help, if needed 💯 

Any use I can be. related to this repo. Lmk

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@ebezzam any chance this issue got solved with your recent PRs? (otherwise I'll repoen it :) )

@gante unfortunately not, I didn't look into Moshi. Running the test @ydshieh mentioned:
```
RUN_SLOW=1 python3 -m pytest -v tests/models/moshi/test_modeling_moshi.py::MoshiIntegrationTests::test_moshika_greedy_unconditional_fp16
```
still yields an error:
```
>           attention_mask = self.decoder.model._prepare_4d_causal_attention_mask_with_cache_position(
                attention_mask,
                sequence_length=sequence_length,
                target_length=past_key_values.get_max_cache_shape(),
                dtype=self.decoder.lm_head.weight.dtype,
                device=device,
                cache_position=cache_position,
                batch_size=batch_size,
                config=self.config,
                past_key_values=past_key_values,
            )
E           TypeError: MoshiModel._prepare_4d_causal_attention_mask_with_cache_position() got an unexpected keyword argument 'device'

src/transformers/models/moshi/modeling_moshi.py:2232: TypeError

FAILED tests/models/moshi/test_modeling_moshi.py::MoshiIntegrationTests::test_moshika_greedy_unconditional_fp16 - TypeError: MoshiModel._prepare_4d_causal_attention_mask_with_cache_position() got an unexpected keyword argument 'device'
```
which is easily fixed by removing `device` [here](https://github.com/huggingface/transformers/blob/main/src/transformers/models/moshi/modeling_moshi.py#L2237).

But then running the test after remove `device` gives following the following error on the expected output:
```
>       self.assertTrue(expected_text_tokens == model_outputs.sequences[0, 1:].tolist())
E       AssertionError: False is not true

tests/models/moshi/test_modeling_moshi.py:1017: AssertionError

FAILED tests/models/moshi/test_modeling_moshi.py::MoshiIntegrationTests::test_moshika_greedy_unconditional_fp16 - AssertionError: False is not true
```

So as @ydshieh mentioned, the expected output may have changed. So let's re-open.

Do you have time to take a look? otherwise I could (would probably have to wait for week after)

@gante FYI below is what I get as the model outputs:
```
# expected [3, 3, 3, 0, 667, 263, 3, 3, 0, 705]
# model_outputs.sequences[0, 1:].tolist()
[3, 1004, 1004, 1004, 1004, 1004, 1004, 1004, 1004, 1004]
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I guess it's not yet resolved ..?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.