Add X-Codec model
Author: Manalelaidouni
Pull request: yes
Labels: name: New model

# What does this PR do?

This PR aims at integrating X-Codec model to `transformers`.

The X-Codec model is a neural audio codec that integrates semantic information from self-supervised models (e.g., HuBERT) alongside traditional acoustic information. This enables :

- **Music continuation** : Better modeling of musical semantics yields more coherent continuations.
- **Text-to-Sound Synthesis** : X-Codec captures semantic alignment between text prompts and generated audio.
- **Semantic aware audio tokenization**: X-Codec is used as an audio tokenizer in the YuE lyrics to song generation model.

X-codec first encodes the audio using an acoustic model (DAC model) and then extracts semantic information using a pretrained Hubert model, this semantic information is further refined with a semantic encoder. The combined features are then fed into a Residual Vector Quantizer (RVQ) that converts the features into discrete codes.

Each individual component reproduce the original X-Codec outputs exactly. Note that I removed the extra padding at the end of the decoded `audio_values` so that the output length matches that of the input, the produced audio closely resembles the original audio with a 1e-5 tolerance; otherwise the forward pass is identical with the output from the original model. 

X-Codec can be now used as a drop in audio tokenizer in YuE as suggested in the design discussion  in  https://github.com/huggingface/transformers/issues/36784.


## Who can review?
@eustlb @zucchini-nlp



Comments:
Hey @ArthurZucker sorry to ping you, it looks like some of the CI tests need to be approved to run, thank you! 🤗​

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38248). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @ArthurZucker thank you for reviewing! I’ve updated based on your suggestions🤗

Hey @Rocketknight1, this has been updated per review and CI is green, could someone take a quick look?

Sorry I was out for 2 weeks!

Hey no worries @ArthurZucker! I’ve rebased and tests have passed, would appreciate your review.

@Manalelaidouni happy to help if you require! 

Thanks for the ping and sorry for the delay, I’ll get on it and push changes soon

run-slow: auto, xcodec

This comment contains run-slow, running the specified jobs: 

models: ['models/auto', 'models/xcodec']
quantizations: [] ...

```python 
    def test_integration(self):
        expected_rmse = {
            "0.5": 0.0065491,
            "4.0": 0.0070978,
        }
        expected_codesums = {
            "0.5": [117262],
            "4.0": [926416],
        }
    
        librispeech = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
        model_id = "Manel/X-Codec"
        model = XcodecModel.from_pretrained(model_id).to(torch_device).eval()
        feature_extractor = AutoFeatureExtractor.from_pretrained(model_id)
    
        librispeech = librispeech.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate))
        audio = librispeech[-1]["audio"]["array"]
    
        inputs = feature_extractor(
            raw_audio=audio, sampling_rate=feature_extractor.sampling_rate, return_tensors="pt"
        ).to(torch_device)
    
        for bandwidth, exp_rmse in expected_rmse.items():
            bandwidth = float(bandwidth)
            with torch.no_grad():
                audio_codes = model.encode(inputs["input_values"], bandwidth=bandwidth, return_dict=False)
                codesum = int(audio_codes.sum().item())
    
                expected_codesum = expected_codesums[str(bandwidth)][0]
>               self.assertEqual(codesum, expected_codesum)
E               AssertionError: 925984 != 926416
```
fails, but it is probably a hardware thing? 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, xcodec

> 
>fails, but it is probably a hardware thing?
> 


I tried to reproduce the failure and rerun the tests on ubuntu and windows  with both cpu and cuda but the test passed consistently, let me know if I should change the equality check 🤗  also I can no longer see the related CI failure.

cf https://github.com/huggingface/transformers/pull/38248#discussion_r2276494292 happy to merge now 🤗 sorry for the late update