Dia model cannot load/ generation error
Author: jiqing-feng
Pull request: no
Labels: name: bug

### System Info

transformers main branch

### Who can help?

@zucchini-nlp 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import set_seed, AutoProcessor, DiaForConditionalGeneration

set_seed(42)

torch_device = "cuda"
model_checkpoint = "nari-labs/Dia-1.6B-0626"

#text = ["[S1] Dia is an open weights text to dialogue model. [S2] You get full control over scripts and voices. [S1] Wow. Amazing. (laughs) [S2] Try it now on Git hub or Hugging Face."]
text = ["[S1] Hello, my dog is cooler than you!"]
processor = AutoProcessor.from_pretrained(model_checkpoint)
inputs = processor(text=text, padding=True, return_tensors="pt").to(torch_device)

model = DiaForConditionalGeneration.from_pretrained(model_checkpoint).to(torch_device)
outputs = model.generate(**inputs, max_new_tokens=3072, guidance_scale=3.0, temperature=1.8, top_p=0.90, top_k=45)

print(outputs)
outputs = processor.batch_decode(outputs)
processor.save_audio(outputs, "dia_example.mp3")
```
I ran the previous script on A100.
The output audio is noise for the input2: ` ["[S1] Hello, my dog is cooler than you!"]`. For input1:`["[S1] Dia is an open weights text to dialogue model. [S2] You get full control over scripts and voices. [S1] Wow. Amazing. (laughs) [S2] Try it now on Git hub or Hugging Face."]`, the output audio is correct. I want to know if it's a bug from the model. Thanks!

BTW, if we load the most common dia model [nari-labs/Dia-1.6B](https://huggingface.co/nari-labs/Dia-1.6B):
```python
from transformers import pipeline

pipe = pipeline("text-to-audio", model="nari-labs/Dia-1.6B")
```
The error will be:
```
>>> pipe = pipeline("text-to-audio", model="nari-labs/Dia-1.6B")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jiqing/transformers/src/transformers/pipelines/__init__.py", line 781, in pipeline
    config = AutoConfig.from_pretrained(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jiqing/transformers/src/transformers/models/auto/configuration_auto.py", line 1352, in from_pretrained
    raise ValueError(
ValueError: Unrecognized model in nari-labs/Dia-1.6B. Should have a `model_type` key in its config.json, or contain one of the following
 strings in its name: afmoe, aimv2, aimv2_vision_model, albert, align, altclip, apertus, arcee, aria, aria_text, audio-spectrogram-trans
former, audioflamingo3, audioflamingo3_encoder, autoformer, aya_vision, bamba, bark, bart, beit, bert, bert-generation, big_bird, bigbir
d_pegasus, biogpt, bit, bitnet, blenderbot, blenderbot-small, blip, blip-2, blip_2_qformer, bloom, blt, bridgetower, bros, camembert, ca
nine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, cod
egen, cohere, cohere2, cohere2_vision, colpali, colqwen2, conditional_detr, convbert, convnext, convnextv2, cpmant, csm, ctrl, cvt, cwm,
...
```

### Expected behavior

Hi @zucchini-nlp . Could you please take a look at this issue? Maybe at the Dia maintainer to look it together. Thanks!

Comments:
@jiqing-feng It does not seem like a hardware or code issue on your side I think the Dia model is trained mainly for multi turn dialogue and longer speaker conditioned scripts so very short single speaker inputs like "Hello my dog...." example as you had given can sometimes collapse during generation and produce noisy audio especially with higher temperature and guidance settings adding a bit of dialogue context or lowering temperature usually fixes it.

However this need to be address in documentation , so @zucchini-nlp if you assign I can include this in documentation

Hi @Haseebasif7 . I tried to lower the temperature and guidance, and extend the input. But the output is still noise:
```python
from transformers import set_seed, AutoProcessor, DiaForConditionalGeneration
import torch

set_seed(42)

torch_device = "cuda" if torch.cuda.is_available() else "cpu"
model_checkpoint = "nari-labs/Dia-1.6B-0626"

# Adjustment 1: Add dialogue context to avoid model collapse on short inputs
# The model performs better with multi-turn context (e.g., adding a preceding sentence from S2).
text = ["[S2] I really like your pet. [S1] Hello, my dog is cooler than you!"]

processor = AutoProcessor.from_pretrained(model_checkpoint)
inputs = processor(text=text, padding=True, return_tensors="pt").to(torch_device)

model = DiaForConditionalGeneration.from_pretrained(model_checkpoint).to(torch_device)

# Adjustment 2 & 3: Further lowered parameters for maximum stability
# temperature: Lowered to 0.4 (makes the model very conservative to prevent noise)
# guidance_scale: Lowered to 1.2 (reduces the forcing strength, which can help with audio artifacts)
outputs = model.generate(
    **inputs, 
    max_new_tokens=3072, 
    guidance_scale=1.2,  
    temperature=0.4,     
    top_p=0.90, 
    top_k=45
)

print(outputs)
outputs = processor.batch_decode(outputs)
processor.save_audio(outputs, "dia_example_stable.mp3")
```

@jiqing-feng 

I ran some analysis and found that the noise is mostly removed when using a higher guidance scale , Very low temperature values tend to make the audio noisy . I already opened a PR to address this behavior .

For now I would recommend using values roughly around these depending on the use case :

max_tokens 3072  
cfg_scale 3.0  
temperature 1.2  
top_p 0.95  
cfg_filter_top_k 45 

Still noise, could you give me a full script that can output the correct audio? (With the input: ["[S1] Hello, my dog is cooler than you!"]) 
Thanks!

@jiqing-feng These settings are working fine for me. Moreover, try to use longer sequences and multi-turn input — your above example will be noisy due to the nature of model training, I think.

```python
from transformers import set_seed, AutoProcessor, DiaForConditionalGeneration
import torch

set_seed(42)

# Device selection - Using CPU for stability (MPS has compatibility issues with this model)
# MPS can cause dimension/shape errors, so we use CPU for reliable operation
if torch.cuda.is_available():
    torch_device = "cuda"
else:
    torch_device = "cpu"

print(f"Using device: {torch_device}")

model_checkpoint = "nari-labs/Dia-1.6B-0626"

# Adjustment 1: Add dialogue context to avoid model collapse on short inputs
# The model performs better with multi-turn context
text = [
    "[S1] I really like your pet. "
    "[S2] Hello, my dog is cooler than you! "
    "[S1] I'm not sure what to say. "
    "[S2] Why can't you say something?"
]

processor = AutoProcessor.from_pretrained(model_checkpoint)
inputs = processor(text=text, padding=True, return_tensors="pt").to(torch_device)

model = DiaForConditionalGeneration.from_pretrained(model_checkpoint).to(torch_device)

outputs = model.generate(
    **inputs,
    max_new_tokens=3072,
    guidance_scale=4,
    temperature=1.5,
    top_p=0.90,
    top_k=50
)

decoded = processor.batch_decode(outputs)
processor.save_audio(decoded, "dia_example_stable2.mp3")
print("Audio saved to dia_example_stable2.mp3")
```


Hi @Haseebasif7 . I still got noise on your example, could you please share your transformers version (or commit id if you build transformers from local)?

@jiqing-feng I am using Version: 4.57.3

> [@jiqing-feng](https://github.com/jiqing-feng) I am using Version: 4.57.3

I still got noise by the same script and the same transformers version on Nvdia A100.

> > [@jiqing-feng](https://github.com/jiqing-feng) I am using Version: 4.57.3
> 
> I still got noise by the same script and the same transformers version on Nvdia A100.

I mean if you are giving a long sequence and had tried multiple parameters combination and still getting noise I don't know what might be the issue exactly . Maybe you can try model provided by Nari Labs themselves [https://github.com/nari-labs/dia](url)

Yes, if the model is generating noise for specific input text only, it might be just the model being wrong. If it is not an actual bug and model from NariLab repo generates the same noise, it is an expected output

As per the pipeline, I am not sure if Dia is supported by audio generation pipeline or not. cc @eustlb @ebezzam 

`pipe = pipeline("text-to-audio", model="nari-labs/Dia-1.6B")` is expected to fail, only the latest `nari-labs/Dia-1.6B-0626` is supported - they slightly differ in architecture using RoPE for the cross attention vs not using it iirc.

The DIA model has poor generalization, and using raw inputs results in noisy outputs. Switching to a new input could resolve this issue.