Why does Anima keep the T5 token/grid structure in the Qwen conditioning path?
Hi! I've been studying Anima's text-conditioning architecture and wanted to ask about one design choice.
As I understand the original pipeline, Anima does not use Qwen hidden states directly as the final conditioning sequence. Instead, the LLMAdapter combines Qwen features with a target sequence constructed from T5 token IDs / tokenization structure, and produces the final [L, 1024] carrier consumed by the DiT.
So roughly:
Qwen3-0.6B hidden states
+
T5-tokenized target/query sequence
↓
LLMAdapter
↓
Anima DiT conditioning
I'm curious why this design was chosen.
Was the T5-side structure intentionally preserved for compatibility with the original Cosmos text-conditioning interface / pretrained DiT, or did you find experimentally that using a Qwen-native token sequence directly performed worse?
In particular, I'd be interested to know whether you tried any alternatives such as:
- Qwen token positions directly as the conditioning rows;
- a learned resampler from Qwen hidden states to the DiT conditioning space;
- removing the T5 tokenizer/query scaffold entirely.
I've been researching this myself and made a T5-free compatibility version of Anima, where the T5 model/tokenizer and the original llm_adapter are not required at runtime:
https://huggingface.co/Disya/Anima-T5-Free-Base
The DiT still receives its existing 512 × 1024 conditioning interface, but the conditioning content is generated from Qwen without using T5 at runtime.
I'd be very interested in hearing the reasoning behind the original architecture, especially whether the T5-shaped carrier was a deliberate requirement discovered during training or mainly inherited from the Cosmos initialization.
Thanks!
The adapter was made to align with T5 tokenization and embeddings primarily because that was the safe, zero-risk way to do it.
Before DiT model training even began, the LLM Adapter was first pre-trained to replicate the T5 output embeddings. It achieved ~0.95 cosine similarity with T5 embeddings. This is so close that you could use Qwen3 + adapter as a drop-in replacement on the original Cosmos model, and it would be genuinely hard to tell the difference in generated images on a blind comparison. I could very well have frozen the adapter at this point, and it would have been just like training on original T5 embeddings, except computed via a tiny qwen model and adapter.
I did briefly test alternatives based on directly feeding the Qwen3 embeddings into the DiT, with no adapter, using the fact that the embedding dimension is equal. I tried three variants of this 1) train everything, 2) train only cross attention blocks, 3) train only the text embedding linear projections inside the cross attention blocks. (3) was very slow training. (1) and (2) were making progress on aligning to the new text embeddings, but still slow. The main problem here is that it qualitatively was almost like training a whole new diffusion model from scratch. You are starting with completely incoherent images and having to re-learn the entire text embedding -> image mapping. A major risk, even if this eventually converges, is that it only learns those concepts well-represented in the anime datasets the model is being trained on, losing a lot of Cosmos's base knowledge and prompt understanding.
With the approach I went with, even on the very first step of training, the text embeddings fed to the DiT are basically T5 embeddings. The model produces coherent images, and training was fast and stable from the very start.
There is an example of a similar model that swapped out the text encoder, and trained the model to handle the new embeddings using "direct alignment" with no adapter: Newbie-Image.
The adapter was made to align with T5 tokenization and embeddings primarily because that was the safe, zero-risk way to do it.
Before DiT model training even began, the LLM Adapter was first pre-trained to replicate the T5 output embeddings. It achieved ~0.95 cosine similarity with T5 embeddings. This is so close that you could use Qwen3 + adapter as a drop-in replacement on the original Cosmos model, and it would be genuinely hard to tell the difference in generated images on a blind comparison. I could very well have frozen the adapter at this point, and it would have been just like training on original T5 embeddings, except computed via a tiny qwen model and adapter.
I did briefly test alternatives based on directly feeding the Qwen3 embeddings into the DiT, with no adapter, using the fact that the embedding dimension is equal. I tried three variants of this 1) train everything, 2) train only cross attention blocks, 3) train only the text embedding linear projections inside the cross attention blocks. (3) was very slow training. (1) and (2) were making progress on aligning to the new text embeddings, but still slow. The main problem here is that it qualitatively was almost like training a whole new diffusion model from scratch. You are starting with completely incoherent images and having to re-learn the entire text embedding -> image mapping. A major risk, even if this eventually converges, is that it only learns those concepts well-represented in the anime datasets the model is being trained on, losing a lot of Cosmos's base knowledge and prompt understanding.
With the approach I went with, even on the very first step of training, the text embeddings fed to the DiT are basically T5 embeddings. The model produces coherent images, and training was fast and stable from the very start.
There is an example of a similar model that swapped out the text encoder, and trained the model to handle the new embeddings using "direct alignment" with no adapter: Newbie-Image.
Thank you for all your hard work tdrussell.