HYDRA

Representation Harmonized Tokenization for
Multimodal Generation and Understanding

🎉 HYDRA is accepted by NeurIPS 2026!

Unified multimodal models struggle to bridge the gap between the abstract representations needed for visual understanding and the detailed primitives required for generation. HYDRA closes that gap with a single representation.

• 🧩 HYDRA-TOK — a representation-harmonized pure-ViT tokenizer, no quantization. The backbone transitions from a Gen-ViT (structure-preserving primitives for synthesis) to a Sem-ViT (semantic encoding), mediated by a Generation-Semantic Bottleneck (GSB) that compresses features to 64 channels, filters noise for robust synthesis, then restores dimensionality for semantic comprehension — all in one architecture.
• 🔗 HYDRA — a native unified framework built on HYDRA-TOK with a dual-head decoder (autoregressive for text, rectified flow matching for images) sharing a single parameter space.

See the paper for the full benchmark tables.

📦 Files in this repo

Path	Size	What it is
hydra_1.5b/	4.9G	HYDRA-1.5B unified model (Qwen2.5-1.5B-Instruct backbone). Single pytorch_model.bin.
hydra_7b/	17G	HYDRA-7B unified model (Qwen2.5-7B-Instruct backbone). Sharded into 2 files.
hydra_tok_gan/	1.1G	HYDRA-TOK, GAN-finetuned — use this one for generation (best perceptual quality).
hydra_tok_base/	1.1G	HYDRA-TOK, base (stage-2, pre-GAN) — used to initialize unified-model training.
hydra_tok_latent_stats.pt	1.6M	Latent mean/std. Required — the flow head consumes normalized latents.

Each unified model is one self-contained checkpoint holding the Qwen backbone, und_vit (Sem-ViT), semantic_proj, both flow-matching heads, diff_proj and the timestep embeddings — 661 tensors. Both were trained with the same stage-3 finetune recipe, so HYDRA-1.5B and HYDRA-7B are directly comparable.

Note on the tokenizer pair

und_vit_weights.pth is byte-identical between hydra_tok_gan/ and hydra_tok_base/ — the stage-2 GAN finetune only updates the generative half (Gen-ViT + Pixel Flow Decoder) and leaves the Sem-ViT understanding half frozen. The latent statistics were computed on the base tokenizer and are reused by the GAN version, which is consistent precisely because the finetune does not move the latent space.

Measured on ImageNet val (50K, 448→256 LANCZOS, paper protocol): the GAN finetune trades ~0.9 dB PSNR for a ~33% rFID improvement, which is why it is the recommended choice for generation and the base version for reconstruction.

Not included — fetched separately

Dependency	Source	Needed for
Qwen2.5-1.5B-Instruct	Qwen	1.5B inference + training
Qwen2.5-7B-Instruct	Qwen	7B inference + training
InternViT-300M-448px-V2.5	OpenGVLab	tokenizer stage-1 teacher (training only)
DINO ViT-S/8	Meta AI	tokenizer stage-2 GAN discriminator (training only)

The Qwen backbones are pulled automatically by from_pretrained — no manual download needed.

🚀 Usage

These are research checkpoints for the HYDRA codebase; they are not loadable via AutoModel.from_pretrained. The model is built from the YAML configs, not from the bundled config.json (which is informational only).

git clone https://github.com/Tencent-Hunyuan/HYdra
cd HYdra && pip install -r requirements.txt

No checkpoint download is needed. configs/inference.yaml points at hf://Xuerui123/HYdra/..., and every entry point resolves those specs into the standard HuggingFace cache on first use ($HF_HOME, default ~/.cache/huggingface). Both commands below run as-is on a clean machine.

Multimodal understanding

cd hydra
python evaluation/inference_mmu.py \
    config=configs/inference.yaml \
    mmu_image_path=./test.jpg \
    question='Describe this image.'

mmu_image_path also accepts a directory, in which case every image inside is processed. Multiple questions can be asked in one pass by separating them with ***, e.g. question='What is in this image? *** What color is the sky?'.

Text-to-image

inference_t2i.py must be launched with torchrun — prompts are sharded across ranks internally.

cd hydra
torchrun --nproc_per_node=8 evaluation/inference_t2i.py \
    config=configs/inference.yaml \
    t2i_prompts_file=prompts/t2i_prompts.txt \
    output_dir=samples

The defaults these two commands pick up are HYDRA-1.5B + hydra_tok_gan + hydra_tok_latent_stats.pt + Qwen/Qwen2.5-1.5B-Instruct. To swap in the base tokenizer (reconstruction comparisons only), add model.hydra_tok_model.pretrained_model_path=hf://Xuerui123/HYdra/hydra_tok_base.

7B

Override the three fields that differ from the 1.5B default:

    model_path=hf://Xuerui123/HYdra/hydra_7b \
    model.llm.llm_model_path=Qwen/Qwen2.5-7B-Instruct \
    model.llm.hidden_size=3584

For T2I, add transport.time_shifting_factor=3.0 — the 7B was evaluated with a shift of 3.0, whereas the config default of 5.0 is the 1.5B setting.

Local weights instead

huggingface-cli download Xuerui123/HYdra --local-dir ckpt

# or just the 1.5B generation stack (~6G)
huggingface-cli download Xuerui123/HYdra --local-dir ckpt \
    --include "hydra_1.5b/*" "hydra_tok_gan/*" "hydra_tok_latent_stats.pt"

The layout matches the ckpt/ directory the code expects; then override with model_path=../ckpt/hydra_1.5b. HYDRA_HF_SKIP_DOWNLOAD=1 turns hf:// resolution off completely.

Sampling protocol

Every sampling knob lives under transport.; a bare guidance_scale=... is rejected rather than silently ignored. configs/inference.yaml ships the HYDRA-1.5B GenEval/WISE setting, so only the 7B and DPG-Bench need overrides.

Setting	HYDRA-1.5B	HYDRA-7B
transport.guidance_scale	5.0	5.0
transport.time_shifting_factor	5.0	3.0
transport.num_inference_steps — GenEval / WISE	50	200
transport.num_inference_steps — DPG-Bench	100	100

Bold entries differ from the shipped config defaults (5.0 / 5.0 / 50).

time_shifting_factor is the only timestep shift that affects generation — it reshapes the sampling grid as t → t / (t + f − f·t). The do_shift flag is training-only and has no effect at inference. Setting time_shifting_factor=0.0 disables the shift and causes a train/test mismatch.

GenEval and DPG-Bench numbers were measured with the detailed caption prompts (prompts/geneval_detailed_caption.csv). The prompt set matters a lot: the same checkpoints score ~0.78–0.80 with rewritten prompts and ~0.64 with the original short prompts.

⚙️ Requirements

transformers must be 4.x. Version 5 removed SlidingWindowCache from transformers.cache_utils, which the Qwen2 modeling file imports at module level, so every entry point fails at import on 5.x.

omegaconf 2.3 also requires the antlr4 4.9.3 runtime — a newer antlr4 makes every OmegaConf.load raise Could not deserialize ATN with version (expected 4). Both constraints are pinned in the repo's requirements.txt.

🙏 Acknowledgements

• Show-o2 — unified multimodal framework
• InternViT / InternVL — ViT backbone & teacher
• Qwen2.5 — LLM backbone
• taming-transformers — GAN / LPIPS losses

📄 License

MIT, matching the code release. The checkpoints inherit the licenses of their base models — Qwen2.5 (Apache-2.0) for the unified models, and for HYDRA-TOK the terms of the InternViT teacher used in stage-1 distillation.

📖 Citation

@article{qiu2026hydra,
  title={{HYDRA}: Representation Harmonized Tokenization for Multimodal Generation and Understanding},
  author={Qiu, Xuerui and Cui, Yutao and Zhang, Guozhen and Li, Junzhe and Zhao, Yaqi and Zhang, Xiao and Li, Yang and Liu, Songtao and Yang, Miles and Shi, Yu and Zhong, Zhao and Bo, Liefeng},
  journal={arXiv preprint arXiv:2603.15228},
  year={2026}
}