anujeremylin/HouseholderMotionEmbedding

Householder Motion Embedding: Noise Modulation for Motion-Controllable Video Generation

NeurIPS 2026

Tao Jun Lin^1,2,*  ·  Jinhong Ni¹  ·  Ruyi Zha¹  ·  Yifu Wang²  ·  Xibin Song²  ·  Pan Ji²  ·  Hongdong Li^1,3

¹The Australian National University    ²Vertex Lab    ³Amazon Inc.
^*Corresponding author

Abstract

Diffusion models for video generation have achieved remarkable success, yet precise motion control under explicit user guidance remains challenging. Recent noise-warping methods introduce temporal correlations by warping the noise according to the user-provided motion signal. However, standard spatial warping generally disrupts the noise’s i.i.d. Gaussian structure, requiring distribution correction, such as Jacobian-based correction, but at a substantial computational overhead. We propose Householder Motion Embedding (HME), which encodes the motion signal into the noise through a sequence of Householder reflections that together form an orthogonal transformation applied to each location’s channel vector. HME exactly preserves the spatial Gaussian structure within each frame while introducing cross-frame correlations that promote temporally consistent video. HME is agnostic to the network architecture and integrates directly into standard video diffusion fine-tuning pipelines. Experiments on different scenarios show that HME matches or outperforms the state of the art, while achieving a 2.7× speedup over noise warping. These results demonstrate that per-location Householder transformations in channel space offer a principled and efficient alternative to noise warping for motion-controllable video generation.

Citation

@inproceedings{lin2026hme,
  title     = {Householder Motion Embedding: Noise Modulation for
               Motion-Controllable Video Generation},
  author    = {Lin, Tao Jun and Ni, Jinhong and Zha, Ruyi and Wang, Yifu
               and Song, Xibin and Ji, Pan and Li, Hongdong},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}

---

license: mit datasets: - MuteApo/RealCam-Vid language: - en base_model: - zai-org/CogVideoX-5b-I2V pipeline_tag: image-text-to-video