NormalCrafter: Learning Temporally Consistent Normals from Video
  Diffusion Priors
Authors: name: Yanrui Bin, name: Wenbo Hu, name: Haoyuan Wang, name: Xinya Chen, name: Bing Wang

Surface normal estimation serves as a cornerstone for a spectrum of computer
vision applications. While numerous efforts have been devoted to static image
scenarios, ensuring temporal coherence in video-based normal estimation remains
a formidable challenge. Instead of merely augmenting existing methods with
temporal components, we present NormalCrafter to leverage the inherent temporal
priors of video diffusion models. To secure high-fidelity normal estimation
across sequences, we propose Semantic Feature Regularization (SFR), which
aligns diffusion features with semantic cues, encouraging the model to
concentrate on the intrinsic semantics of the scene. Moreover, we introduce a
two-stage training protocol that leverages both latent and pixel space learning
to preserve spatial accuracy while maintaining long temporal context. Extensive
evaluations demonstrate the efficacy of our method, showcasing a superior
performance in generating temporally consistent normal sequences with intricate
details from diverse videos.