FastHMR: Accelerating Human Mesh Recovery via Token and Layer Merging
  with Diffusion Decoding
Authors: name: Soroush Mehraban, name: Andrea Iaboni, name: Babak Taati
Organization: Vector Institute

Recent transformer-based models for 3D Human Mesh Recovery (HMR) have
achieved strong performance but often suffer from high computational cost and
complexity due to deep transformer architectures and redundant tokens. In this
paper, we introduce two HMR-specific merging strategies: Error-Constrained
Layer Merging (ECLM) and Mask-guided Token Merging (Mask-ToMe). ECLM
selectively merges transformer layers that have minimal impact on the Mean Per
Joint Position Error (MPJPE), while Mask-ToMe focuses on merging background
tokens that contribute little to the final prediction. To further address the
potential performance drop caused by merging, we propose a diffusion-based
decoder that incorporates temporal context and leverages pose priors learned
from large-scale motion capture datasets. Experiments across multiple
benchmarks demonstrate that our method achieves up to 2.3x speed-up while
slightly improving performance over the baseline.