Training Noise Token Pruning
Authors: name: Mingxing Rao; name: Bohan Jiang; name: Daniel Moyer
Published: 2024-11-27 07:04:00

Abstract: In the present work we present Training Noise Token (TNT) Pruning for vision
transformers. Our method relaxes the discrete token dropping condition to
continuous additive noise, providing smooth optimization in training, while
retaining discrete dropping computational gains in deployment settings. We
provide theoretical connections to Rate-Distortion literature, and empirical
evaluations on the ImageNet dataset using ViT and DeiT architectures
demonstrating TNT's advantages over previous pruning methods.