Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing
Authors: name: Ariel N. Lee, name: Sarah Adel Bargal, name: Janavi Kasera, name: Stan Sclaroff, name: Kate Saenko, name: Nataniel Ruiz

Vision transformers (ViTs) have significantly changed the computer vision
landscape and have periodically exhibited superior performance in vision tasks
compared to convolutional neural networks (CNNs). Although the jury is still
out on which model type is superior, each has unique inductive biases that
shape their learning and generalization performance. For example, ViTs have
interesting properties with respect to early layer non-local feature
dependence, as well as self-attention mechanisms which enhance learning
flexibility, enabling them to ignore out-of-context image information more
effectively. We hypothesize that this power to ignore out-of-context
information (which we name patch selectivity), while integrating
in-context information in a non-local manner in early layers, allows ViTs to
more easily handle occlusion. In this study, our aim is to see whether we can
have CNNs simulate this ability of patch selectivity by effectively
hardwiring this inductive bias using Patch Mixing data augmentation, which
consists of inserting patches from another image onto a training image and
interpolating labels between the two image classes. Specifically, we use Patch
Mixing to train state-of-the-art ViTs and CNNs, assessing its impact on their
ability to ignore out-of-context patches and handle natural occlusions. We find
that ViTs do not improve nor degrade when trained using Patch Mixing, but CNNs
acquire new capabilities to ignore out-of-context information and improve on
occlusion benchmarks, leaving us to conclude that this training method is a way
of simulating in CNNs the abilities that ViTs already possess. We will release
our Patch Mixing implementation and proposed datasets for public use. Project
page: https://arielnlee.github.io/PatchMixing/