This idea of using BERT flavored pretraining in computer vision was also explored in Selfie, but it could not demonstrate strong results. Another concurrent work that explores the idea of masked image modeling is SimMIM. Finally, as a fun fact, we, the authors of this example also explored the idea of \"reconstruction as a pretext task\" in 2020 but we could not prevent the network from representation collapse, and hence we did not get strong downstream performance.