RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Figure 8: Qualitative results showcasing ViTCA’s inductive biases in terms of adapting to masking one or several of its self-attention heads. Gold boxes are inputs, green ground truths, and purple ViTCA outputs (after 2784 iterations). For reference, the first column of outputs does not contain any head masking.; Qualitative results showcasing ViTCA’s inductive biases in terms of adapting to masking one or several of its self-attention heads. Gold boxes are inputs, green ground truths, and purple ViTCA outputs (after 2784 iterations). For reference, the first column of outputs does not contain any head masking.
Comparing denoising autoencoding results between baselines and ViTCA variants. ViTCA variants include: 32 (32 heads), 16 (16 heads), i (inverted bottleneck), xy (xy-coordinate positional encoding). Boldface and underlined values denote the best and second best results. Metrics include Peak Signal-to-Noise Ratio (PSNR; dB), Structural Similarity Index Measure (SSIM; values in [0,1] ) [ 55 ] , Learned Perceptual Image Patch Similarity (LPIPS; values in [0,1] ) [ 39 ] .
Quantitative ablation for denoising autoencoding with ViTCA (unless otherwise stated via prefix) on CelebA [ 47 ] . Boldface and underlining denote best and second best results. Italicized items denote baseline configuration settings. † Trained with gradient checkpointing [ 44 ] , which slightly alters round-off error during backpropagation, resulting in slight variations of results compared to training without checkpointing. See Appendix A.2 .
Linear probe [ 28 ] test accuracies (%) of baseline and variant models. Model variants are labelled as in Tab. 4.1 . All baselines and variants were pre-trained for denoising autoencoding and kept fixed during probing. A linear classifier and 2-layer Multilayer Perceptrons (MLP) were trained on raw image inputs. Parameter counts exclude fixed parameters. Boldface and underlined values denote the best and second best results, respectively. Interestingly, CA-based models trained for denoising autoencoding on increasingly challenging datasets produce an increasingly more useful self-supervised representation for image classification compared to non-CA-based models.
Quantitative ablation on attention neighbourhood size N_{H}\times N_{W} for denoising autoencoding with ViTCA on FashionMNIST. Boldface and underlining denote best and second best results. Italicized items denote baseline configuration settings.
Quantitative ablation comparing test results with ViTCA trained using asynchronous ( \sigma\!=\!50\% ) vs . synchronous ( \sigma\!=\!100\% ) cell updates for denoising autoencoding. During testing, cells are updated at the rate they were trained in. Boldface denotes best results. Italicized items denote baseline configuration settings.
Quantitative ablation comparing test results of ViTCA trained with gradient checkpointing disabled vs . enabled. Boldface denotes best results. Italicized items denote baseline configuration settings.
Profiling results showcasing ViTCA’s runtime performance (forward and backward in milliseconds) and memory usage (in GB) while training on a minibatch of random 32\times 3\times H\times W images with gradient checkpointing disabled vs . enabled. We use T\!=\!32 ViTCA iterations and 16 checkpoint segments. Boldface denotes best results. Italicized items denote baseline configuration settings.