Task: Question Generation
Input (English): The aim of this section is to validate the applicability of our theoretical results—which state that self-attention can perform convolution—and to examine whether self-attention layers in practice do actually learn to operate like convolutional layers, when being trained on standard image classification tasks. In particular, we study the relationship between self-attention and convolution with quadratic and learned relative positional encodings. We find that for both cases, the attention probabilities learned tend to respect the conditions of Lemma UNKREF15, corroborating our hypothesis. Nevertheless, to validate that our model learns a meaningful classifier we compare it to the standard ResNet18 BIBREF14 on the CIFAR-10 dataset BIBREF15.
Output (English): What numerical experiments they perform?