Before revision:  We investigate if a strong V&L representation model can be learned without text-image pairs. We propose Weakly-supervised VisualBERT with the key idea of conducting " mask-and-predict " pre-training on language-only and image-only corpora .
After revision:  We investigate if a strong V&L representation model can be learned through unsupervised pre-training without image-caption corpora. In particular, we propose to conduct `` mask-and-predict " pre-training on language-only and image-only corpora .