Before revision:  Our approach uses a lightweight probing model that learns to map language representations of concrete words to the visual domain . We find that representations from models trained on purely textual data, such as BERT, can be nontrivially mapped to those of a vision model. Such mappings generalize to object categories that were never seen by the probe during training, unlike mappings learned from permuted or random representations .
After revision:  Our approach uses a lightweight probing model that learns to map language representations of concrete words to the visual domain , finding representations of many language models to be useful in retrieving semantically aligned image patches. In control experiments where language and visual representations are intentionally mismatched, we observe much weaker results. Furthermore, we examine the impact of textual context in our experiments, finding, for instance, that nouns accompanied by adjectives lead to more accurate retrieval .