Before revision:  We show evidence of a surprising common ground with the visual domain, finding representations of many language models to be useful in retrieving semantically aligned image patches . In control experiments where language and visual representations are intentionally mismatched, we observe much weaker results. Furthermore, we examine the impact of textual context in our experiments, finding, for instance, that nouns accompanied by adjectives lead to more accurate retrieval. Finally, we show that the examined models substantially under-perform humans in retrieval  .
After revision:  We show evidence of a surprising common ground with the visual domain, finding representations of many language models to be useful in retrieving semantically aligned image patches , and explore the role of context in this process. Much weaker results are found in control experiments, attesting the selectivity of the probe. All examined models greatly under-perform humans in retrieval .