Mapping Language Models to Grounded Conceptual Spaces
Authors: Roma Patel
Ellie Pavlick
Venue: ICLR 2022 Poster (2022)

Abstract: A fundamental criticism of text-only language models (LMs) is their lack of grounding---that is, the ability to tie a word for which they have learned a representation, to its actual use in the world. However, despite this limitation, large pre-trained LMs have been shown to have a remarkable grasp of the conceptual structure of language, as demonstrated by their ability to answer questions, generate fluent text, or make inferences about entities, objects, and properties that they have never physically observed. In this work we investigate the extent to which the rich conceptual structure that LMs learn indeed reflects the conceptual structure of the non-linguistic world---which is something that LMs have never observed. We do this by testing whether the LMs can learn to map an entire conceptual domain (e.g., direction or colour) onto a grounded world representation given only a small number of examples. For example, we show a model what the word ``left" means using a textual depiction of a grid world, and assess how well it can generalise to related concepts, for example, the word ``right", in a similar grid world. We investigate a range of generative language models of varying sizes (including GPT-2 and GPT-3), and see that although the smaller models struggle to perform this mapping, the largest model can not only learn to ground the concepts that it is explicitly taught, but appears to generalise to several instances of unseen concepts as well. Our results suggest an alternative means of building grounded language models: rather than learning grounded representations ``from scratch'', it is possible that large text-only models learn a sufficiently rich conceptual structure that could allow them to be grounded in a data-efficient way.

Published as a conference paper at ICLR 2022
MAPPING LANGUAGE MODELS TO GROUNDED CON- CEPTUAL SPACES
Roma Patel & Ellie Pavlick
Department of Computer Science
Brown University
ABSTRACT
A fundamental criticism of text-only language models (LMs) their lack of grounding- is, the ability to tie word for which they have learned represen- tation its referent the non-linguistic world. However, despite this limitation, large pre-trained LMs have been shown to have a remarkable grasp of the concep- tual structure of language, as demonstrated by their ability to answer questions, generate fluent text, make inferences about entities, objects, and properties that they have never physically observed. In this work we investigate the extent to which the rich conceptual structure that LMs learn indeed reflects the conceptual structure of the non-linguistic world-which is something that LMs have never observed. We do this by testing whether the LMs can learn map entire con- ceptual domain (e.g., direction or colour) onto a grounded world representation given only a small number of examples. For example, we show model what the word "left" means using a textual depiction of grid world, and assess how well it can generalise to related concepts, for example, the word "right", in similar grid world. We investigate range of generative language models of varying sizes (in- cluding GPT-2 and GPT-3), and see that although the smaller models struggle to perform this mapping, the largest model can not only learn to ground the concepts that is explicitly taught, but appears to generalise to several instances of unseen concepts well. Our results suggest alternative means of building grounded language models: rather than learning grounded representations "from scratch", it is possible that large text-only models learn sufficiently rich conceptual structure that could allow them to be grounded in a data-efficient way.
INTRODUCTION
Large pre-trained language models (LMs) trained on text corpora have shown remarkable progress on range of natural language understanding tasks (Radford et al. 2019 Brown et al.) 2020). Such models have demonstrated their ability to generate fluent dialogue (Brown et al.) 2020), make com- monsense inferences Zellers et al. (2019), and reconstruct taxonomies and word relations (Chen et al. 2020). However, has been argued that true meaning cannot be learned from the form of language alone (i.e.. from text) because it is word's use in the non-linguistic world that imparts meaning (Bender & Koller [2020] Bisk et al. (200).. For example, although LMs might learn from textual co-occurrences that the words north and south are opposites, the grounded meaning of these words, i.e., the direction that you should travel if you are told to go north, is something to which these models, by definition, do not have access during training.
While it is indisputable that text-only models do not learn representations of concepts that are grounded in the non-text world, it possible for the structure of relations between concepts in text form to be identical to what a grounded model would learn. In principle, it is therefore possible for a text-only model's conceptual space to be isomorphic to the "true" (i.c., grounded) conceptual space (Merrill et al. 2021). In this work, we investigate whether this is the case by asking whether models can learn to ground an entire domain (e.g., direction) after grounding only a subset of the points in that domain (e.g., left). Specifically, for generative LMs that have been trained only large text corpora, we "orient" the models by showing them how some word forms (that they have learned during training) used in simple text worlds for example, what the direction north maps

---

Published as a conference paper at ICLR 2022
to in a textual representation of a grid world (see Figure D. We then evaluate two types of gener- alisation. First (3.1)). we evaluate generalisation to unseen worlds. For example, if the model has seen several realisations of the word north in different grid worlds, can it correctly identify north in an unseen world (e.g., one of a different size or shape)? Second ($3.2). we evaluate generalisation to unseen but related concepts. For example, if the model has been shown grounded representations of north and east, can it correctly identify south and west, even though it was never shown them? We find that although the small language models (GPT-2 models that contain on the order of 100M parameters) cannot perform either generalisation well, the largest model (a GPT-3 model containing 175B parameters) can indeed learn groundings in the conceptual worlds we build. We analyse the predictions and errors made by models ($3.3) and find that the errors made by the small models are often due to a failure to recognize the domain, and thus generating random words by default. In contrast, the errors made by the largest model are often intuitive, e.g., predicting in-domain concepts that are reasonable substitutes for the target (c.g., maroon versus dark red).
2 EXPERIMENTAL DESIGN
2.1 MODELS
We test five autoregressive Transformer language models (Vaswani et al., 2017) of varying size, specifically the GPT-2 (Radford et al. 2019) and GPT-3 (Brown et al. 2020) models. Our smallest model contains 124M parameters, and the others follow increasing model sizes (355M, 774M 1.5B and 175B parameters). All models are pre-trained on differently filtered versions of the OpenAl Web-Text dataset Radford et al. (2019), composed of 40GB of English web text available on the internet. We generate up to 5 tokens per prompt and, to improve the robustness of our analyses, generate 3 samples per prompt. We use a temperature of 1 during generation and sample from the softmax probabilities produced at each time step using nucleus sampling (Holtzman et al. 2019) with p=0.85. We include more detail on the models and their training data in Appendix A
2.2 IN-CONTEXT LEARNING
Several studies Brownn et al. 2020 Reynolds & McDonell 2021) have shown that instead of fine- tuning generative LMs-ie.. a process that updates parameters learned by the model during pre- training-it is possible to achieve competitive performance by giving the model a small number of training examples within the prompt. This often referred to as "in-context learning" or "few- shot prompting". Specifically, prompt includes task examples that include a question prefix (e.g., "World:") followed by the question, and an answer prefix (e.g., "Answer:") followed by the answer to the question. After giving the model examples in this manner, the prompt ends with new question and only an answer prefix after which the model is expected to generate an answer to the last question, following the prompt format it has seen. By enumerating over all questions in the test set, we can obtain a model-generated answer for every test set question that we wish to evaluate. There are no gradient updates to any model parameters using this approach.
2.3 GROUNDED CONCEPT DOMAINS
The models that we test, by construction, can receive only text inputs. Thus, we focus on a set of grounded domains for which it is possible to faithfully represent the grounded meaning in text form. We briefly describe these domains below, summarise them in Figure and describe in detail, the prompt creation process for each generalisation task in Sections $3.2 and 3.1
We discuss the
possibility of expanding this set of concepts in future work in Section $4
Spatial Terms We consider 6 spatial concepts: left, right, up. down, top, bottom. Each of the above concepts can be represented a grid world using the position of special character (here, a 'I') in the world. To do this, we create grid world environments of varying sizes (where the number of rows and columns ranges from 1 to 8), where each world consists of '0's and a single '1'.
Cardinal Directions We consider eight cardinal directions: north, south, east, west, northeast, northwest, southeast, southwest. These similar to spatial terms, except that they include compo-
2

---

Published as a conference paper at ICLR 2022
Spatial Terms
Cardinal Directions
RGB Colours
above
World:
NW
NE
World:
RGB: (255, 0)) Answer: red
[0. 0. 0.j
[o. O. 1.]
lef
right
[0. O. 1. 0.J
[0. 0.]
[0. 0. O. 0.
[0. 0. 0.]
RGB: (143, 0, 255) Answer: violet
Answer: right
SW
SE
Answer:
northeast
below
dorr
Figure 1: Figure shows example worlds and groundings for three concept categories: colours, cardi- nal directions, and spatial terms. For each of the three domains, the left figure in each shows the full set of grounded concepts in the domain. To the right, we see example world representations with textual instantiations of the groundings that serve as prompts for language models.
sitional terms (e.g., northeust). We use grid-worlds of the same format as spatial terms to represent these concepts.
Colour Terms We consider colour terms in a three-dimensional space, using dataset of 367 RGB colours (Abdou et al. 2021) that contains colour names (e.g., red, cyan, forest green) each associated with an RGB code (e.g.. (255, 0, 0)). Therefore, the world representation in this case is not a grid-world, but an RGB code associated with every colour name. Figure [ shows example RGB codes and colours that serve part of prompt given to the models we test.
2.4 ROTATED WORLDS TO CONTROL FOR MEMORISATION
Motivation The GPT-x models that have been trained on the CommonCraw/ corpus (Rad- ford et al. (2019)), a collection of documents that contains web text freely available on the internet. Since provide instantiations of grounded concepts in text form, it is very plausible that the do- mains described above have been encountered verbatim during training. For example, for spatial terms such left, model might have seen instances of matrices and linear algebra terms with the word left in close proximity; for colour terms, tables that map RGB colour codes to colour names are pervasive web-text. We therefore include control task our experimental setup such that the model cannot succeed using simple memorisation. Rather, success task requires the model to truly perform a conceptual mapping between the ungrounded and grounded representations.
Isomorphism as a Control We use the concept of isomorphism to control for memorisation. tuitively, imagine situation where you are lost in the woods. Once pointed in the direction of north, you instantly know which way is south. However, this ability is not dependent on having been correctly pointed north-if someone would readily infer west to be south. This is because your reasoning depends on your knowledge of the relation between north and south, and between north and the world, rather than on having memorized the "true" grounding of each concept independently.
to incorrectly point east and tell you this
north, you
By the same logic, if model is learning a grounding function, this should be dependent the world in which it is being grounded. For example, in different worlds where the word red grounds different points space, the word blue, by analogy, shares a fixed conceptual relation to the word red. Therefore. it should ground in correspondingly equidistant ways in the two worlds. A model's s ability learn two different grounding functions vs. 9. should not be dependent on what the actual points ground to, as long as the structural relations between concepts in the space are preserved. Further, this should hold for all such isomorphic transformations that preserve the structure of the space, and importantly, it should not hold for random perturbations that distort the structure of the space, since such distortions would break the assumption that the relation between red and blue in ungrounded space is analogous to that in grounded space.
Implementation In the colour domain, since colour concepts exist in 3D world of RGB codes, we rotate each point around a fixed axis by a certain degree to create a new isomorphic world. We repeat this control three times (for 90" 180º and 270" rotations) and average over the rotations in our evaluations. For cardinal directions, we rotate each cardinal concept by 90" in two dimensions, and do this three times and average over rotations. Since the spatial terms exist as pairs, we simply
3

---

Published as a conference paper at ICLR 2022
swap the groundings of alternate terms (e.g., left and right). For random worlds that do not preserve the structure between word forms, we randomly assign a concept name (c.g., red) to a point in the world, and we do this for all concept names and categories obtain random world for each. Figure 2 shows example transformations of colours on rotating by 90", as well as randomly rotating points.
Full Colour Spectrum
Sample Colours
Original World
Original
90" rotation
Random rotation
red
pastel blue
blue
pastel bluc
dark red
160
pastel red
dark blue
pastel blue
20
red
pastel red
bluc
pustel red
dark blue
150
dark
dark blue
dark red
red
f(x)
x(x)
(x
g(x))
blue
Figure 2: Figure shows how colours and modifiers transform in rotated worlds. The leftmost figure shows full 3-D colour space of 367 colours. The three figures on the right, show four sample colours in their original world, world rotated by 90% and a randomly rotated world, showing how the structure of the space preserved or distorted in isomorphic or random rotations respectively.
2.5 EVALUATIONS
Experimental Logic We report model performance in three settings: the original ("true") world (e.g.. that in which red maps to the actual RGB code for red), an average over three rotated worlds (e.g., worlds in which red maps some other RGB code, but relations between colors are pre- served), and a random world (in which relations between mappings are not preserved). If a model is performing the conceptual mapping in the desired way. we expect that performance should be high in the true world and that there should be no significant degradation in performance when moving to rotated worlds. We also expect that performance should be low in the random world.
Metrics When given a prompt, generative LM is free to generate any number of tokens until having generated the EOS token that halts further generation. Since classification tasks usually correspond to labels that contain only few words, the standard approach an entire sequence of text and to then cut off the generation to the first tokens (where typically n < 10). From the prompting mechanism, the model should learn to follow the prompt format to generate one label (e.g., instead of every label in succession), and since it does not receive any gradient updates during this learning, there is no incentive for it to predict all related labels within a n-token generation. We see that this is true in practice and show example generations in Appendix We set n = and then report the following metrics.
let the model generate
TOP-1 ACCURACY the ground-truth answer or any substring thereof lies in the generated answer (i.e., the first n tokens of the full generation), the model gets perfect accuracy. If the ground-truth answer does not exist in the generated answer, or exists in the generation outside of the cutoff limit, the model gets 0 accuracy. For example, for the ground truth answer deep tuscan red, if the model generated answer is tuscan red or red the model gets a perfect accuracy, but if the model generated answer deep red or wine or vermilion. the model gets an accuracy of 0. We also compute the same metric using exact match (i.e., the model is only correct if it generates exactly deep tuscan red). We find the values are lower but the trends are the same; see Appendix ??..
TOP-3 ACCURACY This metric is analogous to Top-1 except that instead of only considering the most probable generation from the model, we collect the second and third most probable answer sequences as well. The model gets perfect accuracy if the correct answer exists in any of the three generated answers. If the correct answer does not exist in any of the generated answers, exists in any of the generations outside of the cutoff limit, the model gets an accuracy of Again, see Appendix ?? for results using exact match
4

---

Published as a conference paper at ICLR 2022
Example Input (20 in-context-learning examples followed by prompt)
Example Model Outputs
World:
World:
Norld:
Norld:
GPT-2 (124M)
[0. 0.]
[0. 0. 0.1
[1. 0.1
[0.
0.1
[0. 0. 0.]
[0. 0.]
world
P=0.09
[0. 0. 1.]
[0. 1.]
[0. 0. 1.]
[C. 0.]
0. 0.11
P=0.08
Answer: right
Answer: right
[0. 0. 0-]
[o. 0.]
P=0.01
[0. 0. 0.]
[o. 0.
World:
World:
Answer: right
Answer: left
[1. 0. 0. 0-]
[o. 0.]
GPT-3 (175B)
Answer: left
[1. 0.]
World:
World:
[o. 0.]
[o. 1. 0. 0.] Answer: left
[1. 0. 0.]
left right leftmost
p=0.20
...13 more...
Answer: left
0. 0-1
P-0.11
Answer:
P-0.01
Figure 3: Figure shows example worlds and groundings for the spatial domain. The left panel shows example grid worlds where the location of the denotes a point grounded in the world, with corresponding text instantiation of that particular concept. The two panels the right show example outputs from the smallest and largest models. We show the top three most probable words from each model along with the probability of that word. When computing Top-1 accuracy, only consider the first, however, we consider all 3 when computing Top-3 accuracy.
GROUNDING DISTANCE For analysis purposes (3.3). we wish to assess how far off models are in cases where they wrong. For this, need quantify the distance between the model's 'ss predicted answer and the ground truth answer. For example, in the colour domain, the distance between two points can be computed as the Euclidean distance between two RGB codes. For every answer generated by the model (e.g., the word pink in the colour domain), if the answer is an acceptable grounded term that exists in the world, we can compute its distance to the true grounding (e.g., the colour red). However, if the generated answer was an unrelated word (e.g., the word cat) that does exist the same domain, no distance can be computed in the world. Therefore, we calculate distance metric the Euclidean distance between two points in space when the generated answer falls in the domain. When the generated answer does not fall the domain, we set the distance to a number significantly higher than the largest distance between two in-domain concepts. We provide equations and details on calculation this metric in Appendix C
Baselines Given that the language models are free to generate any word that exists in their vocab- ulary potential answer, choose two random baselines over vocabulary words against which to compare model performance. We use R-IV (i.e., random in- v--obbuullr) to denote a baseline that randomly selects from among all words in the model's vocabulary. We use R-ID (i.e., random in- domain) to denote a baseline that randomly selects from amongst only the in-domain words (e.g., from colour terms); this is 6, 8 and 367 words respectively for the spatial, cardinal and colour cate- gories. Note that the generative LMS do not have such domain restriction over words, as they are free choose any token from the full vocabulary (like R-IV).
RESULTS
3.1 GENERALISATION TO UNSEEN WORLDS
Our first investigation looks into how well models generalise known concepts to unseen worlds. For example, if a model has seen few examples of concept (such as left) depicted in some grid worlds, can it correctly identify an instance of tleft in a different grid world (e.g., one with a different size or orientation)? Note that can only conduct this type of evaluation in the spatial and cardinal domains, since, for the colour domain, there is only ever one world, with one grounding for each concept (e.g.. the colour red has exactly grounding to 3-digit RGB code in 3-D RGB space).
Data We create prompts that include 20 examples of grounded concepts in set of grid worlds. For each domain (e.g., cardinal directions that contain 8 concepts, and spatial terms that contain pairs of 2 concepts), Then, we append a held-out grid world to the end of the prompt and evaluate whether or not the models generate the correct concept label for these unseen worlds. F example prompts
include (roughly) equal sample of concepts among these 20 examples.
5

---

Published as a conference paper at ICLR 2022
Top-1 Accuracy
Top-3 Accuracy
Spatial
Cardinal
Spatial
Cardinal
Orig.
Rot.
Rand.
Orig.
Rot.
Rand.
Orig.
Rot.
Rand.
Orig.
Rot.
Rand.
R-IV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
R-ID
0.16
0.16
0.16
0.13
0.13
0.13
0.16
0.16
0.16
0.13
0.13
0.13
124
0.11
0.10
0.10
0.13
0.12
0.11
0.23
0.21
0.10
0.25
0.24
0.10
355 M
0.12
0.12
0.10
0.11
0.14
0.10
0.24
0.25
0.15
0.23
0.14
0.12
774 M
0.08
0.09
0.10
0.11
0.12
0.11
0.12
0.19
0.14
0.18
0.17
0.11
1.7 B 0.10
0.11
0.11
0.10
0.11
0.10
0.11
0.18
0.15
0.12
0.12
0.13
175 B
0.45
0.44
0.16
0.46
0.18
0.76
0.19
0.88
0.76
0.21
Table 1: Table shows evaluations for generalisation unseen worlds. Rows show model sizes for GPT-2 models (124M to I.5B) and the 175B GPT-3 model. Columns show metrics for the original, rotated and random worlds for each of the concept categories. We report both Top-1 and Top-3 accuracy in the first and second columns of each rotated world.
given to the model and example generations from three different models. We report results averaged over all generations in Table In general, see the desired trend in which model performance on the original and rotated worlds is well above that on the random world. We do not see consistent or significant performance degradation when moving from the original to the rotated world, suggesting performance not due to simple memorisation. Comparing across models, we see that the smaller models struggle to even learn concepts that taught them in few-shot manner. For example, for the spatial category, the performance of the smallest model is below that of a baseline which guesses randomly among in-domain words, suggesting the model even fails learn the general domain of the task (discussed more in Section 3.3). In contrast, the largest model (GPT-3) has 45% Top-1 accuracy and a 76% Top-3 accuracy for the spatial category.
3.2 GENERALIZATION TO UNSEEN CONCEPTS
Our primary interest here in the model's ability to map ungrounded conceptual structure to grounded world. Specifically, we want to see that the model is able to ground an entire conceptual domain when only taught how to ground a small subset the points in that domain. To test this, we show models example concepts within a sub-space of the domain (e.g., north, east), while holding out other concepts (e.g., south, west). then test them on instances of held-out concepts. Figure 6) depicts this setup in the colour domain: train the model using primarily shades of red, but test on shades of blue.
Example Input (60 in-context-learning examples followed by prompt)
Example Model Outputs
RGB: (48, 213, 200
RGB: (220, 20, 60)
RGB: (0, 0, 128)
GPT-2 (124M)
Answer: orange
Answer: crimaon
color black rgb
B=0.
p=0.08
Training Examples
GPT-3 (175B)
navy
P-0
dark blue
P=0.13
blue
Primary Secondary Colours
57 Colours Within Sub-space
red, blue, yellow, green, orange, violet dark red, crimson, fuchsia, rust, bright red..
Figure shows example worlds and groundings for the colour domain, where the prompt contains colours within sub-space: i.e., training on primary and secondary colors plus shades of red, but then testing navy blue. The left panel shows the full sees. The right shows example outputs from the smallest and largest models, with the top three most probable words from each model along with the probability of that word.
Figure 4:
training examples the model

---

Published as a conference paper at ICLR 2022
Spatial
Cardinal
Colours
Original
Rotated
Random
Original
Rotated
Random
Original
Rotated
Random
R-IV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
R-ID
0.16
0.16
0.16
0.13
0.13
0.13
0.00
0.00
0.00
124 M
0.10
0.11
0.04
0.11
0.10
0.05
0.08
0.09
0.03
Top-1
355 M
0.10
0.10
0.04
0.10
0.11
0.06
0.06
0.07
0.04
Accuracy
774 M
0.09
0.11
0.03
0.13
0.12
0.08
0.11
0.09
0.01
1,5
0.14
0.14
0.12
0.13
0.14
0.10
0.10
0.09
0.06
175 B
0.28
0.27
0.13
0.30
0.29
0.08
0.23
0.21
0.11
R-IV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
R-ID
0.16
0.16
0.16
0.13
0.13
0.13
0.00
0.00
0.00
124 M
0.13
0.12
0.07
0.09
0.09
0.08
0.06
0.05
0.04
Top-3
355 M
0.24
0.17
0.15
0.19
0.17
0.10
0.14
0.11
0.12
Accuracy
774 M
0.19
0.24
0.12
0.17
0.15
0.11
0.15
0.16
0.14
1.5 B
0.32
0.29
0.20
0.21
0.20
0.14
0.19
0.18
0.16
175
0.64
0.65
0.21
0.60
0.61
0.09
0.36
0.13
Table 2: Table shows evaluations for generalisation to unseen concepts for the sub-space split of training/test data. Rows show model sizes for GPT-2 models (124M to 1.5B) and the 175B GPT-3 model. Columns show metrics for the original, rotated and random worlds for each of the concept categories. We report both Top-1 and Top-3 accuracy. R-IV refers random baseline that ran- domly selects a word out the total vocabulary, R-ID refers to a baseline that randomly selects a word out of the set of words in the concept category.
Data To create sub-spaces for the colour domain, we create prompts that contain primary, secondary colours, and 57 other colours within sub-space, determined in the following way. For certain colour (e.g., red) we consider sub-space in the world to be a space colours that lies within Euclidean distance of 150 from that colour. We create 6 such sub-spaces, centered around each of the primary and secondary colours, and report generalisation results averaged over the 6 sub-spaces. Figure 4 shows the sample of colours within sub-space centered around the colour red, that serve as training samples within the prompt. For the cardinal directions, we show models examples of concepts in sub-space of the world (e.g., north, northeast) and then test them concepts in an entirely different sub-space (e.g.. south, west, southwest). The training split therefore contains worlds annotated with sub-space, and we test the remaining held-out concepts. We do this for all sub-spaces and average over them.
We report results all concept categories in Table As before, be exploiting simple memorisation (evidenced by similar performance in original vs. rotated worlds) and that only the largest models appear capable of getting reasonable performance on the task. That said, the largest GPT-3 model achieves impressive results given the difficulty of the task. For example, it achieves over 40% Top-1 accuracy in the color domain, which requires generating label like violet despite having never seen such label during training (Figure 4)
that models do not appear to
3.3 ERROR ANALYSIS
Given the results above, we investigate "how wrong" models are when they fail produce the expected ground-truth label. One clear trend of the large model is the ability to generate "on topic" (i.e., in-domain) responses, regardless of whether or not those responses are correct. For example, the expected response to an input is "left", model which produces right is wrong in very different way than model that produces nonsensical outputs such as "[O[O" or function words such as "the" (see Figure 3). We that the largest 175B parameter model almost always produces in-domain answers. We evaluate this by checking whether the generation lies the set of related words for that category (e.g., all colours, spatial, cardinal words, respectively). We that the smaller models fail to do this. That is, their generations tend to unrelated words that might have had high prominence (e.g., function words). On evaluating accuracy of generations being "in-domain".

---

dark red Published as a conference paper at ICLR 2022
that the smallest model has only 53% accuracy while the largest has 98% accuracy of generating in-domain answers.
Second, evaluate grounded distance 2.5 to measure the degree of correctness of responses. In the colour domain, the colours dark red and wine are close enough in space that they might be intuitive alternate answers for one another. However, our top-1 and top-3 metrics only assess string matches and do not account for this. Thus, we look at the distance between the colour denoted by the predicted label (when it exists, i.e., when the model generated legitimate colour name) and the colour the model was asked to label. The lower this distance is, the "less wrong" the model's prediction is.
Table 3 reports evaluations measured by grounding distance for the colour domain. For every test instance, we compute the distance between the model predicted grounding and the true grounding as defined in Equation C.3 We then average over all computed distances to report one number that tells us how close, on average, model predictions are the true grounding in the world. We show example visualisations of colours in RGB space that lie within certain distance threshold of each other. We see, for the largest model, the distances between model-predicted groundings from the true grounding are significantly lower than random. Such result suggests that the model's errors are often justifiable and the scores given by our string-matching metrics might be an underestimate of the model's true performance.
True
Predicted G & Distance
124M
355M
774M
1.5B
175B
wine (76.5), light crimson (208.1)
C
328.3
309.5
209.6
190.7
96.3
dark slate gray (144.7)
R-IV
334.9
334,9
334.9
334.9
334,9
light green
beige (126.6), light sea green (129.7)
R-ID
174.9
174.9
174.9
174.9
174.9
cerulean (185.7). 7), violet (262.6)
Table 3: Table shows average distance (lower is better) between model-predicted groundings and the world, averaged over
Table 4: Table shows example model predictions (from the GPT-3 model) and distances from the true groundings in RGB space. The first column shows the true concept while the second column shows model predictions and their distances from the true concept.
true groundings instances in the test set. We see that the largest model has an average distance of predictions sig- nificantly lower than random
DISCUSSION
Our empirical results suggest that very large LMs (specifically, GPT-3), even when trained only on text, learn a conceptual space that can, at least in some settings, be "grounded" using only small number of training examples. The fact that these models succeed even in isomorphic rotated worlds suggests that these models not succeeding via naive memorisation. Rather, this suggests that they may be exploiting something about the conceptual structure of the space learned from text in order map onto a new space that was not explicitly encountered during training.
major limitation of our approach is that there are some grounded concepts (e.g., visual and sen- sory inputs) that cannot be easily encoded text form. By construction, the LMs that we use are restricted to text-only inputs, thus our focus is on domains (e.g., colours and directions) that have well-defined textual representation. This is only a small set of all the potential grounded concepts we would wish to teach LMs. Although many forms of data can be coerced into text for- mat (e.g., we represent color using discrete digits represent RGB space), complex concepts may loose fundamental aspects of their meaning when represented in this way. For example, for color, a coarse-graid granned notion of numeric proximity, derivable from text (Wallace et al. 2019 Naik al.) 2019), may be sufficient to differentiate the concepts we explore, but for more complex visual inputs (e.g.. the output of CNN image encoder), a text-based numeric representation is unlikely to capture the necessary degree of nuance. Future work would need to consider ways of adapting GPT-3-like models to accept non-textual inputs while still exploiting the
conceptual structure.
If such limitations were addressed,
results suggestive of potentially promising way in which text-only training could support general purpose, grounded models meaning. Specifically, results imply that the conceptual space model learns from text might be nearly isomorphic to
8

---

Published as a conference paper at ICLR 2022
what it would learn from interacting in grounded world, and that models can be taught map between those conceptual spaces without requiring explicit grounding for every concept. This is exciting, as domain- general text corpora are readily available, while domain general multimodal corpora-e.g, containing sufficient information on abstract concepts such as emotions (happy) or time (used to)-ti be difficult impossible collect. If models like GPT-3 could be adapted to receive non-texi prompts (as discussed above), our results suggest that the rich conceptual structure such models learn from text could be bootstrapped into powerful grounded models of language.
RELATED WORK
There is significant amount of work that focuses on understanding how LMs represent and reason about concepts, well as work that directly attempts to build models that take text inputs and ground them to elements in the world. We situate our work within these two bodies of literature: one that investigates how LMs understand linguistic phenomena and word meaning, and another, that attempts to situate language in models of the world. We describe each body of work below.
Meaning and Understanding in LMs With the advent of large LMs of increasing orders of mag- nitude, there has been speculation on the capabilities of such models, and whether they truly under- stand the meaning of the words they are learning representations for. Several works that attempt to probe linguistic phenomena in LMs, show that the representations learned by such models encode syntactic dependencies and coreference information (Tenney et al., 2019) and word-sense informa- tion (Chen et al., 2020). Work that investigates the ability of models to form word associations (Hwang et al.] 2020). finds that large LMs can indeed perform such a task; suggesting that pre- trained LMs not only recognize that entities are related, but can differentiate how they are related. Especially relevant to our work is recent work that investigates alignment of language models to colours (Abdou et al., (02T) or state changes (Li et al. 2021). Our work is complementary this prior work, and we specifically ask whether the text space can be reliably mapped onto the grounded space.
Natural Language Grounding There is an increasing amount of work, usually at the intersection of NLP and fields like vision and reinforcement learning, that aims to use natural language to instruct agents about aspects of the world. In the case of vision, this could be to learn correspondences between language descriptions and pixels in an image (Eichenberg et al., [021] TTiipouueelii et al.) 2021. or in the case of RL, to build agents that understand natural language instructions in order to take actions in a world that follow the instruction- for example to navigate to a goal (Artzi & Zettlemoyer 2013; Patel al.). or to solve tasks in different languages (Ku et al. 2020). Most of these tasks focus on training LMs from scratch, however usually with inputs that contain both textual information well as grounded world information. Our work is different in that we attempt to take an LM that was previously trained only on text, and attempt to teach it a concept in the world without re-training it. With only few samples of what the concept grounds to, we investigate how well large LMs can use the structure of language and associations between word forms in order to generalise grounded concepts.
CONCLUSION
This work investigates the extent to which large language models, trained only on text can be taught to map previously learned word forms onto conceptual worlds. We investigate several generative language models in colour and direction domains, represented as discrete grid worlds given to the model as text input. With only a few examples of such grounded instances, although smaller models struggle to generalise from text to grounded concepts, we see that the largest model does indeed learn the space of concepts that we test it on. We analyse where models fail and see that the smallest models often produce random, unrelated outputs, but the errors made by the largest model are quite intuitive, for example, predicting colours very close in space to the true grounding. We discuss the limitations of focusing on text-only inputs teach models grounded concepts, well as the implications of such work for allowing large language models to be mapped, in a data- effa-efficient way, to the grounded world.

---

Published as a conference paper at ICLR 2022
REFERENCES
Mostafa Abdou, Artur Kulmizev, Daniel Hershcovich, Stella Frank, Ellie Pavlick, and Anders Søgaard. Can language models encode perceptual structure without grounding? study in color, 2021.
Yoav Artzi and Luke Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactiony of the Association for Computational Linguistics, 1249-62, 2013. doi: 10.1 2..11622acll__000200 URL /tttps://aclanthhtt www.aoonnhoooto.oo...oooooo wwwphooogy..gggg.o........ 0tt.gy.org//111-000000
Emily M. Bender and Alexander Koller. Climbing towards NLU: On meaning, form, and under- standing in the age of data. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 5185-5198, Online, July 2020. Association for Computa- tional Linguistics. doi: 1/t//////11//220allmmmiiia
BRLhttps:: htwp://wwwwww.bbbb........ www.org//o/ooo..oo
aathology/2020.aar aal.aacc-maannaa maan.annn 463
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. Experience grounds language. In Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing (EMNLP), pp. 8718-8735, Online, November 2020. Association for Computational Linguistics. doi: 10.1000......1 1865/////0200.mmlll/l main. 703. URL wwwptchhwllooooog////00 20200 emnlp-ma mm
703
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Catherine Chen, Kevin Lin, and Dan Klein. Constructing taxonomies from pretrained language models. arXiv preprint arXi::2010.12811 2020.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18100408 2018.
Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabeseu, and Anette Frank. multimodal augmentation of generative models through adapter-based finetuning, arXiv preprint arXiv:2//2.0055 2021.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint 19Xivv1904.097511 2019.
Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jeff Da, Keisuke Sakaguchi, Antoine Bosse- lut, and Yejin Choi. Comet-atomic 2020: On symbolic and neural commonsense knowledge graphs. arXiv preprint arXiv:2010.05953, 2020.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-Across-Roome Multilingual vision-and iarion-and-langusse language navigation with dense spatiotemporal grounding. In Conference on Empirical Methods for Natural Language Processing (EMNLP), 2020.
Belinda Z. Li, Maxwell Nye, and Jacob Andreas. Implicit representations of meaning in neural language models. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 1813-1827, Online, August 2021. Association for Computational Linguistics. doi: 10. 1t..1865/11/222111 10.11655/1122221lallloooo lac.-.on... URL 2021 aeeelaaclloogg.. 143.
William Merrill, Yoav Goldberg, Roy Schwartz, and Noah A Smith. Provable limitations of ac- quiring meaning from ungrounded form: What will future language models understand? arXiv preprint Ttt::2004110809 2021.
Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy, Exploring numeracy in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pp. 3374-3380 Florence, Italy, July 2019. Association for Computational URL ht www. aclweb. org argg/antholoyy/
Linguistics. doi: 10.186533v 1/P19-
P F19-1329
10

---

Published as a conference paper at ICLR 2022
Roma Patel, Stefanie Tellex, and Ellie Pavlick. Learning ground language to temporal logical form.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1-7, 2021.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint Xrxiv:1008 0300,00909, 2015.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference Learning Representations, 2019. URL URL URLLbttps:///peereeeee net. new.//rumfff iid rd=SJZSgnRcKX
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multi- modal few-shot learning with frozen language models. Advances in Neural Information Process- ing Systems, 34, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, arXiv preprint 2017.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. Do NLP models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5307-5315, Hong Kong, China, November 2019.
Association for Computational Linguistics. doi: 10.18653/v1/D19/ 1534. ogy -- 1534
acl web. htt/:/gg///aaaaa wogethoooggy/y111
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transformers: State-of-the-ar natural language processing. arXiv preprint 191.:31910 2019.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In ACL, 2019,
APPENDIX
We provide, as supplementary material, additional information about the data generation, models used, examples of model generations, as well as additional results across all models.
A MODELLING DETAILS
We use GPT-3 model Brown et al. 2020) and four GPT-2 Radford et al. (019) models from the Hugging Face Transformer Wolf et al. (2019) library. Each of these pretrained autoregressive transformer model, trained on the OpenAl Web Wextext corpus, containing around 8 million documents. The top domains by volume WebText are: Google, Archive, Blogspot, GitHub, NYTimes, Wordpress, Washington Post, Wikia, BBC, The Guardian, eBay, Pastebin, CNN, Yahoo!. and the Huffington Post. Individual model parameters and layers are shown in Table S The pretrained models use byte-pair encoding (BPE) tokens Sennrich et al. (2015) to represent frequent symbol sequences in the text, and this tokenisation is performed on all new input prompts to generate text from the model. We report the hyperparameters used by the pretrained model in Table
11

---

Published as a conference paper at ICLR 2022
Parameters
Layers
124M
12
355M
24
774M
36
1,5B
48
175B
96
Table 5: Table shows model architecture details for the GPT-3 and four GPT-2 models we use.
Hyperparameter
Selection
number of samples
nucleas sampling p temperature
0.85
max length
Table 6: Table shows model architecture details for the models
B DATA GENERATION
In this section describe how create prompt data for the concept categories wish to evaluate. We describe what the "worlds" look like for each category, as well as detail on the generation of such worlds and division into train and test splits, for colours in Section B. and for both spatial and cardinal concepts in Section B.2
B. COLOUR CONCEPTS
We draw from existing dataset containing RGB codes associated with colour names, for 367 colours. In this concept category, a grounded realisation of a colour in the world, is simply its RGB code i.e., the point that it grounds to in 3-dimensional colour space. Therefore, all grounded examples in prompts follow the format of RGB: (x, y, z) followed by Colour: concept name where the items in red are replaced with instances RGB codes and names from the dataset of colours useAbdou al, (2021). This gives total of 367 samples of RGB codes paired with colour names. We create training and testing splits for different generalisation evaluations in the following way.
B.1.1 GENERALISATION TO UNSEEN CONCEPTS
Random Split We create prompts to perform this experiment in two ways. The first, as reported in $3.2 creates a "train split" i.e., samples within the prompt, by first selecting the 3 primary and 3 secondary colours to always be in the prompt. We then sample 64 other colours from the set of colours to be part of the train split. The prompt therefore contains 70 samples of the question and answer prefixes followed the RGB codes and colour names. For every sample in the test set, we create new prompt that appends question prefix and the RGB code of that sample to the end of the prompt, followed by the answer prefix (with no answer). For each prompt, the model is then required to generate an answer. Figure shows the random sample of colours the prompt and we report results on these in Appendix Table F.T
Sub-space Split We then perform another experiment to evaluate models ability to learn con- cepts within a sub-space, and generalise to a whole new sub-space. Here, the prompt contains primary and 3 secondary colours, well as 57 other colours that select in the following way- For each of the 6 primary and secondary colours, we consider a sub-space in the world to be space of colours that lies within Euclidean distance of 150 from that colour!] Since the model has the primary and secondary colours, as well as a certain space of the colour world, we wish to evaluate how well the model generalise to
sub-space of the world, by using the colour
We show visualisations of such colour sub-spaces in the main paper in Figure 4
12

---

Published as a conference paper at ICLR 2022
word-form associations. We then report generalisation results averaged over the 6 sub-spaces. Fig- ure 4 shows the sample of colours within sub-space centered around the colour red, that serve as training samples within the prompt. We report results these in the main paper in Table
B.2 SPATIAL AND CARDINAL CONCEPTS
We consider "world" be 2D matrix filled with Os and containing exactly one 1, where the posi- tion of the (i.c., the determining object) refers to the concept of interest (for c.g., a spatial location like "left" cardinal direction like "north". We create grid worlds filled with Os and one 1, of row and column ranges from (1, 8) giving us 672 total grid worlds For each grid world, all grounded examples follow the format of World: [1. 0. O. 0.] followed by Direction: concept name where the items in red based on the location of the 1. We create training and testing splits for different generalisation evaluations in the following way.
replaced with instances of different grid worlds and corresponding concepts,
B.2.1 GENERALISATION TO UNSEEN WORLDS
Our primary investigation here is assess whether the models can generalise to new worlds that differ in size, or location of the determining object. Each of the 672 worlds we create differs from one another in either aspect i.e., either the size of the world different, or, when worlds are of the same size, the location of the is in a different position. Therefore, we randomly sample 20 worlds that contain instances of concepts to serve as part the prompt. The prompt therefore contains a question prefix followed by the grid world a new line, followed by the answer prefix and the concept name. We ensure that there is a roughly equal distribution of concepts in the train split (e.g., if there are concepts split over 20 samples, and then fill the remainder of the prompt by randomly sampling from the concepts). We then create prompt for every sample in the test set by appending the world representation and answer prefix to the end. We report results on this in the main paper in Table
ensure that 2 of each concept exist in the prompt,
B.2.2 GENERALISATION TO UNSEEN CONCEPTS
Since we wish to assess a model's ability to generalise to new concepts, here, we hold out concepts in the test set. Specifically, for every domain contain n concepts (e.g., for cardinal directions), the train split contains an equal distribution of n concepts. We then test on samples that contain the last held-out concept, as well as the seen concepts, to assess how well the model has learned seen concepts, and generalises to unseen concepts. We report results on this in Appendix Table 14, and Figure 7 shows an example split of data that is held-out during test time. We note that this is random split, unlike the sub-space split that specifically holds out set of colours based on Euclidean distance.
B.2.3 GENERALISATION TO SUB-SPACES
Similar to the colours, for the cardinal directions, we also report in the main paper, results on a sub- space generalisation task. Specifically,
show models examples of concepts in one sub-space of the world (e.g., north, east, northeast) and then test them on concepts in a different sub-space (e.g., south, west, southwest). We do this for all sub-spaces and average them. As seen in Table 11, we see that the largest model can perform this task to some degree, achieving 60% accuracy.
B.2.4 GENERALISATION TO COMPOSITIONS
Here, we specifically test model performance when holding out any instance of compositions of directions. Therefore, the training split contains examples of worlds annotated with the concepts north, south, east, west, and the test split contains examples of compositions of concepts i.e., north- east, southeast, northwest, southwest. We report results Table 10, and we see that here, even the largest model fails to perform the task. This means that when models have never any instance of a composition, they do not perform these compositions themselves, however, as seen in Table
We exclude grid worlds where the position of the 1 is in ambiguous position for that concept category c.g., directly in the center for cardinal task, in the middle row for a spatial task.
13

---

Published as a conference paper at ICLR 2022
10, and earlier results, when the training data contains examples of compositions, models can indeed generalise to completely new compositions.
Grounded Concept
Size
Example words
Cardinal directions
north, south, east, west northeast, southeast, northwest, southwest
+ compositions
Spatial directions
Icft, right, up, down, top, bottom, above, below
Colours
107
red, blue, yellow, green, violet, aqua, wine, charcoal, brass, cobalt.. antique brass, antique fuchsia, antique ruby, antique white baby blue, baby blue eyes, baby pink
+ antique modifier
+ baby modifier
bright modifier
bright cerulean, bright green, bright lavender, bright maroon burnt orange, burnt sienna, burnt umher
burnt modifier
copper modifier
copper crayola, copper penny, copper red, copper rose, pale copper dark blue, dark brown, dark byzantium, dark cerulean, dark chestnut.. deep carmine. deep carmine pink, deep carrot orange, deep cerise.. electric blue, electric crimson, electric cyan, electric indigo.. fluorescent orange, fluorescent pink, fluorescent yellow french beige, french blue, french lilac, french lime, french raspberry. light apricot, light blue, light brown, light carmine pink, light coral.. medium aquamarine, medium blue, medium carmine.. old gold, old lace, old lavender, old mauve, old pale aqua, pale blue, pale brown, pale carmine, pale cerulean.. dark pastel blue, dark pastel green, dark pastel purple, dark pastel red.. persian blue, persian green, persian indigo, persian orange
+ dark modifier
43
deep modifier
17
+ electric modifier
14
+ fluorescent modifier
french modifier
+ light modifier
24
medium modifier
21
+ old modifier
+ pale modifier
20
+ pastel modifier
16
+ persian modifier
+ rich modifier
rich black, rich brilliant lavender, rich carmine, rich lavender.
rose modifier
12
copper rose, french rose, old rose, persian rose, bonbon
royal modifier
royal azure, royal blue web, royal fuchsia, royal purple, royal yellow vivid auburn, vivid burgundy, vivid cerise, vivid tangerine, vivid violet
+ vivid modifier
Table shows the grounded concepts aim to teach language models. The first column shows the category of related concepts that models might have learnt relational meaning for, the second shows the number of such terms within each category, and the third shows example words.
Table 7:
C EVALUATION METRICS
In this section we describe our evaluation metrics in detail, with examples of how certain generations might be scored by each metric.
C.I SUBSTRING MATCH
For every test instance, there is exactly one string answer that is correct (for e.g., the words "left" or "northeast" or "electric blue"). However, language models are free to generate any number of tokens given prompt-ie.. they might generate exactly one token, or up to 100 tokens for any given prompt. We consider the first 5 tokens generated by the model to be the generated answer- and consider a substring match metric to be one that looks whether or not the model generated answer lies in the ground truth answer. To make this clearer, we provide examples below, denoting a model generatee answer in blue and the ground-truth answer in green. By the substring metric, electric green for electric blue would give an accuracy of O, however electric or green would give an accuracy of 1. In practice, we do not see (the large) models generate half-answers (c.g.. simply saying electric). Similarly green for dark green has an accuracy of 1, but dark pink for dark green would have an accuracy of O.
14

---

Published as a conference paper at ICLR 2022
Test Accuracy
124M
355M
774M
1.5B
179B
(a)
Top-1 1 Accuracy
124M
355M
774M
1.5B
175B
Test Accuracy
(b) Top-3 Accuracy
Figure 5: Figure shows performance curves of models on increasing the number of samples in prompt. From left right, we see the 124M, 355M, 774M 1.5B and 175B parameter models, on increasing the number of samples given to the model to the maximum prompt size. We see that only samples are not enough for models to learn the task, but past a certain threshold (20 samples), models do not have a significant increase in performance on the same test set.
C.2 EXACT MATCH
For this metric, the model only gets perfect accuracy if the model generated answer exactly matches the ground-truth answer (after stripping both for any trailing whitespaces). Therefore saying green
for dark green or electric green for light electric green would have an accuracy of
C.3 GROUNDING DISTANCE
Equation C.3 shows our calculation of distances in the world. For certain concept category C (for example, all colour names existing in the dataset), let e1 be the model predicted concept name and be the true grounding. The grounding distance is the Euclidean distance between the two points when the predicted concept does exist in the space of concepts in the world, and is set to an arbitrarily high number (here, 500, which is higher than the maximum distance between any two points in space) when the predicted concept is some other word does not fall the concept category.
o2) {
Vlc -22 )2x)² (Cly C2g) 2 + (c1= - )2 for E C
500 for 6 & C
D PROMPT SIZE
The size of the input prompt model depends on its maximum content length, which is hyper- parameter fixed before training the model to completion. The question- answer pairs that we use to teach models grounded concepts differ in their lengths based on the concept category. For example, since the spatial and cardinal concepts require grid worlds that could be up to 10 rows and columns wide, prompt for these categories might contain fewer number of samples. Since the colour groundings are 3-digit RGB codes, a larger number of samples can be given in a prompt. We report accuracy curves of models when given an increasing number of samples the prompt in Appendix
D We that past 20 samples for spatial terms, and 60 samples for colours, models have signif- icant increase in performance. When we report results Tables ?? and Table T we report numbers for fixed number of samples in prompt (e.g., 20 60 respectively) for all models.
Interestingly, once we get past a certain number prompts, there is no significant increase in model performance the task. This result hints at two things. First, the larger models seem learn
15

---

Published as a conference paper at ICLR 2022
task, and generalise to new samples, with only small number of sample when fine-tuned in few-shot prompting regime. This seems to hold for the smaller models as well i.e., although they do not learn the task well, increasing the number of samples within a prompt does not significantly increase model performance. Second, the significant difference performance based on model size hints the fact that the limiting factor for good performance is the number of parameters of a model. More concretely, the generalisation extent of each model seems to max out at some number of input prompts, with trend of increasing performance on increasing model size. Therefore, larger model (than the ones we have here) might be required in order to achieve better grounded generalisation.
E
MODEL GENERATIONS
We show example generations from the model in Table
Category
Prompt
True
124M
355M
774M 1.5B
175B
Cardinal
World: [0. 0. 0. 0.]
southeast
world
direction
the south southeast
[0. 0. 0. 0.]
[0. 0. 0. 1.]
World: [0. 0. 1. 0.]
north the direction world south north
[0. 0. 0. 0.)
Spatial
World: [1. 0. 0. 0.]
left
the world an left left
[0. 0. 0. 0.]
[0. 0. 0. 0.]
World: [0. 1.]
right
and world to world right
[0. 0.]
[0. 0.]
[0. 0.1
Colour
RGB: (123, 0, 43)
light blue
color
the
rgb
blue
RGB: (3, 43, 100)
magenta
hex
red
color
green
red
Table Table shows example generations from each the models, cut off for the first five words. The first column shows the concept category while the second column shows the last portion of the prompt given to each model, for which they are required to generate an answer. The last 6 columns show the ground-truth answer, and predicted answers from each of the models respectively.
F PRE-TRAINING DATA
F.1 WHAT HAVE MODELS SEEN IN THEIR TRAINING DATA THAT MIGHT BE RELATED TO GROUNDING?
Pre-trained language models have been trained on large text corpora available the internet- diverse source of information that covers many domains. There is speculation, that the remarkable generalisation performance of these models stems from having seen instances of the task related information at some point in their training data. However, this data has not been made public, which makes this a hypothesis that is hard confirm. In our domain of grounded concepts well, it is unclear what aspects of similar data the model might have had access to during training, or what inductive biases from certain types of data might have learnt, that allow good performance on grounded tasks. We attempt to assess this in the following way. We consider all the concept words every concept category, and all the world representations, and consider these the prompts for the models. For each word, we wish to assess the distribution of text generated, which should, in some sense, reflect the distribution text seen during training. For example, or the world representations (i.e., 2D array), assess how often the model might use spatial cardinal term when simply prompted with an array, thus giving us insights into how often the training data might have contain such an arra in close proximity to such word. Similarly, for the colour domain, evaluate how often the model predicts a correct colour name when prompted with an RGB code. We see that when models simply prompted with world representation, they tend generate similar-
16

---

Published as a conference paper at ICLR 2022
looking world representations, instead of conceptual words (such as "left") that might be associated with it. When prompted with RGB codes, the GPT-3 model only has a 7.3% accuracy of correctly generating the corresponding colour name in 5-token sequence after the prompt.
Cardinal
Cardinal
Original
Rotated
Random
Original
Rotated
Random
R
0.00
0.00
0.00
R 0.00
0.00
0.00
R
0.13
0.13
0.13
R
0.13
0.13
0.13
124 M
0.03
0.02
0.02
124 M
0.11
0.10
0.05
Top-1 Accuracy
355 M
0.03
0.04
0,03
355 M
0.10
0.11
0.06
774 M
0.04
0.03
0.02
774 M
0.13
0.12
0.08
1.5 B
0.07
0.06
0.05
1,5
0.13
0.14
0.10
175
0.12
0.11
0.10
0.30
0.29
R-IV
0.00
0.00
0.00
R-IV
0.00
0.00
R-ID
0.13
0.13
0.13
R-ID
0.13
0.13
0.13
0.03
0.04
0.04
M
0.09
0.09
0.08
Top-3 355 M
0.06
0.05
0.04
355 M
0.19
0.17
0.10
Accuracy
774 M
0.05
0.07
0,05
774 M
0.17
0.15
0.11
1.5 B
0.11
0.10
0.07
1.5 B
0.21
0.20
0.14
0.23
0.21
0.17
175
0.60
0.61
Table Table shows evaluations the cardinal directions domain, where show models ex- amples of single directions (north, south, east, west) and test them on compositions of directions (northeast, southeast, northwest, southwest) that before. Rows show model sizes for GPT-2 models (124M to 1.5B) and the 175B GPT-3 model. Columns show metrics for the original, rotated and random worlds for each of the concept categories. We see that all models fail on this task i.e., when they have never seen compositions of terms before, they never predict them at test-time.
Table 10: In this table, we show models exam- ples of concepts within sub-space (north, east, northeast) and test them on concepts in a differ- ent sub-space (south, west, southwest) that were never seen before. Rows show model sizes for GPT-2 models (124M to 1.5B) and the 175B GPT-3 model. Columns show metrics for the original, rotated and random worlds for each of the concept categories. We see that the largest model can indeed perform on this task i.e., even on only having seen sub-space of the world (along with compositions) it can generalise new sub-space and compositions.
G EXPERIMENTS WITH MASKED LANGUAGE MODELS
The results in the main paper focus GPT-2 and GPT-3 language models i.e., autoregressive left-to- right language models that show impressive capabilities for in-context learning. Here, we provide comparisons to masked language model, specifically BERT model (Devlin al.) 2018). For our implementation we use BERT-base model containing 110 million parameters. Similar to the GPT-x experiments for in-context learning with prompts, we "teach" the BERT model the space on concepts in the following way.
BERT zero-shot (no-context) In this setting, for every sample in the test (e.g., an RGB colour), we create an input prompt that masks the name of the colour, and ask the model fill in the colour name. The prompt, therefore, would look like RGB: (255, 0, 0) Colour: <mask>. For the same test set, we evaluate the Top-1 and Top-3 accuracy of the model filling in the mask with the correct colour, using the same substring-man match metric.
BERT zero-shot (in-context) In this setting, we add several samples into the context the model, and ask the model to fill in the masked token with the colour name, for all samples in the test set. Since the context-width of BERT is smaller than the GPT models, we can only include smaller number of samples (around 7).
17

---

Published as a conference paper at ICLR 2022
Spatial
Cardinal
Colours
Original
Rotated
Random
Original
Rotated
Random
Original
Rotated
Random
R-IV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
R-ID
0.16
0.16
0.16
0.13
0.13
0.13
0.00
0.00
0.00
124 M
0.10
0.11
0.04
0.10
0.10
0.03
0.09
0.09
0.10
Top-1
355 M
0.10
0.11
0.06
0.08
0.07
0.03
0.13
0.12
0.10
Accuracy
774 M
0.09
0.11
0.03
0.10
0.09
0.01
0.11
0.12
0.09
1,5
0.14
0.14
0.12
0.13
0.12
0.11
0.16
0.15
0.11
175
0.28
0.27
0.13
0.29
0.28
0.15
0.42
0.41
0.14
R-IV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
R-ID
0.16
0.16
0.16
0.13
0.13
0.13
0.00
0.00
0.00
124 M
0.13
0.12
0.07
0.10
0.10
0.08
0.16
0.14
0.10
Top-3
355 M
0.24
0.17
0.15
0.20
0.18
0.13
0.25
0.24
0.10
Accuracy
774 M
0.19
0.24
0.12
0.21
0.21
0.17
0.23
0.19
0.11
1.5 B
0.32
0.29
0.20
0.29
0.27
0.20
0.29
0.32
0.23
175
0.64
0.65
0.21
0.63
0.16
0.12
Table 11: Table shows evaluations for generalisation to unseen concepts for random split of train- ing/test data. Rows show model sizes for GPT-2 models (124M to 1.5B) and the 175B GPT-3 model. Columns show metrics for the original, rotated and random worlds for each the concept categories. We report both Top-1 and Top-3 accuracy. R-IV refers to random baseline that randomly selects a word out of the total vocabulary, R-ID refers to a baseline that randomly selects a word out of the of words in the concept category.
Spatial
Cardinal
Colours
Original
Rotated
Random
Original
Rotated
Random
Original
Rotated
Random
R-IV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
R-ID
0.16
0.16
0.16
0.13
0.13
0.13
0.00
0.00
0.00
124 M
0.00
0.00
0.00
0.01
0.00
0.00
1.00
0.00
1.00
Top-1
355 M
0.00
0.01
0.00
0.02
0.02
0.01
0.02
0.03
0.01
Accuracy
774 M
0.00
0.00
0.01
0.00
0.01
0.00
0.02
0.01
0.01
1.5
0.01
0.00
0.01
0.03
0.00
0.04
0.02
0.00
175 B
0.12
0.14
0.07
0.14
0.15
0.06
0.19
0.18
0.08
R-IV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
R-ID
0.16
0.16
0.16
0.13
0.13
0.13
0.00
0.00
0.00
124 M
0.01
0.00
0.01
0.00
0.02
0.00
0.03
0.03
0.01
Top-3
355 M
0.04
0.03
0.00
0.04
0.04
0.01
0.07
0.05
0.03
Accuracy
774 M
0.05
0.04
0.04
0.03
0.04
0.00
0.05
0.05
0.02
1.5
0.09
0.09
0.07
0.07
0.05
0.01
0.10
0.09
0.04
0.24
0.25
0.21
0.13
0.36
0.11
Table 12: Table shows evaluations for generalisation to unseen concepts using an exact substring- match evaluation criteria. Rows show model sizes for GPT-2 models (124M to 1.5B) and the 175B GPT-3 model. Columns show metrics for the original, rotated and random worlds for each of the concept categories. We report both Top-1 and Top-3 accuracy. R-IV refers random baseline that randomly selects word out of the total vocabulary, R-ID refers to a baseline that randomly selects word out of the of words the concept category. This table is equivalent Table F. except using an exact-match metric instead of substring substring-match. match. This table equivalent to Table except using exact-match metric instead of substring-mater
BERT zero-shot (fine-tuned) In this setting, fine-tune the BERT model 67 samples (the same number of samples that the GPT models got as input in prompt) and then report performance when tested all the colours in the set.
18

---

Published as a conference paper at ICLR 2022
Table 13 shows results of the BERT models when tested held-out sub-space of colours and
Table 14 shows results when tested on a randomly held-out split of colours.
Colour
Original
Rotated
Random
R
0.00
0.00
0.00
R#
0.13
0.13
0.13
124 M
0.08
0.09
0.03
Top-1
355 M
0.06
0.07
0.04
Accuracy
774
0.11
0.09
1.5
0.10
0.09
0.06
175 B
0.23
0.21
0.11
BERT zero-shot (no-context)
0.09
0.08
0.08
BERT zero-shot (in-context)
0.08
0.07
0.05
BERT fine-tuned
0.10
0.09
0.07
R-IV
0.00
0.00
0.00
R-ID
0.13
0.13
0.13
124
0.06
0.05
0.04
Top-3
355 M
0.14
0.11
0.12
Accuracy
774 M
0.15
0.16
0.14
1.5 B
0.19
0.18
0.16
175 B
0.34
0.36
BERT zero-shot (no-context)
0.10
0.09
0.06
BERT zero-shot (in-context)
0.11
0.09
0.07
BERT finc-tuned
0.11
0.10
0.09
Table 13: Table shows evaluations for the BERT model when tested in 3 different ways. in the colour domain, where show models examples of colours in sub-space (e.g., all colours in sphere of Euclidean distance 150 from red) and test them on different subspaces i.c., all remaining colours. Rows show model sizes for GPT-2 models (124M to 1.5B) and the 175B GPT-3 model. Columns show metrics for the original, rotated and random worlds for each of the concept categories.
Example Input (60 in-context-learning examples followed by prompt)
Example Model Outputs
RGB: (48, 213, 200)
RGB: (220, 20, 60)
RGB: (143, 0, 255)
GPT-2 (124M)
Answer: orange
Answer: crimson
color
p=0.14
black
p-0.08
All 60 Training Examples
p=0.02
GPT-3 (175B)
1ight pink
P-0.24
purple
P-0.15
pink
p=0.08
57 Randomly Selected Colours red, blue, yellow, green, orange, violet dark red, davy, electric green, light yellow, fuchsia..
Primary and Secondary Colours
Figure 6: Figure shows example worlds and groundings for the colour domain. The left panel shows example RGB codes and associated colour names, while the right shows example outputs from the smallest and largest models, with the top three most probable words from each model along with the probability of that word. When computing Top-1 vs. Top-3 accuracy, we consider only the first vs. all 3 outputs respectively.
19

---

Published as a conference paper at ICLR 2022
Colour
Original
Rotated
Random
R
0.00
0.00
0.00
R
0.00
0.00
0.00
124 M
0.09
0.09
0.10
Top-1
355 M
0.13
0.12
0.10
Accuracy
774 M
0.11
0.12
0.09
1.5
0.16
0.15
0.11
175 B
0,42
0.41
0.14
BERT zero-shot (no-context)
0.03
0.05
0.04
BERT zero-shot (in-context)
0.04
0.05
0.05
BERT fine-tuned
0.08
0.09
0.10
R-IV
0.00
0.00
0.00
R-ID
0.00
0.00
124 M
0.16
0.14
0.10
Top-3
355
0.25
0.24
0.10
Accuracy
774
0.23
0.19
0.11
1.5 B
0.29
0.32
0.23
175 B
0,63
0.63
0.12
BERT zero-shot (no-context)
0.10
0.09
0.09
BERT zero-shot (in-context)
0.09
0.09
BERT fine-tuned
0.10
0.11
0.09
Table 14: Table shows evaluations for the BERT model when tested in different ways. in the colour domain, where show models examples of colours from random split of 70 colours and test them on all remaining colours. Rows show model sizes for GPT-2 models (124M to 1.5B), the 175B GPT-3 model, and the BERT-base model when given samples in 3 different ways. Columns show metrics for the original, rotated and random worlds for each of the concept categories.
Spatial
Cardinal
Colours
Original
Rotated
Random
Original
Rotated
Random
Original
Rotated
Random
R-IV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
R-ID
0.16
0.16
0.16
0.13
0.13
0.13
0.00
0.00
0.00
124 M
0.00
0.01
0.00
0.01
0.00
0.00
0.02
0.01
1.00
Top-1
355 M
0.02
0.01
0.00
0.03
0.02
0.01
0.03
0.03
0.01
Accuracy
774 M
0.02
0.02
0.01
0.03
0.02
0.00
0.07
0.06
0.03
1.5
0.06
0.07
0.07
0.07
0.05
0.09
0.05
175 B
0.09
0.09
0.04
0.10
0.09
0.05
0.19
0.20
0.08
R-IV
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
R-ID
0.16
0.16
0.16
0.13
0.13
0.13
0.00
0.00
0.00
124 M
0.01
0.01
0.00
0.02
0.03
0.01
0.05
0.05
0.02
Top-3
355 M
0.04
0.04
0.02
0.03
0.04
0.02
0.06
0.05
0.04
Accuracy
774 M
0.04
0.05
0.04
0.03
0.04
0.00
0.10
0.09
0.03
1.5 B
0.09
0.09
0.04
0.09
0.10
0.05
0.12
0.11
0.06
175
0.30
0.31
0.10
0.35
0.32
0.13
0.26
0.25
0.10
Table 15: Table shows evaluations for generalisation to unseen concepts using an exact substring- match evaluation criteria. Rows show model sizes for GPT-2 models (1 (124M to 1.5B) and the 175B GPT-3 model. Columns show metrics for the original, rotated and random worlds for each of the concept categories. We report both Top-1 and Top-3 accuracy. R-IV refers randomly selects a word out of the total vocabulary, R-ID refers to a baseline that randomly selects a word of the set of words in the concept category. This table is equivalent Table except using an exact- match metric instead of substring-mate match.
random baseline that
20

---

Published as a conference paper at ICLR 2022
Top-1 Accuracy
Top-3 Accuracy
Spatial
Cardinal
Spatial
Cardinal
Orig.
Rot.
Rand.
Orig.
Rot.
Rand.
Orig.
Rot.
Rand.
Orig.
Rot.
Rand.
R-IV
0.00
0.00
0.00
0.00
0.00 0.00
0.00
0.00
0.00
0.00
0.00 0.00
R-ID
0.16
0.16
0.16
0.13
0.13
0.13
0.16
0.16
0.16
0.13
0.13
0.13
124 M
0.02
0.02
0.02
0.03
0.03
0.02
0.03
0.02
0.01
0.02
0.02
0.01
355 M
0.03
0.02
0.03
0.03
0.03
0.02
0.04
0.05
0.04
0.04
0.03
0.03
774
0.07
0.05
0.06
0.07
0.04
0.09
0.08
0.08
0.04
1.5 B
0.10
0.11
0.11
0.07
0.08
0.06
0.08
0.07
0.07
0.09
0.08
0.08
175 B
0.39
0.39
0.13
0.38
0.39
0.14
0.49
0.50
0.15
0.58
0.59
0.18
Table 16: Table shows evaluations for generalisation to unseen worlds. Rows show model sizes for GPT-2 models (124M 1.5B) and the 175B GPT-3 model. Columns show metrics for the original, rotated and random worlds for each the concept categories. We report both Top-1 and Top-3 accuracy in the first and second columns of each rotated world. This table equivalent to Table [ except using an exact-match metric instead of substring-match.
21


Reviews:
[{"review_id": "lTkm19IP_bC", "reviewer": "Reviewer_BFAr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the modelling work presented is useful and interesting — and importantly: explained well and coherently. However, I find many statements not thought out as much as they deserve. For one, colours are as relative as left/right. Something red looks dramatically different under white light than it does under other colours of light, and so on. Also see visual illusions where the same shade of grey looks like different colours as a function of where it is in the image. This is why categorical cognition plays a role in colours in humans. The authors could at minimum note this, if they wish. Another example is on the equivocation of \"text\" and \"language\" throughout the manuscript. I also find it a bit strange to say \"north\" in humans is not grounded. I assume it is always grounded to where the Sun is and what hemisphere we are in and so if somebody were to lie to us about where north is, we would eventually figure it out when we saw the movement of shadows, etc. There is a lot of research on what humans indeed do here, so some literature review could be done if claims need to be made about actual human behaviour. \n\nOverall, however, I find the modelling here very interesting and compelling. I like that the authors explored a pre-trained model. Perhaps providing a coherent definition of \"grounded\" versus \"ungrounded\" might be useful to help formalise the distinction as used by the authors. ", "summary_of_the_paper": "The authors present their work looking at how text-only input can create what they call a \"grounded\" model. The results support the idea that a small input of text-only data can give rise to the models behaving in such ways as to generalise over things like left and right or colours. This indicates, the perhaps unsurprising conclusion, that language is able to, on a meta level, encode some aspects of groundedness, as defined by the authors, and the perhaps more impressive conclusion that the models can indeed tap into the latent structure found in language using a few examples and thereby generate useful isomophisms. ", "summary_of_the_review": "The work is good quality and well-explained, but could benefit from slightly more details on the rationale behind some jargon and claims about human cognition.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ua85QJpL2aO", "reviewer": "Reviewer_qf1D", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n* I think this is a nice set of experiments. The choices to use both unseen worlds and unseen concepts was good. The rotated baseline is also a great idea, and frankly I find the results on rotated spaces quite surprising, especially when it comes to color. I would have expected a serious performance decrease. It speaks very well to the model's ability to construct an alignment between the spaces.\n\n* I think the framing and conceptual situating this paper does is pretty good. The discussion of the difficulty of \"grounding\" complex concepts that can't be encoded easily in text is important and I'm glad it was included. The principal theoretical complaint that I foresee people having regarding the overall framing of the paper is that it's not \"really\" testing grounding because of the way these the concepts are serialized into text, but I think the authors are very clear about how this approach relies on the faithful encoding of a space into text and that it's a serious limitation particularly when it comes to more complex concepts.\n\n## Weaknesses\n\nI think there are some subtle but pretty important problems with the paper. To fix them, it might be okay to just change the framing and back off from the \"isomorphism\" angle, but that feels like losing the whole point of the paper and leaving it kind of uninteresting. On the other hand, I think maybe adding some tougher experiments and better baselines could patch up the holes and make the paper really nice. Details follow:\n\n* Abdou et al. is mentioned as a data source, but I think deeper comparisons and more credit to that work should be provided. This paper presents itself as providing clear evidence of learning \"isomorphic\" conceptual spaces from text alone, I think the Abdou et al. paper makes a case that is very similar (though meaningfully different, yes) and stronger than this one. At the very least, these results should be put in context of theirs (especially as their data is being used). In particular the Abdou et al. paper already provides pretty good evidence of conceptual grounding of colors using representational similarity analysis and linear mappings on color representations extracted from the LMs. These methods have some underlying assumptions but it seems to me that those assumptions are much weaker than the ones underlying the method used in this paper, which relies on the few-shot learning capabilities of the language model. More on that in the next point.\n\n* I don't think the R-IV baseline is more fair than the R-ID one, as claimed in Sec. 2.5. This paper is about producing the *correct* term for the provided grounding, which is a different problem from producing an *in-domain* term. Generating an in-domain term is *a priori* significantly easier (e.g., by hard-coding it). By including the requirement to generate an in-domain term, these experiments conflate the two problems. So GPT-2, for example, does badly because in the few-shot format, it often can't decide on any answer with high enough confidence to make it to the top of the beam (see in Figure 3 how GPT-2 skips giving an answer at all and just moves on to the next prompt). This is a classic example of \"boring\" sequence generation that happens for sequence models over high entropy output distributions (the literature on this focuses generally focuses on beam search, but I think the argument can reasonably apply in cases like this; see [Holtzman et al. 2019](https://openreview.net/forum?id=rygGQyrFvH), [Stahlberg and Byrne 2019](https://aclanthology.org/D19-1331/), [Eikema and Aziz 2020](https://aclanthology.org/2020.coling-main.398.pdf)), which is an orthogonal issue to the grounding problem. The issue here is a combination of that and the mere fact that the smaller GPT-2 models don't make sense of few-shot prompts in the effective way that GPT-3 does — again, arguably a separate issue from grounding. In short, by relying on the few-shot learning paradigm to extract the conceptual structure of concern, we're limited to extracting conceptual structure that can surface _through few-shot learning_ (just like, for example, linear probes can only extract linearly separable features). This ends up being more a measure of the few-shot learning ability of the model than the conceptual grounding capabilities of the model. This is particularly problematic because it makes the baselines (ie, GPT-2 models) seem weaker than they probably should be, and it limits the baselines that can be used. I would point particularly to Abdou et al., who obtained alignments similar in quality to those from masked LMs (BERT, RoBERTa, ELECTRA) using just PMI statistics and fastText embeddings. It's not clear to me if the large models are exhibiting conceptual generalization/grounding abilities much larger than those afforded by such baselines (once we move out of the few-shot learning format).\n\n* To add to this issue, the headline experiment of generalizing to new concepts has at least one big issue that I find concerning. Including *all* n-1 concepts besides the target one, or many such concepts (eg, 60 colors) provides a very strong constraint in the input (in my view contradicting the \"only a small number of examples\" claim in the paper's abstract). While, yes, generalizing to new concepts does demonstrate that the model understands something about the domain, I think it's important that we understand this in graded terms. The question this paper seems designed to address, based on the framing in the introduction and conclusion, is about whether language models can or do construct conceptual spaces that are roughly \"isomorphic\" to grounded ones. In order to test this, of course, we have to construct the isomorphism. But — and this is one of the classic problems with probing — if we are looking at it in terms of set isomorphism (i.e., without extra homomorphism structure — and we are, if we're measuring prediction accuracy), then it is always possible to reconstruct such an isomorphism if we have enough supervision (See [Pimentel et al.](https://arxiv.org/abs/2004.03061)). The question is *how much* supervision we need versus how much is usefully obtainable from the pretrained model, hence approaches like [Voita and Titov (2020)'s](https://aclanthology.org/2020.emnlp-main.14.pdf) information-theoretic probing. So it's not really a black and white question whether there is an \"isomorphic\" conceptual space inside the model, but a graded one. And when we talk about things in terms of an isomorphism, what we're really implying is that the vast majority of the information required to construct the isomorphism is easily accessible from the model representations, so less supervision is required. This is the basic thinking behind the idea of a \"minimal grounding toehold\" which has been discussed as a prerequisite to grounding language models in conceptual spaces. Some of this discussion in the NLP community subsequent to Bender and Koller's 2020 paper was catalogued in [a blog post](https://blog.julianmichael.org/2020/07/23/to-dissect-an-octopus.html) that goes into a lot more detail on the issue. That post proposed a color experiment extremely close to what is done in this paper, but an important difference in their proposal is that the input used by the LM to reconstruct the grounded lexicon is the *minimal* size that is required to span the conceptual space (i.e., constraining only the *necessary* degrees of freedom). One might use light/dark, red/green, and blue/yellow as the three basis dimensions to align with CIELAB, or even just to try red, green, and blue. I think such a minimal experiment would be required to effectively address the isomorphism question. This paper, by providing many more inputs, is doing something of a mix of probing (because of the few-shot/supervision aspect) and behavioral testing (because it's relying on inherent generalization behavior in the LM distribution).\n  * [Edit Nov 3: One extra thought.] The inclusion of a bunch of distinct concepts in the input can potentially also provide the model with a pragmatic cue that it will be presented with a new/unseen concept label if it receives a new/unseen kind of input, which adds another confound. This may not be much of an issue in the case of color since the space is fairly open and only 1/6 of the set of colors was covered anyway, but it still means a totally ungrounded baseline could do better than random.\n\nTo sum up: to really address the question of interest in this paper, I think the unseen-concept experiments need to more strongly limit their inputs to a very small number (or even better, report curves showing performance dependent on number of inputs), LM experiments which constrain the output space should be included as well, and predictive baselines outside of the few-shot paradigm should be provided. The results of Abdou et al. seem to indicate (to me) that it should be possible to do quite a bit better than the GPT-2 results in this paper using some kind of simple model (for example with word embeddings or a probing model on LM representations). It's important to have these baselines so as not to misrepresent the performance trend with respect to model growth, which I suspect reflects more the ability of the model to do \"few-shot learning\" than to ground concepts.\n\n(And to be clear: I don't think it would be a strike against this paper if few-shot learning for unseen concepts with very few colors didn't give good performance. That's still a cool experiment and an interesting result!)\n\nRelatively minor issues:\n\n* I understand you would probably want to express the color in RGB for the purposes of LM testing, but \nRotations in RGB seems like it isn't the best way of doing the isomorphism control: rotating in a space where distances are more perceptually meaningful (such as CIELAB) seems to me like it would have been a better approach, as while rotation preserves distances, the same distances in different parts of RGB space may correspond to different perceptual distances.\n\n* On that note, I'm not sure if I totally understand the random world baseline. It sounds from the description in the text (Sec. 2.4) that you ground words entirely randomly, but Figure 2 illustrates a \"random rotation\" which is much more constrained than this. It suggests that a random rotation should destroy the underlying structure, but I don't see why this would be the case. As long as distances are preserved (under the assumption necessary for the control that distances are meaningful in the first place) the structure should be preserved. So any rotation should work. In what sense does a random rotation destroy the structure? I don't buy it based just on the example shown in Figure 2.\n\n* The string matching Top-1 accuracy metric seems quite permissive — are there examples, or is there a human verification step you can provide to show that it is reasonable in this case and not overly permissive?\n\n* There seems to be some confusion about the training data. When you first mention it, you use the term OPENAI-WT with no further description or reference (Sec. 2.1). I don't think that is the dataset's name — I found it used as an abbreviation in one paper, Gehman et al. 2020, but nowhere else. The dataset is just called WebText. Can you just call it WebText (or OpenAI WebText) and cite the GPT-2 paper directly when you name it (so it's clear where it's from)? Also, in Section 2.1 it says all models are trained on this data, but this is incorrect — GPT-3 was trained on a different mix of data including two corpora of books, Wikipedia, and Common Crawl. Also, Appendix A calls the data \"OpenWT\", which sounds like \"OpenWebText\", which is an open dataset scraped using the same methods as WebText — but isn't the same dataset as was used to train GPT-2. Then, Section 2.4 does cite the GPT-2 paper for the data, but calls it CommonCrawl, which is incorrect — that's a different web scrape. \n\n## Typos etc.\n\n* Sec. 2.2, fourth line: stray space after \"prompt\"\n* End of page 2: stray comma after \"detail\"\n* The citation to Abdou et al. can be updated — the work is appearing in CoNLL 2021.\n* Last sentence before Sec. 3: stray comma at \"R-IV, a fairer\"", "summary_of_the_paper": "This paper investigates the question of whether language models encode, in some way or another, conceptual spaces that are analogous or isomorphic to grounded ones. The paper reports on the design and results of several experiments to test this in domains which can be easily serialized as text (and hence made accessible to an LM) — spatial directional terms, cardinal directions, and color terms. The ability of a language model to correctly ground conceptual terms is tested in a limited set of unseen environments and for unseen terms, where supervision is provided in the few-shot learning paradigm popularized with GPT-3. The largest tested model, GPT-3 175B, performs appreciably in all settings.", "summary_of_the_review": "I think this is overall a nice paper with some nice ideas and interesting results, but it falls short of adequately addressing the research question that it is designed to. The biggest flaws in my view are the inclusion of too much supervision in the unseen-concept experiments, which make the results less relevant to the \"isomorphism\" question inasmuch as it is interesting to the community, and the exclusive reliance on the few-shot learning paradigm, which adds a big confounding factor to the results that makes it hard to interpret any clear trends. I do think that this is really close to being a very nice paper as long as these issues are adequately addressed at least in the framing of the work if not the experiments.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "KnKL8Y9zHL", "reviewer": "Reviewer_e6Xz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper investigates a very interesting problem. The experiments are rather thorough, with different levels of controls (e.g., semantic-invariant transformations to the world representations, generalization to unseen worlds or unseen concepts). The writing is clear and structured as well.\n\nHowever, I think the concerns that some readers might raise and complain include:\n\n(1) Metric. Is the top-3 accuracy meaningful for the task, especially for the spatial and cardinal problems where the concept space is very small, and for GPT-3 that knows to output in-domain words only? Is the substring metric suitable for the color problem, especially in the “unseen concept” setup? For example, if “light blue” is a seen concept and “dark blue” is the test-time unseen concept, then answering the *seen* concept “light blue” in the unseen concept setup would result in a perfect accuracy. Would that defeat the purpose of testing generalization to unseen concepts, like in Table 2?\n\n(2) Conclusion drawn from the results. The authors argue that if the LM successfully generates the correct concept based on the grounded representation (likely “unseen” in the pretraining data), it means that the model knows to ground the concept to the non-text world. However, is it possible that the model doesn’t understand the relationship between the concepts and the grounded representations, but instead utilizes a similarity between the test grounded representation and the grounded representations in the in-context prompts? For example, upon seeing the test representation (e.g., [0,1,0,0] in the spatial domain, or RGB (140, 0, 255) in the color domain), the model can use a simple strategy: copying the concept of a bunch of most similar representations in the in-line prompt examples (e.g., [0, 1, 0, 0, 0], or RGB (145, 0, 255)). This strategy would not involve the concept of “left” or “pink”, and is robust to the rotation transformation (while not robust to the random transformation, if each point in the world was transformed independently). This would align with the results in Table 1. To check whether this hypothesis is (partially) true, we can look at experiments like:\n\nA: Test on some real unseen concepts. This was done in the paper, like in spatial and cardinal columns in Table 2, Table 9, 10, 11 (in the appendix). But the performance is not very strong in these cases even for GPT-3 (top-1 accuracy).\n\nB: Test with fewer prompts. This is to prevent the model from memorizing similarity with the prompting examples too much. This was also done in the paper (Figure 6 in the appendix). Again, the performance is not strong, if the number of prompts goes below 20 or 60.\n\nC: Replace all of the concept names with concepts in an unrelated domain (e.g., substituting all “left” and “right” with “apple” and “orange”). If the performance is above baseline in this setup, should we conclude that the LM implicitly learns to map fruit concepts to the grounded spatial world? This was not done in the paper and may be a good control experiment.\n\n(3) Though the paper investigates an interesting problem, the overall takeaway of this work is not very clear to me. How is the analysis useful for future work?\n\n(4) Some details in the paper should be checked. For example, in Section 2.1, the authors say that all models are pretrained on the 40GB OPENAI-WT dataset, but this is not true for GPT-3? Also for the color experiments, it is not clear whether 60 or 6+57 or 70 (as mentioned in B.1.1 in the appendix) prompts were used.\n", "summary_of_the_paper": "This works aims at investigating whether large language models (LMs) pretrained only on texts, can implicitly learn grounded concepts of the world beyond texts. Specifically, the authors test whether the LMs can map some conceptual domains (e.g., direction, color) to grounded world representations (e.g., textualized grid world, RGB representation of colors). The authors give a rather small number of examples as prompts to the models in an in-context learning setup. They find that large LMs like GPT-3 can often output the correct concept for the grounded world representations, even though it’s likely that the model hasn’t seen the grounded world representations in its pretraining. They conclude that the text-only LMs may already learn the grounded representations implicitly, without explicit from-scratch training like in the visual language models.\n", "summary_of_the_review": "Overall I think this work investigates an interesting problem, but the main argument needs to be justified more carefully (as mentioned in (1), (2) above). Also, the takeaway and impact of the work are not very clear to me, other than showing the somewhat inscrutable power of GPT-3.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9xedH0OTh7g", "reviewer": "Reviewer_TAjy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOverall I really enjoyed this paper.  The scope was fairly narrow and the experiments were not diverse enough to provide a sense of a thorough understanding of how LM's are forming this space, what possible confounds might be at play, or what other domains might have similar effects.  But apart from these concerns, the paper is well-written and presented in an organized manner, and essentially all the discussion points contributed something to my understanding of the work.  The topic is also very timely and important, and the implications of this work could help unify work in NLP, vision, and other grounded/situated learning.\n\nAn obvious main strength of the paper is that empirically the results are clear and consistent across all experiments.  The similarities between induced and gold spaces, between smaller models and larger models, is very significant.  The experiments themselves are quite simple and straightforward, but my biggest worry -- that confounds from the LM training data may be leaking in and influencing these conceptual spaces -- were strongly considered by the authors.  Their solution of using rotations to test against isomorphic spaces makes intuitive sense.  One thing I am considering is whether it's possible that huge parameter language models may be inducing mechanisms for performing spatial rotations on subspaces, though at some point it might become difficult to separate a model's ability to memorize the data and perform on-the-fly rotations, from something that might be called a non-trivial understanding of these concepts/spaces.  That is mostly an aside as I can't present it as anything more than speculation.\n\nOne of the few questions about the color experiment results was to what extent the model was backing off to predict less specific color names, since it is something that is not well-captured in top-1 accuracy (though maybe some combination of this and distance evaluation starts to get at it).  \n\nAnd why were 67 other colors inserted in the color prompts?  It seemed arbitrary.\n\nHow are grids incorporated into the model?  In Appendix B2, it appears to be just as a string following \"World\", but is it a \"linearized\" string of one row after another?\n\n\nTypos:\nZellers reference\n\"within the prompt .\"\n\n\n", "summary_of_the_paper": "This work seeks to probe large pre-trained language models for indications that they have induced a conceptual space that is structured similarly to one constructed from interactions with the real-world, despite being trained solely from text.  Most of this probing takes the form perceptual tasks -- can LMs quickly learn new concepts related to navigation, or colors, if given some related examples from the space?  In both navigation and color tasks, this seems to be the case, and a consistent trend emerges where larger LMs perform better on this task.  The experiments seem well-controlled, and the authors present an insightful discussion of model performance.\n\n", "summary_of_the_review": "The paper provides new insights into whether large pre-trained LMs induce conceptual spaces that mirror those found in the real world.  This could be an important finding, that together with many related works, gives us a better understanding of what sorts of higher reasoning abilities such LMs are capable of.  I found the experiments straightforward, but well-done, and some foresight was given to what confounding factors may be creeping into these results.  And across all model sizes we learn that similarities between these spaces increases with model size, and that low model sizes, though still big in comparison to many models, here are truly insufficient to capture concept spaces anywhere comparable to the largest models.  The paper is well-written and references are good, although I may have missed an important reference or two since this is slightly outside of my research area, but barring the existence of similar uncited work, I think it's a paper worth publishing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors explore the hypothesis of whether grounded representations can be leaved from text only. They show that a language model trained with relatively little data can make conceptual domains such as color to a grounded world representation such as RGB coordinates. The paper was positively received by the reviewers, specifically after a fruitful discussion to further clarify the points that the authors were making and their conclusions. The authors have already edited some parts of the paper, I ask them to go back and include other points that the reviewers have made. I recommend this paper for acceptance, it will generate good discussion and ideas at ICLR.
Decision: Accept (Poster)