Describe Me an Auklet: Generating Grounded Perceptual Category Descriptions
Authors: Bill Noble
Nikolai Ilinykh
Venue: EMNLP 2023 Main (2023)
Keywords: language-and-vision
grounding
zero-shot
cognitive theories of categorisation
natural language generation
natural language interpretation

Abstract: Human speakers can generate descriptions of perceptual concepts, abstracted from the instance-level. Moreover, such descriptions can be used by other speakers to learn provisional representations of those concepts. Learning and using abstract perceptual concepts is under-investigated in the language-and-vision field. The problem is also highly relevant to the field of representation learning in multi-modal NLP. In this paper, we introduce a framework for testing category-level perceptual grounding in multi-modal language models. In particular, we train separate neural networks to **generate** and **interpret** descriptions of visual categories. We measure the *communicative success* of the two models with the zero-shot classification performance of the interpretation model, which we argue is an indicator of perceptual grounding. Using this framework, we compare the performance of *prototype*- and *exemplar*-based representations. Finally, we show that communicative success exposes performance issues in the generation model, not captured by traditional intrinsic NLG evaluation metrics, and argue that these issues stem from a failure to properly ground language in vision at the category level.

DESCRIBE ME AN AUKLET:
Generating Grounded Perceptual Category Descriptions
Bill Noble" and Nikolai Ilinykh'
Centre for Linguistic Theory and Studies in Probability (CLASP)
Department of Philosophy, Linguistics, and Theory of Science (FLoV)
University of Gothenburg, Sweden [bill.noble, nikolai. ilinykhJ@gu.se
Abstract
ball LeTku Sertale
Human speakers can generate descriptions of perceptual concepts, abstracted from the instance-level. Moreover, such descriptions can be used by other speakers learn provisional representations of those concepts... Learning and using abstract perceptual concepts is under- investigated in the santaage-andvviiioon field. The problem is also highly relevant to the field of representation learning in multi-modaa NLP. In this paper. we introduce framework for testing category-level perceptual grounding in multi-modal language models. In particular, we train separate neural networks to generate and interpret descriptions of visual categories. We measure the communicative success of the two models with the zero-shot classification performance the interpretation model, which we argue is indicator of perceptual ground- ing. Using this framework, we compare the performance prototype- and exemplar-ban representations. Finally, we show that commu- nicative success exposes performance issues in the generation model, not captured by ditional intrinsic NLG evaluation metries, and argue that these issues stem from a failure to properly ground language in vision the cate- gory level.
created spcktar
Waal's crosted makien
list will liight emagg book
Hae mus srzient!
be tetiinniiiee.co oriele
Aamne Jere
Fall CEM
reestied Loekkte
Figure I:
simple learning scenario in which one speaker learns visual concept from another speaker's description; the learner is then able to use their provi- sional representation to classify an entity belonging to the concept.
et al., 2020; Bender and Koller, 2020), and the fast- growing field multi-modal NLP has been work- ing to address this problem (Bernardi et al., 2016; Beinborn et al., 2018). However, multi-modal models have several areas for improvement. Re- cent work suggests that these models are affected by the distribution of items in training data, of- over-representing specific scenarios and under- representing others (Agrawal al., 2018). This, in turn, affects their ability to find a true balance between the levels of granularity in descriptions for novel concepts, as these models are expected to generalise (Hupkes et al., 2023). As a result, these models rely excessively on text and have to be supplied with various mechanisms, enforcing and controlling their attention on modalities such as vision (Lu al., 2017; Thomason al., 2019; Ilinykh al., 2022). This raises questions about the nature of the relationship these models learn between linguistic and non-linguistic information.
1 Introduction
Grounded language use links linguistic forms (sym- bols) with meaning rooted in various perceptual modalities such as vision, sound, and the sensory- motor system (Harnad, 1990). But grounding is not merely a solipsistic mapping, from form to mean- ing; rather, it results from communicative context in which linguistic agents act and have goals in- the real world (Larsson, 2018: Chandu al., 2021; Giulianelli, 2022). Large language models trained on vast amounts of text have been criti- cised for lacking grounded representations (Bisk
"Equal contribution.

---

Exploiting statistical regularities in multi-modal datasets can cause models to hallucinate. Accord- ing to Rohrbach et al. (2018), neural image cap- tioning systems can accurately describe objects in images but struggle to understand the overall situa- tion, often relying common contextual patterns associated with specific objects that co-occur. Sim- ilar problems are common for other multi-modal models, datasets (Alayrac et al., 2022), and tasks such as Visual Question Answering (Antol al., 2015) and Embodied Question Answering (Das et al., 2018). These examples, along with many others, illustrate that perceptual grounding cannot be achieved in the abstract but must be considered in a communicative context, which includes speak- ers' prior common ground, joint perception, and intentions (Clark and Wilkes-Gibbe Gibbs, 1986). One important type of common ground is shared percep- tual world knowledge, which need not necessarily rely on the immediate perceptual context. For in- stance, if someone mentions that red apples are sweeter than green ones, this communicates some- thing, even to someone who is not concurtently looking at or tasting apples. We can acquire and use (provisional) perceptual concept based on natural language description produced by conver- sation partner, process referred to as fast map- ping (Carey, 1981; Gelman and Brandone, 2010). Can modli language models generate a de- scription of a perceptual category that similarly communicates the concept to an interlocutor?
sification on unseen instances using descriptions produced by GEN as auxiliary class information. Zero-shot learning from text descriptions is not a novel task; our focus in this work is on the gen- eration of perceptual category descriptions, using "communicative success" the performance of the IPT model-as as a semi-extrinsic evaluation metric. The proposed evaluation method differs from many standard automatic generation evaluation metrics, such as BLEU (Papineni et al., 2002), which are not designed to capture the level of communicative use- fulness of the generated texts. In contrast to many tasks, we explore the ability of multi-modal models to perform grounding on class-level representations, distinct from instance- level representations, e.g. images.' Additionally, highlight the issue of mismatch between intrin- sic evaluation (generation metrics) and task-based evaluation, as indicated by the performance of the IPT. Our results reveal challenges involved in de- veloping better models with the ability to ground at the class level. We believe that our fineggrained analysis of the task, data and models sheds light on the problems associated with both generating and interpreting class-level image descriptions. We also contribute insights into the extent to which current evaluation methods for generated texts con- sider communicative context. The framework that we propose can be used for evaluating existing models of language grounding and can also aid in building new multi-modal models that perform grounding in communication. To support research in this direction, we have made our code and data available here: https:/ //gs: thub. www.tooo..///CCA.... describe-mar me- an auklu..
Language-and-vee
In this paper, we propose perceptual cate- gory description, which emphasises category- level grounding in a communicative context. This framework models a simple interactive scenario (Figure 1) where (I) a describer, referred to as GEN, generates a description of one or more visual categories, (2) an interpreter, IPT, learns from the generated descriptions, and (3) classifies among both the seen classes, which it already has knowledge of, and the unseen classes described by GEN. During training, the GEN model has ac- cess to images and class labels from both the seen and "unseen" sets, but only receives supervision on ground-truth descriptions from the seen set. This ensures that during testing the generator is evalu- ated based on its ability to use category-level repre- sentations of the unseen classes, rather than mem- orising descriptions from the training data. The IPT model only has access to instances from seen at train time and performs zhoo image clas-
Background
Prototypes and exemplars Cognitive theories of categorisation are peychologically-mottvaa ac- counts of how humans represent perceptual con- cepts and use them for classification. Such theories have challenged the assumption that categories can be defined in terms of set of necessary and suffi- cient features. In contrast, they try account for phenomena like prototypically effects, in which cer- tain members of a category are perceived as more representative of the class than others. In proto- type theory, cognitive categories are defined by a
'See, for example, Bermardi et al. (2016) which presents a survey of image description techniques that rely heavily on the image as part of the input.

---

prototype, an abstract idealisation of the category. Membership in the class, then, is judged in refer- ence to the prototype (Rosch, 1975). In exemplar theory, (e.g.. Medin and Schaffer, 1978; Nosof- sky, 1984), concepts are still defined in relation to an ideal, but this time the ideal is an exemplar, which is particularly representative member of the very category. Put another way, exemplar is of the same kind as the other members of the category, whereas prototypes, in general, are not. Experimental evidence suggests that humans em- ploy both exemplar and prototype-based strategies (Malt, 1989; Blank and Bayer, 2022).
son (2021) demonstrate that for a nominally non- linguistic task (c.g. classification), longer descrip- tions yield better visual representations compared to labels. Image classification can be further im- proved with a better mapping between visual and linguistic features (Elhoseiny et al., 2017; Kousha and Brubaker, 2021).
Innovative language use can be resolved by tak- ing familiar representations and mapping their com- ponents to a new context (Skantze and Willemsen, 2022). Suglia et al. (2020) and Xu et al. (2021) de- velop models that recognise out-of-domain objects by learning compose the attributes of known ob- jects. Also, the descriptiveness and discriminative- ness of gencrated class description influences their utility for interpretation purposes (Young et al., 2014; Vedantam et al., 2017: Chen et al., 2018). We partially explore this phenomenon in our exper- iments; see Section 4.2.
Perceptual categories play role in natural language interpretation and generation. In fact, classifier-based meaning has been proposed as a way to ground language in perception (Schlangen et al., 2016; Silberer et al., 2017). There are both formal and computational interpretations of this approach that support compositional semantics for lexical items with classifier-based perceptual mean- ings (Larsson, 2013; Kennington and Schlangen, 2015). In this paper, explore how classifier- based meaning facilitates the generation of class- level descriptions by testing three different GEN model architectures: one motivated by prototype theory, one by exemplar theory, and one that uses a hybrid approach.
Language games in a multi-agent setup Our setup with two neural networks is somewhat analo- gous to the idea of a multi-agent signalling game (Lewis, 1969). While the idea of multiple agents developing their language to solve tasks has been extensively studies in NLP (Lazaridou et al., 2017; Choi al., 2018), our work differs in that do not have direct learning signal between the mod- els, e.g. the agents are not trained simultaneously. Therefore, our models do not cooperate in a tra- ditional sense. Instead, we focus on developing a more natural and complex multi-network environ- ment by incorporating insights from research on human cognition, perceptual grounding, and com- munication. In particular, we (i) explore the ability of neural language models to learn high-level rep- resentations of visual concepts, (ii) generate and evaluate concept descriptions based on these rep- resentations, and (iii) assess the performance of a separate network in interpreting these descriptions for zero-shot classification.
Zero-shot language-and-vision generation and classification In the perceptual category descrip- tion framework, both models operate with textual deseriptions: one generates them, and the other in- terprets them. The interpretation model performs zero-shot classification, with (in this case) vision the primary modality and text as the auxiliary modality. 2 In zero-shot learning scenarios that use text as auxiliary data, the quality and relevance of the text has been shown to improve model perfor- mance. For example. perceptually more relevant texts might help better learning of novel concepts (Paz-Argaman et al., 2020). Bujwid and Sullivan (2021) show that Wikipedia texts can be used as class descriptions for learning better encoding of class labels. In a similar vein, Desai and John-
In related work, Zhang et al. (2018) train an inter- preter and a speaker to perform continuous learning through direct language interaction. In contrast, our setup is more straightforward as the describer does not receive feedback from the interpreter. Another study by Elhoseiny al. (2013) proposes learn- ing novel concepts without visual representations. They use encyclopedic entries as alternative infor- mation sources when perceptual input unavail- able. Our approach presents a greater challenge as humans often lack access to textual corpora when
"This means that the model has supervised training with visual examples of seen classes, and then the model receives text descriptions (one per class) corresponding to the unseen classes. The model is then evaluated in the generalised zero- shot setting. Le.. to classify new visual examples among both seen and unseen classes. See (Xian et al., 2020) for an introduction to different zero-shot learning sctups and recent survey of the field.

---

interacting in the world. Patel and Pavlick (2022)
task. Visual features of the input, x, are concate- nated with each class vector V, from V before be- ing passed through the network. Consequently, the network produces N scores that are transformed into class probabilities y using soft func- tion applied along the label dimension:
investigate the ability of pre-trained language mod- els to map meaning to grounded conceptual spaces, We are similarly interested in grounding in a struc- tured space of related concepts, but our setup is different, proposing the cemi-interactive task of grounded category description, rather than probing models for their ability to generalise.
y - = (((2((((( @ villss), (1)
3 Models
where
At high level, GEN and IPT each have two connected modules: image classifier, and grounded language module. Both networks learn visual representations which are shared between the classification and language tasks. During train- ing, IPT learns to interpret textual descriptions of seen classes by mapping them into its visual rep- resentation space. If generalises well, textual de- scriptions of unseen classes should then be mapped to useful visual representations at test time, even though no images of unseen classes were avail- able during training. Contrariwise, GEN is trained generate descriptions of seen classes based on its visual representation of those classes. At test time, GEn must extrapolate to generating descrip- tions of unseen classes, for which no ground-truth descriptions were provided during training.
f1(x) = ReLU(Wi x b1),
(2)
52(x' WWa(ReLU( x' b2))
(3)
where W1 22048x11 W2 + and W3 E R is the classification output layer. Both GEN and IPT use h 256 and h2 128.
3.2 Generation model
The generation model has two modules: the classi- fier described in $3.1, and a decoder that generates text from a class representation. Given label yt- the decoder generates text by using the class repre- sentation, c₁2 corresponding to the label. The class representation is computed differently depending whether the model uses prototype class repre- sentations, exemplars, or both:
GEN-PROT simply takes the corresponding row of the label embedding VG, which is also used for classification.
3.1 Label embedding classifier
Both models use label embedding classifier that represents classes embeddings. The embedding matrix V NxD stores visual concept represen- tations. with = 200 being the number of classes and = 512 indicating the size of each single class representation vector.3 The class embedding parameters (VG for GEN model and V, for IPT) are shared between the classification module and language module within each model (no parameters are shared between GEN and IPT). Both models use ResNet visual features, with size of 2048 pro- vided by Schönfeld et al. (2019) as inputs to the classifier. These features were extracted from the standard 101 trained on the ImageNet Ik dataset (Russakovsky et al., 2015). In the follow- ing. x = (esNNe(()) is the encoding of the input image
prot==
prot = V; v Vo[[]
(4)
GEN-EX keeps additional cache of exemplar image features (one per class) which change after each training epoch. The exemplar image for class is computed as the image that most certainly part of that class, according to the classifier:
FX = arg max (9| æ EE (5)
GEN-BOTH uses the concatenation of the proto- type and exemplar representations:
The classifiers simple two-layer forward networks trained on the multi-class classification
CR
both V2 ez
(6)
We train a standard transformer decoder to gener- ate class descriptions (Vaswani et al., 2017). GEN models differ only in the type of input representa- tions provided to the decoder. At each timestep. f,
We also initialise GEN with N = 200 for convenience, but the labels corresponding to the 20 unseen classes are quickly disregarded during supervised training since they never appear in the training data.

---

LIVISS entrupy
entrupe
loss.
Uriess entrupy
model input
y
yi
y
yi
Transformer
classiher
classifer
4
crested aucklet
UF
locking, Large Bizg has with musgy gray bady and pearboal Teathere,
Exemplars
ResNet
BERT cla
ResNet
U
u
2
L
y
GEN
IPT
Training inputs and describer (left) and interpreter (right) model architectures. GEN, the dotted lines indicate options for the type of class representation (cfrot, c, c; choth), If exemplars are used, they updated based on the classifier at the end of each epoch (gray dotted lines), as described in equation 5.
Figure 2:
the model's input is updated with previously gener- ated tokens (w....., We 1) and the current token wy is predicted. We use a standard setup for the transformer: six self-attention layers with eight heads each. The model is trained for 20 epochs in a teacher forcing setup. The learning rate is set to x 10-4. The best model is chosen based on the CIDEr score (Vedantam et al., 2015) on the validation set using beam search with width of 2. Both the classifier and the decoder are trained jointly with the standard cross-entropy loss:
longer generation become repetitive and "boring" (Gu al., 2017).
Beam search is often used as a standard decod- ing algorithm because it suffers much less from the problems occurring during long-text generation. At each generation step i, keeps track of several candidate sequences C = (cI...., cx)) and picks the best one based the cumulative probability score of generated words per sentence:
C; argmaxx log p(ck Ci-1, C; (8) Vz;
of B. |c
lossc = CrossEntropyy (y:))
The parameter k is used control the depth of the search tree, and B is the set of candidate sequences. While beam search generally outper- forms greedy, higher can lead to texts with low diversity (Li et al., 2016). To evaluate whether "more diverse" means "more descriptive" in the context of our two-agent set-up, we generate texts with nucleus sampling method (Holtzman et al., 2020) which samples tokens from the part of the vocabulary defined based the probability mass:
Σ CrossEntropy(ii, (u))), (7)
y is the output of the classifier, ye is the ground- truth label, ú; output of the decoder at position i, and uı is the ground-truth token. For inference, we explore multiple decoding algorithms which we describe below.
3.3 Decoding algorithms
our pnccriber-intrppeetee setup, the quality of the generated texts, particularly their content, of importance. Quality generation depends heavily on the decoding algorithm used to select tokens. "Safer" algorithms may generate more accurate texts, but with poor discriminativity. while other algorithms introduce some degree of randomness, which promotes diversity (Zarrieß et al., 2021). We examine two decoding algorithms, with introduce different conditions for text accuracy and diversity. While greedy search can generate accurate descrip- tions, is sub-optimal at the sentence level, e.g.
p' Σ logp(w; Wes. V); 2 P. W.E
(9)
where p determines the probability mass value, while V is part of the vocabulary V which accu- mulates the mass at the timestamp i. Next, new distribution produced sample the next token:
logp(w; / was va 0)// if wi L otherwise. (10)
P
0

---

With nucleus sampling, we aim to generate more diverse texts than those generated with beam search. By evaluating the interpreter with texts generated by different algorithms, we consider the impact of generation on the success of information transfer from the describer to the interpreter.
cross-entropy loss and the term for the interpreter is computed with the cosine embedding loss, a variation of hinge loss defined below. The overall loss is computed as follows:
lossy CrossEntropy (5r, ye)+ CosineEmbbLsss vk), (12)
3,4 Interpretation model
The IPT model has two modules: a label embedding classifier with a weight matrix V, E R NxD, and an interpretation module that maps texts to vectors of size D. IPT uses [CLS] token vectors extracted from BERT text features. In preliminary experi- ments on the ground-trut tround-truth test data, we observed significant improvements in the performance of IPT by using features from a BERT model (Devlin et al., 2019) which was fine-tuned on descriptions from the seen portion of the training set. We fine-tuned the final layer with a learning rate of 2 x 10 and weight decay of 0.01 for 25 epochs using the Adam optimiser (Kingma et al., 2015). The model was fine-tuned using a text classification task involv- ing the seen classes. Since BERT not visually grounded, we speculate that the pre-training task may assist the model in attending to visually rele- vant information within the descriptions, leading to a more informative [CLS] representation. Given a text description u. we use to denote the [CLS] features (with size 768) extracted from the fine- tuned BERT model.
where
CosineEmbLosss(* vv.)
f1- {1-Cosh Cos(V, viz) max (0, Cos(V, Vk) 8)
k: & if k: # (13)
Like hinge loss, the cosine embedding loss in- cludes margin ö, which we set to 0.1. Intuitively. 8 prevents the loss function from penalising the model for placing its class representation predie- tion close to the representation of a nearby negative class, as long as it isn't too close. After all, some classes are similar. The best IPT model chosen based on the zero-shot mean rank of true unseen classes in the validation set.
Experiments
4.1 Data
We use the Caltech-UCSD Birds-200-2011 dataset (Wah et al., 2011, hereafter CUB), collection of 11 788 images of birds from 200 different species, The images were sourced from Flickr and filtered by crowd workers. In addition to class labels, the dataset includes bounding boxes and attributes, but we do not use those features in the current study, since our focus is on using natural language de- scriptions for zero-shot classification, rather than from structured attribute- features.
The interpretation module is defined as follows:
Tanhi W u u+bb) b)
(11)
where W E 768x and b E IR.
Given a training example (x, Ve, u), the classi- fier makes class prediction y from x and the interpreter predicts the class representation & from Our objective is to improve both on the class predictions and class representations produced by the IPT model. To evaluate the class prediction, we compare to the ground-truth class label ye- As for the class representation, the training objective encourages the model to to predict position in the vector space with is close to the target class, f, and far from randomly selected negative classes. We employ the following sampling strategy. We draw vector Vy from V, so that with frequency of 0.5, it is negative sample (i.e., l + f) and the other half the time = f.
We also use corpus of English-language de- scriptions of the images the CUB dataset, col- lected by Reed et al. (2016). The corpus contains 10 descriptions per image. The descriptions were written to be both precise (annotators were given a diagram labelling different parts of a bird's body to aid in writing descriptions) and very general (anno- tators were asked not to describe the background of the image or actions of the particular bird), This
allows us to treat the captions as class descriptions, suitable for zero-shot classification. We split the dataset into 180 seen and 20 unseen classes and train, test, and validation sets of each (Table 1).
The two modules are trained jointly. The loss term for the classifier is computed with the standard

---

seen
unseen
Total
information of the feature and the bird class, measured on the test set; that is,
Train
8482
948
9430
Test
1060
119
1179
disc(a:) exp(H HY
Val
1060
119
1179
where is a feature and Y is the bird class.
Total
10 602
1186
11 788
The maximum discriminativity of a feature (i.e.. feature that uniquely picks out particular class) equal the number of classes, 200. For example, disc(("bill) { ('long' 'curved') () = 22.9, whereas disc(( ("bill"",,,,, { ('short', 'pointy' })) = 2.9, reflecting the fact that more kinds of birds have short pointy bills than long curved bills. We define two met- rics for the discriminativity, discmaxx and discavg which are the maximum and mean discriminativity of the features included in given description.
Table I: Number of CUB corpus images by data split.
A single training example is triple (æe, y; d), consisting of image, class label, and descrip- tion. Since there are 10 descriptions per image, this gives 84 820 seen training examples for the interpreter. The generator is additionally trained on the 9480 unseen training examples, but with the descriptions omitted. To mitigate the possibil- ity that the unseen split represents a particularly hard or easy subset of classes, we test 5 folds, each with disjoint sets of unseen classes. The results reported are the mean values across the five folds.
Results
Our primary objective is to examine if we can learn models capable of grounding on the category level in the zero-shot image description generation setup. First, address part of this question by exploring the performance of the IPT model when classify- ing new classes given GEN-generated descriptions (Table 2). We evaluate the performance the in- terpreter on the unseen set using both ground-truth descriptions and descriptions generated by the best GEN model. See Table 3 for full comparison of the generation models, including resulting IPT performance. Since multiple descriptions exist per class in the ground-truth texts, we randomly select one for each unseen class in each zero-shot fold. Our first observation is that the model is mod- crately successful on the zero-shot classification task. When learning from the ground truth descrip- tions, the model performs well above the random baseline. While 0.19 is not very high for classi- fication accuracy in general, it is not out of line for unseen results in zero-shot learning. It must be noted that classifying a bird as belonging one of 200 species based only on textual descrip- tion would a difficult task for some humans as well. That the model can use the ground truth text descriptions to learn class representations that are somewhat useful for image classification is encour- aging for the prospect of using it to evaluate the GEN models. However, mance of the model using descriptions generated from the best GEN model quite lot worse than the ground truth. This suggests that while the de- scriptions generated by the best GEN models are not totally useless, they are nevertheless not as
4.2 Evaluation metrics
Generation and classification We evaluate the performance of GEN with BLEU (Papineni al., 2002) and CIDER (Vedantam et al., 2015) the latter has been shown to correlate best with hu- man judgements in multi-modal setup. As is stan- dard in classification tasks with many classes, the interpreter is evaluated with different notions of accuracy: accuracy @ 1, @5 and @ 10, where prediction is considered successful if the true la- bel is the top, in the top 5, or in the top 10 labels, respectively. We also consider the mean rank of the true class to examine how close the model is in cases where its prediction is incorrect.
Discriminativity Our generation model is trained to minimise the cross-entropy of the next token, given the class label. This learning objective may encourage the model generate "safe" descriptions, as opposed to descriptions that mention features that would help to identify birds of the given class. To measure this tendency, we define a notion of the discriminativity of a class description, which evaluates how helpful the description is in picking out instances of the class it describes. To compute the metric, first extract textual features from the descriptions, where each feature consists of the noun and the set of adjectives used in a noun phrase. We define the discriminativity of a feature with respect to a particular class as the exponential of the mutual
note that the perfor-

---

teacher
GEN train data
CE loss
mean rank
acc @
ncc@5
random baseline
5.30
100.5
0.5
2.5
seen
2.50(.18)
5(0)
36.1(1.1)
75.1(2.8)
ground truth
unseen
4.17(.46)
30(6)
19.1(4.1)
44.1(5.6)
seen
2.36(.12)
6(1)
43.7(2.5)
74.1(3.1)
best GEN
unseen
5.28(.54)
46(9)
8.6(5.8)
25.1(3.9)
Table 2: Zero-shot classification results for the IPT model. The results computed macro-averages averaged first over class, then over the fold. Only results of the unseen classes will be interest as an evaluation metric for the GEN model, but here we report both, since is important see that the IPT model still performs well on seen after learining provisional unseen class representations. The ground truth results report zero-shot performance after learning from one randomly sampled ground truth description for each unseen class. The best GEN results report zero-shot performance after learning from the best GEN model (GEN-EX with beam-2 decoding).
discriminativity
mean rank
accuracy @
Bleu1
Bleu4
CIDEr
mean
max
class repr. decoding
both beam
.6(.12)
.55(.09)
1.83(0.30)
1.58(0
2.32(1.35)
94(28)
0.0(0.0)
2.1(1.1)
nucleus
69(.03)
.3((.5)
1.40(0.07)
5.22(1.62)
12.48(5.47)
111(14)
0.8(15)
4.1(4.8
exem
beam
.64(.03)
.58(.02)
1.92(0.08)
1.95(0.41)
3.29(1.24)
16(9)
8.6(5.8)
25.1
nucleus
-65(.04)
.36(.08)
1.42(0.14)
5.10(1.44)
12.07(4.70)
70(7)
6.8(2.6)
18.3(3.1)
prot
beam
.61(.09)
(50(11)
1.80(0.32)
(.650.
2.46(0.79)
73(17)
2.7(2.7)
13.6(6.3)
nucleus
-70(.04)
(38(.0)
1.48(0.08)
5.51(1.83)
13.14(4.14)
75(12)
4.1(3.1)
15. (5.00
Table 3:
Generation results on the unseen set. The models differ only in terms of the input and decoding methods. BLEU and CIDEr scores are reported as micro averages over n-grams produced all 200 class descriptions. Mean rank and accuracy refer to the unseen test performance of the IPT model trained the corresponding zero-shot split, having learned provisional representations from the descriptions provided by the GEN model.
communicatively successful as they could be. We observed intriguing results regarding seen classes: generated texts can be more useful than ground- truth descriptions for the IPT. This suggests either (i) lower quality of generated texts and the inter- preter relying on common bird features and spuri- ous correlations, or (ii) the possibility that human- generated texts not as informative initially assumed. Ultimately, human descriptions were pri- marily intended for interpretation by other humans, which could explain why they may have omitted significant information that listeners already pos- sessed prior knowledge useful for interpretation.
poorly (worse than random baseline in some cases)
in terms of communicative success.
Discussion and conclusion
One of the motivations behind adopting this task was its reliance on class-level representations, which distinguishes from other image-specific language-and and-vision tasks. We wanted to see how well the models can ground language in the absence of image pixels. Our results revealed several inter- esting directions for further exploration in mod- elling, feature representation, and the evaluation of generation and interpretation. Strikingly, the top-perte perforfforing models were the GEN-Ex models, which effectively reintroduce specific images by picking out exemplars to generate from. Of course, the models we used in this study were relatively simple, and more sophisticated neural models may yield different results. But this raises an interest- ing question for future work what does it take to learn grounded representations that are useful in this particular communicative context?
Next, we compare different GEN models, as shown in Table 3. We can see that GEN-EX out- performed the others on most intrinsic metrics (ex- cept for BLEU-I) and also in terms of communica- tive success. Beam search performed better than the nucleus the intrinsic metrics, and particu- larly excelled in the case of GEN-Ex. Interestingly, nucleus- nucleus-generated texts nevertheless scored much higher in terms of discriminativity. GEN-PROT and performed similarly on the intrin- sic metrics, but GEN- -OTH performed extremely
More generally, why do the GEN model descrip- tions fall short comparison to ground-trun trrth for

---

the zero-shot performance on the IPT model? There are two possible explanations. One that the gen- erated descriptions may lack the necessary visual information required for successful classification of unseen classes. Secondly, the texts produced by the GEN model may not be interpretable by the IPT model. Recall that the IPT model was trained on ground truth descriptions from seen. These descriptions have structure to cerm-nertain reg- ularities in conveying visual information. If the GEN descriptions deviate from this structure, IPT may struggle to effectively utilise them, even they do in some sense "contain" visual information. Indeed, there is some evidence that this is what is happening. We see that nucleus sampling resulted in higher discriminativity scores, including for the GEX-PROT and GEN- BOTH models. Although the generator produces sequences with adjective-noun phrases that identify the correct class, IPT cannot properly use them, perhaps because they appear in texts that are "ungrammatical" for the distribution IPT was trained on. As both the GEN and IPT mod- els are simple approximation functions of the data on which they are trained, they may rely too heavily patterns and regularities, which can hinder their ability to learn to recognise and generate better category-level descriptions. This points to an im- portant research direction,
findings suggest that the "groundedness" of class descriptions and their use for the task might not be adequately captured by the set of NLG metrics and one might want to use the generated texts "in con- text" (aka interpretation) to get a clearer picture on how much important information such texts carry.
In future work, we will focus on improving inter- pretation performance by emphasising fine-grained differences between class features. Inspecting how the generation of more descriptive and more dis- criminative class descriptions can be achieved is also important. Additionally, we will examine the impact of more interactive context on the task, which could be studied in reinforcement learning setup (Oroojlooy and Hajinezhad, 2021).
7 Limitations
This paper focused on proposing the task of vi- sual category description, and testing different cognitively-inspire representations for standard neural network architectures. We did not expand our experiments more complex models. Our analysis can also be performed the context of different encoder-decoder combinations. Secondly, the dataset has fine-grained descriptions of cate- gories. However, these descriptions can be so spe- cific that they lack in generality, which depends on the domain and even personal preferences and background of those who interact. While this does correspond the situation in certain real-life situ- ations (bird classification being one of them), the results may look different in a more open-domain setup, such as the dataset from Bujwid and Sulli- van (2021).
might highlight the limitations of many current existing multi-modal models which are built on top of the transformer architecture. Such models might still be useful in various domains but often face challenges in learn- ing higher-level concepts about the world.
A different question is whether generation met- rics reflect the communicative power of texts as measured by the interpreter's performance. In the case of GEN-EX, IPT performs best with texts generated using beam search (Table 3). However, these texts overall score very low on discrimina- tivity. Indeed, we discovered that beam search generates sentences with features common to mul- tiple classes, e.g. "a bird with wings". At the same time, IPT benefits more from nucleus-generated texts produced by the PROT model. These texts are more diverse, possibly describing larger set of class features and our interpreter is able to learn better from that diversity. Intrinsic genera- tion metries rate nucleus-generated texts generally lower, suggesting a mismatch between task-based evaluation (c.g., interpreter's performance) and in- trinsic evaluation (e.g., generation metrics). These
Moreover, the way that the descriptions were col- lected may mean that they differ somewhat from how a human would produce visual category de- scriptions. Experimentation with more datasets of different levels of specificity and with different kinds of ground truth classes descriptions would be necessary to draw more general conclusions. Given that we employ pre-trained transformer model to encode texts, we note that there might be an im- pact of BERT's prior knowledge about birds the interpreter's performance. We recognise this as a promising starting point for exploring the model's performance across other domains, allowing us to assess the general effectiveness of the setup we have introduced.

---

Acknowledgements
Experience grounds language. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 87 8-8735, Online. Association for Computational Linguistics.
The project described in this study was supported by a grant from the Swedish Research Council (VR project 2014-39) for the establishment of the Centre for Linguistic Theory and Studies in Probability (CLASP) at the University of Gothenburg. Authors would like to thank the reviewers and the meta- reviewer for their helpful feedback on the paper.
Helen Blank and Janine Bayer. 2022. Functional imag- ing analyses reveal pro and exemplar represen- tations in perceptual single-category task. Commu- nications Biology, 5(1):1-13.
Sebastian Bujwid and Josephine Sullivan. 2021. Large- Scale Zero-Shot Image Classification from Rich and Diverse Textual Descriptions. In Proceedings of the Third Workshop on Beyond Vision and LANguage: in- TEgrating Real-world kNowledge (LANTERN), pages 38-52, Kyiv, Ukraine. Association for Computational Linguistics.
References
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 4971-4980.
S. Carey. 1981. The child as word learner. In Morris Halle, Joan Bresnan, and George A. Miller, editors, Linguistic Theory and Psychological Reality, first paperback edition edition, pages Press, Cambridge, Mass.
The MIT
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds. Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, An- drew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022. Flamingo: A Visual Language Model for Few-Shot Learning.
Khyathi Raghavi Chandu, Yonatan Bisk, and Alan W Black. 2021. Grounding 'grounding' in NLP, In
Findings of the Association for Computational Lin- guistics: ACI--ICNNLP 2021, pages 4283-4305, On- line. Association for Computational Linguistics.
Fuhai Chen, Rongrong Ji. Xiaoshuai Sun, Yongjian Wu. and Jinsong Su. 2018. Groupcap: Group-based im- age captioning with structured relevance and diversity constraints. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1353-133.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitmick, and Devi Parikh. 2015. Vqa: Visual question an- swering. Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2425- 2433.
Edward Choi, Angeliki Lazaridou, and Nando de Fre- itas. 2018. Compositional obverter communication learning from raw visual input. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Con- ference Truck Proceedings. OpenReview.net.
Lisa Beinbor,, Teresa Botschen, and Iryna Gurevych. 2018. Multimodal grounding for language process- ing. In Proceedings of the 27th International Con- ference Computational Linguistics, pages 2325- 2339, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Re- ferring as a collaborative process. Cognition, 22:1- 39.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NL.U: On meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 5185-5198, Online. Association for Computational Linguistics.
Abhishek Das. Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Embodied question answering, In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 1-10.
Raffaella Bernardi, Ruket Çakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, and Barbara Plank. 2016. Automatic description generation from im- ages: A survey of models, datasets, and evaluation measures. J. Artif. Intell. Res., 55:409-442.
Karan Desai and Justin Johnson. 2021. Virtex: Learning visual representations from textual annotations. In Proceedings of the IEEE/CVE Conference on Com- puter Vision and Pattern Recognition (CVPR). pages
Jacob Devlin, Ming-Wal Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lap- ata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020.

---

Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Meeting of the Association for Computational Lin- guisties and the 7th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 292-301.
Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgam- mal. 2013. Write Classifier: Zero-Shot Learning Using Purely Textual Descriptions. In 2013 IEEE International Conference on Computer Vision, pages 2584-2591. IEEE
Diederik P. Kingma, Jimmy Ba, and Amsterdam Ma- chine Learning lab (IVI, FNWT). 2015. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations (ICLR). arXiv.org.
Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed Elgammal. 2017. Link the head to the "beak":
Shayan Kousha and Marcus A. Brubaker. 2021. Zero- shot learning with class description regularization.
Zero Shot Learning from Noisy Text Description at Part Precision. arXiv:1709 01 148 [cs].
Staffan Larsson. 2013, Formal semanties for perceptual classification. Journal of Logic and Computation. 25(2):335 -399
Susan A. Gelman and Amanda C. Brandone. 2010. Fast- mapping placeholders: Using words to talk about kinds. Language learning and development: the official journal of the Society Language Develop- ment, 6(3):223-240.
Staffan Larsson. 2018. Grounding as a Side-Effect of Grounding. Topics in Cognitive Science, 10(2):389- 408.
Mario Giulianelli. 2022. Towards pragmatic produc- tion strategies for natural language generation tasks, In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 7978-7984, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2017. Multi-agent cooperation and the emergence of (natural) language. In 5th Inter- national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Con- ference Track Proceedings. OpenReview.net
Jiatao Gu, Kyunghyun Cho, and Victor O.K. Li. 2017. Trainable greedy decoding for neural machine trans- lation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1968-1978, Copenhagen, Denmark. Associa- tion for Computational Linguistics.
David Kellogg Lewis. 1969, Convention: A Philosophi- cal Study. Cambridge, MA, USA: Wiley-Blacke -lley-Blackwell.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A jective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 110-119, San Diego, California. Association for Computational Linguistics.
ob-
Stevan Harnad. 1990. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1):335-336.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRevie..net.
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when to look: Adaptive attention via visual sentinel for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 375- 383.
Dieuwke Hupkes, Mario Giulianelli. Verna Dankers. Mikel Artetxe, Yanai Elazar, Tiago Pimentel. Chris- tos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyaghaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske, Ryan Cotterell, and Zhijing Jin. 2023. A taxonomy and review of generalization research in nlp. Nature Ma- chine Intelligence, 5(10):1 1614
Barbara C. Malt. 1989. An on-line investigation pro- totype and exemplar strategies in classification. Jour- nal of Experimental Psychology: Learning, Memory, and Cognition, 15:539-555.
Nikolai llinkkh, Yasmeen Emampoor, and Simon Dob- nik. 2022. Look and answer the question: On the role of vision in embodied question answering. In Proceedings of the 15th International Conference on Natural Language Generation, pages 236-245, Waterville, Maine, USA and virtual meeting. Associ- ation for Computational Linguistics.
Douglas L. Medin and Marguerite M. Schaffer. 1978. Context theory of classification learning. Psychologi- cal Review, 85:207-253
R. M. Nosofsky. 1984. Choice, similarity, and the con- text theory of classification. Journal of Experimen- tal Psychology. Learning, Memory, and Cognition. 10(1):104-114
Casey Kennington and David Schlangen. 2015. Simple Learning and Compositional Application of Perceptu- ally Grounded Word Meanings for Incremental Refer- ence Resolution, In Proceedings of the 53rd Annual
Afshin Oroojlooy and Davood Hajinezhad. 2021. A Review of Cooperative Multi-Agent Deep Reinforce- ment Learning. urXiv:1908.0300 [cs, math, stat].

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Alessandro Suglia, Antonio Vergari. loannis Konstas, Yonatan Bisk, Emanuele Bastianelli, Andrea Vanzo, and Oliver Lemon. 2020. Imagining grounded con- ceptual representations from perceptual information in situated guessing games. In Proceedings of the 28th International Conference on Computational Lin- guistics, pages 1090- 102, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics.
Roma Patel and Ellie Pavlick. 2022. Mapping language models to grounded conceptual spaces. In The Tenth International Conference on Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Jesse Thomason, Daniel Gordon, and Yonatan Bisk. 2019. Shifting the baseline: Single modality perfor- mance on visual navigation & QA. In Proceedings of the 2019 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1977-1983, Minneapolis, Min- nesota. Association for Computational Linguisties.
Tzuf Paz- Paz-Argaman, Reut Tsarfaty, Gal Chechik, and Yuval Atzmon. 2020. ZEST: Zero-shot learning from text descriptions using textual similarity and visual summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 569-579, Online, Association for Computational Lin- guistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. 2016. Learning Deep Representations of Grainer Visual Descriptions. In 2016 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 49-58, Las Vegas, NV, USA. IBEE.
Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 251-260.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object Hallu- cination in Image Captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035-4045, Brussels, Belgium. Association for Computational Linguistics.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image de- scription evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 4575-4775
Eleanor Rosch. 1975. Cognitive reference points. Cog- nitive Psychology, 7(4):532-544 547.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexan- der C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-222.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Be- longie. 2011. Caltech-uesd birds 200. Technical Report ENS-TR-2011-001 California Institute of Technology.
Yongqin Xian, Christoph H. Lampert, Bernt Schiele. and Zeynep Akata, 2020. Zero-Shot Learning - A Comprehensive Evaluation of the Good, the Bad and the Ugly.
David Schlangen, Sina ZarrieB, and Casey Kenning- ton. 2016. Resolving References to Objects in Pho- tographs using the Bords-As-Classifiee Model. In Proceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 1213-1223 Berlin, Germany. Association for Computational Linguistics.
Guangyue Xu, Parisa Kordjamshidi, and Joyce Chai. 2021. Zero-shot compositional concept learning. Proceedings of the 1si Workshop on Meta Learning and Its Applications Natural Language Processing. pages 19 27, Online. Association for Computational Linguistics.
Edgar Schönfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. 2019. Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Carina Silberer, Vittorio Ferrari, and Mirella Lapata. 2017. Visually Grounded Mcaning Representations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39 1):2284-2297
Gabriel Skantze and Bram Willemsen. 2022. Col- lie: Continual learning of language grounding from language-image embeddings. J. Artif. Int. Res. 74.
Sina Zarrieß, Henrik Voigt, and Simeon Schüz. 2021. 1. Decoding methods neural language generation: A survey. Information, 12(9).

---

Haichao Zhang, Haonan Yu, and Wei Xu. 2018. In- teractive language acquisition with one-shot shot visual concept learning through conversational game. In Proceedings of the 56th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 1: Long Papers), pages 2609-2619, Melbourne, Aus- tralia. Association for Computational Linguistics.

---

A Example Appendix
This appendix shows examples of generated descriptions from different model architectures for a sample of 10 unseen and 10 seen classes. All examples were drawn from the first fold of the zero-shot splits. A sample ground-truth description is also shown for each class. The example images are drawn from the test set.
The metrics to the right show the performance of the zero-shot classification model given the description the left. Rank is the mean rank of the correct label, averaged over the images in the test set (the set consist of 5-6 images per class). As before, acc and acc@5 give the percentuge recall of the correct label in the top and top 5 guesses respectively. Note that for the seen examples, the performance of the zero-shot classifier is not directly related to the description on the left, since the model had supervised training for the seen classes and was not provided with text descriptions of them. We show these descriptions anyway, for comparison with the unseen descriptions, since the generation model was not trained on any text descriptions of unseen classes (in contrast to the seen classes, for which it had supervised training).
Model
Decoding
Description
acc
ground Iruth
the large white eyebrows, white belly, and small bill.
107.8
0.0
bearn
this bird wings that are black and has white belly and long black
63.7
0.0
0.0
nucleus
this bird brown
33.3
beam
this bird has wings that are black and has white belly. medium-sized bird that pointed bill.
52.5
0.0
16.7
prot
nucleus
15.0
0.0
16.7
Dinwny Winsspreker unseen
both
beam nucleus
this bird wings that are black and white helly.
82.5
0.0
0.0
this bird has a white and black eyering.
13.5
0.0
33.3
Model
Decoding
Description
mean rank
ace
acc G5
ground truth
this bird has white belly and wings with dark
110.5
0.0
0.0
preen primary feathers.
beam thacluus
this bind hus wings that are brown and has a white belly
136.2
0.0
exem
this bird has wings that are with black head dowswards point
68.2
0.0
0.0
beam
this bind has wings that are grey and bas a white belly. small hird with a short trinngular bill that curves downwards
152.8
0.0
prot
nacleus
114,5
0.0
0.0
Temnessee Warbler
unseen
both
beam mucleus
this bird has wings that are black and white helly.
99.7
0.0
small bird with black bill.
151.3
0.0
Model
Decoding
Deseription
mean rank
acc@95
ground truth
the bird has small black bill and small thighs
15.3
33.3
50.0
bird wings the hird has a small bill and blue hody.
blue
belly
145.3
nucleus
14.0
50.0
66.7
beum
this bird has wings that are black and has white belly brown
2.3
66.7
83.3
nucleus
birdl and yellow color,
Blue Grosbeak unseen
both
beam
has wings that black and white belly.
162.5
nucleus
medium sized hird that is mostly brown color.
60.0
0.0
0.0
Model
Decoding
Description
acc@5
ground
grey and white spocked
, pale yellow abdomen.
black eyes and feet, orunge beak.
bzam
this hird has wings that are brown and has white helly
35.2
0.0
0.0
that gray
beam
bird has wings that are brown and bas white belly
31.2
33.3
prot
nicleus
this hird brown in color with small sharp pointed heak
85.7
0.0
0.0
Olive sdedd Myeatccer
unseen
beam nucleus
bird has wings that black and white belly
84.5
0.0
bird white on black small
42.5
0.0

---

Model
Decoding
Deseription
rank
ace
acc
ground
small
grey color.
beam
this hird has wings that are black and has white helly hird. smooth, moatly black and a small white beak.
140.7
0.0
0.0
boann
this hird has wings that are black and has white helly hird wings of white the bead.
56.7
0.0
0.0
beak going
Gray Cabiid unseen
both
heam
this hird has wings that are black and white helly. white black
117.7
0.0
0.0
wings are
beak.
Model
Decoding
Description
mean rank
acc @1
acc@5
ground tuch
this little bird mostly white feathers with brown speckles.
9.0
16.7
50.0
beam nucleus
Lhis bird has wings that brown bus white belly
192.5
0,0
0.0
breast and beak
11.2
16.7
beam
this hird has wings that are brown and has white belly.
4.2
50.0
66.7
prul
nucleus
this bird bas white belly and rump.
13.0
16.7
Sage Thnsher unseen
heam
hird has that black and white helly
32.2
0.0
nucleus
bind has pointed bill. and long neck of red feathers.
28.8
0.0
0.0
Model
Decoding
Description
mean rank
ace @1
acc
ground truth
small bird with a significant head, needle bill, green crown
23.5
16.7
50.0
back
sccondaries, whise underside.
hird wings that brown
white helly
exem
nucleus
small green bird with black pointed beak belly.
92.3
0.0
0.0
hzam
this hird has wings that are brown and has white helly bird brown crown, black eyerings,
116.2
0.0
0.0
white throst.
Ruby Humminghin unseen
both
heam
this hird has wings that are black and white belly.
58.3
0.0
0.0
biad has white belly orunge crown yellow bill
on the secundary feathers on the distanze.
Model
Decoding
Description
mean rank
acc
acc@5
ground truth
medium sized bird has all black feathers a short, thick beak and (aat tail.
1.0
100.0
100.0
heam
this hird completely black beak medium size bird with black bill, black eyering and crown
18.8
nucleus
1.0
100.0
100.0
heam nucleus
hird has that black and has white helly
2.8
100.0
prot
bird solid black, with hini the bill is black larsus
7,3
0.0
50.0
Crow unseen
and long, the body.
beam nucleus
hird has wings that black and white belly.
2.0
100.0
birdl has small head and a short beak.
10.3
0.0
Model
Decoding
Description
acc@5
pround
small dark yellow colored
stripes on hody
2.5
with the exeception of the wings that are brown.
beam nucleus
this bird bas wings that black and has yellow belly grey
180.3
0.0
0.0
beam
bird has wings that are brown and has white belly..
85.3
0.0
prot
nucleus
this a bird with white belly and grey wings
138.5
0.0
Prairie Warbier unseen
beam
hird has wings that black and white belly. biral has wings that ure grey and hus a white striped tail.
183.7
0.0
nucleus
179.7
0.0

---

Model
Decoding
Deseription
rank
ace
acc@8
ground
has a yellow belly. and
rest of tan color.
beam
this hird has wings that are brown and has yellow helly
193.3
0.0
0.0
nucleus
on
heam
has wings that grey and a yellow
prot
nucleus
this bird has wings that are with yellow and has more
67.3
16.7
16.7
Philadelgghia Vicco
stripe black tip
unseen
both
beam niclens
bird has wings that ard black and white belly this hird is mosrly black excepe for it is heak which slightly
14.5
66.7
66.7
131.8
0.0
0.0
at tip
Model
Decoding
Description
mean rank
scc
acc@5
ground truch
this hird mostly yellow with slightly darker and
2.5
50.0
83.3
black crown and syebrow.
beam
this hird has wings that are brown and has yellow helly
2.3
50.0
83.3
exem
nucleus
small bird with pointed hill and black eyering.
2.2
50.0
83.3
beam
bird wings
stripes
2.2
50.0
Worm Washler
prot
nucleus
this bird has black bill and a crown.
2.0
50.0
100.0
saen
both
beam
this bind bas wings that are black and white belly.
2.3
50.0
83.3
bird
on
Model
Decoding
Description
rank
@
acc
ground truth
bird black crown short pointed white
2.5
and fuzzy brown body.
beam
this bird bas wings that brown and bas white belly
1,7
33.3
100.0
nucleus
has brown
beam
bird has wings that are brown and has white belly.
1.8
$0.0
100.0
prot
nueleus
this bird grey with brown and white on it wings
3,5
66.7
66.7
seen
beam nucleus
hird has wings that black and white belly...
1.8
66.7
100.0
the bird bas a yellow is short and orunge.
1.0
100.0
100.0
Model
Decoding
Description
mean rank
acc
acc
ground truth
black feathers on the top the hind with gray feathers on the breast and underside bird orange color the Eace of bird and long gray clows
40.0
80.0
bearn nucleus
this bird has wings that are black and has an orange hill medium stzed black bird, with short orange bill and
20.0
80.0
exem
20.0
80.0
bourn
this bird has wings that are black and has an orange beak.
20.0
80.0
prot
nucleus
the bird is small with color.
2.6
40.0
80.0
Anblet
saen
both
beam nucleus
this bird has wings that black and white helly. this bird has wings that are brown and has yellow check
20.0
80.0
40.0
80.0
patch
Model
Decoding
Description
acc
acc
ground
small brown and white sported with white beeast long claws
1.5
100.0
besen nucleus
Uhis bird bus wings that are brown and bas white belly this
1.3
83.3
100.0
has white eye with pointed
100.0
wings
botm nucleus
this bird bas wings that are brown and bas white belly. hird
1.7
66.7
100,0
prot
helly long legs
1.7
Beowss Creeper
seen
both
becmn nucleus
this bird has wings thai are black and white bellyy
1.5
83.3
100.0
the small bird has block bill and white eycring of its body.
1.3
83.3
100.0

---

Model
Decoding
Description
aceda
acc
ground
has skinny thighs
10.0
50.0
beam nucleus
this bird wings that are black has yellow belly the hird has a black hill and black eyering breast and brown
8.5
16.7
50.0
exem
9.8
16.7
50.0
retrices.
beam nuclous
wings are
has helly...
9.8
50.0
prot
this bird has wings that are and yellow eychrows and
8.5
16.7
50.0
white
Mystle Warhler
seen
both
bear nnclees
this bird has wings that are black and while belly...
10.2
16.7
16.7
small bird with black hill and white crown the wing.
7.2
16.7
50.0
Model
Decoding
Description
meaan rank
ace
acc@5
ground truth
the small hird has a white belly, brown head and sitt. ing
14.0
33.3
50.0
window scul
heam
this hird has wings that brown and has white helly
12.2
50.0
nucleus
bird is grey with black and has very short beak.
12.7
50.0
66.7
beam nuclens
this bird has wings that are brown and has white belly.
13.2
33.3
50.0
hird wings that brown and white and
12.5
50.0
66.7
House Sparrow
orunge beak.
seen
both
beam nucleus
this bird has wings that are black and white belly.
13.0
33.3
50.0
this bird has yellow and grey head and brown spotted body.
9.3
50.0
66.7
Model
Decoding
Deseription
rank
are
acc
ground
has black yellow body and light grey
1.0
100.0
beann nucleus
this bird bas wings that are black and has yellow belly
1.0
100.
100.0
exem
the bird has spotted belly and small bill crown.
1.3
66.7
100.0
heam
this hird has wings that are black and has yellow helly
1.3
66.7
100.0
prot
nucleus
a bird with small black pointed beak, red underbelly and
1.0
100.0
100.0
white head.
Baltimoee Osiole
seen
both
beam nucleus
this hird has wings that black and white belly. this bird a white bully and breast and neck above it eye
1.0
1.3
100.0
66.7
100.0
100.0
patch.
Model
Decoding
Description
mean rank
acc 81
acc@5
ground much
hird with a thin pointed bill, swepp back brown crown, and white
1.3
66.7
100.0
beam
has wings that brown has long bird with a long pointed bill.
50.0
esem
nucleus
1.3
66.7
100.0
beam nucleus
this bird has wings that black and has white threal...
1,3
66.7
100.0
brown whrie
eyes there
seen
both
beam nucleus
this hirdl has wings that are black and white belly.
1.3
66.7
100.0
this bird has wings that grey and has yellow mark them.
1,3
66.7
100.0
Model
Decoding
Description
mecan runk
ucc @1
acc
ground truhh
this a small red bird with brown wings and a small brown beak.
1.8
66.7
100.0
beam
bird wings that
and red bally
exem
nucleus
this hird has wings that are black and white.
1.8
66.7
100.0
beam
this bird has wings that are black and has red head
1.8
66.7
100.0
nucleus
particular bard belly
brown
Elycatcher
heam
and belly. the bird has white in 's wings and lurge head with brown beak.
has wings that
seen
both
nucleus
1.8
66.7
100.0

---

Model
Decoding
Description
rank
ace
acc@8
ground
gray bird with white
orange legs
inside beak, and black crown.
beam
this hird has wings that white and has a black crown
6.0
0.0
50.0
bird with grey wing and medium
heam
has wings that
crown
prot
nucleus
this bird white and black in color, with small beak
4,7
0.0
66.7
Common seen
beam
this hird has wings that black and white belly
5.7
50.0
nucleus
small bird has large black bill and brown crown white
3.7
83.3
belly.


Reviews:
[{"review_id": "RY34HJPa7T", "reviewer": "Reviewer_uNB9", "summary": "", "questions": "A. What is the performance like against prior works, e.g. CLIP? What is the reason for not  reporting them in the results section?\nB. What is the main motivation for using image captioning model over Wikipedia text descriptions in zero-shot classification, if there is no interaction between GEN and IPT on GEN generated captions?\nC. Why do we expect the proposed pipeline work in zero-shot setting if  the IPT model was trained on ground truth descriptions from seen classes.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! Although we believe we have strong responses to many of your concerns, your comments are very much appreciated and will certainly be useful in the next version of our paper. Below we address your questions and concerns.\n\n> The paper makes no comparison with prior zero-shot learning works in the result section to demonstrate the effectiveness of proposed modules. This also makes it difficult to understand the motivations and contribution of the paper.\n\nIt is not straightforward to compare our pipeline with existing zero-shot learning research as our research questions are quite different. Our primary contribution is a challenging setup that imitates human-human interaction in the process of learning perceptual concepts through language. While we use a zero-shot model (IPT) as part of an end-to-end evaluation pipeline, the experimental focus of the paper is on generating descriptions of perceptual classes (GEN).  The intuition is that if these descriptions are good, they will be useful to a zero-shot learner, so the performance of IPT offers insight into the quality of GEN. With respect to related work, our main points of comparison are other works that use zero- (or few-) shot learning to simulate human-like interaction in a multimodal setup (lines 221--235).\n\n> Why does the captioning model need to be trained over using Wikipedia class descriptions (Paz-Argaman et. al.; Bujwid and Sullivan et. al. especially if the IPT model is not trained with generated captions nor have any interaction with with GEN model?\n\nOur GEN model was not trained with Wikipedia descriptions. We refer to zero-shot image classification with Wikipedia texts in line 178 as part of the related work, but never train our models on this data. The GEN models is only trained with the ground-truth CUB captions described in section 4.1 (Akata et al., 2016). The models interact at the zero-shot learning/evaluation stage through natural language descriptions, but importantly do **not** share any internal weights. This simulates the scenario we are interested in when GEN and IPT model two separate learning agents. \n\n> Why cannot one simply use strong zero-shot image classification models (e.g. CLIP) for this task instead of the proposed pipeline?\n\nOur primary research question is not about zero-shot image classification from natural language descriptions. Rather, the main research question of this work is whether neural networks can use abstract class-level representations of visual concepts to generate class descriptions (GEN) and use such descriptions for classification of visual instances (IPT). While we are also interested in IPT architectures that move away from the image-text pairing paradigm, the IPT model is used primarily as a method of semi-extrinsic evaluation for GEN. CLIP is an example of a model that works in the image-text pairing paradigm, so it is unclear what would be learned by comparing it with our models, since our setup is designed to require abstract visual gronding at the class level. Moreover, we have no way of guaranteeing that instances from the unseen classes are not present in the CLIP pretraining data (in fact it is very likely that they are, we give more details in our comments below).\n\n> Usage of outdated pre-trained GEN and IPT models to support the claim. Here are some straightforward baseline models that the authors could have implemented \n> Feature Encoder: CLIP embeddings and different CLIP model sizes\n\nCLIP does not fit the zero-shot requirements for our setup. Note that is only necessary for CLIP to have seen images from bird _classes_ included in CUB to violate the zero-shot assumption. Specifically, CLIP has seen images from Birdsnap dataset, which overlaps with CUB in its classes. This is an interesting direction, however, we decided to focus on the modelling scenario itself rather then the models. But we do believe that using different pre-trained vision encoders is an important direction and we will investigate it more in the future once the setup and scenario is settled.\n\n> Image Captioning Models: Any pre-trained vision language captioning models, such as BLIP-2 [1]\n\nCaptioning models like BLIP-2 work in the image-text pairing paradigm, producing a description with a particular image as input. There is no way to ask such a model to produce a description of a visual class without reference to a particular instance of it, which is precisely what we are interested in with _perception-based category description_. However we would welcome any suggestions on the models which actually can provide us with class-level embeddings (not just instance-level ones). Our insistence on moving away from the image-text pairing paradgim is based on work in cognitive science that suggests that humans have perceptually grounded category representations that abstract away from particular instances of the category (lines 127--165). \n\n> Language Model IPT: RoBERTA, Generative LLMs such as LLAMA, GPT-3\n\nPre-trained models for interpretation part cannot be used for GEN since they don't take class representations as input (and would therefore need to be retrained from scratch). These models could be used for IPT, however there is little reason to believe they would perform significantly better than BERT given the short and relatively simple input texts. The main difficulty of the GEN model is fine-tuning the text-only pretarained LMs to the task at hand. We point to the comparison of frozen vs. fine-tuned BERT in section 3.4.\n\n> There are too many unnecessarily dense explanation of components that could be move to supplemental (architecture of simple classification models, explanation of beam search, nucleus sampling), leading to lack of space for more results and analysis (1 page of results and analysis).\n\nWe agree that moving description of decoding strategies into the appendix would make the paper more concise and provide space for more analysis. We will implement this in the next version of the paper.\n\n> Experiments are done only on one dataset (CUB) despite numerous benchmarks of of zero-shot classification.\n\nDomain-specific (in this case, bird world) visual concept representation and generating and understanding text based on such representations is our primary goal in the current study. This means that we are focusing on the challenging scenario where data items (seen or unseen) are highly similar and it is really hard for the models to understand fine-grained differences and similarities between them. However, we do plan in the future to expand and use other suitable datasets such as Bujwid and Sullivan (2021). Thank you for your comment.\n\n> A. What is the performance like against prior works, e.g. CLIP? What is the reason for not reporting them in the results section?\n\nWe agree that prior work on multi-modal pre-trained models is highly relevant (e.g., CLIP, ViLBERT, FLAMINGO) and incorporating experiments with such models would diversify our paper from the methodological point of view. For example, one can test whether CLIP-only IPT model can predict a bird class based on the image alone. However, we would like to emphasise that our primary motivation behind the current setup is to explore the limits and capabilities of (simple) neural architectures **without** rich multi-modal knowledge about the world in situations which are highly similar to how humans learn, describe and talk about new categories. The proposed approach is the first exploration of such paradigm in the modelling world of NLP, and as such, requires a thorough overview, evaluation and design. We assure the reviewer that our next step is to explore to what extent the models with prior knowledge about the world can improve on the task performance. As it is somehow clear that they probably will (each model and setup to a different degree), it's also important to establish the task and paradigm first. And this is what we are focusing on this paper.\n\n> B. What is the main motivation for using image captioning model over Wikipedia text descriptions in zero-shot classification, if there is no interaction between GEN and IPT on GEN generated captions?\n\nThe purpose of this paper it to propose _perception based category description_ as a novel task (lines 84--91). We use the zero-shot task primairily as a way of evaluating GEN (by comparing performance of IPT given ground-truth vs. GEN descriptions in the zero-shot learning stage). In this work, the only interaction between GEN and IPT in this zero-shot learning stage. One could implement additional iterations of interaction (for example with reinforcement learning), but thew novelty of this paper comes from the generation task (perception based category description) and from using the zero-shot task as a method of evaluation, rather than from any advances in zero-shot learning from text. \n\n> C. Why do we expect the proposed pipeline work in zero-shot setting if the IPT model was trained on ground truth descriptions from seen classes.\n\nThis question really gets at the main point of the paper. GEN is trained to generate descriptions based on class representations (which it learns through classification). IPT was trained to classify images (which results in class representations) and to (re)produce those class representations based ground-truth descriptions. If the GEN descriptions are (1) sufficiently similar in distribution to the ground-truth descriptions and (2) carry sufficient information to discriminate among classes, then we expect IPT to be able to use them for zero-shot classification. This expectation is validated by IPT's zero-shot performance using ground-truth descriptions. In this way, IPT gives us a way of obliquely comparing the generated descriptions to ground truth (similar to BLEu), but in a way that is sensitive to how useful the descriptions are for discriminating among classes. In the next version of the paper, we will try to make this central premise (lines 83--125) more clear, and we welcome any feedback on how to do so.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Proposes an interesting usage of discriminative language model to perform zero-shot image classification.\n- Provides an analysis of different sampling strategies (beam search, nuecleus) for caption generation and how they affect the classification performance.", "reasons_to_reject": "- The paper makes no comparison with prior zero-shot learning works in the result section to demonstrate the effectiveness of proposed modules. This also makes it difficult to understand the motivations and contribution of the paper.\n   - Why does the captioning model need to be trained over using Wikipedia class descriptions (Paz-Argaman et. al.; Bujwid and Sullivan et. al. especially if the IPT model is not trained with generated captions nor have any interaction with with GEN model? \n   - Why cannot one simply use strong zero-shot image classification models (e.g. CLIP) for this task instead of the proposed pipeline?  \n- Usage of outdated pre-trained GEN and IPT models to support the claim. Here are some straightforward baseline models that the authors could have implemented \n    - Feature Encoder: CLIP embeddings and different CLIP model sizes\n    - Image Captioning Models: Any pre-trained vision language captioning models, such as BLIP-2 [1]\n    - Language Model IPT: RoBERTA, Generative LLMs such as LLAMA, GPT-3  \n- There are too many unnecessarily dense explanation of components that could be move to supplemental (architecture of simple classification models, explanation of beam search, nucleus sampling), leading to lack of space for more results and analysis (1 page of results and analysis).\n- Experiments are done only on one dataset (CUB) despite numerous benchmarks of of zero-shot classification. \n\n[1]: BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models [Li et. al.]", "questions_to_authors": "A. What is the performance like against prior works, e.g. CLIP? What is the reason for not  reporting them in the results section?\nB. What is the main motivation for using image captioning model over Wikipedia text descriptions in zero-shot classification, if there is no interaction between GEN and IPT on GEN generated captions?\nC. Why do we expect the proposed pipeline work in zero-shot setting if  the IPT model was trained on ground truth descriptions from seen classes.", "paper_topic_and_main_contributions": "This paper explores the task of zero-shot image classification using the interaction scheme between the GEN image-captioning model that explains the concepts of class labels and the IPT language model (BERT-based) that interprets the generated descriptions to perform classification. It proposes a new discriminativity metric to analyze the IPT model. The observation is that generating less diverse descriptions with beam search over nucleus sampling leads to a better performance.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The paper includes related works that use Wikipedia class descriptions (Paz-Argaman et. al.; Bujwid and Sullivan et. al.), but do not make any comparison with them in the results section."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Use same font for \"seen\", \"unseen\" classes\n- Use $\\pm$ for std in Table 2, 3.\n- Table 3 should separate out metrics for GEN and IPT evaluations. No bold numbers make it to understand what is the take away message from the table."}}, {"review_id": "rys0bV04yw", "reviewer": "Reviewer_cpaf", "summary": "", "questions": "!. Although the descriptions generated by prototype with nucleus sampling achieve better BLEU score, the accuracy of the prediction is lower than the exemplar. Is there any explanation for this?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments! We address them in our response below and will certainly take them into account when updating this paper for the next version.\n\n> The paper is hard to read; the abstract and introduction are well written however, the method and evaluation is hard to follow (see suggestions section).\n\nWe will try to make our descriptions of methodology and evaluation clearer (more on this below).\n\n> Few details like numbers in the Table 2 are for Gen model with prototype or both are missing.\n\nWe report only the results of the best GEN model in Table 2, w.g. GEN-EX with beam-2 decoding (as described in the table's caption). Since the focus in this table is on the performance of the IPT model, we wanted to make it easier to compare with results from learning on ground-truth data. Comparison of IPT performance on different GEN models can be found in the last three columns of Table 3. \n\n> No other baselines except for random is evaluated, making it difficult to evaluate how good the method is as compared to others. Meta-learning also conceptually creates prototypes and an instance may belong to one of the prototypes, may be some baselines could use that intuition. \n\nWe refer back to our responses to uNB9 regarding strong baselines and existing work from others. We doubt that existing pre-trained models are suitable for use in the new task that we introduce in the paper. We also think that the results in this study can be seen as the \"initial baseline\" resulting from a straightforward architecture that follows the task description as closely as possible. Thank you for baseline suggestions, we think that it would be a valuable addition to the experiments. However, since the task is new and quite different from standard zero-shot scenario, we believe that our models can be considered baselines themselves.\n\n> Ablations to demonstrate the importance of different components and loss functions are missing.\n\nThis is definitely an interesting ablation study, we would like to implement these experiments in future iterations of our experiments. We also would like to note that the point of current study is to establish a general understanding (and a baseline) for the new task that we propose.\n\n> !. Although the descriptions generated by prototype with nucleus sampling achieve better BLEU score, the accuracy of the prediction is lower than the exemplar. Is there any explanation for this?\n\nWe believe that this observation is due to the quality of prototype representations: in our study, they might carry more useful information for classification than exemplar-based representations. While beam-generated texts are better in terms of automatic metrics for exemplar, they are lower in discriminativity than nucleus (lines 556-560). In fact, all beam-generated texts are lower in discriminativity, but they help IPT model more in the exemplar scenario, possibly because they are repetitive and \"safe\", the model sticks with. The fact that prototype+nucleus is more useful for IPT might suggest that diversity in nucleus-generated texts is controlled by prototype representations, making the resulting diversity less _random_ and more _tuned_ to the particular class. We believe that additional experiments are required to see whether nucleus-generated texts are hallucinations (random diversity) or not (class-tuned diversity).\nIn addition, generation evaluation metrics are known for their problems and inability to capture diversity of texts, and we observe such mismatch in our study (lines 621-625), therefore, one has to make careful conclusions about any automatic evaluation metric.\n\n\n> Classifier could be described after the generation model is described with a forward reference. The model description starts with classifer which breaks the flow as classifier suddenly appears in the description.\n\nThank you for this suggestion. We thought it is important to highlight the classifier in the overall architecture since classification is key to the creation of class-level visual categories.\nIn the final version of the paper, we will improve the flow by adopting your other suggestion of writing a high-level overview of the architecture which emphasises this point.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1 The idea of investigating the class-level representations for classification is interesting and intuitive from a human learning perspective.\n2. The model and results have some promise.", "reasons_to_reject": "1. The paper is hard to read; the abstract and introduction are well written however, the method and evaluation is hard to follow (see suggestions section). \n2. Few details like numbers in the Table 2 are for Gen model with prototype or both are missing.\n3. No other baselines except for random is evaluated, making it difficult to evaluate how good the method is as compared to others. Meta-learning also conceptually creates prototypes and an instance may belong to one of the prototypes, may be some baselines could use that intuition. \n4. Ablations to demonstrate the importance of different components and loss functions are missing.", "questions_to_authors": "!. Although the descriptions generated by prototype with nucleus sampling achieve better BLEU score, the accuracy of the prediction is lower than the exemplar. Is there any explanation for this?", "paper_topic_and_main_contributions": "The paper investigates the ability of classification models to learn and operate with class-level representations in a language-vision setting. They train separate generator for class description generation (based on a class label and its representation from a classifier) and an interpreter for class predictions and class representations. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Model: Better to start with a higher level overview of the model and its component. Then, each component is explained starting with the motivation or intuition of that model, then move to the input and output. Specific implementation details could be moved to another section. Such details in between breaks the flow of the reader in terms of higher level model understanding.\n2. Classifier could be described after the generation model is described with a forward reference. The model description starts with classifer which breaks the flow as classifier suddenly appears in the description.\n3. Detailed description of decoding algorithms could be moved to the appendix as these are well-known algorithms.\n\nLine 205: studies->studied\nLine 446: instead of 'different degrees of accuracy'- accuracy @ top 1,5,10 predictions\nLine 449: u instead of d, as it is used earlier for denoting the description to maintain uniformity.\nLine 525: mist->must\nLine 596: sstructure->structure\nTable 3: Bleu->BLEU\nTable 2: learning -> learning (caption)"}}, {"review_id": "DIRVXXALdA", "reviewer": "Reviewer_kZxL", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback! We appreciate your excitement about our work, and below we would like to address your concerns.\n\n> However, I believe that the experimental part is a bit limited. I understand the rationale of the method, but I believe that the author should try to strengthen their findings, maybe implementing some points discussed in the limitation section, e.g. testing on more datasets, or using other methods, like probing the representations for class-level information, input ablation studies, feature attributions, or other XAI methods to explain the ITP and the GEN network.\n\n> My suggestions are in light of the venue the authors are aiming at and the type of paper submitted, i.e. long. For the above-mentioned reasons I believe that in its current shape, this work would maybe fit a short paper submission, but I still encourage the authors in providing more experimental evidence.\n\nWe agree that there are many more points to be discussed and we will definitely continue on expanding the idea that we introduce here. We would like to highlight that this paper is aiming to introduce the task, modelling and evaluation setup and provide insights for future work on that topic.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- very interesting topic\n- well-motivated\n- interesting insights", "reasons_to_reject": "need more experimental evidence", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this work, the authors study the class-level grounding of a neural image captioner. In order to achieve this they train the interpreter network, which has to identify the caption class, based on the caption generated by the image-captioner, experimenting also with enriching the captioner representation with exemplars and prototypes.  Finally, they find discrepancies between traditional NLG metrics, and the zero-shot results based on the interpreter models, highlighting possible issues related to the lack of grounding of class-level information.\n\nThis is a super interesting work, grounding abstract linguistic expression is indeed a very relevant, though understudied problem in VL. The manuscript is overall clear and engaging, and the models and experimental description are quite clear and motivated. The results are well discussed; the authors pinpoint interesting questions, giving some interesting insights and the limitation section fairly highlights possible improvements. However, I believe that the experimental part is a bit limited. I understand the rationale of the method, but I believe that the author should try to strengthen their findings, maybe implementing some points discussed in the limitation section, e.g. testing on more datasets,  or using other methods, like probing the representations for class-level information, input ablation studies, feature attributions, or other XAI methods to explain the ITP and the GEN network. \n\nMy suggestions are in light of the venue the authors are aiming at and the type of paper submitted, i.e. long. For the above-mentioned reasons I believe that in its current shape, this work would maybe fit a short paper submission, but I still encourage the authors in providing more experimental evidence.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "typo lines 525 602"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of generating descriptions for perceptual concepts beyond instance representations. It evaluates communicative success by performing zero-shot classification with generated class descriptions. The proposed evaluation setup allows for fine-grained analysis of class description generation and interpretation separately. This is a very well-motivated and compelling setting for evaluating language grounding and communication, and it opens up the possibility of studying a variety of new questions. One point which reviewers did not make but I (as AC) am curious about is about human evaluation of generated class descriptions (i.e., replacing IPT with a human "listener"). Given the confusion regarding the paper's contribution (i.e., in contrast to contributing a method for the existing task of zero-shot image classification), I would suggest clarifying this point e.g. in the introduction. I also agree that additional experiments and analysis (e.g. on additional datasets) would further strengthen the contribution.
Decision: Accept-Main