I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classification
Authors: Muhammad Ferjad Naeem
Yongqin Xian
Luc Van Gool
Federico Tombari
Venue: NeurIPS 2022 Accept (2022)
Keywords: Zero-shot Learning
Multimodal learning
Transformer
Attention

Abstract: Despite the tremendous progress in zero-shot learning (ZSL), the majority of existing methods still rely on human-annotated attributes, which are difficult to annotate and scale. An unsupervised alternative is to represent each class using the word embedding associated with its semantic class name. However, word embeddings extracted from pre-trained language models do not necessarily capture visual similarities, resulting in poor zero-shot performance.  In this work, we argue that online textual documents e.g., Wikipedia, contain rich visual descriptions about object classes, therefore can be used as powerful unsupervised side information for ZSL. To this end, we propose I2DFormer, a novel transformer-based ZSL framework that jointly learns to encode images and documents by aligning both modalities in a shared embedding space. In order to distill discriminative visual words from noisy documents, we introduce a new cross-modal attention module that learns fine-grained interactions between image patches and document words. Consequently, our I2DFormer not only learns highly discriminative document embeddings that capture visual similarities but also gains the ability to localize visually relevant words in image regions. Quantitatively, we demonstrate that our I2DFormer significantly outperforms previous unsupervised semantic embeddings under both zero-shot and generalized zero-shot learning settings on three public datasets. Qualitatively, we show that our method leads to highly interpretable results where document words can be grounded in the image regions. 

I2Dmer::: Learning Image to Document Attention for Zero-Shot Image Classification
Muhammad Ferjad Naeem 1
Yongqin Xian
Luc Van Gool
Federico Tombari2 2,3
ETH Zürich
2 TUM
Google
Abstract
Despite the tremendous progress in zero-shot learning (ZSL), the majority of existing methods still rely on human-annotate attributes, which are difficult to annotate and scale. An unsupervised alternative is represent each class using the word embedding associated with its semantic class name. However, word embeddings extracted from pre-trained language models do not necessarily capture visual similarities, resulting in poor zero-shot performance. In this work, we argue that online textual documents, c.g., Wikipedia, contain rich visual descriptions about object classes, therefore can be used as powerful unsupervised side informa- tion for ZSL. To this end, we propose 12DFormer, a novel transformer- based ZSL. framework that jointly learns to encode images and documents by aligning both modalities in shared embedding space. In order to distill diseriminative visual words from noisy documents, introduce new cross-modal attention module that learns fine-go aen-grainedd interactions between image patches and document words. Consequently, our 12DFormer not only learns highly discriminative document embeddings that capture visual similarities but also gains the ability to localize visually relevant words in image regions. Quantitatively, 12DFormer significantly outperforms previous unsupervised semantic embeddings under both zero-shot and generalized zero-shot learning settings three public datasets. Qualitatively, we show that our method leads to highly interpretable re- sults where document words can be grounded in the image regions. Code available at https///atttubccc//////a////Doo
demonstrate that our
Introduction
"What does a tiger look like? a fierce animal that looks like scary, big cat with stripes." Tigers are not native to Japan, yet when the travelers coming from China described them in relation to native animals, it inspired a range of historic paintings depicting tigers in Japan. Humans possess an impressive ability to imagine and identify unseen objects from pure language descriptions. In computer vision, the ability to predict unseen classes called zero-shot learning, which can be achieved by transferring knowledge from seen classes using auxiliary side information (or semantic embeddings) e.g., attributes [22), word embeddings 17), etc. Although remarkable progress has been made, most of prior works 22 57 71 [O] rely human annotated attributes the side information. While attributes are appealing, they are often costly to annotate [48] 65 [1] and scale large datasets. Towards unsupervised semantic embeddings [46] [7 2 word embeddings can be easily obtained from pre-trained language models [35]. Yet, they often do not reflect fine-grained visual similarities, thus limiting the performance
The goal of this work is to learn visually aligned unsupervised semantic embeddings from online textual documents for zero-shot image classification. With the advent of the Internet, the collective knowledge of humans about the world has been distilled into online encyclopedias like Wikipedia. These encyclopedias present rich source of fine- inn-ggained auxiliary information for a model. While
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

the entries (referred to as documents) may describe an object class with rich visual details, they tend to contain a lot of noise. For example, an entry for 'horse' can define its appearance as well as interesting historic events it participated in. While the former is helpful for visual model, the latter might introduce noise making it challenging to fully exploit this knowledge.
In this work, we propose a novel model Image to Document Transformer (12DFormer) that learns align image and document pairs with their global representations as well as with token- wise representations, i.e., image patches and document words. As a result, without any image- level language supervision, our model is able to develop an understanding of different parts of an animal, its habitat, etc, leading to a more discriminative semantic embedding. We summarise our contributions as: (1) We propose a novel transformer based framework for ZSL with noisy documents. (2) Our novel Image to Document Attention (I2D Attention) module learns to identify visually discriminative properties in a document leading to a more discriminative semantic embedding. (3) Our model 12DFormer consistently improves the SOTA in unsupervised semantic embeddings on three challenging datasets, i.e., AWA2, CUB and FLO. Moreover, we qualitatively demonstrate that our model learns highly interpretable results. (4) We show that the learned document embedding can be used with any existing ZSL model to significantly improve its performance. To the best of our knowledge, 12DFormer is the first method to learn an attention-based embedding from noisy documents for ZSL without relying on any pretrained part localization model or attribute vocabulary.
2 Related Works
Zero-shot Learning aims to generalize model trained on seen classes onto a disjoint set of unseen classes using shared auxiliary information available for both sets [22]. Several methods in this direction learn a compatibility function between the image and the class embedding space 42 30 8, 27 1
26]. Another competing line of work uses generative models like GANs to learn A complementary line of work and training better image encoders [66] 7T].1.. Semantic embeddings are a crucial building block for all of these methods. However, despite its importance, it is a less studied topic. Human labeled attributes [57] [4] 5 [6] 29) have become the de-facto semantic embedding for most methods. However, they are hard and expensive to scale as they require human experts [88 65 [T].
68 56
the feature space of seen and unseen classes [8 59
focuses on learning improved visual-semantic embeddings
24 68
Learning semantic embeddings with minimal supervision aims to use cheap to obtain side information to learn a semantic embedding with minimal label information. Several works have explored using text corpora
alternative source of semantic embeddings. Some approaches
include using word embeddings from pretrained language models [63] 35| 47 (8) and knowledge graphs 552 19 5
27 to encode semantic similarities. Another line of work aims directly learn semantic embeddings from documents containing information about classes. Earlier works in this direction used TF-IDF 443 to directly embed the document in joint image space [14. Successive works have focused on reducing the noise in the document by using predefined attribute vocabulary (3), learning better weights for TF-IF embeddings (37 or complementing these embeddings with a part detection network 15 69 Recent works have incorporated Transformer based language models to directly embed document to a semantic embedding 220 6). However, all these works either learn the semantic embedding against the global image representation or use pretrained part detector for the human-labeles attributes to filter the relevant details. VGSE [62] instead proposes to directly learn semantic embeddings from images of seen classes and extrapolate them to the unseen classes by measuring their class name similarities. Our model, 12DFormer instead uses both the knowledge in text documents and the images of seen classes to learn a semantic embedding and ZSL model.
30
Learning cross- modal attention between image and text to ground text in images without region level supervision has been a long-studied problem in visual question answering, image captioning, etc.
12 40 4I Methods in this line of work learn a mapping between the region level features from an image and its caption. More recently. Transformers [50] have made breakthrough this field with models like ViLBERT (25 and FILIP 41 that learn a cross- modal attention to learn cross modal embeddings. They show that the grounding of text in the image naturally emerges as a by-product [60]. However, these works rely on having access to image-level text which is expensive to obtain. Our model instead addresses the much more challenging problem of learning a cross-modal embedding and attention from images and their class-level text document.
2

---

12D Attention
Image x
Document d
fareet anound
Document Transformer
Mexican violetear
Image Transformer
K
Desertiption
softnax
d
A
V
ppool
CLS
linear
SX,
CLS
product
Marix multiplication
Image paich enthedding
Jere(x)
( d)
Document word cnlickkling
12D Global
Figure 1: 12DFormer, our novel Transformer based model, uses noisy documents as auxiliary information to learn zero-shot model. The first part of the model, I2D Global, learns to encode images and noisy documents to shared embedding space. In order to distill discriminative local information from the document, we propose novel 12D Attention module that learns fine-grained interactions between image patches and document words. Together, the two modules learn a highly discriminative document semantic embedding I2DEmb.
3 Image to Document Transformer (12DFormer)
The vast majority of existing ZSL works utilize either aunan--nnnoaated attributes or word embeddings auxiliary information. In this work, instead utilize the textual collection of encyclopedia (wiki) entries of classes as side information given the wealth of free document collections describing object classes available the internet. To achieve that, we propose 12DFormer (Figure [ pure-transformer ZSL framework that learns to align image and document pairs with their global representations and with token-wise representations i.e., image patches and document words.
Notations. We define the classes that included in the training set seen classes )" and the classes that are excluded from training as unseen classes y" Let T {(x, y. d) x E X E y d E D*} be our training set where denotes an RGB image from the training images X' y its label belonging to the seen classes ) d is a document e.g., Wikipedia article, containing textual descriptions of the object class y, and D> is collection of documents describing seen classes. At test time, another collection of documents D describing the unseen classes y" will be made available to the model. This simulates an internet query fetch extra information about an unseen class. Those documents will be used as the side information to connect seen and unseen classes. The task of ZSL. is to make prediction among only unseen classes, while GZSL needs to predict both seen and unseen classes.
3.1 12D Global: Learning joint Image-Document Embeddings with Transformer
Our model is dual-stream transformer architecture that learns, respectively, an embedding function F, an image transformer [33 for images, and G, a document transformer (50). for text documents. The first part of our model learns global compatibility between the Image and the Document by our Image to Document(12D) Global module. On the image side, given an input image x E R H xC we reshape it into a sequence of flattened 2D patches Xp E NxP2xX) where (H, W) the size of an input image with C as the RGB channels, (P, P) is the size of each image patch, and N = HW/P p2 the resultant number of patches. Moreover, transformer to learn global image representation. Inspired by LiT [67]. we use a pretrained frozen image transformer [13]. This is followed by a learnable feature projection layer that maps the image embeddings to a joint image-document embedding space with dimensionality r. The image encoder F outputs fcLs(X) (x) E R' as the global image feature and (x) Nxr as the patch-wise image embedding for the input image where r is the feature dimension.
append CLS token to Xp the input to the image
On the document side, given a document d consisting of M words, we get its token-wise input feature representation with pretrained word embedding model. Note that we use words and tokens interchangeably use GloVe word features tokens [35. Since each document consists of
3

---

a typically long sequence of words, we further pass this feature representation through a learnable MLP as a token projection layer reduce the feature dimension and the memory footprint, yielding d, E RMx, where is the feature dimension as the output of the token projection layer. Our learnable document transformer consists of transformer encoder blocks with multi-head attention. We append a CLS E R" token to this sequence and pass it through the document transformer to get ICLS (d) E R' as the global document embedding and gr(d) E RMxx x as the word-wise text embedding for the input document. We later refer to the learned gCLs (d) document embedding (semantic embedding) I2DEmb that can be used by any ZSL method.
We define scoring function s X' X D + R that measures the similarity of any image x and document d pair. The scoring function computes the dot product between global image embedding fx).s()) and document embedding Scr.s(d (d. formulated as
(1) The learning objective is to make the scoring function assign high scores to correct image and document pairs and low scores to incorrect ones. Therefore, for particular training instance (x y, d). and De the collection of documents belonging to seen classes, we minimize the following cross-entropy loss
s(x, d) = fCLs(x) (CLs(d).
LCLS = -log( exp sxp d) Ed'e exp sx. d')
3.2 12D Attention: Learning Image Patch to Document Word Attention
Our I2D Global module essentially aligns image-document pairs using their global representations. While this paradigm has been popularized by influential works like CLIP [38], it relies on large amount of image-text pairs to learn all discriminative local features and represent them the output of CLS token. However, we are dealing with more challenging problem where the number of training images is small (a few thousand) and there is only one document associated with each class. Aligning two modalities at global level will be prone to overfitting and hard to generalize to unseen classes at test time. Moreover, our documents are directly collected from the Internet and therefore are noisy e.g., large portion of the words are irrelevant to visual appearance. To address these challenges, we propose 12D Attention, a novel cross-modality attention module, to learn fine-grained interaction between image patches and document words, capturing local features defined in the document such as body parts of an animal, their habitat in the form of image background, etc. We argue that learning these local mappings allows a model to generalize beyond the seen classes.
Our 12D Attention module takes as inputs the patch-wise embeddings (x(x) E RN of the image and the token- wwse embeddings gr (d) E R of the document. We task the model with searching for the disuallyrrelevnn words in the documents using image patches as the queries. More specifically, we define Q = fo (,),, as the image queries, K = gr ( as the text keys to compare with, and V = 9: (,),, .. as the text values to mix with after the search, where W. We and W. are learnable linear transformations, all in size r x r. The 12D Attention module estimates the mross-modal attention A(x, d) E R. M by computing dot product between every image patch and word pair followed by a softmax,
QKT VT
A(x, d) softmar
(3)
This attention matrix is used to compute new feature representations foa (x. d) E RR-xxr for all image patches as linear combinations of rows of the value matrix V i.c., fpa (x, d) A(x, d)V. Intuitively, this operation recomputes the image patch embeddings using the token-wise embeddings of relevant words in a document. To obtain the image-level embedding. we apply global pooling on the patch dimension of the new patch embeddings fpo (x, d), yielding fpu (x, d) E BRlxr, Afterwards, we compute the local alignment score between an amage-documen pair by applying simple linear layer,
Slocel (x, d) (fppa)
(4)
where H E x a learnable linear layer. Given a particular training example (x, y, d), we optimize the following cross-entropy loss
exp Stooel (x,
TAocal -log(
(5)
Ed'eD exp Stocal (x, d')
4

---

We do not use any skip connection similar to previous cross-modal attention blocks like ViLBERT 225 as we want the attention weighted embedding to directly give us a linearly separable representation. Our I2D Attention Module searches for relevant patch features in the document for each training class y E V's and learns to associate visual concepts with the noisy text in the document. The classification loss Loocol maximizes the contribution of discriminative words in the document and minimizes the contribution of irrelevant details about a class. Furthermore, calculating cross-entropy over the full seen set ensures that the model is aware of similar attributes between fine-grained classes and can pick additional cues to separate such classes. I2D Attention introduces minimal learnable parameters with only 4 additional linear layers and rather forces F and G to minimize irrelevant details in the tokenwise fp(x) and 9: (d) as well as the global fCLS(X) and 9CLS(d) (d) embeddings. This is in contrast to architectures like ViLBERT 25 where several self-attention layers are stacked top of the cross-modal module to further learn the output embedding with self-attention. We later show in our experiments that this hurt the performance in our data constrained zero-shot learning setup. Although documents have been explored before in ZSL., prior work uses fixed document embeddings that are encoded with TF-IDF [14 69 [5] extracted from a pretrained language model [6] (0). In contrast, our document embeddings are aided by the attention module to identify important details and thus are also assisted by this additional visual information.
3.3 Inference
Given an input image x, we search for the document that yields the highest compatibility score,
d arg max d').
d'eD
The search space includes only documents of unseen classes in zero-shot learning i.e., D D and all classes in generalized zero-shot learning (G7SL) i.e., D = D= U The final prediction is simply the class label associated with the document d. For GZSL, we apply calibrated stacking (9] to calibrate the activations of unseen classes on a held-out set to reduce the bias towards seen classes. We only use the output of the global prediction as it is computationally cheaper and has distilled the knowledge of patch-to-toker interactions while training. The attention between image patch and document words is computed as the explainability of the model's decision when required.
4 Experiments
We conduct extensive experiments on Animals with Attributes2 (AWA2) [57] Caltech-UCSI) Birds (CUB) (51) and Oxford Flowers (FI.O) (3). which are widely used datasets in 7SL. We follow the evaluation protocol and data splits proposed by Xian et al. (57]. Since the main focus of this work is to learn unsupervised semantic embeddings, attributes. In the following, we first describe how documents are collected and implementation details. Then, we quantitatively compare against SOTA unsupervised semantic embeddings methods and ZSL methods. Finally, we show quantitative results to demonstrate the interpretability of our method.
do not use any human-annotated
Collecting documents. We use online sources for documents that can be queried with minimal human supervision. These sources contain useful knowledge about each class but might have lot of noise as irrelevant textual details, For AWA2, we use A-Z Animals [53]. an animal encyclopedia. For CUB, we use AllAboutBirds (54). bird-watching encyclopedia. For FLO, we use a collection of gardening blogs and Wikipedia [55] to collect documents for these classes. However, we found documents for flowers to be less focused on the patterns of petals and pistils and rather more focused on the general description of the plant and its taxonomic biological classification. FLO is therefore a challenging dataset to generalize from document-baas based embeddings. We adopt a simple filtering step on these collected articles similar to (20 We look at the documents for 10% of classes of each dataset and identify sections that contain relevant information about the class. The rest of the documents are filtered to only contain these sections. The average size of a document is 4400 words. To put this into perspective, models like CLIP [38 64] use image captions of at max 64 tokens [38] 36). The long length of the documents presents an additional challenge. Document examples and their links are included in the supplementary and are available on the github repository.
Training Details. We implement our model in PyTorch and train on an Nvidia 1100 GPU. We use the VIT/B 16 checkpoint trained on ImageNet Ik by [13] as the pretrained Image Transformer. The
5

---

Zero-Shot Learning
Generalized Zero-Shot Learning
Semantic
AWA2
CUB
FLO
AWA2
CUB
FLO
Embedding
Source
T1
T1
H
uI
$
H
GloVe 35
CLSN
52.1
20.4
21.6
42.1
75.3
54.0
16.2
43.6
23.6
14.4
88.3
24.8
CloVe 35
DOC
61.6
29.0
25.8
49.5
78.1
60.6
23.8
62.6
34.5
14.7
91.0
25.3
LongFormer
DOC
44.2
22.6
8.8
41.6
81.8
55.2
19.9
41.0
26.8
8.8
89.8
16.0
MPNet 19
DOC
61.8
25.8
26.3
58.0
76.4
66.0
20.6
44.3
28.2
22.2
96.7
36.1
TF-IDF
DOC
46.4
39.9
34.0
29.6
44.2
29.0
52.1
37.3
28.9
94.8
44.3
VGSE[62]
IMG CLSN
69.6
37.1
56.9
82.8
67.4
27.6
70.6
39.7
F2DForeer(uur))
IMG + DOC
76.4
45.4
40.0
66.8
76.8
71.5
35.3
57.6
43.8
35.8
91.9
51.5
Table I: Comparing
12DFormer with unsupervised semantic embedding methods using the same image feature and method (our I2D Global module). In ZSL, we report top-1 1 accuracy (TT) on unseen classes, in GZSL on seen/unseen (s/u) classes and their harmonic mean (H). We consider semantic embeddings that are either directly extracted (with a pretrained language model) or learned from different sources including classnames (CLSN), document (DOC), combination of image and classnames (IMG+CLSN), and combination of image and document (IMG+DOC). Our 12DFormer significantly improves on the baselines set new SOTA for unsupervised class embeddings.
Zhro-Shot Learning
Generalized Zero-Shot Learning
Semantic
AWA2
CUB
FLO
AWA2
CUB
FLO
Type
ZSL Model
Embeddings
T1
T1
T1
u
H
H
u
GloVe
56.6
27.1
13.1
41.3
83.4
55.3
14.4
51.6
22.5
4,6
93.2
8.7
Diseriminative
S
VGSE
70.1
31.6
49.9
84.8
62.8
23.1
57.5
33.0
I2DEmb(Oury)
72.6
38.2
33.4
55.8
82.6
66.6
25.0
56.2
34.6
18.5
87.1
30.5
GlaVe
73.8
20.7
15.2
57.6
84.6
68.5
19.6
32.6
24.5
12.8
39.4
19.3
APN 61
VOSE
74.0
34.3
65.0
72.4
68.5
23.2
52.9
32.1
I2DEmb(Ours)
74.5
40.6
35.4
65.5
70.7
49.9
37.5
32.0
85.3
46.5
GlaVe
63.7
37.5
20.9
22.2
90.8
35.6
5.93
36.2
10.2
8,38
97.3
15.4
Generative
GAZSL 69
VOSE
74.7
35.7
29.5
93.8
44,9
10.5
51.8
10.5
I2DEmb(Ours)
83.1
42,9
34.2
56.8
94.7
71.0
15.9
50.4
24.1
28.8
90.1
43.7
CloVe
70.7
31.8
32.1
65.7
69.5
67.6
23.9
55.7
33.5
25.0
99.0
39.9
f- VAEGAG D2 59
VOSE
75.0
40.7
70.8
79.0
74.7
32.7
57.5
41.7
I2DEmb(Ours)
85.1
41.9
36,9
73.2
81.7
77.2
33.4
57.3
42.2
30.0
97.3
45.8
Disc.
(2DFormer(Ours)
I2DEmb (Ours)
76.4
45.4
40.0
66.8
76.8
71.5
35.3
57.6
43.8
35.8
91.9
51.5
Table 2: Comparing I2DFormer with baseline ZSL methods, under various unsupervised semantic embeddings see that model and embeddings I2DEmb set a new SOTA. In ZSL., we report top-1 accuracy (T1) on unseen classes, in GZSL on seen/unseen (s/u) classes and their harmonic mean (H). Best embedding results within method are underlined. Best results overall are bolded.
image patch projection and token projection layers are implemented as a shallow MLP. Maxpool or Mcanpool are chosen global pooling by ablation. The model is trained with Adam optimizer with learning rate -e and takes x24 hours to converge. LCLS and Lieea relative weights are chosen by ablation. More details are available in the supplementary. For baseline methods, we use the CLS features from the same VIT/B16 checkpoint with author's 's implementations. We ablate these methods multiple hyperparameters report the best run. For VGSE, released by the original authors (not available for FLO).
the semantic embeddings
4.1 Comparison with SOTA unsupervised semantic embeddings
In this section, we compare with existing unsupervised semantic embeddings where they are obtained without using human supervision using the same 7.SL. method (our 12D global module).
Compared semantic embeddings. For GloVe (classname) [35 we simply extract Glo vectors of class names. This method has been adopted by many prior ZSL methods [3 45
17
2 30 due
to its simplicity. For GloVe (Document) [35], we average over the feature vectors each word in the document. LongFormer 4 is text transformer model trained for documents and outputs CLS embedding given a document, MPNet[49) the current SOTA Sentence Transformer model(39)
trained to optimize embeddings for natural language classification tasks, Since the original model is

---

trained for short sequences, we average over the individual sentence embeddings similar to [20, 6l. TF-IDF 43] stands for Term Frequency-Inverss Document Frequency. which has been used by some
prior ZSL. methods 14 23 VGSE 62 learns the semantic embeddings from image patches and
word emheddings of class names. Since these embedding models generate one embedding for the whole document, we replace the Document Transformer with an equally deep MLP.
Results. From [ we observe that our method I2DFormer consistently outperforms all semantic embedding methods in both ZSL and GZSI.. Compared to GloVe (Doeument) 35 which also serves as an input to our method (without the average over words), the learned embedding of our model achieves an impressive 76.5% accuracy vs 61.6% on AWA2, 45.4 % vs 29.0 % on CUB and 40.0% vs 25.8% on FLO with relative 1.25 x, 1.5x x and 1.6x improvement each. This shows that our learned document embedding assisted by our I2D attention module significantly improves the zero-shot performance. We see that these improvements are also consistent in GZSL where we see a significant improvement in the HM. Similar results are observed for other pretrained language semantic embeddings Longformer, ormer, MPNet and TF-IDF 4 49 43]. Since the original embedding models for these baselines were only trained on language data, the generated semantic embedding is unlikely to capture the most visually discriminative features deseribed in the document. Our model however is able to learn a more informed semantic embedding thanks to supervision from the images of the seen classes. Comparing rows 1 and 2, we see that the use of documents over classnames leads to a major improvement as documents capture hetter class similarities. Finally, compared to VGSE [62], the current SOTA unsupervised semantic embedding, we observe that our model again substantially outperforms it. While both VGSE and our model exploit patch-wise similarities in images of different classes learn a class embedding, our model is additionally able complement this embedding with localized information available from the documents thanks to our 12D Attention.
4.2 Comparing with SOTA ZSL. methods
In this section, we compare our full model with existing SOTA zero-shot models across baseline embeddings and our learned document embedding. For fair comparison, evaluate those methods with the same VIT/B 16 image features. We show in Table 2 that our method 12DFormer, and our learned document embeddings I2DEmb achieve SOTA performance.
Compared to baselines, our model I2DFormer or our learned embedding I2DEmb consistently outperform all baseline ZSL methods and embeddings to establish a new SOTA. 12DFormer achieves SOTA ZSL performance CUB and FL.O, the fine-grained datasets. On CUB, 12DFormer achieve an impressive 45.4% compared to the closest 42.9% of GAZSL that also uses our I2DEmb. On FLO, 12DFormer achieves 40.0% compared to the closest 36.9% of f- f-VAEGAN-D2 that again uses our I2DEmb. In GZSL, on CUB, 12DFormer achieves 43.8% HM compared to the closest 42.2% of f-VABGAN-D2 (I2DEmb). On FLO, 12DFormer achieves an impressive 51.5% HM compared to the closest 5.8 of f-VAEGAN-D2 (I2DEmb). We would like to emphasize that our model is outperforming both the generative baselines in GZSL on these two datasets. Generative models have previously been shown to be the most competitive baselines in these datasets. However, since 12DFormer learn a fine-grained attention between the image patches and the words in the article, is able to outperform these baselines with this extra knowledge without requiring feature generation. On AWA2, a coarse classification dataset, we see that 12DFormer achieves SOTA performance among the Discriminative baselines. However, the best performance is achieved by the Generative baseline f- VAEGAN-D2 using our I2DEmb on this dataset. f-VAGAAN-22 with I2DEmb achieves the best ZSL accuracy of remarkable 85. 15.1% vs. the closest 83. 1% achieved by GAZSL(I2DEmb), with 12DFormer being the third. In GZSL, f-VAEGAN-D2 with I2DEmb achieves SOTA with an impressive HM of 77.2% followed by 74.7% of the same method with VGSE embeddings. 12DFormer a second this setting surpassing the remaining 3 ZSL methods across all embeddings. However, these basclines are only able to outperform 12DFormer with our learned I2DEmb.
4.3 Ablation study
What kind of Patch to Word Attention is required in ZSL? We study the importance of learning patch to word attention for Document based embeddings in Table [3a We see that while only training 12DGlobal can learn a competitive ZSL. model, it significantly improves and achieves SOTA performance with the introduction of our novel I2D Attention module in 12DFormer. We see
7

---

Model
AWA2
CUB
FLO
Input Embedding
AWA2
CUB
FLO
I2DGiobal
69.4
37.2
37.2
LongFormer
51.5
34.3
21.8
12DGlobal 64
67.3
35.7
38.3
MPNet 49
65.5
36.1
28.3
25
75.0
29.9
21.3
Vee
74.0
43.8
37.8
12DFormer
76.4
45.4
40.0
Ve5
76.4
45.4
40.0
(a)
(b)
Table 3: a) Ablation over 12DFormer. The proposed I2DGiobal module greatly benefits from the addition of 12D Attention to achieve SOTA performance. Comparing against FILIP and VilBERT cross-modal attention, we see that I2D Attention achieves SOTA. b) Ablating over input embeddings for our Document Transformer we see that older models like Word2 and GloVe serve as better input representation than modern Transformer-base language models.
relative 14%, 16%, and 8% improvement over I2DGlobal. This validates our hypothesis that the patch to word attention distills its knowledge to the global I2DEmb, improving its performance. In the same table, we also ablate over competing cross-modal attention modules. FILIP [64] is a recent method that proposes to associate each image patch to its most attended word. We see that this hurts the performance when using noisy Documents. ViLBERT [25 proposes a cross-modal attention module which is paired with self-attention block 50 to learn image embedding. We that while this improves the performance over I2DGlobal on AWA2, it leads to worse performance on our fine-grate ine-grained datasets CUB and FLO potentially due to the bigger model requiring more training data. Our 12D Attention outperforms both these baselines and achieves SOTA performance.
What kind of input text representation works best for I2DFormer? We ablate over several pretrained word/ token representations be used
Document Transformer in Table 3b and note that Glo [35] achieves the best result. We observe that the two Transformed based language models Longerrmer 4 and MPNet [49) perform much worse than older baselines Word2 (28) and GloVe (35]. We conjecture that this is due to the Document Transformer having limited text data for the seen classes while training. Transformer- based models generate different word features for the same word with self-attention [50). Documents of unseen classes use the same and additional vocabulary in new sentences causing distribution shift in their input representation.
input
4.4 Qualitative Results
Document Transformer attention for I2DEmb. We look at the learned attention over documents of unseen classes in Figure 2a and plot the top 8 most attended words across the Document Transformer attention heads for I2DEmb. On AWA2, we see that class name is complemented with human-like labelled attributes for these classes such the color of the animal, type of the feet, and habitat etc. For the fine-grained datasets, CUB and FLO, we see that for similar classes like the two warblers, the model learns similar attributes like "ruby-crowned" "chestnut patterns". We confirm our hypothesis that a learned document embedding will focus on discriminating properties of the class from the noisy document.
well discriminating "tiger stripes" vs
Visualizing Image to Document attention as the row of the attention matrix in Figure 2b we see that the top 3 words contributing to the fp for these patch are visually grounded in the image. I2D Attention is able to develop this localization in the image for unseen classes from the noisy documents without any patch, word associated ground truth. We further see that the class name can repeatedly be the most important cue for the model in multiple sentences of the document for some patches. inssstaan
Visualizing Document word to Image attention as the column of the attention matrix in Figure 3 we see the impressive localization ability of 12DFormer for the top attended words in I2DEmb. We that the model is able to localize the unseen classes horse and giraffe in the image despite never observing them while training. The discriminating properties like the hoofed legs are also localized in the image. For CUB, we see that between the two very similar images of two unseen classes, the model identifies the yellow bottom as an important property from the two different documents of the ground truth class. However, the model further able to identify the discriminative tiger stripes of the Cape May Warbler to differentiate it from the Tropical Kingbird which has gray-green feathers leading to correct classification. Finally, on FLO, the localization ability of our model remains consistent where the Peruvian lily is identified by localizing it as a Lily and identifying its
8

---

Giarlly Rood
Clasanaee
Top atlended 1209bbb
Blue Whale
polar, whale, enomuss
bluemigraaing
AWYA
Shoep
grass, slerker, weely. stocky, mims, hores, shoep. hroves friid, fur, hind, cetaccans, pionjoals, seal, ocean narkings, hooted, giratte, enomoss, mummals,
Lage, Peedied
Scal
Giruffe
woodlands,
Rove,
Green Violetear
bronry, balivis, nicaragin,
chest, gray headed, kinghird preen, venczuela, whitish, flyatcher, gray green feathen, plumage yellowish, breast, short trailed green, tiger sttpes, sirve. raby crowned, cape green. ruby crowee, warbler,
colitei, shining, gliteinn
Tropical Kängbärd
Erilis Codttomes
Cupe May Warbler
Chostmutsided Warbler
markings, winghum,
Nocklaod
Pink Primrose
buttaccupp rammealus, siggns, wildfower, prmnosc four-peusld, evening blooss, tmupola, daiey, wealy, thislles flonts, shurp mothed wrikkled, fily, mher, seipped curving petils flacked, aksrnemeria, strasked, resupinste
Asticuonss Flonas
Thistle
Peruvion Lily
Maapp Flricherd
lily
butbless, capsules, lilim,
pollinated, england
Jubl
(a)
(b)
Figure 2: a) Top attended words for I2DEmb for unseen classes in the Document Transformer consist of discriminative properties available in the document. b) Visualizing Image patch to Word attention, we see that the top 3 most important words are visually grounded in the image.
Glmdlic
Cape Warder
Trapical Kinglee
Usdupan Yalowaa
Tigar Stiper
Vello bakom
Cney gaoe foabas
Glde Thade
Par
Strped
Curvingg
Figure 3: Visualizing Word to Image Attention the most attended document words for I2DEmb, we see that our model 12DFormer has learned to localize them in the image without any paired patch-word supervision. This learned attention differentiates the two similar birds in the second row by identifying and localizing tiger stripes and gray-green discriminative properties.
stripped and curved petals. Similarly for Globe Thistle, the model is able to differentiate the sharp teeth, soft and wrinkled parts of the flower. The prevalence of these words as top attended words in document transformer and their impressive localization verifies our hypothesis that the attention module distills its knowledge to the CLS head. A model that does not learn patch to word attention can miss these properties they are not deemed important among the seen classes.
5 Conclusion
We propose 12DFormer, a fully Transformer based framework for learning semantic embeddings from noisy documents. Our 12D Global module learns shared embedding space between an image and document embeddings. This is assisted by our 12D Attention module learns local features about the class defined in the document without any paired image-level captions. As result, our full model 12DFormer achieves SOTA performance on both ZSL and GZSL with respect to baseline semantic embedding baselines and zero-shot models. In addition, our model develops an impressive ability to identify and localize discriminative properties of a class from the document in the image. Finally, we show that the learned embeddings from our model can further improve all zero-shot methods. Broader Impact. We hope to inspire ZSL works in new fields like medicine, climate change etc. that were limited by expert attribute annotations. Online documents paired with our method can extend the current datasets for these fields. A potential down side of using unsupervised online content is that it can cause unintended biases against gender, race etc. if represented in the collected data. Future works this direction can study using content moderation tool-kits to clean this knowledge. Acknowledgemenn Ferjad is supported by Google Ph.D. Fellowship. We want to thank Wenjia Xu for helpful discussion and support for VGSE related experiments.

---

References
[1] Zeynep Akata, Florent Perronnin, Zaid Harchaoui, and Cordelia Schmid. Label-embedding for image classification. In T-PAMI. IEEE, 2015.
[2] Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embed- dings for fine-grained image classification, CVPR, 2015.
[3] Ziad Al-Halah and Rainer Stiefelhagen. Automatic discovery. association estimation and learning of semantic attributes for a thousand categories. In CVPR, 2017.
[4] Iz Beltagy, Matthew Peters, and Arman Cohan.
Longformer: The long-document transformer. In
arXiv:2044.05 2020.
Maxime Bucher, Stéphane Herbin, and Frédéric Jurie. Generating visual representations for zero-shot classification. In ICCV Workshops, 2017.
[6] Schastian Bujwid and Josephine Sullivan, Large-scale zero-shot image classification from rich and diverse textual descriptions. In LANTERN, 2021.
[7] Yannick Cacheux, Herve Le Borgne, and Michel Crucianu. Modeling inter and intra-class relations the triplct loss for zero-shot learning. In ICCV, 2019.
[8] Soravit Changpinyo, Wei-Lun Chao, Boqing Gong, and Fei Sha. Synthesized classifiers for zero-shot learning. In CVPR, 2016.
[9] Wui-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In ECCV, 2016.
[10] Shiming Chen, Wenjie Wang, Beihao Xia. Qinmu Peng. Xinge You, Feng Zheng, and Ling Shao. Free: Feature refinement for generalized zero-shot learning. ICCV, 2021.
[1]] Abhishek Das. Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. Visual dialog. In CVPR, 2017.
[12] Harm De Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C Courville. Modulating early visual processing by language. In NIPS, 2017.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[14] Mohamed Elhoseiny, Babak Saleh, and Ahmed Elgammal. Write classifier: Zero-shot learning using purely textual descriptions. CVPR, 2013.
[15] Mohamed Elhoseiny. Yizhe Zhu, Han Zhang, and Ahmed Elgammal. Link the head to the" beak": Zero shot learning from noisy text description part precision. In CVPR, 2017.
[16] Ali Farhadi, lan Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In CVPR, 2009.
[17] Andrea Frome, Greg Corrado, Jonathon Shlens. Samy Bengio, Jeffrey Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: deep vesual-semantic embedding model. In NIPS, 2013.
[18] Huajic Jiang, Ruiping Wang, Shiguang Shan, and Xilin Chen. Transferable contrastive network for generalized zero-shot learning. In ICCV, 2019.
[19] Michael Kampffmeyer, Yinho Chen, Xiandan Liang, Hao Wang. Yujia Zhang, and Eric P Xing. Rethinking knowledge graph propagation for zero-shot learning. In CVPR, 2019.
[20] Jihyung Kil and Wei-Lun Chao. Revisiting document representations for large-scale zero-shot learning. In NAACL, 2021.
[21] Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. Generalized zero-shot learning via synthesized examples. In CVPR, 2018.
[22] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. CVPR, 2009.
[23] Jimmy Lei Ba, Kevin Swersky, Sanja Fidler, et al. Predicting deep zero-shot convolutional neural networks using textual descriptions. In ICCV, 2015.
[24] Shichen Liu, Mingsheng Long. Jianmin Wang, and Michael Jordan. Generalized zero-shot learning with deep calibration network. In NewrlPS, 2018.
[25] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-anddlanguugge tasks. NeurlPS, volume 32, 2019.
[26] Massimiliano Mancini, Muhammad Ferjad Naeem, Yongqin Xian, and Zeynep Akata. Open world compositional zero-shot learning. In CVPR, 2021.
10

---

[27]
Massimiliano Mancini, Muhammad Ferjad Naeem, Yonggin Xian, and Zeynep Akata. Learning graph embeddings for open world compositional zero-shot learning. In T-PAMI. IEEE, 2022.
[28] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013.
[29] Muhammad Ferjad Nacem, Evin Pinar Ömek, Yongqin Xian, Luc Van Gool, and Federico Tomhari. 3D Compositional Zero-shot Learning with DeCompositional Consensus, In ECCV, 2022.
[30] Muhammad Ferjad Naeem, Yonggin Xian, Federico Tombari, and Zeynep Akata. Learning graph embed- dings for compositional zero-shot learning. In CVPR, 2021.
[31] Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees GM Snoek, and Ling Shao. Latent embedding feedback and discriminative features for zero-shot classification. In ECCV, 2020.
[32] M-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In ICCVGI, 2008.
[33] Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shiens, Andrea Frome, Greg Corrado, and Jeffrey Dean. Zero-shot learning by convex combination of semantic embeddings. In ICLR, 2014.
[34] Genevieve Patterson, Chen Xu, Hang Su, and James Hays. The sun attribute database: Beyond categories for deeper scene understanding. In IJCV. Springer, 2014.
[35] Jeffiey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
[36] Hieu Pham, Zihang Dai, Golnaz Ghiasi, Hanxiao Liu, Adams Wei Ya, Minh-Thang Luong, Mingxing Tan, and Quoe V I.c. Combined scaling for zero-shot transfer learning. In arXiv:21 10050, 2021.
[37] Ruizhi Qiao, Linggiao Liu, Chunhua Shen, and Anton Van Den Hengel. Less is more: zero-shot learning from online textual documents with noise suppression. In CVPR, 2016.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICLR, 2021.
[39] Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP, 2019.
[40] Anna Rohrbach, Marcus Rohrbach, Ronghang Hu, Trevor Darrell, and Bemt Schiele. Grounding of textual phrases images by reconstruction. ECCV, 2016.
[41] Anna Rohrbach, Marcus Rohrbach, Siyu Tang. Seong Joon Oh, and Bertt Schiele. Generating descriptions with grounded and co-referenced people. In CVPR, 2017.
[42] Bernardioo Romera-Pareee Paredes and Philip Torr. An embarrassingly simple approach to zero-shot Icarning. In ICML, 2015.
[43] Gerard Salton and Christopher Buckley. Tern-weighting approaches in automatic text retrieval. In Information processing & management, 1988.
[44] Edgar Schonfeld, Sayna Ebrahimi, Samarth Sinha, Trevor Darrell, and Zeynep Akata. Generalized zero-and few-shot learning via aligned variational autoencoders. In CVPR, 2019.
[45] Richard Socher, Milind Ganjoo, Christopher Manning, and Andrew Ng. Zero-shot learning through cross modsl transfer. In NIPS, 2013.
[46] Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D Manning, and Andrew
Ng. Zero-shot learning through cross-modal transfer. In NIPS, 2013.
[47] Richard Socher, Milind Ganjoo, Hamsa Sridhar, Osbert Bastani, Christopher D Manning, and Andrew Y Ng. Zero-shot learning through cross-modal transfer. In NIPS, 2013.
[48] Jie Song, Chengchao Shen, Jie Lei. An- Xiang Zeng, Kairi Ou, Dacheng Tao, and Mingli Song. Selective zero-shot classification with augmented attributes. In ECCV, 2018
[49] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tic- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In NeurIPS, 2020.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
[51] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-201 1001,, California Institute of Technology, 2011.
[52] Xiaolong Wang, Yufei Yc, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR, 2018
[53] Website. A-z animals. https: //a- animals.c moimals.com/

---

[54] Website. All about birds. htttp::/www.allaabotbbbiit org/
[55] Website. Wikipedia. htap:://en...ikipeddiee. org/
[56] Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for zhot classification. In CVPR. 2016.
[57] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning-a comprehensiie evaluation of the good, the bad and the ugly. In T-PAMI, 2019.
[58] Yongqin Xian, Tobias Lorenz, Bernt Schiele, Zeynep Akata, Feature generating networks zero-shot learning. In CVPR, 2018.
[59] Yonggin Xian, Saurabh Sharma, Bernt Schiele. and Zeynep Akata, f-vaegan-d2. A feature generating framework for any-shot learning. CVPR, 2019.
[60] Jiarui Xu, Shalini De Mello, Sifci Liu, Wonmin Bycon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In arXiv preprint arXiv:2202.11094, 2022.
[61] Wenjia Xu, Yonggin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Attrihute prototype network for zero-shot learning. In NeurIPS, 2020.
[62] Wenjia Xu, Yongqin Xian, Jiuniu Wang, Bernt Schiele, and Zeynep Akata. Vgse: Visually-grounded semantic embeddings for zero-shot learning. In CVPR, 2022.
[63] Ikuya Yamada, Akari Asai, Jin Sakuma, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji, and Yuji Matsumoto. Wikipedia2vec: An efficient toolkit learning and visualizing the embeddings of words and entities from wikipedia. In ACI, 2020.
[64] Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu. Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image pre- pre-training. ICLR, 2022.
[65] Felix X Yu, Liangliang Cao, Rogerio Feris, John R Smith, and Shih-Fu Chang. Designing category-level attributes for discriminative visual recognition. In CVPR, 2013.
[66] Yunlong Yu, Zhong Ji, Yanwei Fu, Jichang Guo, Yanwei Pang, Zhongfei Mark Zhang, et al. Stacked mamantics-guider attention model for fine-grained zero-shot learning. In NeurIPS, 2018.
[67] Xiaohua Zhai, Xiao Wang, Basil Mustafa. Andreas Steiner, Daniel Keysers. Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022.
[68] Li Zhang. Tao Xiang, and Shaogang Gong. Learning deep embedding model for zero-shot learning. In CVPR, 2017.
[69] Yizhe Zhu, Mohamed Elhoseiny, Bingehen Liu, Xi Peng, and Ahmed Elgammal. generative adversarial approach for zero-shot learning from noisy texts. In CVPR, 2018.
[70] Yizhe Zhu, Jianwen Xic, Bingehen Liu, and Ahmed Elgammal. Learning Seature-to-fcattue translator by alternating prok-pronnnn for generative zero-shot learning. In ICCV, 2019.
[71] Yizhe Zhu, Jianwen Xie, Zhiqiang Tang. Xi Peng, and Ahmed Elgammal. Semantic-guided multi-attention localization for sero-shot learning. In NeurlPS, 2019.
12


Reviews:
[{"review_id": "qSA8bskeCY1", "reviewer": "Reviewer_gQFr", "summary": "The authors propose an attention-based model for zero-shot learning from text documents - knowledge sources such as Wikipedia and bird/flower information websites. The proposed approach relies on two Transformer models: one for processing image patches, and another one for text documents. Each of the Transformers outputs a sequence of features which is fed into two different components of the proposed model. One of them fuses two sequences with an attention mechanism (I2D Attention) and produces an image-text matching score, described as local. The other component (I2D Global) only uses special classification tokens ([CLS]) from both text and image and similarly produces an image-text matching score, which the authors describe as global. The approach optimizes for both local and global matching but only the global one is used later on for classification/generating predictions.\n\nThe authors evaluate their model on standard ZSL datasets: AWA2, CUB, and FLO. They separately evaluate the performance of both their learned text features and their entire model that uses the global matching scores. When using their learned text features as input to existing four different ZSL models they observe quite a consistent improvement over alternative GloVe or VGSE features, often by a big margin. Also, using their entire proposed model, with global text-image matching scores used for classification, generally compares well against the other ZSL models.\n\nAdditionally, the authors argue that their model is more interpretable, showing examples of attention maps over words in a document or matching image-text element pairs.", "questions": "**Questions:**\n\n- (Q1) Table 3 (b): How are different input features (e.g. GloVe) used in the Document Transformer? Is it just an input to a Transformer instead of learning token embeddings (as in the first layer of Transformer)? Are those text features fixed or fine-tuned?\n\n- (Q2) Are any of either the Image or Document Transformer fine-tuned? The Image Transformer seems to be mentioned to be pre-trained but what about Document Transformer? Is it also pre-trained in any way or trained from scratch?\n\n- (Q3) What are the differences between text lengths of different documents? Since the softmax is computed over all classes, and texts lengths can be different, doesn’t it cause significant computational issues? Different lengths would require masking and padding\n\n- (Q4) How does the proposed model compare with other ZSL models with respect to computational requirements - both in training and generating predictions?\n\n- (Q5) How are soft attention maps produced (e.g. Figure 3) if the method only operates on image patches?\n\n- (Q6) Analysis of differences in classification between $s$ and $s_{local}$ predictions (Table 1 in the Appendix): are there any important differences other than just a different level of accuracy? Do they make different types of mistakes? If the two modules are complementary to each other some error analysis from the corresponding scoring function could be insightful\n\n- (Q7) Eq. 4: What is $\\hat{x}\\_{pa}$? Shouldn’t it be $\\hat{f}_{pa}$ instead?\n    - Why H in eq. 4 is a function while in L167 it says it’s a vector?\n\n---\n\n**Suggestions:**\n\n- Ablations in the Appendix, Table 1 seem important - I would recommend at least mentioning the main observations in the main paper\n\n**Minor issues:**\n\n- L50: “consistent” —> “consistently”?\n\n- Table 3: “Modal” —> “Model”?\n\n- Figure 1 doesn’t seem to be referred to anywhere in the text\n\n- Hyperlinks should be highlighted somehow - they do not seem to be displayed in any special way - just as normal text - very easy to miss them\n\n- Terminology: Instead of document “embedding” - I’d rather say encoding/representations/features. Typically, “embedding” refers to basically a look-up for discrete objects (e.g. tokens) - the terminology used in this paper with “embeddings” used as a name for any features or representations makes it very easy to confuse with e.g. Transformer’s embeddings (in the very first layer).", "limitations": "The authors do not discuss the computational cost of training & evaluating their model as opposed to alternative approaches. The qualitative analysis seems to include only selected positive example, with no fail-case analysis (see W3). Additionally, the authors use some very strong, over-exaggerated language - in L45 they claim that their model is “able to develop understanding of different parts of an animal”.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- (S1) The proposed approach is conceptually simple - it consists mostly of dot-product attention. It doesn’t require many extra hyperparameters.\n\n- (S2) The experimental results. The learned document text features (I2DEmb) quite consistently, both among different datasets and different methods, outperform GloVe and VGSE features: often by a large margin. Additionally, the entire proposed model that uses a very simple similarity score between text and image [CLS] token often outperforms the remaining ZSL methods - even models that use the text features proposed in this work (I2DEmb)\n\n- (S3) Rich ablation studies and a good set of experiments: the experiments show the importance of different components of the proposed model. The authors evaluate their model with different types of textual features but also use their textual features with different models. The set of experiments covers the most important aspects of the model.\n\n- (S4) Attention maps provide some extra interpretability which can be important for understanding predictions, especially when using text documents as a source of information about classes (although see W3)\n\n**Weaknesses:**\n\n- (W1) Possibly suboptimal settings for ZSL models used to compare against. GAZSL was introduced using TF-IDF features, e.g. APN was introduced for attribute features. This paper however compares against them when using either GloVe or VGSE features - which might be suboptimal for those models. A more fair comparison should at least additionally include TF-IDF features as well (especially important for GAZSL).\n\n- (W2) The authors seem to miss a little bit of context regarding the attribute data. The usage of text documents instead of attributes, in general, might have many advantages. However, e.g. f-VAEGAN-D2, APN papers using attribute data report much higher performance on e.g. CUB dataset. The authors however do not mention any attribute-based results which might be misleading for the readers, as if attribute features were not competitive even on these datasets. Additionally, the claims of outperforming SOTA should be formulated more precisely - e.g. with respect to a specific type of data/source of information\n\n- (W3) The qualitative evaluation samples are not indicated as random samples, as opposed to selected. Additionally, for more convincing/insightful results an analysis of failure cases would be needed\n\n- (W4) The authors, without justifying, collect their own text documents instead of already standard existing ones already extracted. There is no motivation for it present in the paper and no discussion or analysis of the differences and the impact of using these vs. standard documents. Additionally, the authors mention performing some filtering of document sections, which could potentially be very important but is not discussed/analyzed further\n    - Existing extracted documents (Wikipedia, AllAboutBirds, etc.):\n        - FLO, CUB: Mohamed Elhoseiny, Ahmed Elgammal, and Babak Saleh. Write a classifier: Predicting visual classifiers from unstructured text. TPAMI 2016.\n        - CUB: Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed Elgammal. Link the head to the” beak”: Zero shot learning from noisy text description at part precision. CVPR 2017\n\n(W5) The “direction” of the attention seems somewhat arbitrary and not analyzed. The authors basically weight the text features (attention values) by the attention compatibility between image patches & text sequence elements. One could do the opposite - use image patch features and weight them by attention maps instead. Or do both directions - like some cross-attention works. No motivation behind this choice is discussed and no comparison of alternative choices is present.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mJ6cMXtzHl", "reviewer": "Reviewer_oTDz", "summary": "The authors propose a method to learn a joint representation of an image with a very generic description of the object present in the image. This representation makes it possible to associate the discriminating elements of the textual description with visual elements of the image. The objective is to improve the zero-shot learning approaches and to classify images containing objects not seen during the learning phase only from the textual description. The proposed model is composed of two parts: a first part combines an image-based transformer and a text-based transformer through a scoring function that allows to compute the similarity between text embedding and image embedding. A second part allows alignment using visual queries and textual keys in a combined text and image transformer.\nExperiments are conducted on standard datasets for zero-shot learning. Wikipedia articles were collected to serve as class descriptions. The collected dataset will be made public after the review process.\nExperiments show that the proposed model outperforms state-of-the-art models and allows the classification of images containing objects never seen during the learning process based on their textual description only (unseen classes). Different types of semantic embedding are tested (glove, longformer, mpnet, tfidf). An ablation study and examples of qualitative results are presented.\n", "questions": "I am not familiar enough with the field of zero-shot learning to ask further questions.", "limitations": "NA", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am not familiar with the field of zero-shot learning but it seems to me that the model combining visual queries and textual keys in a transform model is original. The performances of the model seem to exceed the state of the art results and the model obtains interesting performances with simple embedding (Glove), which puts in perspective the contribution of more complex models like longformer for zero-shot learning problems. The bibliography seems to be very complete, again with citations of simple but efficient models (tfidf). The proposed model seems to be able to be trained on a single A100 GPU in one day, which is accessible. The experimental part is very complete with comparison to the state of the art, testing of different embeddings, ablation study and qualitative analysis.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jE2uEkq4F7M", "reviewer": "Reviewer_Sfca", "summary": "In this paper, authors propose to address the zero-shot image classification problem under a more realistic setting, namely each class is descried with one document collected online. To address the issues with online textual documents contain certain noise, and different parts of the document may correspond to the different regions of the images, authors propose a transformer-based ZSL framework that jointly learns to encode images and documents by aligning both modalities in a shared embedding space. The cross-modality attention mechanism is introduced to suppress the noise. And extensive experiments are conducted to validate the proposed modules.", "questions": "- The implementation details regarding the compared methods should be given, since different noisy textual source is utilized under the current experimental settings;\n\n- What about the model complexity, FLOPs of the proposed models compared to other SOTA models?\n\n- What if all tokens (both visual ones and semantic ones) are jointly fed to the transformer instead of utilizing the separate transformer then employing the cross-modal attention, since the transformer contains self and cross attention mechanism itself.", "limitations": "The main concern regarding the limitation is the model complexity.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The transform-based framework is proposed for the ZSL problem.\n\n- The I2D module is proven effective to capture the relevant image regions based on the collected documents.\n\n- The experimental results are considered favorable compared to SOTAs on three benchmark datasets. And the results are highly-interpretable.\n\n- The learned text embedding can be utilized with existing ZSL for further improvement.\n\nWeakness:\n- The discussions regrading the model complexity needs further clarification.\n\n- The novelty of cross modality attention among different tokens is somewhat limited being the main contribution of the whole paper.\n\n- The fairness of utilizing the online document compared to other SOTA ZSL frameworks needs further clarification.\n\nThe more detailed comments I'd like authors to address are summarized in the Questions part.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4FGfMgzAyFV", "reviewer": "Reviewer_uvjr", "summary": "The paper proposes to use multimodal cross-attention to align image representation with word representation by unsupervised training on image-document pairs. The model is later validated on a Zero-Shot Learning task, and an interpretability study is conducted.\n", "questions": "See Cons in the previous section, and please clarify:\n1. how the training proceeds,\n2. how datasets are constructed (especially where the class information comes from),\n3. what is the rationale for processing text with GloVe before passing it to Transformer?", "limitations": "The authors partially analyze the results and provide some details on when the model fails.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The interpretability study demonstrates that the model learned to align modalities together.\n2. The performance over the baselines was improved.\n3. The idea of Image2Document attention seems natural - using the asymmetric attention(image-query) on text-image pairs and aligning them into the same embedding space. \n\nCons:\n1. Motivation: task description is short and appears relatively late in the paper(Zero-Shot Learning is a very vague statement). It is hard to assume what real-life problem the paper is trying to solve, which requires considerable mental effort to judge the model correctly. For example, the article's structure makes it impossible to understand the contribution while reading for the first time (not enough context on them is provided before). Both those issues make it hard to understand the significance of the contribution. \n2. Training: It remains unclear how the training proceeds, e.g., how is the output of the similarity function classified? How are the images aligned to the text? Where does the class information come from? \n3. Model: As I understand, the texts(words) are processed by converting them into GloVe embeddings, then through the MLP layer to be consumed by a Transformer network. This approach added a couple of processing steps, but the decision to do so remained unexplained and is not grounded in the experimental results anywhere in the paper. How much did using GloVe&MLP help? Additionally, in some aspects, the usefulness of this method counters the conclusion the paper reaches, especially the following: \"Documents of unseen classes use the same and additional vocabulary in new sentences causing a distribution shift in their input representation.\" This issue has been previously tackled in other areas of NLP with the subword tokenization (e.g., BPE). Why the authors use static word embeddings in such a situation remains highly mysterious.\n4. Model: Is there any reason not to try to unfreeze the pretrained visual encoder?\t\n5. Model: The proposed model trains a single attention layer while the ablation on scaling the model deeper is lacking but may be significant.\n6. Model: The choice of asymmetric attention(image-query) is not explained nor studied - one can imagine using the coattention or merging the inputs before passing to the self-attention (see, e.g. [1] for comparison). Is there any specific reason to design it that way?\n7. Clarity: The caption in Table 1 is not clear: how were semantic embeddings learned from different sources, and were they represented?\n8. Clarity: The paper presentation is aesthetically pleasing, but the structure makes it hard to understand and stack all the concepts together. Sadly, after reading the paper and having it in front of me, I would not be able to reproduce and reimplement the model's training. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept