Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a method to learn a joint representation of an image with a very generic description of the object present in the image. This representation makes it possible to associate the discriminating elements of the textual description with visual elements of the image. The objective is to improve the zero-shot learning approaches and to classify images containing objects not seen during the learning phase only from the textual description. The proposed model is composed of two parts: a first part combines an image-based transformer and a text-based transformer through a scoring function that allows to compute the similarity between text embedding and image embedding. A second part allows alignment using visual queries and textual keys in a combined text and image transformer.
Experiments are conducted on standard datasets for zero-shot learning. Wikipedia articles were collected to serve as class descriptions. The collected dataset will be made public after the review process.
Experiments show that the proposed model outperforms state-of-the-art models and allows the classification of images containing objects never seen during the learning process based on their textual description only (unseen classes). Different types of semantic embedding are tested (glove, longformer, mpnet, tfidf). An ablation study and examples of qualitative results are presented.

strengths_and_weaknesses: I am not familiar with the field of zero-shot learning but it seems to me that the model combining visual queries and textual keys in a transform model is original. The performances of the model seem to exceed the state of the art results and the model obtains interesting performances with simple embedding (Glove), which puts in perspective the contribution of more complex models like longformer for zero-shot learning problems. The bibliography seems to be very complete, again with citations of simple but efficient models (tfidf). The proposed model seems to be able to be trained on a single A100 GPU in one day, which is accessible. The experimental part is very complete with comparison to the state of the art, testing of different embeddings, ablation study and qualitative analysis.


questions: I am not familiar enough with the field of zero-shot learning to ask further questions.
limitations: NA
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose an attention-based model for zero-shot learning from text documents - knowledge sources such as Wikipedia and bird/flower information websites. The proposed approach relies on two Transformer models: one for processing image patches, and another one for text documents. Each of the Transformers outputs a sequence of features which is fed into two different components of the proposed model. One of them fuses two sequences with an attention mechanism (I2D Attention) and produces an image-text matching score, described as local. The other component (I2D Global) only uses special classification tokens ([CLS]) from both text and image and similarly produces an image-text matching score, which the authors describe as global. The approach optimizes for both local and global matching but only the global one is used later on for classification/generating predictions.

The authors evaluate their model on standard ZSL datasets: AWA2, CUB, and FLO. They separately evaluate the performance of both their learned text features and their entire model that uses the global matching scores. When using their learned text features as input to existing four different ZSL models they observe quite a consistent improvement over alternative GloVe or VGSE features, often by a big margin. Also, using their entire proposed model, with global text-image matching scores used for classification, generally compares well against the other ZSL models.

Additionally, the authors argue that their model is more interpretable, showing examples of attention maps over words in a document or matching image-text element pairs.
strengths_and_weaknesses: **Strengths:**

- (S1) The proposed approach is conceptually simple - it consists mostly of dot-product attention. It doesn’t require many extra hyperparameters.

- (S2) The experimental results. The learned document text features (I2DEmb) quite consistently, both among different datasets and different methods, outperform GloVe and VGSE features: often by a large margin. Additionally, the entire proposed model that uses a very simple similarity score between text and image [CLS] token often outperforms the remaining ZSL methods - even models that use the text features proposed in this work (I2DEmb)

- (S3) Rich ablation studies and a good set of experiments: the experiments show the importance of different components of the proposed model. The authors evaluate their model with different types of textual features but also use their textual features with different models. The set of experiments covers the most important aspects of the model.

- (S4) Attention maps provide some extra interpretability which can be important for understanding predictions, especially when using text documents as a source of information about classes (although see W3)

**Weaknesses:**

- (W1) Possibly suboptimal settings for ZSL models used to compare against. GAZSL was introduced using TF-IDF features, e.g. APN was introduced for attribute features. This paper however compares against them when using either GloVe or VGSE features - which might be suboptimal for those models. A more fair comparison should at least additionally include TF-IDF features as well (especially important for GAZSL).

- (W2) The authors seem to miss a little bit of context regarding the attribute data. The usage of text documents instead of attributes, in general, might have many advantages. However, e.g. f-VAEGAN-D2, APN papers using attribute data report much higher performance on e.g. CUB dataset. The authors however do not mention any attribute-based results which might be misleading for the readers, as if attribute features were not competitive even on these datasets. Additionally, the claims of outperforming SOTA should be formulated more precisely - e.g. with respect to a specific type of data/source of information

- (W3) The qualitative evaluation samples are not indicated as random samples, as opposed to selected. Additionally, for more convincing/insightful results an analysis of failure cases would be needed

- (W4) The authors, without justifying, collect their own text documents instead of already standard existing ones already extracted. There is no motivation for it present in the paper and no discussion or analysis of the differences and the impact of using these vs. standard documents. Additionally, the authors mention performing some filtering of document sections, which could potentially be very important but is not discussed/analyzed further
    - Existing extracted documents (Wikipedia, AllAboutBirds, etc.):
        - FLO, CUB: Mohamed Elhoseiny, Ahmed Elgammal, and Babak Saleh. Write a classifier: Predicting visual classifiers from unstructured text. TPAMI 2016.
        - CUB: Mohamed Elhoseiny, Yizhe Zhu, Han Zhang, and Ahmed Elgammal. Link the head to the” beak”: Zero shot learning from noisy text description at part precision. CVPR 2017

(W5) The “direction” of the attention seems somewhat arbitrary and not analyzed. The authors basically weight the text features (attention values) by the attention compatibility between image patches & text sequence elements. One could do the opposite - use image patch features and weight them by attention maps instead. Or do both directions - like some cross-attention works. No motivation behind this choice is discussed and no comparison of alternative choices is present.
questions: **Questions:**

- (Q1) Table 3 (b): How are different input features (e.g. GloVe) used in the Document Transformer? Is it just an input to a Transformer instead of learning token embeddings (as in the first layer of Transformer)? Are those text features fixed or fine-tuned?

- (Q2) Are any of either the Image or Document Transformer fine-tuned? The Image Transformer seems to be mentioned to be pre-trained but what about Document Transformer? Is it also pre-trained in any way or trained from scratch?

- (Q3) What are the differences between text lengths of different documents? Since the softmax is computed over all classes, and texts lengths can be different, doesn’t it cause significant computational issues? Different lengths would require masking and padding

- (Q4) How does the proposed model compare with other ZSL models with respect to computational requirements - both in training and generating predictions?

- (Q5) How are soft attention maps produced (e.g. Figure 3) if the method only operates on image patches?

- (Q6) Analysis of differences in classification between $s$ and $s_{local}$ predictions (Table 1 in the Appendix): are there any important differences other than just a different level of accuracy? Do they make different types of mistakes? If the two modules are complementary to each other some error analysis from the corresponding scoring function could be insightful

- (Q7) Eq. 4: What is $\hat{x}\_{pa}$? Shouldn’t it be $\hat{f}_{pa}$ instead?
    - Why H in eq. 4 is a function while in L167 it says it’s a vector?

---

**Suggestions:**

- Ablations in the Appendix, Table 1 seem important - I would recommend at least mentioning the main observations in the main paper

**Minor issues:**

- L50: “consistent” —> “consistently”?

- Table 3: “Modal” —> “Model”?

- Figure 1 doesn’t seem to be referred to anywhere in the text

- Hyperlinks should be highlighted somehow - they do not seem to be displayed in any special way - just as normal text - very easy to miss them

- Terminology: Instead of document “embedding” - I’d rather say encoding/representations/features. Typically, “embedding” refers to basically a look-up for discrete objects (e.g. tokens) - the terminology used in this paper with “embeddings” used as a name for any features or representations makes it very easy to confuse with e.g. Transformer’s embeddings (in the very first layer).
limitations: The authors do not discuss the computational cost of training & evaluating their model as opposed to alternative approaches. The qualitative analysis seems to include only selected positive example, with no fail-case analysis (see W3). Additionally, the authors use some very strong, over-exaggerated language - in L45 they claim that their model is “able to develop understanding of different parts of an animal”.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, authors propose to address the zero-shot image classification problem under a more realistic setting, namely each class is descried with one document collected online. To address the issues with online textual documents contain certain noise, and different parts of the document may correspond to the different regions of the images, authors propose a transformer-based ZSL framework that jointly learns to encode images and documents by aligning both modalities in a shared embedding space. The cross-modality attention mechanism is introduced to suppress the noise. And extensive experiments are conducted to validate the proposed modules.
strengths_and_weaknesses: Strengths:
- The transform-based framework is proposed for the ZSL problem.

- The I2D module is proven effective to capture the relevant image regions based on the collected documents.

- The experimental results are considered favorable compared to SOTAs on three benchmark datasets. And the results are highly-interpretable.

- The learned text embedding can be utilized with existing ZSL for further improvement.

Weakness:
- The discussions regrading the model complexity needs further clarification.

- The novelty of cross modality attention among different tokens is somewhat limited being the main contribution of the whole paper.

- The fairness of utilizing the online document compared to other SOTA ZSL frameworks needs further clarification.

The more detailed comments I'd like authors to address are summarized in the Questions part.
questions: - The implementation details regarding the compared methods should be given, since different noisy textual source is utilized under the current experimental settings;

- What about the model complexity, FLOPs of the proposed models compared to other SOTA models?

- What if all tokens (both visual ones and semantic ones) are jointly fed to the transformer instead of utilizing the separate transformer then employing the cross-modal attention, since the transformer contains self and cross attention mechanism itself.
limitations: The main concern regarding the limitation is the model complexity.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The paper proposes to use multimodal cross-attention to align image representation with word representation by unsupervised training on image-document pairs. The model is later validated on a Zero-Shot Learning task, and an interpretability study is conducted.

strengths_and_weaknesses: Pros:
1. The interpretability study demonstrates that the model learned to align modalities together.
2. The performance over the baselines was improved.
3. The idea of Image2Document attention seems natural - using the asymmetric attention(image-query) on text-image pairs and aligning them into the same embedding space. 

Cons:
1. Motivation: task description is short and appears relatively late in the paper(Zero-Shot Learning is a very vague statement). It is hard to assume what real-life problem the paper is trying to solve, which requires considerable mental effort to judge the model correctly. For example, the article's structure makes it impossible to understand the contribution while reading for the first time (not enough context on them is provided before). Both those issues make it hard to understand the significance of the contribution. 
2. Training: It remains unclear how the training proceeds, e.g., how is the output of the similarity function classified? How are the images aligned to the text? Where does the class information come from? 
3. Model: As I understand, the texts(words) are processed by converting them into GloVe embeddings, then through the MLP layer to be consumed by a Transformer network. This approach added a couple of processing steps, but the decision to do so remained unexplained and is not grounded in the experimental results anywhere in the paper. How much did using GloVe&MLP help? Additionally, in some aspects, the usefulness of this method counters the conclusion the paper reaches, especially the following: "Documents of unseen classes use the same and additional vocabulary in new sentences causing a distribution shift in their input representation." This issue has been previously tackled in other areas of NLP with the subword tokenization (e.g., BPE). Why the authors use static word embeddings in such a situation remains highly mysterious.
4. Model: Is there any reason not to try to unfreeze the pretrained visual encoder?	
5. Model: The proposed model trains a single attention layer while the ablation on scaling the model deeper is lacking but may be significant.
6. Model: The choice of asymmetric attention(image-query) is not explained nor studied - one can imagine using the coattention or merging the inputs before passing to the self-attention (see, e.g. [1] for comparison). Is there any specific reason to design it that way?
7. Clarity: The caption in Table 1 is not clear: how were semantic embeddings learned from different sources, and were they represented?
8. Clarity: The paper presentation is aesthetically pleasing, but the structure makes it hard to understand and stack all the concepts together. Sadly, after reading the paper and having it in front of me, I would not be able to reproduce and reimplement the model's training. 

questions: See Cons in the previous section, and please clarify:
1. how the training proceeds,
2. how datasets are constructed (especially where the class information comes from),
3. what is the rationale for processing text with GloVe before passing it to Transformer?
limitations: The authors partially analyze the results and provide some details on when the model fails.
review_title: null

Meta-review: metareview: The authors propose a method to learn a joint representation of an image with a document of the object present in the image. Experiments show that the proposed model outperforms state-of-the-art models. Although the final reviews between reviewers are not aligned, I think authors solved most of their proposed questions.
confidence: Less certain
Decision: Accept