Venue: ICLR 2023 Conference

Reviews:

final_score: rating: null
review_content: Dear authors, reviewers, and AC, 

I thought I would get some discussion started in this paper as I haven't seen any response from the authors and we're in the last portion of the discussion section. Overall, I liked the paper, but I had a few questions and concerns that I was hoping the authors (or other reviewers) could address. 

My primary concern is with the clarity of the method section and less so, the missing comparisons. While I tentatively set my recommendation to borderline accept, I did so with the assumption that the authors could address my clarification requests. Without clarification (from authors or other reviewers), I think the current explanation of the method explanation seems unclear and I fail to see how some of the properties the authors stated are ensured by the method. If I missing something, I would appreciate it if the other reviewers could comment on this. 

I second reviewers o933 and jjXQ questions regarding what the hierarchy is learning: what does it mean for the zebra and patch of grass/sky to be grouped together? are there any (expected/observed) limitations to what hierarchy can provide?  I think the coarse patching makes it unclear what patches belong together or not; eg, in Figure 3, should a patch that's half-plane/half-grass be grouped with the plane or the grass. Furthermore, for longer descriptions, one would expect noun phrases that include both an object and its context (eg, "a zebra grazing") resulting in valid patches that include zerba/patch. However, it's unclear if that was the case. It would be nice if the authors could provide some discussion of this. 


review_title: Initiating paper discussion

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel specialized attention mechanism for CLIP models to induce hierarchical feature discovery in vision and language each modality. 
The authors successfully validate their claims with various experimental results, and they demonstrate the significant performance improvement of CLIP (Contrastive vision-language pretraining) for 11 visual recognition tasks, image-text retrieval task and vision-language reasoning task such as VQAv2 and SNLI-VE.

strength_and_weaknesses: *Strength
- The paper provides enough supports to convince the claims with diverse experimental settings of tasks, pretraining datasets, and downstream datasets.

*Weaknesses
- The authors present (in Appendices) some successful cases of the induced hierarchies. However, it is important to clarify the pros and cons of HiCLIP. I think that it would be more comprehensive to provide unintended results of HiCLIP. 
- It is not clearly explained how the hierarchical features help to learn better representation. As one of suggestions, it might be helpful to comparatively visualize the feature spaces of CLIP, DeCLIP and HiCLIP. 

clarity,_quality,_novelty_and_reproducibility: - Writing is clear, and the organization of manuscript is easy to follow, related work is well-surveyed.
- I think that CLIP models have shown significant progress recently, this work provides novel, plausible, concrete, and reasonable approaches and shows significant performance improvement as expected. I believe that researchers in the ICLR community would be interested in this work.
- The authors provide enough detail information to be reproducible.

summary_of_the_review: See the above.

----------
**Update (Nov 26th)**

Since the authors resolve my major concerns, I've raised my score from 6 to 8.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper makes the observation that classic attention mechanisms do not take advantage in the hierarchical structure implicit within language and vision. 
The propose augmenting the attention operators within transformers with a hierarchy-aware module that progressively discovers "semantic hierarchies" between the tokens. 
The new module is applied within the scope of vision and language pre-training (CLIP) and evaluated primarily on CLIP and DeCLIP (a method that combines CLIP with several other self-supervised tasks) with some comparisons to SLIP (CLIP + SimCLR) and FILIP (A variant of CLIP that computes image-caption similarity on the tokens instead of cosine similarity between global vectors). 
The proposed attention can improve performance of both CLIP and DeCLIP on downstream visual discrimination and vision and language tasks. Furthermore, augmenting CLIP with hierarchical attention can sometimes improve performance more than adding other SSL tasks; HiCLIP > DeCLIP for some vision and language tasks. 
Furthermore, a few qualitative examples are provided that show that the learned hierarchy is meaningful. Several ablations are conducted to evaluate the impact of dataset size (more data helps), backbone parameters (smaller patches resulted in performance improvements), and hierarchy on each domain (hierarchy-attention seems to benefit language encoders more than visual encoders, and using both helps even more). 
strength_and_weaknesses: **Strengths**:
- The idea of taking advantage of the hierarchy implicit within visual and language data is very interesting, and it's nice to see such an observation translated to performance improvements. 
- the details of implementing attention seem to be well-though of. I specifically liked how the authors explicitly listed the hierarchy aggregation priors. 
- I appreciated how the authors went beyond the typical evaluation sets towards some reasoning tasks and analyzed their importance with respect to their modeling contributions. 
- The improvements are impressive, especially in cases where HA outperforms the use of SSL tasks in DeCLIP. 

**Weaknesses:**
- The missing numbers for SLIP and FILIP in Table 1 are problematic. It seems that the numbers match the DeCLIP github repo, however, both models are capable of doing the 0-shot tasks. SLIP only reported numbers for VIT-B/16 which is different from the backbone used in table 1, so one cannot compare numbers across tables. I want to note that I do not think that SLIP is a very important comparison since it's implicit within DeCLIP, however, FILIP is an important comparison. FILIP does a comparison between tokens, and hence, allows the model to implicitly learn features that allow a mapping between both modality-specific tokens. This is a different way to dealing with the structure between tokens: CLIP aggregates all tokens with equal weight, HICLIP learns hierarchical aggregation, FILIP aggregates the loss/similarity through basically though a form of "cross-modal attention." Understanding how it performs would be helpful to understand what aggregation matters for training. Finally, since the paper reports numbers on ViT-B/16 in table 3, evaluating the pre-trained FILIP weights from DeCLIP should be easy as they seem to have the same training setup. 
- The authors do not mention GroupViT by Xu et al. (CVPR 2022). This paper changes the aggregation function through a learned tokenization. As far as I understand, their aggregation is also non-splittable, however, unlike HA-Attention, their grouping block is global rather than local and their computation of affinities are different. GroupViT also uses the same training task and loss as CLIP, so there's a lot of overlap. I think it would be very important for this paper to discuss GroupViT as well as SlotAttention by Locatello et al (NeurIPS 2020). There has been some additional work in that area, so there might be additional work to compare to that I am not aware of, but I think GroupViT is an important comparison and SlotAttention and GroupViT are both important works to discuss as they propose augmentations to attention or transformers with the goal of understanding the composition of the scene to improve learning. 
- The paper makes many claims about how previous approaches in vision overlook hierarchies (first line of second paragraph in intro), however, this is not true. First, convolutional networks were initially proposed to capture hierarchies of features through progressively larger receptive fields. This was shown by many papers and notably discussed by Olah et al (2017). Furthermore, vision datasets are often collected via hierarchies with ImageNet collected using WordNet. Pretrained models also often implicitly learn this hierarchy which can show in confusion patterns as shown by Alsallakh et al (2017). Work on scene graphs and scene compositions defines problems that tries to very explicitly capture different hierarchies. Finally, the paper used to discuss how humans perceive the world hierarchically (Kuzovkin et al) compares brain activations with neural network activations and show they correlate. I do not think any of this requires comparison, but the current tone of the introduction greatly overclaims how hierarchy has been overlooked within vision and multimodal research. 
- While I liked the idea of showing examples of hierarchy in Fig 3, 4, 5 ... I think it would be nice to show more. I am also not sure if there's a way to quantitatively evaluate this via segmentation (similar to GroupViT) or comparing to parse trees for language. This is more of a suggestion than a weakness. 
- I find it very difficult to understand how C is used or computed. Here's my understanding and points I found confusing. I would greatly appreciate some clarification and I think the paper would greatly benefit from an illustrated example of those values (or at least just C) for a small graph (C for the hierarchy of a 4 or 5 word sentence as a toy example would be sufficient I think)
	- For N tokens, you compute affinity values. I am assuming equation 3 can be used for $s_{i, i-1}$ otherwise, it's unclear how one gets those values for equation 4. Furthermore, it's unclear how to compute the similarity for edge tokens; I can imagine using start and end tokens in language, but how do you do that for image tokens? 
	- Equation 5 shows how affinities are updated between layers to ensure that affinities are monotonically non-decreasing, however, that doesn't mean that the order of affinities will not change with time (eg, $a_{0,1} = 0.5 \text{ and }a_{1,2} = 0.4$ at $t_0$ but , $a_{0,1} = 0.55 \text{ and }a_{1,2} = 0.6$ at $t_1$). How would that affect the model? is there something that I am missing that prohibits that from happening? If the ordering changes, does that affect the implied hierarchy. 
	- Most importantly C appears to only be computed for $j>i$ , is this true? if not, I think you need to update the notation to explain that as well as explain how to set the values $a_{i,i}$. Additionally, C appears to be a set of values between 0 and 1 which is multiplied by the QK component of the attention. It does not appear to create a strict hierarchy with definitive edges that only connect parents and children.


 **References:**  
- Xu, J., De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J., & Wang, X. (2022). GroupViT: Semantic Segmentation Emerges from Text Supervision. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (pp. 18134-18144).
- Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., ... & Kipf, T. (2020). Object-centric learning with slot attention. _Advances in Neural Information Processing Systems_, _33_, 11525-11538.
- Olah, et al., "Feature Visualization", Distill, 2017. https://distill.pub/2017/feature-visualization/ 
- Bilal, Alsallakh, et al. "Do convolutional neural networks learn class hierarchy?." _IEEE transactions on visualization and computer graphics_ 24.1 (2017): 152-162.

	  
clarity,_quality,_novelty_and_reproducibility: **Clarity** 
Overall, the paper was easy to read. One very important exception was the second half of Sec 3.1.2: I found it difficult to understand the second portion of HA as noted in the weaknesses. There are also many minor typos and grammatical mistakes which I point out some of them below.

**Reproducibility:**
- What's the batch size? 
- Do you plan on releasing code? While the method is explained well, I don't think it would be easy to implement this or reproducing it without access to code or pseudocode of the method. 

**Minor suggestions/typos:** I noticed several typos while reading the paper. I list some below, but this list is not comprehensive. 
- Abstract: "multimodality content understanding" -> "multimodal content understanding"
- The second sentence in the abstract was very confusing. I encourage the authors to rephrase it. 
- Abstract: "CLIP features can hardly reflect the hierarchy nature" -> "CLIP features hardly reflect the hierarchical nature". Furthermore, I think this claim is unsupported. There are several hierarchies that exist in the word. This paper appears to capture the syntactic trees within language and objects masks within images. However, one could imagine a hierarchy over types similar to WordNet (dogs are a superclass of poodle and bulldog) or graph structure depicting scenes (room has chairs and tables) or part-structure (chair has legs). While the paper shows that adding an explicit attentional mechanism to capture one form of structure improves performance, it does not show that CLIP's feature do not capture any hierarchy. 
- Sec 3, first paragraph, first line: "share a hierarchy nature" -> "share a hierarchical nature" second line: "The lower level hierarchy" -> "the lower level of the hierarchy" 
summary_of_the_review: Overall, I think the paper presents an interesting idea with a good motivation.  The results are compelling and show strong improvements over prior methods. However, I found the explanation of how hierarchical attention operated very confusing, with some unclear notation. Furthermore, there are some missing comparisons. I tentatively set the rating as marginally above acceptance, but I am happy to raise my rating if the points below are adequately addressed. 

Below, I summarize the major weaknesses and clarifications in order of importance that I hope the authors will address:
- The hierarchical attention explanation is a bit confusing, especially how C is computed and used. I would appreciate some clarification in the discussion and I believe the paper would greatly benefit from some additional explanation and a toy example clarifying how C is computed and used.  
- A discussion of GroupViT and SlotAttention would be important to add, as well as a comparison to GroupViT as an alternative approach for aggregating information and a relevant baseline. 
- Table 1 is incomplete for no clear reason. As I noted, there are published numbers that share the setup using a ViT B/16 which was trained by the authors are reported in Table 3. While the comparison against SLIP is not crucial, I think comparing against FILIP and DeFILIP would be very helpful.
- I think the claims about hierarchy being overlooked in computer vision and multimodal representation learning should be adjusted or better contextualized. 

----------------------
**Update (Nov 17th):** I updated my recommendation from 6 to 8 as the authors have responded to my major concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces hierarchy-aware attention in both the visual and textual streams of CLIP. The paper extends the work of 1D Tree transformers and apply it to image patches in 2D.  They show better performance on vision-only and vision-and-language downstream tasks than ClIP style approaches.

strength_and_weaknesses: Strengths
The paper proposes a way to create compact image and text embeddings that consistently outperforms CLIP style approaches for many tasks like classification, image-text retrieval etc. 

Weaknesses
1. The one-turn path taken to compute affinity score between two patches is a pretty rough approximation of shortest path. Is the shortest path really that computationally expensive? I am not sure if shortest path is the right thing to do. Something like connected components (measured by some sort of a threshold) seems like the right thing to do. But concretely, can the author compare these two approaches (the shortest path vs the path with only one turn) and show some empirical result which one does better? 

2. Tree transformer in language seems to be more important than Group Transformer on the vision side, even for visual tasks like image classification (Table 4, Row 2 vs Row 3). For smaller models, the trends seems to be reserved (Row 6 vs Row 8). Can the authors explain this trend a little bit more? 

3. Why not show 11 datasets average accuracy as well in ablations of Table 4. 

4. From the visualizations in Figure 3, 4 and 5, frankly, it's very unclear what the visual heirarchies are capturing. For instance in figure 5, I don't see how patches belonging to zebra and dirt fields are grouped together. 
clarity,_quality,_novelty_and_reproducibility: The experiments are clear and easy to follow, but I thought the approach section (Section 3) could have been restructured a little bit. I think preliminaries (like the tree transformer part) could have been discussed first very explicitly so that it's easy to follow what are the preliminaries and how its extended. I think the figures can also use some more explanation. For instance, in figure 2, what does x mean? What do cyclic arrows represent? 
summary_of_the_review: Overall, the proposed approach shows empirical improvements over CLIP but apart from that, I did not gain any other significant insights for why / how it works. Some of these gains are coming from the tree-transformer-like hierarchy attention in the text encoder, which dilutes the significance of the more interesting contribution in the paper (the hierarchical attention for 2D visual patches). I am on the fence about this paper, and I will update my thoughts based on the author's feedback and other reviews.

Update post-rebuttal: The rebuttal addresses all my concerns - specifically, the connection to connected components, why not to use shortest-path, and explanation of qualitative figures. After reading all the responses to other authors, I am improving the score to 6 (weak accept).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper develops a hierarchy-aware attention mechanism for vision-language pretraining (CLIP-based) models. Motivated by the observation that both vision and language have structural representation, this paper aims to group the similar concepts in a hierarchical manner. It adopts a tree transformer to encode the language structure and designs a novel group transformer to encode image structure. Experimental results show a significant improvement compared to CLIP-based baselines.
strength_and_weaknesses: Pros:
- The idea of exploring vision and language hierarchical representations for better VL pretraining is soundness and reasonable. 
- The method of the group transformer on images is a novel extension of the tree transformer.
-  The experiments results show a large improvement compared with CLIP baselines.

Cons:
- The paper claims to build language structures like unsupervised grammar induction (also in visulization). I'm wondering about the qualitative results on grammar induction (e.g. on COCO dataset like V-PCFG and CLIORA). 
- Computing the hierarchy-aware mask seems complicated and time-consuming. What is the inference time comparing CLIP-based methods? 
- Eq. 9 computes the merging score for two patches by connecting the points along the grid with only one turn. Are there any arguments that use these two paths rather than some other paths like zig-zag? 
- Lack of in-depth analysis on the method though the effectiveness of G-trans and T-trans are well verified in the ablation study. In other words, I would like the authors to discuss the motivation behind the model design and how it works.

Suggestion:
- The paper explores VL structure for better image-text alignment, but ignores region-token alignment (in FILIP) and hierarchical alignment (in CLIORA). I hope the authors can discuss the possibility of extending the current work to the fine-grained cross-modal alignment as mentioned in abstract. This is not a weakness so it won't affect the final score.
clarity,_quality,_novelty_and_reproducibility: Clarity: Overall the paper writing is easy to follow. But I still have some questions regarding some details:
- I cannot find the definition of $a_{i,i+1}$ but only $\hat{a}_{i,i+1}$. 
- Not clear about the visualization process, I cannot find it on the main paper or supplemental materials.

Quality, Novelty: c.f. Pros & Cons

Reproducibility:
Will the authors release the codes? I cannot find any code implementations in the supplemental materials.
summary_of_the_review: I like the idea of exploring the hierarchical representation (in a weakly or unsupervised manner) for VL understanding. This paper presents an interesting design to incorporate VL structural information to representation learning. My overall reviews are positive, and I'm looking forward to more discussions with the authors about my concerns.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents a new attention mechanism for CLIP-like methods which enables hierarchical feature learning. Initial reviews of the paper were mixed; reviewers wanted more discussion about the pros and cons of the method and the role of hierarchy in prior architectures; there were also issues with missing comparisons to prior methods (SLIP and FILIP), insufficient analysis of the Tree Transformer and Group Transformer, and many small details about the attention mechanism and experimental setup. The authors provided an in-depth rebuttal addressing these issues, and after the discussion phase all reviewers felt that their concerns had been largely addressed. In the end all reviewers felt that the paper should be accepted. The authors are strongly encouraged to take the feedback from all reviewers into account when preparing the camera-ready version of the paper.
justification_for_why_not_higher_score: The paper is solid, but feels to me like an incremental improvement on CLIP.
justification_for_why_not_lower_score: All reviewers felt the paper should be accepted.
Decision: Accept: poster