HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention
Authors: Shijie Geng
Jianbo Yuan
Yu Tian
Yuxiao Chen
Yongfeng Zhang
Venue: ICLR 2023 poster (2023)

Abstract: The success of large-scale contrastive vision-language pretraining (CLIP) has benefited both visual recognition and multimodal content understanding. The concise design brings CLIP the advantage in inference efficiency against other vision-language models with heavier cross-attention fusion layers, making it a popular choice for a wide spectrum of downstream tasks. However, CLIP does not explicitly capture the hierarchical nature of high-level and fine-grained semantics conveyed in images and texts, which is arguably critical to vision-language understanding and reasoning. To this end, we equip both the visual and language branches in CLIP with hierarchy-aware attentions, namely Hierarchy-aware CLIP (HiCLIP), to progressively discover semantic hierarchies layer-by-layer from both images and texts in an unsupervised manner. As a result, such hierarchical aggregation significantly improves the cross-modal alignment. To demonstrate the advantages of HiCLIP, we conduct qualitative analysis on its unsupervised hierarchy induction during inference, as well as extensive quantitative experiments on both visual recognition and vision-language downstream tasks.

Published as a conference paper at ICLR 2023
HICLIP: CONTRASTIVE LANGUAGE-IMAGE PRETRAIN- ING WITH HIERARCHY-AWARE ATTENTION
Shijie Geng" Jianbo Yuan', Yu Tian', Yuxiao Chen', Yongfeng Zhang'
Rutgers University *ByteDance Inc. (sg1309, yc984, yongfeng z silanbo.yuan, yutian. httiaa.yyt/bbyeednnc.comm
edu,
ABSTRACT
The success of large-scale contrastive vision-language pretraining (CLIP) has bene- fited both visual recognition and multimodal content understanding. The concise design brings CLIP the advantage in inference efficiency against other vision- language models with heavier cross-attention fusion layers, making it a popular choice for a wide spectrum of downstream tasks. However, CLIP does not explicitly capture the hierarchical nature of high-level and fine-grained semantics conveyed in images and texts, which is arguably critical to vision- laniunl understanding and reasoning. To this end, we equip both the visual and language branches attentions, namely Hierarchy-aware CLIP (HiCLIP),
CLIP with hierarchy-awer to progressively discover semantic hierarchies layer-by-layer from both images and texts in unsupervised manner. As result, such hierarchical aggregation significantly improves the cross-modal alignment. To demonstrate the advantages of HiCLIP, conduct qualitative analysis its unsupervised hierarchy induction during inference, as well as extensive quantitative experiments on both visual recognition and vision-language downstream tasks
INTRODUCTION
In recent years, vision-language pretraining has achieved significant progress pairing with large-scale
multimodal data. Contrastive vision-language pretraining (CLIP) features its generalization ability for zero-shot tasks and robustness to domain shift (Radford al. 2021). Moreover, the spectrum of problems that CLIP can solve range from visual recognition, image-text retrieval, and vision-language reasoning tasks via providing appropriate prompt engineering (Zhou et al. 2022 Gao et al. 2021 Xu al. 2021: Shridhar et al. 2021 Rao et al. 2022 Zhong al., 2022).. Since CLIP is built upon simple cross-modal interactions, it has superior inference efficiency over cross- -ttssttnt based
vision-languaee models (Li et al. 1021 Chen et al. 2020 Li ct al. 2020 TTan & Bansal 2019
Dou et al. 2022). Recent studies including DeCLIP (Li et al. 2022). SLIP (Mu et al. 2021] ) and
FIL.IP Yao et al. 2022) extend CLIPP by either leveraging extra self-supervision training objectives or performing contrastive loss on dense token features.
As humans perceive the world in a hierarchical manner mannerHubell Hubel & Wiesel (9668 Fukushima & Miyake (1982): Kuzovkin et al. (2018), such hierarchical nature in vision and language contents has been explored to assist the design of various model architectures. However, contrastive vision-lanp languug learning methods like CLIP often cannot capture visual and linguistic hierarchies in an explicit way. In the example of right figure in Fig. (a), the pixels first form local patches as image encoder's inputs, and are further merged into semantic groups denoting objects ("traffic lights", "sky"), attributes ("cloudy"), etc. Similarly, syntax hicrarchies can also be observed in natural languages where the caption can be decomposed into constituents shown in Fig, (b). Therefore, we argue that the hierarchical nature (i.e., merging from local to global) in vision and language is critical and can be explicitly utilized for improving CLIP's capability multimodal tasks, especially those requiring high-level understanding and reasoning.
"This work was conducted while interning at ByteDance.
Wee release our implementation of HiCT.IP whtttps: //g thub. joy/ijykiinn jomm/jeyiggugg///eae gung
CLIP

---

Published as a conference paper at ICLR 2023
Visual Hierarchy
Language Hierarchy
truffic
cloudy
"traffic lights sitting under cloudy sky"
lights
stiin
cleady
Figure I: Illustration of hierarchical structures in both (a) vision and (b) language modalities. Based on the affinity scores between adjacent vision patches or word tokens (marked blue boxes), the attention mask C' in hierarchy- aware attention considers both spatial and semantic similarity following the highest valued route (marked in red arrows) between two patches or tokens. The affinity scores evolve layer-by-layer contributing to the different levels of hierarchy granularity.
To this end, we introduce hierarchy-aware attention into CLIP, denoted HiCLIP. Hierarchy-avve aware attention applies an attention mask to the conventional attention mechanism to indicate the tendency merge certain vision patches and language tokens into groups because they are spatially and semantically or visually similar. We generalize hierarchy-aware attention to both images and texts, where its mask is obtained by first calculating the neighbouring affinity score among adjacent patches or tokens, and then propagating the scores across any given patch or token pairs. In addition, we formulate the affinity score with an increasing trend as the layer gets deeper to ensure the merged groups remains the same. In this way, we progressively aggregate hierarchies in layer-by-laye manner for both images and texts.
To be specific, for modeling hierarchies in natural language, we share similar intuitions with the previous studies on unsupervised grammar induction, which aim at unsupervised hierarchical mining (Shi et al. 2019 Drozdov et al. 2019). Tree Transformer (Wang et al. 2019) proposes similar seerarchy-award attention, where
modified attention mechanism which is essentially a special case the attention mask is instantiated constituent prior encourage the merging of semantically similar tokens. Capturing hierarchies in visual contents is more challenging, because spatial correlation should also be considered in addition visual similarities. Therefore, extend the hierarchy- attention to Vision Transformers (Dosovisskiy et al.] 202T) by creating a Group Transformer to progressively aggregate image patches into semantic groups until all patches are merged in one common group which is the original image. Different from the ID scenario in Tree Transformer, the neighboring affinity score computed among the four adjacent neighbors of each image patch (Fig. (a)). Afterwards, we propagate neighboring affinity scores by comparing two special paths connecting image patches on the 2D grid graph.
When we apply such hierarchy-awan attention to both image and text branches in CLIP, we obtain the proposed hierarchy-awarr CLIP (HiCLIP) which features the following advantages: (1) it is able to automatically discover hierarchies in vision and language that matches human intuitions in an unsupervised manner: (2) it generates better multimodal representations especially for vision- language downstream tasks; and (3) it features a comprehensive hierarchy visualization help parse visual and textual hierarchies. To prove the aforementioned advantages, we pretrain HiCLIP and other CLIP-style approaches large-scale image-text pairs, and conduct extensive experiments downstream tasks including visual recognition, image-text retrieval, visual question answering, and visual entailment reasoning. To up, our contributions summarized as follows:
We incorporate hierarchy-aware attention into CLIP (HiCLIP) for both image and text contents, which achieves better performances on vision and lasion- downstream tasks.
To model images in hierarchical manner, we propagate neighboring affinity scores through two
special paths on 2D grid graphs and generalize the hierarchy-aware attention Vision Transformer.
We visualize the evolution of hierarchies in images and texts demonstrate the ability of unsuper- vised hierarchy induction of HiCLIP, which contributes to a better interpretability.
2 RELATED WORK
Vision Languag Models. With the proliferation of multimodal information. exploring the interaction of vision and language information becomes important topic. As result, many vision-
2

---

Published as a conference paper at ICLR 2023
models have flourished recently. Based on how the training objective is designed, they can be divided into three categories. The first category includes early bilinear pooling and attention based multimodal models such as MUTAN (Ben- Younes et al. 2017). BAN (Kim et al. 2018). bottom-up top-down attention (Anderson et al. category is built upon the masked language modeling (MIM) pretraining objective and consists of
2018), and intra-inter modality attention (Gao et al. 2019). The second
approaches such as ViLBERT (Lu et al. 2019). LXMERT (Tan & Bansal 2019), UNITER (Chen
al. 2020) and ALBEF (Li al. 2021). Several recent approaches such as SOHO (Huang et al.
2021) and BEiT WWag et al. 2022 futhur extends MLM to masked visual modeling (MVM) to push
the boundary of multimodal learning In addition, CLIP family models (Radford et al. 2021] Li et al.
2022. Mu et al. 2021: Yao et al. 2022 Chen et al., 2023) that rely on vision-language contrastive
learning and large-scale image-text pairs constitutes the last category.
Unsupervised Grammar Induction. Unsupervised grammar induction is a classic topic in NLP domain aiming at automatically inducing phrase-structure grammars from free-text without parse tree annotations. In the earlier age, probabilistic context free grammars (PCFGs) built upon context- free grammar is widely applied and are solved by inside-outside algorithm (Baker 1979) or CYK algorithm (Sakai 1961). More recently, many deep learning based approaches have been proposed by extending the conventional methods into neural networks such as C-PCFG (Kim et al. (019) and
DIORA (Drozdov et al. 2019), designing special modules to induce tree structures such as PRPN (Shen et 20118) ON-LSTM (Shen et al. 2019), Tree Transformer (Wang et al.) 2019). or assisting unsupervised grammar induction with the help of mods-tty alignment such as VG-NSL (Shi
ct al. 2019), VC-PCFG (Zhao & Titov 2020 and CLIORA (Wan et al. 2022).
Hierarchical Discovery in Vision. Discovering the hierarchy in visual contents is well-established area of vision research. For example, Lin et al. (2017) constructs the hierarchy of feature pyramids in object detection to help the model capture semantics at all scales. More recent work on transformers (Liu et al., [021]] Zhang et al.) (0200 also adopts similar intuition to generate hierarchical feature maps with special local-global attention designs. Meanwhile, another line of research on designing new fine-grained parsing tasks aims to understand hierarchy within scene, such as scene graph parsing (Krishna et al., 2017, Zhang et al. 2019) and action graph parsing (Ji et al. 2020). Recently, more efforts are devoted to automatic hierarchy learning with self-supervised or weakly-supervised objectives (Xie et al. mechanism (Yu et al. 2022 Zheng et al. (202T), and automatically merging semantically similar embeddings (Xu et al.
2021: Dai et al. 202T), designing special inductive bias for the self-attention
2022 Locatello et al.] Our work has the scope of developing special
attention constraint and utilizing contrastive learning objectives for unsupervised hierarchy discovery.
3 HIERARCHY-AWARE ATTENTION IN CLIP
As discussed in Section both vision and language share hierarchical nature in information parsing. The lower level of the hierarchy contains more localized and finer-grained information while the higher levels capture more holistic semantics. These properties are in line with how we humans understand vision (Hubel & Wiesel 1968, Fukushima & Miyake 1982: Kuzovkin et al. 2018) and
language information (Chomsky 1956; Manning 2022).
3.1 A FRAMEWORK OF HIERARCHICAL INFORMATION AGGREGATION
Hierarchy-awee aware attention is based on the attention mechanism in conventional Transformers. Given query Q, key K, and value V. and the scaling factor ddh that maintains the order of magnitude in features where d,, denotes the feature dimension, the general Attention function is defined as:
QK d
Attention K, V) softmax
As illustrated in Fig. we propose to enhance CLIP's vision and language branch with a hierarchy- aware attention. Following the common transformer architecture, given the modality inputs being split into low-level image patches and text tokens, recursively merge patches and tokens that are semantically and spatially similar, and gradually form more image objects and text phrases. First, we define the hierarchy aggregation priors follows:
clusters such as
3

---

Published as a conference paper at ICLR 2023
"traffic lights sittigg under cloudy sky"
Layer 2
Tren-Transformer
Layer
Layer
traffcc
Group Transforree
Layer 0
Layer
Layer
Figure 2: Illustration Hierarchy-aware CLIP (HiCLIP), which employs hierarchy- aaware attention to both vision and language encoders. HiCLIP estimates the affinity scores of neighbouring vision patches or language tokens and progressively groups them into higher-level constituents, encouraging encoders to explicitly capture hierarchical information during training.
Tendency to merge. We recursively merge patches and tokens into higeer-level clusters that are spatially and semantically similar. Intuitively, two nearby image patches have similar appearances, it is natural to merge them
convey the same semantic information.
Non-splittable. Once the patches or tokens are merged, they will never be split at later layers. With this constraint, we aim to enforce that the hierarchical information aggregation will never get degraded, and as result, preserve the complete process hierarchy evolving layer-by-layer.
We then incorporate these hierarchy aggregation priors into attention mask C which serves an extra inductive bias to help the conventional attention mechanism in Transformers to better explore the hierarchical structures adaptive each modality format, i.e., 2D grid images and ID sequence on texts. Therefore, the proposed hierarchy-aware attention can be defined as:
Hierarchy Atteenion = (c softmax
QKT
V.
(2)
Vdn
Note that C' is shared among all heads and progressively updated bottom-up across Transformer layers. We elaborate on the formulations of the hicrarchy-aware mask C for each modality as follows.
3.1.1 HIERARCHY INDUCTION FOR LANGUAGE BRANCH
In this section, we revisit the tree-transformer method from the proposed hierarchy-awarr attention point of view and show how to impose hierarchy aggregation priors on C' with three steps.
Generate neighboring attention score. Neighboring attention score describes the merging tendency of adjacent word tokens. Two learnable key, query matrices W W are adopted to transfer any adjacent word tokens (t,, 1), so that the neighboring attention score 81,44 is defined as their inner-product:
(t,WW) (t++1W7)
81,+1
Here o, is a ceper-parameter to control the scale of the generated scores. Then, for each token softmax function is employed to normalize its merging tendency of two neighbors:
Pt,4+1, Pi, softmax (81,441, 81,4-1)
For neighbor pairs (fi, 1). the neighboring affinity score ăr,+ is defined as the geometric mean of Pi and 1, di,++1 /P1,4+1 Pe+1,5. From graph perspective, it describes the strength of edge 61 1 by comparing it with edges 1,-1, (prs+ V.S. Pi,i-1) and 6,41,42 (Pi+1av... Pi+1,4+2).
4

---

Published as a conference paper at ICLR 2023
Enforcing Non-splittable property. Intuitively, higher neighboring affinity score indicates that two neighbor tokens are more closely bonded. To assure merged tokens will not be splitted, layer-wise affinity scores di.4+1 should increase as the network goes deeper, i.c., ,f 2 d for all I, to help gradually generate hierarchy structure as desired:
d = d + (1 (1-a d d2,41
(5)
Modeling the tendency to merge. To measure the tendency to merge, namely Ct.z for any word token pair (4i, lj), we propagate the affinity scores of neighboring tokens between (4i, (z). Specifically, Ci. derived through the multiplication operation as Ca. IIlk- 0K1 Note that C is symmetric matrix, so we have Ci,5 = Cj.s.
3.1.2 HIERARCHY INDUCTION FOR VISUAL BRANCH
From a graph perspective, it easier to generalize the tierarchy-awand mask C' from the ID sequence in language to the 2D grid in vision domain. First, we also employ query and key matrices W", W to calculate the neighboring attention scores among the four adu-aadjcyecy neighbors of each patch t.y;
() (1i W (4 WK)
(6)
de
where o, is used to control the scale of the generated scores, and the neighborhood b y is limited to the four-adjacency patches of f.j as (f', .j'' E {(i + j.j + n); d, n E {-1, +1}3 A.
Next, for each patch Pt,- the softmax normalizing function is employed to get the merging tendency of ti to its four neighbors:
{P softmax( (((.)) (f'5)) A))
(7)
Similar to the formulation in the language branch, the neighboring affinity score with non-splittable property can be obtained by:
donnee 1-1 (1S)MM + (1 (1- (ed.cee) dasnees
(8)
where (( (PL.)).o's3)) P(r
Lastly, the neighboring affinity scores needs to be propagated to the whole image to acquire (. between any two patches (ti, t2 As the image can be seen 2D-grid graph, natural solution is considering ((1.)1)((22))))) as the length of the shortest path with (1.)) (().)) as the edge weights. To achieve better computational efficiency, we consider two special paths: connecting (ti, (22 along the grid with only one turn. The length of these two paths can be calculated by horizontal and vertical propagation as follows:
C2,C2 II (m.11)11111) II (1g..)))))
II
II ((n.i)) (9)
and C(i (aa = max(C1, C2). Intuitively, Ciny (1 (19.12) finds the maximum merging tendency along two possible paths, either horizontal-first vertical-first. In this way, both spatial and visual similarities have contributed to the attention mask C' for 2D images. Since our approach tries to organize vision patches sharing high similarities into groups, we thus dub it "Group Transformer".
Relation to Recursive Bilateral Filtering, Recursive bilateral filtering Yang 2012) shares similar spirit with our Group Transformer. Given two pixels in an image, recursive bilateral filtering decomposes the calculation of range filtering kernel R into two ID operations horizontal and vertical. For each ID operation, let Iy. 24 denote two pixels on scanline of the 2D image, the ID range filtering kernel RE,; R2 RR+1,642 R
11;-1 RiG+1. where R is computed with a radial basis function kernel: R, = exp (_10,- -1,--,,111)) We can observe the similar property that 20
Tree Transformer possesses. As bilateral filters tend to preserve sharp edges while smoothing other parts of an image, this is in accordance with the goal of our Group Transformer aggregating similar patches into a group. The differences between recursive bilateral filtering and our framework are mainly two perspectives: 1) the basic operation unit is pixel in recursive bilateral filtering while approach uses either patch or word token; 2) radial basis function kernel employed to measure the range similarity in recursive bilateral filtering while we neighboring affinity score instead.
5

---

Published as a conference paper at ICLR 2023
3.2 HIERARCHY-AWAK CLIP
Pretraining with HiCLI. To equip both CLIP branches with the ability of dynamic hierarchy discovery, our Hierarchy-aware CLIP adopts Group Transformer the image encoder and employs Tree Transformer as the text encoder. Let and u denote the image and text feature vectors, the contrastive pretraining objective L can be written as:
exp u,/T) プ exp (v, u//T)
N Σ log
exp (n, V:/T)
L
log
(10)
Σラ= exp v//T//T)
where 7 is a learnable temperature parameter, and N is the total number of image-text pairs.
Unsupervised Hierarchy Induction. During inference, we follow Eq. and Eq. ( to generate all neighboring affinity scores {d and (a(a.)).r..))) from bottom to top layer for the texts and images, respectively. These neighboring affinity scores are then used for hierarchy induction. Intuitively, low affinity score certain layer indicates the two corresponding neighbours remain split within this layer. When repeating such process in a top-down greedy manner, we are able to generate tree hierarchies for texts and similar group hierarchies for images in an unsupervised fashion.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Pretraining Datasets To make fair comparison with the state-of-the-art contrastive vision-languaae pretraining approaches, we adopt the YFCC15M benchmark proposed in (Cui et al. 2022) which builds on subset from YFCC100M (Thomee et al. 2016) consisting of 15M image-text pairs. In addition, we construct a 30M version of pretraining data by including Conceptual Caption 3M (CC3M) (Sharma et al. 2018) and 12M (CC12M) (Changpinyo al. (021). We thus validate model on the two different scales of pretraining data.
Downstream Datasets Following CLIP and DeCLIP, select 11 visual recognition datasets under the zero-shot setting, namely ImageNet (Deng et al,, 2209), CIFAR 10 & CIFAR 100 (Krizheskyy
al. 2009), StanfordCars (Krause et al. (2013). Coltech101 (Fei-Fei et al. 2004), Flowers 102 (Nils-
back & Zisserman 2008), SUN397 (Xiao et al. 2010), DTD (Cimpoi et al. 2014), FGVCAir-
craft (Maji et al. 2013). OxfordPets (Parkhi al. 2012), and FoodToT (Bossard al. 2014). Same
zero-shot classification protocol is applied following Radford et al. (202)) which uses predefined prompts as text inputs. The full list of used prompts provided in the Appendix. Although CLIP and DeCI.IP only evaluates visual recognition, we also provide comprehensive comparisons on vision-language tasks which are more desired in evaluating multimodal models, including: image- text retrieval on MSCOCO Caption (Chen et al., (2015). as well as vision- vision-language reasoning VQAv2 (Antol et al. 2015) and SNLI-VE (Xie et al. 2019).
Implementation Details Two variants of Vision Transformer (Dosovitskiy et al., 2021) are used as the image encoder in our experiments ViT-B/32 and ViT-B/16, while the text encoder is a vanilla Transformer (Vaswani et al. 2017) following CLIP as fair comparison. The embedding size of both image and text features are 512 throughout our paper. To make a fair comparison with CLIP family baselines, hyperparameters including learning rate, warmup steps, weight decay, etc. The input image size is set 224 224, and the input text sequence length is truncated or padded 77. The scaling factor o, and a, of Hierarchy-aware attention are both set to 256 for Group Transformer and Tree Transformer Following CLIP and DeCLIP, the learnable temperature parameter initialized as 0.07.
train all models for 32 epochs under the same of pretraining
4.2 VISUAL RECOGNITION
We first compare HiCLIP with state-of-the-an art CLIP family approaches on YFCC15M benchmark (Cui et al. 2022) containing only ImageNet zero-shot setting. Then we present zero-shot classification results on the other common visual recognition datasets. Both results are presented in Table I
YFCCI5M Benchmark. CLIP Radford et al. (202T). SLIP Mu et al. (202I), and FILIP Yao et al. (2022) all leverage contrastive learning and can be directly compared with HiCLIP. When limiting

---

Published as a conference paper at ICLR 2023
Table 1: Zero-shot classification top-1 accuracy (%) 11 vision datasets against state-of-the-ar
CLIP-style models, including: CIFAR 10/100 (C10/100), Food101 (F101), Flowers (Flow), Caltech
(Cal), Aireraft (Air), and ImageNet (IN). ViT-B/32 used for all compared models.
C100
F101
Pets
Flow.
SUN
Cars
Cal.
Model
4
CLIP
34.6
50.1
35.7
15.5
59.9
1.2
32.8
31.8
SLIP
15M
50.7
25.5
33.3
23.5
49.0
34.7
2.8
14.4
59.9
1.7
34.3
30.0
FILIP
15M
33.5
43.1
24.1
52.7
50.7
24.3
68.8
39.5
37.2
HICLIP
15M
74.1
46.0
51.2
37.8
60.9
50.6
4.5
23.1
67.4
3.6
40.5
41.8 (1 +10.0)
DeCLIP
15M
66.7
38.7
52.5
33.8
60.8
90.3
3.8
27.7
74.7
2.1
43.2
41.3
DeFILIP
15M
70.1
46.8
54.5
40.3
63.7
52.4
4.6
30.2
75.0
3.3
45.0
44.2
HiDeCLIP
15M
65.1
39.4
56.3
43.6
64.1
55.4
5.4
34.0
77.0
4.6
45.9
44.6 +3.3)
CLIP
30M
77.3
48.1
59.1
58.2
52.6
28.0
80.8
48.8
48.4
HICLIP
30M
77.6
56.2
63.9
65.6
62.5
60.7
22.2
38.0
82.4
5.5
$2.9
53.4 (1) +5.0)
DeCLIP HiDeCLIP
30M
84.0
57.1
67.3
71.7
65.0
62.5
23.0
39.5
86.
5.3
55.3
56.1
66.1
44.2
57.4 (1
the comparisons within this scope, HiCLIP achieves the largest performance gain (7.7%) over the other CLIP-style models. Since DeCLIP Li et al. (2022) and DeFILIP Cui et al. (2022) apply multiple single-modal self-supervised tasks in addition to CLIP, we incorporated the same objectives into our hierarchy-awand model for a fair comparison (denoted as HiDeCLIP). By combining the contrastive learning and self-supere supervieee learning loss functions, our HiDeCLIP further improves the zero-shot ImageNet classification performance by 2.7% over DeCLIP, and overall 13.1% % higher than CLIP.
11 Visual Recognition Benchmarks. Note that we included both versions of training data, YFCCI5M (short as 15M) and 30M, in this experiments as discussed in Section 4.1 We observed that the zero-shot performance
Cars and Aireraft datasets very low for all models, because in the YFCC benchmark there are 0.04% and 0% of descriptions contains aircraft and car labels used in these datasets, such "Audi V8 Sedan 1994". HiCLIP achieves significant improvement in average over CLIP on both pretraining datasets, indicating that HiCLIP maintains substantial advantage over CLIP when scaling up the size of training data. Despite the fact that the absolute improvements by incorporating hierarchy-awe self-supervised tasks, it interesting that hierarchy-aware attention compatible with self supe-suee learning (DeHiCLIP) and further achieves performance improvement over DeCLIP.
attentions into CLIP is relatively less significant than adding multiple
4.3 PERFORMANCE COMPARISON ON VISION-LANGUAGG TASKS
In Table 2 we compare different CLyP-style methods on downstream vision-lane lision-language tasks, including image-text retrieval which emphasizes cross- moos- alignment and two vision-language reasoning tasks (VQA and SNLI-VE) which focus more on collaborative multimodal reasoning.
Zero-shot Image-Text Retrieval MSCOCO. On all different algorithms and training datasets, HiCLIP and HiDeCLII improve the retrieval performance by a large margin. is worth noting that without complicated celf-supervised learning objectives. HiCLIP constantly outperforms DeCLIP when merely relying on CLIP's contrastive loss which is different from visual recognition tasks. This finding suggests that the benefits brought by adding self-supervised learning effective within the scope of visual recognition, while our approach fully explores the hierarchical nature of multimodal contents which contributes to a significantly performance boost in vision-lann language tasks.
Fine-tuming on Vision Laiguage Reasoning Tasks. Similar to the results on zero-shot retrieval, we observe consistent performance gains for all visual reasoning tasks and pretraining data, indicating that hierarchy-award attentions are more efficient multimodal learners and is capable of tackling tasks that require content understanding and reasoning capabilities.
4.4 ABLATION STUDY
In this section, we provide additional ablation studies influence factors including the patch granularity of visual encoder and the training data volumes in Table 3 We report the following experimental results including zero-shot accuracy on ImageNet and averaged accuracy all 1
7

---

Published as a conference paper at ICLR 2023
Table 2: Zero-shot image-text retrieval on MSCOCO (5K) dataset and vision-language reasoning on VQAv2 and SNLI-VE with fine-tuning. ViT-B/32 is adopted for all models.
Text Retricval
Image Retrieval
(test-dov)
SNLI (val+test)
Method
Data
RSum
R@
R@
R 10
R 1
R
10
Y/N
Num.
Other
All
Acc.
CLIP
15M
21.4
44.7
56.4
13.7
32.4
42.9
211.5
67.3
30.5
32.7
46.7
62.5
HiCLI
15M
34.2
60.3
70.9
20.6
43.8
55.3
285.1
69.4
33.7
37.2
50.1
67.7
DeCLIP
15M
29.1
55.2
66.6
19.0
41.2
53.1
264.2
70.3
34,9
36,9
50.4
66.1
HiD&CLIP
15M
38.7
64.4
74.8
23.9
48.2
60.1
310.1
72.4
36.1
40,9
53.3
70.5
CLIP
30M
34.8
63.3
73.9
23.3
46.9
58.6
300.8
69.7
34.8
37.8
50.6
66.9
HiCLI
30M
43.9
69.1
78.8
27.0
51.8
62.9
333.5
72.2
36.1
40,9
53.2
70.1
DeCLIP
30M
41.3
68.8
79.3
25.6
50.7
62.3
328.0
71.3
35.4
39.7
52.2
69.0
HiDsCLIP
74.1
82.7
356.2
73.3
42.5
Table 3: Ablations on the patch granularity and pretraining data scale for HiCLIP & HiDeCLIP. Rsum is the summation of the R@I R@5, R@ @ 10 of image-to-Haeeoe sogge--tt-exttt and text-to- tm-ge retrievals.
Method
Fincoder
Data
ImageNet Acc.
II Datasets Avg.
COCO Rsum
VQA Acc.
SNLI Acc.
CLIP
VIT-B/32
15M
32.8
31.8
211.5
46.7
62.5
HICLIP
VIT-B/32
15M
40.5
41.8
285.1
50.1
67.7
DeCLIP
VIT-B/32
15M
43.2
41.3
264.2
50.4
66.1
HiDeCLIP
VIT-B/32
15M
45.9
44.6
310.1
53.3
70.5
CLIP
VIT-B/16
15M
39.3
35.5
245.0
48.8
63.8
HICLIP
VIT-B116
15M
45.2
44.9
313.9
$1.2
69.0
DeCLIP
VIT-B/16
15M
48.2
43.7
290.3
51.5
67.3
HiDeCLIP
VIT-B/16
15M
51.1
48.3
339.6
54.4
71.3
CLIP
VIT-B/32
3UM
48.8
48.4
300.8
50.6
66.9
HICLIP
VIT-B/32
30M
$2.9
53.4
333.5
53.2
70.1
DeCLIP
VIT-B/32
30M
55.3
56.1
328.0
52.2
69.0
HiDeCLIP
VTT-B/32
30M
56.9
57.4
356.2
54.6
72.5
visual recognition dataset, Rsum over recall @ 1, 5, 10 on zero-shot image-text retrieval, as well as accuracy on VQA and SNLI with fine-tuning. addition, we conduct component analysis in Table to show that Group Transformer and Tree Transformer both play important roles in HiCLIP.
On Patch Granularity. We compare all downstream tasks using ViT-B/32 and ViT-B/16 as visual encoders. Since the Group Transformer is based on visual patches and benefits from finer-grained patch segments, we expect HiCLIP and HiDeCLIP achieves consistent performance improvements when directly comparing the same method across different visual encoder variants. When we fix the visual encoder and compare HiCLIP and HiDeCLIP with their corresponding baselines (i.e., CLIP and HiCLIP), HiCLIP and HiDeCLIP constantly outperform CLIP and DeCLIP all tasks with the help of hierarchical -aware attention. It worth noting that HiCLIP alone without complex self-supervised losses outperforms DeCLIP on three out of five tasks, with the exceptions ImageNet and VQA by a small margin. This shows that hierarchical information captured by HiCLIP potentially benefits more to the vision-language contrastive learning paradigm.
On Pretraining Data Scale. As shown in Table 3 for most vision recognition tasks, we observe that the benefits contributed by better modeling strategy saturates when more data is used during pretraining, which is in line with the findings reported by many other works including CLIP Radford al. 202Tp. One possible explanation is that, in order achieve further improvements on visual recognition tasks, more vision-specine training scheme such as self-supervisee learning potentially benefits more, because the ability of multimodal high-level reasoming are not as critical in vision-only tasks. In contrast, by scaling up the pretraining data, the performance improvements achieved on vision-language tasks are more significant and consistent across all methods. Similarly, HiCLIP and HiDeCLIP still enjoys large improvements against CLIP and DeCLIP when the pretraining dataset scales up. In addition, HiCLIP pretrained on 30M data achieves better vision-lang language performances on all three tasks over DeCLIP suggesting a potential better scalability of HiCLIP on vision- reasoning tasks, while DeCLIP features better vision recognition performances.
On Component Analysis. In Table 4 demonstrate that using Group Transformer alone (HiCLIP- Group) for vision modeling yields comparable improvements on visual recognition task (zero- shot ImageNet classification) with using Tree Transformer alone (HrCL.P-Treee. In addition, the
8

---

Published as a conference paper at ICLR 2023
Table 4: Ablations the of Group Transformer (G-Trans) and Tree Transformer (T-Trans) in HICLIP. All models are pretrained on YFCC15M.
Text Retrieval
Image Retrieval
Method
Encoder
G-Trass
T-Trans
ImageNet Acc.
11 Datasets Avg.
COCO Raum
CLIP
VIT-B/32
32.8
31.8
21.4
44.7
56.4
13.7
32.4
12.9
211.5
/
HECLPP
VIT-//22
37.1
38.4
23.7
53.8
63.7
17.2
38.5
50.3
254.2
VITB/32
/
HiCLIP
VIER/32
/
v
40.5
41.8
34.2
60.3
70.9
20.6
43.8
55.3
285,1
CLIP
VIT-B/16
6
39.3
35.5
26.1
52.0
64.6
16.5
37.3
48.5
245.0
HECLIP
VIT-/116
42.3
HICLIP
VITB/166
42.2
37.7
28.5
53.2
65.2
18.1
39.8
256.1
HICLIP
VIT-B/16
45.2
44.9
39.0
65.7
76.4
24.0
48.7
313.9
improvements image-ter retrieval are significant when applying Tree Transformer alone than applying Group Transformer, indicating that language modeling may have more potential impact than visual modeling with regard such vision-language tasks. Moreover, when we activate both Group Transformer and Tree Transformer, substantial performance boosts are obtained against HiCLIP-Grous and HiCLIP-THE Tree, showcasing the synergy between dual hierarchy-aware attentions even under naive cross-modal interactions.
4.5 UNSUPERVISED HIERARCHY INDUCTION WITH PRETRAINED HICLIP MODEL
natural to adopt Tree Transformer because texts are essentially discrete tokens among which certain semantic dependencies are shared. Following the same analogy, since each image is pre- patchified Vision Transformers, we expect the image patches to join semantic groups gradually from bottom layers to top layers for better representation, although it seems be more challenging than the language counterpart. Therefore, in addition to the performance gains achieved over various downstream tasks, also visualize the hierarchies captured in Group and Tree Transformers. As shown in Figure by virtue of explicitly modeling the inputs with hierarchy-awaid attentions during pretraining, our model is able gradually group semantically similar neighbors, showing the ability of performing hierarchical visual and language inductions in an unsupervised manner.
scall
blue
planc
sitting
top
of
lield
small blve plane sitting on top of field"
(+) Hicrarely
Obi Language Hierarchy
Figure 3: An example unsupervised hierarchy induction for a semantically aligned image-text pair.
5 CONCLUSION AND FUTURE WORK
In this work, we equip both the visual and language branches of CLIP with hieracchy-waare attention automatically capture the hierarchies from image-caption pairs. Following the discovered hierar- chical structures, the proposed HiCLIP creates compact image and text embeddings via gradually aggregating spatially and semantically similar patches or tokens into common groups. Supported by extensive experiments on multiple downstream tasks, we show that hicrarchy-aware attention greatly improves the alignment of image and text modalities against several recent CLIP--tyle approaches. Moreover, after pretraining, both branches of HiCLIP can be adopted for unsupervised hierarchy induction by analyzing the generated constituent attention weights. With limited computational resources, we conduct experiments up to 30 million image-text pairs without extensive parameter tuning. As a future direction, we plan to scale up the pretraining dataset as well as the scale of the visual encoder to fully validate the scalability of our approach. In addition, also plan explore the full potential of seraarchy-wan attentions with better multimodal information fusion operations compared with the simple dot product used in CLIP-style models.

---

Published as a conference paper at ICLR 2023
REFERENCES
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6077-6086, 2018.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425 2425-2433, 2015.
James K Baker. Trainable grammars for speech recognition. The Journal of the Acoustical Society of America, 65($1):8132-S133) 1979.
Hedi Ben-You Rémi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal tucker fusion for visual question answering. In Proceedings of the IEEE international conference computer vision, pp. 2612-26200 2017.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-ao 1o-mming discriminative compo- nents with random forests. In European conference on computer vision, pp. 446-46466 461. Springer, 2014.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrenne Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. 2015.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp. 104-120. Springer, 2020.
Yuxiao Chen, Jianbo Yuan, Yu Tian, Shijie Geng, Xinyu Li, Ding Zhou, Dimitris N. Metaxas, and Hongxia Yang. Revisiting multimodal representation in contrastive learning: from patch and token embeddings finite discrete tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Noam Chomsky. Three models for the description of language. IRE Transactions on information theory, 2(3):113-124 1956.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Yufeng Cui, Lichen Zhao, Feng Liang, Yangguang Li, and Jing Shao. Democratizing contrastive language-image pre-training: A clip benchmark of data, model, and supervision. arXiv preprint arXiv:2203.05796, 2022.
Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 1601-1610, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL https: /ttp:://ppeeeeieeeeee net/ffrurm dww.orrmmm....icddd....
10

---

Published as a conference paper at ICLR 2023
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang. Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and- -anguuge transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18166-18176, 2022.
Andrew Drozdov, Patrick Verga, Yi-Pei Chen, Mohit lyyer, and Andrew McCallum. Unsupervised labeled parsing with deep ouside- recursive autoencoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMN.P-IJCNLP)) pp- 1507-1512, 2019.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 178- 178. IEEE, 2004.
Kunihiko Fukushima and Sei Miyake. Neocognitron: self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pp. 267-285. Springer, 1982.
Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
6639-6648, 2019.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint 14.4444,, 2021.
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12976-12985, 2021.
David H Hubel and Torsten Wiesel. Receptive fields and functional architecture monkey striate cortex. The Journal of physiology, 195(1):215-243, 1968.
Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatioltemporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10236-10247, 2020.
Jin-Hwa Kim, Jaehyun Jun, and Byoung- Tak Zhang. Bilinear attention networks. Advances in neural information processing systems, 31, 2018.
Yoon Kim, Chris Dyer, and Alexander Rush. Compound probabilistic context-free grammars for grammar induction. In Proceedingy of the 57th Annual Meeting of the Association for Compu- tational Lingwistics, pp. 2369-2388 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v 1/P 1228. URL httpt:::t////
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine fine-grained categorization. Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ilya Kuzovkin, Raul Vicente, Mathilde Petton, Jean- Shan-Philippe Lachaux, Monica Baciu, Philippe Kahane, Sylvain Rheims, Juan R Vidal, and Jaan Aru. Activations of deep convolutional neural networks are aligned with gamma band activity of human visual cortex. Communications biology, 1(1):1-12, 2018.
11

---

Published as a conference paper at ICLR 2023
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems. 34:9694-999 9700, 2021.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang. Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semanties aligned pre-training for vision-language tasks. European Conference on Computer Vision, pp. 121-137. Springer, 2020.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive language-image pre-training paradigm. In ICLR, 2022.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 10012-10022, 2021.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33: 125-11538, 2020.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL
nww. ffo.mmrrdd
Bkg6RiCqY7
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task task agnostic visiolinguistic representations for vision-an aann-and--ann language tasks. In NeurIPS, 2019.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1rXiv:130..5154, 2013.
Christopher D. Manning. Human Language Understanding & Reasoning, Daedalus, 151 05 2022. ISSN 0011-5266. doi: 10. (0.1112//aed ULL/https:///01..o///00.... daed 01905
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training, arXiv preprint arXiv:2112.127500 2021.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing. pp. 722-729. IEEE, 2008
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: tanguage-guider dense prediction with context-aware prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
18082-18091, 2022.
Itiroo Sakai. Syntax in universal translation. In Proceedings of the International Conference on Machine Translation and Applied Language Analysis, 1961.
12

---

Published as a conference paper at ICLR 2023
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed. image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2566-2565, 2018.
Yikang Shen, Zhouhan Lin, Chin wei Huang, and Aaron Courville. Neural language modeling by jointly learning syntax and lexicon. In International Conference on Learning Representations, 2018. URL htt..httss://.ttm ptttps::/openreveee
net/ rorum? daw.mmt..rog000bbb0000 0W
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree
structures into recurrent neural networks. In International Conference Learning Representations, 2019. UtL.pttpss//ppeerre........
d=B1 16qiR5F7 16
Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. Visually grounded neural syntax acquisition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1842-1861, 2019.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
Hao Tan and Mohit Bansal. Lxmert: Learning cross- modality encoder representations from transform- ers. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5100-5111, 2019.
Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfce100m: The new data in multimedia research. Communications of the ACM, 59(2)664-3, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, 2017.
Bo Wan, Wenjuan Han, Zilong Zheng, and Tinne Tuytelaars. Unsupervised vision-language grammar induction with shared structure modeling. In International Conference on Learning Representa- fions, 2022. URL http:: http:::::///peeevviiiiee
net wote://oommmdddnnnnna.. 0y051BF
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and lasguae tasks. arXiv preprint arXiv:2208. 10442, 2022.
Yau-Shian Wang, Hung- Yi Lee, and Yun-Nung Chen. Tree transformer: Integrating tree structures into self-attention. arXiv preprint
2019.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on IEEE, 2010.
computer vision and pattern recognition, pp.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu, Peize Sun, Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8392-8401, 2021.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding. arXiv preprint arXiv:1901.06706, 2019.
Hu Xu, Gargi Ghosh, Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video- text understanding. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLIP, 2021.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 181 8113-18144 2022.
13

---

Published as a conference paper at ICLR 2023
Qingxiong Yang. Recursive bilateral filtering. In European Conference Computer Vision, pp- 399 413. Springer, 2012.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo
Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image pre-training. In International Conference on Learning Representations, 2022. URL httt:::///ppserrrtt////
net//oorum21G=<pDhcse
Qihang Yu, Huiyu Wang, Siyuan Qiao, Maxwell Collins, Yukun Zhu, Hatwig Adam, Alan Yuille, and Liang-Chiel Chieh Chen. k-means mask transformer. In ECCV, 2022.
Dong Zhang, Hanwang Zhang, Jinhui Tang, Meng Wang. Xiansheng Hua, and Qianru Sun. Feature pyramid transformer. In European conference on computer vision, pp. 323-339. Springer, 2020.
Zhang, Kevin J Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro. Graphical contrastive losses for scene graph parsing. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 11535-11543, 2019.
Yanpeng Zhao and Ivan Titov. Visually grounded compound pefgs. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4369 -369, 2020.
Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. In BMVC, 2021.
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pre- training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16793-16803, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 13009):2337-2348, 2022.
A PROMPTS ENGINEERING FOR ZERO-SHOT VISUAL RECOGNITION
In this work, we follow the 80 prompts proposed in Radford al. 20199) to evaluate zero-shot image
classification on ImageNet dataset. The full list of prompts for ImageNet are presented in Table 5 For other downstream visual recognition datasets,
also domain-specific prompts according Radford et al. (2019). The whole prompts for the 10 downstream datasets can be found in Table 6
B LINEAR PROBE PERFORMANCE
In addition to the zero-shot image classification tasks presented in Table we also perform linear probe on frozen image features to estimate the quality of pretrained image encoders. We follow the settings of CLIP and DeCLIP to train a linear classifier with the L-BFGS optimizer from scikit- learn machine learning library. The linear probe results on downstream visual recognition tasks are presented in Table 7 From the table, we can observe that HiCLIP / HiDeCLIP still outperform CLIP DeCLIP across all varied visual encoder types and pretraining data sizes.
C ADDITIONAL PRETRAINING IMPLEMENTATION DETAILS
Our implementation is based on the open-source PyTorch implementation Following Cui al. (2022), we use Adam W optimizer (Loshchilov & Hutter 2019) with weight decay rate of 0.1 during pretraining. The learning rate is first linearly increased to 0.001 within 2500 warmup steps, and then decayed to 0 following the cosine strategy. For the 15M version pretraining data, set the batch size to 4096 and run all experiments on 32 1000 GPUs. For 30M version pretraining data, we set the batch size to 8192 and run all experiments on 64 A100 GPUs.
httpps: thub. com/ Stte//Snneeeeee Beeinne-GVVTDeee GVt.ee
14

---

Published as a conference paper at ICLR 2023
Table 5:
Full list of prompts to evaluate on ImageNet dataset.
bud photo of (Babel).
pholo of many (label).
sculpture a (libel);
photo of the hard to (labet).
low resolution photo the [lubel].
a rendering of (label).
grathii (label).
photo the [lubel).
cropped photo of the [label].
a tastoo of (label).
the embroidered (label].
photo a hard see {label).
beight photo a [label).
photo clean [label).
of diny [label].
dark photo of (lahel).
drawing a (lhbel).
a photo of my (libel).
the plastic (label).
a photo of the cool {label).
close-up photo of (lahet).
black and white photo of the (label)
painting of the (laiee).
painting of [lahel).
pixelated photo (label).
a sculpture the (label).
bright photo the (label).
cropped photo of (label),
plasric (label).
a photo of the diety (lahel).
jeeg courupted photo of a (label).
a hlury photo of the (lahet).
photo the (lahel).
good photo of the (label].
rendering of the (label).
(label] in video game.
photo one [lubel).
doodle [[ahel).
close-up phato the [lahel).
photo of [lahel).
the origami (label].
the [label] in video game.
sketch of [label].
doodle of the (label).
origani [label).
resolution photoo (label).
the loy [lbbe)).
rendition the (label).
a photo of the clean [lahet).
a photo of large [labet)-
rendition of (lahel).
a photo of nice (lahel).
photo a weird (labbe)..
blurry photo of (label).
cartoon (label).
of (label).
sketch of [lahel).
a embmiideedd [lahel].
pixellted photo of [label].
itap of (lahel).
jpeg comupted photo of the (label).
a good photo a (labol)-
plushic (label).
photo of the nice (label).
photo the small [lahel].
photo the weird (label).
the cartoon (label].
of the [label).
drawing of the (label).
a photo of the large (label).
black and white photo af (label).
the plushic (label).
cark photo a [labet)
itap of' (label).
grattii of the (label).
[lsbel).
ima my [label].
a photo cool [label].
photo a small [label).
tattoo of (lahel).
D MORE VISUALIZATION RESULTS & VISUALIZATION PROCESS
Besides the visualization results illustrated in Figure we also provide eight more cases on unsuper- vised hierarchy induction from Figure to Figure
Moreover, we provide the detailed descriptions of the visualization process for input im- ages in Algorithm [ where we set the list of "break" threshold values (01...., θ12] to (0.35, 0.5, 0.5, 0.6, 0.8, 0.85, 0.9, 0.9, 0.9, 0.9, 0.9, 0.9}. For unsupervised grammar induction, we adopt the same parsing algorithm as in Tree Transformer (Wang et al.) 2019). Based on the visual- ization results, we can conclude that by integrating hierarchy-aware attention into the conventional attention mechanism, our HiCLIP can discover and aggregate spatially and semantically similar visual patches and language tokens in a layer- manner.
However, current unsupervised hierarchy induction of HICLIP (visualization of vision encoder especially) follows a top-down style and relies on the threshold values to decide whether to split two adjacent visual patches and language tokens. For the visual hierarchy, we trivially specify thresholds for different layers (the higher layer also has a higher threshold value). Thus, the threshold list may not be suitable for every image. In addition, changing the threshold values may influence the visual and language induction results. It would be better if the thresholds are adaptive to every input image and sentence. Our future work find better way (e.g.. a deta-depnndent algorithm) to parse the C' matrix for each layer.
E VISUALIZATION OF LEARNED FEATURE SPACE
In Figure 12 and Figure 13
we provide the t-SNE visualization of the learned feature space for CLIP, HiCLIP and DeCLIP pretrained on Y55C-15M and 30M data, respectively. We use the 10 classes of CIEAR-10 dataset to conduct all the visualization experiments.
15

---

Published as a conference paper at ICLR 2023
Table 6: Full list of prompts evaluate on 10 downstream domain-specifie visual recognition datasets.
CIFAR CIEAR
photo of [label].
a bluny pacto of (label).
black and white photo of a (label).
controst photo of (label).
high contrast photo a (label).
a bad phoso of (label).
a good photo of (label).
phoso of a small (label).
photo a big (label),
a photo of the (label).
a blury photo of the (label)
black and white photo of the [ibbel)
low contrast photo of the (label).
high conerset photo of the (lahel).
had photo of the (labet).
photo of the (lahel).
photo of small (label).
photo the big (labet).
Food
photo of [labhl).aann aype of food.
Caltech101
photo a (label).
a painting of (lahel).
a plottic [lubel).
sculpture of (lahel).
sketch a [lubel).
a tuttoo a (lubel).
a toy (labet).
rendition a (lubel).
catbvoidered [Jahel].
a cartion [label).
a (label) in video game.
plushic (lahal).
origami [labet).
of (label).
graftini of [label).
drawing of (label).
doodle of (label).
a photo the (label).
a painting the (label).
the plastic (label).
sculpture the (labe)).
a sketch of the (label)
a tattoo the [label).
the toy (label),
sendition the [lahel).
the exabroidered (lahel).
the cantoon [lahel]..
the (label) is video game.
the plustue (label).
the origani (labet).
art of the (label).
grafti the (lahet)..
drawing of the (label).
doodle the (label).
StansfrrdCrrs
photo (label).
a photo the (label).
photo my (label).
my (label))
photo of my diry (label).
a photo my clion [label).
a photo my new (label).
photo of my (label).
DTD
photo of (label) testure.
a phoso a (lubel) pomten.
a photo of (label) thing.
photo o (label) object.
photo of the (label) texture.
a photo of the (label) postern.
a photo of the (label) thing.
a photo of the (label) object.
PGVCAir<raft
photo of (label), a type of aircraft
photo the (label). type of aircraft.
Flowers102
photo (label), a type of flower.
OxfordPets
photo a (label), atype pet.
SUN39
photo a (labe).
a photo of the (label).
F DETAILED ILLUSTRATION OF THE COMPUTATION OF C
In Figure
we illustrate the detailed computation steps of the attention mask C. For the toy example sentence ' blue cat sitting on bench", we show how the C₁ matrix in each Tree-Transfone layer calculated from neighbourhood affinity scores a',441 through the multiplication operation (i.e.,
Cl IT d.)))) where i {O..... N - 1}, N is the input sequence length.
16

---

Published as a conference paper at ICLR 2023
Table 7: Linear probe performance on downstream datasets.
C10/100 is CIFARI0/100, F101 is
Food101, Flow is Flowers, Cal is Caltech, and Air is Aircraft.
C100
F101
Pets
Flow.
SUN
Ait.
Model
Data
CLIP-VI--B32
86.5
64.7
69.2
64.6
90.6
66.0
79.1
63.0
HICLIP-VT-BB32
15M
89,5
71.1
73.5
70.6
91.9
68.8
30.8
63.9
84,8
27.4
67.2 (1 +4.2)
DxCLIP-VII-B332
1SM
89.2
75.4
72.2
68.8
27.6
68.7
BiDeCLIP-VETEE
15M
88.1
70.7
77.6
73.5
95.6
72.2
36.0
70.1
90.0
32.6
70.8 (1 +2.1)
CLIP B/1
15M
88.5
66.4
77.2
69.3
94,1
69.8
29.0
65.2
82.4
25.5
66.7
HICLIP-VTTB/IC
15M
89.1
70.4
81.0
75.3
95.2
72.5
36.4
68.7
86.4
32.3
70.7 (1 +4.0)
DeCLIR-VIT-B116 HIDECLIP-VIT-B/16
15M
88.7
69.5
83.0
74.3
97.3
74,4
36.9
70.9
89.8
32.2
71.7
15M
88.8
70.3
84.3
80.6
97.1
75.1
42.5
74.3
90.7
38.3
74.2 (1 +2.5)
CLIP-NIT NIT-B/32
30M
92.0
74.7
78.8
80.7
93.7
72.6
55.9
71.4
88.6
29.7
73.8
MICLIP-VIE-B/L
30M
92.8
75.8
80.5
94.4
73.6
59.4
72.2
90.3
33.6
75.4 +1.6)
DxCLIP-VIT-B/32
30M
93.1
76.9
82.0
82.7
96.0
74.9
59.8
74.5
92.6
32.7
76.5
HiDeCLIP
(1 +0.6)
callecin:
vegetiillo
"a collection of fruits and vegetables sitting stove top"
(a) Visual Hierarchy
(b) Language Hierarchy
Figure 4: Visualization results about "a collection of fruits and vegetables sitting on stove top". Our HiCLIP successfully recognizes the green vegetable, fruits like the apples as well as the stove top. In the mean time, the language hierarchy of the input sentence is also created through analyzing the constituent attention weights.
suanding
xahea
othr
"a group of zebra standing each other dirt field"
(a) Visual Hierarchy
Language Hierarchy
Figure 5: Visualization results about "a group of zebra standing next to each other dirt field". Our HiCLIP approach can generate correct parsing tree while aggregating image patches that correspond to the concepts zebra and dirt field into common groups in an unsupervised manner.
17

---

Published as a conference paper at ICLR 2023
during
gume
his
player swings during game"
Visual Hierarchy
by Language Hierarehy
Figure 6: Visualization results about player swings his bat during baseball game". Our HiCLIP approach successfully aggregate the regions of dugout and baseball field, while the batter is not well recognized the visual hierarchy. Meanwhile, the language parsing tree is generally correct by analyzing the constituent attention weights.
DMV7
Oving room
(a) Visual Hierarchy
(b) Language Hierarchy
Figure 7: Visualization results about "a living room with couch and tv near wall", Our HiCLIP successfully generates a correct language hierarchy of the inputs sentence. Moreover, it merges the patches that correspond the couch and tv, as well as the carpet and door regions of the living room.
day
train
possing
flag
train passing by flng clear day"
(i) Visnal Hierarchy
(b) Language Hierarehy
Figure 8: Visualization results about "a train passing by flag on a clear day". Our HiCLIP can successfully recognize the regions of train, flag, and elevated track. For the language hierarchy, HiCLIP can also aggregate these concept words together correctly.
18

---

Published as a conference paper at ICLR 2023
couple
elephants
standing
by somt
trocs
couple elophants standing by some
(i) Visual Hierarchy
Tb) Language Hierarehy
Figure 9: Visualization results about "a couple of elephants standing by some trees". Our HICLIP captures correct language hierarchy of the verb and concept words the imput sentences. Meanwhile, HiCLIP can also aggregate image patches that correspond to the concepts elephant and tree.
building
that
"a stone building that has clock the top"
30 Visual Hicraechy
[b) Language Hierarehy
Figure 10: Visualization results about "a stone building that has a clock on the top". In this case, our HiCLIP can roughly merge the regions of stone building and clock tower. For the language hierarchy, it seems that "a" and "clock" should be aggregated together first and before "has".
people
tennis
"there are people playing a game of tennis"
(a) Visual Hicrarchy
(b) Language Hierarchy
Figure 11: Visualization results about "there are people playing game of tennis". In this case, our HiCLIP didn't it achieve meaningful visual hicrarchy result for the tennis court, even though the patches correspond to the player has been merged during the induction process. For the language hierarchy, it seems that "a" and "game" should be aggregated together first and before "playing".
19

---

Published as a conference paper at ICLR 2023
Algorithm Unsupervised hierarchy induction for input images
Require: All neighboring affinity scores d' (1 for (l,..., N) layers, A list of "break" threshold values (O.,...,ON) 0N) for every layer.
1: I N
Start from the highest layer Store break edges of cach layer
2: Initialize nested list B (Bi,.... Bx}
3: while / do
4: for each edge (i',), ) in the patch graph do if a n.ee
2)) < 0, then
6:
if
then
7:
Append the edge (i.j), (1',5") " to Bi
> Break the edge (i,j), (f,3') 3') in the top layer N
8:
else
edge (i,j), ('))') not in B++1 then
10:
Append the edge (1, (''3')) to B.
Break edge (i, (1",3") in layer I
11:
end if
12:
end if end
13:
end for
15:
|
Move to the next lower layer
16:
end while
17:
Draw visual hicrarchy based on B, then remove redundant edges by finding connected components.
CLIP 15M
HICLIP
DeCLIP 15M
Figure 12: Visualization of the learned feature space via t-SNE on CIFAR-10 dataset. We CLIP, HiCLIP, DeCLIP checkpoints that pretrained on YFCC-15M data.
CLIP HICLIP
DeCLIP 30M
Figure 13: Visualization of learned feature space via t-SNE on CIFAR-10 dataset. We use CLIP, HiCLIP, DeCLIP checkpoints that pretrained on 30M data.
20

---

Published as a conference paper at ICLR 2023
Layer 0:
00.1
0.40
af.2 0.66
023 0.24
<i,, 0.71
a3,5 0.34
5
sitting
bench
6.00
0.28
346
064
0.02
blue
CE
020
0.00
328
Cij
[T 01241
0.08
0.68
0.24
0.72
0.44
0.02
0.94
0.26
026
0.34
Layer 1:
00,1 = 0.43
ai. 0. 99
a3.3 = 0.25
a3,4 0.99
ač.s = 0.54
3
sitting
bench
043
043
0.11
0.11
005
0.25
024
0.13
blue
cat
C. = II
0.22
0.5
0.2
0.20
054
0.28
0.54
024
Layer 2:
45,1
0.79
aF,2 1.00
03.3 0.32
a3 0. 99
ai.s 0.90
sitting
on
bench
0.79
180
blue
cat
2
1,00
079
C. II 06413
22
3
0.25
0.92
032
0.99
0.90
5
0.2
2.38
Figure 14: Detailed illustration of the computation of C. We take short sentence "a blue cat sitting on bench" as toy example. We show real values of af 04 and calculated C matrices in the first three layers of Tree- Transformer. We can clearly see that several words are grouped together when the affinity score between them is high enough and greater than threshold (e.g., 0.8): blue and cat Layer sitting and on in Layer 1; sitting, on, bench in Layer 2.
21


Reviews:
[{"review_id": "qYySvJPjQDt", "reviewer": "Reviewer_WpY2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper makes the observation that classic attention mechanisms do not take advantage in the hierarchical structure implicit within language and vision. \nThe propose augmenting the attention operators within transformers with a hierarchy-aware module that progressively discovers \"semantic hierarchies\" between the tokens. \nThe new module is applied within the scope of vision and language pre-training (CLIP) and evaluated primarily on CLIP and DeCLIP (a method that combines CLIP with several other self-supervised tasks) with some comparisons to SLIP (CLIP + SimCLR) and FILIP (A variant of CLIP that computes image-caption similarity on the tokens instead of cosine similarity between global vectors). \nThe proposed attention can improve performance of both CLIP and DeCLIP on downstream visual discrimination and vision and language tasks. Furthermore, augmenting CLIP with hierarchical attention can sometimes improve performance more than adding other SSL tasks; HiCLIP > DeCLIP for some vision and language tasks. \nFurthermore, a few qualitative examples are provided that show that the learned hierarchy is meaningful. Several ablations are conducted to evaluate the impact of dataset size (more data helps), backbone parameters (smaller patches resulted in performance improvements), and hierarchy on each domain (hierarchy-attention seems to benefit language encoders more than visual encoders, and using both helps even more). ", "summary_of_the_review": "Overall, I think the paper presents an interesting idea with a good motivation.  The results are compelling and show strong improvements over prior methods. However, I found the explanation of how hierarchical attention operated very confusing, with some unclear notation. Furthermore, there are some missing comparisons. I tentatively set the rating as marginally above acceptance, but I am happy to raise my rating if the points below are adequately addressed. \n\nBelow, I summarize the major weaknesses and clarifications in order of importance that I hope the authors will address:\n- The hierarchical attention explanation is a bit confusing, especially how C is computed and used. I would appreciate some clarification in the discussion and I believe the paper would greatly benefit from some additional explanation and a toy example clarifying how C is computed and used.  \n- A discussion of GroupViT and SlotAttention would be important to add, as well as a comparison to GroupViT as an alternative approach for aggregating information and a relevant baseline. \n- Table 1 is incomplete for no clear reason. As I noted, there are published numbers that share the setup using a ViT B/16 which was trained by the authors are reported in Table 3. While the comparison against SLIP is not crucial, I think comparing against FILIP and DeFILIP would be very helpful.\n- I think the claims about hierarchy being overlooked in computer vision and multimodal representation learning should be adjusted or better contextualized. \n\n----------------------\n**Update (Nov 17th):** I updated my recommendation from 6 to 8 as the authors have responded to my major concerns. \n", "strength_and_weaknesses": "**Strengths**:\n- The idea of taking advantage of the hierarchy implicit within visual and language data is very interesting, and it's nice to see such an observation translated to performance improvements. \n- the details of implementing attention seem to be well-though of. I specifically liked how the authors explicitly listed the hierarchy aggregation priors. \n- I appreciated how the authors went beyond the typical evaluation sets towards some reasoning tasks and analyzed their importance with respect to their modeling contributions. \n- The improvements are impressive, especially in cases where HA outperforms the use of SSL tasks in DeCLIP. \n\n**Weaknesses:**\n- The missing numbers for SLIP and FILIP in Table 1 are problematic. It seems that the numbers match the DeCLIP github repo, however, both models are capable of doing the 0-shot tasks. SLIP only reported numbers for VIT-B/16 which is different from the backbone used in table 1, so one cannot compare numbers across tables. I want to note that I do not think that SLIP is a very important comparison since it's implicit within DeCLIP, however, FILIP is an important comparison. FILIP does a comparison between tokens, and hence, allows the model to implicitly learn features that allow a mapping between both modality-specific tokens. This is a different way to dealing with the structure between tokens: CLIP aggregates all tokens with equal weight, HICLIP learns hierarchical aggregation, FILIP aggregates the loss/similarity through basically though a form of \"cross-modal attention.\" Understanding how it performs would be helpful to understand what aggregation matters for training. Finally, since the paper reports numbers on ViT-B/16 in table 3, evaluating the pre-trained FILIP weights from DeCLIP should be easy as they seem to have the same training setup. \n- The authors do not mention GroupViT by Xu et al. (CVPR 2022). This paper changes the aggregation function through a learned tokenization. As far as I understand, their aggregation is also non-splittable, however, unlike HA-Attention, their grouping block is global rather than local and their computation of affinities are different. GroupViT also uses the same training task and loss as CLIP, so there's a lot of overlap. I think it would be very important for this paper to discuss GroupViT as well as SlotAttention by Locatello et al (NeurIPS 2020). There has been some additional work in that area, so there might be additional work to compare to that I am not aware of, but I think GroupViT is an important comparison and SlotAttention and GroupViT are both important works to discuss as they propose augmentations to attention or transformers with the goal of understanding the composition of the scene to improve learning. \n- The paper makes many claims about how previous approaches in vision overlook hierarchies (first line of second paragraph in intro), however, this is not true. First, convolutional networks were initially proposed to capture hierarchies of features through progressively larger receptive fields. This was shown by many papers and notably discussed by Olah et al (2017). Furthermore, vision datasets are often collected via hierarchies with ImageNet collected using WordNet. Pretrained models also often implicitly learn this hierarchy which can show in confusion patterns as shown by Alsallakh et al (2017). Work on scene graphs and scene compositions defines problems that tries to very explicitly capture different hierarchies. Finally, the paper used to discuss how humans perceive the world hierarchically (Kuzovkin et al) compares brain activations with neural network activations and show they correlate. I do not think any of this requires comparison, but the current tone of the introduction greatly overclaims how hierarchy has been overlooked within vision and multimodal research. \n- While I liked the idea of showing examples of hierarchy in Fig 3, 4, 5 ... I think it would be nice to show more. I am also not sure if there's a way to quantitatively evaluate this via segmentation (similar to GroupViT) or comparing to parse trees for language. This is more of a suggestion than a weakness. \n- I find it very difficult to understand how C is used or computed. Here's my understanding and points I found confusing. I would greatly appreciate some clarification and I think the paper would greatly benefit from an illustrated example of those values (or at least just C) for a small graph (C for the hierarchy of a 4 or 5 word sentence as a toy example would be sufficient I think)\n\t- For N tokens, you compute affinity values. I am assuming equation 3 can be used for $s_{i, i-1}$ otherwise, it's unclear how one gets those values for equation 4. Furthermore, it's unclear how to compute the similarity for edge tokens; I can imagine using start and end tokens in language, but how do you do that for image tokens? \n\t- Equation 5 shows how affinities are updated between layers to ensure that affinities are monotonically non-decreasing, however, that doesn't mean that the order of affinities will not change with time (eg, $a_{0,1} = 0.5 \\text{ and }a_{1,2} = 0.4$ at $t_0$ but , $a_{0,1} = 0.55 \\text{ and }a_{1,2} = 0.6$ at $t_1$). How would that affect the model? is there something that I am missing that prohibits that from happening? If the ordering changes, does that affect the implied hierarchy. \n\t- Most importantly C appears to only be computed for $j>i$ , is this true? if not, I think you need to update the notation to explain that as well as explain how to set the values $a_{i,i}$. Additionally, C appears to be a set of values between 0 and 1 which is multiplied by the QK component of the attention. It does not appear to create a strict hierarchy with definitive edges that only connect parents and children.\n\n\n **References:**  \n- Xu, J., De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J., & Wang, X. (2022). GroupViT: Semantic Segmentation Emerges from Text Supervision. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (pp. 18134-18144).\n- Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., ... & Kipf, T. (2020). Object-centric learning with slot attention. _Advances in Neural Information Processing Systems_, _33_, 11525-11538.\n- Olah, et al., \"Feature Visualization\", Distill, 2017. https://distill.pub/2017/feature-visualization/ \n- Bilal, Alsallakh, et al. \"Do convolutional neural networks learn class hierarchy?.\" _IEEE transactions on visualization and computer graphics_ 24.1 (2017): 152-162.\n\n\t  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "k1NrW5xxlfn", "reviewer": "Reviewer_o933", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a novel specialized attention mechanism for CLIP models to induce hierarchical feature discovery in vision and language each modality. \nThe authors successfully validate their claims with various experimental results, and they demonstrate the significant performance improvement of CLIP (Contrastive vision-language pretraining) for 11 visual recognition tasks, image-text retrieval task and vision-language reasoning task such as VQAv2 and SNLI-VE.\n", "summary_of_the_review": "See the above.\n\n----------\n**Update (Nov 26th)**\n\nSince the authors resolve my major concerns, I've raised my score from 6 to 8.", "strength_and_weaknesses": "*Strength\n- The paper provides enough supports to convince the claims with diverse experimental settings of tasks, pretraining datasets, and downstream datasets.\n\n*Weaknesses\n- The authors present (in Appendices) some successful cases of the induced hierarchies. However, it is important to clarify the pros and cons of HiCLIP. I think that it would be more comprehensive to provide unintended results of HiCLIP. \n- It is not clearly explained how the hierarchical features help to learn better representation. As one of suggestions, it might be helpful to comparatively visualize the feature spaces of CLIP, DeCLIP and HiCLIP. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "MgOGubEsjG", "reviewer": "Reviewer_jjXQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces hierarchy-aware attention in both the visual and textual streams of CLIP. The paper extends the work of 1D Tree transformers and apply it to image patches in 2D.  They show better performance on vision-only and vision-and-language downstream tasks than ClIP style approaches.\n", "summary_of_the_review": "Overall, the proposed approach shows empirical improvements over CLIP but apart from that, I did not gain any other significant insights for why / how it works. Some of these gains are coming from the tree-transformer-like hierarchy attention in the text encoder, which dilutes the significance of the more interesting contribution in the paper (the hierarchical attention for 2D visual patches). I am on the fence about this paper, and I will update my thoughts based on the author's feedback and other reviews.\n\nUpdate post-rebuttal: The rebuttal addresses all my concerns - specifically, the connection to connected components, why not to use shortest-path, and explanation of qualitative figures. After reading all the responses to other authors, I am improving the score to 6 (weak accept). ", "strength_and_weaknesses": "Strengths\nThe paper proposes a way to create compact image and text embeddings that consistently outperforms CLIP style approaches for many tasks like classification, image-text retrieval etc. \n\nWeaknesses\n1. The one-turn path taken to compute affinity score between two patches is a pretty rough approximation of shortest path. Is the shortest path really that computationally expensive? I am not sure if shortest path is the right thing to do. Something like connected components (measured by some sort of a threshold) seems like the right thing to do. But concretely, can the author compare these two approaches (the shortest path vs the path with only one turn) and show some empirical result which one does better? \n\n2. Tree transformer in language seems to be more important than Group Transformer on the vision side, even for visual tasks like image classification (Table 4, Row 2 vs Row 3). For smaller models, the trends seems to be reserved (Row 6 vs Row 8). Can the authors explain this trend a little bit more? \n\n3. Why not show 11 datasets average accuracy as well in ablations of Table 4. \n\n4. From the visualizations in Figure 3, 4 and 5, frankly, it's very unclear what the visual heirarchies are capturing. For instance in figure 5, I don't see how patches belonging to zebra and dirt fields are grouped together. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-zcvveQwS8", "reviewer": "Reviewer_79pR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper develops a hierarchy-aware attention mechanism for vision-language pretraining (CLIP-based) models. Motivated by the observation that both vision and language have structural representation, this paper aims to group the similar concepts in a hierarchical manner. It adopts a tree transformer to encode the language structure and designs a novel group transformer to encode image structure. Experimental results show a significant improvement compared to CLIP-based baselines.", "summary_of_the_review": "I like the idea of exploring the hierarchical representation (in a weakly or unsupervised manner) for VL understanding. This paper presents an interesting design to incorporate VL structural information to representation learning. My overall reviews are positive, and I'm looking forward to more discussions with the authors about my concerns.", "strength_and_weaknesses": "Pros:\n- The idea of exploring vision and language hierarchical representations for better VL pretraining is soundness and reasonable. \n- The method of the group transformer on images is a novel extension of the tree transformer.\n-  The experiments results show a large improvement compared with CLIP baselines.\n\nCons:\n- The paper claims to build language structures like unsupervised grammar induction (also in visulization). I'm wondering about the qualitative results on grammar induction (e.g. on COCO dataset like V-PCFG and CLIORA). \n- Computing the hierarchy-aware mask seems complicated and time-consuming. What is the inference time comparing CLIP-based methods? \n- Eq. 9 computes the merging score for two patches by connecting the points along the grid with only one turn. Are there any arguments that use these two paths rather than some other paths like zig-zag? \n- Lack of in-depth analysis on the method though the effectiveness of G-trans and T-trans are well verified in the ablation study. In other words, I would like the authors to discuss the motivation behind the model design and how it works.\n\nSuggestion:\n- The paper explores VL structure for better image-text alignment, but ignores region-token alignment (in FILIP) and hierarchical alignment (in CLIORA). I hope the authors can discuss the possibility of extending the current work to the fine-grained cross-modal alignment as mentioned in abstract. This is not a weakness so it won't affect the final score.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster