RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning
Authors: Xiaojian Ma
Weili Nie
Zhiding Yu
Huaizu Jiang
Chaowei Xiao
Yuke Zhu
Song-Chun Zhu
Anima Anandkumar
Venue: ICLR 2022 Poster (2022)
Keywords: visual relational reasoning
representation learning
systematic generalization

Abstract: Reasoning about visual relationships is central to how humans interpret the visual world. This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of entities, and 3) generalizing to novel object-relation combinations, i.e., systematic generalization. In this work, we use vision transformers (ViTs) as our base model for visual reasoning and make better use of concepts defined as object entities and their relations to improve the reasoning ability of ViTs. Specifically, we introduce a novel concept-feature dictionary to allow flexible image feature retrieval at training time with concept keys. This dictionary enables two new concept-guided auxiliary tasks: 1) a global task for promoting relational reasoning, and 2) a local task for facilitating semantic object-centric correspondence learning. To examine the systematic generalization of visual reasoning models, we introduce systematic splits for the standard HICO and GQA benchmarks. We show the resulting model, Concept-guided Vision Transformer (or RelViT for short) significantly outperforms prior approaches on HICO and GQA by 16% and 13% in the original split, and by 43% and 18% in the systematic split. Our ablation analyses also reveal our model's compatibility with multiple ViT variants and robustness to hyper-parameters.

Published as a conference paper at ICLR 2022
RELVIT: CONCEPT-GUIDED VISION TRANSFORMER FOR VISUAL RELATIONAL REASONING
Xiaojian Ma', Weili Nie?, Zhiding Yu? Huaizu Jiang" Chaoweii Xiao Yuke Zhu
Song-Chun Zhu' Anima Anandkumar
'UCLA 2NVIDIA @Notheesterrn University *ASU UT Austin "Caltech axiacjian.oo mac, sczhužstat. Jucla. (wnic, zhidingy, chaowcix, x,, aanandkumar/ervid... com h. w..iangenrtthhastem..ii yukezêcs www.usteaaec..... edu
ABSTRACT
Reasoning about visual relationships is central to how humans interpret the visual world. This task remains challenging for current deep learning algorithms since it requires addressing three key technical problems jointly: 1) identifying object entities and their properties, 2) inferring semantic relations between pairs of en- tities, and 3) generalizing to novel object-relation combinations, i.e. systematic generalization. In this work, we use vision transformers (ViT') as our base model for visual reasoning and make better use of concepts defined object entities and their relations to improve the reasoning ability of ViTs. Specifically, we intro- duce a novel concept-feature dictionary to allow flexible image feature retrieval at training time with concept keys. This dictionary enables two new concept- guided auxiliary tasks: 1) global task for promoting relational reasoning, and 2) local task for facilitating semantic object-centric correspondence learning. To examine the systematic generalization of visual reasoning models, we intro- duce systematic splits for the standard HICO and GQA benchmarks. We show the resulting model, Concept-guidn -uided Vision Transformer (or ReIViT for short) signif- icantly outperforms prior approaches on HICO and GQA by 16% and 13% in the original split, and by 43% and 18% in the systematic split. Our ablation analyses also reveal our model's 's compatibility with multiple ViT variants and robustness
INTRODUCTION
Deep neural networks have achieved great success in visual recognition. However, their ability for visual relational reasoning, i.e. reasoning with entities and their relationships in visual scene, still falls short of human-level performances, especially in real-world domains. The challenges of common visual relational reasoning tasks, e.g. HICO and GQA benchmarks (Chao al., 2015 Hudson & Manning 2019) are manifested in three aspects: 1) object-centrie learning to identify objects (including humans) well as their visual properties; 2) relational reasoning infer all pairwise relationships between the object entities; and 3) systematic generalization to reason with visual entities and relations novel object-relation combinations and extrapolate longer rea- soning hops (Bahdanau et al. 2018; Hupkes et al. 2020). While existing models have leveraged pre-trained object detectors (Ren et al. 2015 Diang al. 2020) and/or explicit symbolic reasoning methods (Yi al. 2018) to tackle these challenges, they leave ample space for improvement.
Diang
More recently, vision transformers (ViT') have become the new paradigm for visual recognition and have made great strides in a broad range of visual recognition tasks (Dosovitskiy et al.) 2020 Wang et al., 202 Liu et al. 202I). Several properties of ViT make a compelling model choice for visual relational reasoning. First, the self-attention mechanism in VIT offers a strong relational inductive bias, explicitly modeling the relations between input entities. Second, the design of image as patches facilitates the learning of object-centric representations, evidenced by recent works, e.g. DINO and VIT (Caron et al., 2021] Li et al. (02I). that demonstrate ViTs trained with self-supervised learning (SSL.) capture objects in the image without label annotations.

---

Published as a conference paper at ICLR 2022
Concept-feature dictionary D
Ps
loss: Pelog Ps
Pt stop gradient
Global Local projections ht.
Global Local
projections hç
Feature
EMA
student 8s
teacher 8:
<ride, ;iide, horse>
Concept c
In previous
teacher 9t
Iteration...
<ride horse>
crida, horse>
Auxiliary tasks
An overview of our method. Red+Green: the learning pipeline of DINO (Caron et al, 2021) and EsViT (Li al. 202I); Red+Blue: our pipeline. We introduce concept- -feture dictio- nary, where the key is a concept c and its value a queue of image features f with the same concept, to allow flexible feature retrieval with the concept keys. With the proposed dictionary, further develop our concept-guided global and local tasks. EMA denotes the exponential moving average.
Figure I:
To investigate the efficacy of the VIT backbone for visual relational reasoning, in particular on systematic generalization, the ViT backbone with the CNN backbone. Results on GQA show that switching to ViTs in MCAN model (Yu et al.) 2019) brings an immediate 11 % gain in accuracy. However, the performance gap between the original GQA testing split and the new systematic split remains considerable (15% accuracy) for both backbones. It suggests that generic ViTs still need to be improved tackle the reasoning task, especially systematic generalization. Recent works have shown that neural networks can learn representations with better generalization, by learning certain auxiliary tasks of predicting human-specifier concepts (Hill et al. 2020 Koh et al. 2020). A natural question emerges: can we exploit these concepts to improve the reasoning ability of Vil's?
introduce new systematic splits to canonical benchmarks and compare
Our approach is make better of concepts (e.g. the labels in the original training dataset) in the VIT training for better relational reasoning. To this end, we first introduce a novel concept-feature dictionary, where each key is a concept and its value is queue of image features with the same concept, as shown in Figure T It allows dynamic and flexible training-time image feature retrieval during training. Based on this dictionary, we then augment the canonical ViT training pipeline with two auxiliary tasks: First, to facilitate high-level reasoning about relationships, we design a global task that helps cluster images with the same concept together to produce semantically consistent relational representations. Second, to learn better object-centie representations, we develop local task that guides the model discover object-centric semantic correspondence across images (Liu al. 2010). Thanks the plug-and-play feature of our concept-feature dictionary, our auxiliary tasks can be easily incorporated into existing ViT training pipelines without additional input pre- processing. We term the resulting model concept-guide vision transformer (or ReIViT for short).
We evaluate our method on two standard visual relational rea- soning benchmarks: HICO and GQA. Beyond the original in- dependent and identically distributed (I.LD.) training-testing split, we introduce new systematic splits for each dataset to examine the ability of systematic generalization, i.e., recogniz- ing novel object-relation combinations. Our results show that ReI significantly outperforms previous approaches. On HICO, it improves the best baseline by 16%, 43%, and 7% on the original non-systematic and two new systematic splits, respectively, as shown FFigure 2 On GQA, it further closes the gap of overall accuracy between models using visual back- bone feature only and models using additional bounding box features (obtained from pre-trained object detectors) by 13% and 18% on the two splits. We also show that our method is compatible with various ViT variants and robust to hyper- parameters. Finally, our qualitative inspection indicates that ReIViT does improve ViTs learning relational and object-centric representations.
45
Gridher & (2017)
Hou (2020)
Ours
35
25
Original Sys.-easy Sys-hard Figure 2: Results on HICO. Our method improves the best baseline by 16%, 43%, and 7% on the orig- inal non-systematic and two new systematic splits. Sys.: systematic.
2

---

Published as a conference paper at ICLR 2022
Our main contributions are summarized as follows:
We propose ReIViT, by incorporating visual relational concepts to the ViT training with the newly- introduced pnccettgguidd global and local auxiliary tasks, where a concept-feature dictionary proposed to enable dynamic and flexible image feature retrieval with the concept keys.
In extensive experiments on the original non-systematic and new systematic split of the HICO and GQA datasets, we demonstrate the advantages of ReIViT over various strong baselines for visual relational reasoning.
We perform ablation studies on ReIViT to show the contributions of its key components, its com- patibility to various ViT architectures, and its robustness to hyper- parameters. We provide quali- tative results confirm improved learning of relational and object-centric representations.
2
METHODOLOGY
2.1 BACKGROUND
Vision transformers. Here we briefly review the architecture of multi-staged ViTs (Dosovitskiy et al. 2020). Given an image I &&& xc a ViT model g first tokenizes the input into N image tokens (patches) with resolution of (T.T): tokenize (I) = [tı..... ......tll tyl,t, t E =0 .N- HW/T2, where (H, W) and C denotes the original resolution and number of channel of the image, respectively. Then in each stage, patch embedding and a multi-head self attention (MHSA) module is applied to these tokens to produce input for the next stage. The final output of VIT g(I) is a sequence of tokens [21,... zN| that correspond to the aforementioned input tokens. For global prediction tasks, e.g. image categorization, a summary of the input image can be obtained by either inserting an extra [CLS] token to the input sequence of image tokens or performing extra pooling operation over the output tokens (Zhai et al. 2021).
supe-vuee learning with DINO and EVV Our method is developed upon the recently pro- posed self-supervised learning (SSL.) approach self-distillation with no labels (DINO) (Caron et al.) 2027) and its follow-up Es EvViT (Li et al. 202T). As shown in Figure D their main idea is to en- courage the output consistency between a teacher Ir and a student network 9a. parameterized by 0, and 0s, respectively. Given an input image I, both networks map it to a probability distribution (I = (T) and (I h2 (g. (T)) via an extra projection head h(-). The teacher and student network will be updated alternatively by following these two rules: (1) For the student network:
0, + arg ming, LGlobal, where LCtobal - P. P.( log P. (I); (2) For the teacher network, O, is up- dated using an exponential moving average (EMA) on 0,: O, + X0; + (1 - ^)0., where controls the updating momentum. In practice, multiple views of the input image I will be generated via data augmentation and the teacher and student networks will receive different views, preventing the task from being trivialized. Es further extends the image-level loss LGlobal to patch-level by applying dense SSL (Wang et al., 2021c) for learning correspondence between the different views, enhancing the performance on dense prediction. Readers are encouraged to refer to Caron al. (2021) and Li let al. (2021) for more details about these two works.
2.2 RELVIT
ReIViT is a concept-guided VIT that makes hetter of the concepts in the VIT training for the improved relational reasoning. In this section, we first introduce a concept-feature dictionary to store and retrieve image features with their concept keys. We then augment the canonical ViT training pipeline with two auxiliary tasks: a global level task and a local level task, both are concept-guided by resorting to the concept-feature dictionary. Intuitively, the global task help cluster images with the same concept together to produce semantically consistent relational features, while the local task guides the model discover object-centric semantic correspondence across images.
Concept-feature dictionary. We assume the total number of concepts is M, and the set of all concepts = (cI,... (. {(cI, Q.),... (CM, }. where each concept is associated with queue Q; of image fea- tures. During training, each image I may come with multiple concepts, which we denote by G C C. For instance, there may exist several human-object interactions in an image from the HICO dataset, each of which may correspond to a concept. As shown in Figure I whenever a new image-concept pair (I, C1) comes, we uniformly draw concept code from C1 pick up the queue Q from the dictionary that corresponds to c, and then retrieve the image feature f from Q. Meanwhile, we pass the input image to the teacher network St to get the new image feature f' gi (T), and enqueue
concept-feature dictionary is denoted by
=
3

---

Published as a conference paper at ICLR 2022
it Q. Note that if Q is full already, first need to dequeue the oldest image feature from Q- During training. we use the retrieved image feature f for the two auxiliary tasks below, rather than the input image feature f'
Furthermore, the sampling strategy, i.e. how to retrieve image feature f from Q, plays an important role in the overall performance of our method. We consider the following two sampling strategies:
Uniform sampling. Each image feature is drawn with equal probability from the queue, i.e. suppose have N features in the queue, then the probability of each feature being sampled is 1/N. This tactic encourages the diversity of the retrieved image features, benefiting the overall performance. However, some older features in the queue may largely fall behind the current model if the teacher network 9r is updated quickly, eliciting unstable training.
"Most-recent" sampling. The sampling probability mass is allocated based on the freshness of image features, and the most recent feature has the highest chance to be retrieved. Specifically, suppose we have N features in the queue Q (QQ >= N). Then for the i-th newest feature f. we define its weight Wt; N - + 1. Finally, the probability of the i-th newest feature being sampled is wa/ وØ= W/. This tactic ensures we retrieve more up-to-date features and thereby stabilizes the learning. But it may hurt the overall performance due to a lack of feature diversity, as the chance of older features being sampled is small.
Note that the feature queue is empty at the beginning of training. In this case, we simply use the input image feature f' for the auxiliary tasks, and also enqueue it to Q that corresponds to the concept of the input image. As we can show in the next, now our proposed global and local tasks reduce to DINO (Caron et al. (021) and VIT (Li et al. 2021). respectively.
Concept-guided global task. Suppose we have two views {I(), I(2)) of an image I, the main idea of our concept-guided global task is to replace I()) in the DINO loss (Caron et al., (202)) with the image feature f sampled from the concept-feature dictionary, which becomes
(1) where ht and h2 are the projection head of the teacher and student network, respectively, and g, is the student network. Intuitively, minimizing the global loss is equivalent to encouraging the similarity of any two different image features with the same concept. Hence, it can help produce more semantically consistent relational representations, in particular when the concepts stored in the concept-feature dictionary are themselves relational.
LGtabal = -h, log h (Ix))
Similar inter-class representation learning techniques have been explored before (Wang et al. 2017; Caron et al. 2018). However, these approaches require rather complex pre-processing stage, e.g. the images have to be split in terms of the concept before training, making them not directly applicable to existing training pipelines. Rather, with our proposed concept-feature dictionary that dynamically saves & retrieves image features from the running storage, our concept-guided global task becomes a plug-n-play task to existing training pipelines.
Concept-guided local task. As mentioned carlier, our concept-guided local task aims at facilitat- ing object-centric learning, by the means of correspondence learning (Liu et al. [2100] Wang et al.) 2019). Recent studies have unveiled the possibility of learning correspondence with SSL (Wang et al. 2021c; Li et al.] 2021). However, only low-level correspondence between two augmented (e.g. rotated) views of an image can be discovered, while the semantic information of objects is missing. To remedy this, we bring concepts to these methods, endowing them the capability of learning semantic correspondence from images.
Specifically, suppose we have two views (I (22) of an image I, and we also tokenize the image feature into sequence of N local image tokens. Then the output of ViT, we obtain gi (I [2 ..... ..... .2y (1) and g. (1(2) = [z[2] ...................... (,)),,,,)),, where z denotes the local feature. Prior work, such as EsVIT (Li et al. 2021), ). relies on the local features ye (I(2) and ge (T(2)) for the local task. Instead, we replace gr (I() with the image feature f retrieved from the concept-feature dictionary using the concept of the image I. We then split f into multiple local features, i.e. f [2(1) 221 and our concept-guided local loss becomes
LLocal
Σ ke(2))) log (2,2)), jr = arg max CosineDistance ( ( 1=1
(2),, (2)
N
4

---

Published as a conference paper at ICLR 2022
where (..))..())) are the projection heads that map local features to probability distributions Intu- itively, it greedily matches the output between two local regions that have minimal feature distance bootstrapping the object-level semantic correspondence among images with the same concept.
(.)
Overall loss. By combining the global and local tasks, we add an auxiliary task loss Laux to the main loss main (e.g. cross-entropy loss of the reasoning task). The eventual objective is
(3) where a trade-off weight a is added for better flexibility. As we mentioned above, our method will reduce to EsViT, a baseline without concept-guided auxiliary tasks, when we use the current input features 9r (I()) instead of retrieved from our dictionary for computing LGlobal and Local.
L= mainn + aaaux: Laux = Globall + Local:
EXPERIMENTS
We conduct experiments on two challenging visual relational reasoning datasets: HICO (Chao et al., 2013) and GQA (Hudson Manning 2019). Besides their original non-systematic split, we intro- duce the systematic splits of each dataset to evaluate the systematic generalization of our method. First, we compare our method against various strong baselines (Mallya Lazebnik] [016 Girdhar & Ramanan 2017: Hudson & Manning 2018a) on visual relational reasoning, well as state- of-the-art ViTs. Second, we perform the ablation analysis to examine the key components of our method: VIT backbones, concept- feature dictionaries, and auxiliary tasks. Finally, provide qual- itative results to justify the emerging image clustering in terms of concepts and the learned semantic correspondence. Please see more details of all the evaluated tasks in the supplementary material.
3.1 MAIN RESULTS I: HUMAN-OBJECT INTERACTION RECOGNITION
Overview. HICO (Cao al. 2015) features the human-object interaction (HOI) recognition, i.e. predicting all the possible HOI categories of the input image. It contains 600 HOI categories with 117 unique actions and 80 object classes. The training set includes 381 16 images and the test set includes 9658 images. For a fair comparison, we follow the standard practice and mainly focus on those previous methods that do not require extra supervision (Fang et al. 2018) or data (Li eet al. 2020b] 2019b] Jin et al.) 2021). By default, we choose PVTv2-b2 (Wang et al., 202 as the VIT backbone. Regarding the concept-feature dictionary, we use the "most-recent" sampling and a queue length [ of 10. The trade-off weight a in the overall loss is fixed to 0.1. Other syper-parameterd are inherited from DINO (Caron et al. 2021).
Systematic split. The systematic generalization in HICO has been studied before under the name "zoro-shot HOI recognition" (Shen et al.) 2018). The main idea is to remove some HOI categories from the training set while ensuring all the single actions and objects can still be kept in the maining HOI categories. We thereby reuse the systematic splits offered by Hou et al. (2020). There are two splits: systematic-easy, where only the rare HOI classes are removed from the training set; systematic-hard, where only the non-rare HOI classes are removed besides the rare ones. The systematic- split contains much fewer training instances and thereby is more challenging.
Concepts. In HICO, we simply use the 600 HOI categories as our default concepts. We also report results with other concepts (e.g. actions, objects) in the ablation study.
Results. In Table 1. we compare our method with several counterparts. The results read that even a simple model with PVTv2-b2 (25.4M parameters) backbone can outperform many previous meth- ods using ResNet- 101 (44.7M parameters) and lots of bell and whistles. This confirms the great potentials of ViTs visual relation reasoning. By further adding our global and local tasks, we attain 4-6 mAP gain on original and systematic splits. We also observe that EsV (Li et al. 2021 a recently proposed SSL approach, also outperforms the ViT-only baseline. Therefore, we combine their SSL task and our concept-guide guiidd tasks and reach the peak performance (40. 12 mAP) on the original HICO split. Although we do not utilize any extra supervison. ReIViT+Es VIT beats the current state-of-the-art Fang al. (018)) that uses the additional "pose" supervision that does not exist in the HICO dataset. Overall, we raise the results of a fair counterpart (Girdhar & Ramanan 2017) that only exploits extra bbox supervision (which is included in HICO) by 16% (34.6 40.12 on the original split. For systematic splits, we raise the results of Hou et al. (2020) by 43% (26.65
Note that the projection head here is different from DINO's: it works on all output local features. While in DINO, the projection head only works on the summary of input image, i.e. the resulting feature after max-pooling operation or the feature that corresponds [CL.]] in the input.
5

---

Published as a conference paper at ICLR 2022
Systematic-hard Full cls. Unseen cls. Full cls. Unseen cls.
Systematic-easy
Method
Ext. superv.
Backbone
Orig.
Mallya & Lazehnik 2016 *
ResNet-101
33.8
Girchar & Ramanan (011)
bbox pose
ResNet- 101 34.6
Fang et
(018)
ResNet-101
39.9
Hou et al.
2020)
ResNet-101
101 28.57
26.65
11.94
21.76
10.58
ViT-only
PVTv2-b2 35.48
31.06
11.14
19.03
18.85
EsViT (2021)
PVTv2-b2 38.23
35.15
11.53
22.55
21.84
ReIViT (Ours)
PVTv2-b2 39.4 36.99
12.26
22.75
22.66
RelViT + EsVIT (Ours)
PVTv2-b2 40.12 37.21
12.51
23.06
22.89
Table 1: Results on HICO dataset. Some methods requires extra supervision. Bbox/Pose means object-detection or pose estimation is needed. All results are reported in mAP. "Results reported in the original papers; Introduces the systematic split use in the experiments. Full cls.: results reported on all 600 HOI categories; Unseen cls.: results reported on the held-out HOI categories from the training set for testing systematic generalization. Ext. superv.: extra supervision.
+ 37.21) on the systematic-easy split and 7% (21.76 + 23.06) on the systematic-hard split. Fi- nally, although the gap between systematic and non-systematic split still exists (partly due to the much smaller training set for systematic splits), our method makes significant progress, especially on unseen classes (+12.3 mAP on systematic-hane hard). This further demonstrates the advantages of our concept-guider ViT in systematic generalization.
3.2 MAIN RESULTS I: VISUAL QUESTION ANSWERING
Overview. GQA (Hudson & Manning 2019) is recent visual question answering (VQA) dataset with a focus on relational reasoning. Each question is also labeled with semantics. By default, offers both Eeetrained-CNN grid features and region features obtained through Faster R-CNN (Ren eet 2015). For counterparts, we focus on fair comparisons and therefore exclude those that require massive vision-language pretraining (Li et 2019a). Notably, we do not use extra supervision, such as scene graph (Krishna et al. 2016). The ReIViT configuration is almost the same as in HICO, except that we apply the uniform sampling instead as we find it empirically works better. We employ MCAN-Small (Yu et al.] (2019) as our VQA model and the ImageNet I K-pretrained PVTv2- b2 as our vision backbone. The results are reported on the full validation set of GQA.
Systematic split. In GQA, we especially examine the facet of productivity in systematic generaliza- tion, i.e. the ability of reasoning with longer reasoning hops (Hupkes et al., 2020). To this end, we exploit the extra semantics label associated with the GQA questions. We observe that the semantics in GQA break down each question into a sequence of "reasoning hops". where a distribution of reasoning hops can be found in Figure 3. See the supplementary material for examples. Therefore, our idea is to exclude questions with longer reasoning hops from the training set. We end up only keeping questions with less than 5 reasoning hops in the training set. We refer to this setting the systematic split ("Sys.") in the results.
Concepts. Inspired by recent research vision-language pretraining (Tan & Bansal 2019 Li et al., 2019a 2020a). we obtain concepts by parsing the questions into keywords. Specifically, we only keep certain verbs, nouns, and adjectives that contain significant meanings (c.g. actions, objects, characteristics, etc), ending up with 1615 concepts. Due to the space limit, readers may find more details on concept parsing in the supplementary material.
Results. We report the comparison results on the original and systematic splits in Table 2 The main goal of our experiments on GQA mainly to verify if our method can help reduce the gap between models using backbone features only and models using additional bbox features (with dense object detectors). Besides, we also examine to which extent our method can improve systematic generalization. Firstly, we observe that using VIT can largely alleviate the aforementioned gap (51.1 + 56.62), suggesting that the object-centric representations emerge in ViTs. implies the potential of using VITs in eliminating the need for external object detectors. By further adding proposed auxiliary tasks, we achieve the peak performance and raise the results of MCAN- Small w/o bbox features by 13% (51.1 + 57.87) the original split and 18% (30.12 - 35.48) the systematic split. Without any detection pretraining or bbox features, our method achieves very close results to MCAN-Small w/ bbox features on both two splits. The additional results in appendix demonstrate

---

Published as a conference paper at ICLR 2022
Method
Bbox feat.*
Backbone
Orig.
Sys.
Training
BottomUp (2018)
ResNet-101 53.21
0.4
Testing
MAC (201 86
ResNet-101 54,06
0.3
MCAN-Smal 20199
ResNet- 101 58.35 36.21
MCAN-Small (0119
ResNet- 101 51.1 30.12
ViT-only
PVTv2-b2 56.62 31.39
0.1
EvViT (2021)
PVTv2-b2 56.95 31.76
PVTv2-b2 57.87 35.48
0.0
#Resoning hops
Table 2: Results on GQA dataset. All results are reported in
Figure 3:
Histogram of reasoning
overall accuracy. "With extra Faster R-CNN bbox features.
hops over GQA training questions.
that the marginal gap could be further eliminated if we apply larger backbone models (PVTv2-b2 has much fewer parameters than ResNet-101),
3.3
WHY DO OUR AUXILIARY TASKS WORK?
The results the previous section suggest that ReIViT outperforms its counterparts on the challeng- ing relational reasoning tasks. Now we would like provide more insights into method design by answering the question: why do our auxiliary tasks work? To this end, we perform a diverse set of analyses on accessing the impact of key components in ReIViT We also qualitatively justify the intuitions of two auxiliary tasks. These results are reported on the HICO dataset.
3.3.1 ABLATION STUDY
Different VIT architectures. The first aspect examine is the VIT architecture. Besides the default choice on PVTv2-b2, we test our method with the original ViT-S/16 (Dosovitskiy et al. 2020) and another prevalent architecture Swin-Small (Liu al.. 2021). The results are presented in Figure aajand 4b) respectively. These two architectures can both benefit from our auxiliary tasks and we have similar advantages over counterparts in the default setting, which confirms our compatibility to various VIT variants. Full quantitative results are provided in the suppicmentary.
Implementation of concept-feature dictionary. We conduct ablations three facets of concept- feature dictionary: the choice of concepts, sampling tactics, and the size of queue [Q]. In Figure 4c we compare three different concept choices: actions, objects, and HOIs with our best model. The results suggest that all three choices can bring improvement to the baseline without any feature queue (denoted as "None") while using HOIs and objects brings larger improvement. We hypothesize that the proposed auxiliary tasks need more "delicate" concepts to guide the ViT training but actions in HICO tend to be vague and even ambiguous (Shen et al. 20188. Therefore, albeit the consistent ad advantages of our method in terms of different concept selections, careful design of concept space could still be pivotal to achieve the peak performance in relational reasoning.
Furthermore, show the interplay between sampling strategies and queue size |Q] in Figure 4d Interestingly, [Q] has much smaller impact on the performance with the "most-recent sampling than that with the uniform sampling. As mentioned in /Section 2.2, the uniform sampling could help with more diverse features but could also elicit unstable training. Larger [Q] makes the two consequences in the uniform sampling more prominent, thus causing worse performance when stable training is the bottleneck (e.g. in small dataset like HICO). Rather, the "most-recent" sampling can be less sensitive [Q] as only the recent features could be sampled even when [Q] is large.
Auxiliary tasks. In Figure 4e) we show the results of only adding our global or local task in Laux- Surprisingly, just using the local task enough deliver competitive results in the HICO task. This suggests that the real bottleneck in ViTs seems to be better object-centric representations, local task is designed for this. Nonetheless, adding our global task can still elicit clear advantages over other counterparts that do not exploit concept-guided learning.
our
Robustness to a. We sweep the trade-off weight a from 0.02 to 0.5 and report the results in Fig- uue 4f where solid and dash represent our method and the baseline, respectively. It is observed that adding the proposed auxiliary tasks always achieves better performances than the baseline, indicat- ing method is robust to hyper-parameters. Moreover, the improvements become slightly more significant when a is relatively large (but not too large). The peak performances different splits all appear around 0.1, which we thus use our default choice.
7

---

Published as a conference paper at ICLR 2022
ViT-only EMVIT
Object
40
ESVIT
ReVIT
EsVIT-ReMTT
45
Astions
HOls
30
MAP
20
25
10
15
Orignal
Sys.weay
Sya-hhard
Original
Sys.weas
Sya-haard
Orggnal
Sys.waay
Sya-hhard
(a) ReIViT with ViT-S/16.
(b) ReIViT with Swin-Small.
(c) Different choice of concepts.
=
=0
None
Local
46
Original
Syp eaay
Sys-hard
[0]
10
101-50
39
mAP
35
26
35
16
Most-recent
Unifsrm
Original
Sys-eeasy
Sys-hard
(d) Different queue length Q1-
(e) Global local tasks
(f) Robustness a.
Figure 4: Ablation study on HICO. We investigate the impact of VIT architectures, implementation of concept- feature dictionary, auxiliary tasks, and the weight the performance of our method. Sys.: systematic.
ViT-only
ESVIT
RelViT
Figure 5: Visual illustrations of image features against HOI categories on the HICO test via t-SNE. We compare the features obtained by ViT without any auxiliary task (ViT-only), ViT with non-concept auxiliary tasks ViT) and ReIVIT. Besides those clusters that are identified with the other two baselines, clusters that can only be identified with ReIViT are highlighted.
3.3.2 QUALITATIVE INSPECTION
Features vs. concepts. To further justify whether our global task can truly facilitate the learned representation to be more relational, we illustrate the learned output features (max-pout mmx--oooligg all the output tokens) by t-SNE visualization in Figure 5] Different colors correspond different HOI categories, i.e. the concepts we used in ReIVIT. The results read that more clusters can be identified over the image features extracted by ReIViT; therefore our concept-guidee global task can encourage the learned features to be more discriminative regarding the relational concepts than the baselines.
Semantic correspondence. We also probe the learned semantic correspondence that could be en- couraged by our local task, by intuition. We aim at comparing the correspondence extracted from a model trained with different auxiliary tasks, i.e. no auxiliary task, no-concept auxiliary tasks, and auxiliary tasks. We consider two settings: 1) semantic setting (two images that belong to the same concept, e.g. both contains a cat), and 2) non-semantic setting (two views of the same image). Results in Figure 6) highlight the high-similarity matches. Although our method and non-concept baseline (FSV both work fine in the non-semantic setting, our method can identify the semantic correspondence on more objects thanks the concept guidance. Not surprisingly, baseline w/o any auxiliary task (ViT-only) performs the worst as it may suffer from over-smoothness (Gong et al. 2021) and lose all the meaningful spatial information after fine-tuning on the target task.
4 RELATED WORK
Systematic generalization in visual reasoning. Systematic generalization (Hupkes et al. 2020) Bahdanau al. 2018) characterizes which extent a learning agent identify and exploit the
8

---

Published as a conference paper at ICLR 2022
VIT-only
EsViT
ReIViT
Figure 6: Visualization of correspondence. The correspondence is extracted between two views of the same image (upper) and two images that belong to the same concept (lower), using the learned model on HICO. ReIViT can extract correspondence more objects in the two images (seman- tie correspondence) setting. Best viewed on screen.
underlying entities and relations of the training data, and generalize to novel combinations and longer reasoning hops. There has been extensive research inspecting and tackling systematic generalization in visual reasoning (Johnson et al., 2017: Kim & Mnih] 2018: Higgins et al.) 2016]
Kuhnle & Copestake| 2017). However, most of them only focus controlled and synthetic do- mains (Ruis et al.) 2020] Zhang et al. [2019] Barrett et al. 2018 Xie et al.) 2021 Nie et al.) 2200) Jiang al. 2022), while the open-ended real-world domains are largely neglected with very few exceptions (Shen et al. 2018 Teney et al. 2020). In this paper, we tackle systematic generalization in visual relational reasoning with natural images, thereby filling the gap between synthetic and real
domains.
Object-centric and relational representations. Many seminal research reveals that ML models can benefit from object-centric and relational representations with better sample efficiency and gen- eralization (Farhadi & Sadeghi 2013] Ding et al./ 2020; |Mrowaa al.) 2018). However, obtaining such representations from unstructured inputs, i.e. raw images, still remains challenging (Greff jct al. 2019) Locatello et al. 2020; 2019 Yu et al. 2021). Prevalent approaches adopt a latent vari- able model to explicitly infer the foreground srgrround-backgreee background split as well objects & relations (Eslami jet al. [2016] Lin et al., [2200] Zhu & Mumford] (007), while recent findings suggest that they can be emerging property of transformers trained with superviid objectives (Caron et al. 2021 2021). Our goal aligns better with the later regime, as it enables implicit representations and thus could be more versatile and efficient. A key difference is that these methods do not exploit concepts in reasoning benchmarks, making them less capable of learning semantic representations.
Li et al.
Self-supervised learning for ViTs. The recent resurgence on self-supervised learning (SSL) of image models has delivered impressive results on many few-shot or zero-shot tasks (Oord et al, 2018 From a high-level view, these approaches can be categorized into contrastive (He et al., 2020 Chen et al. (020) and non-contrastive (Chen & He (2021). However, not all SSL avenues work well with vision transformers (ViT') and some delicate design may be needed. [Caron et al. (2021) found their non-contrastive learning objective (DINO) manifested better quantitative results and emerging properties on ViTs. [hen et al. (202T) brought similar results on contrastive SSL. Li let al. (2021) further introduced patch-level SSL objective to ViTs for dense prediction tasks. In this paper, instead of proposing a new SSL approach, we make better use of concepts for ViT training, which can be directly applied to the existing SSL objectives for the improved visual reasoning.
5 CONCLUSION
In this paper, our goal is to seek a better inductive bias for visual relational reasoning, especially on real-world data. We found ViTs be a promising candidate due their potential on relational rea- soning, object-centric learning, and systematic generalization. We further presented ReIViT, a sim- ple yet efficient method for exploiting concepts in the visual relational reasoning tasks boost the performances of ViT's. In specific, we proposed two auxiliary tasks in RelViT global task for se- mantically consistent relational representation, and a local task for learning object-centric semantic correspondence. These two tasks are made possible through the use of our proposed concept-feature dictionary. ReIVIT largely outperforms other counterparts on two challenging visual relational rea- soning benchmarks. While mainly focus extending ViTs to visual reasoning using auxiliary tasks, further exploration of combining our work with architectural modification over ViTs to enable better generalization could be new direction for future work.

---

Published as a conference paper at ICLR 2022
REFERENCES
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6077-6086, 2018
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: what required and can it be learned? arXiv preprint arXiv:1811.1288 2018.
David Barrett, Felix Hill, Adam Santoro, Ari Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In International conference machine learning, pp. 511-520.
PMLR, 2018.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. HICO: A benchmark for recog- nizing human-object interactions in images. In Proceedings of the IEEE International Conference on Computer Vision, 2015.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.02055 2021.
David Ding, Felix Hill, Adam Santoro, and Matt Botvinick. Object-based attention for spatio- temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architec- tures. arXiv preprint arXiv:2012.08508, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929 2020.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances in Neural Information Processing Systems, 29:3225-3233 2016.
Hao-Shu Fang, Jinkun Cao, Yu-Wing Tai, and Cewu Lu. Pairwise body-part attention for recogniz- ing human-object interactions. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 51-67,
Ali Farhadi and Mohammad Amin Sadeghi. Phrasal recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12):2854-22 2865, 2013.
Rohit Girdhar and Deva Ramanan. Attentional pooling for action recognition. arXiv preprint arX1v:1711.01467. 2017.
Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Improve vision transformers training by suppressing over-smoothing. arXiv preprint arXiv:2 /2753, 2021.
10

---

Published as a conference paper at ICLR 2022
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pp. 2424-2433. PMLR, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. arXiv preprint arXiv:2009.01719 2020.
Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for human- object interaction detection. In European Conference on Computer Vision, pp. 584-600. Springer, 2020.
Drew A Hudson and Christopher D Manning. Compositional attention networks for machine rea- soning. arXiv preprint arXiv:1803.03060 08360000777 2018a.
Drew A Hudson and Christopher D Manning. Compositional attention networks for machine rea- soning. arXiv preprint 10Xiv:800300066 2018b.
Drew Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Dicuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: how do neural networks generalise? Journal of Artificial Intelligence Research, 67:757-795, 2020.
Huaizu Jiang, Ishan Misra, Marcus Rohrbach, Erik Learned-Miller and Xinlei Chen. In defense of grid features for visual question answering. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10267-10276, 2020.
Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar, Bongard- hoi: Benchmarking few-shot visual reasoning for human-object interactions. In Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Ying Jin, Yinpeng Chen, Lijuan Wang, Jianfeng Wang, Pei Yu, Zicheng Liu, and Jenq-Neng Hwang. Is object detection necessary for human-object interaction recognition? arXiv preprint arXiv:2107.1 13083, 2021.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2900-22190 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International Conference on Machine Learning, pp. 2649-2658, PMLR, 2018.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. International Conference on Machine Learning, pp- 5338-5348 PMLR, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, Michael Bernstein, and Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. 2016.
URL wwt.hhttt://aaattt.. wtt.:/gg///b....o./////o// 07332
Alexander Kuhnle and Ann Copestake. Shapeworld-a new test methodology for multimodal lan- guage understanding. arXiv preprint arXiv: 1744:0/704405/77 2017.

---

Published as a conference paper at ICLR 2022
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self--uuer vision transformers for representation learning. arXiv preprint arXiv:2206.0090 097885, 2021.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint
2019a.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020a.
Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Mingyang Chen, Ze Ma, Shiyi Wang, Hao-Shu Fang, and Cewu Lu. Hake: Human activity knowledge engine. arXiv preprint arXiv:1904.060 2019b.
Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Shiyi Wang, Hao-Shu Fang. Ze Ma, Mingyang Chen, and Cewu Lu. Pastanet: Toward human activity knowledge engine. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 382-391 192-391, 2020b.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv preprint arXiv:2001 022407, 2020.
Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE transactions on pattern analysis and machine intelligence, 33(5)9978-994, 2010.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.
Francesco Locatello, Stefan Bauer, Mario Lucie, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp. 4114-4124. PMLR, 2019.
Francesco Locatello, Dirk Weissenborn. Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. arXiv preprint arXiv:2006. /5055, 2020.
Arun Mallya and Svetlana Lazebnik. Learning models for actions and person-object interactions with transfer to question answering. In European Conference on Computer Vision, pp. 414-428. Springer, 2016.
Damian Mrowca, Chengxu Zhuang, Elias Wang, Nick Haber, Li Fei-Fei, Joshua B Tenenbaum, and Daniel LK Yamins. Flexible neural representation for physics prediction. arXiv preprint 1oX.::/806.08477 2018.
Weili Nie, Zhiding Yu, Lei Mao, Ankit B Patel. Yuke Zhu, and Anima Anandkumar. Bongard- logo: A new benchmark for human-level concept learning and reasoning. Advances in Neural Information Processing Systems, 33, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint /
2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-tim sial-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. A benchmark for systematic generalization in grounded language understanding. arXiv preprint arXiv:2003.05 161, 2020.
12

---

Published as a conference paper at ICLR 2022
Liyue Shen, Serena Yeung, Judy Hoffman, Greg Mori, and Li Fei-Fei. Scaling human-objes inter- action recognition through zero-shot learning. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1568-1576 IEEE, 2018.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. XrXiv preprini arXiv:/908.0740 2019.
Damien Teney, Peng Wang, Jiewei Cao, Lingqiao Liu, Chunhua Shen, and Anton van den Hengel. V-prom: A benchmark for visual reasoning using visual progressive matrices. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12071-12078, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pp. 776-794. Springer, 2020.
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.12122 2021a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. arXiv preprint arXiv:2106. 13797, 2021b.
Xiaolong Wang, Kaiming He, and Abhinav Gupta. Transitive invariance for self-supervised visual representation learning. In Proceedings of the IEEE international conference computer vision, pp. 1329-1338, 2017.
Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pp. 2566-2576, 2576, 2019.
Xinlong Wang. Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for supervipe visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3024-3033, 2021c.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Sirui Xie, Xiaojian Ma, Peiyu Yu, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Halma: Humanlike abstraction learning meets affordance in rapid problem solving. arXiv preprint 1xXi:22022.11344, 2021.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. arXiv preprint
2018
Peiyu Yu, Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song-Chur Zhu. Unsupervised foreground extraction via deep region competition. Advances in Neural Information Processing Systems, 34, 2021.
Zhou Yu, Jun Yu. Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6281-6290, 2019.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv:2106. 106.00000 2021.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chuur Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5317-5327, 2019.
Song-Chun Chun Zhu and David Mumford. A stochastic grammar of images. Now Publishers Inc, 2007.
13

---

used by ReIVIT. We inherit most of the parameters from Published as a conference paper at ICLR 2022
A A FORMAL DESCRIPTION OF LEARNING IN RELVIT
Algorithm formally depicts the execution flow of ReIVIT.
Algorithm ReIVIT: Concept-guided Vision Transformer
Input: set of training images with concepts [(T,, .1). }, an image augmentation function aug(-), mo- mentum update factor A, loss weight o, a concept-feature dictionary D), teacher and student ViT 9r and 9s. parameterized by 0, and 0., respectively.
1: for (I,, C.) in ([I, (.)....) do
2: I 1(() = aL,) aug(L.)
Uniformly draw concept code C.. 4: Retrieve Q from D with c.
5: if Q is not empty then
6:
Sample feature f Q. following some sampling tactics.
7:
Laux = Latobal (f. 9- (1(" Lioca(f, 9. (I.")))
8:
Insert feature 91 (I into Q: if is full, remove the oldest feature.
clse
Laux Latobal (g: ) g. (I end if
(L (I("))
11:
12:
Update 0, with the loss function C = Lmain + aLoux.
13:
Update 0g using EMA: 0, +- 20c (1 2)0.
14:
end for
B EXTRA DETAILS ON RELVIT
B.1 INPUT PIPELINE
We adopt the following data augmentation pipeline for the generating the additional views for our two auxiliary tasks
Randomly crop and resize the image into (224, 224) with scale ratio (0.2, 1.0);
2. Randomly jitter the color of the image on brightness, contrast saturation and hue with probability of (0.4, 0.4, 0.4, 0.1), respectively;
Randomly turn image into gray scale with probability 0.2:
4. Randomly apply Gaussian blur with kernel size 23 and sigma (0.1,2.0) and probability 0.5;
5. Randomly flip the image horizontally.
Notably, we apply a random crop operation to ensure that all the input images for our auxiliary tasks contain the same number of patches.
B.2 HYPER- PAPAME-PRRAMTTES AND BASELINES
Table summarizes the DINO (Caron et al. 2021)
14

---

Published as a conference paper at ICLR 2022
Table 3: Hyperparameters for ReIViT.
Parameter
Value
Optimizer
A with epsilon 1e-1 (HICO) 1e-5 (GQA)
Gradient clipping norm
No grad clipping (HICO) / 0.5 (GQA)
Base learning rate
1.5e (HICO) / 3e -5 (GQA)
Learning rate schedule
0.1 scale with milestones [15, 25] (HICO) [8, 10] (GQA)
Batch size
16 (HICO) 64 (GQA)
Total training epochs
30 (HICO) (GQA)
Temperature t in DINO loss
0.04 for teacher and 0.1 for student, we don't 't use schedule.
Momentum for teacher
0.999
Center m for center features
0.9
Sampling method Queue size [Q]
"most-recent" (HICO) / uniform (GQA)
10
Table 4 summarizes the key details about the loss implementation of different baselines and RelViT
Table 4: Key details about the loss implementation in baselines and RelViT
LClobal
L
Compare student (aug (img) with
DINO X
teacher (aug (img)) ) teacher (aug (img))
ESVIT
X
X
ReIVIT
X
X
queues (concept (img) ] .pop ()
RelVIT ESVIT
X
teacher (aug (img)) ) and
queues [concept (img) ] J. pop ()
C EXTRA DETAILS ON THE DATASETS
C.I HICO
C.1.I ORIGINAL AND SYSTEMATIC SPLITS
Besides the official training/testing split, we adopt the splits for systematic generalization presented in (Hou et al. 2020). It offers splits that follow different strategies select held-out HOI categories. Systematic-easy only select rare HOI categories (with less than 10 training samples), while Systematic-har select non-rare categories instead. Therefore, the training set SSstemati---- hard will contain much fewer samples and become more challenging. Some basic statistics of these training/testing splits can be found in Table 5
Splits
#Training samples #Training HOIs
#Testing samples
#Testing HOIs
Original
38118
600
9658
600
Systematic-easy
37820
480
9658
600
Systematic-hard
9903
480
9658
600
Table 5: Statistics of the splits of HICO dataset.
C.1.2 IMPLEMENTATION OF Liain
In HICO. there might be multiple HOIs for single image. We, therefore, formulate the HOI pre- diction task as a multi-class classification problem. Specifically, the model makes 600 binary clas- sifications and Lmain in equation 3 is binary cross-entropy loss.
15

---

Published as a conference paper at ICLR 2022
C.2 GQA
C.2.1 ORIGINAL AND SYSTEMATIC SPLITS
We introduce systematic split for the GQA dataset that is based reasoning hops. Specifically, we remove those questions that have more than reasoning hops from the training set. Some basic statistics of these training/testing splits can be found in Table 6
Splits
#Taining samples #Testing samples
Original
943000
132062
Systematic 711945
32509
Table 6: Statistics of the splits of GQA dataset.
C.2.2 REASONING HOPS
Since all the questions and answers in the GQA dataset are synthetic, it additionally provides "se- mantics" that characterizes the reasoning procedure that generates the answer from a question and visual scene. These semantics are composed of multiple "reasoning primitives" that act like func- tions: receiving arguments and generating output for the next reasoning step. It is believed they can reflect whether question will require complex multi-hop reasoning pivotal angle of systematic generalization. Therefore we develop our systematic split with Table 7 provides a few examples semantics.
Question
Semantics (Reasoning hops)
relate ([0), pizza, with, (11306744)
filter ([tter([1]] ([1), pizza);
the pizza with the pepper small and covered?
verify (121, covered);
verify size ( small);
and, 4]);
select (11, dreser);
exist(10), ?);
Do you see any tablecloths or dressers?
select tablecleth);
exist ([2], ?);
or 3], ?);
select ( (1), window);
Are there microwave ovens the right of the appliance near the window?
relate [10], appliance, near, s(1297947)
relate ( [[1], microwave, right, 8 (1297947))
exist ([2), ?);
Table 7: Examples of semantics (reasoning hops) in GQA dataset.
C.2.3 CONCEPT PARSING
We obtain the concepts in the GQA dataset by parsing the questions into word tokens. Specifically, we construct a set of concepts that contain nouns, verbs, and adjectives that are with significant meaning. We also manually filter some ambiguous words from this set. The resulting concept set contains 1615 concepts.
We use the python nitk package to process the question. The parsing procedure starts with part-of- speech tagging, where we only keep nouns (NN), verbs (VB) and adjectives (JJ). Then we lemmatize the remaining words to obtain the minimal form of them. Finally, we remove those that do not present in the pre-selected concept list. Additionally, we skip questions with "No" as the answer the question may be unrelated to the image. We provide the statistics of the concepts in GQA in Table 8, The number of associated questions of all the 1615 concepts and a histogram on the number of concepts for each question is presented in Figure aaland 7bl respectively.
16

---

Published as a conference paper at ICLR 2022
Item
Value
Questions without concept
166217 out of 943000 (17.6%)
Concepts without any question Concepts with < 10 questions
14
209
Averaged #questions per concept Median #questions per concept
1030.9
106
Top 20 concepts and their #associated questions
man 52295
animal 44070
furniture 36523
white 33141
front 30779
person 28755
vehicle 26133
woman 25769
bottom 22624
black 22517
device 21962
food 19683
fence 19172
chair 18872
table 18649
hold 18090
shirt 16483
blue 15434
car 14838
Table 8: Statistics of concepts in GQA training set.
00000
250003
50000
200003
350003
30000
300003
30000
420
1000
1200
14000
1000
(a) Histogram of number of questions per concept.
(b) Histogram of number of concepts per question.
Figure 7: Histograms of concepts in GQA training set.
C.2.4 IMPLEMENTATION OF Lmain
GQA is formulated classification problem, i.e. the learner needs select answer from the pre-defined answer set; thus Lmain in equation 3 is a cross-entropy loss.
D FULL QUANTITATIVE RESULTS ON ABLATION STUDIES
We provide the full quantitative results of the ablation studies in Table 9
17

---

Published as a conference paper at ICLR 2022
Method
Orig.
Sys.-easy
Sys.-hard
Method
Orig.
Sys.-easy
Sys.-hard
ViT-only
27.67
26.72
15.23
ViT-only
36.08
31.22
20.88
EsViT
30.83
30.28
17.67
EsViT
38.11
35.22
21.82
ReIViT
31.45
30.88
18.33
ReIViT
39.07
36.27
21.81
ReVi+
33.15
31.09
19.24
ReViT+
39.86
37.17
22.82
(a) ReIVIT with VIT-S/16
(b) ReIVIT with Siwn-Small
Concepts
Orig.
Sys.-easy
Sys.-hard
[Q]
Most-recent Uniform
None
35.48
31.06
19.03
39.71
39.75
Actions
38.8
35.14
22.34
10
40.12
39.93
Objects
39.24
36.59
21.65
30
39.78
39.06
HOIs
40.12
37.31
22.79
50
39.41
38.49
(c) Different choice of concepts
(d) Different queue length [Q]
Orig.
Sys.-easy
Sys.-hard
Tasks
Orig.
Sys.-easy
Sys.-hard
0.02
38.45
36.32
21.85
None 35.48
31.06
19.03
0.05
39.49
36.99
22.62
Global 37.63
34.88
21.07
0.1
40.12
37.31
23.06
Local
39.54
36.74
22.55
0.2
40.04
36.67
22.94
Both 40.12
37.31
22.79
0.5
39.54
35.42
22.79
(e) Global or local tasks
(f) Robustness a
Table 9: Full quantitative results (on full class of HICO) of the ablation studies.
E ADDITIONAL RESULTS
E. RELVIT WITH LARGER BACKBONE MODELS
As we mentioned in Section 3.1 the ViT backbone we use (PVTv2-b2) only has 25.4M parameters, even less than the commonly-used ResNet-101 (44.7M parameters). Therefore, we testify RelViT with larger state-of-the oo-tte-rrtt VIT backbones: PVTv2-b3 (45.2M parameters) and Swin-base (88M
parameters) (Liu et al. 2021) and provide the results on HICO and GQA below:
Table 10: Results with larger ViT models on HICO.
HICO mAP
Fang et al.
2018
ReIViT + ESVIT ReIVIT ESVIT RelVIT + ESViT (Swin-base)
(PVTvv--b)
(PVTv2-b3)
Original Systematic-easy
39.9
40.12
42.61
43.98
37.21
39.92
42.04
Systematic-hard
23.06
25.56
28.36
Table 11: Results with larger ViT models on GQA.
GQA overall accuracy MCAN-Small (w/ bbox)
ReIViT (PVTv2-b2)
ReIViT (PVTv2-b3)
ReIViT
(Swin-base)
original
58.35
57.87 35.48
61.41 36.25
65.54
systematic
36.21
37.51
18

---

Published as a conference paper at ICLR 2022
F EXTRA DISCUSSION ON THE RELATED WORK
F.1 DISCUSSION ON THE MEMORY BANK MECHANISM
Intuitively, the idea of using the concept-feature dictionary the VIT training could be similar to the memory bank mechanism in MoCo (He et al. 2020), where the features are stored in a queue for replaying later. However, the difference is also clear: we have multiple queues that are indexed by concept codes while MoCo only has single queue. Similar use of memory bank can also be found Wu et al. (2018); Tian et al. (2020) but they follow MoCo, and therefore used for providing negative samples when computing the self-supervised contrastive learning loss. Rather, concept- feature dictionary is designed to make better use of the concept supervision via our concept-guided
global and local losses improve the performance visual relational reasoning.
F.2 COMPARISON MAC (HUDSON & MANNING. 20188)
Here we highlight the difference between our cenccet-feattre dictionary and the knowledge base MAC (Hudson & Manning, 2018b): The knowledge base in MAC is used during single VQA reasoning pass (i.e. it will be cleared & initialized with the new image features (visual knowledge) whenever new simage, question on> pair comes), and thus is used in both the training and testing time for the VQA reasoning. However, the concept-feature dictionary in ReIVIT is used to store & retrieve features according the concept of the current input image and help compute our local and global losses that encourage learning better representations. Therefore, training time only as these two losses won't be computed & optimized during testing.
in the
19


Reviews:
[{"review_id": "nYGyK_IPM-o", "reviewer": "Reviewer_fbWL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The proposed method is generic and easily incorporated to existing methods in a plug-and-play manner. \n- Empirical results on HICO and GQA show consistent improvements in performance over baselines. The proposed method also achieves some good results in systematic generalization settings.\n\n\nCons:\n\n(1) Proposed method\n- The scientific novelty of this paper seems limited. The overall structure of the proposed model is almost similar to the \"self-distillation with no labels\" approach (Caron et al.,2021) and EsViT (Li et al.,2021). The contribution is mostly about the use of their concept-feature dictionary as a visual knowledge base to support retrieving information on the fly. The use of the dictionary is more similar to the concept of knowledge base used by [1] rather than memory bank mechanisms as the proposed method only reads out information from the dictionary without refining the information stored in the dictionary.\n- The authors should explain more how they pick the suitable image feature f from a set of a queue Q_i. It is an important step in the proposed model, and the description for their two sampling tactics is not detailed enough.\n- Although the reported numbers look great, it is hard to explain why the visual concepts given by ViT are good for visual relation reasoning. ViT does not actually rely on actual “visual objects”, for example, RoI poolings in Faster R-CNN. I think it is more about the more powerful representation learning pretrained on a large number of images which benefits all tasks regardless. \n\n(2) Experiments\n- My main concern about experimental results is that the improvement of the proposed model may come from the Visual Transformer backbone. It would be fair if the authors also evaluate their concept-guided approach with the ResNet-101 backbone or actual visual objects given by Faster R-CNN.\n- Even using the state-of-the-art ViT backbone, the performance of the proposed model is still worse than state-of-the-art methods on the GQA dataset in both original and systematic settings, and only has sightly improvement on the HICO dataset in the original setting.\n- CLOSURE probably is a better benchmark for evaluating systematic generalization.\n\n[1] Hudson DA, Manning CD. Compositional attention networks for machine reasoning. ICLR 2019.\n\n[2] Bahdanau, Dzmitry, et al. \"Closure: Assessing systematic generalization of clevr models.\" arXiv preprint arXiv:1912.05783 (2019).", "summary_of_the_paper": "The paper proposes a new concept-guided approach for visual relational reasoning. Particularly, it uses a concept-feature dictionary to store and retrieve the feature of an image by the corresponding visual concepts. Then, the concept-guided features are integrated into the training process with the global and local auxiliary tasks. Experimental results on HICO and GQA in both original and systematic settings evidenced the effectiveness of the proposed method against existing works.", "summary_of_the_review": "The paper appears to be limited in novelty and contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hh-YdYbsWdW", "reviewer": "Reviewer_RiEe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n+ Non-trivial motivation to marry the ViT and contrastive learning to advance visual relation reasoning.\n+ Impressive improvements upon the HICO and GQA in the common setting and generalization setting.\n+ Detailed ablations of the proposed method.\n+ Discussion and results about the ViT and visual relation reasoning are inspiring.\n\nCons:\n- Novelty: Model and auxiliary follow the existing works like DINO, EsViT, Liu et.al. 2010, Wang et al., 2021c, etc. Though the feature dictionary is somewhat new, it is still essential to fully discuss the difference between it and the works including memory modules like Wu et al., 2018, Tian et al., 2020, even they are not based on Transformer.\n- Following the above point, this work proposes a useful pipeline by following the existing works, the empirical contribution is more visible than the insight novelty.\n- Compared with EsViT, the improvements are much more marginal, due to the less design difference.\n- Using the images from HICO, the HOI concepts are complex and suitable for relation reasoning. But HOI images usually contain multiple HOI human-verb-object pairs in one image. For example, in HICO, one image usually has more than 2 h-o pairs. So, during the training, what is the effect of the overlapped concepts in contrastive learning, e.g., the largely overlapped pos-neg concepts. Thus, if applying the proposed work to the instance-level concepts (one human-verb-object pair bbox), what would happen? \n- The comparison about the \"complex pre-processing stage\". This point seems unconvincing. In my opinion, these previous settings are a kind of design not for \"efficiency\". And if the proposed method has an obvious advantage on efficiency, supportive results should also be provided. So does the claim about the long-tailed data distribution. In HICO, the long-tailed distribution is very severe. Thus, a discussion about the alleviation of the long-tailed bias would make this work more solid.\n- \"reasoning hops can be found in Table 3.\" --> \"Figure 3\"?\n- \"using ViTs in eliminating the need for external object detectors\", this discussion is very interesting and inspiring, could you please give more analyses combining the discoveries of the proposed model?", "summary_of_the_paper": "This work studies visual relationship reasoning and proposes a model based on the recent contrastive learning methods DINO and EsViT. Compared with the two predecessors, a feature dictionary is proposed to provide the online updated concept features for the teacher model instead. Correspondingly, two auxiliary tasks are introduced to drive semantic concept learning. Experiments on two large benchmarks show the efficacy of the proposed method. And extensive ablations are also made to probe the model components.", "summary_of_the_review": "Overall, this is an interesting work to dig into the performance of ViT on visual relation reasoning and semantic concept learning. Some non-trivial contributions are made to reveal the effect of contrastive learning designs. And the results on the two benchmarks look promising. However, though studying a very different problem, the method contribution seems weak compared to the previous works. Thus, I think the rating of 6 is suitable for this version. I will adjust my rating according to the response and the discussions from the authors and the other reviewers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QC0rBfllYkO", "reviewer": "Reviewer_dCw4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Well-organized paper, it's clear which previous works the method is built upon and what the novel contributions are\n- A relatively simple implementation based on queues of previously-processed images is shown to improve empirical results on two challenging datasets (state-of-the-art mAP in HICO recognition, good results in GQA).\n- Well-detailed appendix, all the info about data pipeline and training hyperparameters should ensure full reproducibility of the results\n- Well-designed ablation studies that cover most of the questions that one can have while reading the description of the method\n\nWeaknesses:\n- The paper lacks a proper comparison with MoCo which is only mentioned briefly in section 4 (K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum Contrast for Unsupervised Visual Representation Learning,”, https://openaccess.thecvf.com/content_CVPR_2020/html/He_Momentum_Contrast_for_Unsupervised_Visual_Representation_Learning_CVPR_2020_paper.html\n). In this paper, in fact, many of the concepts of RelViT are already present, almost with the same names, e.g. \"At the core of our approach is maintaining the dictionary as a queue of data samples\" (MoCo, section 3.2). The main differences are: MoCo uses a single queue and not many concept-based queues, the queue contains images and not features, teacher features are computed on the fly using the current parameters for the teacher.\n- The additional losses in the method are described as weakly-supervised, e.g. \"we use concepts as a source of weak supervision\" in section 4. I think this might be a misrepresentation of the method. In fact, for both use cases demonstrated in the paper, the concepts were extracted from the fully-supervised training labels of each dataset. For example, the basic training signal in HICO already includes the concepts used to build the dictionary, therefore it can not be represented as a form of weak supervision. I suggest rephrasing the relevant paragraphs so that it's clear that RelViT is not a weakly-supervised method but it makes better use of the strong labels present in a dataset.\n- Since the model is applied to relationship detection, I suggest adding experiments on VRD (C. Lu, R. Krishna, M. Bernstein, and L. Fei-Fei, “Visual Relationship Detection with Language Priors” http://arxiv.org/abs/1608.00187) and Unrel (J. Peyre, I. Laptev, C. Schmid, and J. Sivic, “Weakly-supervised learning of visual relations” http://arxiv.org/abs/1707.09472), which are two popular datasets for visual relationship detection\n- One ablation study on the concept-based queues is missing: using a single queue of old images. This would verify whether a replay buffer alone (akin to keeping track of multiple teachers) already brings an improvement over using the features of the current teacher only. However, maybe this is already present and corresponds to the \"None\" label in the plot, it's not very clear from the text whether None means no queue at all or a single non-concept-based queue.\n\nQuestions:\n- $L_{main}$ is left unspecified in section 2.2, it would help if the loss was detailed in full for the tasks used in the experiments. At a first read, I missed the fact that HICO is used for a recognition task (just predict the list of relationships present in an image) and not for detection (detect object bounding boxes and predict relationships). It might be my bias because I'm used to seeing HICO for detection, but also because $L_{main}$ for HICO is not properly specified.\n- For the local task, patch-based features from the student's image and the teacher's queued image are greedily matched in the loss. Is the match guaranteed to be 1:1? Is this desirable or not desirable? Can you add a discussion about this?\n- Is \"RelViT\" in figure 6 only RelViT or RelViT+EsVit? If it's only RelViT, without the local loss, why is it so good at matching local regions?\n\nMinor points:\n- Equation numbers would help reference specific parts of the paper\n- From section 2.2 it's not clear which part of the method corresponds to the experiment rows in table 1: EsViT, RelViT, RelViT+EsViT. The confusion arises also in other parts of the paper. I suggest making the similarities and differences explicit, e.g. with the table below\n\n|              | $L_{global}$ | $L_{local}$ | Compare `student(aug(img))` with |\n|--------------|--------------|-------------|----------------------------------|\n| DINO         | x            |             | `teacher(aug(img))`              |\n| EsViT        | x            | x           | `teacher(aug(img))`              |\n| RelViT       | x            |             | `queues[concept(img)].pop()`     |\n| RelVit+EsViT | x            | x           | `queues[concept(img)].pop()`     |", "summary_of_the_paper": "The paper extends the training objective of EsViT to make it better suited to visual relationship detection. This is done by replacing the teacher's features used in both local and global losses with features taken from a queue. The queue is populated with teacher features extracted from previous images. Furthermore, the queue is partitioned by \"concepts\", e.g. HOI labels or VQA concepts, so that the current student image can be compared with a previous teacher image that contains the same concept instead of any random image.\n\nThe novel queue-based formulation is tested in various experiments, both for visual relationship detection and visual question answering. Individual components of the method are ablated and discussed, namely the dictionary of queues, the EsViT-style local loss, and other hyperparameters.", "summary_of_the_review": "This is a good paper in terms of ideas, writeup and experiments. My suggestions and critics would require just minor adjustments to the paper that can be done in the rebuttal phase. Unless major issues arise in the discussion with other reviewers, e.g. the method is not as novel as I thought due to related works that I am not aware of, my recommendation is to accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three experts reviewed the paper and all recommended acceptance. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance. However, the reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. For example, the discussion about or comparison with related works, clarity of the writing, suggested experiments, etc. The authors are encouraged to make the necessary changes to the best of their ability. We congratulate the authors on the acceptance of their paper!
Decision: Accept (Poster)