ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation
Authors: Yangyi Chen
Xingyao Wang
Manling Li
Derek Hoiem
Heng Ji
Venue: EMNLP 2023 Main (2023)
Keywords: Visual Structural Knowledge Extraction
Code-Vision Representation
Curriculum-based Learning
TL;DR: A framework to improve the visual structural knowledge extraction ability of vision-language models.

Abstract: State-of-the-art vision-language models (VLMs) still have limited performance in structural knowledge extraction, such as relations between objects. In this work, we present ViStruct, a training framework to learn VLMs for effective visual structural knowledge extraction. Two novel designs are incorporated. First, we propose to leverage the inherent structure of programming language to depict visual structural information. This approach enables explicit and consistent representation of visual structural information of multiple granularities, such as concepts, relations, and events, in a well-organized structured format. Second, we introduce curriculum-based learning for VLMs to progressively comprehend visual structures, from fundamental visual concepts to intricate event structures. Our intuition is that lower-level knowledge may contribute to complex visual structure understanding. Furthermore, we compile and release a collection of datasets tailored for visual structural knowledge extraction. We adopt a weakly-supervised approach to directly generate visual event structures from captions for ViStruct training, capitalizing on abundant image-caption pairs from the web. In experiments, we evaluate ViStruct on visual structure prediction tasks, demonstrating its effectiveness in improving the understanding of visual structures. The code will be made public to facilitate future research.

ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code- Vision Representation
Yangyi Chen', Xingyao Wang', Manling Li', Derek Hoiem', Heng Ji' University of Illinois Urbana-Champaign Northwestern University [yangyic3, xingyao6, manling2, dhoiem, hengji lillinois.ede
Abstract
Generate Visual Concepts
object11 horae(
loc=[pos_2,, attributes=! "hungry')
pos,4),
State-of-the-art vision-language models (VLMs) still have limited performance in structural knowledge extraction, such as relations between objects. In this work, present ViStruct, a training framework to learn VLMs for effective visual structural knowledge extraction. Two novel designs are incorporated. First, we propose to leverage the inherent structure of programming language to depict visual structural information. This approach enables explicit and consistent representation of visual structural information of multiple granularities, such as concepts, relations, and events, in well-organized structured format. Second, we introduce curriculum-baan learning for VLMs to pro- gressively comprehend visual structures, from fundamental visual concepts to intricate event structures. Our intuition is that lower-level knowledge may contribute complex visual structure understanding. Furthermore. we compile release collection of datasets tailored for visual structural knowledge extraction. We adopt a weakly-supervised approach directly generate visual event structures from captions for ViStruct training, capitalizing on abundant image-caption pairs evaluate
object2 grass
loc- pes_e, pos_3, 90,_4),
attributes" "green', 'sparse']
Genorate Relations
walk, on(ssuroojjecct, obj-object2)
Generate Visual Event with Argunents (ingestorrhorre, ingestiblessgrass)
Figure I: Programming language enables unified representation of visual structural information code- vision representations, including concepts, attributes, locations, relations, and visual events.
et al., 2023), and multimedia misinformation detce- tion (Fung et al., 2021; Luo et al., 2021). The recipe lies in the pre-training on web-scale data, enabling substantial enhancements in vision-languag repre- sentation (Radford al., 2021; Li et al., 2022a). However, recent evaluations expose the limita- tions of existing VLMs models capturing visual structural information (Hendricks and Nematzadch, 2021; Zhao et al., 2022: Yükkskkönüü et al., 2022), rendering them ineffective for visual structural knowledge extraction. For example, Yüksekgönül et al. (2022) argue that VLMs represent images as bag-of-words, disregarding relations among ob- jects. In this work, we present ViStruet, a training framework for VLMs to improve their visual struc- tural knowledge extraction ability. There are two key challenges multi-granularity visual structural knowledge learning: (1) capturing the structures at each granularity, for which we propose code-viee visiovisin representations, and (2) recognizing the dependen- cies between different granularities, for which we propose curriculum pyramid to guide the knowl- edge learning.
from the web. In experiments, ViStruct on visual structure prediction tasks, demonstrating its effectiveness in improving the understanding of visual structures. The code is public at https: //ggtttttt.///////////// com/ Yangyi-Chent Chen/ Chthhheeiiisruett.....eeeessseee
Introduction
Vision- models (VLMs) exhibit impres- sive performance across various multimodal down- stream tasks (Li et al., 2019a; Tan and Bansal, 2019; Chen al., 2023b: Wang al., 2022a), such as visual question answering (Antol et al., 2015; Yuan et al., 2023), image captioning (Karpathy and Fei- Fei, 2015), visual event extraction (Li et al., 2022b; Wang et al., 2022c), chart understanding (Zhou
The motivation of employing code-vision rep-

---

resentations is to mitigate the negative impact of commonly employed techniques such as con- trastive learning (Radford et al., 2021) and image- conditioned language modeling (Li et al., 2023), which limit VLMs to focus solely on primary ob- jects, resulting in "bag-of-words" representations. we draw inspiration from prior successful appli- cation of programming language in representing linguistic structures (Madaan al., 2022; Wang al.,, 2023b). We propose to extend this "code for structure representation" principle to multimodal scttings for explicit representation of visual struc- tures. Initial attempts to employ code represen- tation in visual reasoning (Gupta and Kembhavi, 2023; Surís et al., 2023) focus solely on object in- formation and lack perception abilities, but relying on the off-the-shelf object detectors tools. As a result, establish multi guati-granularity code rep- resentation framework to capture structured visual information, including visual concepts, attributes, relations, and events (see Figure I).
in a scalable manner. To ensure consistent class label definitions, we automate the alignment of all concepts from various datasets to WordNet (Miller, 1995) and FrameNet (Baker et al., 1998) synsets, with minimal manual assistance.
For downstream task evaluation, we find that ViStruct consistently outperforms various baselines three downstream structure prediction tasks (vi- sual relation detection, scene graph classification, and situation recognition). In addition, we find that code representation and curriculum learning both provide consistent gains on downstream tasks and demonstrate their effectiveness on zero-shot visual structure prediction.
Parsing visual data into various levels of detail has been longstanding problem. To the best of our knowledge, we are the first extract multi- granularity visual structured knowledge via code- vision representation learning, which effectively captures the hierarchy among different levels of visual knowledge, offering a comprehensive and compositional understanding of overall structure.
To further capture the interdependencies among various granularities, we propose the curriculum guided visual structural knowledge learning. Our intuition is that foundational knowledge (such as concepts) may enhance comprehension of more complex visual structures (such as visual events). Thus, unlike previous work that involves pre- training VLMs on mixture of tasks, we build a visual structural knowledge pyramid (see Fig- ure 2), aligning with the pedagogical strategy of progressively addressing simpler to more complex tasks during training (Soviany et al., 2022; Wang et al., 2022b). However, in contrast to conven- tional curriculum learning methods that focus on arranging data samples to facilitate training for specific task (Jiang et al., 2014a; Spitkovsky et al.. 2010b), our objective is to enable VLMs to un- derstand visual structures comprehensively at all levels. To accomplish this, we incorporate a replay buffer mechanism that selectively reserves a small amount of data at each level to preserve lower-level knowledge throughout the training process.
2 Related Work
2.1 Vision-Language Models (VLMs) for Visual Knowledge Extraction
Visual knowledge extraction has been long opti- mized as separate tasks for each granularity of ex- traction, such as object detection, relation detec- tion, scene graph parsing, situation recognition and event extraction, further detailed in Appendix B. Recent years witness great success through VLMs to obtain knowledge of visual concepts (Sadhu et al., 2021) and events (Wen et al., 2021; Desh- pande et al., 2022; Li ct al., 2022c). The primary goal of VLMs is learning representations from mul- tiple modalities to capture visual details and their alignment with text semantics (Gan et al., 2022; Uppal et al., 2022; Wang et al., 2022c), and the corresponding research can be broadly classified into three stages.
The initial stage of VLMs primarily relies on off-the-shhe shelf object detectors to represent visual modality (Li et al., 2019b; Tan and Bansal, 2019; Lu et al., 2019; Li et al., 2020a) and learn the align- ment across modalities (Li et al., 2020c, 2021b).
Moreover, we curate an extensive collection of datasets for ViStruct, called ViStruct Suite. This suite includes multi-level structural knowledge, aligned with the curriculum pyramid. We first collect existing labeled datasets and large-scale unlabeled image-caption pairs. Then, we adopt a approach to facilitate the cre- ation of visual event structure labels from captions
The second phase of VLMs aims to eliminate the use of resource-intensive object detectors and ad- dress the issue of unseen objects (Dou et al., 2022; Huang et al., 2020; Kim et al., 2021; Huang et al., 2021; Xu et al., 2021; Yang et al., 2022; Yao et al.,

---

Concepts Recognition
Objects Grounding
Altribute
Object-Objan Relation
Event
object1 horsel loc= [pes_],
object1 horse( locs [pos_k, ("breen' "hungry"
walk subnobject,, abf-objec12
eati ingestoraherse fngestiiles--gras
pos_4], object2 grasst
4os_4],
attributes-ione
jject2 grass(
object2 grasst
locehone,
loc: (pos_1, pos_2, pos_3, pos_4], attributes-None
loc: (oc=ipos_A,, pos_2, pes_a, pos_4], attriibutss- groen", "sparse")
attrnbbtes-looee
Increasing Complexity
Figure 2: The curriculum pyramid in ViStruct that incorporates five different levels of multimodal knowledge acquisition, progressing from basic to more advanced stages. All levels of visual structures can be uniformly represented using the programming language. The code generated based on the image in Figure 1.
2022). Large-scale multimodal data is required and emphasized to better align two modalities (Li et al., 2021a; Radford et al., 2021; ; Jia et al., 2021).
ViStruct
We propose ViStruct that learns a rision-language model that takes images input and generates set of structural knowledge, such as concepts, rela- tions, and events involved in the images (see Fig- ure 2). We first define code-vision representation (Sec. 3.1), which serves as the output decode vi- sual structural knowledge as code blocks. We then propose a curriculum pyramid to guide the train- ing objectives from low-level towards high-level semantics (Sec. 3.2). To train the model, we con- struct a comprehensive multi multi-granullarity dataset ViStruct Suite (Sec. 3.3), and optimize the model only on information that describe relavant visual streutural information (Sec. 3.4).
The current stage of VLMs moves towards a more unified pre-training goal to perform various tasks without task-specific adaptations (Wang al., 2021, 2022a; Li et al., 2023). Some research leverages programming language (Gupta and Kembhavi, 2023; Surís et al., 2023), but lacks perception ability and relies on pre-trained object detectors. Although we witness great success of existing VLMs on benchmark evaluations, in-depth diagnostic investigations have exposed their shorteomings in fundamental visual concept identification (Hendricks and Nematzadch, 2021; Zhao et al., 2022: Yükkseköönül al., 2022), raising doubts about whether current VLMs are overemphasizing objects to represent images while ignoring other visual structures, akin to the initial stage VLMs, making them inadequate in structural knowledge extraction. Thus, in this work, we design a training framework to improve VLMs visual structural knowledge extraction.
3.1 Code-Vision Representation
Visual structural information depicts both visual knowledge elements (such as visual concepts, locations, attributes) and complex structural connections between them (such as relations, and visual events with associated arguments). To en- code visual structural knowledge, the first question is to define the appropriate structured representa- tion. Such representation is required to not only have robust capabilities for illustrating structures and their intricate interconnections, but also be unified and consistent across multiple levels semantic granularity. It is also expected to be adapt- able and extendable to accommodate other future semantic granularities. In order to address these re- quirements, we leverage the inherent capabilities of programming language (PL). We adopt Python the PL this paper due its straightforwarddess and popularity, allowing us effectively harness its structured representation power and well-resourced data for code-vision pretraining. Similar to PL, we define ViStruct including both class definition and
2.2 Curriculum Learning
Curriculum learning has proven be useful tech- nique for improving the learning process by grad- ually exposing the learner to more challenging ex- amples (Bengio et al., 2009; Karras et al., 2018; Wang et al., 2020; Su et al., 2021; Platanios et al., 2019; Yu et al., 2022). However, it has been stud- ied mainly in a more limited form, ordering and presenting examples to train single task (Bengio al., 2009; Spitkovsky et al., 2010a; Kumar et al., 2010; Jiang et al., 2014b: Graves al., 2017). In this work, we propose the first task-level, multi- faceted, multimodal approach to curriculum leamn ing for visual structural knowledge learning.

---

Visual Structural Information & Definition
Pl. class definition
Algorithm 1 The training algorithm described in We omit the warm-up process for brevity.
Visul
Class
Sec. 3.4.
Event uqumentesubbectt (object) in relution
Arguments for cluss instamiation Doestring
Human writeen definition.
Po b pre-trained vision-language model
Table 1:
The alignment of visual structures, visual se- mantic roles, annotated definitions, and programming language (PI.) class definitions. The concrete examples are shown in Figure 3.
Daune [DRR, DoG, DOA, DOR DE]
D
Stage-specific datasets defined in Sec. 3.2
3: DRB = 0
P Replay buffer in Sec. 3.2
4: for D, in Dstage do
5: Dunin = D3 U DRB
Definition of Visual Concept
6: Finetune M using Drrain
Class (dbject) nen
DRB = DRB U raadomm_sstt handom__ubset_uue (D.
Dafinitian: solid-hoofad herbivaraus quadruped donesticated since prehfstoric times nen
end for
def init.. (Lac=Nana, -tact___aaa-,, attributes:lane);
super Inft... (loc=loc, attribures-attrriuutes)
niently represented in unified and structured man- ner. In this work, we consider the most salient vi- sual structural information, including concepts, lo- cations, attributes, relations, and visual events with associated arguments. The structure-code mapping is detailed in Table 2. It is noteworthy that our code- vision representation can be effortlessly expanded encompass supplementary visual details, such affordance, object states, and similar aspects.
Definition Visual Event
class building(Activity)::
Definition: This frane describes assembly constructian actions, where Agent joins
Corponents tagether to form Created entily.
The Agent builds Created_entie entitt. created, entity: Created entity identifies the entity that is created in building. new
def inftt.. (agent-Maa,,,,,,,, created, antity-None):
self.agent, agent self.created_entityy created_entityy
3.2 Curriculum-Based Training for Knowledge Extraction with Replay Buffer
Figure Examples of the class definitions of basic visual concepts and visual events.
Existing vision-language pre-training approaches are mostly based on end-to-end training on mix- ture of tasks (Li et al., 2023), which operates dif- ferently from the way humans learn. In human learning, for example, the well-validated Montes- sori method (Lillard al., 2017) focuses on cur- riculum self-directed learning stages, where demonstrated mastery of elementary concepts is required before advancing to those more complex. Based on this insight, we propose a curriculum pyramid that directs VLMs towards progressively learning to extract visual structural knowledge from simpler to more complex concepts. In particular, introduce the replay buffer mechanism to pre- vent the forgetting of foundational structural knowl- edge obtained at the lower levels of the curriculum. The training algorithm is described in Alg. 1.
instantiation of visual structural information.
Class Definition of Visual Structural Informa- tion. To define a general set of multi- gulli-grramllarity visual structures, we combine two external knowl- edge resources: WordNet (Miller, 1995) for basic visual concepts and relations; and FrameNet (Baker al., 1998) for visual events. Class definitions PL provide a unified and structured way to represent the visual structures, semantic roles, and human-written definitions in WordNet and FrameNet. Table details the structure-class map- ping we defined. For each visual concept, we de- fine object location loc for object grounding, and attribbu attribbues for object properties, shown in the "horse" example in Figure 3. On top of that, we define visual events, which have visual concepts as participants, characterized by their semantic roles (such as agent and Create_entity. y in the ex- ample visual event "building" in Figure 3). As discussed in Sec. 3.4, these definitions are used to warm up the model through finetuning.
Curriculum Pyramid. We present our curricu- lum pyramid in Figure 2. We categorize the process of learning complete visual structural knowledge into five levels, based the assumption that knowl- edge acquisition at lower levels (such as concepts) can facilitate the acquisition of higher-level knowl- edge (such as visual events). For example, Fig- ure 1, identifying the "horse" and "grass" concepts contributes to understanding the "eat" event with
Instantiation of Visual Structural Information. As shown in Figure 1, the multi caaltiigranulartty visual structural information of each image can be conve-

---

Structural Information
Python Code Representation
Concept
object robject_synsee>> (loceNone, attribbute-tiioe)
Grounded Object
object <objectt rooject__yyneet syst__sssstt>ecceeee (Acce (pos_k, pot_á, pos_3, pes_4], actributes-tione)
Attribute
object sobrjcct_aynsetz (1oce [poa_1, 1, po9_2, poa_3, po9_4], 4), attributes= atcr
Relation
<relatiion fon_ syrsett souueedeeen I1l poddjooeeet
seeddefrree__obbb je012)
Event
seventt Ke rgguent nemeeoooo joojswooojjee jeot peet_r Kangument remeett_ramesssssee jeul name?)
Table 2: The correspondence between visual structural information and Python code representations.
Caption: kitten sitting dish the table.
cepts from the learning of task-specific skills is ef- fective. We adopt this approach and anticipate that VLMs will gain a range of fundamental visual con- cepts, including rare object categories, at this level. At the second level, VLMs are required to perform Object Grounding (OG), which involves the iden- tification of objects with specific, concrete loca- tions in the images. The third level places emphasis on Object Attribute (OA) detection, with the goal of inferring the attributes associated with particular objects (e.g., the horse is brown and hungry).
#ttCagenbski Loen, deptict tepicctrendish, locattor=table)
We generate the visual events with argument roles from the captions using the off-the-shelf system.
Figure 4:
these two participants.
Conditioned on an image input, we train the model to reconstruct masked visual structures rep- resented using code (c.g., arguments). Specifically, we mask out targeted visual information in the code and treat this masked sequence along with the image input to the model. For example (Figure 1), in object attributes learning stage. we mask out the target attributes: object horse (loc= pos_4], attributes= (<mask>]) Then we train the model to generate the code sequence replacing the <mask> with "brown" and "hungry". Formally, given an image i, the code sequence and mask function mask(.) to mask different information for different stages. We use ruto-regressive loss to train the code-vision representation model on the dataset D, that is, we are minimizing the following loss function:
The initial three levels facilitate the precise wn- derstanding of objects. We then comprehend the interaction between objects and the deep seman- fics embedded within the images. The fourth level centers on visual Object Relation (OR) detection. with the objective of identifying the basic relations between two objects (e.g., predict a walk-on rela- tion between horse and grass). In the final level, the focus is on high-level comprehension of the images such as Event (E). Beyond the perceptual tasks, visual event detection requires VLMs to pro- vide summary of what is happening in the images and the participants involved in the event (e.g., the horse eating grass).
pos_2, pos_3,
We highlight the flexibility of our curriculum pyramid that can be casily extended to incorporate new structural information at different levels with the goal of improving overall multimodal knowl- edge acquisition. This adaptability is facilitated by the flexible code-vision representations employed.
((0) - Σ logPo(sli, mask(s))
(b,))DD
where A refers to the model parameters, D is the dataset we used for training, which composed of a stage-specific dataset Depesitic E (DeR, DoG, Doa, DOR, Dr. for all the five stages in Figure 2 and a replay buffer DRB. that is, D = Dspecific U DRB for each stage. We use Figure as running example to introduce each stage in the pyramid.
Replay Buffer. We aim empower VLMs with the capacity to proficiently acquire and maintain structural knowledge across various levels, aligning with the underlying motivation of lifelong learn- ing research (Biesialska et al., 2020; Wang et al., 2023a). Although the acquisition of lower-level knowledge remains vital for higher-level knowl- edge attainment, directly training VLMs high- level tasks can lead catastrophic forgetting, espe- cially when dealing with infrequent and rare con- cepts. To prevent this, we build the replay buffer by random sampling at the level of the curricu- lum pyramid. During training, VLMs utilize the
The initial stage involves basic Concept Recog- nition (CR), which applies VLMs to identify the most prominent objects in the images presented (e.g., horse and grass). recent study by Kamath et al. (2022); Whitehead et al. (2021), it is demon- strated that decoupling the acquisition of basic con-

---

data samples specific their respective level (i.e.. samples from previous stages stored in the replay buffer). The amount of data we sampled from each stage can be found in Table 7 in the Appendix.
Appendix C). For the initial three levels in the cur- riculum, the model is trained for one epoch, while for the last two levels involving deep-semantic un- derstanding, the model undergoes three epochs of training. Specifically, we introduce a focusing optimization trick that only optimize loss for the masked tokens (see Appendix D for more details).
3.3 ViStruct Suite Construction
We assemble a comprehensive dataset collection consisting of both labeled and unlabeled datasets for ViStruct corresponding to each level in the cur- riculum pyramid. The selected datasets along with their statistics are listed in Table 7 in the Appendix. To curate the object recognition dataset, exclude rare classes with less than 500 samples and retain a total of 10,450 categories. On top of it, we add object grounding with 80.6M categories, object attributes with 14.6M attribute annotations, and ob- ject relations, using the resources Table 7. For visual event understanding, we propose weakly- supervised method to generate training data from the large-scale image-caption pairs in Table 7. As shown in Figure 4, for each image, we leverage the off-the-shelf semantic role labeling model to perform semantic parsing on the paired caption. Aligning with ViStruct, we adopt the FrameNet for the event and corresponding argument definitions. More details are included in Appendix E.
4 Experiment
We evaluate ViStruct on three structure predic- tion tasks, including visual relation detection (Yu et al., 2023), scene graph classification (Zellers et al., 2018; Yao et al., 2021), and situation recog- nition (Yatskar et al., 2016, 2017). We refer to Appendix A for implementation details.
4.1 Visual Relation Detection
Dataset & Metrics. We evaluate the visual re- lation detection task on Visual Genome (Krishna et al., 2017). Following Krishna et al. (2019), we adopt the refined version that only contains 20 well- defined relation categories. Each image is anno- tated with identified objects and their relations. The task is to detect the relation between two given ob- jects (provided along with their bounding boxes).
We gather the aforementioned datasets and con- duct subsequent post-processing procedure. To establish consistency with ViStruct across various datasets, we align all categories from different datasets the WordNet and FrameNet synsets through the automatic mapping process imple- mented in the NLTK toolkit (Bird et al., 2009). In cases where multiple categories in a dataset are assigned to the same synset, we manually adjust the mapping based on our prior knowledge.
For evaluation metrics, we measure two types of model performance following previous work (Yao et al., 2021). First, we report the micro-averaged recall in top K relation predictions (R @ K). Second, evaluate the marco-averagee recall of all 20 lation types across the top K predictions (mR @ K), intending to measure the performance on long-tail relations. In our study, we select K=50, 100.
Baselines. We compare ViStruct against three types of baseline methods. (1) Task-specific mod- els: We consider the Neural motifs (Zellers et al.. 2018) (Motif) that incorporates task-specific de- signs (c.g., off-the-shelf object detectors); (2) Ex- tra supervision-enhanne the distant supervision method proposed in Yao et al. (2021), which introduces extra signals for pre- training (Motif+P) and denoising (Motif+D); (3) Ablations of ViStruct: We aim to measure the sepa- rate gains of two designs within ViStruct. Specif- ically, we quantify the gains of code-vision rep- resentations by measuring the performance of the backbone model OFA- which is trained on the same datasets but with natural language (i.e., image-conditioned language generation) (ViStruct- Text). In assessing curriculum learning ablation,
3.4 Model Training
ViStruct is based on continued pretraining, and can work any pre-trained VLMs. In this work, we adopt OFA-base (Wang al., 2022a) as back- bone model since offers simple seq-to-seq mod- cling approach. We train the model by feeding im- ages and the masked sequence mask(s) ) as defined in Equation 3.2 as input to the encoder, and train the decoder to produce s conditioned on the encoder outputs. Before the curriculum training, we include a warm-up pretraining stage to facilitate the mod- cls' acquisition of fundamental code syntax struc- tures and visual concepts with their definitions (see
methods: We consider
'tttps: //g thub. com/ chan ind nd/ frame- semant o--taans- transformen former

---

Method
R6050
R 100
mk 60
mR @ 100
Mean
Method
R@
R@ 100
ml
mR @ 100
Mean
67.93
70.20
52.65
55.41
61.55
Motif
31.14
31.92
23.53
25.27
27.97
Motif+P*
73.22
75.04
60.44
63.67
68.09
Motif+Pr
34.11
34.88
26.51
27.94
30.86
Motif+D*
76.28
77.98
60.20
63.61
69.52
Motif+D*
35.93
36.47
28.07
30.09
32.64
ViStruct-Tea 73.39
75.16
63.27
67.42
69.81
USStructTexxt
30.55
31.04
26.26
28.73
29.15
ViStrict-Mix 74.71
76.32
61.41
68.26
70.10
WSStruct-Mix
32.34
33.65
26.22
27.87
30,02
ViStruct 75.71
78.53
63.71
70.71
72.17
ViStruct
34.72
35.27
30,51
32.26
33.19
Table 3: The results (%) of the visual relation detection. denotes results from Yao et al. (2021).
Table 4: The results (%) of the scene graph classifica- tion. denotes results from Yao et al. (2021).
we measure the performance of pre-training on the ViStruct Suite mixture (ViStruct-Mix).
4.3 Situation Recognition
Dataset & Metrics. We evaluate ViStruct SWiG dataset (Pratt et al., 2020). Each image the dataset annotated with the major activity and corresponding participants (a.k.a,, objects). The models must sequentially predict both the activity and the participants involved. Following Cho et al. (2022), we measure whether the gold-standard ac- tivity is predicted in the top-1 (Top-1 Predicted Verb) and top-5 (Top-5 Predicted Verb) model predictions. The evaluation of participant pre- diction entails two aspects: Value-all examines whether all predicted participants for an activity are correct, while Value evaluates the accuracy of each predicted participant in their assigned role. Participant prediction is considered correct if it matches any of the ground truth annotations. All metrics are calculated on per- ver-verb-specing basis and subsequently averaged across all verbs.
Experimental Results. As shown in Table ViStruct consistently improves the visual relation detection performance regarding both the common and long-tail relation types. The reason lies in the diverse relation types ViStruct encounters during the structural knowledge learning phase, which pre- vents it from overfitting to some frequent relation types. Additionally, demonstrate the separate gains of two novel designs in ViStruct.
4.2 Scene Graph Classification
Dataset & Metries. We continue to utilize the refined Visual Genome dataset for evaluation pur- poses. The scene graph classification task involves predicting object categories and relation rypes based on bounding box annotations, thereby as- sessing the models' concept recognition capabili- ties besides the ability to detect object interactions. We measure the @@K and mR @ K as described in Sec. 4. and the correctness of the predicted relation is determined solely by the accuracy of the predicted relation types and object categories.
Baselines. We compare ViStruct against three types of baseline methods. (1) Traditional task- specific models: We include GraphNet (Li et al., 2017), CAQ w/ RE-VGG (Cooray et al., 2020), and Kernel GraphNet (Suhail and Sigal, 2019), studied in Cho et al. (2022); (2) State-of-the-ar task-specific models based Transformers: We consider the CoFormer model that consists of two interactive modules (Cho et al., 2022). (3) Abla- tions of ViStruct: We consider the same ablations of ViStruct introduced in Sec. 4.1.
Baselines. We adopt the same baselines Sec. 4.1.
Experimental Results. As shown in Table ViStruet consistently outperforms baseline meth- ods. Additionally, the performance gap between ViStruet and baseline methods including task- specific models and two ablations increases com- pared to the results Table 3. This can be at- tributed to (1) the extensive coverage of fundamen- tal visual concepts in ViStruct, aligning with the requirements of scene graph classification for ob- ject category identification, and (2) the carefully arranged curriculum that guides the model to pro- gressively comprehend the visual structures, pre- venting the model from overfitting on the large dataset visual concepts.
Experimental Results. As shown in Table 5, ViStruct demonstrates substantial enhancements in both activity and participant detection, which can be attributed to its weakly-supervised training approach utilizing web-scale data. Our approach effectively utilizes the captions generate visual event structures depicted in the web images. The benefit lies in the increased inclusivity of infre- quent events and visual concepts, facilitated by the extensive volume of image-caption pairs. In addi-

---

Metric
Top-1 Predicted Verb
Top-5 Predicted Verb
Ground -ruuh Verb
Set
Method
verb
value
value-all
verb
value
value-all
value
value-all
GraphNet*
36,93
27.52
19.15
61.80
45.23
29.98
68.89
41.07
CAQ w/ RE-VGG*
37.96
30.15
18.58
64.99
50.30
29.17
73.62
38.71
Kernel GraphNet*
43.21
35.18
19.46
68.55
56.32
30.56
73.14
41.68
Dev
CoFormer
44.41
35.87
22.47
72.98
57.58
34.09
76.17
42.11
ViStruct-Teae Text
44.10
35.04
21.19
71.04
55.24
31.76
75.04
40.18
ViStruct-Mii
44.68
35.58
21.73
71.15
55.65
32.61
75.31
40.82
ViStruct
46.23
38.82
24.29
74.81
60.91
37.41
79.23
46.23
GraphNet*
36,93
27.52
19.15
61.80
45.23
29.98
68.89
41.07
CAQ RE- VGG*
37.96
30.15
18.58
64.99
50.30
29.17
73.62
38.71
Kernel GetphNet*
43.21
35.18
19.46
68.55
56.32
30.56
73.14
41.68
Test
CoFormer
44.41
35.87
22.47
72.98
57.58
34.09
76.17
42.11
ViStruct- Text
44.94
35.86
21.99
71.19
55.70
32.69
75.29
40.83
ViStruct-Mix
44.72
35.60
21.77
71.03
55.51
32.62
75.23
40.81
ViStruct
46.89
38.58
24.91
74.57
60.40
37.53
78.99
46.62
Table 5: The results (%) of the situation recogmition task.
denotes results from Cho et al. (2022).
Mothod
R@:50
R@ 100
m:
mR 100
Mcan
imprevoment Curtculum
ViStract-Test
7.05
10,23
3.02
4,43
6.18
ViStmec
18.22
24.43
6.66
8.89
14.55
75
Table 6: The results (%) of the zero-shot visual relation detection task.
70
85
tion types. This demonstrates the benefits of the curriculum learning paradigm for generalization low-resource settings.
5.2 Zero-shot Capability
Figure 5: The effectiveness of the curriculum learning framework for visual relation detection. The metrics are introduced in Sec. 4.1. The dotted lines denote the results of ViStruct-Mix.
We evaluate ViStruct on the zero-shot visual rela- tion detection task. The results are presented in Table 6. We compare with the results of training natural language (ViStruct-Textt introduced in Sec. 4.1, and show that the code-vision represen- tation exhibits significantly better zero-shot per- formance (i.e., 2.4x better on average). The code- vision representation naturally facilitates the zero- shot visual structure prediction ability, owing to the unified format in both the pre-training and the downstream tasks.
tion, observe the concrete gains associated with the utilization of curriculum learning. In alignment with the conclusion in Sec. 4.2, curriculum learn- ing aids the VLMs to focus on high-level visual structure comprehension in the later stages, while simultancously mitigating the risk of overfitting.
5 Further Analysis
Conclusion
5.1
Curriculum Learning
We conduct an ablation study to measure the per- formance of models trained at each stage of the curriculum on the visual relation detection task. We also compare the results of directly pre-training the mixture of ViStruct Suite (ViStruct-Mix). As shown in Figure 5, the training progresses, the evaluation metrics regarding two aspects continue to increase, gradually surpassing the performance of ViStruct-Mix, especially for the tail rela-
We present visual structural knowledge extrac- tion framework ViStruet as new step towards multi-granularity semantic parsing of visual data. First, utilize the inherent structured of programming language to represent visual structural knowledge of different levels unified way. capture the interdependencies among various granularitics, we then leverage linguistic cues and propose a curriculum learning pyramid

---

that guides the models to effectively acquire visual structural knowledge of each level. Results show significant and consistent improvements on structure prediction tasks across all levels.
learning in natural language processing: survey. In Proceedings of the 28th International Confer- ence Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 6523-6541. International Committee on Com- putational Linguistics.
Limitations and Future Work
Steven Bird, Ewan Klein, and Edward Loper. 2009, Nat- ural language processing with Python: analyzing text with the natural language toolkit. " "O' O'Reilly Media, Inc.".
This paper utilizes the programming language to represent visual structures, with a focus on knowl- edge representation. However, due to the con- strained capability of the base model, direct code generation is not feasible, resulting in reduced per- formance downstream tasks. One potential solu- tion is to integrate more advanced vision-language models, such as BLIP-2 with more capacity. By doing so, the trained models can effectively gen- crate code representations of visual structures for diverse images in downstream tasks.
María Alejandra Bravo, Sudhanshu Mittal, Simon Ging, and Thomas Brox. 2022. Open-vocabulary attribute detection. CoRR, (ttp//211..19911
Keyan Chen, Xiaolong Jiang, Yao Hu, Xu Tang, Yan Gao, Jianqi Chen, and Weidi Xie. 2023a. Ovarnet: Towards open- veee-voocabllryy object attribute recognition. CoRR, 0bs/2301.095500
Yangyi Chen, Karan Sikka. Michael Cogswell, Heng Ji, and Ajay Divakaran. 2023b. Measuring and improv- ing chain-of-thought reasoming in vision-language models. arXiv preprint arXiv:2309, 04461..
Acknowledgement
We thank the anonymous reviewers for their sugges- tions and comments. This research is based upon work supported by U.S. DARPA ECOLE Program No. HR001 12390060 and U.S. DARPA KAIROS Program No. 9A8750-19-2-1004 The views and conclusions contained herein those of the au- thors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
Junhyeong Cho, Youngseok Yoon, and Suha Kwak. 2022. Collaborative transformers for grounded situa- tion recognition. In IEEE/CVF Conference Com- puter Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 19627-19636. IEEE.
Thilini Cooray, Ngai-Man Cheung, and Wei Lu. 2020. Attention- based context aware reasoning for situa- tion recognition. In 2020 IEEE/CVF Conference Computer Vision and Puttern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pages 4735 4744. Computer Vision Foundation IEEE.
Paria Davoodi, Mehdi Ezoji, and Naser Sadeghnejad. 2023. Classification of natural images inspired by the human visual system. Neurocomputing, 518:60-69.
References
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: visual question an- swering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425-222222222 2233. IEEE Com- puter Society.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEEE conference computer vision and patlernn recognition, pages 248-28 255. Icee.
Shachi Deshpande, Kaiwen Wang, Dhruv Sreenivas, Zheng Li, and Volodymyr Kuleshov. 2022. Deep multi-modal structural equations for causal effect estimation with unstructured proxies. Advances Neural Information Processing Systems, 35:10931- 10944.
Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. Framenet: A knowledge base for natural lan- guage processing. In Proceedings of the 17th inter- national conference on Computational linguistics, pages 86-90. Association for Computational Linguis- tics.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang. Shuohang Wang, Lijuan Wang. Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng. 2022. An empirical study of training end- end-to-end vision-an aad--and--aneee lan-uane transform- ers. In IEEF/CYF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 18145-18155 IEEE
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009, Curriculum learning. In Proceedings of the 26th annual international confer- ence on machine learning, pages 41-48.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà, 2020. Continual lifelong

---

Yi Fung, Christopher Thomas, Revanth Gangi Reddy, Sandeep Polisetty, Heng Ji, Shih-Fu Chang, Kathleen McKeown, Mohit Bansal, and Avi Sil. 2021. Infos- urgeon: Cross- media fine-grained information con- sistency checking for fake news detection. In Proc. The Joint Conference of the 59th Annual Meeting of the Association for Computational L Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-ICNLPP 2021).
Lu Jiang, Deyu Meng. Teruko Mitamura, and Alexan- der G. Hauptmann. 2014b. Easy Samples First: Self-paced Reranking for Zero-Example Multimedia Search. In Proceedings of the 22nd ACM Interna- tional Conference on Multimedia, pages 547-556, Orlando Florida USA. ACM.
Amita Kamath, Christopher Clark, Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi. 2022. Webly supervised concept expansion for gen- eral purpose vision models. In Computer Vision ECCV 2022 17th European Conference, Tel Aviy, Is- rael, October 23-27, 2022, Proceedings, Part XXXVI, volume 13696 of Lecture Notes in Computer Science, pages 662-681. Springer.
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, and Jianfeng Gao. 2022. Vision-language pre- training: Basics, recent advances, and future trends. Found. Trends Comput. Graph. Vis., 14(3-4):163- 352.
Alex Graves, Mare G. Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. 2017. Auto- mated Curriculum Learning for Neural Networks. arXiv:1704.0300 [cs].
Andrej Karpathy and Li Fei-Fei. 2015. Deep visual- semantic alignments for generating image descrip- tions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3128-3137,
Tanmay Gupta and Aniruddha Kembhavi. 2023. Vi- sual programming: Compositional visual reasoning without training. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recog- nition, pages 14953-14962.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehti- nen. 2018 Progressive growing of gans for improved quality, stability, and variation. In Proceedings of the 6th International Conference on Learning Represen- tations.
Lisa Anne Hendricks and Aida Nematzadeh. 2021. Probing image-language transformers for verb un- derstanding. Findings of the Association for Com- putational Linguistics: ACI-IJCNLP 2021, pages 3635-3644, Online. Association for Computational Linguistics.
Wonjac Kim, Bokyung Son, and Ildoo Kim. 2021. Vilt: Vision-and laiion-an--laguuge transformer without convolu- tion or region supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5583-5594, PMI.R
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang. Bei Liu, Dongmei Fu, and Jianlong Fu. 2021. Seeing out of the box: End-to-end pre-training for vision- language representation learning. IEEE Confer- ence on Computer Vision and Puttern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 12976- 12985. Computer Vision Foundation IEEE.
Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jer- nite, Margaret Mitchell, Sean Hughes, Thomas Wolf, et al. 2022. The stack: 3 th of permissively licensed source code, arXiv preprint arXiv:2211.1555:
Zhicheng Huang, Zhaoyang Zeng. Bei Liu, Dongmei Fu, and Jianlong Fu. 2020. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. CORR, Abs/2004400849.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-EI-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings. Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng. Dhyanesh Narayanan, and Kevin Murphy. 2017. Openimages: A public dataset for large-scale multi-label and multi- class image classification. Dataset available from https://github.co htt.hhab..oooooeiiaaee...
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc V. Lc. Yun-Hsuan Sung, Zhen Li, and Tom Duerig. 2021. Scaling up vi- sual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th In- ternational Conference Machine Learinng, ICML 2021, 18-24 July 2021, Virtual Event. volume 139 of Proceedings of Machine Learning Research, pages PMLR.
Ranjay Krishna, Vincent S. Chen, Paroma Varma, Michael S. Bernstein, Christopher Ré, and Li Fei- Fei. 2019. Scene graph prediction with limited la- bels. In 2019 IEEE/CVF International Conference Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 2580-2590. IEEE.
Lu Jiang, Deyu Meng, Teruko Mitamura, and Alexan- der Hauptmann. 2014a. Easy samples first: Self- paced reranking for zero-example multimedia search. In Proceedings of the ACM International Conference on Multimedia, MM '1, Orlando, FL. USA, Novem- ber 03 133-77 07, 2014, pages 547-556. ACM.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma,

---

Bernstein, and Li Fei-Fei. 2016. Vi- sual genome: Connecting language and vision us- ing crowdsourced dense image annotations. CoRR, abs/1602.07332.
Michael S.
Human Language Technologies. NAACL-HLT 2021, Online, June 6-11, 2021, pages tion for Computational Linguistics.
Associa-
Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. 2022b. Clip-event: Connecting text and images with event structures. In Proc. Conference on Computer Vision and Pattern Recognition (CVPR2022).
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. 2017. Visual genome: Connecting language and vi- sion using crowdsoureed dense image annotations. International journal of computer vision, 123(1):32- 73.
Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. 2022c. Clip-cvent: Connecting text and images with event structures. In Proceedings of the IEFF/CVE Conference on Com- puter Vision and Pattern Recognition, pages 16420- 16429.
M. Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-Paced Learning for Latent Variable Models. In Advances in Neural Information Processing Systems, volume 23. Curran Associates, Inc.
Gen Li. Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020a. Unicoder-vl: A universal en- coder for vision and language by cross-modal pre- training. In The Thirty- Foorthh AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirly-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 11336- 1886--1344, 11344, AAAI Press.
Manling Li, Alireza Zareian, Qi Zeng. Spencer White- head, Di Lu, Heng Ji, and Shih-Fu Chang. 2020b. Cross-mede media structured common space for multime- dia event extraction. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 2557-2568. Association for Computational Linguistics.
Ruiyu Li, Makarand Tapaswi, Renjie Liao, Jiaya Jia, Raquel Urtasun, and Sanja Fidler. 2017. Situation recognition with graph neural networks. In IEFE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 4183- 4192. IEEE Computer Society.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. CoRR, htbs/3001.1299.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H.
Hoi. 2022a. BLIP: bootstrapping language-image pre-training for unified vision-language understand- ing and generation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Bul- timore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages PMLR.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020c. Oscar: Object-semantics aligned pre- training for vision-language tasks. In Computer Vi- sion ECCV 2020 16th European Conference, Glas- gow; UK, August 23-28, 2020, Proceedings, Part XXX, volume 12375 of Lecture Notes in Computer Science, pages 121- Springer.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare. Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 202 la. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 119694-9705.
A. S. Lillard, M. J. Heise, E. M. Richey, X. Tong, A. Hart, and P. M. Bray. 2017. Montessori preschool elevates and equalizes child outcomes: A longitudi- nal study. In Frontiers in Psychology.
Liunian Harold Li, Mark Yatskar, Da Yin. Cho-Jui Hsich, and Kai- wei Chang. 2019a. Visualbert: A simple and performant baseline for vision and lan- guage. CORR, abs/1908.03557.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. In Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V. volume 8693 of Lecture Notes in Computer Science, pages 740-755. Springer.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsich, and Kai-Wei Chang. 2019h. Visualbert: simple and performant baseline for vision and lan- guage. arXiv preprint arXiv: 190:::398.07
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision- and language tasks. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8- 14, 2019, Vancouver, BC, Canada, pages 13-23.
Liunian Harold Li. Haoxuan You, Zhecan Wang, Alireza Zareian. Shih-Fu Chang, and Kai- Wei Chang, 2021b. Unsupervised vision-and- lannauage pre-training with- out parallel images and captions. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics:

---

Grace Luo, Trevor Darrell, and Anna Rohrbach. 2021. NewsCLIPpings: Automatic Generation of Out-of- Context Multimodal Media. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6801-6817, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Me Manor. 2021. Imagenet-21k pretraining for the masses.
Arka Sadhu, Tanmay Gupta, Mark Yatskar, Ram Neva- tia, and Aniruddha Kembhavi. 2021. Visual semantic role labeling for video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5589-5600.
Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang. and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 1384-1403, Association for Computational Linguistics.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 2019. Objects365: dataset for object detection. In 2019 JEEE/CVF In- ternational Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 8429-8438. IEEE.
large-scale, high-quality
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38339444--4
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic im- age captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 2556-2565.
Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg. 2011. Im2text: Describing images using 1 million captioned photographs. In Advances in Neural In- formation Procexsing Systems 24: 25th Annual Con- ference on Neural Information Processing Systems 2011. Proceedings of meeting held 12-14 December 2011, Granada, Spain, pages 1143-1151.
Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sehe. 2022. Curriculum learning: A survey. Int. J. Comput. Vis., 130(6):1526 1565.
Charulata Patil and Aditya Abhyankar. 2023. Generat- ing comprehensive scene graphs with integrated mul- tiple attribute detection. Mach. Vis. Appl., 34(1):11.
Valentin I. Spitkovsky, Hiyan Alshawi, and Dan Ju- rafsky. 2010a. Baby steps: How "less is more" in unsupervised dependency parsing. In Human Lan- guage Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguisties.
Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen. Quan Tran, and Abhinav Shrivastava. 2022. Improving closed and open-vocabulary attribute pre- diction using transformers. In Computer Vision ECCV 2022 17th European Conference. Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXV, volume 13685 of Lecture Notes in Computer Science, pages 201-219. Springer.
Valentin I. Spitkovsky, Hiyan Alshawi, and Daniel Ju- rafsky. 2010b. From baby steps to leapfrog: How "less more" in unsupervised dependency parsing. In Human Language Technologies: Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, June 2-4, 2010, Los Angeles, California, USA, pages 751-759. The Association for Computational Linguisties.
Emmanouil Antonios Platanios, Otilia Streteu, Graham Neubig. Barnabas Poezos, and Tom Mitchell. 2019. Competence-ben based curriculum learning for neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pages 1162-1177 1172, Minneapolis, Minnesota. Association for Computational Linguistics.
Yixuan Su, Deng Cai, Qingyu Zhou, 7ibo Lin, Simon Baker, Yunbo Cao, Shuming Shi, Nigel Collier, and Yan Wang. 2021. Dialogue response selection with hierarchical curriculum learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1740-1751, Online. Association for Computational Linguistics.
Sarah M. Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. 2020. Grounded situation recognition. In Computer Vision ECCV 2020- 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV, volume 12349 of Lecture Notes in Computer Science, pages 314-332. Springer.
Mohammed Suhail and Leonid Sigal. 2019. Mixture- kernel graph attention network for situation recogni- tion. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pages 10362-10371. IEEE
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- ur, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763, PMLR.
Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiw:2303.08728.

---

Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross- modsattodllyyy encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNN 5100-5111, Hong Kong, China. Association for Com- putational Linguistics.
Heng Ji. 2021. Resin: A dockerlized schema- schema-guided cross docsmmt cross- pooss-lnguaal cross-media informa- tion extraction and event tracking system. In Proc. The 2021 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies (NAACI-HITT2221) Demo Track.
pages
Spencer Whitehead, Hui Wu, Heng Ji, Rogerio Feris, and Kate Saenko. 2021. Separating skills and con- cepts for novel visual question answering. In Proc. Conference on Computer Vision and Pattern Recog- nition (CVPR2021).
Shagun Uppal, Sarthak Bhagat, Devamanyu Hazarika, Navonil Majumder, Soujanya Poria, Roger Zimmer- mann, and Amir Zadeh. 2022. Multimodal rescarch in vision and language: A review of current and emerging trends. Inf. Fusion, 77:149-171.
Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Song- fang Huang. Wenming Xiao, and Fei Huang. 2021. E2E.VLLP end-to-end vision-language pre-training enhanced by visual learning, In Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/LICNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 503-513, Association for Computational Linguistics.
Junwen Wang, Katayoun Farrahi, and Mahesan Niran- jan. 2020. Curriculum based cascade learning for image classification.
Liyuan Wang. Xingxing Zhang, Hang Su. and Jun Zhu. 2023a. A comprehensive survey of continual learning: Theory, method and application. CoRR, abs/2302.00487.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022a. OFA: unifying ar- chitectures, tasks, and modalities through a simple sereencceto-squenne learning framework. In Inter- national Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA. volume 162 of Proceedings of Machine Learning Research, pages 23318-23340. PMLR.
Zhengyuan Yang. Zhe Gan, Jianfeng Wang. Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022. Unitab: Unifying text and box outputs for grounded lision-language modeling. In Com- puter Vision - ECCV 2022 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Proceed- ings, Part XXXVI, volume 13696 of Lecture Notes in Computer Science, pages 521-539. Springer.
Xin Wang, Yudong Chen, and Wenwu Zhu. 2022b. survey on curriculum learning. IEEE Trans. Pattern Anal. Mach. Intell., 44(9):455554455 4576.
Yuan Yao, Qianyu Chen, Ao Zhang, Wei Ji. Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. 2022. PEVL: position-enhanced pre-training and prompt tuning for vision-language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 11104- 11117. Association for Computational Linguistics.
Xingyao Wang. Sha Li, and Heng Ji. 2023b. Code4struct: Code generation for few-shot event structure prediction. In Proc. The 61 Annual Meet- ing of the Association for Computational Linguistics (ACI22023).
Zhenhailong Wang. Manling Li, Ruochen Xu, Lu- owei Zhou, Jie Lei, Xudong Lin, Shuohang Wang, Ziyi Yang, Chenguang Zhu, Derek Hoiem. Shih-Fu Chang, Mohit Bansal, and Heng Ji. 2022c. Language models with image descriptors are strong few-shot video-language learners. In Proc. The Thirty- Sixth Annual Conference on Neural Information Process- ing Systems (NeurIPS2022).
Yuan Yao, Ao Zhang, Xu Han, Mengdi Li, Comeeliss Weber. Zhiyuan Liu, Stefan Wermter, and Maosong Sun. 2021. Visual distant supervision for scene graph generation. In 2021 IEEF//CV International Confer- ence on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pages 15796-15806. IEEE.
Mark Yatskar, Vicente Ordonez, Luke Zettlemayer, and Ali Farhadi. 2017. Commonly uncommon: Semantic sparsity in situation recognition. In 2017 IEEE Con- ference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA. July 21-26, 2017, pages 6335-6344. IEEE Computer Society.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yu- lia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak super- vision, arXiv preprint arXiv:2 10904
Haoyang Wen, Ying Lin, Tuan M. Lai, Xiaoman Pan, Sha Li, Xudong Lin, Ben Zhou, Manling Li, Haoyu Wang, Hongming Zhang, Xiaodong Yu. Alexan- der Dong, Zhenhailong Wang, Yi R. Fung, Piyush Mishra, Qing Lyu, Didac Surís, Brian Chen, Susan W. Brown, Martha Palmer, Chris Callison- Burch, Carl Vondrick, Jiawei Han, Dan Roth, Shih-Fu Chang, and
Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 5534-5542, IEEE Computer Society.

---

Chen Yu and Dana H. Ballard. 2004. On the integra- tion of grounding language and learning objects. In Proceedings of the Nineteenth National Conference on Artificial Intelligence, Sixteenth Conference on In- novative Applications of Artificial Intelligence, July 25-29, 2004, San Jose, California, USA, pages 488- 494. AAAT Press The MIT Press.
Pengfei Yu. Heng Shih-Fu Chang, and Kevin Duh. 2022. Multimedia curriculum learning for language acquisition. In Journal of Science and Technology on Information and Communications.
Tianyu Yu, Yangning Jiaoyan Chen, Yinghui Li, Hai- Tao Zheng, Xi Chen, Qingbin Liu, Wenqiang Liu, Dongxiao Huang, Bei Wu, and Yexin Wang. 2023. phowledge-augmentt few-shot visual relation de- tection. CoRR, abs/2303.05342.
Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi R Fung. Hao Peng, and Heng Ji. 2023. Craft: Customiz- ing lims by creating and retrieving from specialized toolsets. arXiv preprint arXiv:2309. /7428.
Mert Yüksekgönül, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022. When and why vision-language models behave like bags-of-words, and what to do about it? CoRR, abs/2210.01936.
Rowan Zellers, Mark Yatskar. Sam Thomson, and Yejin Choi. 2018. Neural motifs: Scene graph parsing with global context. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 5831-5840. Computer Vision Foundation IEEE Computer Society.
Tiancheng Zhao, Tianqi Zhang, Mingwei Zhu, Haozhan Shen, Kyusong Lee, Xiaopeng Lu, and Jianwei Yin. 2022. Vl-checklist: Evaluating pre-trained vision- language models with objects, attributes and relations. CoRR, htbs/200..002211
Mingyang Zhou, Yi Fung, Long Chen, Christopher Thomas, Heng Ji, and Shih-Fu Chang. 2023. En- hanced chart understanding via visual language pre- training plot table pairs. In Findings of the As- sociation for Computational Linguistics: ACL 2023, pages 1314-13 11146226, Toronto, Canada. Association for Computational Linguistics.

---

A Adaptation of Downstream Tasks
input the mask prompt with annotated location
to ViStruct:
All evaluations are conducted within the supervised learning paradigm, involving training models using the training dataset, and selecting superior models as dictated by key performance indicators in the validation dataset. The ultimate performance of these models evaluated on the testing dataset. We enumerate the specifics of adapting to downstream tasks as detailed below.
Generate Concepts
ob jecti <mask>
location=
[bounding_box],
attribute-None
ViStruct then trained to predict the ground truth object categories, for example, the "horse". The visual relation detection task is trained in the same vain as in Sec. A.I.
A.1 Visual Relation Detection
In this task, the models are provided with ground- truth annotations of the object bounding boxes and their respective categories. The requirement is for these models effectively predict the visual inter- relations between each pair of objects in the images. our implementation, each object pair is supplied with their respective annotated bounding boxes, in conjunction with mask prompt as input parame- ters for the ViStruct. ViStruct is then tasked with predicting the visual relation between the specified objects. Assuming the pair of objects provided are a horse and grass, the imputs introduced ViStruct would as follows:
A.3 Situation Recognition
In this task, the models are required to predict the visual event (a.k.a, the activity) in the image with corresponding participants. We treat it as two different tasks, namely the visual event prediction and the event role prediction. For the visual event prediction, we input the prompt as:
Generate Visual Event with Arguments
ViStruct is trained to predict the visual event, for example, "eat". For the event role prediction task, the models are provided with the ground-trute predictive visual event with mask prompt as input:
Generate Concepts
object1 horse(
(ocation=(bounding__oo attribute=None
Generate Visual Event with Arguments s eat (ingestor=<mask)
ViStruct is then trained to predict the object for the argument "ingestor". Specifically, in this task, perform an automatic mapping to convert each potential visual event and object to a single token in the OFA model for training and evaluation. The automatic mapping is based on the word embed- ding of the OFA model. We select the most similar token, measured by the cosine similarity, with the target verbalizers of visual events and objects.
object2 = (
(bouddigg attribute=None
I Generate Relations
<mask> (sub=objectl, obj=object2)
Correspondingly, ViStruct is trained to predict the ground truth relation: "walk_on" as output. We meticulously select and refine appropriate verbal- izers for every unique relation to ensure that each relation within the label space can be represented by a single, meaningful token in the OFA model.
B Visual Structure Learning
Several studies have been conducted consecutively on learning multimodal structural knowledge, com- mencing with visual concept recognition as ba- sic task (Deng et al., 2009; Davoodi et al., 2023; Kamath et al., 2022), progressing to object ground- ing (Yu and Ballard, 2004; Shao et al., 2019; Krasin al., 2017), object attribute detection (Chen al., 2023a; Patil and Abhyankar, 2023: Bravo al., 2022), object-object relation detection (Krishna et al., 2016), and finally visual event understand- ing (Yatskar et al., 2016, 2017: 7; Cho et al,, 2022:
A.2 Scene Graph Classification
In this task, the models are provided with the bounding boxes and are required to forecast both the classifications of the objects and their corre- a multi-
sponding visual relations. We treat task learning problem, training ViStruct to perform both the object recognition task and the visual rela- tion detection task. For the object recognition task,

---

Type
Pretraining Task
Source
#Sample
#B_ 1B_Samll
Object Recognition
ImageNei2IK Object365, Openlmages, COCO
1M
104K
Object Grounding
2.5M
1M
Supervised
Attribute Detection
LSA
0.5M
0.1M
Relation Detection
Openlmages, Visual Genome
0.5M
0.1M
Unsupervised
Event Detection
COCO, SBU, CC3M
4M
Table 7: The datasets included in ViStruct Suite. #B 1_Samplle denotes the number of samples in the replay buffer.
E Details of ViStruct Suite
Generate Visual Concepts objectl horse( [po=_p,,_,, pos_2, pos_3, pos_4], bosbbutte=("bbownn [hungry!]
The selected datasets along with their statistics are listed in Table 7. For object recognition, we utilize the pre-processed ImageNet-218 21K (Ridnik et al., 2021). To curate the dataset, we exclude rare classes that consist of less than 500 samples and re- tain total of 10,450 categories, each with sample size of 100. For object grounding, we leverage ob- ject detection datasets, including COCO (Lin al., 2014), OpenImages (Krasin et al., 2017), and Ob- ject365 (Shao et al., 2019), which contain 80, 600, 365 categories respectively. For object attribute de- tection, we adopt the open-vocabulary LSA (Pham et al., 2022) that contains 14.6M attribute annota- tions in total. For visual object relations detection, we adopt the OpenImages (Krasin et al., 2017) and Visual Genome (Krishna al., 2016). We adopt the training split of these labeled datasets to avoid data contamination.
Generate Relations
walkkonnsuub=ole (sb==objjet obj=object2)
Generate Visual Events eat cotre,,stto=hhor,,, ingestiblessggras)
The focusing optimization trick prioritizes the semantic content of generated code (highlighted portion) while disregarding the code's syntactic structure.
Figure 6:
Li al., 2020b, 2022b). In this work, we de- sign curriculum learning pyramid that progres- sively teaches VLMs the aforementioned structural knowledge and assemble a dataset collection that encompasses multi-level structural knowledge.
C Warm-up Pretraining
For visual event understanding, we propose a superiy-upervised method to generate training data from the large-scale image-caption pairs in COCO (Lin et al., 2014), SBU Captions (Ordonez et al.. 2011), and CC3M (Sharma al., 2018). As shown in Figure for each image, we leverage the off-the-shelf semantic role labeling model2 perform semantic parsing on the paired caption. Aligning with ViStruct, we adopt the FrameNet for the event and corresponding argument definitions.
Prior to engaging in multimodal structural knowl- edge learning, warm-up stage is implemented to facilitate the models' acquisition of fundamental code syntax structures and visual concepts with their definitions. To accomplish this, we inte- grate code training data (Kocetkov et al., 2022), code-formmlated class definitions of WordNet and FrameNet synsets, and all datasets within the ViStruet Suite, then subject the models to sin- gle training epoch.
We gather the aforementioned datasets and con- duct a subsequent post-processing procedure. To establish consistency with ViStruct across various datasets, we align all categories from different datasets to the WordNet and FrameNet synsets through the automatic mapping process imple- mented in the NLTK toolkit (Bird et al., 2009). In cases where multiple categories in a dataset are assigned to the same synset, we manually adjust the mapping based on our prior knowledge.
D Focusing Optimization
Through our analysis, we observe that the code out- put contains redundant information regarding the syntax structure, which can be quickly learned dur- ing the warm-up stage. Consequently, we propose to focus solcly on the semantic content of the gen- erated code, meaning that the loss computation will be limited the semantic content exclusively, As depicted in Figure 6, this approach can direct mod- cls to prioritize structural knowledge and accelerate the training process.
Pttpps: / thub. wwt.huu..cm//hoooa cww/ccanin//llccccccccooo. Srame-semant ie- transformer


Reviews:
[{"review_id": "iN85PRiqUA", "reviewer": "Reviewer_cMsY", "summary": "", "questions": "See weaknesses above, all my questions/concerns are included there.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that you like our paper! We have carefully considered your comments and would like to address them below.\n\n# Reason to Reject\n**R1: Relatively low performance increase.**\n\nA: We will correct the incorrect bold highlighting of the results. For the relatively low performance increase, the visual relation detection task (Table 3) may be too simple for evaluating VLMs. Thus, the gap between various methods is relatively small. The scene graph classification (Table 3) is a more challenging task given the performance of existing methods. Although the absolute improvement of ViSutrct seems not significant, the average relative improvement compared to the previous model is 7.55%. We conduct the student t-test to reevaluate the results and show that our approach shows significant improvement compared to the baselines with p<0.1. In addition, the best-performing baseline model in Table 4 (Motif+D) uses extra augmented data for training, while our approach only relies on the original training dataset.\n\n\n**R2: Insufficient comparison with baselines for the visual relation detection and scene graph classification tasks.**\n\nA: Thank you for your feedback and for pointing out the relevant references [1,2,3]. We have diligently reviewed these papers, along with some of the more recent publications related to these tasks [4,5]. \n\nThe primary distinction between the methodologies described in the references and the ones in our experiments pertains to the stages of improvement they target. In our work, we primarily focus on enhancing the vision-language representations in the backbone models using continued pretraining. In contrast, the predominant thrust of the studies [1,2,3,4,5] is geared towards refining the finetuning phase. This includes augmentations in the training data [4], refining training objectives [1,2], and adopting novel training paradigms [3,5].\n\nIt is important to note that the advancements made in the training stages by the recent work [1,2,3,4,5] can be seamlessly integrated with our methodology. Namely, our method and the related work are complimentary. This is based on the premise that our approach augments the visual structural knowledge extraction capability of the backbone model. We will ensure clarity on this matter and provide a more comprehensive discussion regarding related research in our subsequent revision.\n\n\n[1] Graphical Contrastive Losses for Scene Graph Parsing; Ji Zhang, Kevin J. Shih, Ahmed Elgammal, Andrew Tao, Bryan Catanzaro\n\n[2] The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation; Lin Li, Long Chen , Yifeng Huang, Zhimeng Zhang, Songyang Zhang, Jun Xiao\n\n[3] RelTR: Relation Transformer for Scene Graph Generation; Yuren Cong, Michael Ying Yang, Bodo Rosenhahn \n\n[4] Fine-Grained Scene Graph Generation with Data Transfer; Ao Zhang, Yuan Yao, Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, Tat-Seng Chua\n\n[5] Panoptic Scene Graph Generation Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, Ziwei Liu\n\n\n**R3: Limited datasets used. This work uses visual genome for benchmarking the VRD and SGC tasks. Prior methods routinely used datasets like OpenImages.**\n\nA: Thanks for pointing it out! Actually, the OpenImages dataset is included in ViStruct Suite, which is used for pretraining. So, we exclude this dataset for evaluation to prevent the data leakage. For the visual genome considered in our work, we spent great efforts on excluding examples that appear in the evaluation set for fair comparison. \n\n\n**R4: Generalizability to different pre-trained models**\n\nA: We agree that evaluating our method based on various VLM backbones can make the work more solid. But we only have limited computing resources in the academic lab. In our work, the pretraining takes around 1 month, including some trials and errors. So it would be very difficult for us to conduct experiments using 2 or 3 VLM backbones. \n\n**R5: More details are required to claim performance improvements. for eg. is the current ordering of curriculum necessary? are same computation resources/epochs/data size used for the ablation which compares curriculum based learning with non curriculum based learning?**\n\nA: We will add those details in the next revision. For the design of the curriculum, we do not try enumerating all diverse orders to finalize the curriculum in our work. The design is based on two factors: (1) Empirical evidence drawn from human learning [1,2]; (2) A certain amount of trial and error in the preliminary experiments. We do not present the experimental results of (2) in our paper because we can not draw reliable conclusions from the obtained results due to the variance. For example, we have tried: (a) first performing the object relation detection task before the concept recognition task; (b) reversing the curriculum order due to the findings that reverse curriculum (from hard to easy) may contribute to the improvement. (Considering the best performance we observed in these two designs) Both of them achieve worse performance compared to directly training on the mixture of datasets (ViStruct-Mix) without curriculum guidance. We will discuss these in the next revision. For a fair comparison of our approach and ablations, we utilize the same datasets and same epochs for training. \n\n[1] Curriculum learning; Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston\n\n[2] Progressive growing of gans for improved quality, stability, and variation; Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.\n\n\n\n**R6: Applications of visual structure**\n\nA: Thanks for your insightful and interesting suggestions! In this work, we show that our approach can improve the ability of VLM to perform visual structural knowledge extraction. In addition, we are currently working on exploiting the concrete applications of ViStruct. As you suggested, we also consider the open-domain visual relationship extraction. Besides, we also consider mixing the visual structural knowledge learning in ViStruct with the typical end-to-end pretraining to further leverage the web-scale data to improve performance.\n\n\n# Missing References:\nThanks for pointing them out! We will add these citations in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel idea: Representing visual knowledge as code is a new idea as per my knowledge, including masked code prediction, and curriculum learning. \n1. Scalable w.r.t new/larger/better models: I believe as VLMs improve, this technique should also improve.\n1. The ViStruct Suite might be useful for benchmarking/training of future visual structure extraction models.\n1. Evaluation is performed on multiple tasks", "reasons_to_reject": "1. Relatively low performance increase:  Although Table 3 and Table 4 show best results are achieved by Vistruct, the numerical values do not seem to be consistent with the value which are made bold in all columns (some non bold numbers are the highest it seems?). I would request the authors to correct this. Also see my next point regarding baselines considered.\n1. Insufficient comparison with baselines: There are numerous methods for scene graph generation/relation prediction which do not seem to be included in Table3 and Table 4. They are for example [1,2,3] and it is possibly many other scene graph generation methods (also see this somewhat old repository for baselines https://github.com/microsoft/scene_graph_benchmark). In light of these advances in SGG, the utility of the current method is questionable. Comparison with all are not necessary, but some latest/recent baselines should be included.\n1. Limited datasets used: This work uses visual genome for benchmarking the VRD and SGC tasks. Prior methods routinely used datasets like OpenImages. Benchmarking on an extra dataset can improve this work. \n1. Generalizability to different pre-trained models: It seems necessary to study the effect of changing pre-trained models. Currently, table 3 baselines and OFA based model of this paper is not really comparable I believe. I do not have a direct solution to this, but having more pre-trained backbones can improve this work and give the reader more clarity of the performance of the model w.r.t pretrained backbone.\n1. Curriculum learning: more details are required to claim performance improvements. for eg. is the current ordering of curriculum necessary? are same computation resources/epochs/data size used for the ablation which compares curriculum based learning with non curriculum based learning?\n1. Applications of visual structure: Although there are many papers that tackle the problem of visual structure/scene graph generation (and probably some are better than this method as per the previous point), the utility of generating a scene graph alone is questionable. Instead, applications of using scene graphs might be more interesting to improve this work, for eg. this work cites Yüksekgönül et al., 2022 which shows that VLMs have a bag of words nature. Can the current method extract visual relation ships which can then be used to make progress on ARO task provided by Yüksekgönül et al. Joint training for image-text matching and code completion as done in this work, is a possible direction. This point is largely a suggestion and not exactly a reason to reject.\n\nReferences\n1. Graphical Contrastive Losses for Scene Graph Generation https://arxiv.org/abs/1903.02728\n1. RelTR: Relation Transformer for Scene Graph Generation: https://arxiv.org/abs/2201.11460v3\n1. Noisy label correction for robust scene graph generation https://openaccess.thecvf.com/content/CVPR2022/papers/Li_The_Devil_Is_in_the_Labels_Noisy_Label_Correction_for_CVPR_2022_paper.pdf", "questions_to_authors": "See weaknesses above, all my questions/concerns are included there.", "paper_topic_and_main_contributions": "This paper primarily targets the visual structure prediction/extraction - like tasks where, given an image, the model should extract concepts, objects, their attributes, relations etc. The paper uses “code” for representing visual structure. The paper can be categorized in the line of work that has shown that using “code” for representing structural knowledge can prove to be very useful for large language models (especially in the few shot prompting setting). Even though prior work that uses code representations for natural language (common sense reasoning, event extraction) is an inspiration for this work, there are significant differences for eg. 1. This work targets the multimodal setting which is significantly more challenging and 2. Unlike previous works that use code representations, this work does not use prompting, since prompting VLMs might not be as good as prompting of LLMs. Training techniques (curriculum learning) are designed in this paper for generating visual structure in the form of code.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Some more baselines can be considered for table 3 and 4. for eg:\n1. Graphical Contrastive Losses for Scene Graph Generation https://arxiv.org/abs/1903.02728\n1. RelTR: Relation Transformer for Scene Graph Generation: https://arxiv.org/abs/2201.11460v3\n1. Noisy label correction for robust scene graph generation https://openaccess.thecvf.com/content/CVPR2022/papers/Li_The_Devil_Is_in_the_Labels_Noisy_Label_Correction_for_CVPR_2022_paper.pdf\n\nMore baselines can be found from citations of the above papers.(there should be many in this line of work)"}}, {"review_id": "sQkTaUMrL5", "reviewer": "Reviewer_CE6x", "summary": "", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback and comments. Our responses are as follows. \n\n# Reason to Reject\n**R1: Only one backbone (OFA-base) is evaluated and the model size is too small**\n\nA: We agree that evaluating our method based on various VLM backbones can make the work more solid. But we only have limited computing resources in the academic lab. In our work, the pretraining takes around 1 month, including some trials and errors. So it would be very difficult for us to conduct experiments using 2 or 3 VLM backbones. \nIn addition, we discuss the model scale in the limitation section of our paper and will do try implementing our method on a more powerful model, like BLIP-2, in future work.  \n\n**R2: Too few previous works are compared in Tab1 and Tab2**\n\nA: Thank you for your feedback. We have diligently reviewed relevant and latest papers related to these tasks [1,2,3,4,5]. \n\nThe primary distinction between the methodologies described in the references and the ones in our experiments pertains to the stages of improvement they target. In our work, we primarily focus on enhancing the vision-language representations in the backbone models. In contrast, the predominant thrust of the studies [1,2,3,4,5] is geared towards refining the training phase. This includes augmentations in the training data [4], refining training objectives [1,2], and adopting novel training paradigms [3,5].\n\nIt is important to note that the advancements made in the training stages by the recent work [1,2,3,4,5] can be seamlessly integrated with our methodology. This is based on the premise that our approach augments the visual structural knowledge extraction capability of the backbone model. We will ensure clarity on this matter and provide a more comprehensive discussion regarding related research in our subsequent revision.\n\n[1] Graphical Contrastive Losses for Scene Graph Parsing; Ji Zhang, Kevin J. Shih, Ahmed Elgammal, Andrew Tao, Bryan Catanzaro\n\n[2] The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation; Lin Li, Long Chen , Yifeng Huang, Zhimeng Zhang, Songyang Zhang, Jun Xiao\n\n[3] RelTR: Relation Transformer for Scene Graph Generation; Yuren Cong, Michael Ying Yang, Bodo Rosenhahn \n\n[4] Fine-Grained Scene Graph Generation with Data Transfer; Ao Zhang, Yuan Yao, Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, Tat-Seng Chua\n\n[5] Panoptic Scene Graph Generation Jingkang Yang, Yi Zhe Ang, Zujin Guo, Kaiyang Zhou, Wayne Zhang, Ziwei Liu\n\n\n\n\n**R3: Evaluation of grounding and attribute prediction**\n\nA: In the author response period, we conduct experiments on grounding due to the time limits. We compare our approach with task-specific models (MDETR [1]) and the ablation that directly using text representation. We evaluate the methods on RefCOCO and RefCOCO+, and report the Acc@0.5 results on the validation split. The results are shown below. We observe that ViStruct can still outperform baselines in the grounding task. \n\n| Method/Dataset | RefCOCO | RefCOCO+ |\n| :-----| :----: | :----: |\n| MDETR | 86.75 | 79.52 |\n| ViStruct-Text | 87.89 | 80.94 |\n| ViStruct | 88.57 | 82.07 |\n\n[1] Mdetr - modulated detection for end-to-end multi-modal understanding; Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, Nicolas Carion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. I like the idea of structuralizing the vision information (concept, attribute, location, relation, etc) into codes. Compared with text, codes are more organized and more efficient to express high-level logic. \n2. The curriculum learning framework propagates from the basic concept level up to the event level according to the complexity, which is intuitively reasonable. Although almost no new methods are proposed, combining previous methods to adapt to the new problem is still meaningful.", "reasons_to_reject": "1. Main concern: Only one backbone (OFA-base) is evaluated and the model size is too small. At least two or three different pre-trained VLM backbones are required to demonstrate the generalizability of the proposed method as it's a plug-in method on top of pre-trained VLMs. Also, the model scale is too small (just base). I wonder if the conclusion that curriculum learning helps can still hold with larger pre-trained VLM, i.e., more pre-trained natural text knowledge injected.\n2. Too few previous works are compared in Tab1 and Tab2 and they are too outdated.\n3. I didn't see any evaluation on grounding and attribute prediction. As they are also one of the curriculum levels, it's necessary to evaluate them.", "questions_to_authors": "See weaknesses.", "paper_topic_and_main_contributions": "This paper proposed ViStruct, a novel curriculum framework for training vision-language models (VLMs) with code representation of visual structural knowledge. It leverages the inherent structure of programming languages and converts visual information to Python-style code which ranges from basic concepts to complex events. Additionally, they employ curriculum-based learning, progressively teaching VLMs to understand varying visual structures, they also collect a new dataset for pre-training. The experimental result shows that the proposed method can output baselines and many other previous works.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BlR9jfm4BU", "reviewer": "Reviewer_4qDb", "summary": "", "questions": "I think this paper is very interesting and exciting. I have no more question besides the one mentioned in the reasons to reject.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are excited that you like our paper! We thank you for your valuable feedback and suggestions.\n\n1. **Inference Stage and Section A from Appendix**:\nWe understand the importance of clarity in explaining the inference stage, especially given its critical role in the methodology. In response to your suggestion, we will move Section A from the appendix to the main text. This will allow readers to have a seamless understanding of the mask prediction process without the need to refer to the appendix.\n\n2. **Implementation Details**:\nWe apologize for any oversight regarding the provision of implementation details. To ensure reproducibility, we will enhance our paper by adding a more comprehensive section. \n\n3. **Clarification on Generated Program Language**:\nThank you for pointing out this potential point of confusion. We will make it explicit in the main text that the generated program language is not intended for direct execution in typical python environments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of encoding an image using programming language is quite interesting.\n\nThis programming language based approach could potentially further achieve improved performance using code based LLM.\n\nThe paper converts the visual relation prediction task into a masked language prediction problem. This is very interesting.", "reasons_to_reject": "I would recommend either add a section describing the inference stage or move the section A from appendix to main text. For several rounds of reading the main text, I couldn't figure out how to do the inference. After reading the appendix, I realized it is doing mask prediction.\n\nI hope the author could provide more implementation details for how to reproducing the results in either main text or appendix. I found it might be a little bit hard to find those information.\n\nI think the author might also want to mention that the program language (generated) is not intended for directly running from some python environment. In the first round of reading, I misunderstood that the generated code is intended for directly running.", "questions_to_authors": "I think this paper is very interesting and exciting. I have no more question besides the one mentioned in the reasons to reject.", "paper_topic_and_main_contributions": "This paper studies visual structural knowledge extraction. Concretely, given an image with questions about the visual relations between two objects (or multiple objects), the model expected to provide the correct answer. To study this problem, the paper proposed a very interesting approach that encodes the image into a programming language and leveraged this language for visual relation understanding. The approach could also leverage the pre-trained code based LLM to further improve the performance. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "A4jdslRJo0", "reviewer": "Reviewer_YaB4", "summary": "", "questions": "- Question A: To validate the effectiveness of the code-vision representation, the experimental setting of ViStruct-Text would be clear. Does the warm-up pretraining (depicted in Appendix C) is also utilized for ViStruct-Text?\n- Question B: For in-depth analysis of curriculum learning, additional experiments would be needed. Have you conducted experiments under the mixed order of curriculum? About the replay buffer, please report the experimental results without the replay buffer. How did you set the size of the replay buffer of each curriculum? Is the random sampling strategy enough to deal with the infrequent and rare concept (L373)?\n- Question C: The authors claim the adopted curriculum prevents the model from overfitting on the dataset of visual concepts (L506-510). Could you explain the connection between the experimental results and the overfitting in detail?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our paper and for providing invaluable feedback. We sincerely appreciate the insights and recommendations you've provided.\n\n# Reason to Reject\n**R1: Lacks an in-depth ablation study on the proposed method. For example, the additional experiments on the diverse order of curriculum and the design of a replay buffer help to convince the effectiveness of the proposed method.**\n\nA: Thanks for pointing it out! For the design of the curriculum, we do not try enumerating all diverse orders to finalize the curriculum in our work. The design is based on two factors: (1) Empirical evidence drawn from human learning; (2) A certain amount of trail and error in the preliminary experiments. We do not present the experimental results of (2) in our paper because we can not draw reliable conclusions from the obtained results due to the variance. For example, we have tried: (a) first performing the object relation detection task before the concept recognition task; (b) reversing the curriculum order due to the findings that reverse curriculum (from hard to easy) may contribute to the improvement. (Considering the best performance we observed in these two designs) Both of them achieve worse performance compared to directly training on the mixture of datasets (ViStruct-Mix) without curriculum guidance. We will discuss these in the next revision. \n\nIn reference to the replay buffer's design, our initial approach did not include this strategy. Consequently, for the average performance, we observed results of 69.42% for visual relation detection and 29.87% for scene graph classification tasks. Notably, these outcomes were even less favorable than those from our other two ablations. It was these findings that motivated us to incorporate the replay buffer into our methodology. We will discuss these in the next revision.\n\n\n\n\n\n**R2: A detailed analysis of the curated dataset would be needed.**\n\nA: Thanks for pointing it out! Yes, we acknowledge that the curated visual event detection dataset does contain noise because the off-the-shelf semantic role labeling system is not perfect. Our strategy is increasing the dataset size of visual event detection to mitigate the noise, which aligns with the common strategy adopted in pretraining on web-scale datasets [1,2]. Specifically, we utilize a mix of COCO, SBU, and CC3M that contains around 3M examples to create our visual event detection dataset. In the author response period, we conduct a human evaluation of our automatically generated visual event detection dataset. We sample 100 samples from the dataset, evaluating the accuracy of (1) visual event, and (2) corresponding argument roles. For the visual event, we find that 93% of samples are correct, 4% of samples are ambiguous (may contain other correct labels), and only 3% are incorrect. For the argument roles, we find that 87% of samples are correct, 8% of samples overlook some entities mentioned in the captions, and 5% of samples contain wrong extracted arguments. We will include these results in the revision.\n\n\n[1] ALIGN: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision. Chao Jia and Yinfei Yang.\n\n[2] Learning Transferable Visual Models From Natural Language Supervision. Alec Radford, Jong Wook Kim, Chris Hallacy, et al\n\n\n\n\n\n\n\n# Questions\n**QA: Does the warm-up pretraining (depicted in Appendix C) is also utilized for ViStruct-Text?**\n\nA: In our experiments, we have tried this strategy but observed worse results (-0.43 on average) on the evaluated visual relation detection task. So the results of ViStruct-Text we report in our paper are without the warm-up pretraining. \n\n\n**QB: Have you conducted experiments under the mixed order of curriculum? About the replay buffer, please report the experimental results without the replay buffer. How did you set the size of the replay buffer of each curriculum? Is the random sampling strategy enough to deal with the infrequent and rare concept (L373)?**\n\nA: For the mixed order of curriculum and replay buffer, please refer to our response to the first reject reason. We report the size of the replay buffer of each curriculum in Table 7 of our paper. The size is determined to ensure that 100 samples of each concept are maintained in the replay buffer for each dataset (e.g., ImageNet21K, Object365). \n\nSorry for the confusion about the random sampling strategy. We do not employ a purely random sampling approach across the entire dataset. Instead, our sampling strategy is category-specific. That is, for every category, we randomly select a predetermined number of examples to guarantee an equal representation of both rare and common concepts. We will make it clear in our next version.\n\n\n\n\n\n\n**QC: Explain the connection between the experimental results and the overfitting in detail.**\n\nA: In Table 4, we present a comparison between the performance of our proposed method, ViStruct, and its ablation, ViStruct-Mix, which is directly pretrained on the ViStruct Suite mixture. The results clearly indicate that ViStruct surpasses ViStruct-Mix with an average relative improvement of 10.55%. This significant difference underscores the advantage of curriculum training, suggesting its ability to progressively guide the model in learning complex visual structures in the later stages of training. Directly training on the ViStruct Suite mixture poses challenges, especially given that the concept recognition dataset incorporates a multitude of rare and long-tail visual concepts. Such complexities necessitate significant modeling capacity. Without the guided progression of our curriculum, models risk being overwhelmed by these intricacies, which can impede their ability to learn and represent more complex visual structures like visual events. This, we believe, would increase susceptibility to overfitting, especially on rare concepts.\n\n# Typos\nThanks for pointing them out! We will correct them in the next version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper defines a visual structural knowledge pyramid and collects ViStruct Suite including multi-level structural knowledge aligned with the pyramid. This knowledge pyramid and the curated dataset would help to promote future studies.\n- This paper devises an elaborated training framework for VLMs to improve their visual structural knowledge extraction ability.", "reasons_to_reject": "- Even though the curriculum-guided code-vision representation is an interesting idea, this paper lacks an in-depth ablation study on the proposed method. For example, the additional experiments on the diverse order of curriculum and the design of a replay buffer help to convince the effectiveness of the proposed method.\n- In ViStruct Suite, annotations in the visual event detection stage might contain noises since they are obtained by the off-the-shelf semantic role labeling system. A detailed analysis of the curated dataset would be needed.", "questions_to_authors": "- Question A: To validate the effectiveness of the code-vision representation, the experimental setting of ViStruct-Text would be clear. Does the warm-up pretraining (depicted in Appendix C) is also utilized for ViStruct-Text?\n- Question B: For in-depth analysis of curriculum learning, additional experiments would be needed. Have you conducted experiments under the mixed order of curriculum? About the replay buffer, please report the experimental results without the replay buffer. How did you set the size of the replay buffer of each curriculum? Is the random sampling strategy enough to deal with the infrequent and rare concept (L373)?\n- Question C: The authors claim the adopted curriculum prevents the model from overfitting on the dataset of visual concepts (L506-510). Could you explain the connection between the experimental results and the overfitting in detail?", "paper_topic_and_main_contributions": "This paper presents ViStruct, a training framework for Vision-Language Models (VLMs) to learn visual structural knowledge. The proposed framework adopts curriculum learning on code-vision representations for effective multi-granularity visual structural knowledge extraction.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- L413: continued pertraining\n- L463: for pertaining"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers are generally appreciative of the novel idea of representing visual knowledge as code and using curriculum-based learning to guide the vision-language models training process in different complexity stages. The ViStruct Suite is seen as potentially useful for training and benchmarking future visual structure extraction models.

However, one key remaining concern is the limited number and outdated nature of the pre-trained models used for evaluation. The reviewers also suggest further improvements, such as better exposition, detailed analyses of ViStruct Suite and the curriculum learning design.

Given its novelty and potential impact on future work, all reviewers are pretty excited about this work.
Decision: Accept-Main