Adaptive Visual Scene Understanding: Incremental Scene Graph Generation
Authors: Naitik Khandelwal
Xiao Liu
Mengmi Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Scene graph generation
Continual learning
In-context symbolic replay
Long tail distribution
Compositional scene graphs
Primary area: machine_vision
TL;DR: We introduce a continual scene graph generation dataset and a model to tackle this problem.

Abstract: Scene graph generation (SGG) analyzes images to extract meaningful information about objects and their relationships. In the dynamic visual world, it is crucial for AI systems to continuously detect new objects and establish their relationships with existing ones. Recently, numerous studies have focused on continual learning within the domains of object detection and image recognition. However, a limited amount of research focuses on a more challenging continual learning problem in SGG. This increased difficulty arises from the intricate interactions and dynamic relationships among objects, and their associated contexts. Thus, in continual learning, SGG models are often required to expand, modify, retain, and reason scene graphs within the process of adaptive visual scene understanding. To systematically explore Continual Scene Graph Generation (CSEGG), we present a comprehensive benchmark comprising three learning regimes: relationship incremental, scene incremental, and relationship generalization. Moreover, we introduce a ``Replays via Analysis by Synthesis" method named RAS. This approach leverages the scene graphs, decomposes and re-composes them to represent different scenes, and replays the synthesized scenes based on these compositional scene graphs. The replayed synthesized scenes act as a means to practice and refine proficiency in SGG in known and unknown environments. Our experimental results not only highlight the challenges of directly combining existing continual learning methods with SGG backbones but also demonstrate the effectiveness of our proposed approach, enhancing CSEGG efficiency while simultaneously preserving privacy and memory usage. All data and source code will be made public.

Adaptive Visual Scene Understanding: Incremental Scene Graph Generation
Naitik Khandelwal! Xiao Liu Mengmi Zhang
College of Computing and Data Science, Nanyang Technological University (NTU), Singapore Deep NeuroCognition Lab, Agency for Science, Technology and Research (A*STAR), Singapore Address correspondence to mengmi.Zhang ntu..du..s
Abstract
Scene graph generation (SGG) analyzes images to extract meaningful information about objects and their relationships. In the dynamic visual world, it is crucial for AI systems to continuously detect new objects and establish their relationships with existing ones. Recently, numerous studies have focused on continual learning within the domains of object detection and image recognition. However, a limited amount of research focuses more challenging continual learning problem in SGG. This increased difficulty arises from the intricate interactions and dynamic relationships among objects, and their associated contexts. Thus, in continual learning, SGG models are often required to expand, modify, retain, and reason scene graphs within the process of adaptive visual scene understanding. To systematically explore Continual Scene Graph Generation (CSEGG), we present a comprehensive benchmark comprising three learning regimes: relationship incremental, scene incremental, and relationship generalization. Moreover, we introduce a "Replays via Analysis by Synthesis" method named RAS. This approach leverages the scene graphs, decomposes and re-composes them to represent different scenes, and replays the synthesized scenes based on these compositional scene graphs. The replayed synthesized scenes act as a means to practice and refine proficiency in SGG in known and unknown environments. Our experimental results not only highlight the challenges of directly combining existing continual learning methods with SGG backbones but also demonstrate the effectiveness of our proposed approach, enhancing CSEGG efficiency while simultaneously preserving privacy and memory usage. All data and source code are publicly available here)
Introduction
Scene graph generation (SGG) aims to extract object entities and their relationships in scene. The resulting scene graph, carrying semantic scene structures, can be used for a variety of downstream tasks such object detection[64 image captioning (20 and visual question answering (17). Despite the notable advancements in SGG, current works have largely overlooked the critical aspect of continual learning. In the dynamic visual world, new objects and relationships are introduced incrementally, posing challenges for SGG models to account for new changes without forgetting previously acquired knowledge. This problem of Continual ScenE Graph Generation (CSEGG) holds great potential for various applications, such as real-time robotic navigation in dynamic environments and adaptive augmented reality experiences.
The field of continual learning has witnessed significant growth in recent years, with a major focus tasks such as image classification [3), object detection [67, and visual question answering 28 However, these endeavors have largely neglected the distinctive complexities associated with CSEGG. Here, we highlight several unique challenges of CSEGG: (1) In contrast to object detection,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Scene Graph
Home & Hotel
Outdoor
Shopping & Dine
looking at
dog
dog
sitting on
scat, under, thair>
eman, besides, car> cear, parked on. street> besides, sidewalks
cwaitress, holdnn, menuz
chair
In
grass
ochar, on, foor-
cwaitress, serving, customerio
<subsect,
object>
cdola besides chairs
Objectt
Relations
New objects and new relations emerge in new scenes
Time
Figure (a) A scene graph is a graph structure, where objects are represented as nodes (red boxes), and the relationships between objects are represented as edges connecting the corresponding nodes (green boxes). Each node in the graph contains information such as the object's s class label. and spatial location. The edges in the graph indicate the relationships between objects, often described by predicates. A scene graph can be parsed into a set of triplets, consisting of three components: a subject, a relationship predicate, and an object that serves as the target or object of the relationship. The graph allows for a compact and structured representation of the objects and their relationships within a visual scene. (b) An example CSEGG application is presented, where robot continuously encounters new objects (blue) and new relationships (yellow) over time across new scenes.
SGG involves understanding and capturing the relationships between objects, which can be intricate and diverse. Consequently, in CSEGG, conveying the spatial and semantic relationships between objects demands adaptive reasoning from the dynamic scene. (2) SGG introduces a higher level of combinatorial complexity than object detection and image classification because each detected object pair may have multiple potential spatial and functional relationships. Thus, as new objects are introduced to the scenes, the complexity of relationships among all the objects increases significantly in non-lineat fashion. (3) The long-tailed distribution in both objects and relationships in SGG can be attributed to the inherent characteristics of real-world scenes, where certain objects are more prevalent than others. Consequently, CSEGG requires the computational models to adapt continually to the evolving long-tailed distributions over different scenes. Due to a scarcity of research specifically addressing these challenges of CSEGG, there is pressing need for specialized investigations and methodologies to enable computational models with the ability of CSEGG.
In this study, we re-organize existing SGG datasets 25
27 to establish a novel and comprehensive CSEGG benchmark with 3 learning protocols as shown Fig. 2 (S1). Relationship-incremne
setting: an SGG agent learns to recognize new relationships among familiar objects within the same scene. (S2). ecene-incrementll setting: an SGG agent deployed in new scenes where it has to jointly learn to detect new objects and classify new relationships. (S3). Relationship generalization setting: an SGG agent generalizes to recognize known relationships among unknown objects, as the agent learns to recognize new objects.
We curate a set of competitive CSEGG baselines by directly combining three major categories of continual learning methods with two SGG backbones and benchmark them in our CSEGG dataset. Their inferior performances show the difficulties of our benchmark tasks, which require the ability to expand, modify, retain, and reason scene graphs within the process of adaptive visual scene understanding. Specifically, the weight-reguulariztton methods fail to estimate the importance of learnable parameters given the complicated model design in SGG backbones. Although image-replay methods retain knowledge from prior tasks through replays, the extensive combinatorial complexity of relationships among objects surpasses the complexity accommodated by restricted set of replay images with efficient storage. Additionally, none of these baseline methods consider the shifts inherent in long-tailed distributions in dynamic scenes.
To address the CSEGG challenges, we present a method called "Replays via Analysis by Synthesis". abbreviated as RAS. RAS employs scene graphs from previous tasks, breaks them down and re- composes them to generate diverse scene structures. These compositional scene graphs are then used for synthesizing scene images for replays. Due to its nature of symbolic replays, RAS does not require the storage of original images, which often carry excessive and redundant details. This also ensures data privacy preservation and data efficiency. Furthermore, by synthesizing scenes using composable scene graphs, RAS maintains the semantic context and structure of previous scenes and also enhances the diversity of scene generation. To prevent biased predictions stemming from long-tailed distributions, we moderate the distribution of replayed scene graphs by balancing tail and head classes. This ensures uniform sampling of relationships and objects during replays. Extensive
2

---

Eval. metries
S.
#Tasks
#Objs
#Rels
SGG Backbone
CL basc.
Kn.
Unk.
Objs
S1
5
150 (All)
10 per task
R.
Transformer based (SGTR)
Joint, Naive,
Rels
(bbox, labels)
mF, mR,
Replay M%, FWC.
Task 1: 100
Task
Rels and Ohjs (bbux, labels)
S2
2
FWT, BWT,
None
Task 2: 25
Task 2:
Gen Robor Gen R
CNN based (IMP)
Pec
Objs (bhox, labels)
S3
4
30 per task
35 per task
RAS_GT
Rels
Table 1: Overview of three CSEGG learning scenarios. This table summarizes the three learning
scenarios (Column 1) in CSEGG, including the number of tasks, the number of object (#Objs) and relationship (#Rels) classes, the evaluation metrics, the NGG-Backbones used, and the continual learning (CL) baselines. The Kn. and Unk. columns provide information regarding what is known to the CSEGG models during training in that scenario and what is being incrementally learned by the models. Unknown information is being incrementally learned by the models. See Sec. 3 for details.
experiments underscore the effectiveness of our approach. Network analysis reveals our crucial design choices that can be beneficial for the future development of CSEGG models.
2 Related Works
Scene Graph Generation Datasets. Visual Phrase [59] stands as one of the earliest datasets in the field of visual phrase recogmition and detection. Over time, various large-scale datasets have emerged to tackle the challenges of Scene Graph Generation (SGG) on static images [23] 42
25
27
37
78
74 72
80 12
35 81] Subsequent works further extend the SGG to dynamic videos 22
50, 56
Despite the significant contributions of these datasets SGG, none focuses continual learning in SGG. As the preliminary efforts towards CSEGG, we start with fundamental and straightforward settings of SGG on static images. Among all the SGG datasets on static images, the Visual Genome dataset (25 has played a pioneering role by providing rich annotations of objects, attributes, and relationships in images. Thus, we re-structure the Visual Genome dataset [25] and establish a novel and comprehensive CSEGG benchmark, where AI models are deployed to dynamic scenes where new objects and new relationships are introduced.
Scene Graph Generation (SGG) Models. SGG models are categorized into two main approaches: top-down and bottom-up. Top-down approaches 38 77 typically rely on object detection precursor to relationship prediction. They involve detecting objects and then explicitly modeling their relationships using techniques such as rule- reasoning 42] graph convolutional networks 773
On the other hand, bottom-up approaches focus on jointly predicting objects and their relationships in an end-to-end manner (34 35 72). These methods often employ graph neural networks 33 82
message-passing algorithms 72 to capture the contextual information and dependencies between
objects. Furthermore. recent works have explored the integration of language priors [48] 42 69
and attention mechanisms in transformers [3] to enhance the accuracy and interpretability of scene graph generation. However, none of these works evaluate SGG models in the context of continual learning. In our work, we directly combine continual learning methods with SGG backbones and benchmark these competitive baselines in CSEGG. Our results reveal the limitations of these methods and highlight the challenges of our CSEGG learning protocols.
Continual Learning Methods. Existing continual learning works can be categorized into several approaches. (1) Regularization-basse -based methods 24 9 79 aim to mitigate catastrophic forgetting by employing regularization techniques in the parameter space. (2) Dynamic trchitecture-basee
approaches (66] [66 21 (7) adapt the model's architecture dynamically to accommodate new tasks without interfering with the existing ones. (3) Replay-based methods [57] I0 55 65
52 7 utilize a
memory buffer to store and replay past data during training, cnabling the model to revisit and learn from previously seen examples, thereby reducing forgetting. The special variants of these methods [1], where synthetic data is generated and replayed. Although these generative replay methods, as well as other continual learning methods,
include generative replay methods, such as [6] 71 75
have been extensively studied in image classification 8 77 43 and object detection (68 60 45). few works focus on the challenges in CSEGG, such as adaptive reasoning from the dynamic scenes, the evolving long-tailed distribution across scenes, and the combinatorial complexity involving objects and their multiple relationships. In this work, we introduce a continual learning method, abbreviated as RAS (Replays via Analysis by Synthesis). To address the distinct challenges of CSEGG, RAS
3

---

S1: Rel. Incre.
S2: Scene Incre.
S3: Rel. Gen. In Obj. Incre.
<oup, on,
<woman, shtting on, chair>
<chalr, cabinet>
<glass, on, tables
at.
<tlwwrr in, vase> cwoman, near, table> cwomen, neer, vase>
<bottle, on, tables
cman, at, table>
clogo, on, bottle>
<womBn, sitting on, cheir>
<lady carrying box>
<short, on, player>
<kid. on, surfboards
Task
clady, has, hel>
<layeer, wearrng, short-
<pizza, on, plate> <man, wearing, shirt- <logo, shirt=
clady, carrving, beaket>
<logo. on. sign- <sock, player-
<basket, attached to. poles
Time
Training set
Test set
Figure 2: Three learning scenarios are introduced. From left to right, they are S1. relationship (Rel.) incremental learning (Incre.); S2. scene incremental learning: and S3. relationship generalization (Rel. Gen.) in Object Incre.. In S1 and S2, example triplet labels in the training (solid line) and test sets (dotted line) from each task are presented. The training and test sets from the same task are color-coded. Blue color indicates task and orange color indicates task 2. The new objects or relationships in each task are bold and underlined. In S3, one single test set (dotted gray box) is used for benchmarking the relationship generalization of object incre. learning models across all the tasks.
involves creating in-context synthetic scene images based re-composable scene graphs from previous tasks to reinforce continual learning. The components RAS facilitate memory-efficient training and preserve privacy while maintaining the scene diversity and scene context for SGG in dynamic environments. With the rise of pretrained vision-language models (VLMs) [49 31 83 40),
various SGG methods [1T 30] have been proposed to tackle apen-voccbbllry and zero-shot SGG challenges. However, these settings differ fundamentally from CSEGG, where we aim to simulate scenarios where the model encounters novel predicates objects, unseen by any models, including LLMs or multi-modal models. Using pre-learned information from LLMS or frozen encoders conflicts with the continual learning setting we address in CSEGG.
3 Continual ScenE Graph Generation Benchmark
In CSEGG, to cater to the three continual learning scenarios below, we re-organize the Visual Genome [25] dataset and follow its standard image splits for training, validation, and test sets specified in (72). In each learning scenario,
consider sequence of T tasks consisting of images and corresponding scene graphs with new objects, new relationships, both. Let Dj [(T,, ) represent the dataset at task f, where Ię denotes the i-th image and G; represents the associated scene graph. The scene graph G; comprises a set of object nodes O; and their corresponding relationships R. Each object node 0j is defined by its class label cj and its bounding box locations and sizes bj. Each relationship r'k is represented by triplet (o., Pk, Pk,,,), where o, and o, denote the subject and object nodes, and pk represents the relationship predicate.
3.1 Learning Scenarios
Scenario 1 (S1): Relationship Incremental Learning. To uncover contextual information and go beyond studies of object detection and recognition, we introduce this scenario consisting of 5 tasks where 10 new relationship classes are incrementally added in every task (Fig. 2 left: Fig. ST] Tab. I]] All object classes and their locations are made known to all CSEGG models over all the tasks. This scenario resembles human learning scenario where a parent gradually teaches a baby to recognize new relationships among all objects in the same room, focusing one new relationship at a time during continual learning. This scenario also has implications in medical imaging where identical cell types may form new relationships with nearby cells depending on the context (Sec. A.1.I)
Scenario (S2): Scene Incremental Learning. To simulate the real-world cases when there are demands for detecting new objects and new relationships from old to new scenes, we introduce this scenario where new objects and new relationships are incrementally introduced over tasks (Fig. 2, middle: Fig. ST Tab. []. There are 2 tasks in total with the first task containing 100 object classes and 40 relationship classes with 25 more object classes and 5 more relationship classes the second task. This aligns with the real-world use cases where common objects and relationships are learned in the first scene, and incremental learning in the second scene only happens on less frequent relationships and objects. See Sec. A.1.2 for details.

---

Scenario 3 (83): Relationship Generalization. Humans have no problem at all recognizing the relationships of unknown objects with other nearby objects. This scenario is designed investigate the relationship generalization ability of CSEGG models. This capability is essential for real-world implications. such as robotic navigation where it often encounters unknown objects and requires classifying their relationships. In total, there are four tasks, each introducing an incremental addition of 30 new object classes, All relationship classes are made known to all CSEGG models over all the tasks (Fig. 2 right; Fig. ST: Tab. D.. Different from scenarios S1 and S2, a standalone generalization test set is curated, where the objects are unknown but the relationship classes among these unknown objects are common to the training set of every task. The CSEGG models trained after every task are tested on this standalone generalization test set to predict relationships among the unknown objects. See Sec. A.1.3 for details.
Data sampling and distributions. To allocate data for every task of each scenario, we perform the following sampling strategies. In S1 and S3 above, either object or relationship classes are randomly sampled from the Visual Genome dataset and incrementally added to every task. Due to the inherent characteristics of real-world scenes, the long-tailed class distribution is present in SI and S3. However, in S2, only tail classes are sampled and added in subsequent tasks. The number of tasks in each scenario is experimentally determined to optimize the training data configuration, ensuring sufficient training samples in each task while maximizing the number of tasks. For detailed statistics, see Fig. S2 and Sec. A.2
3.2 Competitive CSEGG Baselines
Due to the scarcity of CSEGG works, we contribute diverse set of competitive CSEGG baselines and implement them on our own. Each CSEGG baseline requires three components: a backbone model for scene graph generation (SGG), continual learning (CL) method to prevent the SGG model from forgetting, and an optional data sampling technique to deal with imbalanced data at every task for training SGG models. Next, we introduce the 2 SGG backbones, the 5 continual learning methods, and the optional data sampling techniques. See See. A.3 for implementation and training details of CSEGG baselines.
SGG Backbones. We use the two statc-of-the-e backbones: (1) one-stage Scene graph Generation TRansformer (SGTR) (32) and (2) the traditional Two-stage SGG model (TCNN) (72). Briefly,
SGTR (Fig. left) uses a transformer-baser architecture for image feature extraction and fusion.
During training, 32 formulates SGG as bipartite graph construction and matching problem. In contrast, TCNN detects objects with Faster-RCNN(i8 backbone and infers their relationships
separately via Iterative message passing (72). We use implementations from 32] and [68 with default hyperparameters.
Baselines. We include the following continual learning methods (Fig. S3 right): (1) Naive (lower bound) trained on each task in sequence without any measures to prevent catastrophic forgetting. (2) EWC(24) is a weight-regularizztion method, where the weights of the network are regularized in the parameter space, based on their "importance" " the previous tasks, (3) PackNet[44] is a parameter-isolation method, iteratively pruning the network parameters after every task, so that it can sequentially pack multiple tasks within one network. (4) Replay @ 557 includes a memory buffer with the capacity of storing M percentages of images in the entire dataset as well as their corresponding ground truth object and predicate notations depending on the task at each learning scenario. We vary M- = 10%, 20%, and 100%. (5) Joint Training is an upper bound where the SGG model is trained on the entire CSEGG dataset. (6) RAS_ is a baseline in which we use the ground truth scene graph labels from each task to create replay buffers using an image generation model explained in detail in Sec. 4 See Fig. S4 for schematics of CSEGG basclines. We provide mathematical formulations of these baselines Sec. S.c.A.4.
Sampling Methods to Handle Long- Distribution. We adopt the five data sampling tech- niques to alleviate the problem of imbalanced data distribution during training. (1) LVIS 19 is an image-level over-sampling strategy for the tailed classes. (2) Bi-level sampling (BLS) 33
balances the trade-off between image-level oversampling for the tailed classes and instance-level under-sampling for the head classes. (3) Equalized Focal Loss (EFL. 29 is an effective loss function, re-balancing the loss contribution of head and tail classes according to their imbalanced distribution.
EFL. enabled all the time for all the CSEGG baselines. In addition to applying data sampling techniques to the training sets, we can also apply LVIS and BLS techniques to the data stored in the
5

---

Selecting Unique Triplets
<man horse> <mn infrontof hou se>
Selecting Triplets for Prompé
Context Checker
Prompe Gien.
Training Data
Generating
"Reulistic Image horse
Notations
Gen. Img. Model
behind horse and man in frant of howse.
Figure 3: Schematic of our proposed Replays via Analysis by Synthesis (RAS) method. At task t+ our RAS stores all the triplet labels Uz, such as <man, on, horse>, from the previous tasks. It then re-composes these triplet labels to create in-context prompts, utilizing them as inputs to generative image models to synthesize images for replays. For predicting scene graphs on these synthesized images, we employ the frozen model Mr from the preceding task t, marked with "snowflakes" Subsequently, these predicted scene graph notations, along with their corresponding synthesized images, contribute to "pseudo" replays, preventing the current model M,+1 from experiencing forgetting. See Sec. for more details.
replay buffer. We name these data sampling techniques applied during both training and replays as (4) LVIS@ LeIl@Rellay and (5) BLS @ Replay.
3.3 Evaluation Metries
we adopt the evaluation metric @ (R@K) on the top K predicted triplets in the scene graphs G. As CSEGG is long-tailed, we further report the results in mean recall (mR @ K) over the head, body, and tail classes. Forgetfullness (F), Average (Avg.) performance, Forward Transfer (FWT) (39] and Backward Transfer (BWT) [1]] are standard evaluation metrics used for continual learning in image recognition and object detection tasks. In Scenario and 2, we adapt these metrics to recalls @@K and introduce F@ K, Avg.R @ K. FWT @ K. and BWT@ K respectively for CSEGG settings. Similarly, we also adapt these metrics to mR K. We explored CSEGG with K=20, 50, and 100. Since our results are consistent among Ks, omit "@K" and analyze all the results based on K=20 the entire text.
Same as existing SGG works 72 32,
In scenario S3, we evaluate all CSEGG methods in the standalone generalization test set, shared over all the tasks. To benchmark generalization abilities in unknown object localization and relationship classification among these unknown objects, we introduce two evaluation metrics: Gen Riboz @ K and Gen R@K K. As the CSEGG models have never been taught to classify unknown objects, we discard the class labels of the bounding boxes and only evaluate the predicted bounding box locations with Gen Rioor K. To evaluate whether the predicted bounding box location is correct, we apply a hard threshold of Intersection over Union (IoU) between the predicted bounding box locations and the ground truth. Any predicted bounding boxes with their IoU values above the hard threshold are deemed to be correct. We vary IoU thresholds from 0.3, 0.5, to 0.7.
To assess whether the CSEGG model generalizes to detect known relationships over unknown objects. we evaluate the recall Gen R@K of the predicted relationships 7 only on correctly predicted bounding boxes. See Sec. A.5 for details. All results are averaged over runs.
4 Replays via Analysis by Synthesis (RAS)
To address the complexities in CSEGG, we introduce our "Replays via Analysis by Synthesis" method, dubbed RAS. Our RAS belongs the group of generative replay methods for continual learning. We create an exemplar set E, for replays. At task t, we jointly train the scene graph generation model M, on E, and the current training dataset D.. However, different from the existing generative replay methods [6]] [5] [6), our RAS leverages symbolic replays with state-of offte-oflin siiee--o-theaaat the-art diffusion models. Moreover, rather than generating any random images for replays, our RAS capable of generating in-context images following semantic rules, such as object co-occurrences. The schematic our RAS is presented in Fig. 3 Next, we focus on how RAS creates F, containing the generated images and their SGG annotations these images for replays.

---

Image Generation. At the current task ++ 1, our RAS requires storing the frozen old model snapshot M, at the end of the previous task and all the triplet labels U, which are parsed from all the scene graphs aggregated from all the previous tasks. These triplets contain object labels, subject labels, and relationships among them. For example, <man, on, horse> and <man, in front of, horse> are two unique triplet labels. Unlike the traditional replay methods in continual learning literature, our method refrains from storing original images 1; or scene graphs G, in the training sets, thereby eliminating storage issues and privacy concerns.
To generate images I for replays, our RAS feeds text prompts, which are formed by a set of chosen triplet labels and describe the diverse in-context scenes, into the state-of-the-an Stable Diffusion model [58]. As previous works suggests context plays important roles in visual perceptions [5]. To generate text prompts describing context-congruent scenes, we employ context checker. First, the context checker uses the pre-trained large language model BERT [4 to extract embeddings for each triplet label in U.. As BERT has been pre-trained on a large corpus of text data, it learns to capture context-relevant representations of words. Next, hierarchical clustering is performed on these embeddings using the agglomerative clustering algorithm [46]. This ensures that each cluster contains only embeddings that are semantically close. The threshold for the agglomerative clustering algorithm is set to 0.6. As real-world images often contain complex scenes involving multiple triplets, we select any cluster with more than 3 triplet labels to create a text prompt for image generation.
In practical applications, conducting agglomerative clustering on all triplet labels in U2 is computa- tionally demanding. as requires computing pairwise embedding similarities among all the triplet labels. To address this, RAS opts for more efficient approach during replays by selecting a subset of triplet labels and clustering their embeddings. Recognizing that real-world scenes often exhibit a long-tailed distribution with certain objects or relationships being more prevalent, we introduce the LoigeTailed Distribution (LTD) module for balancing this distribution in U.. Unlike image-level sam- pling methods like BLS and LVIS 33 19 discussed in Sec. 3.2 our LTD module in RAS operates at the triplet level. For each triplet label, its dropout rate is determined proportionally to its frequency in U.. Specifically, we define the drop-out rate dx for the k-th triplet as: d4 = / ft) a, where N is the total number of triplets in U,, 0.7 is scaling factor, and fa is the frequency of the i-th triplet in Uy. This sampling formula enables RAS to select triplets from tail classes more frequently compared to those from head classes.
To generate a text prompt from the chosen triplet labels, we employ a straightforward English language construct using the conjunction "and". This involves combining all the selected triplet labels into a sentence by starting with "Realistic Image of". For instance, if the triplet labels are <man, on, horse>, <house, behind, horse>, and <man, in front, house>, the generated prompt becomes "Realistic Image of man on horse and house behind horse and man in front of house." " To increase exemplar diversity for replays, use the Stable Diffusion model [58] to generate y number of images for the same text prompt. In practice, we set = 10 over all the learning scenarios.
We provide the visualization examples of some synthesized images along with the corresponding text prompts in Fig. SS From these examples, we found that the composed text prompts and the synthesized images are often of high quality and contextual coherence.
Scene Graph Prediction on Synthesized Images. During replays, to train the model M4+1 on IJ. we also need to predict their corresponding scene graph notations Go on 7/. As the frozen model snapshot M; at the end of task t carries prior knowledge for SGG from the previous tasks, we use it to predict notations Gj, on I These G'; comprises object nodes O2 with their respective classes c/j along with object locations U/, Additionally, it includes corresponding relationship nodes Rt formed by triplets <0',, P/s. df> representing subject, predicate, and object nodes, respectively. These generated notations G¹ along with I2, serve to construct the exemplars E,, used for replays.
Results
5.1 RAS outperforms all the CSEGG baselines in Scenarios 1 and 2
The results for Avg. R, F. mR, mF, FWT, and BWT in learning scenarios (S1) and 2 (S2) are presented in Tab. [ Our observations align with established research in continual learning, especially in image classification and object detection: regardless of the SGG architectures, over both learning scenarios, Naive consistently performs the worst, showcasing significant catastrophic forgetting.
7

---

B2
Learning Scenario (SI)
Learning Scenario (S2)
Methods
Avg.R
Ft
mRt
mFt
FWTT BWTT
Avg.Rt
Ft
mRt
mFf
FWTT
BWT1
Joint
20.15
0
4.6
0
12.64
0.
9.84
0
Replay @ 100%
16.17
-12.24
3.32
-1.34
-1.77
-11.72
4.56
-4.13
4.56
-5.61
-1.045
-30.25
Naive
1.33
-28.7
0.86
-1.74
-2.03
-60.67
0.51
-23.22
0.05
-11.31
-3.77
-62.34
EWC 24
1.89
-28.4
0.96
-1.72
-1.17
-52.45
-23.22
-11.31
-2.65
-50.12
RAS GT
5.78
-26.51
1.43
-1.54
-1.2
-44.27
0.98
-23.11
0.76
-10.86
-1.6
-43.25
PackNet [4
7.19
-25.67
1.35
-1,64
-1.03
-42.35
1.67
-22.77
0.9
-10.33
-1.4
-42.45
Replay @ 10%
8.55
-22.21
4.33
-1.44
4.29
-38.35
1.81
-20.72
1.15
-9.64
-0.9
-40.67
Replay 20%
9.25
-20.35
4.78
-1.42
3.21
-31.98
2.57
-17.17
1,56
-8.07
-0.67
-38.27
Ours*
10.78
-18.92
5.6
-1.39
2.3
-25.56
3.45
-10.23
2.75
-6.57
-0.54
-35.67
TCNN
72
Learning Scenario 1
(SI)
Learning Scenario (S2)
Methods
Avg.Rt
Ft
mRt
mFt
FWTT BWTT
Avg.Rt
Ft
mR
mFf
FWT1
BWT1
Joint
19.53
0
3,9
0
4.3
0
3.7
Replay @ 100%
13.45
-8.83
3.6
-0.35
-1.5
-10.45
12.45
4.13
3.2
-0.56
-2,1
-00.4
Naive
0.98
-21.2
0.74
-1.35
-3,45
-4387
18.22
0.45
-2.67
-4.12
-53.12
EWC[24
2.36
-21.55
0.67
-1.34
-2.34
-3.899
-18.2
0.03
-3.77
-51.67
PackNet
3.2
-19.7
1.1
-1.13
-1.3
-32.45
1.1
-17.82
0.84
-1.97
-2.84
-40.34
Replay @ 10%
5.67
-18.9
3,21
-1.05
1.45
-28.44
1.81
-16.72
1.03
-1.74
-1.4
-43.56
Replay @00%
6.23
-17.45
3.5
-1.01
1.01
-24.32
2.37
-15.17
1.45
-1.53
-1.1
-38.56
Ours
7.8
-15.67
3.9
-0.95
0.5
-19.83
4.67
-11.31
2.2
-0.89
-0.97
-29.65
Table 2: Results of CSEGG for various continual learning methods applied on the two SGG backbones (SGTR and TCNN) in Learning Scenarios and 2. See Sec. 3.2 for continual learning baselines. See Sec. 3.3 for evaluation metrics. The higher the evaluation metrics, the better. The best are in bold, * means the experiment still running, we will report the results in the final version.
Model
mRt mFt
InU-0.3
1oU-0.5
IoU-0.7
Narve
Replay@10%
RAS
LVIS @ Replay @ 10%
3.98 -1.54
BLS@ @eseleyplaee @ 10%
4.34 -1.47
Gen Rebort
60
RAS (ours)
5.6 -1.39
Gen
40
Table 3: Results at Task 5 in Learning Scenario I when sampling techniques are applied to long- tailed distribution data. See Sec. 3.2 for the sampling techniques on long-tailed distributions. We copy the results of our RAS from Tab. [2 for easy comparisons. The best results are in bold.
T4
20
12
T4
(a)
(b)
Figure 4: Results in Scenario 3. See Sec. 3.3 for evaluation metrics. The higher the values, the better. Line colors indicate continual learning methods. Line types denote the IoU thresholds
Replay-based methods, such Replay @ 10% and Replay @ @20% outperform techniques like EWC and PackNet. However, none of them surpass our RAS.
RAS achieves superior performance compared to Replay @ 20% (~2 Gb) while requiring less storage (~1.22 Gb), equivalent to storing 15% exemplary images. This storage efficiency is due to only needing to store the old model snapshot, triplet labels in U2, and the image generation model, thus avoiding privacy concerns. Also, RAS outperforms RAS GT (Tab. ( indicating that decomposing scene graphs into smaller, more diverse ones, along with more comprehensible prompts, is more effective than storing the ground truth scene graphs and directly using them for image generation.
In Learning Scenario 2 (S2), the task involves classifying both new objects and new relationships, significantly escalating the level difficulty compared to S1. As evident from Tab. 2 all CSEGG models, including Replay @ 100%, exhibit a decline in overall performance when compared to the upper bound Joint. Although our RAS achieves the leading performance among all the baselines, its performance is still far from Joint. This underscores the persistent challenge posed by S2 in the context of CSEGG. Future work should explore new approaches to address this gap.
To tackle the issue of imbalanced data distribution in real-world scenarios, we incorporate two established data sampling techniques (LVIS@ L.VIS@Replay @ 10% and BLS Replay @ 10%) into our experiment (Sec. 3.2). The outcomes in learning scenario S1 are presented in Tab. 3 We observed that their performance falls short of our RAS, underscoring the efficacy of RAS addressing long- tailed distribution during generative replays. We also noticed that BLS @ Replay @ 10% significantly
8

---

?
Context
LTD
Triplet
Avg.R 0
F
mR 1
mF 1
A1
10
x
/
Multiple
8.23
-21335
3,98
-1.8
A2
10
/
X
Multiple
9.75
20.65
4.12
-1 -1.65
A3
10
/
Single
7.75
-22.45
3.12
-2.48
2
Multiple
2.45
-27.43
0.45
-9.84
A5
4
/
Multiple
5.67
-26.42
2,41
-3.26
A6
8
/
Multiple
7.89
-22.42
3,89
-2.17
Ours
10
7
Multiple
10.78
-18.92
5.6
-1.39
Table 4: Ablation results of our RAS on learning scenario S1 reveals key design insights. This table presents the results of ablation studies conducted to identify key components our method, as discussed in Sec. 5.3 Results in Avg.R, F, mR, mF are reported after the last task in Scenario S1.
outperforms LVIS Replay 10%, contrary to findings in the classical SGG problem where BLS is considered more effective than LVIS [33]. The performance difference may stem from variations in the number of replay instances between the two approaches after applying these data re-sampling methods to exemplar images in the memory buffer (Sec. (.2)). This observation suggests that the original sampling methods designed for addressing long-tailed distributions in the classical SGG problem may not be effective when applied to CSEGG.
We explored the impact of task sequence permutations CSEGG performance, finding effect consistent with existing literature [63] (Fig. S6 Sec. A.6.1). We also observed that fine-tuning DETR S1 has minimal impact on forgetting, indicating that any forgetfulness in S1 is solely due to relationship incremental learning (Fig. S7 Sec. A.6.2). Moreover, to gain qualitative insights, we provide visualizations of predicted scene graphs for all CSEGG baselines in Scenario I (Fig, S8 and Sec. A.7.I and Scenario 2 (Fig, $9 and Sec. A.7.2).
5.2
CSEGG Models Can Generalize in Unknown Scenes
Fig. 4 illustrates the generalization results for detecting unknown objects and classifying known relationships among these objects in Learning Scenario (S3). In Fig. 4 (a), an increasing trend in Gen RMoo is observed with the increasing task number for all CSEGG methods, indicating improved generalization in detecting unknown objects. Notably, even with minimal training in Task 1, all CSEGG methods propose 23% reasonable object regions with threshold IoU = 0.7, showcasing the SGTR model's 's ability to generalize to locate "objectness". As expected, with an increase in loU threshold from 0.3 to 0.7, we found that Gen Ríboz decreases due to fewer bounding boxes being considered correct. Moreover, we also compared the generalization performance in object detection between Replay @ 10% and Naive. Contrary previous observations in S1 and $2 (Tab. 2), found that Replay @ 10% show a decline in Gen Rybcr. possibly due to a fixed number of detected object bounding boxes output by CSEGG methods. Similarly, our RAS exhibits reduced G Raboz compared to Replay @ 10% and Naive, likely for the same underlying reason.
In Fig. (b), Replay @ 10% outperforms Naive in Gen R when considering correctly detected unknown object locations, emphasizing that minimizing forgetting in continual learning enhances the SGTR model's overall relationship generalization in unknown performance of Replay @ 10% is still inferior to our RAS method; implying that our RAS is more proficient in generalizing classify relationships among unknown objects. Interestingly, noted that even with minimal training in Task 1, all the CSEGG methods achieve 45% recall of known relationships among unknown objects, demonstrating the SGTR model's ability generalize to classify "relationship". Visualization examples, when CSEGG models can generalize to recognize and Sec. A.7.3
understanding. However, the
also
relationships, are presented Fig. and
5.3 Ablation Studies on Our RAS Reveal Key Design Insights
We introduce our default method designs Sec. Here, we vary the components in our RAS to reveal key design insights. We propose context checker in RAS. Here, we conduct an ablation by removing this module. Triplet labels are randomly selected and combined for text prompts. of Tab. 4 we observe a performance decrease of approximately 2% across all evaluation metrics, compared with our RAS. This suggests that generating images adhering real-world context rules is crucial for replays. The lower performance may be attributed to the challenge of generating

---

good-quality out-of-context images for Stable Diffusion Models and the potential domain differences affecting the SGG model M, in predicting out-of-context SGG notations.
The LTD sampling module in our RAS is designed to balance the distribution of head and tail triplet labels from U.. Here, we remove the LTD sampling module and report the performance of the ablated method in A2 of Tab. 4 Compared to our RAS, we observe an absolute decrease of 1-2% across all metrics. Notably, the relative decrease is more pronounced in mR and mF than Avg.R and F. As mR and mF indicate mean Recall and mean Forgetfulness over both tail and head classes, the larger drops in these metrics suggest that the absence of LTD sampling significantly hinders the SGG model's ability to predict tail classes from previous tasks.
In our RAS, employ multiple triplet labels to construct text prompts for image generation. In contrast to single triplet labels, our approach yields rich text descriptions of complex scenes, allowing the SGG model to capture intricate relationships among multiple objects in the same scene. Additionally, using multiple triplets is more efficient in rehearsing, as it enables the model to practice predicting multiple triplets simultancously within the same number of synthesized images. Indeed, when we replace multiple triplet labels with single triplet labels for text prompts, we note a decrease of approximately 3% across all metrics (compare A3 with ours in Tab.
Lastly, we investigate the impact of generating y images using the same text prompt in RAS, varying from 2 to (Tab. 4 A4-6). As expected. performance improves with higher y, showing that increased sample diversity enhances CSEGG performance. With ample computing resources, dynamically synthesizing more images could further improve performance. This highlights RAS's advantage in generating numerous images for replays without expanding storage usage.
Discussion
In the dynamic world, adapting scene graph generation (SGG) models to new objects and relationships poses challenges. Despite progress in SGG and continual learning, there is still a gap in understanding Continual Scene Graph Generation (CSEGG). We address this by operationalizing CSEGG, and introducing benchmarks, datasets, and evaluation protocols. Our study explores three learning scenarios, analyzing continual object detection and relationship classification in long-tailed class- incremental settings for CSEGG baselines. Our findings show that integrating sampling methods with CSEGG baselines to address long-tailes distributions moderately eliminates forgetting; however, a large performance gap between current CSEGG baselines and the joint training upper bound persists. To address CSEGG challenges, propose RAS, Replays via Analysis by Synthesis method. RAS parses previous task scene graphs into triplet labels for diverse in-context scene graph reconstruction. Based on these re-compositional context-congruent scene graphs, RAS synthesizes images with Stable Diffusion models for replays. Unlike other image replay methods, RAS stores only triplet labels and the model snapshot, maintaining constant memory usage and preserving privacy. Extensive experiments demonstrate our RAS's ' superior performance over current CSEGG basclines in knowledge transfers and reducing forgetting. Interestingly, our RAS model is also capable of generalizing to classify known relationships among unseen objects.
Moving forward, there are several key avenues for future research. First, our current endeavors focus on tackling CSEGG problems from static images in an Independent and Identically Distributed (i.d.d) manner, diverging from how humans learn from video streams. Future research can look into CSEGG problems on video SGG datasets. Second, our plans also involve expanding the set of continual learning baselines and integrating more long-tailed distribution sampling techniques. Third, we aim to construct a synthetic SGG dataset to systematically quantify the aspects of SGG that influence continual learning performance under controlled conditions. In RAS, SGG annotations for synthesized images in the replay buffer are predicted by the preceding SGG model, which can lead to error propagation across training iterations. In the future work, integrating a generative model with fine- fine-grained control signals (such as bounding boxes and captions) (36] could provide more precise supervision, potentially mitigating these accumulated errors and further enhancing the performance of our approach. Although the CSEGG method holds promise for many downstream applications like monitoring systems, medical imaging, and autonomous navigation, we should also be aware of its misuse in privacy, data biases, fairness, security concerns, and misinterpretation (see Sec. A.8 for an expanded discussion). We invite the research community to join us in expanding and updating the safe use of CSEGG benchmarks, thereby fostering its advancements in research and technology.
10

---

Acknowledgement
This research supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: LSSG2-RP-2021- 2021-22)- and its NRFF award NRF-NRFFI5-2023 0001. We also acknowledge Mengmi Zhang's s Startup Grant from Agency for Science, Technology. and Research (A (A*ST,R), Startup Grant from Nanyang Technological University, and Early Career Investigatorship from Center for Frontier Al Research (CFAR), A*STAR.
References
[1] Somak Aditya, Yezhou Yang, Chitta Baral, Cornelia Fermuller, and Yiannis Aloimonos. From images to sentences through scene description graphs using commonsense reasoning and knowledge, 2015.
[2] Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuyte- laars. Memory aware synapses: Learning what (not) forget. In Proceedings of the European conference on computer vision (ECCV), pages 139-154, 2018.
[3] Martin Andrews, Yew Ken Chia, and Sam Witteveen. Scene graph parsing by attention graph. arXiv preprint arXiv: 1et.:9999900000 00293, 2019.
[4] Frederik Benzing. Unifying importance based regularisation methods for continual learning. In International Conference on Artificial Intelligence and Statistics, pages 2372-2396. PMLR,
2022.
[5] Philipp Bomatter, Mengmi Zhang, Dimitar Karev, Spandan Madan, Claire Tseng, and Gabriel Kreiman. When pigs fly: Contextual reasoning in synthetic and natural scenes. In Proceedings of the IEEF/CV International Conference on Computer Vision, pages 255-264, 2021.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 213-229, Springer, 2020.
[7] Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pages 233 248, 2018.
[8] Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In Proceedings of the IEEE/CVF International conference computer vision, pages
2021.
[9] Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European conference on computer vision (ECCV), pages 532-547,
[10] Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arxiv preprint arXiv: 902. 2019.
[11] Zuyao Chen, Jinlin Wu, Zhen Lei, Zhaoxiang Zhang, and Changwen Chen. Expanding scene graph boundaries: Fully open-vocabulary scene graph generation via visual-concept alignment and retention. arXiv preprint arXiv:2311.10998 2023.
[12] Bo Dai, Yuqi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks. In Proceedings of the IEEE conference on computer vision and Puttern recognition, pages 3076-3086. 2017.
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEFE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.

---

[14] Jacob Devlin, Ming- deep bidirectional transformers for language understanding. arXiv preprint 2018.
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of
[15] Rui Gao and Weiwei Liu. Ddgr: continual learning with deep diffusion-based generative replay, In International Conference on Machine Learning, pages 10744-10763. PMLR, 2023.
[16] Chandan Gautam, Sethupathy Parameswaran, Ashish Mishra, and Suresh Sundaram. Generative replay-based continual zero-shot learning. arXiv preprint arXiv:2/01.08844 2021.
[17] Shalini Ghosh, Giedrius Burachas, Arijit Ray, and Avi Ziskind. Generating natural language explanations for visual question answering using scene graphs and visual attention, 2019.
[18] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440- 1440, 2015.
[19] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 5356-5366 2019.
[20] Muhammad Umair Hassan, Saleh Alaliyat, and Ibrahim A. Hameed. Image generation models from scene graphs and layouts: A comparative analysis. Journal of King Saud University Computer and Information Sciences, 35(5):101543, 2023.
[21] Ching-Y? Yi Hung, Cheng-Hao Tu, Cheng-En Wu, Chien-Hung Chen, Yi-Ming Chan, and Chu- Song Chen. Compacting, picking and growing for unforgetting continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[22] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10236-10247, 2020.
[23] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1668-3678, 2015.
[24] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):521-3326 2017
et al.
[25] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
[26] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[27] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7)1196-11981 2020.
[28] Stan Weixian Lei, Difei Gao, Jay Zhangjie Wu, Yuxuan Wang, Wei Liu, Mengmi Zhang, and Mike Zheng Shou. Symbolic replay: Scene graph as prompt for continual learning on vqa task, 2022.
[29] Bo Li, Yongqiang Yao, Jingru Tan, Gang Zhang, Fengwei Yu, Jianwei Lu, and Ye Luo. Equalized focal loss for dense long-tailed object detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 6990- (0000,9999000 2022.
[30] Lin Li, Jun Xiao, Guikun Chen, Jian Shao, Yueting Zhuang, and Long Chen. Zero-shot visual relation detection via composite visual cues from large language models. Advances Neural Information Processing Systems, 36, 2024.
12

---

[31] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, al. Grounded language-image pre-training. In Proceedings of the IEEF/CVE Conference on Computer Vision and Puttern Recognition, pages 10965-10975, 2022.
[32] Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19486-19499 2022.
[33] Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11109-11119, 2021.
[34] Yikang Li, Wanli Ouyang, Xiaogang Wang, and Xiao' ou Tang. Vip-cnn: Visual phrase guided convolutional neural network. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 1347-1356, 2017.
[35] Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xiaogang Wang. Scene graph generation from objects, phrases and region captions. In Proceedings of the IEEE international conference on computer vision, pages 1261-1270, 2017.
[36] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511-225211 2023.
[37] Yuanzhi Liang. Yalong Bai, Wei Zhang, Xueming Qian, Li Zhu, and Tao Mei. Vrr-vg: Refocus- ing visually-relevant relationships. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10403-10412, 2019.
[38] Wentong Liao, Bodo Rosenhahn, Ling Shuai, and Michael Ying Yang. Natural language guided visual relationship detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops, pages 0 0-0, 2019.
[39] Sen Lin, Li Yang, Deliang Fan, and Junshan Zhang. Beyond not-forgetting: Continual learning with backward knowledge transfer. Advances in Neural Information Processing Systems, 35:16165-16177
2022.
[40] Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-traine pre-training for open-set object detection. arXiv preprint 0xXiv:2303.054999 2023.
[41] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
[42] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 852-869. Springer, 2016.
[43] Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey, 2021.
[44] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages
2018
[45] Angelo G Menezes, Gustavo de Moura, Cézanne Alves, and André CPLF Carvalho. Continual object detection: A review of definitions, strategies, and challenges. Neural Networks, 2023,
[46] Daniel Müllner. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint 110. 2011.
13

---

[47] Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen. and Moin Nabi. Learning remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 11321-11329, 2019.
[48] Bryan Plummer, Arun Mallya, Christopher M Cervantes, Julia Hockemmaier, and Svetlana Lazebnik. Phrase localization and visual relationship detection with comprehensive image- language cues. In Proceedings of the IEEE international conference on computer vision, pages 1928-1937, 2017.
[49] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 9748-8763 PMLR, 2021.
[50] Nishant Rai, Haofeng Chen, Jingwei Ji, Rishi Desai, Kazuki Kozuka, Shun Ishizaka, Ehsan Adeli, and Juan Carlos Niebles. Home action genome: Cooperative compositional action understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11184-11192 2021.
[51] Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Tch, and Raia Hadsell. Continual unsupervised representation learning. Advances in Neural Information Processing Systems, 32, 2019.
[52] Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2010, 2017.
[53] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[54] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[55] Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. aaXiv preprint arXiv: 19900 2018
[56] Ivan Rodin, Antonino Furnari, Kyle Min, Subarna Tripathi, and Giovanni Maria Farinella. Action scene graphs for long-form understanding of egocentric videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8662-18633 2024.
[57] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap. and Gregory Wayne. Experi- ence replay for continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[58] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684- 10695, 2022.
[59] M. A. Sadeghi and A. Farhadi. Recognition using visual phrases. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, CVPR '11, page 1745-1752, USA, 2011. IEEE Computer Society.
[60] Jeng-Lun Shieh, Qazi Mazhar ul Haq, Muhamad Amirul Haq. Said Karam, Peter Chondro, De-Qin Gao, and Shanq-Jang Ruan. Continual learning strategy in one-stage object detection framework based on experience replay for autonomous driving vehicle. Sensors, 20(23):6777, 2020.
[61] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
14

---

[62] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:149. 2014.
[63] Parantak Singh, You Li, Ankur Sikarwar, Weixian Lei, Daniel Gao, Morgan Bruce Talbot, Ying Sun, Mike Zheng Shou, Gabriel Kreiman, and Mengmi Zhang. Learning to learn: How to continuously teach humans and machines. arXiv preprint arXiv:221 1.1477 2022.
[64] Christian Szegedy, Alexander Toshev, and Dumitru Erhan. Deep neural networks for object detection, Advances neural information processing systems, 26, 2013.
[65] Jeffrey S Vitter. Random sampling with reservoir. ACM Transactions Mathematical Software (TOMS), 11(1):37-57, 1985.
[66] Fu-Yun Wang, Dei Zhou, Han-Jia Ye, and De-Chuan Zhan. Foster: compression for class-incremental learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXV, pages 398-414. Springer, 2022.
Feature boosting and
[67] Jianren Wang, Xin Wang, Yue Shang-Guan, and Abhinav Gupta. Wanderlust: Online continual object detection in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11829-10836 October 2021.
[68] Jianren Wang, Xin Wang, Yue Shang-Guan, and Abhinav Gupta. Wanderlust: Online continual object detection in the real world. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10829-10838, 2021.
[69] Wenbin Wang, Ruiping Wang. Shiguang Shan, and Xilin Chen. Exploring context and visual pattern of relationship for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8188-8197, 2019.
[70] Zifeng Wang, Zizhao Zhang, Chen-Yu Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 139-149, 2022.
[71] Chenshen Wu, Luis Herranz, Xialei Liu, Joost Van De Weijer, Bogdan Raducanu, et al. Memory replay gans: Learning to generate new categories without forgetting. Advances in Neural Information Processing Systems, 31, 2018.
[72] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5410-5419, 2017.
[73] Jianwei Yang. Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In Proceedings of the European conference computer vision (ECCV), pages 670-685, 2018
[74] Kaiyu Yang, Olga Russakovsky, and Jia Deng. Spatialsense: An adversarially crowdsoureed benchmark for spatial relation recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2051-2060, 2019.
[75] Fei Ye and Adrian G Bors. Learning latent representations across multiple data domains using lifelong vaegan. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Pari XX 16, pages 777-795. Springer, 2020.
[76] Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. arXiv preprint arXiv:1708 1eee1778.005555 01577 2017.
[77] Ruichi Yu, Ang Li, Vlad Morariu, and Larry S Davis. Visual relationship detection with internal and external linguistic knowledge distillation. In Proceedings of the IEEE international conference computer vision, pages 1974-1982, 2017.
15

---

[78] Alireza Zareian, Zhecan Wang, Haoxuan You, and Shih-Fu Chang. Learning visual common- sense for robust scene graph generation. In Computer Eiion-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pages 642-657 Springer, 2020.
[79] Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International conference machine learning, pages 3987-3995. PMLR, 2017.
[80] Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. Visual translation embed- ding network for visual relation detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5532-5540, 2017.
[81] Zhang, Yannis Kalantidis, Mareus Rohrbach, Manohar Paluri, Ahmed Elgammal, and Mo- hamed Elhoseiny. Large-scall scale visual relationship understanding. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9185 91949,994,-9999 2019.
[82] Zhang, Kevin Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro. Graphical contrastive losses for scene graph parsing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11535-11543, 2019.
[83] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16793-16805 2022.
16

---

A
Appendix
A.1
Introduction to Three Learning Scenarios
Within this section, we present more details of three learning scenarios and their practical applications.
A.1.1
Scenario 1 (SI): Relationship Incremental Learning
While existing continual object detection literature focuses on incrementally learning object attributes
43 70 8 68 60 45), incremental relationship classifications are equally important as it provides a deeper and more holistic understanding of the interactions and connections between objects within scene. To uncover contextual information and go beyond studies of object attributes, we introduce this scenario where new relationship predicates Pę: are incrementally added in each task (Fig. S1). There are 5 tasks in S1. To simulate the naturalistic settings where the frequency of relationship distribution is often long-tailed, we randomly and uniformly sample relationship classes from head, body and tail categories in Visual Genome
[25], and form a set of 10 relationship classes for each task. Thus, the relationships within task are long-tailed; and the number of relationships from the head categories of each task is of the same scale. To tackle this issue, we allow CSEGG models to the same images over tasks, but the relationship labels are only provided in their given task
The design of such images and label splits over tasks aligns with human learning scenarios where parent teaches the baby to recognize different toys and objects in the bedroom. Though the baby is exposed to the same bedroom scenes multiple times, the parent only teaches the baby to detect and recognize one object at time in continual learning setting. In the future, will expand our studies to cases where the SGG models learn from non-overlapping sets of training images for each task. The same reasoning applies in S2 and S3. Example relationship classes from each task and their distributions are provided in Fig. S1
Here, we provide a concrete example application of Scenario 1 in medical imaging. Within medical imaging, an agent must acquire the ability to detect cancerous cells within primary tumors, like colon adenocareinoma. Subsequently, it must extend this proficiency to identifying the same cell types within metastatic growths that manifest in different bodily regions, such as lymph nodes or the liver. In this instance, the identical cancer cell disseminates to fresh organs or tissues, progressively establishing new relationships with other cells over the course of time.
A.1.2 Scenario 2 (S2): Scene Incremental Learning
To simulate the real-world scenario when there are demands for detecting new objects and new relationships over time old and new scenes, we introduce this learning scenario where new objects O, and new relationship predicates pk are incrementally introduced over tasks (Fig. S2).. To select the object and relationship classes from the original Visual Genome motivations in mind. First, in real-world applications, such as robotic navigation, robots might have already learned common relationships and objects in one environment, Incremental learning only happens on less frequent relationships and objects. (2) Transformer- based AI models typically require large amounts of training data to yield good performances. Training only on a small amount of data from tail classes often leads to cosse-tt-chance performances. Thus, we take the common objects and relationships from the head classes in Visual Genome as one task, while the remaining less frequent objects and relationships from tail classes the other task. This results in tasks in total with the first task containing 100 object classes and 40 relationship classes. In the subsequent task, the CSEGG models are trained to continuously learn detect 25 more object classes and more relationship classes. Same as S1, both the object class and relationship class distributions are still long-tailed within a task (Fig. ST
25 for S2, we have two design
Next, we provide two real-world example applications in robot collaborations on construction sites and video surveillance systems.
The CSEGG model's capacity to incorporate new objects and new relationships while retaining existing knowledge finds pivotal application in video surveillance contexts. Consider a company developing video-based security systems for indoor environments, capturing prevalent indoor objects and relationships. Expanding to outdoor settings like parking lots or restricted compounds demands retraining the model with new outdoor data alongside previous indoor data, ensuring operational effectiveness in both realms. The outdoor context introduces new objects like "cars" and relationships
17

---

others
building
others
has
on
dog
chair
ca
for
waae
beard
glass
shirt
man
others
others
helding
laptop
Jacket
horse
gir
bike
umbrella
bench
others
table
fower
leg
bullding
bed
woman
man
wearing
riding others
in
sitting on
others
near
behind
with
motorcycle
people
pant
pizza
pola
skateboard
ghint
snow
stres
rable
elephant
leg
fence
sign
person
hat
head
beach
person
tree peoplit
others
window!
head
hair
others
Task
Task 2 Tk
Task3 (24K)
Task2
Tassk (18KK
Task 2 (45K)
Task (11K)
(117K) (64K)
(52K)
(66K)
(662K)
(14K)
(13K)
Figure S1: Label distribution in each task in each learning scenario is presented. In scenario S1 (a) and scenario S3 (c), we use different colors to denote different tasks. The color gradient indicates the frequency of data within task, with the lighter color denoting the smaller frequency of data in that category. Only the most frequent labels (relationship labels in (a) and object labels in (c)) are provided. See the legend for the total data size per task. In (b) scenario S2 on both objects and relationships, data distributions are presented in the form of small-world networks, where nodes denote object categories and the edges linking object pairs indicate relationships. Thickness in edges implies the diversity relationships between object pairs. Same color conventions as (a) and (c) are applied. See the legend for triplet sizes.
like "driving", distinct from indoor scenarios featuring "chair" and "sitting." Employing CSEGG allows the company to focus on new objects and relationships while retaining indoor insights.
Another real-world example would be a construction site where a team of rohots is tasked with assembling various components build complex structure. Initially, during the foundation-laying phase, the robots are introduced to objects like "concrete blocks" and relationships like "stacking". As the construction advances the wiring and installation phase, they encounter new objects like "wires" and relationships like "connecting," which were absent from earlier stages. The SGG model deployed in these robots needs to adapt incrementally to learn these new relationships without forgetting the existing ones. This ensures that the robots can effectively communicate and collaborate while comprehending the evolving scene and tasks, optimizing their construction efficiency and accuracy.
A.1.3 Scenario 3 (S3): Scene Graph Generalization In Object Incremental Learning
We, as humans, have no problem at recognizing the relationships unknown objects with other nearby objects, even though do not know the class labels of the unknown objects. This scenario is designed to investigate whether the CSEGG models can generalize as well as humans. Specifically, there are 4 tasks in total with each task containing 30 object classes and 35 relationship classes. In each subsequent task, the CSEGG models are trained to continuously learn to detect 30 more object classes and learn to classify the same set of 35 relationships among these objects. The class selection criteria for each task follow the same as S1, where the selections occur uniformly over head, body, and tail classes. Example object classes and their label distributions for each task provided in Fig. S1 Different from S1 and S2, a standalone generalization test set is curated, where the objects unknown and their classes do not overlap with any object classes in the training set but the relationships among these unknown objects are common to the training set of every task. The CSEGG models trained after every task are tested on the same generalization test sets.
Here, we provide two real-world applications of Scenario in the deep sea and space explorations for autonomous navigation systems.
prime example is the ongoing research on deep sea exploration for autonomous navigation systems, where undiscovered flora and fauna reside beneath the ocean's 's surface. Encountering new and unidentified species becomes manageable through SGG's ability to understand spatial relations. The robot discerns the object's proximity or orientation even without precise identification of the species, enhancing its autonomous navigation ability. Likewise, in deep space exploration, SGG aids in recognizing spatial relationships with previously unseen space debris, aiding in maah-planning. In
18

---

sarging
000
lest
(a)
(b)
(d)
Figure S2: Data Statistics for all Learning Scenarios: (a) Distribution of objects in the entire training set of Visual Genome during Stage I of S1. (b) Distribution of relationships during Stage 2 for each task in S1. (c) Distribution of objects during Stage I for each task in S2. (d) Distribution of relationships during Stage 2 for each task in S2. (e) Distribution of objects during Stage 1 for each task in S3. (f) Distribution of relationships during Stage 2 for each task in S3 The numbers in brackets in the legend in (b-f) denote the number of training images in the particular task. Zoom in to the figure to get the exact labels and the frequency associated with them.
essence, SGG's relationship generalization empowers robots to navigate and plan routes in unfamiliar terrains, such as deep sea and deep space, where novel encounters demand adaptable responses.
A.2 Data Statistics
In this section, we provide various types of data statistics for all three learning scenarios. Specifically, we present statisties regarding the number of images, objects, and relationships involved in each task of each learning scenario. This information is provided in Fig. S2
19

---

Inpul image
Feature Extraction
Object/Subject Node Generation
Low enrer for task
Low emort for task
Output scene graph
Graph Construction
Predicate Node Generation
EWC
Realay
Natve
PocklNet
Figure S3: Introduction to backbone SGG models and continual learning baselines. We use Scene graph Generation TRansformer (SGTR) (32) as the SGG backbone (Sec. 3.2p. SGTR consists of four modules indicated by each blue box. Arrows indicate the signal flows among modules. (b) Four continual learning baselines are listed: EWC [24,, Replay [57], Naive (Sec. 3.2) and PackNet 44 (Sec. 3.2). 8% denotes the optimal network parameters after learning on task A. The arrows in colors indicate the shifts of network weights in the parameter space when learning Task for different baselines.
A.3 Implementation and Training Details
A.3.1 SGTR Backbone
(a), the approach uniquely formulates the task as a bipartite graph construction problem. Starting with a scene image (T,), SGTR utilizes a 2D-CNN and transformer-based encoder to extract image features. These features are then incorporated into a transformer- -based decoder, predicting object and subject nodes (O;). Predicate nodes (R;) are formed based on both image features and object node features, and a bipartite graph (G;) is constructed to represent the scene collectively. The correspondence between object nodes (ox) and predicate nodes (r%) is established using the Hungarian matching algorithm [26). Experimental results are based on the average over three runs, and the implementation leverages public source codes from 32 and 68 with default hyperparameters.
For SGTR in Fig. S
The SGTR is trained in two stages in supervised manner. In stage 1, only object detection losses in DETR is (6) applied on O,. In stage 2, only predicate entity prediction loss is applied on R; which can be further decomposed into L1 and GIOU losses for objeet/subjecc/predieese and cross-entropy loss for object/subjeec//prdicaal classification. In learning scenario S1, we skip Stage I, and directly load pre-trained weights of DETR for object detection the entire training set of Visual Genome (32]. In stage of S1, we freeze the feature extractor, and fine-tune the rest parts of SGTR for predicate entity predictions. As only relationship classes are incrementally introduced in S1, we freeze the entire weights of DETR for detecting all the objects in the scene over tasks. However, empirical results suggest that fine-tuning transformer-he based encoders in DETR helps downstream predicate predictions [32. Even with fine-tuning DETR in S1, we verify that there is minimal forgetting of detecting all objects in the scene over tasks (see Fig. S7 and Sec. A.6.2.. Thus, the forgetting observed in S1 could only be attributed to incremental relationship learning. In Stage I of S2 and S3 where object classes are also incrementally introduced over tasks, we load weights of the feature extractor, pre-trained on ImageNet [13], and fine-tune the entire DETR (6) over all the tasks. Stage S2 and $3 the same as $1.
localization [54]
Training the SGTR model involves two stages:
Object Detection Training: In this stage, a batch size of 32 is used. All methods optimized using the Adam optimizer with a base learning rate of x 10 and a weight decay of x 10-4, Object detection training is conducted only in the S2 and S3 scenarios. Each task in S2 is trained for 100 epochs, while each task in S3 is trained for 50 epochs. To expedite convergence, pre-trained weights on ImageNet are utilized before training Task for both S2 and S3.
In this stage, the entire SGTR model is fine-tuned while keeping the 2D-CNN feature extractor frozen. A batch size of 24 is employed, and the Adam optimizer is used with a base learning rate of x 10-5, In S1 and S3, each model is trained for 50 epochs per task, while in S2, 80 epochs per task are used. All models are trained on 4 A5000 GPUs.
SGG (Scene Graph Generation) Training:
20

---

Tasks
t-1
Dt-1
Et
Dt
Et+1
Dt+1
some
aine dala points
Cosa poins
trains
traits
Exemplar based methods (Replay, RAS)
Wi-1
W
(a)
M--1
Mj
Mc+1
taits with Lewo
Eaiis with
Lowe
Dt-1
Ft
Dt
1++1
Dt+1
traine
Calcolatin
tnairs
Calcolating Fisher Matris
tnaiss
Fieher
Marris
W Wt-l
W
EWC (b)
M₂-1
M++1
D1-1
D
D
tnvins
Wt-1
Wi-
prinos
W
PackNNct
(c)
M
M₁
Mp+1
Figure S4: Detailed Schematic of three CSEGG Baselines. In this schematic, we show the operations of each baseline at tasks L - 1, + 1. (a) shows the schematic for exemplar-based methods like replay and RAS. (b) shows the schematic for EWC. (c) shows the schematic for PackNet. Here, D; refers to the dataset of task i. E, denotes the exemplar set for replays at task W; is the weights M, at task
A.3.2 TCNN Backbone
As for TCNN, it employs Raster-RCNN 18 to generate object proposals from scene image (7:). The model extracts visual features for nodes and edges from these proposals. Through message passing, both edge and node GRUs output structured scene graph. Experimental results are based on the average over three runs.
Given scene image I, TCNN utilizes Faster-RCNN 18 to generate set of object proposals. The model subsequently extracts visual features nodes and edges from the set of object proposals. Finally, both edge and node GRUs output structured scene graph via message passing, The TCNN is trained two stages supervised manner. stage only object detection losses in Faster-RCNN 5 are applied on O, We use the cross entropy loss for the object class and L1 loss for the bounding box offsets. stage 2, the visual feature extractor (VGG-16 pre-trained on ImageNet [13) and GRUs layers trained predict the final object classes, bounding boxes, and
21

---

"Realistic Image of man on horse and house behind horse and man front of house."
"Realistic Image of man sitting on sofa and man sitting on chair and chair near sofa and person near man.
(a)
(b)
Figure S5: Visualizations of example generated images given the text prompt from our RAS. Each row presents two examples of generated images. Given the input text prompt on the left, the generated image is displayed on the right. These visualizations provide qualitative validations into the capability of RAS to produce diverse and contextually relevant images based on our designed text prompts.
relationship predicates using cross-entropy loss and L1 loss. In Learning Scenario (SI), similar to the implementation details of SGTR in Sec. A.3.1 we skip Stage 1, and directly load pre-trained for object detection on the entire training set of Visual Genome [32]. In
weights of Faster-RCNe stage of S1, we load the pre-trained weights of the visual feature extractor (pre-trained on ImageNet) and fine-tune the rest parts of the model. In stage of S1, we load the pre- pre-trained weights visual feature extractor (pre-traing trained ImageNet) and fine-tune the rest parts of the model. In Stage 1 of S2 and S3 where object classes are also incrementally introduced over tasks, we load weights of the Faster-RCNN, pre-trained on ImageNet 13. and fine-tune it over all the tasks. Stage of S2 and $3 follows the same training regimes as Stage 2 of SI.
optimized using the SGD optimizer with base learning rate x 10-2 and a weight decay of 1 x 10-4. For training on the entire VG dataset, we train the model for 60 epochs with batch size of 8 for both S2 and S3. To expedite convergence, pre- trained weights on ImageNet are utilized before training on Task for both S2 and S3.
Object Detection Training: All methods
SGG (Scene Graph Generation) Training: A batch size of 12 is employed, and the SGD optimizer used with a base learning rate of x 10-2 and a weight decay of x 10-1 In S1, each model is trained for 30 epochs. In S2, each model is trained for 15 epochs. In S3, each model is trained for 25 epochs. All models are trained on A5000 GPUs.
A.4 Mathematical formulations of Continual learning baselines
Here, we introduce mathematical formulations of these baselines: (1) To train Naive baseline on CSEGG, task t, take the previously trained model M,-1 with weights Wi-1 and train it on the dataset Dr at task t, to obtain weights W, of the model Mr. (2) For EWC, we use We and M, to calculate Fi+1, where F is the Fisher information matrix using the equation, F++ log
W.). During the training of task 1 + 1, we add LEwG to the training loss, where LEWe Σ F241 (Wa-1- W.)². (3) For PackNet, after training of task t, we take W,, and apply a pruning algorithm to obtain the pruned weights W/. At task ( 1, we obtain M by training M, with W/ on D (4) For Replay M, we create exemplar set E, the replay buffer by storing data points from D after training Mp-1 on task t - 1 At task t, we obtain weights We of M, by training Mt on E, and Dr.. (5) For the joint training, we train one model Mr jointly on all the datasets {D.}. where t E {1. ..., T). (6) For RAS_ GT, we adopt the similar math formulations as Replay@ @ M. See more details in Sec.
A.5 Evaluation Metrics
To assess the catastrophic forgetting of CSEGG models, we define Forgetfullness (F@K) as the difference in R @@K K on D,-1 between the CSEGG models trained at task / and task 1. An ideal CSEGG model could maintain the same RaK on Dz=1 over tasks; thus, F for all tasks. The more negative F is, the more severe in forgetting an model gets.
22

---

Order
Order 2
Order 3
30
30
30
30-
Avg R@20
Avg. R@00
Avg. R@20
Avg. R@20
20
20
10
10
10
T2
T3
T4
T5
T2
74
T5
(a)
Naive
(b)
EWC
(c) Replay@10%
(d) Replay8100%
F@20
-10
F@20
-10
F@20
F@20
-20
-20
-20
-30
-30
T1
T2
T3
T4
T5
T1
T2 T3 T4
T5
T1
T2
T3
T4
T5
T1
T2
T3 T4 T5
(e) Naive
(f) EWC
(g)
Replay@ 10%
(h) Replay@1009
Figure S6: Results of F@ K=20, Avg. R K=20 over tasks on CSEGG models with the SGTR backbone in Learning Scenario 1 with different permutations of task sequences. (a).(e) denotes Avg.R@K and F@ K for naive baseline. (f).(f) denotes Avg. K and F@K for EWC baseline. (g) denotes Avg.R@ @ K and F@ K for Replay @ 10% baseline. (d), denotes Avg.R@ K and F@ K for Replay @ 100% baseline. See Sec. for introduction to continual learning baselines. See Sec.
3.3 for explanations about evaluation metrics. X-axis indicates the task numbers. The higher Avg. @20, the better.
To assess the overall recall of CSEGG models over tasks, we also report the continual average recall (Avg. R R@ Avg. R@K K is computed as the average recall on all the data at the previous and current tasks D,, where E (1.2.....t).
To assess whether the knowledge at previous tasks facilitates learning the new task and whether the knowledge new tasks enhances the performances at older tasks, we introduce Forward Transfer (FWT @ K) 33 and Backward Transfer (BWT@) @ K) []. BWT@@ K is defined as BWT @K =
T ROKT, R@ Ki,, where T denotes the total number of tasks in a learning scenario and Ro RaK,, denotes the continual learning model trained after task i and tested in task j. FWT is defined as FWT@K --1 ΣT_ RüK,; büK,, where K,; is the @@K for an independent model with random initialization trained in task and tested in task i.
In learning scenario S3, we evaluate CSEGG models on their abilities to generalize to detect unknown objects and classify known relationships on these objects, in the standalone generalization test set over all tasks. To benchmark these, we introduce two evaluation metrics: the recall of the predicted bounding boxes on unknown objects (Gen Ribor @ K) and the recall of the predicted graph G; (Gen R@K K). As the CSEGG models have never been taught to classify unknown objects, we discard the class labels of the bounding boxes and only evaluate the predicted box locations with Gen Rabor @K. To evaluate whether the predicted box location is correct, we apply a hard threshold of Intersection over Union (IoU) between the predicted bounding box locations and the ground truth boxes. If the IoU exceeds the predetermined threshold, it is considered a true positive (TP). We used IoU thresholds of 0.3, 0.5, and 0.7 for our experiments. If the IoU exceeds the threshold multiple times for the same predicted bounding box, we consider it single positive prediction. This is because the metric aims to define the model's performance in locating bounding boxes of objects that have not been learned during training. Counting multiple times for the same box would be mislcading, as it would inflate the number of TPs and recall, while the actual number of unknown bounding boxes the model can generate might be low. The total possible positives (TP + FN) are determined by the total number of ground truth boxes in the image. In general, Gen Ryber @ K helps evaluate how well the CSEGG model locates unknown objects within an image. True positives (T represent successful identification of the location of an unknown object, while TP + FN represents all possible unknown objects the CSEGG model could locate. Thus, object classification labels are not required to calculate Gen Rebor @ K. Instead, we need the total number of ground truth bounding boxes in an image and the number of predicted boxes that meet the IoU thresholds.
23

---

mAp
Pre-Trained-DETR Task 1
Task 2
Task 3
Task
Task 5
Figure S7: mAP performance on the entire Visual Genome Test Set. We used a pre-trained
DETR checkpoint [2] well naive baseline of SGTR continually trained on each task from S1, evaluated them on the entire Visual Genome's ' test set, and reported their mAP performances. We observed minimal decrease in mAP across all tasks.
To assess whether the CSEGG model generalizes to detect known relationships over unknown objects, we evaluate the recall Gen R@ K the predicted relationships 7k only on correctly predicted bounding boxes.
A.6 More Result Analysis on Continual SGTR Methods
A.6.1
Results for Different Task Sequences
Recent work 663 has highlighted the consistent and substantial curriculum effects in class-incremental learning and continual visual guestion-answenn
tasks. Inspired by these findings, we conducted experiments to assess the impact of the curricula in the context of CSEGG. To delve into its potential influence, we trained baselines with three distinct task sequences for Learning Scenario Our results demonstrated that curriculum learning indeed shapes CSEGG performance within class-incremental settings. Notably, Fig. ())d), a large difference in Avg. @ 20 between Order and Order emerges for the Replay (100%) baseline. Similarly, Fig. S6(a) reveals substantial Avg. R @ 20 disparity between Order and Order 3 for the Naive baseline. This trend extends to F@ 20, as depicted in Fig. ())) These insights collectively affirm the significance of the curricula within CSEGG.
A.6.2 Minimal Forgetting in DETR
To validate the impact of fine-tuning the DETR model in training Stage 2 of learning scenario S1 on relationship predicate predictions and to ensure minimal forgetting occurs in object detection (Sec
A.3) we compare the mean Average Precision (mAP) for object detection on the entire test set of Visual Genome between the pre-trained DETR checkpoint from the paper 332 and the DETR models after fine-tuning on each task of S1.
As shown in Fig. S7 the results indicate slight decrease of 0.4 in mAP from the pre-trained checkpoint to DETR models over 5 tasks. This study provides evidence that fine-tuning DETR in SI has negligible effects on forgetting. The forgetfulness observed in S1 can only be attributed to relationship incremental learning.
A.7 Visualization Examples for All Learning Scenarios for Continual SGTR-based Models
In this section, present visualization examples from each learning scenario showcase the performance of the three continual SGTR-hased models, namely Replay 10%, EWC, and Naive, in three learning scenarios.
A.7.1 Learning Scenario 1 (S1)
From Fig. observe that, in Task 1, the ground truth scene graph contains triplets of "on" relationship: "plate on table" and "hair on women". After training on task 1, all three models (Replay @ 10%. EWC, Naive) can accurately predict these triplets of "on" relationship.
In Task 2, triplets of "has" relationship are introduced: "plate has food" and "women has hair". After training on task 2 data, the Replay@ @ 10% model successfully remembers the triplets of "on" relationship ("plate on table". "hair on women") from Task and predicts "women has hair". The Naive model forgets the triplets of "on" relationship and only predicts "women has hair". The EWC

---

Ground Truth
10
Tosk
Tosk
Tosk
Tosk
Taok
Time
Figure S8: Visualization example for Learning Scenario 1 (S1). The leftmost column in the figure displays the ground truth bounding boxes and scene graphs for each task in Learning Scenario 1 (S1). The remaining columns, from left right, represent the bounding boxes and scene graphs generated by each baseline model (Replay @ 10%, Naive, and EWC). In all the scene graphs, red boxes indicate objects, while green boxes represent relationships. The direction of the arrows between the red (object) and green (relationship) boxes indicates the subject and object ordering in the triplet. For example, in the scene graph predicted by the EWC model after Task 5, the triplet "trunk behind women", as the arrow goes from "trunk" to "behind" to "women". The time arrow on the left side of the figure demonstrates that the model is exposed new data over time, with new relationships incrementally added, as described in Sec. 3.1
model remembers the triplets of "on" relationships and predicts "women has hair". None of the models predict "plate has food".
In Task 3, triplet of "at" relationship is introduced: "women at table". After training on task 3 data, the Replay@ @ 10% model remembers the previous triplets ("plate table", "women has hair", "hair on women") and predicts "women at table". The Naive model forgets the previous triplets and only predicts "women at table". In contrast its previous performance, the EWC model forgets the previous triplets and only predicts "women at table".
In Task 4, triplets related to "of" relationship are introduced: "hand of women", "neck of women", "arm of women", and "head of women". After training on task data, the Replay @ 10% model remembers the triplets related "on" and "has" relationships ("plate table". "women has hair". "hair on women") from previous tasks but forgets the "at" relationship triplet ("women at table"). ) only predicts "head of women" from the triplets introduced in Task 4. The Naive and EWC models both forget the "at" relationship triplet from the previous task but predict "head of women" and "arm of women" from the triplets introduced in Task 4.
In Task 5, triplet belonging to the "bchind" relationship is introduced: "trunk behind women". After training task data, the Replay @ 10% model forgets the triplets related to "on" relationship ("plate on table", "hair on women") and only remembers the triplets related to "has" and "of" relationships ("women has hair", "head of women") learned from the previous task. It is not able to predict "trunk behind women". The Naive model, similar to its performance after previous tasks, fails to remember any triplets previously learned and only predicts "trunk behind women". The EWC model also fails to remember any triplets from the previous task and only predicts "trunk behind women".
A.7.2 Learning Scenario 2 (S2)
From S9 we observe that, in Task 1, the ground truth scene graph contains triplets: "man riding skateboard". "man above skateboard", and "shoe of skatcboard". After training task data,
25

---

Replay@10%
Task
Task
Time
Figure S9: Visualization example for Learning Scenario (S2). The leftmost column shows the ground truth bounding boxes and scene graphs in each task of Learning Scenario 2 (S2). The remaining columns, from left to right, represent the bounding boxes and scene graphs generated by each baseline model (Replay @ 10%, Naive, and EWC). In all the scene graphs, red boxes indicate objects, while green boxes represent relationships. As explained in Fig. [S8 caption, the direction of the arrows between the red (object) and green (relationship) boxes indicates the subject and object ordering in the triplet. The time arrow on the left side of the figure demonstrates that the model is exposed to data over time, with new objects and relationships incrementally added, as described in Sec. 3.1
all three models (Replay@ 10%, Naive, EWC) predict "man riding skateboard" and "man above skateboard". None of the models predict "shoe of skateboard".
In Task 2, new triplets introduced are: "leg of person", "person wearing sock", "person has head",, and 'head of person". After training on task 2 data, the Replay @ 10% model only remembers "man riding skateboard" from the previous task, forgetting "man above skateboard". Moreover, Replay @ 10% model can only predict "leg of person" from the triplets introduced in task 2. The Naive model forgets all the triplets from task 1 ( ("man riding skateboard", "man above skateboard") and only predicts "leg of person" from the triplets introduced in task 2. Similar to Naive model, EWC model forgets all the triplets from task ''man riding skateboard", "man above skateboard") and only predicts "leg of person" from the triplets introduced in task 2.
A.7.3 Learning Scenario 3 (S3)
Fig. S1O illustrates the performance of the Replay @ 10% and Naive models in locating unknown objects and recognizing the relationships between these objects and other nearby unknown objects. The ground truth in Fig. (S10 consists of three unknown objects: "mountain", "sheep". and "house", along with three relationships: "near", "bchind", and "infront of"" (between "mountain" " and "house"), and "near" relationship (between "sheep" and "house").
After training on Task data, the Naive model can accurately locate the objects "mountain" and "house". However, no new object, such as "sheep", is located by the Naive model after training on Task 2 data. After training on Task 3 data, the Naive model is also able to locate the object "sheep" in addition to "mountain" and "house". Even after training on Task 4 data, the Naive model continues to locate all three objects: "mountain", "house", and "sheep". contrast, the Replay @ 10% model, after training on Task 1 data, can only locate "mountain" and "house". This remains the same even after training on Task 2 and Task 3 data, where the Replay @ 10% model can still only locate the objects "mountain" and "house". However, after training Task 4 data, the Replay @ 10% model is able to locate all the objects: "mountain". ", "house", and "sheep".
Regarding relationship generalization on unknown objects, the Naive model, after training on Task can only predict the "near" relationship between the located objects "mountain" " and "house" out of the three possible relationships. This performance remains the same even after Task 2. However, after training on Task 3, the Naive model can predict the "behind" relationship in addition to the "near" relationship between the located objects "mountain" and "house". After Task 4, the Naive model can predict the "behind" and "near" relationships between the located objects "mountain" and "house", as well as the "near" relationship between the located objects "sheep" and "house".
26

---

Ground Truth
10p
Task
mountoia
Task
sheep K0
Task 3
bahked
Task
Time
Figure S10: Visualization example for Learning Scenario 3 (S3). The leftmost column shows the standalone ground truth bounding boxes and scene graphs in the generalization test set regardless of which task it is in Learning Scenario 3 (S3). The remaining columns, from left to right, represent the bounding boxes and scene graphs generated by each baseline model (Naive. Replay 10%). Similar to Fig. S8 and S9 the red boxes in all scene graphs indicate objects, while green boxes represent relationships. As explained in Fig. [S8 caption, the direction of the arrows between the red (object) and green (relationship) boxes indicates the subject and object ordering in the triplet. The time arrow on the left side of the figure demonstrates that the model is exposed to new objects over time as described in Sec. 3.1 For easy referral of object instances in the predicted scene graphs, we numbered the unknown bounding boxes in this figure, where the numbers are not actually present in the model predictions. Concretely, unk refers "mountain"; unk 2 is "house"; and unk 3 is "sheep".
In contrast, the Replay @ 10% model, after training on Task 1, can predict the "near" and "behind" relationships between the located objects "mountain" and "house". After Task 2. it can also predict the "infront of" relationship between the located objects "mountain" and "house" along with "near" and "behind" relationships. Even after Task 3, the Replay @ 10% model is still able to predict the "near", "behind". and "infront of" relationships between the located objects "mountain" " and "house". After Task 4, as it can now locate the object "sheep". the Replay @ 10% model can also predict the "near" relationship between the objects "house" and "sheep", in addition to the existing relationships between "mountain" and "house".
A.8 Ethical Concerns
The development and deployment of Scene Graph Generation (SGG) technology present potential negative societal impacts that warrant careful consideration (32) Firstly, privacy concerns arise as SGG may inadvertently capture sensitive information from images, potentially violating privacy rights and raising surveillance issues. Secondly, bias and faimess challenges persist, as SGG algorithms can perpetuate biases present training data, leading to discriminatory outcomes that reinforce societal inequalities. Misinterpretation and misclassification by SGG algorithms could result in misinformation and incorrect actions, impacting maaiiin.-making... The risk of manipulation and misuse of SGG-generated scene representations for malicious purposes is also concern. For example, attackers might manipulate scene graphs to deceive systems or disrupt applications that rely on scene understanding.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Sec. 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
28

---

Justification:
We don't oon't have any theoritical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sec. A.3
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
29

---

Answer: [Yes]
Justification:
We have provided the code as the Supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee 7/nips.com / cassiicyncccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Sec.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Sec. A.3 the experiments.
have provided information about the statistical significance of
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
30

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Sec.
A.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwuu.pcccppp...cccccccaa
Answer: [Yes]
Justification:
We have followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See. A.8
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We have used Stable Diffusion models for the image generation. So have ensured to put their safegaurds during the image generation process.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have credited the authors of the models and methods are using for our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
32

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Sec.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We have not done any crowdsourcing or research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We have not done any crowdsourcing research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "IT9859GEy9", "reviewer": "Reviewer_Y6eG", "summary": "This paper proposes a benchmark and a framework for incremental scene graph generation using continual learning. They curated the benchmark over an existing benchmark SGG dataset, VG. They proposed three learning scenarios such as relationship incremental,  scene incremental and generalization on relationship. Each of these learning regimes has different continual learning challenges such as learning new relationships, learning both new objects and relationship etc. The authors proposed a generative replay method which can deal with the catastrophic forgetting of continual learning. They reported detailed experimental results along with ablation studies.", "questions": "1. For Gen Rbbox@K metrics, could you please share some details on how you calculated the metric without the labels and how you ran the inference for predicting the relationship?", "limitations": "yes, the authors has addressed the limitations and potential negative societal impacts", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Applying continual learning to scene graph generation could greatly benefit tasks like robotic navigation etc. where the agent need to adapt in new scenerios with new objects and relationships \n3. The data and codes of the experiment will be publicly available", "weaknesses": "1. While the paper proposes a benchmark and framework for continual learning for SGG with detailed experiemtns and evaluation, the \npresentation of the paper is very difficult to follow. \n\n2. The problem formulation and formal mathematical definition of the overall methods are missing for the most of the parts. Section 3 has some definitions at the beginning and then explained learning scenarios and competitive baselines followed by the explanation of the evaluation metrics. Section 4 explains the continual learning framework 'Replays vai Analysis by Synthesis', but how this framework work in conjuction of SGG backbones (one-stage and two-stage existing SGG approach) needs to be clarified and formulated in formal equations.\n\n3. For Gen Rbbox@K metrics, could you please share some details on how you calculated the metric without the labels and how you ran the inference for predicting the relationship?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[Y6eG.Weakness.1-Poor paper presentation]** We appreciate the reviewer’s feedback regarding the presentation of our paper. We would like to note that other reviewers did not mention any issues with the clarity or presentation of our work. However, we value your input and it would be great if you could specify which parts of the paper were difficult to follow. This would help us make the necessary revisions and improvements for the final version. Thank you for your constructive feedback.\n\n**[Y6eG.Weakness.2-Lack of Mathematical Formulation]** For each scenario in CSEGG, we have task-specific dataset $D_t \\$, where  $t = 1, 2, 3, 4, 5$  for Learning Scenario 1 (S1), $t = 1, 2 $ for Learning Scenario 2 (S2), and $ t = 1, 2, 3, 4 $ for Learning Scenario 3 (S3).\n\nAt task $ t $, we take the previously trained model $ M_{t-1} $ (with weights $ W_{t-1} $) and train it on the dataset for task $ t $, $ D_t $, to obtain weights $ W_t $ and model $ M_t $. If using an exemplar-based approach (Replays and RAS), the model is also trained on $ E_t $, where $ E_t $ is the exemplar for task $ t $. Note that $ E_1 $ doesn’t exist as no exemplar is needed for task 1 in all scenarios.\n\nFor replay-based methods, we create exemplar $ E_t $ after training of task $ t-1 $ by storing data points from $ D_{t-1} $. Therefore, after training of task $ t $, we create $ E_{t+1} $ from $ D_t $. At task $ t+1 $, we obtain $ M_{t+1} $ by training $ M_t $ (with weights $ W_t $) on $ E_{t+1} $ and $ D_{t+1} $.\n\nFor RAS, we create $ E_{t+1} $ as shown in Fig 3 in the manuscript. Following that, at task $ t+1 $, we obtain $ M_{t+1} $ by training $ M_t $ (with weights $ W_t $) on $ E_{t+1} $ and $ D_{t+1} $.\n\nIf we are using EWC, we use $ W_t $ and $ M_t $ to calculate $ F_{t+1} $, where $ F $ is the Fisher information matrix using the equation, $ F_{t+1} = -E[\\frac{\\partial^2 }{\\partial W_{t}^2}log(M_t(x)|W_t)] $, where $ M_t $ is the model of task t and $ W_t $ is the weights of task $ t $. Now during training of task $ t+1 $, we add $ L_{EWC} $ to the training loss. We calculate $ L_{EWC} $ using the equation, $ L_{EWC} = \\sum_{}^{}F_{t+1}(W_{t+1} - W_{t})^2 $. While training for task $ t+1 $ we modify the loss $ L_{t+1}  = L_{train} +  L_{EWC} $. \n\nIf we are using PackNet, after training of task $ t $ we take $ W_t $ and apply a pruning algorithm to obtain $ (W_t)’ $, where $ (W_t)’ $ are the pruned weights. At task $ t+1 $, we obtain $ M_{t+1} $ by training $ M_t $ with $ (W_t)’ $ and training on $ D_{t+1} $.\n\nWe are also attaching a **Figure R1**, in the rebuttal pdf to show the entire schematic of CSEGG.\n\n\n**[Y6eG.Weakness.3-Gen Rbbox@K explanation]** Gen Rbbox@K is specifically defined for Learning Scenario 3 (S3) to evaluate the CSEGG model’s performance in locating bounding box locations. As described in Section 3.1, S3 consists of 4 tasks, with each task incrementally introducing 30 new objects. The relationship classes remain consistent across all tasks. In S3, there is a standalone test set where novel objects, absent from any training set of S3, appear with the same set of common relationship classes used in training.\n\nDuring evaluation, to calculate Gen Rbbox@K, for each predicted bounding box, we calculate the Intersection over Union (IoU) between the predicted bounding box and the ground truth boxes. If the IoU exceeds the predetermined threshold, it is considered a true positive (TP). We used IoU thresholds of 0.3, 0.5, and 0.7 for our experiments. If the IoU exceeds the threshold multiple times for the same predicted bounding box, we consider it a single positive prediction. This is because the metric aims to define the model’s performance in locating bounding boxes of objects that have not been learned during training. Counting multiple times for the same box would be misleading, as it would inflate the number of TPs and recall, while the actual number of unknown bounding boxes the model can generate might be low. The total possible positives (TP + FN) are determined by the total number of ground truth boxes in the image. \n\nThis metric helps evaluate how well the CSEGG model locates unknown objects within an image. True positives (TP) represent successful identification of the location of an unknown object, while (TP + FN) represents all possible unknown objects the CSEGG model could locate. Thus, object classification labels are not required to calculate Gen Rbbox@K. Instead, we need the total number of ground truth bounding boxes in an image and the number of predicted boxes that meet the IoU threshold as explained in Section 3.3. \n\n**[Y6eG.Question.1- Gen Rbbox@K explanation]** Same as **[Y6eG.Weakness.3]**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wu1K5SGMx8", "reviewer": "Reviewer_myHM", "summary": "The paper introduces the problem set up of continual learning for image scene graph generation. To this end, they  reorganize existing SGG datasets to establish this new benchmarks on three learning scenarios. Next, they present a \"Replays via Analysis by Synthesis\" (RAS) for generate diverse scene structure, followed by synthesizing scene graphs for replays.", "questions": "What do the authors think about creating a similar benchmark for dynamic scene-graphs such as Action Genome or others?", "limitations": "Apart from simple geometric/locations, most important visual relations are associated with action, and those actions evolve over time. A good examples of such dynamic scene graphs would be action genome, HOMAGE similar, and more recently EASG. e.g. a single image may not be enough to understand if a person is getting down or getting up for riding a horse. \n\nI'd like to understand the scenario for the continual learning setup can work for video/dynamic scene graphs. \n\n[1] Ji. et al, Action Genome: Actions as Compositions of Spatio-temporal Scene Graphs, CVPR 2022\n[2] Rai et al, \"HomeAction Genome: Cooperative Compositional Action Understanding\", CVPR 2021\n[3] Rodin, Furnari, Min et al, \"Action Scene Graphs for Long-Form Understanding of Egocentric Videos\", CVPR 2024", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The scenarios and benchmarks experiments using both transformer-based and CNN-based backbones are exhaustive. The proposed RAS outperforms baselines for two scenarios. \nRAS parses previous task scene-graphs into triplet labels for diverse in-context scene-graph reconstruction. RAS then synthesizes images with Stable Diffusion models for replays from the above re-compositional graphs.", "weaknesses": "For the continual learning setup, I'd imagine a video/dynamic scene graph generation will be more useful. However, currently it's focused on image scene graphs alone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[myHM.Weakness.1 - Focus on Image SGG explanation]** We agree with the reviewer that advancing continual learning in Scene Graph Generation (SGG) to include dynamic or video SGG is a natural and beneficial progression, as it more closely aligns with real-world settings. \n\nHowever, this research topic on continual learning in SGG is relatively unexplored, as no one has studied it before. Therefore, we need to begin with basic and straightforward settings, i.e. continual learning in scene graph generations on static images. Here, we aimed to establish a comprehensive benchmark of baselines and datasets to provide a foundation for the research community to start developing more advanced continual learning methods for SGG on static images. With all these foundations developed in this work,  we can then tackle more complex aspects, such as dynamic video SGG in the next phase.\n\n**[myHM.Questions.1 - Opinion on Dynamic SGG Benchmark]** We agree that creating a similar benchmark for dynamic scene-graphs is an excellent idea, and we definitely plan to extend our benchmark dataset and to incorporate dynamic scene-graph models. As mentioned in our previous response, we are the first team to address SGG in a continual learning setting, and our initial goal was to begin with basic and straightforward settings on static images. Only after this, we can move on to incorporate dynamic SGG models into our benchmark. Thank you for bringing Action Genome to our attention; it has inspired us for future developments in our benchmark. We will certainly cite this work in our final version.\n\n**[myHM.Limitations.1 - Continual learning for dynamic SGG]** We agree with the reviewer that advancing continual learning in Scene Graph Generation (SGG) to include dynamic or video SGG is a natural and beneficial progression, as it more closely aligns with real-world settings. As mentioned earlier, we are the first team to address SGG in a challenging continual learning setting, and our initial goal was to establish a comprehensive benchmark for image-level SGG models. Only from here, we can then move on to develop continual learning SGG in dynamic and video settings. \n\nWe thank the reviewer for the clear directions in continual learning for dynamic scene graphs. In response, we will include a paragraph in the related works section to survey all the dynamic scene graph generation works, such as the three works here: Action Genome, HOMAGE, and EASG. We will cite these papers in the related works. However, do note that all these three works do not study continual learning problems. Neither do these works look into the importance of studying continual learning in SGG in a dynamic setting. Expanding our benchmark to incorporate continual learning in dynamic SGG models will be a key focus of our future work. Thank you for the valuable suggestions and for highlighting important works in this area.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vuKZIi6fET", "reviewer": "Reviewer_XGhv", "summary": "This paper proposes a new task, Continual Scene Graph Generation (CSEGG), where SGG models must dynamically update to recognize new objects and relationships. It introduces a benchmark with three learning regimes: relationship incremental, scene incremental, and relationship generalization. In addition, the paper proposes a learning method called Replays via Analysis by Synthesis (RAS), which utilizes textual triplets to synthesize new images for model training. Performance evaluation on a dataset shows the effectiveness.", "questions": "1. What does the \"long-tailed distribution\" in L43 refer to? long-tailed predicate distribution or long-tailed object distribution?", "limitations": "The authors discussed the limitations in the appendix.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is clear and easy-to-understand.\n2. The effectiveness of key components has been verified.", "weaknesses": "1. The significance of CSEGG is limited. In CSEGG, the model should be trained for new categories. However, such training processes are unstable, considering the data, hyperparameters, and hardware requirements. What are the advantages of CSEGG compared to open-vocabulary SGG [a] and zero-shot SGG [b]?\n2. In RAS, the training targets are predicted by the previous SGG model, which may cause error propagation. Why not use a generative model [c] with fine-grained control signals (e.g., given boxes and captions)? \n3. The two-stage SGG baseline (IMP, CVPR'17) is very old. Why not use the latest models (e.g., PE-Net [d])?\n4. The model is evaluated on VG only. More experiments are needed to verify the generality, e.g., cross-domain evaluation.\n\n\n\n[a] Expanding Scene Graph Boundaries: Fully Open-vocabulary Scene Graph Generation via Visual-Concept Alignment and Retention. arXiv, 2023.\n[b] Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models. NeurIPS, 2023.\n[c] Open-Set Grounded Text-to-Image Generation. CVPR, 2023.\n[d] Prototype-based embedding network for scene graph generation. CVPR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[XGhv.Weakness.1 - CSEGG is limited]** We would like to address the concerns regarding the difference of CSEGG compared to open-vocabulary SGG [a] and zero-shot SGG [b]. \n\nFirst, we respectfully disagree with the reviewer that the significance of CSEGG is limited. The problem setting in open-vocabulary SGG [a] and zero-shot SGG [b] differs fundamentally from the problem setting addressed by CSEGG. In CSEGG, we aim to simulate scenarios where the “new” predicates or “new” objects encountered by the model are completely novel and have not been seen by any models before including the existing language models or models with multiple modalities. \n\nFor example, if a new drug called XYZ is discovered, this drug has not been introduced to any AI models, whether they are language, vision, or multi-modal models. As a result, all the zero-shot and open-vocabulary models referenced in the two citations would fail to identify this new drug, as there is NO prior knowledge about this drug to transfer knowledge at the first place.\n\nSecond, open-vocabulary SGG [a] uses a frozen text encoder with pre-learned weights, assuming prior knowledge of objects or predicates. Similarly, zero-shot SGG [b] relies on large language models for description-based prompts, assuming these models have encountered the new objects or predicates. In contrast, CSEGG simulates scenarios where completely novel objects or predicates are introduced. Relying on pre-learned information from large language models or frozen text encoders would violate the continual learning setting.\n\nBoth [a] and [b] are fantastic works. We will cite them and discuss the differences of these works from our work in the related work section. However, our work focuses on continual learning in SGG, which is NOT addressed by these methods as it is a different problem setting altogether.\n\n\n**[XGhv.Weakness.2 - Fine-grained controls for RAS]** We appreciate the reviewer’s suggestion. However, we would like to point out that even without fine-grained controls as suggested in [c], our approach has already demonstrated decent performance outperforming all the existing continual learning baselines.\n\nAs we are the first to address continual learning in SGG, we introduced a proof-of-concept model called RAS. Our main goal was to show that generative modeling can prevent catastrophic forgetting in CSEGG. \n\nWe agree that using generative models with fine-grained control signals like boxes and captions could improve our method. We will cite the suggested work in the related works section and incorporate it in our model in the final version. \n\n**[XGhv.Weakness.3 - Limited Choice of SGG Models]** We understand the reviewer's concern that the two-stage SGG baseline (IMP, CVPR'17) is old. However, with 1359 citations, it remains a pioneering model in SGG and a strong baseline for comparison in subsequent works. We included it to show that our method works on classical SGG models and that the CSEGG benchmark can be extended to other SGG models. Additionally, we included a one-stage baseline (SGTR, CVPR'22) to validate our benchmark with more recent models. \n \nWe agree that PE-Net [d] is a very interesting work, and we plan to try and include this in the CSEGG benchmark for the final version.\n\nWe included only one model from each category (one-stage and two-stage) due to hardware and time constraints. For Learning Scenario 1 (S1), we conducted 120 experiments per model, including 5 tasks for each of the 8 baselines (excluding joint models), with 3 runs per baseline for statistical significance. For Learning Scenario 2 (S2), we performed 48 experiments per model, involving 2 tasks for each of the 8 baselines, also with 3 runs per baseline. This totals 168 experiments per SGG model, and with 2 models, we conducted 336 experiments. Each experiment takes about 2 days on a single machine with 4 A5000 GPUs. Using 3 machines, training for S1 and S2 alone took approximately 7 months. Including Learning Scenario 3 (S3) and additional ablation studies, the entire set of experiments extended to around 9-10 months. Thus, we included only one model from each category (one-stage and two-stage) in our study. We invite the community to contribute by incorporating more SGG backbones.\n\n**[XGhv.Weakness.4 - No generality]** We agree with the reviewer that studying model generalization in scene graph generation is interesting. This includes training a SGG model in VG dataset and testing on other SGG datasets, such as OpenImages. However, we argue that model generalization and continual learning are completely different problems. In our work, we focus on continual learning. \n\nIdeally, we would like to include other SGG datasets such as OpenImages, for our Continual SGG problem. However, our current analysis is limited to the Visual Genome (VG) dataset due to hardware restrictions and time constraints as mentioned in **[XGhv.Weakness.3]**. \n\nWe appreciate the list of references related to open-vocabulary, open-set, and zero-shot problems in SGG. However, we would like to emphasize again that none of these works look into the problem of continual learning, which is the main focus of our work. We will cite these papers and discuss the differences from our work in the related work section. As mentioned in **[XGhv.Weakness.1]**, our work studies continual learning problems in SGG, which is valuable, unique, and challenging. It is different from open-vocabulary, open-set, and zero-shot SGG. \n\n**[XGhv.Question.1 - \"long-tailed distribution\" in L43]** We refer “long-tailed distribution” to distributions in both objects and relationships; and hence, predicates altogether. We will clarify this statement in the final version.  We present histograms of object and relationship distributions in each task of each learning scenario in Fig. 5 in the Appendix. The histograms show that the long-tailed distribution of objects and relationships (predicates) persists across all tasks and scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all the reviewers' feedback. We encourage reviewers to refer to the PDF file containing additional figures. To differentiate these new figures in the rebuttal from those in the main text, we have prefixed them with "R" in the rebuttal. For example, Fig R1 corresponds to Fig 1 in the rebuttal PDF. We've included a point-by-point response for each of the three reviewers.

Meta-review: The paper initially received mixed-to-negative reviews. While the reviewers appreciated the exhaustive experiments and the clarity in presentation, there were some concerns over (i) the problem statement, (ii) the limited explanation of the metrics, (iii) the lack of comparison with newer baselines, and (iv) the lack of fine-grained control signals. The authors' response addressed many of the concerns effectively. While concerns remain over the utility of the problem setting and somewhat limited baselines for two-stage networks, the paper offers an interesting perspective for SGG and will serve as a solid baseline for continual learning for SGG. The authors are strongly encouraged to incorporate the discussion from the rebuttal into the final version for completeness.
Decision: Accept (poster)