SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Authors: Nikolaos Chaidos
Angeliki Dimitriou
Maria Lymperaiou
Giorgos Stamou
Venue: ICML 2025 poster (2025)
Keywords: Scene Graph Retrieval
Unsupervised Graph Autoencoders
Visual Semantic Similarity
Primary area: deep_learning->graph_neural_networks
TL;DR: We present SCENIR, an unsupervised scene graph retrieval framework based on Graph Autoencoders, to efficiently tackle semantic image similarity.

Abstract: Despite the dominance of convolutional and transformer-based architectures in image-to-image retrieval, these models are prone to biases arising from low-level visual features, such as color. Recognizing the lack of semantic understanding as a key limitation, we propose a novel scene graph-based retrieval framework that emphasizes semantic content over superficial image characteristics. Prior approaches to scene graph retrieval predominantly rely on supervised Graph Neural Networks (GNNs), which require ground truth graph pairs driven from image captions. However, the inconsistency of caption-based supervision stemming from variable text encodings undermine retrieval reliability.
To address these, we present *SCENIR*, a Graph Autoencoder-based unsupervised retrieval framework, which eliminates the dependence on labeled training data. Our model demonstrates superior performance across metrics and runtime efficiency, outperforming existing vision-based, multimodal, and supervised GNN approaches.
We further advocate for *Graph Edit Distance* (GED) as a deterministic and robust ground truth measure for scene graph similarity, replacing the inconsistent caption-based alternatives for the first time in image-to-image retrieval evaluation. 
Finally, we validate the generalizability of our method by applying it to unannotated datasets via automated scene graph generation, while substantially contributing in advancing state-of-the-art in counterfactual image retrieval.
The source code is available at https://github.com/nickhaidos/scenir-icml2025.

SCENIR:
Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Nikolaos Chaidos'
Angeliki Dimitriou
Maria Lymperaion
Giorgos Stamou
Abstract
EmORREEVIT
Ours (SCENIR)
Despite the dominance of convolutional and transformer-based architectures in image-to- image retrieval, these models are prone to biases arising from low-level visual features, such as color. Recognizing the lack of semantic under- standing as a key limitation, we propose a novel scene graph-baser retrieval framework that em- phasizes semantic content superficial im- age characteristics. Prior approaches to scene graph retrieval predominantly rely supervised Graph Neural Networks (GNNs), which require ground truth graph pairs driven from image cap- tions. However, the inconsistency of caption- based supervision stemming from variable text en- codings undermine retrieval reliability. To address these, we present SCENIR, Graph Autoencoder- based unsupervised retrieval framework, which eliminates the dependence on labeled training data. Our model demonstrates superior perfor- mance across metrics and runtime efficiency, out- performing existing vision-based, multimodal, and supervised GNN approaches. We further advocate for Graph Edit Distance (GED) deterministic and robust ground truth measure for scene graph similarity, replacing the inconsis- tent caption-based alternatives for the first time in image-to-image retrieval evaluation. Finally, we validate the generalizability of our method by applying unannotated datasets via auto- mated scene graph generation, while substantially contributing in advancing state-of-the-art in coun- terfactual image retrieval. The source code is available at https://ithhb.co/niickaaaaaa icm12025.
TDP
TOP-2
TCn-3
Quary
Alldietirobertal
Al-Mini M-LE
Alt-M-Net base
TDP
TOP
Example of visual biases (color bias) in image retrieval when using visual models ours. Bottom: Example of the retrieval variability of SBERT models (top-1 retrieved items).
Figure Top:
1. Introduction
With the advent of deep visual models, transiting from Con- volutional Neural Networks (CNNs) to Vision Transformers (ViT), unprecedented performance has been reported across a variety of vision tasks. However, it is still unclear whether related models fully rely on correlational patterns or acquire an understanding of semantics, i.e. objects, attributes and their relationships. For example, CNNs lack understanding of contextual dependencies (Lin et al., 2020), while ViTs widely exposed to correlation biases (Ghosal & Li, 2024), revealing that true knowledge of visual semantics has not yet been conquered (Wang et al., 2023b). To this end, there is an extensive line of work exposing erroneous results on saate-of-the-an (SotA) visual models due to biases (Shi et al., 2022: Park & Kim, 2022; Wei al., 2023; Murali al., 2023; Menon et al., 2023; Izmailov al., 2024; Puaduraru et al., 2024; Zheng et al., 2025).
Artificiil Intelligence and Learning Systems Laboratory, Na- tional Technical University of Athens. Correspondence Niko- laos Chaidos <nchaidos@e wawiiidds@@ils.ccee.. .r.
We briefly demonstrate an example of biased image-to- image retrieval in Figure 1. Efficient- ViT (Cai et al., 2023) a SotA image classifier retrieves results that are predom- inantly based on color attributes, such as "black & white'
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
rather than semantics (as seen in the top-1 and top-2 po- sitions). This behavior strongly suggests that dispropor- tionate focus is placed visual features like color. In turn, semantic relationships are overlooked - i. the fact that the depicted group of people is riding a form of sports equip- ment (bikes, pairs of skis).
ing in different top-k matchings for a given query caption- as briefly demonstrated in Fig. 1. In general, SBERT mod- els tend to return disagreeing rankings, when compared to each other; in Fig. 2, we calculate the agreement in top- I retrieved results using some of the best-perforning SBERT models for embedding captions', As evidenced in the ma- jority of results, there is significant disagreement (240%) between the top-1 retrievals across models. Such inconsis- tencies in the supposed ground truth will inevitably cause error cascuding in the trained GNN, ultimately leading to inconsistent outputs, as we demonstrate empirically in Sec- tion 4.2. On top of that, language as ground truth modality should not be considered without apprehension. Captions are usually short descriptions, providing a very high-level representation of the image without comprehensive seman- tic details. Moreover, there no definitive way to measure the similarity between two sentences, leading dispute even among humans (Wang & Dong, 2020). All these is- sues suggest that deviating from captions as a ground truth GNN supervision signal is likely a more effective solution.
We argue that harnessing se- mantic information and mit- igating visual biases can be tackled by adopting scene graphs in visual pipelines, as they provide structured repre- sentations of images, where the objects, attributes and re- lationships are explicitly pre-
1000
MiniLM RoBERTa MPNet
1.000
0.605
0.597
0.95
0.5
0.605
1.000
0.582
0.10
0.597
0.582
1.000
0.00
sented (Chang et al., 2021).
MPNet
RoBERTa
MiniLM
As a result, when a scene graph is employed for vi- sion task, such as image simi- larity -the focus of this work- concepts drive the decision- making of the underlying model. Thus, advanced inter- pretability and robustness is offered (Wang et al., 2023b), while redundant visual details (lighting, camera angle, black & white image etc.) become less influential. Ultimately, the problem of tnage-to--mmae retrieval translates to that of scene graph retrieval.
Figure 2. Agreement he- tween top-1 I retrieved items with various SBERT models (MPNet: all- aal-mmpet-base-e RoBERTa: all all-distilroberta- v1, MiniLM: all-minilm-16- v2) for caption retrieval.
In this work, we propose a solution to bypass relying both on GED as well as image captions for GNN supervision. Instead, we favor unsupervised methods for scene graph retrieval, topic that has been largely underexplored in previous research. Additionally, we highlight the need to scrutinize the evaluation strategies used for assessing scene graph retrieval systems, as they often lack reliable and ro- bust ground-truth measures. To this end, we contribute to the following: 1 We propose SCENIR, an Unsupervised Graph Autoencoder for scene graph retrieval that eliminates the need for similarity labels, while surpassing supervised methods in both performance and computational efficiency.
However, this domain is not devoid of computational obsta- cles. Scene graph retrieval essentially requires solving graph matching and ultimately calculating Graph Edit Distance (GED) (Sanfeliu& & Fu, 1983), an NP-hard problem (Zeng et al., 2009). To accelerate GED calculation, Graph Neu- ral Networks (GNNs) have been utilized to acquire scene graph embeddings, facilitating efficient retrieval in lower- dimensionality spaces. Existing endeavors for GNN-basee GED acceleration harness pre-calculated similarity scores between graph pairs, which demands obtaining ground truth GED scores beforehand (Dimitriou et al., 2024).
We advocate the utilization of GED as the standard evaluation framework to compute reliable similarity scores without variability, therefore deterministically assessing the capacity of the models to capture important semantic infor- mation. 3
2
We present the extendability of our method on unannotated images via Scene Graph Generation (SGG) and offer use cases such as counterfactual image explanations.
Staying within the supervised training spectrum, and by acknowledging the computational restrictions of GED cal- culation, other works resort to the utilization of captions as the ground truth supervision signals for GNN training (Yoon et al., 2021; Maheshwari et al., 2021; Wang et al., 2023b). Caption matching is based on pre-trained Sentence- BERT (SBERT) models (Reimers& & Gurevych, 2019), which provide embedding representations of captions, allowing the computation of cosine similarity scores between their embeddings. Even this seemingly simple solution poses various disadvantages. To elaborate, there is non-negligible variability depending on the chosen SBERT model, result-
2. Related work
Graph Autoencoders (GAEs) are the center of work. Despite the novel contribution of our proposed model, we utilize and build upon well-known GAE architectures, such as VGAE (Kipf & Welling, 2016) which learns graph representations by attempting to reproduce their original ad- jacency matrix from latent node embeddings in an encoder- decoder fashion. The Adversarially Regularized Variational Graph Autoencoder (ARVGA) (Pan et al., 2018) further
Captions from 3000 images, from the PSG dataset(Yang et al., 2022) (under license CC-BY 4.0).
2

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
boosts the regularization of the latent embeddings via ad- versarial training. While both methods focus exclusively on topological aspects of the graph, the Graph Feature Autoen- coder (GFA) (Hasibi & Michoel, 2021) implements new decoder that solely regards node information, which learns to reconstruct the original feature matrix. In this paper, we propose a carefully designed combination and extension of architectural components to enhance the representative power of GAEs for semantic graphs.
we present a brief comparative use case within the counter- factual context. In doing so, we demonstrate that SCENIR is more competent in the inductive setting, a scenario not explored in their work.
3. Method
3.1. Notation & Problem Formulation
In this paper, we focus on image-to-mmage retrieval using scene graphs; thus, suitable dataset comprises (T, pairs, where / is image, and G the corresponding scene graph. Formally, scene graph G = (V, E) consists of set of nodes V and a set of edges E C V x V. It is specifically formulated as a feature matrix X E RRadd (information about objects in image I), and an adjacency matrix A E Ra** (information about relations in image I), where n |V is the number of nodes, and d the feature vector dimensionality per node. A scene graph retrieval framework receives as input the query pair (Iq: Gg) and the can- didate pairs {(Les Ge.). ), (Leu Gez).... (G,,,GG.. Ge.)]. The output is a permutation of the candidates { Gk,), (Ika, Gkz).... (Ik),...(II... (Ik.,, to approximate the ranking of a surrogate similarity measure sim(G G..)) defined on the scene graphs. For this similarity measure we utilize GED, as proposed in Dimitriou et al. (2024), where, instead of exclusively utilizing it for evaluating counterfactual explanations, we extend it to evaluate scene graph retrieval in general.
Graph similarity methods leverage the topological struc- ture and semantic content of graphs to measure their dif- ferences. While often a step in retrieval pipelines, their relevance here is complementary, frameworks are distinct field. Often employed in graph similarity endeavors, deterministic similarity/distance met- ries such as GED (Sanfeliu& & Fu, 1983) are computationally expensive, ultimately rendering the use of approximation frameworks like GNNs an imperative choice. Most widely used GNN models are supervised, i.e. they are trained us- ing either a large amount of pre-computed similarity labels (Bai et al., 201 0a;;; Zhuo & Tan, 2022), or ground-truth positive/negative pairs (Li et al., 2019; Ying et al., 2020). In contrast, SCENIR mitigates the extensive data requirements by employing unsupervised GAES to encode scene graphs and then compute embedding similarity. GED is only used in the evaluation stage as a ground truth measure.
scene graph retrieval
which aims
Scene graph retrieval though well-defined as a field, re- mains relatively underexplored in literature. In the general context of similarity, scene graphs have primarily been uti- lized for cross-modal text-image retrieval (Wang et al., 2019; Peng& & Chi, 2020; Zhong et al., 2021; Wang et al., 2023a). However, despite its semantic advantages, taage-to-image retrieval through scene graphs, the focus of our work, has received limited attention so far (Yoon et al., 2021; Mahesh- wari et al., 2021; Wang et al., 2023b). Yoon et al. (2021) (IRSGS) train a three-layer siamese GNN leveraging simi- larity labels derived from SBERT-embedded image captions. The same caption- caption-based similarities are treated as ground truth for evaluation. Their probabilistic training method requires labels that scale quadratically with respect to the available scene graphs. Our approach differs from their framework in two major ways: a) relying entirely on unsu- pervised models for greater efficiency in training time and label requirements, and b) using the more robust GED as ground truth. Recent work by Wang et al. (2023b) merges visual and graph features for multi-modal image retrieval using images and scene graphs input. Finally, the ap- proach of Dimitriou et al. (2024) (Graph Counterfactuals GC) explores GNN-based methods to produce counterfac- tual explanations (CEs) for scene classifiers, utilizing GED as the evaluation framework, similar to us. Although our primary focus is on scene graph retrieval rather than CEs,
Here, we harness GNNs to extract global embeddings for query and candidate scene graphs, and then calculate the fi- nal rankings through cosine similarity. Despite this retrieval pipeline being conceptually simple, using unsupervised rep- resentations for scene graph retrieval is surprisingly un- derexplored topic. To this end, our experimentation revolves around GAE models, which we integrate to our retrieval framework as sub-modules to compute embeddings.
3.2. Proposed Model
While existing GAE approaches have shown promise in graph-based tasks, they face key limitations with scene graph retrieval, including struggles with complex node-edge representations, limited expressiveness of inner- product decoders, and loss of discriminative power due to over- smoothing. Our proposed SCENIR (SCene-graph auto- ENcoder for Image Retrieval) framework addresses these challenges through a carefully designed combination of split encoders, MI.--aased decoders, and adversarial training.
Encoder The encoder comprises two GNN modules (GNN,, y and GNN,), whose goal is to encode the input graph into a learned latent space. It takes as input the scene graph's 's feature matrix X and adjacency matrix A,
3

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
fale?
Skale
board
64y
Iiln -ce-ssheet
beard
sea
persort Nind
st0re
chid
Encoder
Dhoody
Query (Go)
Canaidate SGs (Gra)
Quey
boland
Seard
SUF
dky
OWEY
GUIT
cocine Similrrty
GNN,
Peetig
Rank
persART
peraccT
MIM
OVEY
SMM
Test SGs
Figure 3. Overall Scene Graph Retrieval pipeline: training (top) and inference (bottom), with scene graphs linked to images in the dataset. architecture proposed SCENIR model depicted. The only loss term that not originate from the Discriminator or the Decoder's modules is LKL for the variational regularization, that is applied directly to the encoder output.
and outputs latent node embeddings matrix Z Rexd The authors of the original VGAE (Kipf & Welling, 2016) suggest leveraging a shared GNN layer followed by sepa- rate GNN, N, and N, for variational training, which is a widely adopted model option. We instead find that it is more beneficial to completely split GNN,, and GNN,, into two independent 3-layer branches.
product decoder, in VGAE and ARVGA (Pan al., 2018). This design choice allows the autoencoder to learn more sophisticated relations in the encoded latent space compared to a simple inner-product, which proves to have great im- pact in the case of semantic similarity between scene graphs. Mathematically. the formulation of the decoder is:
Ze = We,zWe,t1
The split between GNN,, and GNN, branches is moti- vated by the distinct roles of mean and variance parameters in variational inference, where mean embeddings capture structural features while variance embeddings model uncer- tainty. Thus, the independent branches allow these distinet aspects to be learned more effectively. Ablation studies (offered in App. D) showcase that this architectural choice enhances retrieval accuracy compared to the shared-layer approach. Formally, the encoder is expressed as:
Ap a (Z..Tee))
(2)
Z = W,,2W
where W. are learnable weights (bias and RELU activation included in the implementation). is the sigmoid function, Ap E R" xn, with Ap.if E [0, 1). is the predicted adjacency matrix and Z E T Igaxd the predicted feature matrix.
The incorporation of MLP instead of GNN in the decoder addresses the challenge of embedding oversmoothing that occurs when stacking multiple graph convolution layers (Li et al., 2018). Our choice of 2-layer MLP decoder, and 3-layer GNN encoder architecture additionally aligns with recent findings (Luo et al., 2024) that demonstrate the ef- fectiveness simpler, shallow architectures over complex ones in the majority of graph- tasks. This balanced design achieves both model expressiveness and computa- tional speed, demonstrating superior performance in graph retrieval, while also maintaining significant computational efficiency (further details in Section 4.4).
Z, = GNNN (GNN (X))))) Z, = GNN. (GNN..2 (GNN.,1 (X, A)))
All GNN functions can be implemented using any message- passing module, such as GCN (Kipf & Welling, 2017) or GIN (Xu al., 2019). Bias terms and RELU activation func- tions are included for each GNN layer in the implementation. Notably, Z, is only employed during training.
Decoder A key innovation in our architecture is the de- coder design, which comprises two parallel branches: an Edge Decoder and Feature Decoder, both implemented as 2-layer MLPs instead of the traditional no-parameter inner-
Discriminator
To enhance learned representation quality, we integrate adversarial training through a discriminator

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
module, as proposed by Pan et al. (2018). This module distinguishes between real samples from a prior distribution (typically Gaussian ~ (0, I)) and fake latent embeddings generated by the encoder. This adversarial component helps to further regularize the latent space and improve the overall representation quality. The discriminator is implemented as a 2-layer MLP, with a binary output (real/fake).
6K
# #NoDES
6.29
# EDCES
5.30
Count
4K
DENSITY
0.22
DEGREE
1.66
2K
0K
Training The model is trained end-to-end with a com- prehensive loss function that combines multiple objectives from all the aforementioned modules:
0
1
4
5
Max Path Length
Figure 4. Maximum path lengths and mean values for graph met- rics, for the preprocessed PSG graphs.
L [= + Ledgeerecon)
Azhade + X3LKL
where feeatreecn is the Mean Squared Error (MSE) loss between the original X and predicted y feature matrices, while Ledge. Ledge_recon precon is the reconstruction loss between the original A and the predicted A, adjacency matrices defined as .A) [log p [z.]] Lade is the adversarial loss implemented as Binary Cross Entropy Loss for the discrim- inator prediction, and L is the Kullback-Leibler Diver- gence of the latent embeddings to the prior Gaussian dis- tribution N(O, I). To further improve training stability, we implement Exponential Learning Rate Scheduler with gamma of 0.95, and we apply loss tradeoff terms X; (details in Appendix D) resulting in the final loss equation 3.
our approach, motivated by recent work adjacent to our field (Dimitriou et al., 2024) that emphasizes semantic similarity over low-level features, such as pixels. In accordance to their analysis and our experimental findings, GED's robust- ness eliminates ambiguity in generating golden rankings, unlike methods such as captioning (Fig. 2). Our experi- ments are based on an inductive retrieval setting, i.e. both the test query and candidates are not available to the model during training. We compute GED between all pairs of the 1K test graphs, and finally extract the ground-truth rank- ings. In total, we have 1K test queries, each accompanied by 999 retrieved and ranked objects. Using these GED rank- ings as ground truth, we evaluate all models on NDCG@K k, MAP@K, and MRR (k 1, 10).
Inference At inference time, global scene graph represen- tations are generated by applying sum-pooling to the latent node embeddings Z, from the trained GNN N₁ encoder (Fig. 3). This pooling strategy is selected for its proven superior discriminative power (Xu et al., 2019) in capturing graph- level representations (especially in cases with distinct node types, such as in scene graph representations), when com- pared to other pooling functions (e.g. mean or max).
We initially compare our proposed architecture, to SotA pre-trained Vision and Vision-Language (VL) mod- els, supervised GNNs, and basic GAEs. For the Vision mod- els, we use two pre-trained convolution- architectures, (Woo et al., 2023) and InceptionNeXt. Base (Yu et al., 2024), two pretrained ViT-based models, EfficientViT-L3 (Cai et al., 2023) and DeiT-IlI-Lare (Tou- vron et al., 2022), and four pre-trained VI. models, CLIP (ViT-L-14) (Radford et al., 2021), BLIP (base) (Li et al., 2022), BLIP-2 (COCO finetuned) (Li et al., 2023b) and AL- BEF (base) (Li et al., 2021). For the inference stage of vision models, extract the pooled last-layer feature vectors per image, while for the VL models we encode the query image and the candidate captions for image-tex retrieval. The final rankings are calculated via cosine similarity between these vectors.
Baselines
4. Experiments
Datasets For our experiments we leverage the PSG scene graph dataset (Yang et al., 2022), more curated version of the traditional scene graph dataset, Visual Genome (Krishna et al., 2017), as it is based on more advanced panoptic seg- mentation masks, containing almost 49K annotated image, caption and scene graph samples. We select 11 K scene graphs for training, and 1K scene graphs for testing. Statis- ties for the finally preprocessed scene graphs are presented in Fig. 4. We also experiment on images from Flickr30K (Young al., 2014) evaluate SCENIR in real-world use case, where caption and scene-graph annotations are unavailable, requiring us to generate synthetic ones (details about dataset preprocessing in Appendix A).
Regarding GNNs, we compare with IRSGS- BOGG-GGN//INN the current SotA for supervised scene graph retrieval retaining the original training details (Yoon et al., 2021). We also maintain caption similarity labels as a supervision signal for training, as originally proposed by the authors, harnessing two SBERT models (MPNet and RoBERTa), in order to assess whether SBERT disagreements affects the models' output. As for the EA,s, we evaluate VGAE and ARVGA,
Ground Truth and Evaluation We employ approximate GED as the ground truth distance/similarity for evaluating
5

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Table Retrieval results (as occurring with comparison to ground-truth GED ranks) using supervised and unsupervised GNNs, well Vision & VL models. Bold denotes the best overall result, underlined denotes best results within each model category.
NDCG
MAPT
MODEL
MRRI
@3
@5
1
@ 3
@5
@ 10
CONVNEXT
12.33
12.50
12.47
13.06
24.60
34.87
36.83
35.33
41.74
VISION
NNCEPTIONNEXT
12.63
12.67
12.90
13.64
23.90
33.92
36.03
35.33
41.27
EFFICIENT-VIT
13.47
13.49
13.40
13.89
25.70
35.86
37.27
36.36
42.66
DEIT-III
12.75
13.25
13.05
13.78
25.40
35.74
37.32
36.27
42.48
CLIP
15.63
14.64
14.58
14.93
28.80
38.21
39.38
38.25
44.79
BLIP2
13.64
14.23
14.60
15.29
25.90
36.62
39.14
38. 18
43.73
BLIP
15.62
15.19
1...00
15.38
28.50
38.57
40.01
38.86
45.10
ALBEF
15.77
15.37
15.36
15.69
28.10
38.58
39.99
38.73
45.11
IRSGS-GCN MPNet
27.50
25.84
24.70
23.53
41.50
51.33
51.82
49.18
56.42
OURS EXISTING GNN
IRSGS-GIN PNN
29.83
27.68
26.75
26.04
44.10
53.17
53.64
50.49
58.73
IRSGS-GCN Roberta
28.93
26.03
25.20
24.17
44.10
53.01
53.62
50.05
58.19
IRSGS-GIN
Roberta
29.64
27.96
27.13
26.00
43.80
54.12
54.22
50.67
59.16
VGAE-GCN
27.30
25.68
24.77
23.91
40.90
50.63
51.11
48.35
55.76
VGAE-GIN
27.27
26.26
25.27
24.41
41.80
50.84
51.43
48.73
56.58
ARVGA-GCN
26.59
25.10
24.26
23.84
39.70
49,46
50.07
47.44
$5.09
ARVGA-GIN
25.09
24.93
24.36
24.08
40.20
50.53
51.34
48.71
55.92
SCENIR-GCN
26.42
25.05
23.92
22.55
39.30
48.37
48.80
46.61
53.61
SCENIR-GIN
31.39
28.77
27.59
26.28
44.60
54.16
54.27
51.70
59.01
as well as our SCENIR, implementing each one with GCN and GIN modules ensure fair comparison.
Focusing on the GNNs, we observe that the supervised IRSGS performs competitively over the basic unsupervised VGAE and ARVGA, holding consistent advantage across all metrics. Notably, the GIN variants consistently outper- form their GCN counterparts for all architectures, which can be attributed to the theoretically proven expressive power of the GIN module (Xu et al., 2019). Indeed, our SCENIR-GIE variant achieves 2-3% increase in NDCG, 3-5% in MAP, and 2-5% in MRR compared its GCN variant. This pattern holds for both supervised and unsupervised approaches, sug- gesting that GIN's structural advantages generalize across different training paradigms.
Implementation We utilize PyTorch Geometric (Fey & Lenssen, 2019) for supervised and unsupervised GNNs, training them on a single P100 GPU. The mean-pooled node embeddings of the last layer serve as the graph em- beddings for IRSGS variants, while sum-pooling on the latent node embeddings (encoder output) is used for the GAEs. For Vision/VL pre-trained model implementations, we leverage open-source libraries (Wightman, 2019; Li et al., 2023a). For all models, predicted rankings obtained via cosine similarity through one-step retrieval (no secondary pre-ranking), to isolate the model's retrieval abilities. More details are presented in Appendix C.
Overall, our proposed SCENIR architecture demonstrates superior performance, surpassing even the supervised IRSGS across most metries. Our comprehensive unsuper- vised approach, combining an MLP-based decoder with a powerful GIN backbone, can effectively substitute for the absence of training labels while providing SotA retrieval.
4.1. Quantitative Results
In Table 1 we present test set retrieval results, comparing Vision and VI. models, supervised and unsupervised GNNs.
First and foremost, there is a clear performance gap be- tween Vision models and GNNs, with Vision models scor- ing about half as much in NDCG and 15%-20% less in MAP and MRR. Among Vision models, ViT-based archi- tectures (Efficient-ViT, DeiT--III) consistently outperform convolution-basee ones. While VL models show modest improvements of 1%-3% over pure Vision ones, both cate- gories fall significantly behind GNN-based architectures on scene graph (and equivalently (nageto--mmage) retrieval.
Ablation Studies To understand the contribution of each integrated architectural component, we conduct ablation experiments (Table 2), gradually subtracting the main con- stituents analyzed in Section 3.2. As evidenced in Table 2, the full benefit of the selected architectural components is only realized when properly integrated within our entire pro- posed framework. For example, excluding the MLP decoder immediately hurts performance by 44% in NDCG, MAP and MRR, while further removing the discriminator slightly
6

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Query
0.275
0.250
0.225
NDCGB5
0.200
Top-1
lap-2
Top-3
0.175
SCENIR
(-MLP Dec.)
YORK W SART
U5
0.150
((Eiscriminator)
DeT
0.125
(-GNN Edge. Dec.)
(-GNN Feat.
0.100
Number Layers
IRSIS IRSCS-RRBEE RUSSS-RBBRRRR
Figure 5. 5DCG@5 score for different number of GNN layers.
Ours (SCENIR)
deteriorates results. However, when subtracting the GNN Feature Decoder and the GNN Edge Decoder we observe some marginal gains, without however approximating the full SCENIR performance. Additional experiments with varying numbers of GNN layers in Figure 5 (further results in Appendix D) show that SCENIR uniquely benefits from deeper architecture, achieving optimal performance with 3 layers, while other models exhibit decreased performance beyond a single layer. This aligns with the path lengths of our scene graphs (Fig. 4), allowing SCENIR to achieve semantic representation without over-smoothing.
Figure 6. Qualitative results: VL (DeiT) supervised GNNs (IRSGS-GIN w RoBERTa-base caption similarity) vs SCENIR.
supervised GNNs, even though the top-1 result displaying cat is reasonable, subsequent ones depict dogs instead of cats, revealing the inability of the best IRSGS variant (with GIN and RoBERTa for caption embeddings) to return the key semantics of the query. Still, the semantic understand- ing is elevated in comparison to DeiT, with displayed con- cepts staying within the 'animal' category. On the contrary, SCENIR successfully returns cat images in all top-three positions, also qualitatively surpassing its competitors.
Table 2. Ablation study results showing the impact of different modules for SCENIR. Higher values better for all metrics.
MODEL
NDCG@31
M
MRR1
SCENIR
28.27
54.16
59.01
(-MLP DEC.)
24.95
49.27
54.95
(-DISCRIMINATOR)
24.37
49.02
54.22
Caption-driven disagreements Fig. 7, the impact of
GNNNN EDGE DEC.)
25.81
50.99
56.61
disagreeing SBERT captions and in turn ground -ruth ranks becomes prominent, proven by the diverging outputs of IRSGS variants. Specifically, RoBERTa and MPNet-based supervision signals lead to a discrepancy in all positions, with the MPNet-based one also returning an irrelevant im- age at top-2, which does not contain any of the 'bird' or "bench" query semantics. Other than the compared super- vised GNNs, EfficientViT, lying the Vision models cate- gory, is also entirely confused top-2 and top-3 positions, resulting in striking failure to bring images containing least one relevant semantic. On the contrary, SCENIR suc- cessfully retrieves bird-related images in all three positions.
(.GNN FEATURE DEC.)
25.68
50.63
55.76
4.2. Qualitative Results
Some qualitative results that demonstrate the superiority of approach are presented in Figures 6 and 7. In Figure 6, DeiT, VL model, completely fails to retrieve rele- vant image to the query, which depicts a black cat wearing a white bow tie. Derailed by color-related black& white patterns, DeiT returns a boy wearing a tuxedo top-1 posi- tion. Failed retrievals follow the top-2 and top-3 places, portraying people in athletic situations, the semantics of which are totally unrelated to the query ones. It is uncertain how DeiT decision-making works in these situations, since the query image accurately falls within DeiT's pre-training distribution on ImageNet-IK (Deng et al., 2009) (which includes cat classes, denoting that related features should be well-imbued within De representation). Regarding
4.3. Extendability
In-the-wild Retrieval To validate our model's practical applicability. we evaluate SCENIR's performance on im- ages from Flickr30K. This represents a more challenging real-worke scenario where ground truth scene graphs, or cap- tions are unavailable. We employ PSGTR (Yang et al., 2022)

---

its between them; particularly, graph edits between scene graphs to instruct CEs are suggested in the SotA framework of Graph Counterfactuals (GC) (Dimitriou et al., 2024). SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Query
Table Counterfactual Scene Graph Retrieval on PSG.
MODEL
NDCG(B)@ It
MAP(B)@31
MRR(B)1
VGAE
8.79
11.09
14.23
ARVGA
8.2
10.63
13.79
Top-1
Top-2
Tup-3
IRSGS
8,9
11.63
14.96
GC
7.0
8.7
10.67
Ethcentti
SCENIR
9.7
11.83
14.99
IRSCS-MI Pool
Due the critical role of the scene graphs in this use case, demonstrate the versatility of SCENIR when adopted as retrieval component in a CE pipeline: given query scene graph in class A we seek the most similar scene graph of class * A according to Places365 classifier (Zhou et al., 2017), which ultimately serves as the CE between the cor- responding images. For evaluation metries, we utilize the binary versions (denoted by 'B'), introduced in GC, where only the top-1 ground-truth instance considered relevant? Unlike GC's supervised transductive approach that requires test graphs during training, our evaluation follows a more challenging and realistic inductive setting with completely unseen test graphs. As shown in Table 4, SCENIR outper- forms all supervised and unsupervised baselines along with the previous SotA GC framework (which is optimized for calculating CEs) across all metrics, illustrating our superi- ority in counterfactual retrieval, without requiring explicit supervision or exposure to the test set during training.
RISOS-ROBE-R
Ours (BCENIR)
Figure 7. Additional qualitative results. Inconsistent ground truth matchings lead inconsistent IRSGS-GIN outputs.
for automated scene graph generation and BLIP- -.PP-Captioner-- Base (Li et al., 2022) for caption generation to process the raw images. As shown in Table 3, SCENIR maintains its superior performance even this challenging setting, achieving the highest scores across all retrieval metrics. These results demonstrate SCENIR's robustness and gener- alizability beyond curated datasets, rendering it particularly valuable for real-world scene graph retrieval applications.
Table 5. Complexities scene graph retrieval frameworks, with respect to the dataset size (PSG dataset, 11klk train/test graphs).
MODEL
PREPROC.
TrAINING
INFERENCE
TOTAL TIME
GC IRSGS SCENTR
O(n²)
O(z²)
O(n)
~ hr.
Table Image Retrieval without pre-annotated scene graphs and captions on Flickr30K.
O(n²)
O(n)
O(n)
50 min.
O(n)
O(n)
O(n)
8 min.
MODEL.
3
MAP@31
MRR
IRSGS
21.02
49.08
54.30
4.4. Computational speedup
VGAF
17.92
44.81
50.45
ARVGA
17.51
44.17
49.54
As shown in Table 5, SCENIR achieves significant computa- tional advantages over competitors. Prior SotA supervised methods like IRSGS and GC require quadratic complexity with respect to the sampled PSG dataset size in preprocess- ing and/or training with runtimes of ~550 minutes and ~33 hours respectively. SCENIR though maintains linear com- plexity, executing the entire pipeline just ~ minutes. This efficiency stems from our unsupervised approach that eliminates the need for expensive preprocessing of similarity
SCENIR
22.75
50.31
56.20
Counterfactual Retrieval Retrieval with emphasis on depicted semantics objects and relations is essential in tasks such as counterfactual explanations (CEs) (Browne & Swift, 2020). Semantic-driven CE frameworks identify the minimum changes required to transit between classification labels. For image classifiers, this involves finding the most similar image from a different class and computing the ed-
Thee CE setting instructs that only the closest instance is rele- vant, due to minimality constrains of counterfactual theory.
8

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
labels or caption embeddings. The linear complexity across preprocessing, training, and inference makes SCENIR par- ticularly suitable for large-scale retrieval applications.
in deep neural networks, 2020. URL Utthtts:://aran org aaa/bb///2 /201..001211111 10076.
Cai, H., Li, J., Hu, M., Gan, C., and Han, S. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. 2023 IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 17256-17267, 2023, URL https: /ppi. larppissemannnn semanticscholar. org/ / /tt//CoruussD 2670221226792288
5. Conclusion
In this work, we propose SCENIR, a novel unsupervised framework for scene graph retrieval based on graph au- toencoders. We emphasize the importance of using Graph Edit Distance, a deterministic graph similarity algorithm, for evaluating scene graph retrieval. Our proposed model achieves superior performance in retrieval metrics when compared to SotA Vision and VL models, as well as su- pervised GNNs, while also being significantly faster than its GNN competitors. The qualitative results showease the robustness and effectiveness of our approach in retrieving relevant images, contributing to the underexplored domain of scene graph retrieval and highlighting the potential of unsupervised approaches in this field. Through our extend- ability experiments on unannotated datasets and counterfac- tual explanation applications, we further highlight the broad applicability of our framework in real-world scenarios.
Chang, X., Ren, P., Xu, P., Li, Z., Chen, X., and Haupt- mann, A. G. A comprehensive survey of scene graphs: Generation and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:1-26, 2021. URL https: /api. semanticscholas org CorpusID: :2:544458:
Deng. J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. pp. 248-255, 2009, doi: 111.1///VRR.220.5206666
Dimitriou, A., Lymperaiou, M., Filandrianos, G., Thomas, K., and Stamou, G. Structure your data: Towards se- mantic graph counterfactuals. In Forty-first International Conference on Machine Learning, 2024.
Acknowledgments
Fey, M. and Lenssen, J. E. Fast graph representation learn- ing with pytorch geometric. ArXiv, abs/1903.02428, 2019. URL attps://app semanticscholar.
This work was supported by the Hellenic Foundation for Re- search and Innovation (HFRI) under the 5th Call for HFRI PhD Fellowships (Fellowship Number 19268). We thank all reviewers for their insightful comments and feedback.
Ghosal, S. S. and Li, Y. Are vision transform- ers robust to spurious correlations?, 2024. ISSN 1573-1405. URL attps://doi.oo 1007/ s s11263-028 01916-5.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning and Image-to- Image Retrieval. There are no societal consequences of our work concerning the utilization of data such as image datasets, which are publicly available and of general use.
Hasibi, R. and Michoel, T. A graph feature auto- encoder for the prediction of unobserved node fea- tures on biological networks. BMC Bioinformatics, 22, 2021. URL httts:// htpp.api..s// wet.iiseeaaniiicciicce....... org/ Corpus TD : 240006620.
References
Bai, Y., Ding, H., Bian, S., Chen, T., Sun, Y., and Wang, W. Simgnn: A neural network approach to fast graph simi- larity computation. Proceedings of the Twelfth ACM In- ternational Conference Web Search and Data Mining, 8018a. URL https::tpttt.........t /ppi. swwanticcinno icscholar. Ptc/CooppuID::55220
Izmailov, P., Kirichenko, P., Gruver, N., and Wilson, A. G. On feature learning in the presence of spurious correla- tions. NIPS '22, Red Hook, NY, USA, 2024. Curran Associates Inc. ISBN 9781713871088.
Kipf, T. and Welling, M. Semi-supervised classification with graph convolutional networks. In Proceedings of International Conference on Learning Representations, ceemanticscooaat
Bai, Y., Ding, H., Gu, K. Sun, Y., and Wang, W. Learning-basee efficient graph similarity com- putation via multi-scale convolutional set matching. 2018b. URL https: /appi.apppapp semantisscholar. org/CorpusID: :tC.CoppuSI::1092277710 210928750.
2017. URL nittps://at org/ CorppsID::1142111
Kipf, T. N. and Welling, M. Variational graph auto- encoders. In Proceedings of NIPS' Bayesian Deep Leam- ing Workshop, 2016. URL https: /tt.aaixvvvv..o/////// abs/1611.073C.
Browne, K. and Swift, B. Semantics and explanation: why counterfactual explanations produce adversarial examples

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J,, Shamma, D. A., Bernstein, M. S., and Fei-Fei, L. Vi- sual genome: Connecting language and vision using crowdsourced dense image annotations, 2017. ISSN 1573-1405, URL https: /tts://ddo...o 016 11126-006-0990
Maheshwari, P., Chaudhry, R., and Vinay, V. Scene graph embeddings using relative similarity supervision. Proceedings of the AAAI Conference on Artificial In- telligence, 35(3):2328-233
May 2021. ISSN 2159-
5399. doi: 10.1609/aaii 16333. URL http:
//dx. //tp./dii.rgg...111111.... www.ooggg/..... wat.:/1aa...........///// aaai wwai.v33i....................... w35... 163333.1.33.
Menon, S., Chandratreya, I. P., and Vondrick, C. Task bias contrastive vision-language models. In- ternational Journal of Computer Vision, Decem- ber 2023. ISSN 1573-1405. doi: 10.1007/ s1 012226622-01945555 URL /ttp:///xxxooi.ooo 0..1007/s111233-222 00923-00-14-0....
Li, D... Li, J. Le, H., Wang, G., Savarese, S., and Hoi, S. C. LAVIS: one-stop library for language-vision intelligence. pp. 31-41, Toronto, Canada, July 2023a. Association for Computational Linguisties. htts: //aacaantoooo/
org/2023 acl demo. 3.
Li, J., Selvaraju, R. R., Gotmare, A. D., Joty. S. R., Xiong. C., and Hoi, S. C. H. Align before fuse: Vision and language representation learning with momentum distillation. 2021. URL https: //api. semanticscholest 236034189.
Murali, N., Puli, A., Ranganath, R., and Bat- manghelich, K. Beyond distribution shift: Spu- rious features through the lens of training dynam- ics. Transactions on machine learning research, 2023, 2023. URL https: api semanticscholas org/ /CoppooTD:25003999
htw.://ccccola..ooooooopsst org//orpssD:
Li, J., Li. D., Xiong, C., and Hoi, S. C. H. Blip: Bootstrapping language-image pre-training for uni- fied vision-language understanding and generation. 2022. URL https:/ sea...sstttiicccoo....... org/Coo:pSSD::24411
Pan, S., Hu, R., Long, G., Jiang, J., Yao, L., and Zhang, Adversarially regularized graph autoencoder for graph embedding, 2018.
Park, N. and Kim, S. How do vision transformers work? 2022.
Li, J., Li, D., Savarese, S., and Hoi, S. C. H. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. 2023b. URL https: /api. semanticscholar. org
Peng, Y. and Chi, J. Unsupervised cross-media retrieval using domain adaptation with scene graph. IEEE Transactions on Circuits and Systems for Video Technology, 30:4368-43799 2020. URL https:
//api. www.nticsccoo..................e www.aattccccola.oooooooooooooppp org/CorppiI 209103205.
Li Q. Han, Z., and Wu, X.-M. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the Thirty-Second AAAI Conference on Ar- tificial Intellgence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium Educational Advances in Artificial Intelli- gence,
Puaduraru, C. D., Buarbualau, A., Filipescu, R., Nico- licioiu, A. L., and Burceanu, E. Conceptdrift: Un- covering biases through the lens of foundation models. 2024. URL https: http://ppps///////eee pesmanticschllas org http:CooopIII::23545555
8/EAA/18.. AAAI Press, 2018. ISBN 978- 1-5-1--7735--000 800--.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021.
Li, Y., Gu, C.. Dullien, T., Vinyals, O., and Kohli, P. Graph matching networks for learning the similarity of graph structured objects. 2019. URL https: //api. wwwant org/CorpusID: 139102215.
Reimers, N. and Gurevych, I. Sentence- bert: Sentence em- beddings using siamese bert-networks. In Conference Empirical Methods in Natural Language Processing, 2019. URL https: / //pii. semanticschen org/ http/CoppsID:::000 646009,
Lin, X., Ma, L,, Liu, W., and Chang, S.-F. Context- gated convolution, 2020. URL https:t///// /doi.org/////o/oo//o//ooo//o/ 10. 0000/978-30 111999978830---0 -00000000-55555.2
Luo, Y., Shi, L., and Wu, X.-M. Classic GNNs are strong baselines: Reassessing GNNs for node classification. 2024. URL https://///// https:///peerevee..roo Forum?
Sanfeliu, A. and Fu, K.-S. A distance measure between attributed relational graphs for pattern recognition. JEFE Transactions on Systems, Man, and Cybernetics, SMC-13 (3):353-362, 1983. doi: 10.1109/TSMC. 9983.63116....
d=xx1]jKdGed
10

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Shi, Y., Daunhawer, I., Vogt, J. E., Torr, P. H. S., and Sanyal, A. How robust is unsupervised representation learning to distribution shift? 2022. URL https: //api. www.nt 254823652.
Yang, J., Ang, Y. Z., Guo, Z., Zhou, K., Zhang, W., and Liu, Z. Panoptic scene graph genera- tion. In European Conference on Computer Vision, 2022. URL https: htp.. www..ssemmtticcchhoo org/CorpusTD::0 (10orppuTTD:2101863333
Org/CorpusID:
Touvron, H., Cord, M.. and J'egou, H. Deit ii:: Revenge of the vit. 2022. URL https: //api. serinteeeantttmmmm
Ying, R., Lou, Z., You, J., Wen, C., Canedo, A., and Leskovec, J. Neural subgraph matching. ArXiv, 0bs/2007.03099 2020. URL https:
248178188.
www.ssmatnncccoaar...r..... 220381221.
Wang, G., Shang, Y., and Chen, Scene graph based fusion network for image-text retrieval. 2023 IEEE International Conference Multimedia and Expo (ICME), pp. 138-143, 2023a. URL https: //api. www.sttesscsnnnnoeeeet 257632083.
Yoon, S., Kang, W. Y., Jeon, S., Lee, S., Han, C., Park, J., and Kim, E.-S. Image-to-image retrieval by learn- ing similarity between scene graphs. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12): 10718-10726, May 2021. ISSN 2159-5399, doi: 10.1669/2aii URL http: doi. org 0055/10.1609/aaa0 135112.17288
org/CorpusID:
Wang. J. and Dong, Y. Measurement of text similarity: A survey. Information, 11(9), 2020. ISSN 2078-2489, doi: 10.3390/info1 1090421. URL https: com/2078-201 200--22499/10 /119////// //21.
Young. P., Lai, A., Hodosh, M., and Hockemmaier, J. From image descriptions to visual denotations: New similarity metries for semantic inference over event descriptions. Transactions of the Association for Computational Lin- guistics, 2:67-78, 2014. doi: 10.1162/tael...00166. URL https: / hatp//allaltee awwaacaaaatolooooooooooooooo wwwplllyyyoooogggggoo/oo wwwg/1g/.. 20te:/214110000 4-1006/.
Wang, S., Wang, R., Yao, Z., Shan, S., and Chen, X. Cross Pross-modal scene graph matching for relationship-aware image-text retrieval. 2020 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 1497-1506, 2019. URL https: http::::/ppaaammmm... wetpp.ssematiicoool..... org Veg/Cozze Cttp/CrrpssDDD:. 204402762.
Yu, W., Zhou, P., Yan, S., and Wang, X. Inception- next: When inception meets convnext. 2024 IEEE/CVF Conference Computer Vision and Pattern Recogni- tion (CVPR), pp. 5672-5683, 2024. doi: 10.1109/ CVPR52733.2024.0077
Wang, Y., Dai, P., Jia, X., Zeng, 7., Li, R., and Cao, X. Hi-sigir: Hierachical semantic-guided image-to- image retrieval via scene graph. Proceedings of the 31st ACM International Conference on Multimedia, 2023b. URL https: semppeissaaniiichol.... org/CorpusIDD 264492649.
Zeng. 7.,, Tung, A. K. H., Wang, J., Feng, J., and Zhou, L. Comparing stars: on approximating graph edit distance. Proc. VLDB Endow. 2(1):25-36, aug 2009. ISSN 2150- 8097. doi: 10.117788/1688767116666666
Wei, J., Narasimhan, H., Amid, E., Chu, W.S.,, Liu, Y., and Kumar, A. Distributionally robust post-hoc classifiers under prior shifts, 2023.
URL https: / //doi. oiddi.orgg/0.. 14 777 /1////61666272 687631
Zheng, G., Ye, W., and Zhang, A. Benchmarking spurious bias in few-shot image classifiers. pp. 346-364, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031- 72989-8.
Wightman, R. Pytorch image models. https: -mae--immemem.mee......
com/ htt:/wiight www.th/nann/aao..mt..oo.....nana 2019.
Woo, S., Debnath, S., Hu, R., Chen, X., Liu, Z., Kweon, L.-S., and Xie. S. Convnext v2: Co- designing and scaling convnets with masked autoen- coders. 2023 IEEE/CVF Conference
Zhong, X., Yang, Z., Ye, M.. Huang, W., Yuan, J., and Lin, C.-W. Auxiliary bi-level graph representation for cooss-modal image-text retrieval. 2021 IEEE Interna- tional Conference on Multimedia and Expo (ICME), pp. 1- 6, 2021. URL https:ttp/// //tts://pppp //ppi.....//////// semant semanticscholar: org /coooocsDD:23227473300
Computer Vi-
sion and Pattern Recognition (CVPR), pp. 16133-16142, 2023. URL https: httppaaappaaa///a///.aa/ semant icchoocc http://oppuI:::55333::
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor- ralba, A. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
Xu, K., Hu, W., Leskovec, J., and Jegelka, S. How powerful are graph neural networks? 2019. URL https:// openrevicn met/forum?? d=ryGs 61A5km.

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Zhuo, W. and Tan, G. Efficient graph similarity computation with alignment regularization. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp.
Curran Associates, Inc., 2022.
12

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
A. Dataset Preprocessing
For the preprocessing of the PSG dataset, we removed the isolated nodes (no incoming and no outgoing edges) of all the scene graphs, mainly for two reasons: Firstly, the isolated nodes do not contribute in any way during the propagation through graph convolutional layers (neither do they provide, nor receive any information). Secondly, the PSG dataset defines special type of grouped node (noted as "merged"), which basically represents multiple nodes of the same type. We only retain these merged nodes, implicitly keeping all the important information (object types and interactions), without obstructing the training process of the GNNs. Examples of the final scene graphs can be seen in Fig. 11. We used 768- -68-dimensional Sentence-Transe Transformer embeddings for the 189 object and predicate classes to construct the feature matrix X for each scene graph, which is required as input for all the GNN models.
B. Ground Truth and Retrieval Metrics
For the computation of the ground-truth GED scores, we need to define costs for each edit operation on the input graphs iussertio,,dlltionn and substitution). Specifically, we define the node substitution cost as the cosine distance between the node embeddings, while the node insertion/deletion cost is defined as the cosine distance to the mean node embedding (average of all the 133 object class embeddings). Additionally for the retrieval metrics, we focus on the top-50 items out of the 999 retrieved by GED to avoid inflated MAP and MRR scores. For NDCG, we scale the inverse GED score to a range of [1, 10] manage outlier instabilities.
It is worth noting that the retrieval metrics behave differently, when changing the number of top-k retrievals. Specifically, we can see in Table that NDCG steadily decreases as k goes from 1 to 10. This is expected, as it is harder task to have perfect top-m retrieval, than a perfect top-k retrieval, where m > k. This is not the case with MAP, where we have significantly lower scores when a: = 1, compared to = 3, 5, 10. This exception is a direct result of the definition of MAP, because it only considers whether a retrieved object is relevant or not. It does not use any relevance scoring system, like NDCG, something that renders it vulnerable to situations where the top retrieved items vary in relevance. this case, if there are highly relevant retrieved top-1 items, NDCG increases according to their relevance score, but MAP will remain constant independent of their relevance. This phenomenon is especially evident with MAP@1 1, because the final score is entirely determined by single retrieved item.
C. GNN Training Details
Regarding the Graph Autoencoders, they were all trained for 30 epochs, with batch size 64, Adam Optimizer (Ir = 0.001, 31 = 0.9, 832 = 0.999, weight decay = 0.01), 1000 latent space dimension, 32 output dimension for the Edge Decoder, 768 output dimension for the Feature Decoder, and output dimension for the Discriminator (real/fake). Concerning the models that employ adversarial training, we followed the training algorithm proposed in Pan et al. (2018), with two separate AdamW optimizers, one for the Discriminator, and one for the rest of the model parameters. Also, we used an Exponential Learning Rate Scheduler (7 0.95), and Loss Tradeoff terms in order to stabilize the training. Experiments for Loss Tradeoff terms can be found in the next section.
0.30
0.20
0.26
0.50
0.550
NOC383
MARg3
0.525
0.500
0.20
[-MLP Dec.)
SCEMIR
0.43
(ML Dec
SCENIR
0.475
( MLP Dec.)
0.18
#Dischiminstor
Edge. Dac.)
Discricninttor)
GNN Edge.
0.450
(GNAN Edge. Dec.)
HDischiinstor)
0.16
GNN Foot. Dec.)
0.35
(-GN Feal. Dec.)
(-SAN Feat. Dec.)
2 Number of Layors
2
Number of Layers
Number of Layers
(a) Layer-wise 3
(b) Laser-wise MAP@3.
(c) Layer-wise MRR.
Figure 8. Variation in NDCG@3, MAP@3 and MRR for different number of layers, for each GAE The 3-layer variant of SCENIR performs the best in every metric.
13

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
0.30
0.28
0.575
0.26
0.50
0.550
NDC000
0.24
NARg3
0.525
0.500
0.475
A:
0.40
0.18
A:
A2
0.425
0.26
1/6
1/2
1/2
Lass
L035 Term Velue
(a) NDCG@3 performance.
(b) MAP@3 performance.
(c) MRR performance.
Figure 9. Variation in NDCG@3, MAP@3 and MRR for different values of each loss term, in the final loss function.
D. Further GNN Architecture Experiments
In Figure 8, report the results on NDCG@3, 3, MAP@3 and MRR for different number of GNN layers, the SCENIR- GIN architecture. Results follow the same trajectory depicted in Fig. 7 of the main paper, showing that the 3-layer GNN outperforms the rest, every metric.
We also report the results on Loss Trade-off Term tuning. Specifically, the final loss function of SCENIR, can be written as:
L Aı (L recem + Ledge recmn) + A2Ladv + Agh
(4)
Here, A denotes the loss weight for the graph reconstruction, X2 the loss weight for adversarial regularization, and A3 the loss weight for KL Divergence. Each loss term was individually tuned by fixing the other two to their empirically determined optimal values and varying the remaining term to identify its best value. The final chosen parameters where À 3, A2 = and Ag 3.2 Results for NDCG@3, MAP@3 and MRR of the SCENIR- GENNI model, reported in Figure
0.60
GNN,
GNN
0.50
Uelled
0.45
GNN
GNN
G.N.V
0.35
GNN
G.N.V
G.N.N GAV Erooddr
0.30
Spl
Unified-GMM
MAPG3
MRR
NDCG@3
Figure Left: illustration the encoder architecture in original GAE (unified), and in the proposed SCENIR (split). Right: Performance comparison between split and unified architecture, on MAP@3, MRR and NDCG@3.
Finally, regarding the splitting of the GNN Encoder, the original GAE (Kipf& & Welling, 2016), proposed unified encoder architecture (as seen in Figure 10), where all the layers are shared, except for the output. We proposed using completely Split variation for the GNN encoder, without any shared layers, which ultimately surpasses the original variant across every metric, Architecture illustration and metrics results can be seen in Figure 10.
E. Preprocessed Scene Graphs of Qualitative Results
In Figures 12 and 13, we present the scene graphs accompanying the qualitative results of Figures 5 and 6 of the main paper respectively. Viewing the graphs with the image retrieval outcomes provides a better understanding of our graph-based based semantic retrieval system and its merits. In both cases, it is apparent that GNN-based models (supervised and unsupervised) retrieve structurally similar graphs varying in semantic content; our method leads to consistently better semantics.
important to highlight that PSG graphs have been curated so that they properly represent semantics and relationships
14

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Query
Query
Top-1
Top-2
Top-3
Top-1
Top-2
Top-3
Figure 11. Additional qualitative results ohtained from SCENIR. Scene graphs of retrieved images are also provided.
present in images. To this end, our graph based method does not favor scene graphs being similar to the query because of merely having similar annotations; on the contrary, it is ensured that similarity in terms of scene graph annotations denotes actual similarity in terms of semantics and relationships.
F. Additional Results
In Fig. 11 we provide some additional ranked images retrieved from our proposed model, SCENIR.
We can observe general semantic consistency among the retrieved results. Starting from the example on the left side of Fig. 11, we can see that our framework successfully returns semantically similar images in all three positions. Specifically, in the first two positions, we can see a train on the railroad, something that is also present in the scene graphs. For the image in the third position, while the vehicle itself is different ("bus" instead of "train"), ") there is still significant similarity in the corresponding scene graphs.
As for the example on the right side of Fig. 11, we can see more clearly the structural similarities of the query and the retrieved scene graphs. In all cases, semantics of nodes and edges are very similar. Specifically, in the first two positions, the retrieved images depict multiple cows, something that is also evident in the scene graphs with the "cow" nodes. Since the test set was randomly chosen, it is evident that finding exact-match images is an uncommon scenario. Therefore, the image in the third position, depicts zebras instead of cows. It is important to note that many of the semantics remain similar (both structurally and conceptually), while the concepts "cow" and "zebra" are particularly close.
G. Details on Semantic Counterfactual Retrieval
Counterfactual explanations provide insights into why machine learning model made particular decision by identifying the smallest possible changes to an input that would have led to a different outcome. For example, in the context of loan approval, a counterfactual explanation might state: "Had your income been $5,000 higher, your loan would have been approved." This method is particularly useful for interpretability, as it helps users understand not just the current decision, but what would need to change to alter
Semantic counterfactual explanations extend this idea by ensuring that the generated modifications are meaningful and realistic within the context of the data. While counterfactuals in general focus any minimal change that flips the model's prediction, semantic counterfactuals ensure that these changes preserve the logic and constraints of the domain. For instance, in an image classification model, semantic counterfactual would modify features in way that aligns with real-wordd variations (e.g., changing a dog's s breed rather than distorting the image with unrealistic pixel noise). Within the context of semantic graph counterfactuals proposed by Dimitriou et al. (2024), images are represented as scene graphs, and the goal is identify all meaningful edits required transform scene graph belonging to one source class to the most similar scene
15

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
graph/image classified differently. Therefore, their case can directly leverage our scene graph retrieval framework.
Their method relies Graph Edit Distance (GED) supervision signal, which measures the smallest number of edit operations (node/edge additions, deletions, or label changes) required to transform one graph into another. By leveraging GED, one can generate counterfactuals by using GNNs to learn to approximate GED and thus minimize the number of modifications needed to alter the model's decision. However, computing GED is computationally expensive, particularly in a supervised learning setting where large datasets require repeated counterfactual evaluations. Even within the GC paper highlighted that this brute-force approach is somewhat inefficient, making impractical for real-wordd applications that require fast and scalable explanations.
Our proposed framework, SCENIR, directly addresses these shortcomings. Its efficiency stems from its fully unsupervised design, which not only reduces training time but also removes dependencies on labeled data, making it more scalable for large-scale retrieval applications. This aspect is evident in the significantly smaller training and inference times, as shown in Table 5. Additionally, our approach differs in that GED is used as a ground truth measure while maintaining an unsupervised learning paradigm, allowing for more flexible and inductive generalization.
16

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Query
Top-1
Top-2
Top-3
BLIP2
DeiT
IRSGS-RoBERTa
Ours (SCENIR)
Figure 12. Underlying preprocessed scene graphs directly corresponding to the qualitative results of Figure 5 in the main paper.
17

---

SCENIR: Visual Semantic Clarity through Unsupervised Scene Graph Retrieval
Query
Top-1
Top-2
Top-3
IRSGS-MPNet
RSGS-RoBERTa
Ours (SCENIR)
Figure 13. Underlying preprocessed scene graphs directly corresponding to the qualitative results of Figure 6 in the main paper.
18


Reviews:
[{"review_id": "nmQDT0WFnf", "reviewer": "Reviewer_s1fN", "summary": "This paper tackles the problem of image-to-image retrieval, focusing on improving the retrieval performance by emphasizing semantic content over low-level visual features. The authors argue that current models often rely on visual biases (e.g., colors, lighting conditions) rather than semantic contents/understanding. To avoid bypassing semantic information, the authors recast this task into scene graph retrieval and  propose SCIENIR, which is an unsupervised graph autoencoder framework for scene graph retrieval. The model architecture is essentially a branched VGAE, which is trained with a combination of losses such as the reconstruction loss, adversarial loss, and KL loss (against Gaussian).\n\nIn the experiments, this approach is evaluated in two data settings, with/without scene graph annotation. For the dataset without scene graph annotation, captions and scene graphs are automatically generated with existing tools (BLIP and PSGTR). Graph edit distance is used as the similarity score throughout the experiments. The quantitative results show that the proposed method outperforms various baselines ranging from existing GNN-based approaches to VLMs. Additionally, the authors provide qualitative analysis, counterfactual retrieval results, and speed analysis, indicating SCIENIR's effectiveness and efficiency.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer s1fN for their thoughtful feedback and for the care they took in understanding our claims and evaluation methods. We greatly appreciate their recognition of the validity of our work and their positive assessment of its presentation. We would like to address their main concern to provide further clarity and ensure a shared understanding.\n\nWe appreciate the reviewer’s suggestion and acknowledge the relevance of strong captioning models such as LLaVA and large language models (LLMs) like GPT and Gemini. However, to ensure a fair comparison, we adhere to replicating baseline models as proposed by their respective authors without extending them beyond their intended scope (e.g., substituting the captioner with an LLM-based alternative). More broadly, we believe that modifying the captioning module is slightly beyond the scope of our work. While a more advanced captioner could generate richer visual descriptions - potentially benefiting prior approaches that rely on caption-based supervision (e.g., IRSGS) - our core contribution is to move away from textual descriptions altogether. As discussed in the Introduction (lines 091-107) and further validated in Section 4.2 (Caption-driven disagreements), caption-based similarity models (e.g., SBERT variants) exhibit high variability, leading to inconsistent matching results depending on the specific SBERT model employed. Importantly, this issue persists regardless of improvements in caption quality.\n\nBy instead leveraging scene graphs, we utilize Graph Edit Distance (GED) as a more deterministic and reliable supervision signal, reducing the ambiguity inherent in text-based representations. This is a key aspect of our second contribution (highlighted at the end of the Introduction) and is further supported by recent literature (Dimitriou et al., 2024). Additionally, our approach enables counterfactual retrieval via conceptual edits - a direction explored in recent explainability research (Dimitriou et al., 2024) - which would not be as naturally facilitated by caption-based methods.\n\nMoreover, our decision not to rely on LLM-based captioning models aligns with our goal of reducing computational burden while maintaining strong performance. As the reviewer has acknowledged, LLM-based methods tend to be significantly more resource-intensive. Our approach provides a more efficient alternative, demonstrating that high-quality retrieval can be achieved without the need for large-scale caption generation.\n\nWe once again thank the reviewer for their insightful feedback and for encouraging further discussion on this aspect of our work.", "claims_and_evidence": "In my opinion, the most important and interesting claim is that an unsupervised graph autoencoder can effectively retrieve images, even surpassing its supervised counterparts. The quantitative results clearly support that the proposed approach has advantages over the baselines. I feel the baselines look a bit old. I discuss this point in the “Methods And Evaluation Criteria” section. Also, the authors advocate that GED is a more reliable measure for scene graph similarity than caption-based approaches. Figure 2 and Figure 7 demonstrate disagreement of those approaches and support this claim. Overall, this paper clarifies its claims and provides support evidence to them properly.", "methods_and_evaluation_criteria": "The proposed framework and the motivations behind it sound reasonable. My only concern is the choice of baselines. As of now, there are many strong captioning models available (e.g., LLaVA). Also, it would be nice to make some connection with SOTA LLMs such as GPT and Gemini models. It would be a reasonable baseline for the idea of scene graph retrieval (e.g., generate detailed image descriptions for all images and retrieve them in the text space using any text encoder). The model sizes are too different, but it would be great if the proposed approach achieves comparable results.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See my comments in the “Methods And Evaluation Criteria” section.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- Overall, this paper is well-written and easy to follow. Supplementary materials typically provide additional information to clarify ambiguous points in the main text.\n- To clarify my position, my only concern is the choice of baselines. I had hoped that this paper would use more up-to-date ones.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dt024uvbf6", "reviewer": "Reviewer_kfau", "summary": "This paper presents SCENIR, an unsupervised framework for scene graph retrieval that aims to improve semantic understanding in image-to-image retrieval tasks. It introduces a Graph Autoencoder-based architecture, eliminating the dependence on supervised ground truth labels like captions, which suffer from variability and inconsistencies. Key contributions include advocating Graph Edit Distance (GED) as a deterministic ground-truth measure, superior retrieval performance compared to both vision and supervised GNN baselines, and demonstrated applicability to real-world datasets and counterfactual retrieval scenarios. Experimental results on PSG and Flickr show SCENIR outperforming state-of-the-art methods in terms of accuracy and computational efficiency.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer kfau for their thorough feedback and for recognizing the validity of our evaluation methods, the soundness of our experiments, and the clarity of our presentation. We address their reported limitations and respond to their questions below.\n\n- Other comments and suggestions: Regarding the runtime environment specifics, we note that all experiments were conducted on an NVIDIA Tesla P100 GPU (line 269). Given the page limit constraints, we prioritized presenting our core contributions concisely. However, if the paper is accepted, we will include additional details in the camera-ready version to further enhance reproducibility. For reference, our setup includes: a single NVIDIA Tesla P100 GPU (16GB VRAM), a preprocessed scene graph dataset (~2GB), Python 3.10, PyTorch Geometric 2.4.0, PyTorch 2.2.0, and CUDA 11.8.\n- Questions For Authors - 1: Regarding the impact of embedding dimension on retrieval accuracy and computational cost, we would like to emphasize that these results stem from hyperparameter tuning, where we empirically determined the embedding dimension alongside other hyperparameters (e.g., loss terms, encoder architecture). To address the reviewer’s valid concern, we would like to emphasize that the significant computational efficiency demonstrated in Table 5 is primarily due to SCENIR’s unsupervised nature. Specifically, its linear training time stems from the fact that it does not regress on a pair’s similarity value, and its linear preprocessing time arises because we do not compute similarity values for every pair—an obligatory step in the other two frameworks. This distinction is further explained in Section 4.4 of the paper.\n- Questions For Authors - 2/W1: Regarding SCENIR’s robustness against noisy or incomplete scene graphs, we would like to clarify that the primary focus of our work is to evaluate and compare different retrieval frameworks rather than to conduct an in-depth robustness analysis for noisy or out-of-domain inputs. To ensure a high-quality benchmark, we use the PSG dataset, which significantly refines annotations from previous datasets (as mentioned in lines 254-258). Additionally, we apply preprocessing steps (detailed in Appendix A) to construct the final train/test sets. That being said, some level of noise remains inevitable. For example, this has been highlighted by the deliberate inclusion of Figure 11 (left). Specifically, it can be observed that background objects may still appear in the scene graph. However, SCENIR effectively identifies the primary semantic object of interest (e.g., “train”) despite the presence of surrounding noise, demonstrating its ability to focus on key scene elements.\n\nWe once again thank the reviewer for their valuable insights and suggestions, which help strengthen the clarity and impact of our work.", "claims_and_evidence": "The major claims, improved retrieval performance and computational efficiency, are supported by experimental results, comparisons, and ablation studies.", "methods_and_evaluation_criteria": "The methods (GAE-based unsupervised learning with adversarial training, graph pooling strategies, GED evaluation) are suitable for addressing biases arising from low-level visual features in scene graph retrieval tasks. Using GED as the evaluation metric is sensible due to its deterministic nature, reducing ambiguity in retrieval evaluations.", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "The experimental designs are thorough and valid. Various models including Vision, Vision-Language (VL), supervised GNNs, and unsupervised GAEs were fairly compared using clear and appropriate evaluation metrics.\nA slight limitation might be the lack of an ablation study on diverse graph embedding dimensions.", "relation_to_broader_scientific_literature": "SCENIR's contributions are clearly positioned against related literature. Its novelty primarily lies in extending unsupervised graph autoencoders for scene graph retrieval tasks and advocating GED over caption-based supervision. The authors have clearly discussed prior related works such as IRSGS and Graph Counterfactuals.", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "Strengths:\n1. Clear method contributions in using unsupervised methods effectively.\n2. Robust and comprehensive experimental validation.\n3. Effective demonstrations of practical applicability.\n\nWeaknesses:\n1. Limited analysis of robustness to dataset variability or extreme cases, especially in the Flickr dataset.", "other_comments_or_suggestions": "1. Clarify runtime environment specifics (hardware configurations clearly in the main paper for reproducibility).", "questions_for_authors": "1. How does the choice of embedding dimension affect SCENIR’s retrieval accuracy and computational cost?\n2. Have you explored the robustness of SCENIR against noisy or incomplete scene graphs? If so, what were the findings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oweq1vrekv", "reviewer": "Reviewer_tG8k", "summary": "The paper introduces an unsupervised framework for scene graph retrieval using GNN to prioritize semantic content over low-level visual biases. It employs a graph autoencoder to learn scene graph embeddings without labeled data and advocates for Graph Edit Distance as a deterministic evaluation metric.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer tG8k for their thoughtful comments and for acknowledging the strengths of our approach, methodology, and evaluation process. We now address your concerns systematically\n\n- W1: To address the reviewer’s valuable concern regarding the comprehensiveness of the datasets used in our work, we offer the following clarification. Since our approach focuses on scene graphs, and after a thorough investigation of potentially related datasets, we found that PSG and Flickr30K serve as a superset or an improved version of other similar datasets (e.g., Visual Genome, as mentioned in lines 254-255). Experimenting with additional datasets like Visual Genome would lead to redundancy, as it contains the same images from MSCOCO. The richness of scenes, objects, and relationships in the PSG and Flickr30K datasets already covers a wide range of domains and scenarios, thereby eliminating the need to explore other datasets that would essentially reflect the same data distribution. Furthermore, our experiments on Flickr30K with synthetic scene graphs and PSG (or Visual Genome) with annotated scene graphs follow the experimental setup used in prior work (Yoon et al., IRSGS).\n- W2: Regarding the impact of adversarial training and the feature/edge decoder design, we would like to clarify that we have already conducted ablation studies on both components in Section 4.1. The effect of adding or removing the adversarial loss is shown in line 3 of Table 4, while the decoder ablations are detailed in lines 4 and 5 of the same table, further supporting our design choices for the final framework architecture. While we appreciate the reviewer’s perspective, given our focus on proposing a novel end-to-end semantic image retrieval framework, rather than optimizing specific GNN components, a more fine-grained ablation analysis (e.g., tuning the normalization of the adversarial training module) falls beyond the intended scope of our work. \n- Questions For Authors: Despite the irrefutable validity of such a query, we believe that evaluating SGG frameworks’ quality falls outside the scope of this work. In our study, we report findings using the most effective SGG module available, as older frameworks may struggle to accurately represent scenes, potentially limiting retrieval quality. Therefore, we recommend adapting SCENIR to the strongest SGG framework for meaningful results. That said, SGG is not a core component of the SCENIR pipeline but is included primarily to demonstrate the effortless extendability of our approach to unannotated datasets.\n\nWe once again thank the reviewer for their valuable feedback and hope this clarification addresses their concerns.", "claims_and_evidence": "NA", "methods_and_evaluation_criteria": "method.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "NA", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "S:\n1\\ Scene graphs explicitly model objects and relationships, mitigating biases from superficial features like color\n2\\ Extends to unannotated datasets using automated scene graph generation\n3\\ Introduces GED as a deterministic ground truth, addressing variability in caption-based evaluation.\n\nW:\n1\\ Experiments focus on PSG and Flickr30K, broader validation across diverse domains is needed.\n2\\ The impact of adversarial training and decoder design could benefit from deeper analysis. This part need more ablations.", "other_comments_or_suggestions": "c.f., Weakness.", "questions_for_authors": "What is the sensitivity of SCENIR to different scene graph generation models?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "If8MsTssdm", "reviewer": "Reviewer_PwzX", "summary": "SCENIR is a novel unsupervised scene graph-based retrieval framework that prioritizes semantic content over low-level visual features. SCENIR uses a Graph Autoencoder to eliminate the need for labeled data. It outperforms vision-based, multimodal, and supervised GNN approaches in both accuracy and efficiency. Additionally, it introduces Graph Edit Distance (GED) as a robust metric for scene graph similarity, improving retrieval reliability and enabling generalization to unannotated datasets for counterfactual image retrieval.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate Reviewer PwzX's thoughtful feedback and their recognition of the validity of our method and the soundness of our evaluation. We now provide clarifications for each of the concerns raised.\n\n- W1: We appreciate the reviewer’s suggestion regarding Figure 1 and would like to clarify that our design choices for the teaser figure were intentional. The selected images specifically illustrate the color bias present in Efficient-ViT, which SCENIR effectively mitigates. This example highlights SCENIR’s ability to retrieve images based on true semantic understanding rather than superficial color similarities. Examining the ranked results in detail (all three positions), we can observe Efficient-ViT’s color bias when analyzing the semantic relevance of the retrieved images, as discussed in lines 47-56 of the Introduction:\n    1. The **Top-1 result** from Efficient-ViT does contain a ‘group of people,’ but they are not engaged in ‘riding’ any form of ‘sports equipment’ or ‘wheeled vehicle’ on the ‘street’ - key semantic elements of the query.\n    2. While the **Top-2 result** might appear somewhat relevant due to the presence of a car, it lacks critical query concepts such as ‘group of people,’ ‘riding,’ and ‘sports equipment,’ demonstrating a reliance on color rather than semantics (as noted in lines 49-53 of the Introduction).\n    3. The **Top-3 result** from Efficient-ViT moves away from a black-and-white image but still fails to capture the essential semantics, as it lacks both a ‘group of people’ and any form of ‘sports equipment.’ In contrast, SCENIR retrieves an image of people snowboarding, which, while differing in activity, aligns more closely with the core query semantics than Efficient-ViT’s selections.\n- W2: We appreciate the reviewer’s request for clarification and would like to reiterate the rationale behind using GED as an evaluation metric, as detailed in the Ground Truth and Evaluation section (starting at line 267). GED provides a deterministic method for identifying the most similar graph pairings in a dataset by measuring structural differences. Specifically, it measures structural differences: It quantifies the dissimilarity between two graphs by counting the minimum number of edit operations (node/edge insertion, deletion, substitution) needed to transform one graph into another. This ensures that retrieval is based on meaningful changes in object compositions and relationships, rather than just superficial features. Unlike purely visual metrics (e.g., pixel similarity) or text-based evaluations (e.g., caption similarity), GED directly measures the semantic coherence between two scene graphs. Semantically related objects or objects with similar roles (e.g., \"person sitting on a chair\" vs. \"child sitting on a bench\") will have lower edit distances, reflecting semantic similarity. Motivated by these advantages, as well as non-negligible evaluation shortcomings stemming from leveraging captions and caption similarity (see inconsistent top-1 SBERT retrievals in Figure 1, as well as low agreement between varying SBERT models in Figure 2), together with the support of prior literature in scene graph similarity (Dimitriou et al, 2024), we conclude that GED arises as an ideal evaluation measure in scene graph retrieval, promoting semantic preservation while eliminating ambiguity in defining ground-truth pairings.\n- W3: We appreciate this interesting suggestion; however, we strongly believe it aligns with a parallel research direction rather than the core focus of our work. One of our main claims is computational efficiency (see Section 4.4. Computational speedup): SCENIR achieves fast retrieval by executing all stages in the pipeline in around 8 minutes, while requiring minimal computational resources (single NVidia Tesla P100 16Gb VRAM GPU). On the other hand, utilizing LLMs/Multimodal LLMs already increases the computational budget, even if we exclude the pre-training stage, since most potent LLMs require larger GPUs than the one used in our experiments, or paid APIs (e.g. as in the case of ChatGPT). Other than that, harnessing scene graphs directly fuses semantic information within the pipeline, ensuring determinism in results, meaning that multiple runs of the pipeline yield consistent results. On the other hand, multimodal LLMs introduce variability, as they are not explicitly guided on which features to prioritize, potentially leading to biases similar to those observed in visual and vision-language models (e.g., Efficient-ViT in Figure 1 and DEiT in Figure 6), along with additional prompt-based variability. In any case, we regard this comment as a future work direction.\n\nWe once again thank the reviewer for their valuable feedback and hope this clarification addresses their concerns.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Yes", "essential_references_not_discussed": "It seems that essential references are discussed.", "other_strengths_and_weaknesses": "S1: SCENIR overcomes visual biases in vision models by using scene graphs to enhance semantic understanding.  \n\nS2: It introduces an unsupervised Graph Autoencoder for scene graph retrieval, removing the need for labeled data.  \n\nS3: SCENIR ensures robust evaluation with GED and extends to unannotated datasets for broader applications.\n\nW1: Figure 1 could use clearer examples, as the Top-1 result of Efficient-ViT appears more similar to SCENIR's Top-1 result, where the sports-related semantic relationship is not prominent.\n\nW2: Clarify the rationale for using GED as an evaluation metric in scene graph retrieval, emphasizing its ability to capture structural and semantic differences effectively.\n\nW3: The paper could further explore or compare the capabilities of recent large language and multimodal models in addressing the scene graph retrieval problem.", "other_comments_or_suggestions": "Refer to the weaknesses mentioned above", "questions_for_authors": "Refer to the weaknesses mentioned above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a scene graph-based retrieval framework emphasising semantic content over superficial image characteristics. The motivation is meaningful, and the proposed method is novel. All reviewers considered the paper as acceptance. Therefore, I recommend the acceptance.
Decision: Accept (poster)