VISTA: Visual-Textual Knowledge Graph Representation Learning
Authors: Jaejun Lee
Chanyoung Chung
Hochang Lee
Sungho Jo
Joyce Jiyoung Whang
Venue: EMNLP 2023 Findings (2023)
Keywords: Knowledge Graph
Multimodality
Representation Learning
Knowledge Graph Completion
Transformer

Abstract: Knowledge graphs represent human knowledge using triplets composed of entities and relations. While most existing knowledge graph embedding methods only consider the structure of a knowledge graph, a few recently proposed multimodal methods utilize images or text descriptions of entities in a knowledge graph. In this paper, we propose visual-textual knowledge graphs (VTKGs), where not only entities but also triplets can be explained using images, and both entities and relations can accompany text descriptions. By compiling visually expressible commonsense knowledge, we construct new benchmark datasets where triplets themselves are explained by images, and the meanings of entities and relations are described using text. We propose VISTA, a knowledge graph representation learning method for VTKGs, which incorporates the visual and textual representations of entities and relations using entity encoding, relation encoding, and triplet decoding transformers. Experiments show that VISTA outperforms state-of-the-art knowledge graph completion methods in real-world VTKGs.

VISTA: Visual-Textual Knowledge Graph Representation Learning
Jaejun Lee
Chanyoung Chung Hochang Lee
Sungho Jo Joyce Jiyoung Whang"
School of Computing. KAIST
Cjjlee98, Chanyoung.chumg chung, hochang. 1ee91, shjo, jjwhang)ekaist.acck
Abstract
cart
pull
person
Knowledge graphs represent human knowledge using triplets composed of entities and relations. While most existing knowledge graph embed- ding methods only consider the structure of a knowledge graph, few recently proposed mul- timodal methods utilize images or text descrip- tions entities in a knowledge graph. In this paper, we propose visual-textual knowledge graphs (VTKGs), where not only entities but also triplets can be explained using images, and both entities and relations accompany text descriptions. By compiling visually expressible commonsense knowledge, we construct new benchmark datasets where triplets themselves are explained by images, and the meanings of entities and relations described using text. We propose VISTA, a knowledge graph repre- sentation learning method for VTKGs, which incorporates the visual and textual representa- tions of entities and relations using entity en- coding, relation encoding, and triplet decoding transformers. Experiments show that VISTA outperforms state-of-the-art knowledge graph completion methods in real-world VTKGs.
ride
horse
transport
pull
sclishtoffedd
thing
quadruped
(a) Existing Multimodal Knowledge Graph
pull causse to move by pulling
person human being
cart
ride
wagan
pull
and tratel the back aniimal unusily while controlling its motions
cause to move by pullig
transport
mow something somebady amonnd osually mer long distanoes
soishhoofed hatbiverous
thing atti
quxtraped domorticated since prshitoons limes
(b) Visual-Textus Knowledge Graph (VTKG)
Figure 1: While existing methods assume only entities have their images and text descriptions, our VTKG ad- ditionally considers cases where a triplet itself is repre- sented by an image, and relations have text descriptions.
recently proposed multimodal knowledge graph completion methods (Zhao et al., 2022; Li et al.. 2023) consider the case where entities can have their images and text descriptions; Figure (a) is form of knowledge graph existing multimodal knowledge graph completion methods consider.
1 Introduction
Knowledge graphs provide diverse human knowl- edge in a structured form, representing each fact as a triplet consisting of entities and a relation. Knowledge graph representation learning meth- ods (Ji et al., 2022) aim to convert the entities and relations into set of representation vectors which can be utilized in predicting missing triplets or in other applications such as commonsense rea- soning (Lin et al., 2019) and question answering models (Liu et al., 2020). While most existing knowledge graph embedding methods focus solely on the structure of a knowledge graph to learn the representations (Lacroix et al., 2018; Sun et al., 2019), using additional images text descriptions can result in much better representations. A few
We focus on the fact that some triplets can be more intuitively expressed by their images. For example, in Figure (b), (person, ride, horse) and (person, pull, cart) have images, where the triplets themselves are represented by images, pro- viding visual insights. On the other hand, exist- ing knowledge graphs do not provide images for triplets and may lack visually expressible triplcts because they are constructed by extracting infor- mation mainly from texts. Therefore, we propose forming knowledge graph by extracting infor- mation from images and utilizing those images to represent visual commonsense knowledge.
*Corresponding author

---

To this end, we compile visual phrases in triplets and their images from diverse computer vision tasks, including visual relationship detec- tion (Lu et al., 2016), human-object interaction detection (Chao et al., 2018), and visual knowl- edge extraction (Sadeghi et al., 2015). We propose Visual-Textual Knowledge Graphs (VTKGs) where the entities and triplets can be represented by im- ages, and the entities and relations have their text descriptions as shown in Figure (b). We build new VTKG benchmark datasets containing visually expressible commonsense knowledge, the entities' and triplets' images, and detailed descriptions of the entities and relations.
recently, the Visual Genome dataset (Krishna al., 2017) has been released, where diverse computer vision datasets are merged. However, this dataset contains heterogeneous information which not in the form of triplets or is hard to be considered as commonsense knowledge. On the other hand, our VTKG datasets provide visual commonsense knowledge in the form of triplets with images and can seamlessly enlarge existing knowledge bases. We believe our work can be utilized in visual com- monsense reasoning (Zellers et al., 2019) and visual question answering (VQA) (Antol et al., 2015).
Knowledge Integration Ilievski et al. have at- tempted to examine the characteristics of informa- tion from different sources by manually categoriz- ing the kind of knowledge (llievski et al., 2021). From multimodal learning point of view, adding different modalities to existing knowledge bases has been considered (Zhu et al., 2022), e.g., adding images or texts to entities in knowledge graph. Different from these approaches, our VTKGs are proposed to represent visually expressible knowl- edge, and our benchmark datasets are created by a fine-level alignment of entities and relations from different sources using WordNet synsets (Miller, 1995). Details are described in Section 3.2.
To learn representations of entities and rela- tions in VTKGs, we propose the VISual-TextuAl (VISTA) knowledge graph representation learn- ing method that utilizes not only the structure of a visual-textual knowledge graph but also the vi- sual and textual features extracted from images and text descriptions. In particular, when triplet is provided with an image, VISTA learns the vi- sual representation of the relation in the given triplet. The resulting visual representation of the relation also utilized when the relation appears in other triplets, which enhances the entire rep- resentation learning process. To the best of our knowledge, VISTA is the first knowledge graph representation learning method that learns the vi- sual representations for visually expressible rela- tions. We design VISTA by proposing three trans- formers: entity encoding, relation encoding, and triplct decoding transformers. The entity and rela- tion encoding transformers represent entities and relations using their visual and textual feature vec- tors, whereas the triplet decoding transformer pre- dicts a missing entity in a triplet using masking scheme. Experimental results on four real-world datasets demonstrate that VISTA outperforms 10 different state-of-the-ar knowledge graph comple- tion methods. Our datasets and codes are available at https:: httpt//tttttttteee hoom/bdi-aa lad/viiAb..
Multimodal Knowledge Graph Completion While some knowledge graph embedding meth- ods utilize images of entities (Xie al., 2017; Wang et al., 2021; Dñoro-Rubin et al., 2019), some recently proposed multimodal methods consider both images and text descrip- tions of entities (Pezeshkpour et al., 2018; Wang et al., 2019). For example, MoSE (Zhao et al., 2022) and IMF (Li et al., 2023) learn modality- specific representations and make predictions ing the representations from different modalities. Also, OTKGE (Cao et al., 2022) proposes an op- timal transport to align multi-modal embeddings, while MKGformer (Chen et al., 2022) conducts multi-level fusion using a hybrid transformer. Un- like VISTA, all these existing methods assume that only entities can have images or descriptions and do not consider the cases where image repre- sents triplet itself or relations have deseriptions.
al., 2019; Liu
2 Related Work
Visual Commonsense Reasoning There have been some attempts to extract visual knowledge from images by using visually verifiable rela- tions (Sadeghi al., 2015; Chen et al., 2013). How- ever, they were not studied in the context of mul- timodal knowledge graph representation learning, and some datasets are not accessible now. More
3 Visual-Textual Knowledge Graphs
We describe how we collect visual commonsense knowledge and create VTKG benchmark datasets.

---

[V
[
[T]
Tent
[nn
Peat
Peil
VTKG-I
VTKG-C
VRD
130
88
842
14,945
10,372
129
80
UnRel
31
19
44
1,636
939
31
18
HICO-DET
96
157
516
232,830
129,936
92
155
VisKE
605
563
2,422
0
591
552
ConceptNerW
6,260
2,421
14,919
5,922
2,205
WNI8RR++
41,105
11
93,003
70,349
41,105
0
Table I: The original sources of two VTKG datasets. The first four rows datasets representing visual commonsense knowledge, and the last two rows are knowledge bases.
of triplets. I is set of images, and D is set of text descriptions. Since an image can be attached to an entity or a triplet, = Tent U where Tent indicates a set of images attached to entities and Tui is a set of images attached to triplets. Also, D = Dent U Drel where Dent and Diel indicate the descriptions of entities and relations, respectively. Throughout the paper, we seript upper cases for sets, boldfaced upper cases for matrices, and boldfaced lower cases for vectors.
(giraffe, eat, grass)
Visual Relationship Detection. Given an im- age, short visual phrase created using triplet.
Figure 2:
To create VTKG datasets, we use well-known knowledge bases, ConceptNet (Speer et al., 2017) and WN18RR (Dettmers al., 2018). We note that some tail entities in ConceptNet include predicates. For example, (spider, CapableOf, kill_a_fly should be converted to (spider, kill, fly) be consistent with other triplets in the other datasets; this consistency is important because we com- bine triplets from different sources to create our VTKG datasets where the same entities and rela- tions should be uppropriately aligned across differ- ent datasets. We manually inspected the triplets in ConceptNet for three weeks, resulting in the ConceptNetW dataset, which a subset Con- ceptNet, containing 14,919 triplets. On the other hand, in the original WN18RR, the entity types are missing. Thus, some entities are not mapped into unique synset ID which is defined in Word- Net (Miller, 1995), resulting in incorrectly repre- senting different entities one entity. To fix this, manually mapped those entities to of the five types: noun, verb, adjective, adjective satellite, and adverb. The resulting dataset W
3.1 Representing Visual Commonsense Knowledge as Triplets and Images
There can be many different design choices for ex- tracting information from images, depending on the type of information we want to extract. Among others, we look into computer vision tasks return- ing a visual phrase in the form of (objectı, predi- cate, object2). We assemble visual commonsense knowledge using four benchmark datasets from three different computer vision tasks: VRD (Lu al., 2016) and UnRel (Peyre al., 2017) for visual relationship detection, HICO-DET (Chao et al., 2018) for human-object interaction detec- tion, and VisKE (Sadeghi et al., 2015) for visual knowledge extraction. All these tasks return short visual phrase in the triplet format to describe given image. For example, Figure 2 shows out- put of visual relationship detection. Among the four datasets, VRD, UnRel, and HICO-DET pro- vide the triplets with images, whereas VisKE only provides triplets but not images. When collecting the visual phrases from these datasets, we filter out image-specific phrases that are hard to be con- sidered as commonsense knowledge, e.g., (dog, on on_the_right, vase).
Using the visual commonsense knowledge datasets and knowledge bases, create two VTKG datasets shown in Table 1. In VTKG-I, combine visual commonsense datasets where all triplets have their images. In VTKG-C, we merge all visual commonsense datasets and two knowledge bases, ConceptNetW and WNI8RR++.
3.2 Defining and Creating VTKGs
We define a VTKG as G (V, R, T, I, D) where V is a set of entities, R is a set of relations, T is a

---

bird
X
0 Rood
war miblooded eag-leving war virtebrates cha- racterord by feather and fonwimbs moditedd wings
(1
X
v551
V...
ViT-Base
Duisc
Entity Encoding Transformer
BERT Bast
Dixt
Pats
Pvis
Post
W
vis
vis
W
txt
X
Prediction
take msdldd food
Frrury
h
Byss
Triplet
x
Decoding
Prel
Transformer
e/p1
ViT-Base
Relation Encoding Transformer
(2)
VIT-Base
Fuis,
O
Pais
Prus
O
Post
x
vii
Weis
X
O-
:
ViT-Base
Product
Fvis
Tvis&
(2)
Puil
califfeed
BERT
Fest
Mask Vector
Figure 3: Overview of VISTA. We extract visual and textual features using ViT-Base and BERT The entity and relation encoding transformers compute the entity and relation representations, respectively. The resulting representations are fed to the triplet decoding transformer, which predicts missing entity in triplet.
Constructing VTKG datasets requires much more effort than simply unioning the triplets because triplets from different sources are usually written using different vocabularies, and the semantics of the same word should be determined depending on the context. For example, the same concept be written using different words, e.g., human person. Also, the same word can indicate dif- ferent meanings depending on the context, e.g., (diamond, scratch, glass) or (bartender, fill, glass). To refine the expressions, we mapped each entity and relation into a synset provided in Word- Net. This mapping requires us to select the most appropriate synset of each word in a triplet by ex- amining the meaning of each synset and the se- manties of the word in context. We replaced the entity and the relation with the corresponding synset ID so that triplets are rephrased using the synsets defined in WordNet. this way, we consolidated the vocabularies used in VRD, UnRel, HICO-DET, VisKE, and ConCeptNetW. Details about this process are described in Appendix A. Since WNISRR+ is subset of WordNet, it did not require this post-processing.
scriptions using pretrained VIT (Dosovitskiy et al., 2021) and BERT (Devlin et al., 2019). By utilizing the extracted features, VISTA learns representa- tions of entities and relations, which are computed by the entity and relation encoding transformers, respectively. Based on the resulting entity and re- lation representations, VISTA predicts missing entity in triplet using the triplet decoding trans- former.
4.1 Extracting Visual and Textual Features
In VTKGs, images can be associated with an entity or triplet. To extract visual features from the images, we employ ViT pretrained on ImageNet- 21k (Deng al., 2009), We resize an input of ViT- Base 224 x 224 and consider the final hidden state corresponding to [class] token as the visual feature vector. an entity has multiple images, the entity can have multiple visual feature vectors. one from each image. In practice, we select the top images per entity by computing pairwise dot product similarities of the visual feature vectors.
When an image is given for triplet (s, r, o). we first extract the visual features of entities, and o, by cropping bounding boxes, each of which con- tains s and and feeding each bounding box ViT-Base. Let and o denote the resulting vectors. To consider the visual attribute relation r, we define the minimal union bounding box that con-
4 VISTA: Learning Representations in Visual-TTxtual Knowledge Graphs
We propose VISTA shown in Figure 3. We extract visual and textual features from images and text de-

---

tains both entities and and compute the visual feature vector u by feeding this bounding box to ViT-Base. Then, we define Ivis = [s: o; where is a vertical concatenation, s, E Rds and dvis = 768. The visual representation vector of is computed by learning a projection matrix acting on Ivis. Details are described in Section 4.3. For each relation, we select the top à images based on the pairwise dot product similarities of the rvis values.
4.3 Relation Encoding Transformer
Given a relation т, consider Ivis,1:"" 1. vis E Ii described in Section 4.1. Let us consider images containing the relation r. Specifically, con- sider the j-th image containing triplet (sj, 7y))- Recall that rvis.j is created by concatenating the visual features of šj. oj. and the union bounding box containing sj and 0j. We compute the initial visual representation vector of for the j-th image by W visEvis where learnable projection matrix W vis is introduced. A visual representation vector of is learned per image while sharing the pro- jection matrix. The way we compute the visual representation of can be considered as a gener- alization of UVTransE (Hung et al., 2021) since Wvis allows flexible operations between the fea- ture vectors of 8, o, and their union bounding box, whereas UVTransE employs fixed constraint. On the other hand, let W Rdxdo be learnable matrix for converting the textual feature vector Fett of r to its textual representation vector.
To extract textual features from descriptions, we employ the pretrained BERT. For each given text description of an entity or a relation, we first tok- enize the description, feed the tokenized sequence to BERTBASE, and use the final hidden state cor- responding [CLS] token as the textual feature vector. For entities and relations without descrip- tions, we feed their labels in natural language to BERT The dimension of a textual feature vec- tor is set to dus = 768.
4.2 Entity Encoding Transformer
Let us consider an entity h with visual features, 1 ... k E Rdx,, and the textual feature hia E oR The input of the entity encoding trans- former is defined by
The input of the relation encoding transformer is
(0)
MIEEL) (1
vvi,1 + Pvis) ... P)
(
Vis P W Wat + PAX)
2 10 | ris.
(0)
(1 v Fxxt
(0)
x()
ZIENT|
|| [h +
+ Pvis)l
([ hvi,k + Pvis)] Wixchar + Prx))
where Z(REL| E I2d is learnable vector that is shared across all relations, E Rd is learnable vector for r. P, Pvis; PAu E Rd are the positional encodings of the learnable vector, visual represen- tation vectors, and the textual representation vector, respectively. After L transformer encoder layers, we get the final representation matrix
(0) 2h |b 1i ... vis.k bixi
where Z(ENT] Rd is a learnable vector that is shared across all entities, the dimension of a representation vector, h E Rd is learnable vec- tor for h, Wvis ddxd and Wixt E dddx are learnable projection matrices for converting the visual and textual features to their representation vectors, respectively, P. Pvis: Pixt E Rd are the po- sitional encodings of the learnable vector, visual representation vectors, and the textual representa- tion vector, respectively, and denotes a horizontal concatenation. After L transformer encoder lay- ers (Vaswani et al., 2017), we get the final repre- sentation matrix
i
(L) 2x
viv.
Fett
v
and we use z(²i as the representation vector of the relation r.
4,4 Triplet Decoding Transformer
Given a triplet (h, r, ?), the triplet decoding trans- former predicts the missing entity using Z 2y and learnable mask vector Z|MASK] E Rd. The input of the triplet decoding transformer is
(I.) (T)
x
(L) 4
(L)
(Z)
vis.
visk hxt
(L) and we use 2 as the representation vector of the entity h.
x (0) =[(([ ((() (Mm)ll(2pue + [un)]
/ (X]0]] ||

---

where Phead Prel, Puil Id are the positional en- codings of the head entity, relation, and tail entity, respectively, After applying L transformer encoder layers, we have
[V]
[R]
[T]
[Z]
[
VTKG-I
181
217
1,316
390,658
383
VTKG-C
43,267
2,731
111,491
461,007
45,401
WNISRR+
41,105
11
93,003
70,349
41,105
FB15K237
14,541
237
310,116
145,944
14.515
(
Table 2: Statistic of Datasets
head
X
and caleulate the score Yj of an entity Ej by Uj (D)
five methods are multimodal knowledge graph rep- resentation learning methods that consider images and text descriptions of entities. We run MoSE with three different options, denoted by MoSE-AL, MoSE-BI, and MoSF-MI. Details about running the baseline methods are described Appendix B.
tail -Ze; where ( the representation vector of
ej computed from the entity encoding transformer. We predict the missing entity to be the entity with the highest score. Similarly, we can also make a prediction for (?, r).r t). We use the cross entropy loss in our implementation.
We divide the training, valid, and test sets with ratio of 8:1:1 for VTKGs; we use the provided split for the existing benchmark datasets. When training model using triplets' images, only use the images associated with the triplets in training set. For knowledge graph completion in the VTKG datasets, the task to predict a head or a tail entity in (h, r, t). we do not provide any image associated with the triplet rh, t) to a model at testing time because its image can directly hint at the missing entity. From training to testing, we never use images of test triplets. We explain the hyperparameters of VISTA in Appendix C. For fair comparisons, we
5 Experiments
We evaluate the performance of VISTA using four real-world datasets by comparing it to 10 different knowledge graph completion methods. We imple- ment VISTA using PyTorcc (Paszke et al., 2019) with the Adam optimizer (Kingma and Ba, 2015). We apply dropout (Srivastava et al., 2014) and use the cosine annealing scheduler (Loshchilov and Hutter, 2017).
5.1 Datasets and Experimental Setup
256 on all datasets for all methods except MKGformer; the dimension in MKGformer is fixed to d 768 since it directly uses ViT and BERT.
We use four datasets shown in Table 2; VTKG-1 and VTKG-C are real-works VTKG datasets intro- duced in Section 3.2. While WN18 (Bordes et al., 2013) and FB15K237 (Toutanova and Chen, 2015) are benchmark datasets used in other multimodal knowledge graph completion research (Zhao et al., 2022; Chen et al., 2022), WN18 has test leak- age issue, and WNISRR (Dettmers et al., 2018) has been proposed to resolve the issue. In our ex- periments, we use WNISRR++ which is the fixed version of WN18RR as described in Section 3.2. Our VTKG datasets include images for triplets and entities, whereas the existing benchmark datasets provide images only for entities.
5.2 Knowledge Graph Completion
We evaluate the knowledge graph complction per- formance using MR (4), MRR (t), and H (t. N=1,3,10) in Table 3. The best results are boldfaced, and the second-best results are under- lined. Overall, VISTA significantly outperforms all baseline methods. While the second-best method varies depending on the dataset and the metric, VISTA consistently shows the best performance. In VTKG-1 and VTKG-C, there a substantial per- formance gap between VISTA and the best base- line method in all metrics. While VISTA utilizes relations' visual representation vectors and their text descriptions, none of the baseline methods is capable of dealing with them. This difference at- tributes to the performance gap between VISTA and the baseline methods since the VTKG datasets include visually expressible relations provided by images of triplets and abundant text descriptions for relations. On the other hand, WN18RR++ and
We use ten different baseline methods: ANAL- OGY (Liu et al., 2017), ComplEx-N3 (Lacroix et al., 2018), RotatE (Sun et al., 2019), PairRE (Chao et al., 2021), RSME (Wang et al., 2021), TransAE (Wang et al., 2019), MKG- former (Chen et al., 2022), OTKGE (Cao et al., 2022), MoSE (Zhao al., 2022), and IMF (Li al., 2023). The first four methods knowledge graph embedding methods only considering the structure of the given knowledge graph, whereas RSME can deal with images of entities. The rest

---

VTKG-1
VTKG-C
MR
MRR
Hit@1
Hit@
Hit@ 10
MR
MRR
Hit@
Hit@3
Hit@ 10
ANALOGY
39.5
0.3040
0.2328
0.3015
0.4466
10392.1
0.2963
0.2609
0.3180
0.3532
ComplEx-N3
31.8
0.3911
0.3168
0.4046
0,5191
3668.0
0.3944
0.3515
0.4079
0,4815
RotatE
24.5
0.3131
0.2099
0.3473
0,5267
4121.2
0.3893
0.3473
0.4062
0,4704
PairRE
20.9
0.4104
0.3015
0.4504
0.6145
2736.8
0.3876
0.3431
0.4013
0.4782
RSME
32.0
0.4027
0.3321
0.4122
0,5573
4401.4
0.3942
0.3513
0.4096
0.4776
TransAE
19.5
0.2437
0.0687
0,3092
0.6374
3063.3
0.0751
0.0053
0.1053
0.2072
MKGformer
29.9
0.3884
0.3397
0.3740
0,4885
668.5
0.4227
0.3531
0.4487
0.5580
OTKGE
27.5
0.4278
0.3588
0.4466
0.5458
2606.8
0.3939
0.3446
0.4152
0.4881
MoSE-AI
22.5
0.4306
0,3473
0.4466
0.6221
854.3
0,3929
0.3186
0,4301
0,5210
MOSE-BI
23.7
0.4297
0.3473
0.4466
0.6221
615.9
0.3933
0.2825
0.4578
0,5864
MoSE-MI
34.0
0.4235
0.3397
0.4504
0.6107
$27.0
0.4056
0.2913
0.4762
0,5977
IMF
35.6
0.4184
0.3282
0.4656
0.5649
2951.6
0,4116
0.3706
0,4261
0,4935
VISTA
17.3
0.4650
0.3626
0,5076
0.6641
220.8
0,4675
0.3918
0.4961
0,6157
WN18RR++
FB15K237
MR
MRR
Hit
Hit
Hit@ 10
MR
MRR
Hit
Hit@3
Hit@ 10
ANALOGY
7453.8
0.4128
0.3969
0.4175
0.4438
371.9
0.2420
0.1516
0.2677
0.4301
ComplEx-N3
3545.0
0.4745
0.4292
0.4895
0.5675
172.7
0.3510
0.2584
0,3847
0.5391
RotatE
1946.1
0.4606
0.4274
0,4754
0,5230
246.1
0.3099
0.2183
0.3433
0,4932
PairRE
3676.2
0.4529
0.4127
0.4663
0.5351
184.3
0.3326
0.2399
0.3675
0.5193
RSME
4062.1
0.4567
0.4175
0.4751
0.5300
174.9
0.3445
0.2523
0.3780
0,5286
TransAE
2308.2
0.0900
0.0040
0.1291
0,2511
234.6
0.2122
0.1432
0,2267
0,3459
MKGformer
352.3
0,5308
0.4697
0.5557
0.6560
297.6
0.3095
0.2278
0.3356
0,4740
OTKGE
1993.1
0.4327
0.3722
0.4663
0.5407
168.1
0.3411
0.2511
0.3739
0,5192
MoSE-AI
303.2
0.4857
0.4255
0.5094
0.5996
149.1
0.3247
0.2384
0,3532
0,4965
MoSE-BI
108.0
0.5026
0.4151
0.5461
0.6670
132.3
0.3466
0.2570
0.3755
0,5303
MoSE-MI
205.3
0.4969
0.4026
0.5466
0.6681
148.9
0.3275
0.2416
0.3568
0.4984
3774.0
0.4749
0.4397
0.4845
0.5469
151.8
0.3677
0.2735
0.4040
0.5573
VISTA
177.6
0.5526
0,4871
0,5799
0.6755
114.2
0,3808
0.2873
0.4158
0,5718
Table 3: Knowledge Graph Completion Performance. VISTA outperforms baseline methods all datasets.
5.3 Qualitative Analysis
Visual Representations of Relations
We conduct qualitative analysis using VISTA trained on VTKG-C. In Section 4.3, VISTA learns W vis to compute the visual representation vectors of relations from images. Figure 4 shows the vi- sual representation vectors of three relations, Feed, Repair, and Carry, visualized by t-SNE. From a testing set, we select images corresponding triplets containing the three relations and compute the visual representation vectors using the learned Wvis- We see that the visual representations cor- responding the same relation are well clustered even though VISTA computes them based un- images and triplets during training.
Repair Carry
10.
5-
0-
-15
-20
Figure 4: Visual representation vectors of relations, Feed, Repair, and Carry, where the vectors are com- puted on images of triplets containing the three relations.
We qualitatively compare the representation vectors created by BERT (Devlin et al., 2019), ViT (Dosovitskiy et al., 2021), and VISTA in VTKG-C. Given query entity (or relation), we retrieve the top 3 similar entities (or relations) to the query based the representation vectors turned by BERT, VIT, and VISTA. Table 4 shows the results where the query entities are jar and dark_red, and the query relations are have and buy. We see that BERT returns some abstract con-
FB15K237 do not include images for triplets and descriptions for relations. In WN18RR++, even though VISTA shows the second-best performance in MR, the MRR, Hit@ @3, and @it@ 10 scores of VISTA are better than any other meth- ods. In FB FB15K237, VISTA outperforms all base- line methods.

---

Query
BERT
VIT
VISTA
Entities without Images
Relations without Images
eyelet
lid
bucket
10
2EREL)
jar
tent top
barrel
10-
d
3
gravy_boat
coquilla_nut
bowl
Denstry
Density
Foxt
incense
leisure_wear
orange
6
dark_ren red
coloring
sportswear
red
3 buffer
sweatshirt
crimson
move straddle
keep
have influence hop_on hold
0.2
0.2
3
begin inspect incorporate
Attention Weights
Attention Weights
1
stīpulate pluck share
(a) Attention weights in the entity and relation encoding trans- formers for entities and relations without images
buy
incorporate jab price
use
own
trade
Entities Image
Relations Image
Table 4: Top three similar entities relations the query in VTKG-C based on the representation vectors created by BERT, VIT, and VISTA.
Z/enTI
25
DEEL
10
Donstty
Demsity
15
10-
cepts, e.g., incense and influence whereas ViT returns visually expressible concepts, e.g., lid and straddle. VISTA successfully returns the most se- mantically close entities and relations to the queries by utilizing both texts and images.
0.0
0.2
0.4
0.0
0.4
Attention Weights
Weights
(b) Attention weights in entity and relation encoding trans- formers entities and relations with one image
Figure 5: Distributions of attention weights in the entity and relation encoding transformers.
As deseribed in Section 4.2, the entity encod- ing transformer of VISTA learns the representation vector of each entity based on [ENTT- h. I. I*",, hvis,k. and has. Similarly, the relation encoding transformer learns the representation vector a v and Tix described in Section 4.3. To analyze the con- tribution of each of these terms, we visualize the at- tention weights using kernel density estimate plots. Figure 5 shows the distributions of the attention weights for entities and relations for (a) the case where the entities and relations do not have images and (b) the case where the entities and relations have images. When we consider the case when images are not given, textual features Fixt play the most crucial role in relations. On the other hand, in entities, the learnable vector h has relatively high attention weights. When images given, the vi- sual attributes of relations 1 tend have high contributions, sometimes even exceeding the con- tributions of textual descriptions Fix-- This verifies that learning visual representations of relations are effective in VISTA, which contributes to the perfor- mance of VISTA. For entities, the learnable vector h still has high importance when images are given, and the visual features h tend to have slightly higher importance than textual features h
binations of the modalities (the "Modality" rows). Also, show the impact of different modalities for entities (the "Entity Modality" rows) and for relations (the "Relation Modality" rows). On both VTKG-I and VTKG-C, we see that adding visual or textual features leads to better performance than the case without those features. More importantly, using all modalities presented in the last row of Table 5 (VISTA) leads to the best performance.
relation based on [EEL]- r, 1v
The impact of the visual features is more promi- nent on VTKG-I than on VTKG-C. To analyze it, wc count the number of entitics and relations with and without visual features, shown in Table 6. While all entities and relations have their visual features on VTKG-I, many entities and relations do not have visual features on VTKG-C. As result, visual features tures VTKG-I, while textual features are more important than visual features on VTKG-C.
important than textual fea-
On the other hand, the performance decreases of VISTA with 'replace rvis w/ u' indicate that our way of constructing Evis is more effective than just using the union bounding box of triplet when creating a visual representation vector of r.
Furthermore, we remove or replace some com- ponents in VISTA. Specifically, for the "Encoder" part, we consider the cases where (1) we remove Z[ENT] and Z[REL], (2) we remove h and r, (3) we
5.4 Ablation Studies
We present ablation studies of VISTA in Table 5. We show the results of VISTA using different com-

---

VTKG-I
VTKG-C
MR
MRR
@it@1
Hit
@it 10
MR
MRR
Hit@1 1 Hit
Hit 10
w/o hes. Har. Frix. Fint
23.3
0.398
0.309
0.431
0.573
3163.9
0.365
0.325
0.381
0.441
Modality
w/o hyi Feis
21.0 0.442
0.347
0.454
0.645
238.7
0.456
0.383
0.481
0.601
w/o har,
19.5 0.453
0,363
0.473
0.634
2641.1
0.374
0.333
0.389
0.452
w/o his hua
22.2 0.397
0.294
0,439
0.592
3036.6
0.366
0.327
0.382
0.440
Entity Modality
w/o hys
20.5 0.415
0.305
0.454
0.622
229.1
0.457
0,383
0.484
0.603
w/o hea
17.2 0.459
0,359
0.496
0.664
2599.5
0.372
0.331
0.387
0.449
w/o Fvis, Fuu
18.8 0.430
0.324
0.466
0.641
235.8
0.460
0.386
0.487
0.610
w/o Feis
19.0 0.455
0.359
0.492
0.641
223.7
0.466
0.391
0.493
0.613
Relation Modality
w/o Fia
17.3 0.459
0,351
0.500
0.664
236.3
0.461
0.385
0,490
0.609
replace rvs w/ u
18.3 0.433
0.336
0,458
0.649
318.1
0.416
0.341
0.445
0.557
w/o Zienra and Z
18.6 0.402
0.290
0.454
0.622
316.8
0..402
0,325
0.434
0.549
w/o h and w/o trans.
19.0 0.438
0,328
0.485
0.649
250.9
0.450
0.372
0,484
0.598
Encoder
18.4 0.417 0,302
0.473
0.622
369.3
0.389
0.312
0.420
0.538
wfo ent. trans.
25.2 0,406
0.332
0.424
0,546
4368.7
0,362
0.319
0.377
0.444
w/ DistMult decoder
21.0 0.391
0,282
0,416
0.634
420.6
0.420
0.357
0.438
0.542
Decoder
replace dot prod. w/ lin.
29.5
0.406
0.317
0,435
0.580
964.5
0.430
0.380
0.448
0.524
VISTA
17.3 0.465
0.363
0,508
0.664
220.8
0.467
0,392
0.496
0.616
Table 5: Ablation Studies of VISTA. The best performance is achieved when modalities are considered. The performance of VISTA degrades when remove any component replace component with other alternatives.
VTKG-I
VTKG-C
prediction. The two rows of the "Decoder" in Ta- ble indicate that making changes in the way of predicting a missing entity for knowledge graph completion leads to degrading the performance of VISTA. This validates the effectiveness of our triplet decoding transformer.
Entities w/ hyis
181 (100%)
7,863 (18.2%)
Entities w/o hois
0 (0%)
35,404 (81.8%)
Total
181
43,267
Relations w/ Ivis
217 (100%)
217 (7.9%)
Relations w/o rvis 0 (0%)
2,514 (92.1%)
Total
217
2.731
Conclusion & Future Work
Table 6: Entities and relations with and without visual features on VTKG-I and VTKG-C.
We propose VTKGs, where visually expressible triplets augmented by images, and both entities and relations have textual descriptions. By appro- priately utilizing all this rich information, VISTA substantially outperforms 10 different state-of-the- art knowledge graph completion methods in real- world VTKG datasets. Our VTKG datasets and VISTA model can be utilized in diverse applica- tions and scenarios (Sckuboyina et al., 2019; Kwak et al.. 2022; Lee et al., 2023), including those re- quiring visual commonsense knowledge such VQA (Marino al., 2021) scene graph genera- tion (Chang al., 2023; Zareian al., 2020) and commonsense reasoning (Lin al., 2019).
remove the relation encoding transformer and just leave the learnable vector r, and (4) we remove the entity encoding transformer and just leave the learnable vector h. At the first two rows of the "Encoder" in Table 5, we see that Z[ENT]; Z[REL]. h, and are all important inputs of the entity and re- lation encoding transformers. At the last two rows of the "Encoder" in Table 5, we note that removing either the entity encoding transformer or the rela- tion encoding transformer greatly impacts VISTA's ss performance. Regarding Hit@ removing the re- lation encoding transformer leads more crit- ical decrease in performance than removing the entity encoding transformer. On the other hand, for the other metrics (i.e., MR, MRR, Hi and Hit@ 10), the entity encoding transformer is more critical than the relation encoding transformer.
We will extend our work to hyper-relational knowledge graphs (Galkin et al., 2020; Chung et al., 2023) or bi-level knowledge graphs (Chung and Whang, 2023) where more information is added to each triplet using qualifiers, or higher-level relation- ships are considered to enrich information between triplets. By structuring an image or description as auxiliary information or qualifiers, VISTA can be casily extended to hyper-relational knowledge graphs with images and texts.
For the "Decoder" part, we consider the cases where (1) we replace the triplet decoding trans- former with DistMult (Yang et al., 2015) and (2) we replace the dot product with a linear layer for

---

Limitations
on Neural Information Processing Systems, pages 99090-391022.
When constructing our VTKG datasets, we aligned the same entities and relations using WordNet synsets (Miller, 1995) described in Section 3.2; and this process was manually conducted, which required much human effort. While utilizing auto- mated tools would make the process more scalable, we could not find reliable and accurate annotation tools for our alignment task. Developing automated tools that can align triplets from different sources as precisely as human annotators will accelerate the convergence of diversely-source knowledge.
Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiaojiang Chen, and Alex Hauptmann. 2023. A com- prehensive survey of scene graphs: Generation and application. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):1-26.
Linlin Chao, Jianshan He, Taifeng Wang. and Wei Chu. 2021. PairRE: Knowledge graph embeddings via paired relation vectors. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing, pages 4360- 4369.
Our model, VISTA, comprises three transform- ers and thus might not be considered as lightweight compared to classical shallow encoder- based knowledge graph embedding methods such as Ro- tatE (Sun et al.. 2019) and ComplEX-N3 (Lacroix et al., 2018). However, we note that VISTA is faster than another transformer-basee former; to process VTKG-C, VISTA took 3 hours, whereas MKGformer took 15 hours. We will ex- plore how we can make our implementations more scalable and make our model lighter.
Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng, 2018. Learning to detect human-object interactions. Proceedings of the 2018 IEEE Win- ter Conference on Applications of Computer Vision, pages 381-389.
Xiang Chen, Ningyu Zhang, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang. Luo Si, and Huajun Chen. 2022. Hybrid transformer with level fusion for multimodal knowledge graph completion. Proceedings of the 45th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pages 904-915,
baseline, MKG-
Acknowledgements
Xinlei Chen, Abhinav Shrivastava, and Abhinav Gupta. 2013. NEIL: Extracting visual knowledge from web data. In Proceedings of the 2013 IEEE International Conference on Computer Vision, pages 1409-1416.
This research was partly supported by NRF grants funded by MSIT (2022R1 A2C4001594 and 2018RIA5A 11A59991......... This work was also sup- ported by an IITP grant funded by MSIT 2022- 0-00369 (Development of AI Technology to sup- port Expert maaiinn that can Explain the Reasons/Grounds for Judgment Results based on Expert Knowledge).
Chanyoung Chung, Jaejun Lee, and Joyce Jiyoung Whang. 2023. Representation learning on hyper- relational and numeric knowledge graphs with trans- formers. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing, pages 310-322.
Chanyoung Chung and Joyce Jiyoung Whang. 2023. Learning representations of bi-level knowledge graphs for reasoning beyond link prediction. In Pro- ceedings of the 37th AAA/ Conference on Artificial Intelligence, pages 4208-4216
References
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, Charles Lawrence Zit- nick, and Devi Parikh. 2015. VQA: Visual question answering. In Proceedings of the 2015 IEEE In- ternational Conference on Computer Vision, pages 2425-2433.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hier- archical image database. In Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255.
Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. In Proceedings of the 27th Con- ference on Neural Information Processing Systems. pages 2787-2795
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp. and Sebastian Riedel. 2018. Convolutional 2d knowl- edge graph embeddings. In Proceedings of the 32nd AAAI Conference Artificial Intelligence, pages 1811-1818.
Jacob Devlin, Ming-Woi Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. Proceedings of the 2019 Conference of the North American Chapter of the Association
Zongsheng Cao, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. 2022. OTKGE: Multi-modal knowledge graph embeddings via opti- mal transport. In Proceedings of the 36th Conference

---

for Computational Linguisties: Human Language Technologies, Volume I. pages 4171-4186.
Xinhang Li, Xiangyu Zhao, Jiaxing Xu, Yong Zhang, and Chunxiao Xing. 2023. IMF: Interactive mul- timodal fusion model for link prediction. In Pro- ceedings of the ACM Web Conference 2023, pages 2572-2580.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x 16 words: Transformers for image recogni- tion at scale. In Proceedings of the 9th International Conference on Learning Representations.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. KagNet: Knowledge-awan graph net- works for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language P Processing and the 9th International Joint Conference on Natural Language Processing, pages 2829-28333
Mikhail Galkin, Priyansh Trivedi, Gaurav Maheshwari, Ricardo Usheck, and Jens Lehmann. 2020. Mes- sage passing for hyper-relational knowledge graphs. In Proceedings of the 2020 Conference on Empiri- cal Methods in Natural Language Processing, pages 7346-7359.
Hanxiao Liu, Yuexin Wu, and Yiming Yang. 2017. Ana- logical inference for multi-relational embeddings. In Proceedings the 34th International Conference Machine Learning, pages 2168-2178.
Zih-Siou Hung, Arun Mallya, and Svetlana Lazebnik. 2021. Contextual translation embedding for visual relationship detection and scene graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, (3(11):3820-3832.
Weijie Liu, Peng Zhou, 7he Zhao, Zhiruo Wang. Qi Ju Haotang Deng, and Ping Wang. 2020. K-BERT: Enabling language representation with knowledge graph. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, pages 2901-2900
Filip Niivskii, Alessandro Oltramari, Kaixin Ma, Bin Zhang. Deborah Louise McGuinness, and Pedro Szekely. 2021. Dimensions of commonsense knowl- edge. Knowledge-Bast Systems, 229:107347.
Ye Liu, Hui Li, Alherto García-Durán, Mathias Niepert, Daniel Oñoro- Oñoro-Rubio, and David S. Rosenblum. 2019. MMKG: Multi-modal knowledge graphs. Pro- ceedings of the 16th European Semantic Web Confer- ence, pages 459-474
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Mart- tinen, and Philip Shi lung Yu. 2022. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 3342):494-514
Ilya Loshchilov and Frank Hutter. 2017. SGDR:
Stochastic gradient descent with warm restarts. Proceedings of the 5th International Conference on Learning Representations.
Diederik Pieter Kingma and Jimmy Ba. 2015. Adam: method for stochastic optimization. Proceed- ings of the 3rd International Conference on Learning Representations.
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. 2016. Visual relationship detection with language priors. In Proceedings of the 14th European Conference on Computer Vision, pages
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Ayman Shamma, Michael Scott Bernstein, and Li Fei-Fei. 2017. Vi- sual Genome: Connecting language and vision us- ing crowdsourced dense image annotations. Interna- fional Journal of Computer Vision 123(1):32-73.
Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. 2021. KRISP: Inte- grating implicit and symbolic knowledge for open- domain knowledge- based VQA. In Proceedings of the 2021 IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14106-14116.
George Armitage Miller. 1995. WordNet: a lexical database for English. Communications of the ACM, 38(11):39 41.
Ji Ho Kwak. Jaejun Lee, Joyce Jiyoung Whang, and Sungho Jo. 2022. Semantic grasping via knowl- edge graph of robotic manipulation: A graph rep- resentation learning approach. IEEE Robotics and Automation Letlers, 7(4):9397-9404
Daniel Oñoro-Rubio, Mathias Niepert, Alberto Garefa-Durán, Roberto González-Sáncher and Roberto Javier López-Sastre. 2019. Answering visual-relational queries in web-extracted knowledge graphs. Proceedings of the 1st Conference on Automated Knowledge Base Construction.
Timothee Lacroix. Nicolas Usunier, and Guillaume Obozinski. 2018. Canonical tensor decomposition for knowledse base completion. In Proceedings of the 35th International Conference on Machine Learn- ing, pages 2863-2872.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. 2019. PyTorch:
Jaejun Lee, Chanyoung Chung, and Joyce Jiyoung Whang. 2023. InGram: Inductive knowledge graph embedding via relation graphs. In Proceedings of the 40th International Conference on Machine Learning, pages 18796-18809.

---

An imperative style, high performance deep learn- ing library. In Proceedings of the 33rd Conference on Neural Information Processing Systems, pages 8026-8037.
ACM International Conference on Multimedia, pages
Zikang Wang, Linjing Li, Qiudan Li, and Daniel 7eng. 2019. Multimodal data enhanced representation learning for knowledge graphs. In Proceedings of the 2019 International Joint Conference on Neural Networks, pages 1-8.
Julia Peyre. Ivan Laptev, Cordelia Schmid, and Josef Sivic. 2017. Weakly-supervised learning of visual relations. In Proceedings of the 2017 IEEE Interna- tional Conference on Computer Vision, pages 5189- 5198.
Ruobing Xie, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2017. Image-embodied knowledge represen- tation learning. In Proceedings of the 26th Inter- national Joint Conference on Artificial Intelligence, pages 3140-3146.
Pouya Pezeshkpour, Liyan Chen, and Sameer Singh. 2018. Embedding multimodal relational data for knowledge base completion. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages
Bishan Yang, Wen tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In Proceedings of the 3rd International Conference on Learning Representations.
Fereshteh Sadeghi, Santosh Kumar Divvala, and Ali Farhadi. 2015. VisKE: Visual knowledge extraction and question answering by visual verification re- lation phrases. In Proceedings of the 2015 IEEE Conference Computer Vision and Pattern Recog- nition, pages 1456-1464.
Alireza Zareian, Svebor Karaman, and Shih-Fu Chang, 2020. Bridging knowledge graphs to generate scene graphs. In Proceedings of the 16th European Confer- ence on Computer Vision, pages 606-623.
Anjany Sekuboyina, Daniel OñooroRubbia Jens Kleesiek, and Brandon Malone. 2019. A relational- learning perspective multi-label chest x-ray classi- fication. Proceedings of the 2021 IEEE 18th Inter- national Symposium on Biomedical Imaging, pages 1618-1622.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition cognition: Visual commonsense reasoning. In Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 6713-6724.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. ConceptNet 5.5: An open multilingual graph of gen- eral knowledge. In Proceedings of the 31st AAAI Conference on Artificial Intelligence, pages 4444- 4451.
Yu Zhao, Xiangrui Cai, Yike Wu, Haiwei Zhang, Ying Zhang, Guoqing Zhao, and Ning Jiang. 2022. MoSE: Modality split and ensemble for multimodal knowl- edge graph completion. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, pages 10527-10536.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Re- search, 15(56):1929-1955
Xiangru Zhu, Zhixu Li, Xiaodan Wang. Xueyao Jiang. Penglei Sun. Xuwu Wang. Yanghua Xiao, and Nicholas Jing Yuan. 2022. Multi-modal knowledge graph construction and application: A survey. IELF Transactions on Knowledge and Data Engineering, pages 1-20.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Yun Nie, and Jian Tang. 2019, RotatE: Knowledge graph embedding by relational rotation in complex space. In Proceed- ings of the 7th International Conference on Learning Representations.
Kristina Toutanova and Danqi Chen. 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Composi- tionality, pages 57-66.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Nicholas Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Atten- tion is all you need, In Proceedings of the 31st Con- ference on Neural Information Processing Systems, pages 3998-6008.
Meng Wang, Sen Wang, Han Yang, Zheng Zhang, Xi Chen, and Guilin Qi. 2021. Is visual context really helpful for knowledge graph? representa- tion learning perspective. In Proceedings of the 29th

---

Appendix
CV
person paint boat
Details about Constructing and Validating vTKGs
As described in Section 3.2, we examined all triplets in VRD, UnRel, HICO-DET, VisKE, and ConceptNerW to map each entity and relation to a synset in WordNet (Miller, 1995). We devel- oped data labeling tool shown in Figure that allows a human expert to select the most appropri- ate synset of each word in triplet by examining the meaning of each synset and the semantics of the word in context. It took six weeks to examine all the triplets. In the data laheling process, also verified whether each triplet valid or not; we re- moved the invalid or -onteer-nte triplets from our datasets, e.g., (mechanic, winter, car). Dur- ing this process, 269 triplets were removed among 19,065 original triplets. When we sampled 2,000 triplets and Nouble-checked. the labels, the labeling accuracy was 95.01%.
OI
Figure 6: Interface of our data labeling tool for map- ping each word to the most appropriate synset and also checking the validity of each triplet.
lation is represented using a synset of WordNet, e.g., 'airplane.n.01" or 'have.v.01', where 'n' in- dicates a noun, 'v" indicates verb, and the last two digits indicate the sense number. The descrip- tions of entities and relations from WordNet. Figure shows some examples of triplets in our VTKG datasets.
Additionally, we checked the matching of each triplet and its image. In our datasets, 1,316 triplets are expressed by images, and there are 141,247 images for triplets. The triplet images came from VRD, UnRel, and HICO-DET. To examine whether triplet is well matched with its image, ran- domly sampled 100 images for each triplet. If triplet has less than 100 images, we examined all the images for the triplet. As result, we exam- ined 36,919 images, and each image was examined by two different annotators. We consider a triplet is matched with its image only if both annotators agree on it, i.e., if one of the annotators disagrees, we consider the triplet is not matched with its im- age. When measuring the matching degree in this way, the accuracy was 98.62%.
Merging triplets from different sources
One of the critical points of creating VTKGs is properly combining triplets from different sources. Since triplets from different sources are usually written using different vocabularies, as discussed in Section 3.2, we had to find a way to consolidate those vocabularies. To resolve this issue, we re- place each entity/relation in a triplet with synset ID defined in WordNet to express triplets using the vocabulary defined in WordNet. An alterna- tive method we considered was to replace each en- tity/relation with a word defined in the Cambridge Dictionary. However, using the Cambridge Dictio- nary resulted in creating redundant triplets due to various synonyms, failing to merge semantically identical triplets. Therefore, we decided to use WordNet synsets when we merge triplets.
When constructing our VTKG datasets, we did not use FB15K237 because FB15K237 de- livers different level of information from VRD, UnRel, HICO-DET, VisKE, Con- ceptNerW, and WNISRR++. For example. while VTKG-C contains generic relation- ships between concepts, e.g., (person, write, story), most triplets in FB15K237 provide specific details about particular individuals or facts, e.g., (James_Camerron, graduate_from. iiiiforni_Sttate_niivvriss
B Experimental Details about Running the Baseline Methods
We run bascline methods using GeForce RTX 2080 Ti or RTX A6000 depending on the dependency of each method. For all methods, we use the default hyperparameters provided in their implementations unless otherwise stated. We choose the best con- figuration for each method based the validation results. In what follows, we use the notation of the original papers.
We provide some examples of the triplets and their images and descriptions in our VTKG datasets. As explained above, each entity or re-
For ANALOGY (Liu et al., 2017), we tried

---

(Head Entity,
Relation,
Tail Entity)
Image #1
Image #2
(airplane.n.) .n.,
have.v.0
wheel.n.01)
an aircraft that has a fixed wing and powered propellers or jets
have or possess, either concrete sense
a simple machine consisting of circular
spakes solid disc) that can rotate shaft axle (as in vehicles or other machines)
(kite.n.03)
fly.v.01,
sky.n.01)
plaything consisting of light frame covered tissue paper; flown wind at end of string
travel through the air; be airborne
the stmosphere and outer space as viewed from earth
(person.n.0)
drive.v.01, control vehicle
car.n.01)
human being
motor vehicle with four wheels; usually propelled by an internal combustion engine
Figure Examples of Triplets in VTKGs
the learning rate & {0.001, 0.01, 0.05, 0.1) and À E {0.001, 0.005, 0.01, 0.05, 0.1}. We tuned ComplEx-N3 (Lacroix et al., 2018) using the reg- ularization coefficient& {0.0, 0.01, 0.05, 0.1, 0.2). In RotatE (Sun et al., 2019) and PairRE (Chao al., 2021), (0.00005, 0.0001) and ? C [6.0, 9.0, 12.0). RSME (Wang et al., 2021) is tuned with the learn- ing rate E (0.005, 0.01, 0.01,0.05) 0.05] and the regular- ization coefficient E {0.05, 0.1, 0.15, 0.2}. We tuned TransAE (Wang et al., 2019) with A e {0.00005, 0.0001, 0.0005, 0.001, 0.01} and 7 E {2.0, 5.0, 7.5, 10.0). In MKGformer (Chen al., 2022), tried the learning rate (0.00005, 0.0005) in the fine-tuning stage. When we ran MKGformer, we had to decrease the batch size from 96 32 due to the memory constraint of our GPUs. For OTKGE (Cao et al., 2022), we tried the batch size E {2000, 5000} and the regularization coefficient (0.001, 0.005). For MoSE (Zhao et al., 2022) and IMF (Li et al., 2023), used the default parameters provided in their im- plementations. The results reported in the original IMF paper are not reproducible.
rate of the embedding matrices is 0.9, the dropout rate of the textual representation vectors is 0.1, and the step size of the cosine learning rate scheduler is 50. We validated the model performance every
50 epochs.
tried the learning rate
We tuned VISTA on VTKG-I with I. E {2, 3, 4), İ {1, (1,2,3). L E {1,2), number attention heads E {2, [2, the dropout rate of the embed- ding matrices E (0.5, 0.6, 0.7], the dropout rate of the visual representation vectors E {0.3, 0.4). and == C (1,3,5). We used 768 as the hidden di- mension of the transformers, 128 the batch size, and 0.01 as the dropout rate of the transformers. NVIDIA GeForce RTX 2080 Ti was used to run VISTA on VTKG-I, and it took approximately one minute for a single run with the best hyperparame- ters.
For VTKG-C, VISTA was tuned with İ E [1,2), number of attention heads {2, 1). the dropout rate of the visual representation vectors (0.3, 0.4), and the dropout rate of the textual rep- resentation vectors E {0.00 0.1}. We set the batch size to be 512, of the transformers to be 0.01. NVIDIA GeForce RTX 3090 was used to run VISTA on VTKG-C, and it took approximately three hours for single run with the best hyperparameters.
3, and the dropout rate
C Hyperparameters of VISTA
In VISTA, the learning rate was set to 0.0001 for VTKG-I and VTKG-C, and 0.001 for WN18RR++ and FB1152277. Unless otherwise stated, the hid- den dimension of the transformers was fixed to 2,048. We set 2, = 1, and & = 1. The dropout rate of the transformers 0.1, the dropout
We tuned VISTA WNISRR+ with number of attention heads E {8, 16), the dropout rate of the visual representation vectors E {0.2, 0.3), the dropout rate of the textual representation vectors E {0.0, 0.1}, and the step size of the cosine learning rate scheduler E {50, 100]. We set = 1 and

---

the batch size is 1024. We used NVIDIA Geforce RTX 3090 and NVIDIA RTX A6000 to run VISTA on WNISRR++, and it took approximately hours for single run with the best hyperparameters.
For FB15K237, VISTA was tuned with L E {1,2}, number of attention heads {32, 64), the dropout rate of the embedding matrices E {0.8, 0.9), the step size of the cosine learning rate scheduler с (50, 100), and the batch size E {512, 1024). We the dropout rate of the visual representation vectors and the textual representation vectors to 0.3 and 0.0, respectively, and 1. We used NVIDIA GeForce RTX 3090 and NVIDIA RTX A6000 to run VISTA on FB 1K5337,, and it took approximately 15 hours for single run with the best hyperparameters.


Reviews:
[{"review_id": "tixIcmttB1", "reviewer": "Reviewer_K6Ph", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reading our paper. In what follows, we provide our answers to your concerns. Please also consider that we prepared our answers to **seven different reviewers** during the rebuttal period. By incorporating the comments from seven reviewers, including yours, we got a chance to improve our paper. We hope the improvements can be considered in the final assessment of our paper.\n\n$\\rule{18cm}{0.05cm}$\n> 1. The results of the baselines deviate from what was presented in the original paper.\n\nAs described in Lines 424-427, we reproduced the results of the baselines by setting the dimension to be $d=256$ for all methods for a fair comparison; in MKGformer, $d$ is fixed to 768 since it directly uses ViT and BERT. Details about how we run the baseline methods are described in Appendix B. Among the datasets, FB15K237 is the only dataset on which we can check the baselines’ performances in their original papers because the other datasets were introduced in our paper. Among the baselines, ComplEx-N3, RotatE, PairRE, MKGformer, MoSE, and IMF reported their results on FB15K237; the other baselines, ANALOGY, RSME, TransAE, and OTKGE did not use FB15K237 in their papers. In Table A1, we present the baselines’ performances on FB15K237 reported in their original papers and the reproduced results; ‘N/A’ indicates the case where the original paper does not provide the result. We run VISTA with $d=256$ (reported in our original submission) and $d=512$ (added results during the rebuttal period) for comparisons with the baseline methods.\n\n### **Table A1: Performances of baseline methods and VISTA on FB15K237**\n\n$\\begin{array}{|c|c|c|}\n\\hline\n&&\\begin{array}{c}\n\\text{FB15K237}\\\\\\\\\n\\text{\\hspace{-12mu}MR\\hspace{33mu}MRR\\hspace{31mu}Hit@1\\hspace{26mu}Hit@3\\hspace{22mu}Hit@10\\hspace{-15mu}}\n\\end{array}\\\\\\\\\n\\hline\n\\text{ComplEx-N3}\n&\\begin{array}{c}\n\\text{Original Paper ($d=4000$)}\\\\\\\\\n\\text{Reproduced Results ($d=4000$)}\\\\\\\\\n\\text{Reproduced Results ($d=256$)}\n\\end{array}\n&\\begin{array}{ccccc}\n\\hspace{-1mu}\\text{N/A}\\hspace{7mu}&0.3700\\hspace{7.5mu}&\\text{N/A}\\hspace{7.5mu}&\\text{N/A}\\hspace{7.5mu}&0.5600\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}143.5\\hspace{7mu}&0.3666\\hspace{7.5mu}&0.2713\\hspace{7.5mu}&0.4018\\hspace{7.5mu}&0.5587\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}172.7\\hspace{7mu}&0.3510\\hspace{7.5mu}&0.2584\\hspace{7.5mu}&0.3847\\hspace{7.5mu}&0.5391\\hspace{4mu}\n\\end{array}\\\\\\\\\n\\hline\n\\text{RotatE}\n&\\begin{array}{c}\n\\text{Original Paper ($d=2000$)}\\\\\\\\\n\\text{Reproduced Results ($d=256$)}\n\\end{array}\n&\\begin{array}{ccccc}\n\\hspace{-1mu}177.0\\hspace{7mu}&0.3380\\hspace{7.5mu}&0.2410\\hspace{7.5mu}&0.3750\\hspace{7.5mu}&0.5330\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}246.1\\hspace{7mu}&0.3099\\hspace{7.5mu}&0.2183\\hspace{7.5mu}&0.3433\\hspace{7.5mu}&0.4932\\hspace{4mu}\n\\end{array}\\\\\\\\\n\\hline\n\\text{PairRE}\n&\\begin{array}{c}\n\\text{Original Paper ($d=2000$)}\\\\\\\\\n\\text{Reproduced Results ($d=256$)}\n\\end{array}\n&\\begin{array}{ccccc}\n\\hspace{-1mu}160.0\\hspace{7mu}&0.3510\\hspace{7.5mu}&0.2560\\hspace{7.5mu}&0.3870\\hspace{7.5mu}&0.5440\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}184.3\\hspace{7mu}&0.3326\\hspace{7.5mu}&0.2399\\hspace{7.5mu}&0.3675\\hspace{7.5mu}&0.5193\\hspace{4mu}\n\\end{array}\\\\\\\\\n\\hline\n\\text{MKGformer}\n&\\begin{array}{c}\n\\text{Original Paper ($d=768$)}\\\\\\\\\n\\text{Reproduced Results ($d=768$)}\n\\end{array}\n&\\begin{array}{ccccc}\n\\hspace{-1mu}221.0\\hspace{7mu}&\\text{N/A}\\hspace{7.5mu}&0.2560\\hspace{7.5mu}&0.3670\\hspace{7.5mu}&0.5040\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}297.6\\hspace{7mu}&0.3095\\hspace{7.5mu}&0.2278\\hspace{7.5mu}&0.3356\\hspace{7.5mu}&0.4740\\hspace{4mu}\n\\end{array}\\\\\\\\\n\\hline\n\\text{MoSE-AI}\n&\\begin{array}{c}\n\\text{Original Paper ($d=4000$)}\\\\\\\\\n\\text{Reproduced Results ($d=4000$)}\\\\\\\\\n\\text{Reproduced Results ($d=256$)}\n\\end{array}\n&\\begin{array}{ccccc}\n\\hspace{-1mu}135.0\\hspace{7mu}&\\text{N/A}\\hspace{7.5mu}&0.2550\\hspace{7.5mu}&0.3760\\hspace{7.5mu}&0.5180\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}134.9\\hspace{7mu}&0.3397\\hspace{7.5mu}&0.2514\\hspace{7.5mu}&0.3704\\hspace{7.5mu}&0.5144\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}149.1\\hspace{7mu}&0.3247\\hspace{7.5mu}&0.2384\\hspace{7.5mu}&0.3532\\hspace{7.5mu}&0.4965\\hspace{4mu}\n\\end{array}\\\\\\\\\n\\hline\n\\text{MoSE-BI}\n&\\begin{array}{c}\n\\text{Original Paper ($d=4000$)}\\\\\\\\\n\\text{Reproduced Results ($d=4000$)}\\\\\\\\\n\\text{Reproduced Results ($d=256$)}\n\\end{array}\n&\\begin{array}{ccccc}\n\\hspace{-1mu}117.0\\hspace{7mu}&\\text{N/A}\\hspace{7.5mu}&0.2810\\hspace{7.5mu}&0.4110\\hspace{7.5mu}&0.5650\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}117.8\\hspace{7mu}&0.3735\\hspace{7.5mu}&0.2792\\hspace{7.5mu}&0.4091\\hspace{7.5mu}&0.5635\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}132.3\\hspace{7mu}&0.3466\\hspace{7.5mu}&0.2570\\hspace{7.5mu}&0.3755\\hspace{7.5mu}&0.5303\\hspace{4mu}\n\\end{array}\\\\\\\\\n\\hline\n\\text{MoSE-MI}\n&\\begin{array}{c}\n\\text{Original Paper ($d=4000$)}\\\\\\\\\n\\text{Reproduced Results ($d=4000$)}\\\\\\\\\n\\text{Reproduced Results ($d=256$)}\n\\end{array}\n&\\begin{array}{ccccc}\n\\hspace{-1mu}127.0\\hspace{7mu}&\\text{N/A}\\hspace{7.5mu}&0.2680\\hspace{7.5mu}&0.3940\\hspace{7.5mu}&0.5400\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}140.9\\hspace{7mu}&0.3513\\hspace{7.5mu}&0.2613\\hspace{7.5mu}&0.3854\\hspace{7.5mu}&0.5298\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}148.9\\hspace{7mu}&0.3275\\hspace{7.5mu}&0.2416\\hspace{7.5mu}&0.3568\\hspace{7.5mu}&0.4984\\hspace{4mu}\n\\end{array}\\\\\\\\\n\\hline\n\\text{IMF}\n&\\begin{array}{c}\n\\text{Original Paper ($d=256$)}\\\\\\\\\n\\text{Reproduced Results ($d=256$)}\n\\end{array}\n&\\begin{array}{ccccc}\n\\hspace{-1mu}134.0\\hspace{7mu}&0.3890\\hspace{7.5mu}&0.2870\\hspace{7.5mu}&\\text{N/A}\\hspace{7.5mu}&0.5930\\hspace{4mu}\\\\\\\\\n\\hspace{-1mu}151.8\\hspace{7mu}&0.3677\\hspace{7.5mu}&0.2735\\hspace{7.5mu}&0.4040\\hspace{7.5mu}&0.5573\\hspace{4mu}\n\\end{array}\\\\\\\\\n\\hline\n\\text{VISTA}\n&\\begin{array}{c}\n\\text{Original Submission ($d=256$)}\\\\\\\\\n\\text{Added Results ($d=512$)}\n\\end{array}\n&\\begin{array}{ccccc}\n122.3&0.3732&0.2796&0.4081&0.5633\\hspace{4mu}\\\\\\\\\n\\bf{115.3}&\\bf{0.3788}&\\bf{0.2854}&\\bf{0.4144}&\\bf{0.5670}\\hspace{4mu}\n\\end{array}\\\\\\\\\n\\hline\n\\end{array}$\n\nIn Table A1, the key things to note are as follows:\n - The model performances are affected by the dimension ($d$); thus, using the same dimension for all methods is needed for a fair comparison, as we did in our paper.\n - VISTA outperforms the baseline methods even when it uses a smaller dimension than the baseline methods.\n - The authors of MKGformer[1] reported that their original implementation had a severe bug leading to data leakage problems. After fixing the bug, the performance of MKGformer was degraded. In Table A1, we use the fixed results reported on arXiv[2]. When we ran MKGformer, we had to decrease the batch size (from 96 to 32) due to the memory constraint of our GPUs.\n - The results of IMF are not precisely reproducible.\n\n**References:**\\\n[1] X. Chen et al., Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion, SIGIR 2022\\\n[2] X. Chen et al., Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion, arXiv:2205.02357v4\n\n$\\rule{18cm}{0.05cm}$\n\n> 2. The benchmark is collected from other datasets and lacks more analysis and validation.\n\n## Validations of VTKGs\nWhen creating VTKGs, we utilized the existing benchmark computer vision datasets and well-known knowledge bases, which are already widely used in research. We manually inspected and validated all triplets (except those from WN18RR++ since WN18RR++ is a subset of a standard lexical database, WordNet) in our VTKGs, as described in **Appendix A**. We developed the data labeling tool shown in Figure 6 to inspect each triplet to determine whether the triplet is valid and map each entity/relation in a triplet to an appropriate synset ID in WordNet (Lines 812-826). During this process, 269 triplets were removed among 19,065 original triplets; we removed invalid or hard-to-interpret triplets, e.g., $\\langle \\texttt{slave}, \\texttt{hang\\\\_from}, \\texttt{st} \\rangle$. When sampling 2,000 triplets and double-checking the labels, the labeling accuracy was 95.01%.\n\n## Additional Validations of VTKGs\nAdditionally, we checked the matching of each triplet and its image. In our datasets, 1,316 triplets are expressed by images, and there are 141,247 images for triplets. The triplet images came from VRD, UnRel, and HICO-DET, well-known computer vision benchmarks. To examine whether a triplet is well matched with its image, we randomly sampled 100 images for each triplet. If a triplet has less than 100 images, we examined all the images for the triplet. As a result, we examined 36,919 images, and each image was examined by two different annotators. We consider a triplet is matched with its image only if both annotators agree on it, i.e., if one of the annotators disagrees, we consider the triplet is not matched with its image. When measuring the matching degree in this way, the accuracy was 98.62%. In a revised paper, we will describe this process of validating the matching between triplets and their images.\n\n## Examples of Triplets in VTKGs\nWe provide some examples of the triplets and their images and descriptions in our VTKG datasets. As explained above, each entity or relation is represented using a synset of WordNet, e.g., ‘airplane.n.01’ or ‘have.v.01’, where ‘n’ indicates a noun, ‘v’ indicates a verb, and the last two digits indicate the sense number. The descriptions of entities and relations are from WordNet. Since we cannot use an external link here, we provide the image IDs of the source dataset for the triplet images. If our paper is accepted, we will release the full datasets and codes.\n\n--------------------------\n### **(airplane.n.01, have.v.01, wheel.n.01)**\n\n#### **[Descriptions]**\n\n**airplane.n.01**: an aircraft that has a fixed wing and is powered by propellers or jets\\\n**have.v.01**: have or possess, either in a concrete or an abstract sense\\\n**wheel.n.01**: a simple machine consisting of a circular frame with spokes (or a solid disc) that can rotate on a shaft or axle (as in vehicles or other machines)\n\n#### **[Images]**\n\n5406472568_78d687217f_b.jpg, 6994124346_d4813d9602_b.jpg, 8059265370_6843fc7ba4_b.jpg from **VRD**\n\n--------------------------\n\n### **(kite.n.03, fly.v.01, sky.n.01)**\n\n#### **[Descriptions]**\n\n**kite.n.03**: plaything consisting of a light frame covered with tissue paper; flown in wind at end of a string\\\n**fly.v.01**: travel through the air; be airborne\\\n**sky.n.01**: the atmosphere and outer space as viewed from the earth\n\n#### **[Images]**\n\n4433556373_ddfabedff5_o.jpg, 8642248228_d2187daf86_b.jpg, 9759509291_143489b32a_b.jpg from **VRD**\n\n--------------------------\n\n### **(person.n.01, drive.v.01, car.n.01)**\n\n#### **[Descriptions]**\n\n**person.n.01**: a human being\\\n**drive.v.01**: operate or control a vehicle\\\n**car.n.01**: a motor vehicle with four wheels; usually propelled by an internal combustion engine\n\n#### **[Images]**\n\nHICO_train2015_00002944.jpg, HICO_test2015_00005622.jpg from **HICO-DET**\\\n3141015389_d1dd4fa2ee_b.jpg from **VRD**\n\n____\n\n## More Analysis of VTKGs\nThe following tables show some additional statistics about our VTKG datasets. \n### **Table A2: Entities and relations with and without visual features**\n\n$\\begin{array}{rcc}\n\\hline\n & \\text{VTKG-I} & \\text{VTKG-C} \\\\\\\\\n\\hline\n\\text{Entities w/ $\\bf{h}\\_\\text{vis}$} & 181\\ (100\\\\%) & 7,863\\ (18.2\\\\%) \\\\\\\\\n\\text{Entities w/o $\\bf{h}\\_\\text{vis}$} & 0\\ (0\\\\%) & 35,404\\ (81.8\\\\%) \\\\\\\\\n\\text{Total} & 181 & 43,267 \\\\\\\\\n\\hline\n\\text{Relations w/ $\\\\mathbf{r}\\_\\\\text{vis}$} & 217\\ (100\\\\%) & 217\\ (7.9\\\\%) \\\\\\\\\n\\text{Relations w/o $\\\\mathbf{r}\\_\\\\text{vis}$} & 0\\ (0\\\\%) & 2,514\\ (92.1\\\\%) \\\\\\\\\n\\text{Total} & 217 & 2,731 \\\\\\\\\n\\hline\n\\end{array}$\n\n### **Table A3: Categorizing triplets based on whether their entities and relations have visual features**\n\n$\\begin{array}{rcc}\n\\hline\n & \\text{VTKG-I} & \\text{VTKG-C} \\\\\\\\\n\\hline\n\\text{two entities w/ visual features \\\\& relation w/ visual features} & 1,316\\ (100\\\\%) & 2,569\\ (2.3\\\\%) \\\\\\\\\n\\text{one entity w/ visual features \\\\& relation w/ visual features} & 0\\ (0\\\\%) & 1,743\\ (1.6\\\\%) \\\\\\\\\n\\text{two entities w/o visual features \\\\& relation w/ visual features} & 0\\ (0\\\\%) & 522\\ (0.5\\\\%) \\\\\\\\\n\\hline\n\\text{two entities w/ visual features \\\\& relation w/o visual features} & 0\\ (0\\\\%) & 10,166\\ (9.1\\\\%) \\\\\\\\\n\\text{one entity w/ visual features \\\\& relation w/o visual features} & 0\\ (0\\\\%) & 21,106\\ (18.9\\\\%) \\\\\\\\\n\\text{two entities w/o visual features \\\\& relation w/o visual features} & 0\\ (0\\\\%) & 75,385\\ (67.6\\\\%) \\\\\\\\\n\\hline\n\\text{Total} & 1,316 & 111,491 \\\\\\\\\n\\hline\n\\end{array}$\n\nFrom Table A2 and Table A3, we note that all entities and relations have their visual features on VTKG-I, whereas many entities and relations do not have visual features on VTKG-C. This analysis was beneficial to interpreting the results of our additional ablation studies requested by Reviewer bFL1 and Reviewer V1gZ.\n\n$\\rule{18cm}{0.05cm}$\n\n> 3. VISTA appears to be a mere combination of existing methods, and less innovative. Like Entity Encoding and Relation Encoding Transformer.\n\nTo the best of our knowledge, our entity and relation encoding transformers and the triplet decoding transformer are different from any existing method. Since VISTA is the first method that can handle images of triplets and visual/textual features of relations, the model architecture should be fundamentally distinguished from the existing methods. We would be pleased to discuss this point further when you provide specific references.\n\n$\\rule{18cm}{0.05cm}$\n\n> 4. The overall structure seems heavy and not easily scalable.\n\nWe first want to remind you that we already mentioned the scalability issue of VISTA in our Limitations Section (Lines 558-568). We present the runtimes of the baselines and VISTA on VTKG-C in the following figure. For a fair comparison, we compare the runtimes of methods that can handle different modalities. Even though making VISTA more scalable is one of our future works, we note that VISTA is the second best in terms of runtime and is much faster than MKGformer, OTKGE, and IMF.\n\n### **Figure A1: Runtimes of multimodal methods on VTKG-C**\nTransAE$\\hspace{30.3mu}$▒▒▒▒▒▒▒ (3 hours 40 minutes) \\\nMKGformer$\\hspace{4.2mu}$▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ (15 hours) \\\nOTKGE$\\hspace{38.6mu}$▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ ≈ ▒▒▒ (80 hours) \\\nMoSE$\\hspace{47.2mu}$▒▒ (40 minutes) \\\nIMF$\\hspace{59.8mu}$▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ (13 hours) \\\n**VISTA**$\\hspace{44.6mu}$▒▒▒▒▒▒ (3 hours) \\\n-----------------|-----------|-----------|-----------|-----------|-----------|-----------|----- ≈ -----|----- \\\n$\\hspace{76.5mu}$0h$\\hspace{43.0mu}$3h$\\hspace{43.0mu}$6h$\\hspace{43.0mu}$9h$\\hspace{40.0mu}$12h$\\hspace{34.5mu}$15h$\\hspace{34.5mu}$18h$\\hspace{45.5mu}$80h", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Detailed experiment and full ablation studies\n2. Clear and well-written", "reasons_to_reject": "1. The results of the baselines deviate from what was presented in the original paper.\n\n2. The benchmark is collected from other datasets and lacks more analysis and validation.\n\n3. VISTA appears to be a mere combination of existing methods, and less innovative. Like Entity Encoding and Relation Encoding Transformer.\n\n4. The overall structure seems heavy and not easily scalable.", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. This paper proposes a visual-textual knowledge graphs (VTKGs) benchmark, where triplets themselves are explained by images and the meanings of entities and relations are described in detail.\n2. Then propose VISTA, a knowledge graph representation learning method that incorporates the visual and textual representations of entities and relations using entity relation encoding. The visual features are extracted by ViT-Base, and the textual features are extracted by BERT.\n3. Experimental results on four real-world datasets demonstrate that VISTA outperforms 10 different SOTA methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "argX8FVXlf", "reviewer": "Reviewer_jTyB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reading our paper. In what follows, we provide our answers to your concerns. Please also consider that we prepared our answers to **seven different reviewers** during the rebuttal period. By incorporating the comments from seven reviewers, including yours, we got a chance to improve our paper. We hope the improvements can be considered in the final assessment of our paper.\n\n$\\rule{18cm}{0.05cm}$\n\n> The writing of the paper is not always clear, especially the description in the beginning part, making it not easy to understand the overview of the method.\n\nWe will refine the writing and more clearly describe the overview of our method in a revised paper. Any specific suggestions would be appreciated.\n\n$\\rule{18cm}{0.05cm}$\n\n> Maybe you can use a more illustrative Figure 1 to elaborate the difference of the KG representation discussed in this paper from previous multimodal representations.\n\nThank you for the suggestion! We will add a direct comparison between our VTKG and an existing multimodal knowledge graph (MKG) in Figure 1. The main similarities and differences between our VTKGs and existing MKGs are as follows.\n - Both can have images and text descriptions for entities.\n - While VTKGs can have images for triplets, MKGs do not assume that triplets can be represented using images. For example, in Figure 1, $\\langle \\texttt{person}, \\texttt{pull}, \\texttt{cart}\\rangle$ and $\\langle \\texttt{person}, \\texttt{ride}, \\texttt{horse}\\rangle$ are represented by images in our VTKG. These triplet images are utilized only in VTKGs but not in MKGs.\n - While VTKGs can have text descriptions of relations, most MKGs do not consider describing the lexical meanings of relations using text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The introduction of visual-textual knowledge graphs (VTKGs) marks an innovative step in knowledge graph representation. It goes beyond conventional approaches by incorporating both images and text descriptions for entities and triplets.\n* The creation of benchmark datasets with visually explainable triplets is an important contribution which could interest the researchers in the KG realm.\n* The development of the VISTA method, encompassing entity encoding, relation encoding, and triplet decoding transformers, showcases a sophisticated approach to learning knowledge graph representations in VTKGs.\n* They conducted extensive and comprehenisve experiments on four datasets with 10 different baselines. Experimental results revealed the real-world efficacy of the VISTA method by outperforming all baseline methods of knowledge graph completion.", "reasons_to_reject": "* The writing of the paper is not always clear, especially the description in the beginning part, making it not easy to understand the overview of the method. \n* Maybe you can use a more illustrative Figure 1 to elaborate the difference of the KG representation discussed in this paper from previous multimodal representations.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The research question of the paper revolves around enhancing knowledge graphs by incorporating visual and textual descriptions, specifically through the concept of visual-textual knowledge graphs (VTKGs). The authors introduce a novel type of knowledge graph representation VTKGs where not only entities, but also triplets, can be represented using images. They propose a method called VISTA for learning the representation of VTKGs, utilizing transformers for entity and relation encoding as well as triplet decoding. The main contributions of the paper include the introduction of VTKGs, the creation of benchmark datasets with visually explainable triplets, and the development of the VISTA method, which outperforms existing knowledge graph completion methods when applied to real-world VTKGs.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lc9Dppk2N5", "reviewer": "Reviewer_bFL1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reading our paper. In what follows, we provide our answers to your concerns. Please also consider that we prepared our answers to **seven different reviewers** during the rebuttal period. By incorporating the comments from seven reviewers, including yours, we got a chance to improve our paper. We hope the improvements can be considered in the final assessment of our paper. For better readability, we split and reordered the comments without missing any.\n\n$\\rule{18cm}{0.05cm}$\n\n> The presentation of this paper can be improved. Section 3.2 contains a long description of how they create VTKGs, without any sub-heading or text in bold, making others hard to capture the key points in their dataset constructing process.\n\nWe will add sub-headings and text in bold to let readers better catch the overall procedure of our dataset generation process. We summarize the key steps of our VTKG construction process as follows.\n - Step 1. To assemble visually expressible commonsense knowledge, we collected visual phrases from four computer vision benchmark datasets: VRD, UnRel, HICO-DET, and VisKE, where the first three datasets also provide images (Lines 158-180).\n - Step 2. We constructed ConceptNetW from ConceptNet by converting the triplets into a form consistent with the other datasets (Lines 196-208).\n - Step 3. We constructed WN18RR++ from WN18RR by mapping each entity into a unique synset ID (Lines 209-217).\n - Step 4. We constructed VTKG-I by combining VRD, UnRel, and HICO-DET. Also, we constructed VTKG-C by combining all the above datasets (Table 1). When we combined the datasets, we replaced each entity/relation in a triplet with a synset ID defined in WordNet to express triplets using the vocabulary defined in WordNet. All triplets (except those from WN18RR++ since WN18RR++ is a subset of WordNet) in our VTKGs were inspected and validated by human annotators (Lines 218-248 & Lines 812-826).\n\n$\\rule{18cm}{0.05cm}$\n\n> The quality of VTKG is unclear. In this paper, rather than building a new multi-modal knowledge graph from scratch, the authors design a pipeline to extract and merge triplets from CV and CSKG datasets. This kind of method may cause error accumulation in each step of the pipeline and thus reducing the overall quality of the final knowledge graph. Additionally, both evaluation of VTKG (E.g. matching degree of triplet and its image) and validation of each step in their pipeline are missing in this paper. All these things above make me worried about the quality of their constructed VTKG.\n\n## Validations of VTKGs\nWhen creating VTKGs, we utilized the existing benchmark computer vision datasets and well-known knowledge bases, which are already widely used in research. We manually inspected and validated all triplets (except those from WN18RR++ since WN18RR++ is a subset of a standard lexical database, WordNet) in our VTKGs, as described in **Appendix A**. We developed the data labeling tool shown in Figure 6 to inspect each triplet to determine whether the triplet is valid and map each entity/relation in a triplet to an appropriate synset ID in WordNet (Lines 812-826). During this process, 269 triplets were removed among 19,065 original triplets; we removed invalid or hard-to-interpret triplets, e.g., $\\langle \\texttt{slave}, \\texttt{hang\\\\_from}, \\texttt{st} \\rangle$. When sampling 2,000 triplets and double-checking the labels, the labeling accuracy was 95.01%.\n\n## Additional Validations of VTKGs\nAdditionally, we checked the matching of each triplet and its image. In our datasets, 1,316 triplets are expressed by images, and there are 141,247 images for triplets. The triplet images came from VRD, UnRel, and HICO-DET, well-known computer vision benchmarks. To examine whether a triplet is well matched with its image, we randomly sampled 100 images for each triplet. If a triplet has less than 100 images, we examined all the images for the triplet. As a result, we examined 36,919 images, and each image was examined by two different annotators. We consider a triplet is matched with its image only if both annotators agree on it, i.e., if one of the annotators disagrees, we consider the triplet is not matched with its image. When measuring the matching degree in this way, the accuracy was 98.62%. In a revised paper, we will describe this process of validating the matching between triplets and their images.\n\n## Examples of Triplets in VTKGs\nWe provide some examples of the triplets and their images and descriptions in our VTKG datasets. As explained above, each entity or relation is represented using a synset of WordNet, e.g., ‘airplane.n.01’ or ‘have.v.01’, where ‘n’ indicates a noun, ‘v’ indicates a verb, and the last two digits indicate the sense number. The descriptions of entities and relations are from WordNet. Since we cannot use an external link here, we provide the image IDs of the source dataset for the triplet images. If our paper is accepted, we will release the full datasets and codes.\n\n____\n\n### **(airplane.n.01, have.v.01, wheel.n.01)**\n\n#### **[Descriptions]**\n\n**airplane.n.01**: an aircraft that has a fixed wing and is powered by propellers or jets\\\n**have.v.01**: have or possess, either in a concrete or an abstract sense\\\n**wheel.n.01**: a simple machine consisting of a circular frame with spokes (or a solid disc) that can rotate on a shaft or axle (as in vehicles or other machines)\n\n#### **[Images]**\n\n5406472568_78d687217f_b.jpg, 6994124346_d4813d9602_b.jpg, 8059265370_6843fc7ba4_b.jpg from **VRD**\n\n____\n\n### **(kite.n.03, fly.v.01, sky.n.01)**\n\n#### **[Descriptions]**\n\n**kite.n.03**: plaything consisting of a light frame covered with tissue paper; flown in wind at end of a string\\\n**fly.v.01**: travel through the air; be airborne\\\n**sky.n.01**: the atmosphere and outer space as viewed from the earth\n\n#### **[Images]**\n\n4433556373_ddfabedff5_o.jpg, 8642248228_d2187daf86_b.jpg, 9759509291_143489b32a_b.jpg from **VRD**\n\n____\n\n### **(person.n.01, drive.v.01, car.n.01)**\n\n#### **[Descriptions]**\n\n**person.n.01**: a human being\\\n**drive.v.01**: operate or control a vehicle\\\n**car.n.01**: a motor vehicle with four wheels; usually propelled by an internal combustion engine\n\n#### **[Images]**\n\nHICO_train2015_00002944.jpg, HICO_test2015_00005622.jpg from **HICO-DET**\\\n3141015389_d1dd4fa2ee_b.jpg from **VRD**\n\n____\n\n## More Statistics of VTKGs\nThe following tables, Table A1 and Table A2 show some additional statistics about our VTKG datasets, which will be used for interpreting the results of our ablation studies shown below.\n\n### **Table A1: Entities and relations with and without visual features**\n\n$\\begin{array}{rcc}\n\\hline\n & \\text{VTKG-I} & \\text{VTKG-C} \\\\\\\\\n\\hline\n\\text{Entities w/ $\\bf{h}\\_\\text{vis}$} & 181\\ (100\\\\%) & 7,863\\ (18.2\\\\%) \\\\\\\\\n\\text{Entities w/o $\\bf{h}\\_\\text{vis}$} & 0\\ (0\\\\%) & 35,404\\ (81.8\\\\%) \\\\\\\\\n\\text{Total} & 181 & 43,267 \\\\\\\\\n\\hline\n\\text{Relations w/ $\\\\mathbf{r}\\_\\\\text{vis}$} & 217\\ (100\\\\%) & 217\\ (7.9\\\\%) \\\\\\\\\n\\text{Relations w/o $\\\\mathbf{r}\\_\\\\text{vis}$} & 0\\ (0\\\\%) & 2,514\\ (92.1\\\\%) \\\\\\\\\n\\text{Total} & 217 & 2,731 \\\\\\\\\n\\hline\n\\end{array}$\n\n### **Table A2: Categorizing triplets based on whether their entities and relations have visual features**\n\n$\\begin{array}{rcc}\n\\hline\n & \\text{VTKG-I} & \\text{VTKG-C} \\\\\\\\\n\\hline\n\\text{two entities w/ visual features \\\\& relation w/ visual features} & 1,316\\ (100\\\\%) & 2,569\\ (2.3\\\\%) \\\\\\\\\n\\text{one entity w/ visual features \\\\& relation w/ visual features} & 0\\ (0\\\\%) & 1,743\\ (1.6\\\\%) \\\\\\\\\n\\text{two entities w/o visual features \\\\& relation w/ visual features} & 0\\ (0\\\\%) & 522\\ (0.5\\\\%) \\\\\\\\\n\\hline\n\\text{two entities w/ visual features \\\\& relation w/o visual features} & 0\\ (0\\\\%) & 10,166\\ (9.1\\\\%) \\\\\\\\\n\\text{one entity w/ visual features \\\\& relation w/o visual features} & 0\\ (0\\\\%) & 21,106\\ (18.9\\\\%) \\\\\\\\\n\\text{two entities w/o visual features \\\\& relation w/o visual features} & 0\\ (0\\\\%) & 75,385\\ (67.6\\\\%) \\\\\\\\\n\\hline\n\\text{Total} & 1,316 & 111,491 \\\\\\\\\n\\hline\n\\end{array}$\n\n$\\rule{18cm}{0.05cm}$\n\n> The discussion of VISTA’s each component in section 5.4 is too brief, considering VISTA is quite complex with many different components. Conclusions and analysis like which component play a more critical role in VISTA should be concluded.\n\n## More Experiments for Ablation Studies\n\nWe added more ablation studies in Table A3. The blue colored rows indicate the new experiments. The leftmost column indicates a higher-level category of our ablation study. Note that Table A3 also includes all results shown in Table 5 for complete analysis.\n - Modality: We show the results of VISTA using different combinations of the modalities. Recall that $\\bf{h}\\_\\text{vis}$ indicates the visual feature of an entity, $\\bf{h}\\_\\text{txt}$ indicates the textual feature of an entity, $\\bf{r}\\_\\text{vis}$ indicates the visual feature of a relation, and $\\bf{r}\\_\\text{txt}$ indicates the textual feature of a relation.\n - Entity Modality: We show the impact of different modalities for entities.\n - Relation Modality: We show the impact of different modalities for relations.\n - Encoder: We added the ablation study of removing the entity transformer in VISTA, where we replace the entity transformer with a simple learnable vector $\\bf{h}$ for each entity.\n - Decoder: We copied the results regarding the decoder from Table 5 for complete comparisons.\n\n### **Table A3: Additional ablation studies**\n\n$\\begin{array}{|c|c|c|c|}\\hline&&\\text{VTKG-I}&\\text{VTKG-C} \\\\\\\\&&\\text{MR\\hspace{23mu}MRR\\hspace{23mu}Hit@1\\hspace{18mu}Hit@3\\hspace{15mu}Hit@10} & \\text{\\hspace{-2.5mu}MR\\hspace{30.5mu}MRR\\hspace{24mu}Hit@1\\hspace{18mu}Hit@3\\hspace{15mu}Hit@10\\hspace{-5mu}}\n\\\\\\\\\n\\hline\n\\begin{array}{c} \\text{Modality} \\\\\\\\ \\text{(Entity + Relation)} \\end{array} & \\begin{array}{c}\\text{\\textcolor{blue}{w/o $\\bf{h}\\_\\text{vis}$, $\\bf{h}\\_\\text{txt}$, $\\bf{r}\\_\\text{vis}$, $\\bf{r}\\_\\text{txt}$}} \\\\\\\\ \\text{\\textcolor{blue}{w/o $\\bf{h}\\_\\text{vis}$, $\\bf{r}\\_\\text{vis}$}} \\\\\\\\ \\text{\\textcolor{blue}{w/o $\\bf{h}\\_\\text{txt}$, $\\bf{r}\\_\\text{txt}$}} \\end{array} & \\hspace{6mu}\\begin{array}{ccccc} 23.3 & 0.3979 & 0.3092 &0.4313 & 0.5725 \\\\\\\\ 21.0& 0.4423& 0.3473& 0.4542& 0.6450 \\\\\\\\ 19.5& 0.4525& 0.3626& 0.4733& 0.6336 \\end{array}\\hspace{10mu} & \\begin{array}{ccccc} 3163.9 &0.3651 & 0.3252 & 0.3806 & 0.4406 \\\\\\\\  238.7& 0.4556& 0.3827& 0.4811& 0.6014 \\\\\\\\ 2641.1& 0.3743& 0.3331& 0.3893& 0.4523 \\end{array}\\hspace{10mu}\\\\\\\\\n\\hline\n\\text{Entity Modality}& \\begin{array}{c}\\text{\\textcolor{blue}{w/o $\\bf{h}\\_\\text{vis}$, $\\bf{h}\\_\\text{txt}$}} \\\\\\\\ \\text{\\textcolor{blue}{w/o $\\bf{h}\\_\\text{vis}$}} \\\\\\\\ \\text{\\textcolor{blue}{w/o $\\bf{h}\\_\\text{txt}$}} \\end{array} &\\hspace{6mu}\\begin{array}{ccccc} 22.2& 0.3970& 0.2939& 0.4389& 0.5916 \\\\\\\\ 20.5& 0.4149& 0.3053& 0.4542& 0.6221 \\\\\\\\ 17.2& 0.4590& 0.3588& 0.4962& 0.6641 \\end{array}\\hspace{10mu} & \\begin{array}{ccccc} 3036.6& 0.3661& 0.3268& 0.3816& 0.4403 \\\\\\\\ 229.1& 0.4569& 0.3832& 0.4842& 0.6028 \\\\\\\\ 2599.5& 0.3721& 0.3312& 0.3871& 0.4490 \\end{array}\\hspace{10mu}\\\\\\\\\n\\hline\n\\text{Relation Modality}& \\begin{array}{c} \\text{\\textcolor{blue}{w/o $\\bf{r}\\_\\text{vis}$, $\\bf{r}\\_\\text{txt}$}} \\\\\\\\ \\text{\\textcolor{blue}{w/o $\\bf{r}\\_\\text{vis}$}} \\\\\\\\ \\text{\\textcolor{blue}{w/o $\\bf{r}\\_\\text{txt}$}} \\\\\\\\ \\text{replace $\\bf{r_{\\text{vis}}}$ w/ $\\bf{u}$} \\end{array} &\\hspace{6mu}\\begin{array}{ccccc} 18.8& 0.4297& 0.3244& 0.4656& 0.6412 \\\\\\\\ 19.0& 0.4546& 0.3588& 0.4924& 0.6412 \\\\\\\\ 17.3& 0.4587& 0.3511& 0.5000& 0.6641 \\\\\\\\ 18.3& 0.4334& 0.3359& 0.4580& 0.6489 \\end{array}\\hspace{10mu} & \\begin{array}{ccccc} 235.8& 0.4601& 0.3856& 0.4865& 0.6097 \\\\\\\\ 223.7& 0.4661& 0.3910& 0.4935& 0.6127 \\\\\\\\ 236.3& 0.4608& 0.3845& 0.4903& 0.6088 \\\\\\\\ \\hspace{4.5mu}318.1\\hspace{4.5mu}& 0.4159& 0.3415& 0.4454& 0.5573 \\end{array}\\hspace{10mu}\\\\\\\\\n\\hline\n\\text{Encoder}& \\begin{array}{c} \\text{w/o $\\bf{z_\\text{\\[ENT\\]}}$ and $\\bf{z_\\text{\\[REL\\]}}$}\\\\\\\\ \\text{w/o $\\bf{h}$ and $\\bf{r}$} \\\\\\\\ \\text{w/o rel. trans.} \\\\\\\\ \\text{\\textcolor{blue}{w/o ent. trans.}} \\end{array} &\\hspace{6mu}\\begin{array}{ccccc} 18.6&0.4022&0.2901&0.4542&0.6221 \\\\\\\\ 19.0&0.4376&0.3282&0.4847&0.6489 \\\\\\\\ 18.4&0.4168&0.3015&0.4733&0.6221 \\\\\\\\ 25.2&0.4065&0.3321&0.4237&0.5458  \\end{array}\\hspace{10mu} & \\begin{array}{ccccc}\\hspace{4.5mu} 316.8 \\hspace{4.5mu}& 0.4018& 0.3251& 0.4340& 0.5486 \\\\\\\\ 250.9&0.4497&0.3719&0.4835&0.5976 \\\\\\\\ 369.3&0.3889&0.3119&0.4203&0.5381 \\\\\\\\ 4368.7&0.3615&0.3194&0.3775&0.4442 \\end{array}\\hspace{10mu}\\\\\\\\\n\\hline\n\\text{Decoder} & \\begin{array}{c} \\text{w/ DistMult decoder} \\\\\\\\ \\text{replace dot prod. w/ lin.} \\end{array} &\\hspace{6mu}\\begin{array}{ccccc} 21.0& 0.3911& 0.2824& 0.4160& 0.6336 \\\\\\\\ 29.5& 0.4063& 0.3168& 0.4351& 0.5802 \\end{array}\\hspace{10mu} & \\begin{array}{ccccc} \\hspace{4.5mu}420.6\\hspace{4.5mu}& 0.4195& 0.3572& 0.4385& 0.5419 \\\\\\\\ 964.5& 0.4297& 0.3796& 0.4478& 0.5236 \\end{array}\\hspace{10mu}\\\\\\\\\n\\hline\n&\\text{VISTA} & \\hspace{6mu}\\begin{array}{ccccc} 17.3& 0.4650& 0.3626& 0.5076& 0.6641 \\end{array}\\hspace{10mu} & \\begin{array}{ccccc} \\hspace{4.5mu}220.8\\hspace{4.5mu}& 0.4675& 0.3918& 0.4961& 0.6157 \\end{array}\\hspace{10mu}\\\\\\\\\n\\hline\n\\end{array}$\n\n## Conclusions and Analysis of the Additional Ablation Studies\nWe summarize our conclusions drawn from our ablation studies as follows; we will add the results of our additional experiments and analysis in a revised paper.\n - On both VTKG-I and VTKG-C, we see that adding visual or textual features leads to better performance than the case without those features. More importantly, using all modalities presented in the last row of Table A3 leads to the best performance.\n - The impact of the visual features is more prominent on VTKG-I than on VTKG-C. To understand this point more thoroughly, we focus on the statistics about categorizing triplets based on whether their entities and relations have visual features, shown in Table A2. While all entities and relations have their visual features on VTKG-I, many entities and relations do not have visual features on VTKG-C. As a result, visual features are more important than textual features on VTKG-I, while textual features are more important than visual features on VTKG-C.\n - When we compare the impact of the entity modality and the relation modality, removing a modality of entities is more critical than removing a modality of relations on both VTKG-I and VTKG-C.\n - The performance decreases of VISTA with ‘replace $\\bf{r}\\_\\text{vis}$ w/ $\\bf{u}$’ indicate that our way of constructing $\\bf{r}\\_\\text{vis}$ is more effective than just using the union bounding box of a triplet when creating a visual representation vector of $r$.\n - By looking at the first two rows of Encoder in Table A3, we see that $\\bf{z_\\text{[ENT]}}$, $\\bf{z_\\text{[REL]}}$, $\\bf{h}$, and $\\bf{r}$ are all important inputs of the entity and relation encoding transformers. By looking at the last two rows of Encoder in Table A3, we note that removing either the entity encoding transformer or the relation encoding transformer greatly impacts VISTA's performance. Regarding Hit@1, removing the relation encoding transformer leads to a more critical decrease in performance than removing the entity encoding transformer. On the other hand, for the other metrics (i.e., MR, MRR, Hit@3, and Hit@10), the entity encoding transformer is more critical than the relation encoding transformer.\n - The two rows of Decoder in Table A3 indicate that making changes in the way of predicting a missing entity for knowledge graph completion leads to degrading the performance of VISTA (Lines 517-520). This validates the effectiveness of our triplet decoding transformer.\n\n$\\rule{18cm}{0.05cm}$\n\n> They do not clearly validate the effectiveness of introducing image information of triplet into the multimodal knowledge graph. One of the most important contributions they make in this paper is proposing to express triplet in knowledge graph with its image. However, they fail to quantitatively validate the effectiveness of doing so in the experiment section. I suggest adding a new ablation study to train each model w/ and w/o visual information of triplet.\n\nWe present the results by removing triplet images $\\mathcal{I}\\_\\{tri}$ in Table A4, where we also present the ratios of the affected triplets by removing $\\mathcal{I}\\_\\{tri}$ (denoted by **'Aff.'**). Note that the numbers presented in the Aff. column are computed based on Table A2. The performance drops in VTKG-I confirm the effectiveness of triplet images. Even though the performance drops in VTKG-C seem not as significant as in VTKG-I, that is mainly because the ratios of the affected triplets (Aff.) by removing the visual information of triplets are not substantial in VTKG-C.\n\n### **Table A4: Effect of triplet images**\n\n$\\begin{array}{|c|c|c|}\\hline&\\text{VTKG-I}&\\text{VTKG-C} \\\\\\\\\n&\\text{Aff.\\hspace{24mu}MR\\hspace{23mu}MRR\\hspace{21mu}Hit@10} & \\text{\\hspace{-1mu}Aff.\\hspace{30mu}MR\\hspace{30mu}MRR\\hspace{21mu}Hit@10}\n\\\\\\\\\n\\hline\n\\begin{array}{c}\n\\text{w/o $\\mathcal{I}\\_\\text{tri}$}\n\\end{array}\n& \\begin{array}{cccc}\n100\\\\% & \\hspace{1.5mu}19.0\\hspace{1.5mu} & \\hspace{1.5mu}0.4546\\hspace{1.5mu} & 0.6412\n\\end{array}\\hspace{10mu}\n& \\begin{array}{cccc}\n\\hspace{3mu}4.3\\\\% & \\hspace{6mu}223.7 & \\hspace{3mu}0.4661\\hspace{3mu} & 0.6127\n\\end{array}\\hspace{10mu}\\\\\\\\\n\\text{VISTA}\n&\\hspace{6mu}\n\\begin{array}{ccccc}\n\\hspace{3mu}\\text{N/A}& \\hspace{1mu}\\bf{17.3} & \\hspace{-3mu}\\bf{0.4650} & \\hspace{-5mu}\\bf{0.6641}\n\\end{array}\n\\hspace{10mu}\n&\\begin{array}{ccccc}\n\\hspace{7.5mu}\\text{N/A} & \\hspace{4mu}\\bf{220.8}& \\hspace{-2mu}\\bf{0.4675} & \\hspace{-5mu}\\bf{0.6157}\n\\end{array}\n\\hspace{10mu}\\\\\\\\\n\\hline\n\\end{array}$\n\n$\\rule{18cm}{0.05cm}$\n\n> They could also consider training models in VTKG and another existing multi-modal knowledge graph respectively, and then testing in a new knowledge graph dataset (out-of-domain setting).\n\nThroughout the paper, we assume the conventional transductive setting for knowledge graph completion. That is, all entities and relations are assumed to be observed during training, and what we predict during testing is a plausible combination of the entities and relations. This is the setting that most existing multi-modal knowledge graph embedding methods assume. To make predictions in the out-of-domain setting, we should consider an inductive learning scenario, which is out of the scope of our paper; this can be an interesting future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of expressing the triplets with their images sounds novel and useful. This approach empowers the multi-modal knowledge graph to encompass richer and more comprehensive information, thereby potentially benefiting downstream tasks like visual question answering.\n2. Their VISTA achieves promising results in both multi-modal and traditional knowledge graph completion tasks. The subsequent qualitative analysis makes their method more direct and explainable.", "reasons_to_reject": "1. The presentation of this paper can be improved. Section 3.2 contains a long description of how they create VTKGs, without any sub-heading or text in bold, making others hard to capture the key points in their dataset constructing process. The discussion of VISTA’s each component in section 5.4 is too brief, considering VISTA is quite complex with many different components. Conclusions and analysis like which component play a more critical role in VISTA should be concluded.\n\n2. The quality of VTKG is unclear. In this paper, rather than building a new multi-modal knowledge graph from scratch, the authors design a pipeline to extract and merge triplets from CV and CSKG datasets. This kind of method may cause error accumulation in each step of the pipeline and thus reducing the overall quality of the final knowledge graph. Additionally, both evaluation of VTKG (E.g. matching degree of triplet and its image) and validation of each step in their pipeline are missing in this paper. All these things above make me worried about the quality of their constructed VTKG.\n\n3. They do not clearly validate the effectiveness of introducing image information of triplet into the multimodal knowledge graph. One of the most important contributions they make in this paper is proposing to express triplet in knowledge graph with its image. However, they fail to quantitatively validate the effectiveness of doing so in the experiment section. I suggest adding a new ablation study to train each model w/ and w/o visual information of triplet. They could also consider training models in VTKG and another existing multi-modal knowledge graph respectively, and then testing in a new knowledge graph dataset (out-of-domain setting).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors of this paper present a novel approach aimed at enhancing the multi-modal knowledge graph by incorporating image information into each triplet. To assemble such visual-textual knowledge graph (VTKG), they leverage both computer vision (CV) datasets and commonsense knowledge graphs (CSKG). To be specific, they extract images and corresponding visual phrases (triplets) from the datasets in different computer vision tasks. And then, they rewrite and complement triplets of two large commonsense knowledge graphs, ConceptNet and WN18RR. Finally, they manually merge the triplet from both CV datasets and CSKG according to a group of pre-defined sunsets.\n\nThey also design a method of learning representations in visual-textual knowledge graph (VISTA). It first separately encodes visual (image) and textual descriptions with ViT and BERT respectively. With the visual and textual features extracted, they utilize transformer-based encoders to merge multi-source information (visual, textual and triplet) and then learn to predict the missing entities with a triplet decoder. They conduct knowledge graph completion (KGC) task to evaluate their VISTA with VTKG and other knowledge graphs. Experiment results demonstrate their VISTA outperforms the compared models in both multi-modal and traditional knowledge graph. They also conduct qualitative analysis to further validate and explain the effectiveness of VISTA.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F4Sux7rY9Q", "reviewer": "Reviewer_6NUN", "summary": "", "questions": "● At line 290, does the [CLS] is the last token?\n● Why do you put the implementation details in section 4, not section 5, about lines 371-375?\n● What do the up and down arrows mean at lines 430-431?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reading our paper. In what follows, we provide our answers to your concerns. Please also consider that we prepared our answers to **seven different reviewers** during the rebuttal period. By incorporating the comments from seven reviewers, including yours, we got a chance to improve our paper. We hope the improvements can be considered in the final assessment of our paper.\n\n$\\rule{18cm}{0.05cm}$\n\n> There are no demo datasets provided.\n\nWe provide some examples of the triplets and their images and descriptions in our VTKG datasets. As explained in the paper (Lines 225-246), each entity or relation is represented using a synset of WordNet, e.g., ‘airplane.n.01’ or ‘have.v.01’, where ‘n’ indicates a noun, ‘v’ indicates a verb, and the last two digits indicate the sense number. The descriptions of entities and relations are from WordNet. Since we cannot use an external link here, we provide the image IDs of the source dataset for the triplet images. If our paper is accepted, we will release the full datasets and codes.\n\n----------------\n\n### **(airplane.n.01, have.v.01, wheel.n.01)**\n\n#### **[Descriptions]**\n\n**airplane.n.01**: an aircraft that has a fixed wing and is powered by propellers or jets\\\n**have.v.01**: have or possess, either in a concrete or an abstract sense\\\n**wheel.n.01**: a simple machine consisting of a circular frame with spokes (or a solid disc) that can rotate on a shaft or axle (as in vehicles or other machines)\n\n#### **[Images]**\n\n5406472568_78d687217f_b.jpg, 6994124346_d4813d9602_b.jpg, 8059265370_6843fc7ba4_b.jpg from **VRD**\n\n____\n\n### **(kite.n.03, fly.v.01, sky.n.01)**\n\n#### **[Descriptions]**\n\n**kite.n.03**: plaything consisting of a light frame covered with tissue paper; flown in wind at end of a string\\\n**fly.v.01**: travel through the air; be airborne\\\n**sky.n.01**: the atmosphere and outer space as viewed from the earth\n\n#### **[Images]**\n\n4433556373_ddfabedff5_o.jpg, 8642248228_d2187daf86_b.jpg, 9759509291_143489b32a_b.jpg from **VRD**\n\n____\n\n### **(person.n.01, drive.v.01, car.n.01)**\n\n#### **[Descriptions]**\n\n**person.n.01**: a human being\\\n**drive.v.01**: operate or control a vehicle\\\n**car.n.01**: a motor vehicle with four wheels; usually propelled by an internal combustion engine\n\n#### **[Images]**\n\nHICO_train2015_00002944.jpg, HICO_test2015_00005622.jpg from **HICO-DET**\\\n3141015389_d1dd4fa2ee_b.jpg from **VRD**\n\n____\n\n$\\rule{18cm}{0.05cm}$\n\n> Some parts of the paper writing are not rigorous and incoherent.\n\nWhile any specific suggestions would be appreciated, we can confirm that we will refine and revise our paper by incorporating the comments suggested by all reviewers.\n\n$\\rule{18cm}{0.05cm}$\n\n> At line 290, does the [CLS] is the last token?\n\n[CLS] is the first token of a sequence. In Line 290, by saying “the last hidden state”, we meant “the hidden state at the last layer”. We will paraphrase the expression.\n\n$\\rule{18cm}{0.05cm}$\n\n> Why do you put the implementation details in section 4, not section 5, about lines 371-375?\n\nAs you suggested, we will move the implementation details (Lines 371-375) to Section 5.\n\n$\\rule{18cm}{0.05cm}$\n\n> What do the up and down arrows mean at lines 430-431?\n\nIn Lines 430-431, the down arrow next to MR indicates that a lower MR implies better performance, whereas the up arrows next to MRR and Hit@N indicate that higher MRR and Hit@N imply better performance.\n\n$\\rule{18cm}{0.05cm}$\n\n> Incoherent descriptions appear at lines 251-257.\n\nWould you mind letting us know what kind of incoherence you observed in Lines 251-257?\n\n$\\rule{18cm}{0.05cm}$\n\n> improper title for Section 5.\n\nWe will rename the title of Section 5 as “Experiments”. Other suggestions would also be welcomed if available.\n\n$\\rule{18cm}{0.05cm}$\n\n> The caption of Figure 4 is ambiguous and hard to understand.\n\nTo show Figure 4, we consider three relations, $\\texttt{Feed}$, $\\texttt{Repair}$, and $\\texttt{Carry}$, and select images representing triplets having those three relations in the test set. Among these selected images, assume that the $j$-th image contains $\\texttt{Feed}$. Then, we compute $\\widehat{\\bf{W}}\\_\\text{vis} \\bf{r}\\_{\\text{vis},j}$ which is the visual representation vector of $\\texttt{Feed}$ for the $j$-th image. In this way, we compute the visual representation vectors of the three relations on the selected images and visualize them in Figure 4 (Lines 458-469). Note that $\\widehat{\\bf{W}}\\_\\text{vis}$ is the projection matrix learned during training that converts visual features to visual representation vectors of relations (Lines 320-334). We used the images of the test triplets (those having the above three relations) only for visualization in Figure 4; we never used images of test triplets from training to testing of VISTA, as described in Lines 417-423. We will add more details to the caption of Figure 4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "● This paper proposes a VTKG representation learning method.\n● The SOTA performance on various datasets.\n● The description of the experiment is detailed.\n● The figures of the paper provide effective explanations for the idea.", "reasons_to_reject": "● There are no demo datasets provided.\n● Some parts of the paper writing are not rigorous and incoherent. ", "questions_to_authors": "● At line 290, does the [CLS] is the last token?\n● Why do you put the implementation details in section 4, not section 5, about lines 371-375?\n● What do the up and down arrows mean at lines 430-431?", "paper_topic_and_main_contributions": "This paper proposes a method VISTA for visual-textual knowledge graphs and conducts new VTKGs benchmark datasets. Extensive experiments validate the effectiveness of VISTA. ", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "● Incoherent descriptions appear at lines 251-257.\n● improper title for Section 5.\n● The caption of Figure 4 is ambiguous and hard to understand."}, "extra_text": {}}, {"review_id": "UrNvj3DkDo", "reviewer": "Reviewer_iJiR", "summary": "", "questions": "Q1: while reading section 4, the operator || appears constantly. I understand it as an element list separator. Am I wrong? Maybe Figure 3 could show the parts that corresponds to some of the equation (e.g. X^(0)). An idea could be to put a color shade that highlight the elements that corresponds to X^(0), X^L,...\nQ2: I'm missing a table that shows the final statistics of VTKG. I can see table 2 does the same for the DB that were use as sources, but does it means that VTKG is just the straight combination of them?\n\n", "limitations": "", "rating": 5, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reading our paper. In what follows, we provide our answers to your concerns. Please also consider that we prepared our answers to **seven different reviewers** during the rebuttal period. By incorporating the comments from seven reviewers, including yours, we got a chance to improve our paper. We hope the improvements can be considered in the final assessment of our paper.\n\n$\\rule{15cm}{0.05cm}$\n\n> If the code of the model and the dataset is not release, it could be difficult to replicate the results. However, this alone wouldn't be a strong reason to reject the paper.\n\nWe will release all our datasets and codes if this paper is accepted. In what follows, we provide some examples of the triplets and their images and descriptions in our VTKG datasets. As explained in the paper (Lines 225-246), each entity or relation is represented using a synset of WordNet, e.g., ‘airplane.n.01’ or ‘have.v.01’, where ‘n’ indicates a noun, ‘v’ indicates a verb, and the last two digits indicate the sense number. The descriptions of entities and relations are from WordNet. Since we cannot use an external link here, we provide the image IDs of the source dataset for the triplet images.\n\n____\n### **(airplane.n.01, have.v.01, wheel.n.01)**\n\n#### **[Descriptions]**\n\n**airplane.n.01**: an aircraft that has a fixed wing and is powered by propellers or jets\\\n**have.v.01**: have or possess, either in a concrete or an abstract sense\\\n**wheel.n.01**: a simple machine consisting of a circular frame with spokes (or a solid disc) that can rotate on a shaft or axle (as in vehicles or other machines)\n\n#### **[Images]**\n\n5406472568_78d687217f_b.jpg, 6994124346_d4813d9602_b.jpg, 8059265370_6843fc7ba4_b.jpg from **VRD**\n\n____\n\n### **(kite.n.03, fly.v.01, sky.n.01)**\n\n#### **[Descriptions]**\n\n**kite.n.03**: plaything consisting of a light frame covered with tissue paper; flown in wind at end of a string\\\n**fly.v.01**: travel through the air; be airborne\\\n**sky.n.01**: the atmosphere and outer space as viewed from the earth\n\n#### **[Images]**\n\n4433556373_ddfabedff5_o.jpg, 8642248228_d2187daf86_b.jpg, 9759509291_143489b32a_b.jpg from **VRD**\n\n____\n\n### **(person.n.01, drive.v.01, car.n.01)**\n\n#### **[Descriptions]**\n\n**person.n.01**: a human being\\\n**drive.v.01**: operate or control a vehicle\\\n**car.n.01**: a motor vehicle with four wheels; usually propelled by an internal combustion engine\n\n#### **[Images]**\n\nHICO_train2015_00002944.jpg, HICO_test2015_00005622.jpg from **HICO-DET**\\\n3141015389_d1dd4fa2ee_b.jpg from **VRD**\n\n____\n\n$\\rule{15cm}{0.05cm}$\n\n> Q1: while reading section 4, the operator || appears constantly. I understand it as an element list separator. Am I wrong? Maybe Figure 3 could show the parts that corresponds to some of the equation (e.g. X^(0)). An idea could be to put a color shade that highlight the elements that corresponds to X^(0), X^L,... \n\nYou understood it correctly. We will revise Figure 3 as you suggested. Thank you for the suggestion!\n\n$\\rule{15cm}{0.05cm}$\n\n> Q2: I'm missing a table that shows the final statistics of VTKG. I can see table 2 does the same for the DB that were use as sources, but does it means that VTKG is just the straight combination of them?\n\nThe first two rows of Table 2 show the final statistics of VTKGs, and Table 1 shows the statistics of the source datasets of VTKGs. Our VTKG datasets are not straight combinations of the source datasets; how we merged the datasets is described in Lines 225-248.\n\n$\\rule{15cm}{0.05cm}$\n\n> The worked reminded me a bit to this AKBC paper:\\\nDaniel Oñoro-Rubio, Mathias Niepert, Alberto García-Durán, Roberto Gonzalez-Sanchez, Roberto Javier López-Sastre: Answering Visual-Relational Queries in Web-Extracted Knowledge Graphs. AKBC 2019\\\nFor the Multimodal Knowledge Graph Completion section: Ye Liu, Hui Li, Alberto García-Durán, Mathias Niepert, Daniel Oñoro-Rubio, David S. Rosenblum: MMKG: Multi-modal Knowledge Graphs. ESWC 2019: 459-474\\\nAnother visual KG application in medical domain: Anjany Sekuboyina, Daniel Oñoro-Rubio, Jens Kleesiek, Brandon Malone: A Relational-Learning Perspective To Multi-Label Chest X-Ray Classification. ISBI 2021: 1618-1622\n\nThank you for the suggestions! We will mention the first two papers in Section 2, explaining how our work differs from theirs: while their works assume that only entities have images, our VTKGs assume that triplets can also be represented using images, and the relations have visual/textual features. We will mention the last paper in Section 6, describing possible applications of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper could provide to the community a new cross-domain benchmark. The paper is well written, the model and datasets seems technically good. The experimental setup is reasonable, and the results are good.", "reasons_to_reject": "If the code of the model and the dataset is not release, it could be difficult to replicate the results. However, this alone wouldn't be a strong reason to reject the paper.", "questions_to_authors": "Q1: while reading section 4, the operator || appears constantly. I understand it as an element list separator. Am I wrong? Maybe Figure 3 could show the parts that corresponds to some of the equation (e.g. X^(0)). An idea could be to put a color shade that highlight the elements that corresponds to X^(0), X^L,...\nQ2: I'm missing a table that shows the final statistics of VTKG. I can see table 2 does the same for the DB that were use as sources, but does it means that VTKG is just the straight combination of them?\n\n", "paper_topic_and_main_contributions": "The paper VISTA and VTKG. VTKG is a new dataset that builds a KG in which the entities could be represented by text and/or images. VISTA is a new multi-modal model that is designed to solve the problem proposed. They proposed a complete experimental setup. The results shows consistency, and they seems to achieve the state of the art.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The worked reminded me a bit to this AKBC paper:\n\nDaniel Oñoro-Rubio, Mathias Niepert, Alberto García-Durán, Roberto Gonzalez-Sanchez, Roberto Javier López-Sastre:\nAnswering Visual-Relational Queries in Web-Extracted Knowledge Graphs. AKBC 2019\n\nFor the Multimodal Knowledge Graph Completion section:\nYe Liu, Hui Li, Alberto García-Durán, Mathias Niepert, Daniel Oñoro-Rubio, David S. Rosenblum:\nMMKG: Multi-modal Knowledge Graphs. ESWC 2019: 459-474\n\nAnother visual KG application in medical domain:\nAnjany Sekuboyina, Daniel Oñoro-Rubio, Jens Kleesiek, Brandon Malone:\nA Relational-Learning Perspective To Multi-Label Chest X-Ray Classification. ISBI 2021: 1618-1622\n\n"}}, {"review_id": "HloO89fOKb", "reviewer": "Reviewer_pVWp", "summary": "", "questions": "1) See Reasons to Reject 1), 2), and 3).\n\n2) How is the ablation study \"w/o $z_{ENT}$ and $z_{REL}$\" conducted? After removing them, what is the output of the encoder? What is the input of the decoder and Is it only $z_{[MASK]}$? Where is the $z_{[MASK]}$ from? What is the input and the model when obtaining $z_{[MASK]}$?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reading our paper. In what follows, we provide our answers to your concerns. Please also consider that we prepared our answers to **seven different reviewers** during the rebuttal period. By incorporating the comments from seven reviewers, including yours, we got a chance to improve our paper. We hope the improvements can be considered in the final assessment of our paper. We believe most of your concerns may be resolved by the clarifications below. In particular, please read our first two responses together since they answer closely related questions.\n\n$\\rule{18cm}{0.05cm}$\n\n> The VTKG experiment setting with triple images may lack practical application. The paper utilizes triple images during training, which are not available during testing. If the triplet image is available in testing, the two entities that appeared in one image directly reveal the ground truth and do not need further inference. If they are not available, I could not think of one application scenario. Moreover, the KGs organize the entities and relations separately and the triple image may be contrary to it. \n\nLet us clarify how we extract and utilize the visual features of relations. For an image representing $\\langle s,r,o \\rangle$, we define a visual feature of relation $r$ as $\\bf{r}\\_\\text{vis}=[\\bf{s}\\;\\bf{o}\\;\\bf{u}]$ (Lines 270-281). For each relation, we select the top $\\hat{k}$ images in the training set based on the pairwise dot product similarities of the $\\bf{r}\\_\\text{vis}$ values (Lines 283-285). When we select the $\\hat{k}$ visual features of a relation $r$, i.e., $\\bf{r}\\_{\\text{vis},1}, \\cdots, \\bf{r}\\_{\\text{vis},\\hat{k}}$, we consider all training triplet images containing the relation $r$ regardless of their head or tail entities. Since a learnable projection matrix $\\widehat{\\bf W}\\_\\text{vis}$ is applied to each $\\bf{r}\\_\\text{vis}$ vector to construct an initial visual representation vector of $r$, the $\\bf{r}\\_\\text{vis}$ vectors do not need to share $s$ or $o$. Once the initial visual representation vectors of relations (e.g., $\\widehat{\\bf W}\\_\\text{vis} \\bf{r}\\_{\\text{vis},1}$) are fed into the relation encoding transformer (Figure 3), the final representations of relations are the ones that reflect the visual features extracted from training triplet images. **Note that we never use images of test triplets, as we already explained in Lines 412-423.** Using the final representations of entities and relations, we predict a missing entity in a triplet at testing time. While VISTA is different from existing methods in that it can incorporate the visual and textual features of relations when learning the representations of relations during training, our prediction task itself is the same as a general multimodal knowledge graph completion problem and does not need a separate application scenario.\n\n$\\rule{18cm}{0.05cm}$\n\n> How VISTA work in the test phase is ambiguous. In the test phase, how is the relation embedding and $r_{vis}$ calculated? Is the input of the relation encoder only $r_{txt}$? If the $r_{vis}$ is calculated by concatenating head and tail images, there may be information leakage. I think the decline of \"replace $r_{vis}$ w/ $u$\" may confirm the problem.\n\nThe representation vector of a relation is computed by using the textual feature of $r$ (Lines 286-295) and at most $\\hat{k}$ different $\\bf{r}\\_\\text{vis}$ vectors extracted from the training set. **We do not use the images from the test set, so there is no information leakage.** Note that there is no overlap between the training and testing triplets. Regarding how we compute and utilize the $\\bf{r}\\_\\text{vis}$ vectors, please refer to our response to the first comment described above. We extract the visual features of relations from images of training triplets, and the final representation vectors of the relations reflect the training visual features. We will further clarify this point in a revised paper. The experiment of “replace $\\bf{r}\\_\\text{vis}$ w/ $\\bf{u}$” is the ablation study about our way of constructing the visual features of relations and has nothing to do with information leakage.\n\n$\\rule{18cm}{0.05cm}$\n\n> How VISTA work in MKGs without triple images is not clear. How are $r_{vis}$ and $u$ calculated in the FB15K-237 and WN18RR++ datasets?\n\nSince FB15K237 and WN18RR++ do not have images for triplets (Lines 393-394), ${\\bf r}\\_\\text{vis}$ and $\\bf{u}$ are not calculated. For relations without visual features, the representations are learned based on $\\bf{z}\\_\\text{[REL]}$, $\\bf{r}$, and $\\bf{r}\\_\\text{txt}$ in the relation encoding transformer. For entities without visual features, the representations are learned based on $\\bf{z\\_\\text{[ENT]}}$, $\\bf{h}$, and $\\bf{h}\\_\\text{txt}$ in the entity encoding transformer.\n\n$\\rule{18cm}{0.05cm}$\n\n> How is the ablation study \"w/o $z_{ENT}$ and $z_{REL}$\" conducted? After removing them, what is the output of the encoder? What is the input of the decoder and Is it only $z_{[MASK]}$? Where is the $z_{[MASK]}$ from? What is the input and the model when obtaining $z_{[MASK]}$?\n\nIn the ablation study \"w/o $\\bf{z}\\_\\text{ENT}$ and $\\bf{z}\\_\\text{REL}$\", we use $\\bf{h}^{(L)}$ and $\\bf{r}^{(\\hat{L})}$ for the outputs of the encoding transformers. To solve a problem $\\langle h,r,? \\rangle$, we use the representation of $h$ returned from the entity encoding transformer, the representation of $r$ returned from the relation encoding transformer, and $\\bf{z}\\_\\text{[MASK]}$ (Lines 353-370); these are the inputs of the triplet decoding transformer. We employ a standard masking scheme in a transformer, and $\\bf{z}\\_\\text{[MASK]}$ is a learned vector during training representing a missing entity in a triplet (Line 355).\n\n$\\rule{18cm}{0.05cm}$\n\n> Writing needs improvement. Some details of the method and the experiment setting are not clear. The notations in Section 4 are messy. It would be easier to read to simplify some notations.\n\nWe will refine and revise the writing by incorporating all the comments from seven reviewers. We will elaborate more on how we construct $\\bf{r}\\_\\text{vis}$ and the details about the ablation studies and the problem setting. Also, we will add more details about how VISTA works when images are not available. For notations, even though the notations in Section 4 may seem messy, we believe every notation is needed to describe our model precisely. If you could let us know a specific notation that is redundant, we will simplify it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The association of triple images in MKGs is novel. \n\n2) The dataset construction is solid.\n\n3) The experiments on VTKG datasets and MKG datasets outperform the state-of-the-art baselines.", "reasons_to_reject": "1) The VTKG experiment setting with triple images may lack practical application. The paper utilizes triple images during training, which are not available during testing. If the triplet image is available in testing, the two entities that appeared in one image directly reveal the ground truth and do not need further inference. If they are not available, I could not think of one application scenario. Moreover, the KGs organize the entities and relations separately and the triple image may be contrary to it. \n\n2) How VISTA work in the test phase is ambiguous. In the test phase, how is the relation embedding and $r_{vis}$ calculated? Is the input of the relation encoder only $r_{txt}$? If the $r_{vis}$ is calculated by concatenating head and tail images, there may be information leakage. I think the decline of \"replace $r_{vis}$ w/ $u$\" may confirm the problem.\n\n3) How VISTA work in MKGs without triple images is not clear. How are $r_{vis}$ and $u$ calculated in the FB15K-237 and WN18RR++ datasets? ", "questions_to_authors": "1) See Reasons to Reject 1), 2), and 3).\n\n2) How is the ablation study \"w/o $z_{ENT}$ and $z_{REL}$\" conducted? After removing them, what is the output of the encoder? What is the input of the decoder and Is it only $z_{[MASK]}$? Where is the $z_{[MASK]}$ from? What is the input and the model when obtaining $z_{[MASK]}$?", "paper_topic_and_main_contributions": "The paper proposes visual-textual knowledge graphs (VTKG) in which the entity, relation, and triple all associate textual descriptions and images, while the existing multimodal knowledge graph (MKGs) methods do not consider the triple images. The paper proposes new datasets for VTKG named VTKG-I and VTKG-C. Moreover, the paper proposes a transformer-based VTKG representation learning method VISTA. The experiments on both VTKG datasets and MKG datasets demonstrate the effectiveness of the method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Writing needs improvement. Some details of the method and the experiment setting are not clear. The notations in Section 4 are messy. It would be easier to read to simplify some notations."}, "extra_text": {}}, {"review_id": "81Ddqb22ob", "reviewer": "Reviewer_V1gZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your time and effort in reading our paper. In what follows, we provide our answers to your concerns. Please also consider that we prepared our answers to **seven different reviewers** during the rebuttal period. By incorporating the comments from seven reviewers, including yours, we got a chance to improve our paper. We hope the improvements can be considered in the final assessment of our paper.\n\n$\\rule{18cm}{0.05cm}$\n\n> The experiments about impact of different modal information for different baseline models are required to better understand the benefits of multimodal KG.\n\nAmong the baseline methods, OTKGE[1], MoSE[2], and IMF[3] presented the impact of different modal information in their original papers, showing that utilizing different modalities is beneficial to improving the performance of knowledge graph completion. To analyze the impact of each modality in VISTA, we added two tables below. In Table A1, we show the results of VISTA using different combinations of the modalities. In VISTA, we consider $\\bf{h}$ in the entity encoding transformer and $\\bf{r}$ in the relation encoding transformer as the structural representations of an entity and a relation, respectively (Figure 3). On both VTKG-I and VTKG-C, we see that adding visual or textual features leads to better performance than the case without those features. We also note that the impact of the visual features is more prominent on VTKG-I than on VTKG-C. To understand this point more thoroughly, we also analyze the statistics about categorizing triplets based on whether their entities and relations have visual features, shown in Table A2. While all entities and relations have their visual features on VTKG-I, many entities and relations do not have visual features on VTKG-C. This explains why visual features are more critical on VTKG-I. More importantly, **Table A1 shows using all the modalities (structural+textual+visual) leads to the best performance of VISTA**.\n\n### **Table A1: Performance of VISTA using different combinations of modalities**\n\n$\\begin{array}{|c|ccccc|ccccc|}\n\\hline\n&&&\\hspace{21mu}\\textclap{VTKG-I}&&&&&\\hspace{6mu}\\textclap{VTKG-C}&& \\\\\\\\\n&\\text{MR} & \\text{MRR} & \\text{Hit@1} & \\text{Hit@3} & \\text{Hit@10} & \\text{MR} & \\text{MRR} & \\text{Hit@1} &  \\text{Hit@3} & \\text{Hit@10} \\\\\\\\ \n\\hline\n\\text{structural} & 23.3 & 0.3979 & 0.3092 &0.4313 & 0.5725 & 3163.9 &0.3651 & 0.3252 & 0.3806 & 0.4406 \\\\\\\\ \n\\text{structural + textual} & 21.0& 0.4423& 0.3473& 0.4542& 0.6450 & 238.7& 0.4556& 0.3827& 0.4811& 0.6014 \\\\\\\\ \n\\text{structural + visual} & 19.5& 0.4525& 0.3626& 0.4733& 0.6336  & 2641.1& 0.3743& 0.3331& 0.3893& 0.4523 \\\\\\\\\n\\hline\n\\text{structural + textual + visual} & \\bf{17.3} & \\bf{0.4650} & \\bf{0.3626} & \\bf{0.5076} & \\bf{0.6641} & \\bf{220.8} & \\bf{0.4675} & \\bf{0.3918} & \\bf{0.4961} & \\bf{0.6157} \\\\\\\\\n\\hline\n\\end{array}$\n\n### **Table A2: Categorizing triplets based on whether their entities and relations have visual features**\n\n$\\begin{array}{rcc}\n\\hline\n & \\text{VTKG-I} & \\text{VTKG-C} \\\\\\\\\n\\hline\n\\text{two entities w/ visual features \\\\& relation w/ visual features} & 1,316\\ (100\\\\%) & 2,569\\ (2.3\\\\%) \\\\\\\\\n\\text{one entity w/ visual features \\\\& relation w/ visual features} & 0\\ (0\\\\%) & 1,743\\ (1.6\\\\%) \\\\\\\\\n\\text{two entities w/o visual features \\\\& relation w/ visual features} & 0\\ (0\\\\%) & 522\\ (0.5\\\\%) \\\\\\\\\n\\hline\n\\text{two entities w/ visual features \\\\& relation w/o visual features} & 0\\ (0\\\\%) & 10,166\\ (9.1\\\\%) \\\\\\\\\n\\text{one entity w/ visual features \\\\& relation w/o visual features} & 0\\ (0\\\\%) & 21,106\\ (18.9\\\\%) \\\\\\\\\n\\text{two entities w/o visual features \\\\& relation w/o visual features} & 0\\ (0\\\\%) & 75,385\\ (67.6\\\\%) \\\\\\\\\n\\hline\n\\text{Total} & 1,316 & 111,491 \\\\\\\\\n\\hline\n\\end{array}$\n\n**References:**\\\n[1] Z. Cao et al., OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport, NeurIPS 2022\\\n[2] Y. Zhao et al., MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion, EMNLP 2022\\\n[3] X. Li et al., IMF: Interactive Multimodal Fusion Model for Link Prediction, TheWebConf 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " 1. New benchmark datasets for multimodal knowledge graph, which is useful for studies on multimodal knowledge graph.\n 2. A visual-textual (VISTA) method is proposed and extensive experiments are conducted on the new datasets.", "reasons_to_reject": "The experiments about impact of different modal information for different baseline models are required to better understand the benefits of multimodal KG.", "questions_to_authors": "", "paper_topic_and_main_contributions": " This paper constructs new benchmark datasets for multimodal knowledge graph, which contains visually expressible commonsense knowledge, the images of entities and triples and detailed descriptions of entities and relations. And a visual-textual (VISTA) method is proposed to for knowledge graph representation learning.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a visual-textual knowledge graphs (VTKGs) benchmark datasets, where the triplet itself is explained using images and provides a detailed description of the meaning of entities and relationships. Meanwhile, they propose a knowledge graph representation learning method named VISTA that incorporates the visual and textual representations of entities and relations using entity relation encoding. The visual features are extracted by ViT-Base, and the textual features are extracted by BERT. Experimental results on four real-world datasets demonstrate that VISTA outperforms 10 different SOTA methods. As the reviewers mentioned, the paper writing should be improved and the authors should add the new results mentioned in rebuttal stage to revised version.
Decision: Accept-Findings