Topic and Hyperbolic Transformer to Handle Multi-modal Dependencies
Authors: Noriaki Kawamae
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-modal search
Hyperbolic space
Hyperbolic geometry
Lorentz model
Transformer
Topic models

Abstract: As multi-modal search relies on jointly learning image-text representations and has been investigated in the literature,
our innovation is to develop Chimera, a framework in which to learn their representations and similarities.
Because the core of multi-modal search is learning the modalities in a shared semantic space and measuring their similarities,
search quality depends on which expressive space is utilized in learning.
This motivates us to identify the space that can elucidate their semantic and complex relationships with small information loss.
Novelty is assured by introducing the topic and hyperbolic as spaces,
and performing contrastive/metric learning tasks to ensure the cooperation of these spaces with Transformer.
Experiments show that Chimera empowers pre-trained models for multi-modal search tasks and demonstrate the ability of the layers it introduces.

Under review as a conference paper at ICLR 2023
TOPIC AND HYPERBOLIC TRANSFORMER TO HAN- DLE MULTI-MODAL DEPENDENCIES
A Anonymous authors Paper under double-blind review
A ABSTRACT
As multi-modal search relies on jointly learning image-text representations and has been investigated in the literature, our innovation to develop Chimera, a framework in which to learn their representations and similari- ties. Because the core of multi-modal search is learning the modalities in a shared semantic space and measuring their similarities, search quality de- pends on which expressive space is utilized in lcarning. This motivates us identify the space that clucidate their semantic and complex rela- tionships with small information loss. Novelty is assured by introducing the topic and hyperbolic as spaces, and performing contrastive/ /metric learning tasks to ensure the cooperation of these spaces with Transformer. Exper- iments show that Chimera empowers pre-trained models for multi-modal search tasks and demonstrate the ability of the layers it introduces.
1 INTRODUCTION
While conventional search systems find relevant documents to a query, multi (cross) -modal search systems retrieve relevant instances to answer the query. The text-based image trieval models return images matched to a given text, where images are ranked based on their similarity
the text. Inferring the semantic alignment between images and words in the associated sentences allows the fine-grained interplay between vision and language data to be captured, and makes image-text matching more interpretable. As this search model needs semantie-nnernneditte expressions for both images and text, our challenge is to identify spaces that can learn their appropriate representations and define similarities.
Since joint image-text embedding is the bedrock for most Visionaand-Language (V+L) tasks, many studics have explored how to learn their embeddings, project them into a semantic space, and measure their similarity. Many existing multi-modal approaches employ either embedding Lee el al. (2018) or classification Huang al. (2017). As evidenced by the embedding approach, the semantic space is generally of low dimension allowing representa- tion by vectors and the direct computation of modalities by conventional distance metrics (e.g., cosine similarity Euclidean distance) space. The literature describes the use of a variety of matching functions, such as metric learning Hsieh al. (2017); TTay et al. (2018b) and/or neural networks He et al. (2017). Although the studies are primarily focused on designing distance functions over objects (i.e., between users and items), and have empirically demonstrated reasonable success in collaborative ranking with implicit feedback, matching functions work only in the scope of Euclidean space Tran et al. (2020). Since there hypernym, and an entailment rela- lionship within words, and texts Vendrov el al. (2016), one is able to perceive semantic hierarchical relationship between sentences and images.
the similarity function in this embedding
This intuition motivates us to explore spaces that explicitly illuminate both semantic and hierarchical relationships and project them for representation learning. That is, our frame- work, Chimera, focuses topics and hyperbolic geometry as the semantic space and the roomier space, respectively; the contrastive and metric learning approach are used as train- ing objectives. As topic models describe the statistical relationships of word occurrences as global information (e.g., over a given data), Chimera adopts their concept to complement Transformer with global information, and mitigate the strong conditional dependencies be-

---

Under review as a conference paper at ICLR 2023
tween inputs and modalities. As recent studies Hui & Ku (2022 Yang et al. (2022) show that hyperbolic space is roomier than Euclidean space, and can better represent complex relationships, Chimera employs both this space and its operation to realize simpler represen- tations rather than looking for good image/text matches in Fuclidean space. Unlike other models, Chimera is framework in which Transformer-basee models are directed to jointly learn representations in different rooms (i.e., topics and hyperbolic space). Chimera applies metric learning in both Euclidean and hyperbolic space utilize the benefits of both these spaces, and provides higher similarity scores to the positive pairs than the negative ones.
The key advantages of our approach are two-fold:
Theoretical contribution: Chimera (1) derives a way to incorporate both topic and hy- perbolic space with Fuclidean space to learn more semantic and complex relationships over modalities, in end-to-end fashion. (2) trains objectives in the multi-task learning, and adopts the simplicity and effectiveness of the metric learning paradigm to combine Fuclidean space based pre-trained models and hyperbolic space model in the fine-tuning stage.
Practical contributions: Chimera is a plug and play framework, that can utilize the benefits of pre-trained models by fusing Euclidean/ cyyperbolicttopp spaces through trastive/ment metric learning paradigm in fine-tuning, as shown in 4.3 and 4.4.
RELATED WORK
Cross-modal or multi-modal retrieval Balaneshin- korddan & Kotov (0188); Carvalho et al.
(2018;; Wang et al. (2019) finds a set of objects response to textual queries, or assigns understandable terms to objects; multimodality inputs are processed simultancously for joint
vision + language (V+L) understanding. Recent. pre-trained language models Devlin et al.
(2019): Radford et al. (2019); Yang et al. (2019) Liu et al. (2019b); Lan et al. [2020 use Transformer Vaswani et al. (2017) for learning contextualized text representations; they have vielded great advances in NLP tasks, and have been applied to multi-moaal tasks (Singh et al.
(2022); Hu & Singh (2021) ViLBERT Lu et al. (2019) and LXMERT Tan & Bansal (2019)
introduced the two-stream architecture, where separate Transformers are applied to images and text independently; the results of which are fused by a third Transformer in the second stage. Unicoder- VL Li et al. (2020a) takes the visual regions of the image and textual tokens of the sentence as the input. and then encodes the input to yield the linguistic embedding and image embedding, respectively. As with pre-training, VL-BERT Su et al. (2020) performs pre-training on both disual-linguisti and text-only datasets jointly, but does not address the task of Sentence-Image Relationship Prediction unlike other works Lu et al. (2019). UNITER Chen et al. (2020b) uses conditional masking on Masked Language Modeling and Maskod Region Modcling, and introduces a novel Word-region Alignment pre-training task. It can realize heterogeneous downstream V+L tasks with joint multi-modal embeddings. Zhou et al. Zhou el al. (2020) present a unified Vision-Language Pre-training model that also allows generation via unified Transformer with various self-attention masks Dong et al.) (2019). Murahari et al. Murahari et al. (2020) proposes HisDial-BERY that employs pre- trained models for visual dialog. VD-BERT Muraharii et al. (2020) relies the self-attention mechanism within a single-stream Transformer encoder to capture the complex interactions in a unified manner. Oscar Li et al. (2020b) is motivated by the observation that the salient objects in an image can be detected, and are often mentioned in the accompanying text, and uses object tags as anchor points to align the image and language modalities in shared semantic space. VinVL /Zhang et al. (2021) feeds the visual features generated by the new object detection model, and utilizes an improved version of OSCAR+ to pre-train the V+L model and fine-tune it for wide range of downstreem VL. lasks.
Hyperbolic representation learning has recently demonstrated great potential across di- verse range of applications such as learning entity hicrarchies Nickel & Kiela (2017) and/or natural language processing Tay et al. (2018a), hyperbolic graph neural network Chami et al. 2019 Liu et al. (2019a). Hyperbolic geometry has already been applied to the recommender systems domair Tran et al. (2020); Wang et al. (2022a) or image recognition applications Liu et al. (2020): Khrulkov et al. (2020). Hyperbolic space is a kind of manifold space studied in Riemannian geometry, in which basic mathematical operations (e.g., dis- lance measurement) are defined differently from Fuclidean space. Nickel and Kiela Nickel &
2

---

Under review as a conference paper at ICLR 2023
propose method that exploits the individual strengths of both these models by building embeddings using the Lorentz model and mapping them into the Poincaré ball. Hyperbolic Visual Embedding Learning Networks Chen et al. (2020a) learn the hierarchy- aware image embedding features in hyperbolic space, where image labels are projected into hyperbolic space with the Poincaré hierarchy embedding model Nickel & Kiela (2018) and Poincaré Globe Tifrea et al. (2019). Law et.al Law et al. (2019) explain why the squared Lorentzian distance is a better choice than the Poincaré metric.
Kiela 2018
Despite the advances made in hyperbolic embeddings, how to use hyperbolic embeddings for downstream tasks such as classification is challenge due to the absence of corresponding hyperbolic neural network layers Hui & Ku (2022). This motivates us to explore a space that identifies more semantic and complex relationships with small information loss, and introduce the topic and hyperbolic as alternative spaces. As with training tasks, Chimera
can jointly fine-tune models with other training frameworks Radford et al. (2021): Wang
let al. (2022b); Huang et al. (2022]; Zeng et al. [0022;; Zhong et al. 2022
Ln Obajciou lasis
Euclidtan space
Trnsformer Layers
H. MUJM/eatt image)
atantteriion topic spacs
Natching
RTT T(teat)
Trimage)
HLM
Cy (
Toslc Layer (060))
Hyanbbrr Layer
Ippn Lanir
Litet)))
H. (isst-)maae)
R. (.iimgg)))
Hypernaloid space
Traraforing Loyar
d_(k)
Transtomme
typerboli"
weight
Eviadann
Transformer Layar
Klenn mochal
Mispgint
[cls] aamn beiing formard [MASK] you 1.7. [60S] [MMo] padaa). [600)
Image
Input sequence
Figure 1: Architecture of (left) Chimera, and (right) various spaces and models adopted in Chimera for the attention layer: Chimera consists of a text/image topic layer, a hyperbolic layer, matching-layed and objective function (e-g., (0), LD, TLM, and HT.M) which can be injected into pre-trained Transformer-baser models without modifying their architecture. Upon receiving the output of top Transformer layer, HA, the hyperbolic layer feeds HA to the matching layer. The text/image topic layer receives H2 (text)/ HH, (image) and feeds Hr (tert) (tert)/HL Hr (imagee to the matching layer, where (tezt), and H2 (image) denotes the Hy associated with text, and image, respectively. That is, Chimera decomposes H2 E Rlaxdds into HA, HT(let)), and Himaage)) and integrates them into Has E Rlaxxd.
CHIMERA
3.1
MODEL OVERVIEW
The architecture of Chimera is illustrated in Figure 1 Chimera exists between the top of Transformer layers and model specific objectives or output functions. receives the hidden states of last Transformer layer as its input, transforms them via the topic and Hyperbolic space with objectives, and feeds its output into the model-specific objectives or functions.
As Chimera based on the understanding that hyperbolic space can better represent complex relationships than Euclidean space Nickel & Kiela (2018); Zhang & Gao (2020); Gülçehre et al. (2019), represents sets of images and texts in both topic and a hyperbolic space, to illuminate their semantic and hierarchical relationships. This direction leads us to adapt Transformer-basee
models for performing an attentive read operation over different representation vectors in both topic and hyperbolic space, while maintaining the simplic- ity and effectiveness of the metric learning paradigm. Tn practice, this framework enables Chimera to connect topics, and Euclidean and hyperbolic space, while retaining the benefits of pre- tre-trained models and alleviating catastrophic forgetting Ramasesh et al. 2021
3.2 TRANSFORMER LAYERS
Transformer has layers, each of which contains two blocks. While the core of the first block is multi-head attention with k-heads, the core of the sccond block is feedforward network with ReLU activation Nair & Hinton (2010); it embeds inputs with |o] length, X e gla1, to the hidden representation, Hp E R xd,, and projects it using inner dimension f, with layer normalization Ba et al. (0066)
3

---

Under review as a conference paper at ICLR 2023
H. EM)(X), H LayerNorm(HH..) H, MultiHead (H Hi-1, Multi K, [ W H. Layer.Norm(HH)
H FF,((H)) Hi, FFN(HH) mar(0, HUV,, h Attention(Q,, Kj, Vi), where U E Rdxx,, V E R/xdi, and W. E R xdx are Icarnable weights, Qji Ky V E Blalxde are obtained by transforming the output of the 1-th layer, Hi, us- ing W w w/'s E pdd xdx, respectively.
3.3 TOPIC LAYER
The motivation for introducing the topic layer is to model the uncertainty in the genera- live process, and strike a balance between visual information in the image and linguistic knowledge acquired from the text. Given input sequence X f Id]j= and dataset D (x xD), ) non-autorogressive gencration can be achieved by minimizing the following independence assumption in the decoding process:
IDI lel
mångC(0) ming ΣΣ
log (o(a,a(XxA)))
(2)
where 0, and Xdxx represents model parameters, and Xa without Id, respectively.
To mitigale the gap between inputs and modalities, our framework introduces topics, 2, into this process and so modifies Eq
( to:
|
(i)
ming ΣΣ log Σ
===1
where 2; denotes the topic of the f-th token, and Z is the number of topics. That is, Po (xax)))) the prior distribution over latent topic 2, and Ps (a ze) is the "genera- tive" distribution over words, V. This formulation ensures that sequence x can be generated by random process involving latent topic variable z: (1) ze first generated from the con- ditional distribution Po(x)(xarr). (2) Id,ı is finally generated from Po (o (xA.e 21). Note that (a(=((aar) a multinomial distribution over discrete variables, not The Gaussian distribution used in variational autoencoders Kingma & Welling (2014).
By placing this topic layer on both the text and the visual encoder (Transformer layers), mitigate the discrepancy between the cross-modal representations learned by the encoder and the representation needed by the decoder for ienerating text.. As Chimera injects topics on the top blocked attention layers, it maps hidden representation vector h E RR to latent topic vector z E TRZ and then projects this topic vector into the topic-specific distribution over words. This yields Eq (1) by defining topic matrix, W E 1d,x2 and word generation function, Fhh,,), where V the size of the vocabulary. This method is used to sample the next token according to the following probability:
H2 (textt/imaee) = NagerNorm([]] H W =(n,))==22))
(4)
Polaa(xa.1n-1)
Pa(=a,
where Wz is a learnable weight. As with F. F(hr,)) 2), we propose three (ransfor- mations to generate Idd that accords with the given & and xd,1:-11
h
residual if = 0 addition if
(1- (-wwhr. + wgz
F. Wvix
(5)
h 0 gz
multiplication if =
h WRZz + b
affine if 2 > 0,
where Is E dd,, Way nts= Rdd. and b, E Rdd are the learnable weights specific to topic
z. We prepare the residual to select the input if = 0, to select Euclidean space. Note that just as Eq (2) is transformed into Eq (3) through the introduction of topics, Wviz E d used in previous Transformer is decomposed into the product of Wz and F(hL,)) in Eq
4

---

Under review as a conference paper at ICLR 2023
3.4 HYPERBOLIC LAYER: LORENTZ MODEL AND ATTENTION MECHANISM
Formally, n-dimensional hyperbolic space is a manifold of constant negative curvature. As the number of objects grows exponentially with semantic distance from the query, hyperbolic geomelry can, unlike Euclidean geomelry, encode those objects without creating interfer- ence Gülgehre et al. (2019). Among the five most. common models for hyperbolic space, we primarily make use of the Lorentz model because its distance function avoids numerical instabilities, unlike the Poincaré equivalent. Note that one-to-one isometric transformations can be defined between cach different model of hyperbolic space. The Lorentz model is the only unbounded hyperbolic model and is defined as ~ (IH" ,c) with points con- strained by H = {x R=+1 (x, x)x = -1, -1,20 > 0}, where the Riemannian metric tensor (c(x) = diag(-1, I,., 1), and (x, y) is the Lorentzian scalar product. The associaled distance function de is then given
dz(x,y) = (xxoss(((xxxx)a)) (x,y) - To30
(6)
As shown in Eq 1. h corresponds to scaled dot-product attention, and is written as R softmar( OK* VV, where Q K, V denotes the queries, keys, and values, respectively; vi
d, is the shared dimensionality of the queries and keys. The attentive read operation over keys, k, by query q has the following form:
r= = Σ, aij 7 Wijs Oi = exp( Va qqi kj)), Uij = Vij = E,0s...
where qi is a vector called the query and k,s are the keys for the memory locations being read from, ogg is the function that compules scalar matching score between qe and k,, Vsy is value to be read from j-th location by i-th query, and ((> 0) is a normalization factor for the full sum.
As the /onggl/in features are extracted as vectors in the Euclidean space, they are projected into the hyperbolic space align with the word representation in this phase. The most natural way to exploit hyperbolic geometry for matching pairs of points is to use the hyperbolic distance belween them Gülçehre el al. (2019). To gain the weighted value in the Euclidean Transformer, to compute the aggregation weights appropriately [Unarr] (2005) and Eq (6) Thus, the
project Vi.j on Klein model and adopl the Einstein midpoint
hyperbolic attention weights, aij, and each entry, Ti,6, of R that result from an attentive read can be expressed as:
Ci = Oi (( "ia,,,,)))) loy, as = f(- f(-Bdc(qn. kj) c), (W:n))
1
(8)
V1 12
where 7(-) are the Lorentz factors, and f(-) denotes the sigmoid function; B and e are manually set parameters and learned along with the rest of the network. The above yields H
3.5 MATCHING LAYER
This layer is to integrate the output of the topic layers and the Hyperbolic layer into the hidden representation, HM, which can be fed to objective tasks of previous Transformer based models. Thus, the output of this layer, Hi, can be expressed as:
HM (1 w/ H Hy (image) + (1 w., HT (text))
where w, and We is the learnable weights of topic and visual, respectively. That is, both the topic layers and the hyperbolic layer can adopted by and work seamlessly within existing Transformer based models.
3.6 TRAINING TASKS
Topic Level Matching (TLM): Inspired by contrastive learning Khosla et al. (2000
Radford et al. (022)), Topic Level Matching (TLM) aims to bring similar instances closer
5

---

Under review as a conference paper at ICLR 2023
and push away dissimilar instances further from each other. Assuming that similar instances in the Euclid space are considered similar at the topic level, topics are learned to explain the latent semantic similarity; accordingly, we apply TLM to the topic layer. We use t, and i to refer to text and image on the topic layer, respectively.
As cach image has multiple sentences and these sentences refer to the same image, they are considered to be semantically similar. Since this task aims to utilize this semantic similarity for representation learning, it is designed to predict whether one linguistic sentence can be semantically matched with other sentences attached to the same image. Like Image- Text Matching (ITM) in other models Chen et al. (2020b), this model samples both sentences (positive) with the same image and those with the other images (negative) and learns their matching scores, where it creates the negative pair by replacing the sentence in paired sample with randomly-selectee equivalent from the other sample.
exp exp (l,
(C) = - - log exp (l, En exp (o(
(10)
where (t) denotes the inner product of t and i, N the size of negative samples, and (in) are N negative samples.
Hyperbolic Level Matching (HLM): To realize fine-tuning for image-text and text- image retrieval, we formulate the task as ranking problem. The fine-tuning inputs share the same data preprocessing procedures used in pre-training, except that we do not mask words or regions in the fine-tuning stage. Following the V+L task, we denote the score function as 8. Here, we are motivated by Visual Linguistic Matching (VLM) in Li et al. (2020a), and ITM. Both define the similarity between two inputs with the ranking loss constraint widely used in embedding- baaeedddng-aedddd methods to construct a bi-directional max-margin ranking loss instead if treating the "match" and "mismatch"
binary classification problem. Similar to Hsich et al. (2017). for a given query, the nearest neighborhood images are: 1) the images previously referred to by this query, and 2) the images referred to by other words that have similar semantics to this word. That is, we are also able to indirectly observe the relationships between word-word pairs and image-image pairs through the pull-push mechanism of metric learning in Euclidean space.
Using the exponential map allows us to project image features v; into the hyperbolic space. Here, we transform this feature vector into its polarity form as (d, r) E R=+1 where r = || and d = v,, ie. ||d|| = 1, and denote this transformed feature as dy. Similarly, we get the transformed feature, Wi from the word embedding w/. Thus, we can project these different features in the same Lorentz space. As the model trained in Lorentz hyperbolic space, we use the distance, de, defined in Eq G and define our pull-and-puse loss function as:
CHLM Σ
Σ
[m + de 03) (2 0y)]+
(11)
(1.)ED (K.D)ED
where D contains all observed implicit foedback, i.c. positive image- word pairs; [x]+ =)) is the standard hinge loss and m >> 0 is the safely margin size.
Because the goal when embedding one space into another is to preserve distances while maintaining complex structures/ wawauurrs/eelaionsiiii rww.erroooooossssssseessssssss Sala et al. (2018), HyperMI, et al. (2020) explores metric learning in hyperbolic space for recommender systems. defines the dis- tortion optimization function while embedding user-item pairs into hyperbolic space with the constraint of preserving good structure quality for metric learning. As our goal is to coordinate the Euclid space with the Hyperbolic space, our approach adapts this distortion optimization function as follows:
|dc(wi, (g)) de(ws, 07)1
Ide Idd((Ek, 17)) dz 27y)
LD= =
Σ (.,)ED
dg(w,, &z)) 1. + Σ
(12)
dg(wk. v2))
(K.D)ED
where defines the absolute value, and di is the embedding representation in Euclid space. Thus, fine-tuning aims to preserve the distances by minimizing Cp, as this function associates lower distortion with better preservation.

---

Under review as a conference paper at ICLR 2023
Table 1: Basic statistics of the datasets used in fine-tuning
Datasets
# Training
# Walidation
#est
Flickr30
30K
IK
IK
MSCOCO
83K
5K
5K
3.7 FINE- TUNING
Chimera can adopt pre-trained Transformer-base models and can be fine-tuned for their downstream multi-modal tasks, because Topic/Hyyperbolcc/ Htttyyerroob////ahhiiinng Matching Layer and TLM/HLM/Lo can be casily plugged into these models without changing their structure. By performing the pre-training over suitable visual-linguiste corpus, Chimera can be cus- tomized and applied to visual-linguistic tasks, while keeping the size of the input and output representations in line with those of the pre-trained Transformer-basel models.
While jointly training tasks, (r), LTLM, LHLM and LD contribute to boost the model performance while providing good representations: there is, however, an unavoidable trade- off between these functions Sala et al.] (2018); Tran et al. (2020). Accordingly, we integrate these tasks, the model-specific objective task sct, Lo, in an end-to-end fashion a multi- task learning framework. The objective function is defined as:
min Le = XTCr(0) + ATLM + CHLMCHIM + ADDD +
doLo
(13)
Chimera
pre twe-trraiedmodels
where θ the total parameter space that covers all embeddings and variables of the net- works, and X, is the weight of cach task.
EXPERIMENTS
4.1 DATASETS
We use two datasets, MSCOCO Lin et al. (014] and Flickr30 D
widely used dataset for al. (2020a), as follows. We follow Karpathy
multi-modal search Wang et al. (2019): y Li et
& Li (2015) Faghri et al. 20018) to split these datasets into training/ training/validation /vt/idaaiioon////////a /test subsets.
Their statistics are shown in Table
4.2 IMPLEMENTATION DETAIL
We implemented Chimera using Pytorch 2 with Nvidia A Ape and Horovod speed up training. We will release this code later. As shown in Figure Chimera can be inserted as additional layers with original objective tasks, Lo Eq [3 into various pre-trained Trans- former models without changing their architecture. Chimera works with the Transformer layers and their objective tasks. To get the best out of the combination of the pro-trained models and Chimera rather than comparing fairly over various models, follow the settings of each pre-trained model.
Experimental design:
As with Transformer-baseer models (UNITER, VL-BERT, VD- BERT, ViT)), we modify the implemented model and optimize their parameters. With regard to training Transformer in our framework,
Adaptive Moment Estimation (Adam) Kingma & Ba (2015) with 31 = 0.9 and 32 0.999 used for optimization, over mini-batches update parameters and adopt the dropout strategy SSivastava al. (2014) to optimize networks. Then we follow the fine-tuning of [Radford et al. (2021); Jia et al. (2021) and set number Eq (10) the batch size. After performing hyperparameter
httpe://pytorchh.og
httppe///gthhhl..cc//ooooo///oos
https:

---

Under review as a conference paper at ICLR 2023
Table 2: EX1: The contribution of Chimera in image-text matching tasks over Flickr30 and MSCOCO datasets using RoN: The value is the improvement (+%).
Data
Flickr30
MSCOCO
Method
12T
T2I
12T
T2I
(N =)
10
10
1
5
10
1
5
10
UNITER
1.6
3.1
8.3
2.5
4.8
10.1
1.8
3.6
8.9
2.9
5.4
11.4
VL-BERT
1.4
3.3
8.4
2.3
4.6
9.8
1.6
3.3
9.2
2.7
5.2
11.2
ViLBERT
1.5
3.4
8.2
2.4
4.4
9.6
1.5
3.1
9.1
2.8
5.0
11.1
VD-BERT
1.4
3.2
8.1
2.2
4.3
9.7
1.5
3.2
9.1
2.8
5.1
10.9
VIT
1.5
3.3
8.2
2.2
4.4
9.8
1.5
3.3
9.2
2.7
5.0
11.1
Table 3: EX2: The contribution Chimera in image-text matching tasks over Flickr30
and MSCOCO datasets: Different from the previous experiment shown in Table 2 using RON, we randomly shuffle the order of the words in each input sentence. The value is the
improvement (+%)
Data
Flickr30
MSCOCO
Method
I2T
T2I
I2T
T2I
(N =)
1
5
10
1
5
10
5
10
1
5
10
UNITER
1.9
3.9
8.9
3.2
5.7
12.2
2.1
4.1
9.8
3.5
6.2
12.6
VL-BERT
1.8
3.7
9.1
3.1
5.8
12.1
2.2
4.2
10.1
3.4
6.3
12.4
VILBERT
1.7
3.6
9.0
3.0
5.7
12.2
2.1
4.1
10.0
3.2
6.2
12.4
VD-BERT
1.7
3.7
9.1
3.1
5.5
12.1
2.1
4.2
10.1
3.3
6.3
12.5
VIT
1.7
3.6
9.0
3.0
5.5
12.2
2.1
4.1
10.0
3.2
6.2
12.4
optimization using grid search, we set them as Xv,, ÀTLM AHLM, AD, and A0=0.5, 0.5, 0.5, 0.5, and, 1.0, and apply these values to other models. Like other state-of-the-art pre-trained V-L models, our framework can fine-tune for various downstream visual-linge linguiiie tasks, by simply modifying the inpul format, output prediction, loss function and training strategy.
4.3 EVALUATION METRICS AND RESULTS
For the V-L task, we perform two image-text matching tasks:
12T: sentence retrieval, i.e., retrieving ground truth sentences given query image. T2I: image retrieval, iee, retrieving ground truth images given a query text. Since these tasks based on rankings, candidates are ranked according to their scores, and the number of candidates varies with the query. For both tasks, the commonly used evaluation metric is R@N, which is defined as the recall rate for the top N results for the query.
Table shows the results these tasks, EX1. We confirm that the contribution of Chimera is proportional to the increase in N; this tendency stronger for T2I than I2T.
As the accuracy was virtually the same in the previous experiment, we performed an addi- tional experiment with different conditions. That is, randomly permuted words in each sentence to determine the robustness of the model. As shown in Table 3 Chimera offers significantly better performance than the other models than in experiment EX2. This table indicates that not only the hyperbolic space but also TLM/HLM, which are newly intro- duced as the training tasks our architccture, contributed to this superiority, as shown by the ablation analysis. Since the difference is greater for T2I than for 12T, the latter task considered be more difficult than the former. This result suggests that hyperbolic space is betler suited to the visual-lingustti task. Chimera is more robust against word reordering because Topic and Hyperbolic layers learn representation on each input level.
4.4 ABLATION ANALYSIS
To investigate the respective contributions of pre-training tasks to overall performance, conducted an ablation analysis over the same datasets with the same metric following 2 In order to compare the effect of each component, Chimera used ViT the pre-trained model. To emphasize the difference between the effects of these components, we performed fine-tuning over the data with 20% of each of the images and texts randomly masked, the
8

---

Under review as a conference paper at ICLR 2023
Table 4: Ablation analysis of pre-training of image-text matching tasks over Flickr30: In this table, TL, HI,, Z, and N denote Topic Layer, Hyperbolic Layer, #topics, and # dimensions of Hyperbolic space, respectively. In TLM, ad, mu, and af denote addition, multiplication, and affine in Eq (3), respectively. The best results marked in bold, where the bold value denotes the statistical significance for p < 0.01, compared to the best baseline. TLM and HLM/Lp need the topic layer and the Hyperbolic layer, respectively.
Data Tasks
Flickr30
MSCOCO
12T R@1,5,10
T2I R@1,5,10
12T
T2T
TL
HL
LD
R
R@1,5,10
Z
TLM
N
HLM
20
af
20
W
W
67.8,91.2,97.1
58.1,77.1.89.9
66.3.91.4.9775
58.7,77.6,90.2
20
ad
20
w/o
2,.888
56.9,75.3,86.8
64.2,84.8,92.2
54.8,74.2,8661
20
mu
20
W
W
96.1,89.1,96.2
57.3,75.9.87.9
65.1,86.6,9355
55.6,75.3,87.2
20
af
20
w/o
64.2288.6,90..3
86.5,74.8,86.22
63.7,84.1.91.7
53.9,73.1.85.2
20
w/o
20
67.2,90.1,96.
56.9,76.2.88.8
66.3,90.3,96.6
58.3,76.8,89.4
10
af
20
W
W
65.1,88.2,91.2
57.1,75.7,86.99
64.22,84.892..9
54.2,73366,85.7
0
w/o
20
63.2,84.1,88.3
56.2,75.2.86.1
63.4,83.9,92.1
53.7,73.1,855.
20
al
20
w/o
w/o
67.5,90.3,97.0
57.2,76.8,88.9
66.3,90.8,96.9
38,4,76.9,89.5
20
af
10
67.1,90.2,96.8 66,7,89.8,6.22
56,8,76.8,88888
65.8,88.4,95.7
57.3,76.1,88
20
af
0
W/O
w/o
56.3,76.7,89.3
65.2,87.9,94.2
56.7,75.7,87.8
order of the words in each sentence were randomly shuffled. A comparison of improvements achieved is shown in Table 4 The lower the value, the greater the effectiveness of that excluded task. A comparison of the magnitude of the decrease shows that the contribution of both the Topic layer and the Hyperbolic layer to Chimera higher, and supports our assumption that both topic space and hyperbolic space contribute to providing better rep- resentations through the discovery of the semantic relationships and the summarization of the complex relationships. This table shows that among these lasks, the topic layer is more sensitive to both topic specific tasks (e.g., TLM and HLM) and decrease the number of topics, which corresponds to the number of Hyperbolic dimensions, than the Hyperbolic layer.
DISCUSSION
Our approach uses topic and hyperbolic space and metric learning to focus on the character- isties and differences of multi-modal dependencies. Fach image and its corresponding text differ not only in the number of features but also in the amount of information they contain. The relationship between images and text is asymmetric, and this relationship is observed as task asymmetry in Table 3 EX2. Table 3 implics that the mcaning of sentence in a text does not change much even if a word is replaced, but images lose their meaning regions are shuffled in each image, that is the degree of freedom is low. As the V+L task requires a representation space into which both images and lext can be embedded without any other bias, our architecture learns their representations and projects them in topie and hyperbolic space. To jointly learn image-text representations more effectively, Chimera em- ploys metric learning to coordinate representations in the Euclid space with those in the hyperbolic space, and can move data point certain distance in hyperbolic space with smaller force than is nceded in Euclidcan space.
CONCLUSION
To develop semantic intermediate expressions for both images and texts, our proposed frame- work, Chimera, bridges the gap between Euclidean and hyperbolic geometry through topics and the metric learning approach. This leads our framework to project image-text rep- resentations into topic and hyperbolic space while preserving semantic similarity in the Euclidean space through metric learning. Experiments various data sets showed that Chimera achieved better results on the Aidion-and-Langeee task than the baselines and showed its ability to learn the image-tett dependencies.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalizationg. CoRR, abs/ 0es/1607.063 06440,, 2016.
Saeid Balaneshin-kordan and Alexander Kotov. Deep neural architecture for multi-modal retrieval based on joint embedding space for text and images. In WSDM, pp. 28 36, 2018.
Micael Carvalho, Rémi Cadène, David Picard, and etc. Cross-modal retrieval in the cooking conlext: Learning semantic lext-image embedding. In SIGIR, pp. 35-44-44 45-44, 2018.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolu- tional neural networks. In NeurIPS, 2019.
Jingjing Chen, Liangming Pan, Zhipeng Wei, Xiang Wang, Chong-Was Wah Ngo, and Tat-Seng Chua. Zero-shot ingredient recognition by multi-relational graph convolutional network. In AAAI, pp. 10542-10550, 2020a.
Yen- Chun Chen, Linjie Li, and etc. UNITER: universal image-text representation learning. In ECCV, pp. 104-120, 2020b.
Jacob Devlin, Ming-Weirr Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACLM, pp. 4171-4186, 2019.
Li Dong, Nan Yang. Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Esiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In NewrIPS, pp. 13042-13054, 2019.
Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: improving visual-semantic embeddings with hard negatives. In BMVC, pp. 12, 2018.
Çaglar Gülçehre, Misha Denil, and etc Mateusz Malinowski and. Hyperbolic attention networks. In ICLR, pp.
2019.
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In WWW, pp. 173 182, 2017.
Cheng-Kang Hsieh, Longqi Yang, Yin Cui, Tsung-Y Yi Lin, Serge J. Belongie, and Deborah Estrin. Collaborative metric learning. In WWW, pp. 193 201, 2017.
Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. In ICCV, pp. 1419 1429, 2021.
Yan Huang, Wei Wang, and Liang Wang. Instance-aware image and sentence matching with selective multimodal Istm. In CVPR, pp. 7254 7262, 2017.
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu: Seeing out of the box: End-to-end pre-training for vision-language representation learning. In CVPR, pp. 12976 12985, 2022.
Bo Hui and Wei-Shinn Ku. Low-rank nonnegative tensor decomposition in hyperbolic space. In KDD, pp. 646 654, 2022.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language rep- resentation learning with noisy text supervision. In ICML, pp. 4904-4916, 2021.
Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. In CVPR, pp. 3128-3137 3137, 2015.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NewrIPS, 2020.
10

---

Under review as a conference paper at ICLR 2023
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, and etc. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR, 2020.
Mare Teva Law, Renjie Liao, Jake Snell, and Richard S. Zemel. Lorentzian distance learning for hyperbolic representations. In JCML, pp. 3672-3681, 2019.
Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. CoRR, abs/ obs/1803.08024, 2018.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: universal encoder for vision and language by cross-modal pre-training. In AAAI, pp. 11336 11344, 2020a.
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, 2020b.
Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECVV, pp. 740-755, 2014.
Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. In NewrIPS, 2019a.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong- Wah Ngo, Tat-Seng Chua, and Yu- Gang Jiang. Hyperbolic visual embedding learning for zero-shot recognition. In CVPR, pp. 9270 9278, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1190..1192 2019b.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for lision-ndd-angguae tasks. In NeurIPS, pp. 13 23, 2019.
Vishvak Murahari, Dhruv Batra, Devi Parikh, and Abhishek Das. Large-scale pretraining for visual dialog: A simple state-of-the baseline. In ECCV, pp.
2020.
Vinod Nair and GeofTrey E. Hinton. Rectified linear units improve restricted boltznann machines. In ICML, pp. 807-814 2010.
Maximilian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical repre- sentations. In NIPS, pp. 6338 6347, 2017.
Maximilian Nickel and Douwe Kicla. Learning continuous hicrarchics in the lorentz model of hyperbolic geometry. In ICML, pp. 3776 3785, 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Tly Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, and etc. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8748-8763, 2021.
Vinay Venkatesh Ramasesh, Ethan Dyer, and Maithra Raghu. Anatomy of catastrophic forgetting: Hidden representations and task semantics. In ICLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Frederic Sala, Christopher De Sa, Albert Gu, and Christopher Ré. Representation tradcoffs for hyperbolic embeddings. In ICML, pp. 4457-4466, 2018.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, and etc. FLAVA: A foundational language and vision alignment model. In CVPR, pp. 16638-15650, 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhut- dinov. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):1929-1988, 2014.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vi-bert: Pre-training of generic visual-linguisti representations. In ICLR, 2020.
Hao Tan and Mohit Bansal. Lxmert: Learning rrods-modality encoder representations from transformers. In EMNLP-IJCNLLS pp. 5099 6099-5110, 2019.
Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. Hyperbolie representation learning for fast and efficient neural question answering. In WSDM, pp.
2018a.
Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. Latent relational metric learning via memory- based attention for collaborative ranking. In WWW, pp. 729 739, 2018b.
Alexandru Tifrea, Gary Bécigneul, and Eitavian-Eugert Ganea. Poincare glove: Hyperbolic word embeddings. In ICLR, pp. 3776 3785, 2019.
Lucas Vinh Tran, Yi Tay, Shuai Zhang, Gao Cong, and Xiaoli Li. Hyperml: A boosting metric learning approach in hyperbolic space for recommender systems. In WSDM, pp. 609 617, 2020.
Abraham A Ungar. Analytic Hyperbolic Geometry: Mathematical Foundations and Appli- cations. World scientific, 2005.
Ashish Vaswani, Noam Shazeer, Niki Parmar, and etc. Attention all you need. In NIPS, pp. 5998 6008, 2017.
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. In ICLR, 2016.
Hao Wang, Defu Lian, Hanghang Tong, Qi Liu, Zhenya Huang, and Enhong Chen. Hy- persorec: Exploiting hyperbolic user and item representations with multiple aspects for social-aware recommendation. ACM Trans. Inf. Syst., (:22):2::124:25 2022a.
Tan Wang, Xing Xu, and etc. Matching images and text with multi-modal tensor fusion and re-ranking. In MM, pp. 12 20, 2019.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkvv and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In ICLR, 2022b.
Menglin Yang, Zhihao Li, Min Zhou, Jiahong Liu, and Irwin King. Hicf: Hyperbolic infor- mative collaborative filtering. In KDD, pp. 2212-2221, 2022.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Lc. Xlnet: Generalized autorcgressive pretraining for language understanding. In NIPS, pp. 5754-5764, 2019.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Align- ing texts with visual concepts. In ICML, pp. 25994-26009, 2022.
Chengkun Zhang and Junbin Gao. Hype-han: Hyperbolic hierarchical attention network for semantic embedding. In IJCAT, pp. 3990-3996, 2020.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Making visual representations matter in vision-language models. In CVPR, 2021.
12

---

Under review as a conference paper at ICLR 2023
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-based language- image pretraining. In CVPR, pp.
2022.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J. Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and VQA. In AAAI, pp. 13041-13049 2020.
13


Reviews:
[{"review_id": "xPTK06zh6O", "reviewer": "Reviewer_YYrR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author proposes a way to combine topic space, hyperbolic space, and Euclidean space in an end-to-end manner with learning semantic and complex relationships across modalities. ", "summary_of_the_review": "Considering hyperbolic space for multi-modal search is a novel attempt. However, there are some concerns that I mentioned before", "strength_and_weaknesses": "## Strength\n\n- Considering hyperbolic space for multi-modal search is a novel attempt given that there is a hypernym, and an entailment relationship within words, and texts.\n- The performance is also improved compared with baselines.\n\n## Weakness\n\n- The author only compares some small dimensional results. How about the high-dimensional performance comparisons?\n- The author's comparison is limited and lacks sufficient comparison to demonstrate the superiority of his method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "op8FChOVln", "reviewer": "Reviewer_Gm2Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors focus on the multi-modal representation learning and cross-modal similarity estimation topic and propose the Chimera framework. The proposed framework uses the hyperbolic space to better represent complex multi-modal relationships. The topic model is also utilized. The proposed Chimera method is plug and play. Experiments verify its effectiveness when combine with existing multi-modal representation learning framework.", "summary_of_the_review": "This paper proposes a plug-and-play framework, Chimera, for enhancing the multi-modal representation learning and cross-modal similarity estimation. The proposed method can improve the performance of pre-trained modal. However, the novelty of this paper both technical and empirical is limited. More detail is stated in the last part.", "strength_and_weaknesses": "Strength\n+ This paper combines the topic model, hyperbolic space and Euclidean space to better model the complex relationships between modalities.\n+ The proposed Chimera method is plug-and-play. Diverse experiments are provided, validating that it can benefit the existing pre-trained models.\n\nWeaknesses\n- The authors claim that the hyperbolic space can better learn the complex cross-modal association. However, there is lacking strong evidence to support it. In table 4, the results of w HL (hyperbolic space) do not consistently outperform its w/o HL (hyperbolic space) counterpart. In addition, experiments only include the module effectiveness of HL. If using attention module in the Euclidean space replaces HL, could similar results be achieved?\n- In the introduction section, it states that Chimera has theoretical contribution to learn more semantic and complex relationships over modalities. However, it seems that the method section only provides the statement of the proposed framework, and the theoretical support does not well proved. \n- The ablation studies about the hyper-parameters are missing. Does the selection of hyper-parameters affect the model performance a lot?\n- This paper focuses on multi-modal representation learning, but the experiments only contain vision + language dataset.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eTNfWvQRO1Z", "reviewer": "Reviewer_hwZK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a framework for image-text retrieval task, by introducing matching on topic and hyperbolic space. Experiments are conducted on MSCOCO and Flickr30 datasets. Combined with some well-known models, improvements on these two datasets are shown.", "summary_of_the_review": "See \"Strength And Weaknesses\".", "strength_and_weaknesses": "Strength\n1. Improvements are observed.\n\nWeaknesses\n1. Only 2 datasets are used to evaluate the proposed method. More dataset, more settings and more tasks are necessary for illustration the advantages of the proposed method.\n\n2. Fig.1 is not clear. Please use a high quality image.\n\n3. The equations in Eq.(1) are well-known. It is not needed to mention them all.\n\n4. About the novelty. It seems that the proposed method is just a combination of several losses.\n\n5. How to choose lambdas in Eq(13)?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject