Multimodal Language Learning for Object Retrieval in Low Data Regimes in the Face of Missing Modalities
Authors: Kasra Darvish
Edward Raff
Francis Ferraro
Cynthia Matuszek
Venue: Accepted by TMLR ()

Abstract: Our study is motivated by robotics, where when dealing with robots or other physical systems, we often need to balance competing concerns of relying on complex, multimodal data coming from a variety of sensors with a general lack of large representative datasets. Despite the complexity of modern robotic platforms and the need for multimodal interaction, there has been little research on integrating more than two modalities in a low data regime with the real-world constraint that sensors fail due to obstructions or adverse conditions. In this work, we consider a case in which natural language is used as a retrieval query against objects, represented across multiple modalities, in a physical environment. We introduce extended multimodal alignment (EMMA), a method that learns to select the appropriate object while jointly refining modality-specific embeddings through a geometric (distance-based) loss. In contrast to prior work, our approach is able to incorporate an arbitrary number of views (modalities) of a particular piece of data. We demonstrate the efficacy of our model on a grounded language object retrieval scenario. We show that our model outperforms state-of-the-art baselines when little training data is available. Our code is available at https://github.com/kasraprime/EMMA.

Published in Transactions on Machine Learning Research (10/2023)
Multimodal Language Learning for Object Retrieval in Low Data Regimes in the Face of Missing Modalities
Kasra Darvish University of Moryland, Boltimore County
kasrdaruussh@umbbeedd
Edward Raff
Raff Edward&@aah.
University of Moryland, Baltimore County
Booz Allen Hamillon
Francis Ferraro
ferraro@rumbc.cm edu
Universily of Moryland, Boltimore County
Cynthia Matuszek
emat-Gumbee.c edu
University of Moryland, Baltimore County
Reviewed on OpenReview: https: // opwrrevie........ net/ forum? id= cXa6Xdm0v7
Abstract
Our study is motivated by robotics, where when dealing with robots or other physical systems, we often need to balance competing concerns of relying on complex, multimodal data coming from a variety of sensors with general lack of large representative datasets. Despite the complexity of modern robotic platforms and the need for multimodal interaction, there has been little rescarch integrating more than two modalities in low data regime with the real-world constraint that sensors fail due to obstructions or adverse conditions. In this work, we consider case in which natural language is used as a retrieval query against objects, represented across multiple modalitics, in a physical environment. We introduce extended mullimodal aligament (EMMA), a method that learns to select the appropriate object while jointly refining modality-specffic embeddings through geometric (distance- based) loss. In contrast to prior work, our approach able incorporate an arbitrary mumber of views (modalities) of particular piece of data. We demonstrate the efficacy of our model on grounded language object retrieval scenario. We show that our model outperforms state-of-the-art baselines when little training data is available. Our code is available at stiped/aa.//t//////eebb/ooo////r EEMMA
1 Introduction
Humans use multiple modalities to understand and interact with the world around them. Although physical agents such as robots typically have access to multiple sensory inputs, most approaches use at most two senses (e-g., depth images along with standard RGB images) with single labels. Simultancously using additional modalities and labels is an underexplored area. The benefits of integrating multiple modalities are twofold: first, complementary information from different modalities second, additional modalities can help when one more sources of data become unavailable. Hardware can become damaged or defective, sensors can get blocked or obstructed, and various adverse conditions can remove a modality from use. However, designing an approach that integrates an arbilrary number of modalities while being robust to the loss of one or more modalities at test time an important, yet non-trivial. problem. Mcanwhile, the ability to learn from small amounts of training data has a significant impact on the adaptability and usefulness of robots. When they are deployed, they do not necessarily have access to large datasets to fine-tune themselves, particularly when they are attempting interact with people. Therefore,
help with understanding the world, and

---

Published in Transactions on Machine Learning Research (10/2023)
guided learning method that makes use of signals in low-data regime is crucial. Grounded language understanding, in which natural language is used as query against objects in physical environment, allows an intuitive mechanism by which users can instruct agents to engage in tasks. Visuolinguistic approaches to such object inference tasks typically involve training on large pools of image/text pairs and then using language to subselect elements of the sensed environment (Hong el. al. 2021 Zhuge el al. 2021
In order to fully leverage small amounts of multimodal training data, we propose generalized distance- based loss function that can be extended to learn retricval models that incorporate an arbitrary number of modalities, while being robust to missing information. We consider the domain of grounded language- based object retricval (Hu et al. 2016 Nguyen et al. 2021), in which objects in an environment must be identified based on linguistic instructions. Previous approaches have explored combinations of sensors with labels provided by language (Richards et al., 2020). However, much of the existing work has not previously included an arbitrary number of modalities. The ultimate aim of this work is to take arbitrary input modalitics about novel objects, including both spoken and written language, and build a model that allows a system to correctly relrieve that object given a subset of those input modalities.
In this work, propose our new method called extended multimodal alignment (EMMA), generalized geometric method combined with a pross-entropy-basee supervised contrastive loss function (Khosla et al. 2020). Our met can be used to learn retrieval models that incorporate an arbitrary number of views of a particular piece of data. We demonstrate steps towards this goal with a mechanism that learns quickly from a variety of imput signals and is robust to the ablation of inputs. Our contributions are as follows:
1. We propose new approach to multimodal learning that combines geometric and cross-entropy methods. Our method is designed to be applicable to additional modalities they become available.
2. Our method performs better than tt-teatf- contrastive learning (Chen al. 2020) and su- pervised contrastive learning (Khosla et al., 2020) especially when limited training data is available.
3. During training, our method converges about 2.3 times faster than the individual baselines.
4. We demonstrate robustness in the face of missing input signals when one or more modalities are ablated at test time, and propose a straightforward averaging method to quantify performance for multimodal object retrieval scenarios.
We demonstrate the separate utility of speech and text as sources of label information by treating them as sensory input modalities, rather than explicitly as labels.
2
Related Work
Grounded language learning (Roy & Pentland 2002 Mooney 2008 Matuszek et al., 2012) is intrinsically multimodal since its task is to learn the associations among at least two modalitics of language and visual inputs. Prior work has focused on language and visual modalities, using a combination of language and vision to perform visual retrieval (Vo et al. 2019). While recent works have focused on grounding with respect to knowledge set (Zheng & Zhou 2019 Meng et al. 2020), our work focuses on robust multimodal learning to perform grounding over an arbitrary number of input modalities.
The growing number of datasets that contain modalities beyond images and text demonstrates the impor- tance and the challenge of multimodal learning. Bagher Zadeh et al. (2018) introduce large dataset of multimodal opinion sentiment and emotion intensity (CMU-MOSEI) for sentiment analysis and emotion recognition. Kébé et al. (2021) present a multimodal dataset of household objects containing RGB images, depth images, written language, and spoken language, which has been used to support learning grounded language directly from speech given small datasets (Kébé et al., 2022).. [Baltruaaiiis et al. (2019) propose a taxonomy of multimodal machine learning by introducing five technical challenges including alignment the process of identifying the direct relationship between elements across modalitios-whiee is our focus here. Liang et al. 2022) also provide a thorough survey on multimodal learning challenges and approaches.
To the best of our knowledge, there has been limited work on incorporating more than three modalities into a general learning approach. Veit et al. (2018) train a joint model of images, hashtags, and users to perform image retrieval. This work differs from ours in that it becomes computationally complex as more and higher-dimensional modalities are added. Lei et al. (2020) use inputs from three modalities to perform image
2

---

Published in Transactions on Machine Learning Research (10/2023)
retrieval given a sketch using co-attention mechanism. Mittal et al. (2020) use Canonical Correlational Analysis (CCA) to differentiate between ineffective and effective modalities for the emotion recognition task from multiple imput modalities including face, speech, and text. Abouelenien et al. (2014) use language, physiological response, and thermal sensing to detect deceit. However, recent work in the literature has given more attention to this problem (Liang et al. 2023 Reed el. al.] 2022). Liang el al. (2223) provide two new metrics to measure how similar two modalities are and how different pairs of similar modalities interact. The authors propose HighMMT, multimodal transformer-based approach that can incorporate up to 10 modalities by fusing modalities that include unique information. Reed et al. (2022) proposed GATO, generalized multimodal model that can perform multiple different tasks using the same network.
Variational autoencoders or VAEs (Kingma & Welling 2013) are generative latent variable models that have had great success in generating images. Originally designed for unimodal tasks, later work has extended the
technology to work in multimodal scenarios (Wu Goodman 2018 Shi et al. 2019 Palumbo et al.) 2022). Transformer (Vaswani ct al., 2017 is an attention-based neural network architecture consisting of an encoder and a decoder, originally proposed as unimodal architecture for sequence modeling. Different, varieties of transformer models gained attention after they proved to be successful in many different tasks (Lu et al.,
2019 Raffel et al. 2020 Brown et al.
2020 Dosovitskiy ct al. 2021 Lewis et al. 2020). It makes sense to
use them for multimodal learning. Recent works have used multimodal transformers for a variety of tasks and modalities (inter alia, (Tu et al.
2019 Li et al. 2019 Likhosherstov et al.
2023 Zhang et al.
2023
Contrastive loss is mostly used in self-supervised learning (Bui et al. 2021 Alayrac et al.
2020 Chen
le al. 2020), ; with some exceptions such as Khosla et al. (2020). However, a number of different approaches
have been proposed for linguistic-bseed contrastive learning. [Radford et al. (2021) introduced CLIP, which
is designed to align two modalities of text and image using contrastive loss similar to SimCLR (Chen ct al. 2020). Alayrac et al. (2020) use a self-supervised contrastive learning method to learn and combine representations from images, audio, and language. The loss function they define differs from ours in that they do not handle arbitrarily many modalities and do not focus on robustncss in the face of modality drop-outs. Nguyen et al.] (2020) lake a similar approach and perform pairwise cosine similarity to align images and natural language text descriptions to perform object retrieval tasks. Nguyen et al. (2021) take a cross-modal manifold alignment approach to grounded language learning using triplet loss (Chechik et al. 2010 and perform object retrieval by connecting text descriptions of objects to their corresponding RGB-D images.
In contrastive learning, there are two broad families of approaches in the literature. One family is based on cross-entropy losses (Khosla et al. 2020 Chen et al., 2020), and the other is purely geometric (Poklukar ct al. 2022 Carvalho et al.] 2018 Salvador et al.) 2017; Nguyen et al.) 2021). In this paper we marry the two families and show that combining a geometric approach with cross-entropy based supervised contrastive loss superior. The geometric approach is based on the idea of similarity (or distance) of elements of training data (Carvalho et al., 2018 Nguyen et al., 2021 Salvador al. 2017). Our geometric loss function formulation is similar to lifted structured loss (Song et al (0066) and quadruplet loss (Chen et al. 2017] Tursun et al.] 2021), but we apply our idca to four modalities (and could incorporate more) and do not compute the distance among all items in a batch. PPolukar et al. (2022) take a geometric approach which is similar to our work, but differs in that they fuse all embeddings as a central embedding and then minimize distances, whereas we align each modality separately.
3 Problem Description
Given a language command (either text or speech) that describes an object, we want our model to retrieve the correct object from a set. This problem is an exemplar of tasks found in the area of grounded language learning in the fields of robotics and natural language processing. Intuitively, the goal is to take unconstrained natural language queries and select the appropriate object based on the complete set of sensor inputs available to the agent. We demonstrate on domain containing four modalities, all of which refer to objects in the environment: spoken language, written text, RGB (image) inputs, and depth camera inputs.
More formally, given spoken language command F,, a textual language command 24, a set of RGB images X, (.),,,.)), and set of depth images Xa = {x12 (1..m) }, the task is to retrieve the correct object by choosing
3

---

Published in Transactions on Machine Learning Research (10/2023)
the index that has the minimum distance from either of the language commands across all modalities. Depending on which modalities are ablated, we consider up to four distances: sr, a vector of distances between I2 and all RGB images in Xx; sd, a vector of distances between and all depth images in Xa; tr, a vector of distances between 24 and all RGB images in X,: and td, a vector of distances between 2, and all depth images in Xa- In order to select the correct object, we first perform a component-wise average of the relevant modality pair distances for the available modalities. Then, we select the object which has the minimum distance, i.e, we perform an argmin on this average vector of multiple-modality distances. Depending on the missing/ available sensors during test time, we might have any combination of these four distances. For example, if no written instructions are available at test time we compute the component-wist average of sr and sd, and then select the object whose coordinate resulted in the lowest average distance. This method allows us to extend our model to support arbitrary modalities while remaining robust. to test. cases in which some modalities are missing or incomplete.
4 Approach
In keeping with previous work the closely related problem of image retrieval, focus contrastive loss approaches, in which the goal is to learn embedding of data where similar samples in our case, samples belonging to the same class of object-ard are embedded 'close' to one another, while dissimilar samples are farther apart. Since contrastive learning is based on similarity, it is aligned with our goal of training a multimodal model for grounded language learning: contrastive learning enables us to teach the model to map different modalities of the same object closer to each other in the latent space. We describe a novel geometric loss function, GEOMETRIC ALIGNMENT, that simultaneously minimizes intra-class distances and maximizes inter-class distances across cach pair of modalities, yiclding a model that is both effective at the retrieval task defined above and robust to modality dropouts at test time. We additionally combine this GEoMETRIC ALIGNMENT loss with classification-based (cross-entropy) loss function which results in a superior model compared to either geometric or cross-entropy losses alone; we refer to this combination as Extended Multi-Modal Alignment, or EMMA.
Core Concepts. As per typical contrastive loss, we refer to three primary kinds of sample: anchors, data points that acts as a point of reference; and positives and negatives, which refer to a data psint(s) that are similar and dissimilar to the anchor, respectively. For example, to learn the meaning of the concept "book," " the anchor might be an RGB image of a book; the corresponding text description and depth image of that book are in the positive set; and the negative set includes the text description and RGB image of an apple. The hods we discuss differ both in how they choose these three sets and the objective function used.
4.1 Baselines
We compare both EMMA and GEOMETRIC ALIGNMENT against contrastive learning (Chen et al. 2020) and supervised contrastive learning (Khosla et al. 2020), which for conciseness refer to SUPCON. We consider SUPCON as the main baseline since it is general version of multiple contrastive loss functions including triplct loss and N-pair loss (Sohn 2016
4.1.1 Contrastive Loss
We compare our model against the contrastive learning method presented by Chen el al. (2020) where they use the normalized temperature-scaled cross-entropy loss (NT-Xent).. In order to implement this loss function, we use cosine similarity as suggested in Chen et al. (2020p. Another possibility is to use an inner dot product (Khosla et al.) 2020); if not normalized, this can lead to numerical instabilities and overflow ///ndenddoowwww since the dot product is not bounded, but the normalized inner dot product is similar to cosine similarity. We can treat different modalities of an instance as different augmentations of it and consider any combination of two of them as a positive pair. For each item i in a batch and each positive pair
This setting particularly salient. While large bodies of text are frequently available at training time, a person interacting directly with a physical agent may well prefer use only spoken instructions.
4

---

Published in Transactions on Machine Learning Research (10/2023)
of i and j(1 the contrastive loss function is then formulated in eq.
exp(sim(2,,2 2(())//)
log Eneroy exp(sim 2i, 2g )/)
(1)
where A(1) are all negative and positive indices (except 1), sim is a similarity function, and f()).
4.1.2 Supervised Contrastive Learning
Khosla et al. 2020 extend the contrastive learning method (NT-Xent) and propose supervised way of performing contrastive learning to treat not only augmentations of the anchor but also every item that shares the same label with the anchor as positives. This loss function is shown in eq. (2
expl2 2p/T)
Σ NEI P(O
log PEP()
(2)
EREA()) exp(2; za/T)
Although this loss function does not use cosine similarity, embeddings are normalized before performing the dot product, which is equivalent to cosine similarity.
The authors applied this baseline to a unimodal dataset. In this paper, we extend it to work with a multimodal dataset and show that is slower than FMMA to learn and requires more data converge. Since SUPCON considers all pairwise distances in each batch, with M modalities and a batch of size each batch contains Bx. items, and SUPCON computation involves (BM)² pairwise distance terms. However, the computations of our GEOMETRIC ALIGNMENT approach are agnostic with respect to batch size which makes it scalable. As originally proposed, the SUPCON approach we choose as our baseline was applied to unimodal datasets such as ImageNet (Deng et al., 2009), CIFAR-10, and CIFAR-100 (Krizhevsky & Hinton 2009p. We both demonstrate that is possible to use SUPCON for multimodal datasets and extend with the addition of geometric component, which gives benefits to performance and training speed.
4.2 EMMA: Extended Multimodal Alignment
Our proposed multimodal method is composed of two complementary parts. The first part is geometric loss based on distances in the latent space, and the second is a supervised contrastive loss based on cross- entropy (SUPCON). The geometric loss is faster to learn and is more aligned with the downstream task of object retrieval, while the SuPCON loss gives the model the classification ability when the model is trained on enough data. Hence we combine them into a single model, EMMA.
4.2.1 Geometric Alignment Loss
We define distance-based loss function that can be used for arbitrary number of modalitics. Our proposed method is inspired by similarity-basetr triplet loss, and is similar to contrastive loss (Chen et al, 2020 Khosla et al. 2020g under some settings. Triplet loss- learning works by forcing similar concepts from different domains 'together' in some shared embedding space while forcing dissimilar concepts 'apart.' However, standard triplet loss cannot be used for more than two modalities. We note that triplet learning often require an expensive mining step to find harder negative samples enable effective learning (Hoffer & Ailon 2015 Schroff al.] 2015 Zhao et al. 2018 Zhai et al., 2018). This is problematic for throughput and scalability, as run time becomes quadratic in batch size. Our approach does not require mining to obtain good performance, alleviating these issues along with their complicated history in reproducibility and parameter tuning (Musgrave et al.] 2020
Raff 2021 2019).
To address these issues, we use pairwise distance optimization among all modalities of an instance and a sampled negative example (rather than fixed negative examples). We modify the concept of triplet loss as follows. training, we sample two different instances and their corresponding representations from all modalities into two sets-a positive set (referring to a specific object) and a negative set (referring some other object), shown in Figure Unlike prior triplet loss methods, the anchor is not randomly chosen.
5

---

Published in Transactions on Machine Learning Research (10/2023)
This is red apple.
A coffee mug.
BERT
FC layers
3 FC layers
BERT
RGB
ResNer152
FC layers
PC layers
ResNer152
Depth
layers
FC layers
ResNerl
"Black and
Speoch
apple."
2ae 2.0
layers
gold mug."
Figure high-level prototype of our approach and the distances used the GEOMETRIC ALIGNMENT loss. Gray arrows indicate frozen pre-trained models (the parameters are fixed and not trained). The black arrows show 3 fully connected layers with ReLU activation function (Nair & Hinton 2010) after the first two layers. These networks trained. Orange rectangles negative embeddings and green rectangles are positive embeddings. Dashed lines indicate distances to be maximized while dotted lines minimized.
Instead, every item in the positive set becomes anchor once. We minimize the distance between that item and other items in the positive set while maximizing the distance from all items in the negative set.
The objective is to first minimize the distance between each pair of positive points from heterogeneous modalities, and sccond maximize the distance between each positive and negative point from all modalitios. We refer to this approach as GEOMETRIC ALIGNMENT, formulated in Equation 3 and shown in Figure 1
M
M Σ Mig -
C
Σ mi Lmz=11
Σ
|- max(disl(zzt, ama) + az 0)]
Inax(disl(z))) ).))
(3)
In eq. [3), M is the number of modalities, the superscripts + and represent positive and negative objects, a represents an enforced margin between each positive and negative point (set to 0.4 for all modalities without tuning), and = is the embedding from applying a mapping function f, which is a neural network on our input data. In other words, 2m = fm(xm), where each modality m has a specific model fm that different from the models for other modalities. These models do not share their weights. Cosine similarity the opposite of distance, and need reverse the logie for maximization and minimization. We use cosine similarity between pairs of embeddings. It is good choice for high-ddmensional data it is bounded between -1 and 1. Other distance metrics, such Euclidean distance, grow in value with respect their dimensionality, resulting in very large distances for data points. The cos(-) function is a measure of similarity, not distance.
To make the equations more readable, the generic dist function is replaced with the specific cos(.) function. Also, we use pash(x,)) y) = max(cos(x, y) - + a, 0) for maximizing all unique pairwise distances between modalitics positive and negative instances, and pull(z, y)
(1-c0111;;; y), 0) for minimizing the unique pairwise distances among the modalities of positive instances. Hence, GEOMETRIC ALIGNMENT loss function in eq. (3) can be written as L ΣM EM push( 2mg,) IM EM Em pull( ( > =
Our proposed GEOMETRIC ALIGNMENT loss function in eq. (3) can be rewritten as in [4] by fully specifying the summations, showing how our objective function can be reduced to triplet and pairwise loss.
M-
M
==
pull(2t .2)) push(2t,,2 + push(2; mansh(2,2ff) (2,,))) + push (nsh(2t,27)).
(4)
1=
If M (the number of modalitics is 2) and we ignore the last two terms in the derived objective function, it results in the triplet loss method. If M 2, then our objective function reduces to the quadruplet loss method (Tursun et al. 2021 Chen al. 2017 if multiply the first term by 2, ignore the third term, and

---

Published in Transactions on Machine Learning Research (10/2023)
change the last summation to be up to M - (which results in single term). If M = 1, only the last term remains in the loss function which is exactly the pairwise distance- based loss function. This loss function can be seen as a contrastive loss, usually used in the domain of self-supervised learning (Chen et al. 2020).
Our proposed loss function has two advantages over the traditional loss expressed in eq. The first is that our loss function does not need to compute the distance between all items the batch as opposed to SUPCON (Khosla et al., 2020). Instead, we sample only two objects (positive and negative), each of which has M modalities, giving 2M data points (or embeddings). Hence, model can be trained using smaller batch sizes and reduces the mumber of negative samples we need. The second advantage is that this loss function can be used in a multimodal sctting with an arbitrary number of modalities. Although our GEOMETRIC ALIGNMENT is quadratic in the number of modalities, we observe that experimentally, training time increases only by 10 minutes with each additional modality. It is noteworthy that our training procedure does not perform any stochastic dropout of modalities to obtain test-time robustness to missing modalities.
4.2.2 Combining Geometric Loss and Cross- Cross-Entropy based SupCon Loss
The main difference between GEOMETRIC ALIGMMENT and SUPCON is that in GEOMETRIC ALIGNMENT focus the geometric interpretation of similarity using cosine distances, while in SUPCON, cosine distances are used to compute classification--asee loss function similar to cross-entropy. Each method offers unique advantages. In GEOMETRIC ALIGNMENT, the advantages are an intuitive learning objective in terms of distance, interpretability of the learned embedding space, and faster convergence. The advantago of SUPCON is that with enough training data points, it provides the model with complimentary classification- based signals which can be helpful, especially with more chaotic modalities such as speech (see fig. 77. We next combine the GEOMETRIC ALIGNMENT defined in eq. (4) with SUPCON defined in eq. (2), and refer to this approach EMMA, for extended multimodal alignment. The combination is done by taking the sum of the equations for GEOMETRIC ALIGNMENT and SUPCON (shown in eq-
Enforcing Representational Similarity for Multi-modal Views of the Instances
Aligu different modalities similar objects
Push dissimilar representations apart
M
M
L=
(2 2(.k)
push(2 (2 2T,e) + push( 27,y, 24,0) + push(2 p
KE
k k=j+1
(5)
Grounding through Contrastive Predictive
M
log exp(zim 23/T) 7EA(i.) exp(zi.m 2//T)
Σ
mi)
BEP(i.)
where A(i, m) includes all items in the batch except for the zi, itself and P(i, includes all the modalities of all instances that have the same label current instance excluding 2,m itself. In other words, the
collection of embeddings indexed by pii,m) is UU Zi,,, U U 3.
TAMEM
Although both SUPCON and GEOMETRIC ALIGNMENT are trying to bring target objects together (and push dissimilar ones away), there a trade-off in SUPCON where there is a normalized ranking across them. In contrast, in GEOMETRIC ALIGNMENT distances can be arbitrarily near or far. Our approach tries respect the distinct modalitics of cach instance. While our formulation is similar to SUPCON notationally, conceptually SUPCON uses different augmentations of RGB images positive examples (coming from the same underlying empirical distribution), while we use different modalities drawn from different distributions.
4.3 Network Architecture
Transformers (Vaswani et al., 2017 have become de facto architoctures in the natural language pro- cessing community and have shown great success across different tasks. We use BERT (Devlin et al., 2019) embeddings contained in the FLAIR library (kkik et. al. 2019a bb to featurize textual input, and

---

Published in Transactions on Machine Learning Research (10/2023)
way2vec2 (Bavvski al. 2020 to extract audio embeddings from speech. Both BERT (Devlin et al. 2019 and wav2vec2 (Baevski et. al. 2020) are self-supervised language models using transformers (Vaswani et al. 2017). To process images, we use 15sNet152 (He et al. 2016) for both RGB and depth images which gives us a 2048-dimensional embedding vector. Depth images are colorized before passing to ResNet152. We then use different. instantiations of the same multi-layer perceptron (MLP) consisting
fully connected layers with ReLU activation [Nair & Hinton 2010) to map each of these embeddings to a shared 1024-dimensional space where we can compute the distances between all embeddings. These networks do not share weights.
5 Experiments and Results
In this section, we describe the experiments and provide quantitative and qualitative results by comparing our methods against supervised contrastive lcarning (Khosla et al.) 2020) and contrastive loss (Chen et al., 2020). Moreover, we evaluate the quality of object retrieval models learned using EMMA loss function. We first describe the dataset we use and the setup of the experiments, then describe the metrics by which we evaluate performance. We demonstrate the effectiveness of our approach on a publicly available multimodal dataset called GoLD (Kébé et al. 2021), which contains RGB images, depth images, written text descriptions, speech descriptions, and transcribed speech descript.ion for 207 object instances across 47 object classes (see Figure There are a total of 16,500 spoken and 16,500 textual descriptions. The training set, validation set, and test set contain 7380, 4160, and 4960 items (each consisting of four modalities), respectively. Speech is converted to 16 Hz to match the wav2vec2 speech model.
5.1 Setup
To evaluate model we measure different performance metrics on retrieval task where the model selects object from set given language description. Only one of the objects corresponds to the description. Similar to Khosla et al. (2020), we use stochastic gradient descent (SGD) optimizer with momentum (Ruder, 2016) with a flexible learning rate starting at 0.05. All models are trained for 200 epochs with a batch size of 64 on a Quadro RTX 8000 GPU. Following the literature, we used a temperature of 0.1 for training the contrastive learning method (section 4.1.1). and a temperature of 0.07 for training SUPCON (section 4.1.2).
We compute the distance between the given natural language description and all available sensory modalities for 5 randomly selected objects (one of which corresponds to the description, with the others from different object classes), as described in section 5.4 For example, if RGB and depth are present, we compute the distance between the language embedding and all candidate RGB and depth embeddings. We then average these distance matrices and choose the closest image embedding (average distance of RGB and/or /rr depth from language) as the prediction. In order to use cosine distance, we subtract the cosine of the angle between two embeddings (which represents similarity) from
5.2 Metrics
The best metric to capture the performance in such scenario is mean reciprocal rank (MRR) for Q queries. For each query, we predict the rank of all objects based on their distance from the language command, and then the inverse rank of the desired objects in all queries are averaged. We compute MRR by 1 Σ rank, [ MRR is an informative metric because it captures the idea that having the correct object as the second choice should be considered better than having as a last choice. Because our approach is designed to be robust to missing modalities, summarizes the results of all experiments. To provide a better sense of the metric, consider model that always ranks the correct object in the second place: such a model would have an MRR of 0.5.
report MRR for different combinations of modality dropouts. Table 1
5.3 Learning from Limited Data
We consider a case where lower amounts of data are available to train the model to replicate a real-world constraint. When a personal robot is deployed, it does not have access to huge amounts of data to finetune its model and should be able to learn and adapt using the minimum amount of data possible. Figure 2 depicts
8

---

Published in Transactions on Machine Learning Research (10/2023)
Methorss-
spepch
RGB
text dexthhepth
test /RGB
text/speschy
text/spoccli
specchy RRB/
text, RGB/
all
percent
depth
RGB
depth
degeh
EMMLA-25
70.16-11.79
71.8310:25
65.44=0.37
90.2110.49
87.2810.52
59.26-00.55
73.07
91.36.10
90.72-0.31
SupCon-25
46.49.10.13
46.35.10.18
83.77.0.79
84.87
80.71110.78
81.48.0.6
46.39
85.73.10.22
83.25-10.69
Goom.225
68.11+0.52
67.51
65.60+40.41
87.4+00.9
65.48++0.26
87:20--0.44
68.17++0.30
87.94+0.31
88.12-40.6
SimCLRR-22
30.99:0.52
51.21 1.058
50.3740.43
50.51-0.7
08.5910.221
08.77
19.131.0.56
48.6:0..0
47.75=0.117
EMMA--10
77.63=0.229 29
T8l 76.6620.61
89.87:005
91.26:01.8
89.6620038 36
90. 90.97:0.666
50.3240. 15
92.71:0.5
92.72=0.47
SupCon- 100
78.18200558
79.69:1054
89.04110.55
90.5620.74
88.75110.66
90..5:0..00
81.210039
91.9640.02
92.0320.7
Goom.-100
76.82:00.34
18.34410.29
80.64:10.38
91.13-0.77
60.21:10.45
90.95-00.83
79.37:-0.29
92 29:10.1
02.14
SinCLR--100
71.7420073
13.3740.39
89.72:00.54
90.82=11.37
59.13:10.61
90.26:00.58
74.9620.44
91.9240.41
91.72=00.3
Table 1: Average and standard deviation of mean reciprocal rank (MRR) over 5 runs on a held-out test set with different modalities ablated during testing. For 5 objects, MRR ranges from 20% to 100% (higher is better). A random guess would have MRR of 0.46. The batch size is 64 and the optimizer is SGD with momentum for all experiments. Column headers describe which modalities are present at query time for example, in column 2 'speech/deptie both text and RGB are ablated. Models were trained with cither 25% or 100% of the available training data (shown in Column 1). EMMA is either betler than or very close lo state-of-the- the-art methods for most cases. With 25% of the training data used, EMMA outperforms other methods in all cases. Statistically significant results are shown in bold.
the MRR for the models trained with different amounts of training data. As expected, less training data means lower performance. However, the drop in performance is significantly higher in SUPCON compared to EMMA when less data is available. With 369 training data points, SuPCON does not learn and results in an MRR consistent with a random baseline, while EMMA achieves an MRR of 0.82.
While this demonstrates the empirical benefit of EMMA, it raises the question of what is contributing to this dif- ference. We hypothesize that by decreasing the number of training examples, SUPCON does not get enough examples of positive pairs in the contrastive loss. We note that the number of negative pairs in each batch is about 15 times larger than the number of positive pairs in SUPCON when we have 369 training data points. However, this ratio is about 1 in FMMA. In order to test this, reduce the batch size from 64 to and the MRR score of SUPCON goes from 0.47 up to 0.61.
1.0
0.9
0.8
0.7
0.6
EMMA
0.5
SupCon
Geometric
0.4
This is an example of how carefully using minimal signals can result in a better model, similar to what we proposed in our GEOMETRIC AlIGNMent method. This especially important in low-data regime. SUPCON, on the contrary, is useful when trained more data since the loss function contrasts all items in the batch. When little data is avail- able, contrasting all items in the batch confuses the model, however, with large amounts of data it leads to classification abilitics since SUPCON is based on cross-entropy.
51
147
738 2214
3321 5535
Figure 2: Mean Reciprocal Rank (g-axis) when all modalities are available, shown as func- tion of the amount of training data (æ-axis). All averaged over 5 runs for the downstream task of object retrieval. Higher is better. Or- ange is SUPCON, green is our proposed GE- OMETRIC ALIGNMENT, and blue is our pro- posed EMMA loss function. We train all mod- els for 200 epochs. EMMA outperforms other models, especially when only small amounts of training data are available.
The literature is not consistent about the effect of the number of negative pairs on the contrastive loss function. Awasthi et al. (202)) state that more negative examples do not necessarily hurt performance; however, they assume non-overlapping latent classes. Other work (Saunshi et al. the number of negative points beyond a Uhreshold can hurt performance, as it becomes more likely to see negative samples that share the same latent features as an anchor and positive samples. They refer to this as 'collision' and argue that makes it harder to train the model to learn good representations. In our case, the dataset we use consists of everyday objects that share overlapping latent features (c.g., same color, same size, etc.). Hence, more negative examples when the total number of examples is low hurts learning.
2019 Ash et al. 2022) argues that incrcasing
9

---

Published in Transactions on Machine Learning Research (10/2023)
1.0
1.0
1.0
1.0
0.9
0.9
0.9
0.9
0.8
0.8
0.7
0.7
0.6
0.6
0.6
0.6
EMMA SupCon Gesmetric
EMMA SupCon
MMA SupCon
EMMA
0.5
0.5
0.5
0.5
SupCon
Geometnic
Geometric
Geometric
0.4
147
735
2214
3321
5535
0.4
738
2214 3321
5535
0.4
147 2214
3521
5535
0.451
147
738
2214
3521
5535
(a) All modalities available.
(b) Text modality ablated.
(c) Speech/depte ablated.
(d) Speech /RGB ablated.
Figure 3: Mean Reciprocal Rank (g-axis) for selected modalities of the held-out test set, shown as function of the number of training samples (I-axis) and averaged over 5 runs. Higher is better. Orange is SUPCON, green represents our proposed GEOMETRIC ALIGNMENT, and blue our proposed EMMA loss function. In all cases, EMMA outperforms other models when smaller amounts of training data are available.
5.4 Modality Ablation
There may be different interpretations of "missing modalities" and hence different design choices. One of the advantages of our approach is that when different sensory modalities become unexpectedly unavailable during operation (test time), its performance degrades only minimally. In Figure B and table ! we show that in such ablations, FMMA outperforms SUPCON in low data scenarios. The loss function requires no changes beyond adjusting the value of M in eq. B according to the number of modalities available during training. Our goal the non-trivial downstream prediction task: determining what objects are being referred to by arbitrary language from a small number of examples. When we consider only text, RGB, and depth, written language is used as the query modality, and compute the distance of RGB and depth modalitics from it and then average them. However, when speech is incorporated as an additional modality, distance of RGB and depth from text and from speech which gives us 4 distance matrices, which we then take the average of. At test time, depending on which modalities are available to the model, we can use speech, text, or both to take the average distance of RGB and depth embeddings from the linguistic query.
compute the
There are a total of nine possible modality dropouts and corresponding distance computations. In all cases we compute the distances of available modalities (and average them if there are more than two modalities). Figure g shows the MRR per number of training data for four example cases in which different modalities are ablated. At 25% of the training data, SUPCON struggles to learn and the converged MRR is at 0.46, while EMMA achieves an MRR of 0.73. A random model would have an average MRR score of 0.46.
When we drop the text modality (table D performance decreases from about 0.93 to about 0.82, showing that speech cannot completely replace text in our current effort.. Tn Figure 7 the alignment of shared embeddings for randomly sampled set of classes is visualized; this visualization suggests that the spoech modality not aligned well as the lext modality. This supports our hypothesis that a geometric alignment of the latent space is crucial to good performance for object retrieval and multimodal understanding. There is slight gap in performance when depth or RGB are dropped (Figures 3c and 3d) compared to when all modalities are available (Figures 3a and 4 showing that our model is robust when RGB or depth sensors fail.
In Figure 3bb when speech is used as the query and the text modality is ablated, the SUPCON baseline works slightly better than EMMA as more training data becomes available, while EMMA still learns faster. Although the difference is negligible, the reason is that SUPCON optimizes for the classification task and when enough data is available the classifier can be trained. But, GROMETRIC ALIGNMENT is trained to align modalities properly in the latent space, which does not necessarily help the downstream task especially if one of the modalities is chaotic, and in our case, the speech modality has high variance which makes it harder to align. However, both GEOMETRIC ALIGNMENT and EMMA work significantly better than SUPCON when less training data is available. Future research will consider strategics to align more chaotic modalitics.
10

---

Published in Transactions on Machine Learning Research (10/2023)
5.5 Convergence Speed
Figure shows the MRR per epoch of EMMA and GEO- METRIC ALIGNMENT against SUPCON (Khosla et al. 2020 and SimCLR (Chen et al.j 2020) when all modalities are available. EMMA learns faster and results in better per- formance compared to both SUPCON (Khosla et al.) 2020 and SimCLR (Chen et al. 2020) with all modalities avail- able at test time. We observe that not only does SimCLR learn more slowly, but that it is prone to overfitting; while this can be addressed with careful tuning, an approach that is innately robust to overfitting without tuning is preferable.
0.9
0.8
0.7
SupCon
0.5
Geometric
SImCLR
EMMA takes almost 8 epochs converge and each epoch takes roughly 0.7 minutes, yielding 5.6 minutes until con- vergence, while SuPCON takes 36 epochs to converge and each epoch takes approximately 0.52 minutes, totaling 18.7 minutes. When we ablate modalities or use less training data, training takes less time.
Figure 4: Mean Reciprocal Rank (y-axis) on the held-out test with the ablation of se- lected modalities against training epochs (x- axis), averaged over 5 runs. In almost all cases, EMMA learns in fewer epochs.
5.6 EMMA Component Contributions
As FMMA combination of geometric approach and cross-entropy--based contrastive learning, it is worth considering what each component contributes to the algorithm's performance. One of the obvious advantages of GEOMETRIC ALIGNMENT over SUPCON is that the loss function of GEOMETRIC ALIGNMENT is designed based on distances bet ween embeddings. This design is more aligned with the downstream task of object detection based on the distance of candidate object embeddings from the embedding of the language query. This is evident when small amounts of training data are used (fig. 2
Figure shows the impact of each of GEOMETRIC ALIGNMENT and SUPCON (represented by color) the overall MRR of ob- ject retrieval when the weight of each component's 's contribu- tion to EMMA varies. The y-axis shows the MRR. score, and higher value means better object retrieval ability. The r-axis shows the percentage of dissimilar object representations being mapped outside the specified margin from each other in the latent space. higher percentage means more of the dissimi- lar object embeddings in the latent space respect the enforced margin. The lighter colors indicate less weight for GEOMETRIC ALIGNMENT and more weight for SUPCON, and vice versa. The figure shows three positive correlations between each two com-- binations of the three dimensions (I-axis, y-axis, and color).
0.90
1.0
0.85
0.80
Mhit al Heal
0.7
0.75
Weighh: Gacom
0.70
0.5
0.65
0.3
0.60
0.2
0.55
0.1
0.50
0
10
40
50
60
73
80
0.0
Percentage Dissimllrr Bemms With a Cerlain Distarce Beineeen Them
The impact of GEOMETRIC ALIGNMENT and SUPCON on the MRR and the percentage mapped outside a margin from each other, as a function of their contribution (weights) in EMMA. Higher is better for both x and axes.
Figure 5:
In GEOMETRIC ALIGNMENT we enforce a margin between neg- ative and anchor points make sure that dissimilar items are not mapped within that margin from each other. In fig. 5 see that increasing the weight of GEOMETRIC ALIGNMENT in EMMA increases both the MRR and the number of dissimi- lar data points being mapped such that the distance between them is greater than the minimum margin. Figure 5 also shows
dissimilar items being
a positive corrclation between MRR and the percentage of dissimilar items mapped outside margin from each other, which suggests that enforcing a margin between anchor and negative points in the loss function of GEOMETRIC ALIGNMENT helps the model learn better representations in the latent space.
11

---

Published in Transactions on Machine Learning Research (10/2023)
Text
RGB
Depth
Speech
bowl
toothbrush
food_can
book
apple toothpaste
plate
shampoo
lemon
eye_glasses
Figure 7: T-SNE (Van der Maaten Hinton 2005) projection of test embeddings of randomly selected classes of objects using EMMA. Each modality separately projected into a three-dimensional space. Each RGB and depth image is associated with several language descriptions, leading to denser plots for text and speech. In perfect embedding, all instances of a class would be clustered in identical areas of the embedding space across all modalities. We can see that EMMA successfully encourages all four modalities to live in a common manifold, allowing accurate retrieval even when modalities are missing.
5.7 Qualitative Results
An example of the need to consider multiple modalities jointly shown in fig. 6 showing how EMMA able correctly select an object instance from several similarly shaped and describ- able objects. EMMA more accurately ranks objects and handles ambiguities with respect re- opener. It is light blue
where the query is "This is in color." and the candidate objects are opener, potato, soda bottle, book, and light bulb, EMMA correctly seleets the can opener as the desired object, while SUPCON selects the light bulb. To help understand the embeddings EMMA learns,
trieval queries. In a retrieval
in fig. 7 we consider projections of a randomly selected subset of classes the high-dimensional learned embeddings into 3-dimensional space uss- ing t-SNE (Van der Maaten & Hinton, 2008).
Object
RGB
Depth
EMMA
SurCow
Can Opener
Rank
Rank
Combined with the quantitative results, these pro- jections demonstrate that our model is learning to map instances of the same class closer to each other regardless of their modalitios. As qualitative ex- emplars, we note that toothbrush and toothpaste are mapped almost on top of each other in the text modality, showing semantic and syntactic similar- ity, but not in the visual modalities. Also, apples and lemons are mapped close to each other in all modalities.
Potato
Rank
Rank
Soda Bottle
Rank
Rank
Book
Rank
Rank
Lightbulb
Rank
Rank
"This is a can
There remains room for improvement. Specifically, the speech modality harder to handle. Figure shows that although the relative position of in- stances is correct in the speech space, the distinction and clustering of different objects are not good as the other three modalities. Text is better clus- tered, which makes sense, as the variation in written text is much smaller than the other three modali- ties. Variation speech is higher because there are number factors affecting speech understanding, inchuding different accents, native language, gender,
Query:
opener. It is light
blue color."
Figure 6: An example of the downstream task of object retrieval. The green rectangle shows a correct retrieval and the orange rectangle indicates wrong onc. Both models trained on only 25% of the training dataset using the same setup. While SUPCON confuses "light blue" in the query with "light bulb" and predicts light bulb, EMMA correctly identifies the can opener as the intended object
12

---

Published in Transactions on Machine Learning Research (10/2023)
and age (Kébé et al. 2022). Also, it will be valuable to explore using the presented system to support object
retrieval performed by an embodied (real or simulated) agent Mo et al. (2023): Bejjani et al. (2021).
6 Conclusion
In this work, we have demonstrated the effectiveness of our approach to learning from high-dimensional information from an arbitrary number of input modalities, both when training data is scarce, and when one or more modalities are unavailable at test time. On an object retrieval task from a testbed that contains four separate modalities, consistent with the information that might be available to a physical agent, our model EMMA achieves significantly better results when only small amounts of training data are available. This gap even larger when the text modality is missing during the test and speech is used as the query, Moreover, FMMA is more than twice fast SuPCON and requires about 30% of the time that SUPCON needs to converge. The requirements for agents in different settings may differ, and training speed may not be critical in offline learning scenarios; however, since models often need to be fine-tuned for different tasks when it comes to transfer learning, the training speed is relevant. More importantly, in scenarios where language labels are provided by human interaction, learning from small amounts of data helps address a potential bottleneck in the learning process. Our proposed method is general enough to be applied to a variety of multimodal retrieval problems.
Acknowledgments
We would like to thank the anonymous reviewers for their helpful comments, questions, and suggestions. This material is also based on research that is in part supported by the NSF under Grant Nos. 2007290, 2024878, and 2145642; the Army Research Laboratory, Grant No. 1911 and by the Air Force Research Laboratory (AFRL), DARPA, for the KAIROS program under agreement number FA8750-19-2- 1003. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either express or implied, of the Air Force Research Laboratory (AFRL), DARPA, or the U.S. Government.
References
Mohamed Abouelenien, Veronica Pérez-Rosas, Rada Mihalcea, and Mihai Burzo. Deception detection using a multimodal approach. In Proceedings of the 16th International Conference on Mullimodal Interaction, ICMI '14, pp. 58 65, New York, NY, USA, 2014. Association for Computing Machinery.
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. Flair: An casy-to-use framework for statc-of-the-art nlp. In NAACL Annual Conference of the North American Chapler of the Associalion for Compulational Linguislics (Demonstrations), pp. 54 54-59, 2019a.
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. Pooled contextualized embeddings for named entity recognition. In NAACL Annual Conference of the North American Chapter of the Association for Com- putational Linguistics, Minncapolis, Minnesota, June 2019b. Association for Computational Linguistics.
lean-Baptise Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelovié, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-Supervised MultiModal Versatile Networks. In Newr/PS, 2020.
Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Dipendra Misra. Investigating the role of negatives in contrastive representation learning. In Gustau Camps-Vall Francisco J. R. Ruiz, and Isabel Valera (eds.), 25th International Conference on Artificial Intelligence and Statistics, 2022.
Pranjal Awasthi, Nishanth Dikkala, and Pritish Kamath. Do more negative samples necessarily hurt in contrastive learning? In Kamalika Chaudhuri, Stefanic Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1101-1116. PMLR, 2022.
13

---

Published in Transactions on Machine Learning Research (10/2023)
Alexei Bacvski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. In NewrIPS, 2020.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Mul- timodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph. In The 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, July 2018. Association for Computational Linguistics.
Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41(2), 2019.
Wissam Bejjani, Wisdom Agboh, Mehmet R. Dogar, and Matteo Leonetti. Occlusion-awarre search for object retricval in clutter. In 2021 IEEE/RS, REE/SJJ International Conference on Intelligent Robots and Systems IEEE, 2021.
(TROS), pp.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christo- pher Berner, Sam McCandlish, Alec Radford, Πly Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrlPS, 2020.
Nghi DQ Bui, Yijun Yu, and Lingxiao Jiang. Self-supervised contrastive learning for code retrieval and sum- marization via semantic- preservin transformations. In The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2021.
Micael Carvalho, Rémi Cadène, David Picard, Laure Soulier, Nicolas Thome, and Matthieu Cord. Cross- modal retrieval in the cooking context: Learning scmantic text-image embeddings. In The 41st Interna- lional ACM SIGIR Conference on Research and Develoyment. in Information Retrieval, 2018.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large Scale Online Learning of Image Similarity Through Ranking. Journal of Machine Learning Research (JMLR), 2010.
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. arXiv preprint arXiw:2002.057 05709, 2020.
Weihua Chen, Xiaotang Chen, Jianguo Zhang, and Kaiqi Huang. Beyond triplet loss: deep quadruplet network for person re-identification. In IEEE Conference Compuler Vision and Pallern Recogrition, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognétion, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pro-training of deep bidirec- tional transformers for language understanding. In The Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Represendations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016.
Elad Hoffer and Nir Ailon. Deep Metric Learning Using Triplet Network. In Aasa Feragen, Marcello Pelillo, and Marco Loog (eds.), SIMBAD 2015: Eimilarity-Basee Based pottern Recognition, pp. 84 84-92, Cham, 2015. Springer International Publishing.
14

---

Published in Transactions on Machine Learning Research (10/2023)
Weixiang Hong, Kaixiang Ji, Jiajia Liu, Jian Wang, Jingdong Chen, and Wei Chu. Gilbert: Generative vision-language pre-training for image-text retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1379 1388, 2021.
Ronghang Hu, Huazhe Xu, Marcus Rohrbach, Jiashi Feng, Kate Saenko, and Trevor Darrell. Natural language object retrieval. In The IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Gaoussou Youssouf Kébé, Padraig Higgins, Patrick Jenkins, Kasra Darvish, Rishabh Sachdeva, Ryan Barron, John Winder, Donald Engel, Edward Raff, Francis Ferraro, and Cynthia Matuszek. spoken language dataset of descriptions for speech-based grounded language learning. In NeurIPS Systems Dalasels and Benchmarks Track, 2021.
Gaoussou Youssouf Kébé, Luke E. Richards, Edward Raff, Francis Ferraro, and Cynthia Matuszek. Bridging the gap: Using deep acoustic representations to learn grounded language from percepts and raw speech. In The AAAI Conference on Artificial Intelligence. AAAI Press, 2022.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), NewrIPS, 2020.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. ar preprint arXiv:1312.6114, 2013.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Jianjum Lei, Yuxin Song, Bo Peng, Zhanyu Ma, Ling Shao, and Yi-Zhe Song. Semi-heterogeneous three way joint embedding network for sketch-basep image retrieval. IEFE Transactions on Circuils and Systems for Video Technology, 30(9):322633)) 3237, 2020.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-oo-sqquence pre-training for natural language generation, translation, and comprehension. In The 58th Annual Meeting of the Association for Compu- tational Linguistics, 2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant bascline for vision and language. arXiv preprint arXio:1900 190..09587,, 2019.
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions. arXie preprint ar Siv:2209.004300 2022.
Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis- Philippe Morency, and Russ Salakhutdinov. High-modality multimodal transformer: Quantifying modality & interaction heterogeneity for migh-mo representation learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL / net wttp:///ww......oeoooottttttttaa
Valerii Likhosherstov, Amurag Arnab, Krzysztof Marcin Choromanski, Mario Lucic, Yi Tay, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. TMLR, 2023.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representat ions for vision-and-languagee tasks. NewrIPS, 2019.
Cynthia Matuszek, Nicholas FitzGerald, Luke Zettlemoyr,, Liefeng Bo, and Dieter Fox. A joint model of language and perception for grounded attribute learning. In The International Conference on Machine Learning, June 2012.
Chuan Meng, Pengjic Ren, Zhumin Chen, Weiwei Sun, Zhaochun Ren, Zhaopeng Tu, and Maarten de Rijke. DukeNet: A Dual Knowledge Interaction Network for Knowledge-Ground Grounded Conversation, pp. 1151-1160. Association for Computing Machinery, 2020.
15

---

Published in Transactions on Machine Learning Research (10/2023)
Trisha Mittal, Uttaran Bhattacharya, Rohan Chandra, Aniket Bera, and Dinesh Manocha. M3er: Multi- plicative multimodal emotion recognition using facial, textual, and speech cues. Proceedings of the AAAT Conference on Artificial Intelligence, 34(02))1359 1367, Apr. 2020.
Yuchen Mo, Hanbo Zhang, and Tao Kong. Towards open-world interactive disambiguation for robotic grasping. In 2023 IEEE International Conference on Robolics and Aulomalion (ICRA), pp. 8061 8067, 2023. doi: 10.1109 AA..999IAAA891111 10161333.
Raymond J. Mooney. Learning to connect language and perception. In The 23rd National Conference Artificial Intelligence, AAAT '08, pp. 1598-1601. AAAI Press, 2008.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A Metric Learning Reality Check. In ECCV, 2020.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In JCML, pp. 807-814, 2010.
Andre T. Nguyen, Luke E. Richards, Gaoussou Youssouf Kebe, Edward Ralf, Kasra Darvish, Frank Ferraro, and Cynthia Matuszek. Practical cross- modal manifold alignment for robotic grounded language learning. In The IEEE/CKE Conference on Computer Vision and Pattern Recognition Workshops, 2021.
cross-modal
Thao Nguyen, Nakul Gopalan, Roma Patel, Matthew Corsaro, Ellie Pavlick, and Stefanie Tellex. Robot Object Retrieval with Contextual Natural Language Queries. In Robotics: Science and Systems, 2020.
Emanuele Palumbo, Imant Daunhawer, and Julia Vogt. MMVAE+: Enhancing the generative quality of multimodal VAFs without compromises. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022.
Petra Poklukar, Miguel Vasco, Hang Yin, Francisco S. Melo, Ana Paiva, and Danica Kragic. Geometric mul- timodal contrastive representation ion learning. In The 39th International Conference on Machine Learning, volume 162. PMLR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, 2021.
Edward Raff. A Step Toward Quantifying Independently Reproducible Machine Learning Research. In NewrIPS, 2019.
Edward Raff. Research Reproducibility as a Survival Analysis. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 116140):1-67, 2020.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth- maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research, 2022. ISSN net 2835-8856. URL https://openreviae wwwp:e/wwwwwwteeeeoooooeeeeeeeee hett/formm1dd1110kttj Featured Certification, Outstanding Certification.
Luke E. Richards, Kasra Darvish, and Cynthia Matuszek. Learning Object Attributes with Category- Free Grounded Language from Deep Featurization. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), October 2020.
Deb Roy and Alex Pentland. Learning words from sights and sounds: computational model. Cognitive science, 26(1):113 1141166,, 2002.
16

---

Published in Transactions on Machine Learning Research (10/2023)
Sebastian Ruder. An overview of gradient descent optimization algorithms. arxiv preprint 2016.
Amaia Salvador, Nicholas Hynes, Yusuf Aytar, Javier Marin, Ferda Ofli, Ingmar Weber, and Antonio Tor- ralba. Learning cross-modal embeddings for cooking recipes and food images. In Proceedings of the IEFF Conference Computer Vision and Pattern Recognition, 2017.
Nikunj Saunshi, Orestis Plevrakis, Saujeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. theo- retical analysis of contrastive unsupervisod representation learning. In The 36th International Conference on Machine Learning, 2019.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. FaceNet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Yuge Shi, Brooks Paige, Philip Torr, et al. Variational mixture-of-experts autoencoders for multi-modal deep generative models. Advances in Neural Information Processing Systems, 32, 2019.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2016.
Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Computer Vision and Pullern Recognition (CVPR), 2016.
Osman Tursun, Simon Denman, Sridha Sridharan, Ethan Goan, and Clinton Fookes. An efficient framework for zero-shol sketch-based image retrieval. arXio preprint arXiv:2102. 2022102200000 04016, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 911)), 2008.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 5998 6008, 2017.
Andreas Veit, Maximilian Nickel, Serge Belongie, and Laurens van der Maaten. Separating self- exfreesprssion and visual content in hashtag supervision. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays. Composing text and image for image retrieval-an empirical odyssey. In The IEEE/CVF Conference on Computer Vision and Pallern Recognition, 2019.
Mike Wu and Noah Goodman. Multimodal generative models for scalable seakly-supervised learning. Ad- vances in neural information processing systems, 31, 2018.
Yao Zhai, Xun Guo, Yan Lu, and Houqiang Li. Tn Defense of the Triplet Loss for Person Re-Identificatio. Arxio e-prints, 2018.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tumed audio-visual language model for video understanding. arXie preprint arXiv:2306.0283 2023.
Yiru Zhao, Zhongming Jin, Guo-Jun Qi, Hongtao Lu, and Xian-Sheng Hua. An Adversarial Approach to Hard Triplet Generation. In European Conference on Computer Vision (ECCV). Springer, 2018.
Wen Zheng and Ke Zhou. Enhancing conversational dialogue models with grounded knowledge. In Proceed- ings of the 28th ACM International Conference on Information and Knowledge Management, CIKM '19. Association for Computing Machinery, 2019.
Mingehen Zhuge, Dehong Gao, Deng- Ping Fan, Linbo Jin, Ben Chen, Haoming Zhou, Minghui Qiu, and Ling Shao. Kaleido-bert: Vision-language pre-training on fashion domain. In Proceedings of the IEEE/CVP Conference Computer Vision and Pattern Recognition (CVPR), pp. 12647-12657, June 2021.
17


Reviews:
[{"review_id": "sU9s5K9Bue", "reviewer": "Reviewer_NCbe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "note: I had reviewed the previous version of this submission. My main concerns from the previous version were:\n\n1. It was not clear when the proposed approach (using both geometric and contrastive alignment objectives are useful) actually works, and there should be a deep discussion on when we need alignment, when geometric and contrastive alignment is necessary, and when we should combine both.\n\n---> The authors have addressed this concern with through adding section 5.6 which varies the weight of the two components of EMMA (Geom and SupCon) to study their effects on performance and other qualitative measures of the representation space. The results here are quite useful and can be broadly used in the community for investigating various distance measures and supervised/unsupervised versions of contrastive learning. These experiments also show the effectiveness of their proposed methods which combine both objectives.\n\n2. The results were not very convincing, and there was a lack of standard deviation and statistical analysis.\n\n---> The authors have addressed this partially by focusing on the low data regime where it seems likes performance boosts are higher, and by adding multiple runs and deviations. However, there is a concern that the bolding of numbers in Table 1 is misleading, since some of the bolded numbers fall into the same standard deviation as unbolded numbers. I would suggest a statistical test and only bolding if the increase in performance is significant, for example referring to https://aclanthology.org/P18-1128.pdf.\n\n3. There were some concerns about the computational tradeoffs since you are computing and optimizing 2 objective functions, and how that affect learning. There can also be deeper analysis of how learning should be performed - ablating training for both at the same time, or 1 then the other?\n\n---> The authors have addressed computational concerns and added running/training time details. The authors have also added suitable ablations wrt training objectives and modalities.\n\nFor the new version, I am mostly satisfied with the changes. There are some other concerns/suggestions I have for the authors:\n1. The authors motivate the problem setting of many modalities, but if I'm not wrong there is nothing specifically new about their methods that make it suitable for dealing with many modalities, they just take pairwise contrastive objectives and do it pairwise for M>2 modalities. More clarification here would be useful to make the motivation more clear. There have also been some studies on models for many modalities, both in TMLR last year, which could be good references under the related work paragraph 'To the best of our knowledge, there has been limited work on incorporating more than three modalities into a general learning approach.'\nhttps://openreview.net/forum?id=ttzypy3kT7\nhttps://openreview.net/forum?id=1ikK0kHjvj\n\n2. Likewise the method isn't really tailored for missing modalities, despite the authors motivating that front. It would be good to get some clarification here too, and similarly references for recent work in dealing with robustness for missing modalities should be valid baselines if that is a main contribution of the paper. For example there are some baselines for noisy/missing modalities listed in the recent multimodal survey paper: https://arxiv.org/abs/2209.03430\nRight now the main comparisons are to other CL-based objectives, but this class of methods may not be the most suitable for dealing with low-data and missing modality regimes in the first place.\n\n3. Is there something deeper you can say about the 2 objectives either theoretically or via experiments? Is it mostly about the impact of margin loss vs contrastive loss on representation space? Or is the distinction mostly about the contrastive objective being unsupervised or supervised? If its the first one I should be able to replace supCon part in the objective with unsupervised contrastive learning and still have it work well. If its the second one then I should be able to combine CL with supCon (without geometric) and still do really well. Or if I really need to do supCon and I have access to labels why don't I just directly do supervised learning/fine-tuning after I train with unsupervised geometric objective. Some clarification here would be great.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the setting where there are more than two modalities (one of them a natural language query and other modalities representing physical objects), in a low data regime, and with the real-world constraint that sensors fail due to obstructions or adverse conditions. The authors propose a method called extended multimodal alignment (EMMA) that uses a combination of geometric alignment loss and supervised contrastive learning loss. Results show strong results in grounded language object retrieval based on four modalities including vision, depth sensing, text, and speech, and faster convergence.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see the 3 points in suggestions above.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Dc0NoxDRlG", "reviewer": "Reviewer_LDTU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- **[Updated]** EMMA achieves compelling quantitative gains over prior methods in low-data regimes. These new results (in Table 1) fit much better with the narrative in the introduction.\n- **[Updated]** The paper presents an ablation study investigating the contributions of individual components in EMMA. This new analysis is very helpful in understanding how EMMA works. \n- **[Updated]** The introduction, methods, and results sections have greatly improved in writing. The context for low-data multimodal learning is much better motivated. \n- **[Updated]** The authors have done a good job of incorporating feedback from previous reviews. \n- EMMA tackles an important problem in real-world perception. Robots and other embodied agents in physical environments often suffer from sensory failures and occlusions. Learning joint representations of multiple modalities that are robust to modality dropouts could potentially improve the reliability of such systems.    \n- Figures 3 and 4 are particularly informative. The qualitative results in Figure 4 could be further explored in future works to study the generalization capabilities of multi-modal representations over uni-modal representations.  \n\nWeaknesses\n- Despite being motivated by agents with faulty sensors, the choice of the GoLD dataset – a static classification dataset – seems odd. For embodied agents, classifying among 5 possible objects is rarely a relevant problem formulation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**[Updated]** This paper presents EMMA, a method for learning joint representations of multiple modalities (images, text, speech) with limited data and missing modalities. EMMA integrates contrastive learning with a geometric alignment loss to learn representations that minimize distances between instances and maximize distances between classes. Experiments are conducted on the GoLD dataset, which consists of simple language-grounding classification tasks with multi-modal inputs: RGB images, depth images, text descriptions, and speech descriptions. Results indicate that EMMA substantially outperforms prior methods like contrastive-learning and supervised-contrastive-learning in low-data regimes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**[Updated]** No major changes. But nice to have:\n- Some experiments outside classification datasets. Perhaps with some object-retrieval task in an embodied simulator.", "broader_impact_concerns": "The reviewer does not see any major ethical or moral concerns regarding the method or dataset used in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "q6YuseLJLT", "reviewer": "Reviewer_5qzW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. This paper extends the contrastive loss to a multimodal setting where an arbitrary number of views and certain modalities can be unavailable. \n2. Extensive experiments show that EMMA can converge faster than prior methods and learn better representations when limited data is available.\n3. The paper is well written and easy to follow.  \n\nWeaknesses:\n\n1. The proposed method shows better results when less data is available (e.g., 369 data points). Is the training data the bottleneck for agents in a real world scenario?\n\n[Updated] The authors have addressed concerns and it will be good if they can clarify the motivation in the final revision.\n\n2. \"At 25% of the training data, SupCon struggles to learn and the converged MRR is at 0.46, while EMMA achieves an MRR of 0.73. A random model would have an average MRR score of 0.46.\" => Using 25% of the training data, SupCon performs as a random model?\n\n[Updated] According to the experimental results and analysis, it seems that objectives based on contrastive learning are not suitable for low-data settings. It will be beneficial if the authors could incorporate some baselines capable of handling low-data situations and missing modalities effectively. This would enhance the soundness of the proposed model.\n\n3. The authors claim EMMA achieves faster convergence speed (5.6 minutes) compared to SupCon (18.7 minutes). As all the models are trained offline, it is unclear to see the benefits of faster training speed, especially SupCon can achieve comparable results with a much simpler formula.\n4. \"Although both SupCon and the Geometric Alignment approaches are trying to bring target objects together (and push dissimilar ones away), there is a trade-off in SupCon where there is a normalized ranking across them. In contrast, in Geometric Alignment distances can be arbitrarily near or far.\" Can you elaborate the difference between the SupCon and triplet loss?\n\n[Updated] Thanks for the clarification.\n\n5. In figure 7, can you provide the 3D T-SNE of the ablations of geometric loss and SupCon loss?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an extended multimodal alignment (EMMA) to learn multimodal retrieval models. The authors propose a novel loss function that combines the supervised contrastive loss and modified triplet loss (geometric alignment), and extend its application to the realm of multimodal scenarios. The conducted experiments show that EMMA converges faster, achieves better results under constrained data conditions, and comparable or slightly improved results when more data is accessible.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses above, especially justifying why training speed and limited data matter with concrete problem settings.", "broader_impact_concerns": "The reviewer did not see any major ethical concerns in this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is both an audience for this work and the results are sufficient to demonstrate the claims of the paper.  

Most of the remaining concerns revolve around issues of presentation -- which have been acknowledged by the authors in the response -- and can be easily addressed.  If the authors choose to expand their evaluation it will strengthen the final paper but is not necessary for acceptance.
Decision: Accept (poster)