Title: VSE++: IMPROVING VISUAL-SEMANTIC EMBEDDINGS WITH HARD NEGATIVES

Abstract: We present a new technique for learning visual-semantic embeddings for crossmodal retrieval. Inspired by the use of hard negatives in structured prediction, and ranking loss functions used in retrieval, we introduce a simple change to common loss functions used to learn multi-modal embeddings. That, combined with fine-tuning and the use of augmented data, yields significant gains in retrieval performance. We showcase our approach, dubbed VSE++, on the MS-COCO and Flickr30K datasets, using ablation studies and comparisons with existing methods. On MS-COCO our approach outperforms state-of-the-art methods by 8.8% in caption retrieval, and 11.3% in image retrieval (based on R@1).

1. INTRODUCTION:

Joint embeddings enable a wide range of tasks in image, video and language understanding. Examples include shape-image embeddings (Li et al. (2015b) ) for shape inference, bilingual word embeddings (Zou et al. (2013) ), human pose-image embeddings for 3D pose inference (Li et al. (2015a) ), fine-grained recognition (Reed et al. (2016a) ), zero-shot learning (Frome et al. (2013) ), and modality conversion via synthesis (Reed et al. (2016b; a) ). Such embeddings entail mappings from two (or more) domains into a common vector space in which semantically associated inputs (e.g., text and images) are mapped to similar locations. The embedding space thus represents the underlying structure of the domains, where locations and often direction are semantically meaningful.

In this paper we focus on learning visual-semantic embeddings, central to tasks such as imagecaption retrieval and generation Kiros et al. (2014) ; Karpathy & Fei-Fei (2015) , and visual questionanswering Malinowski et al. (2015) . One approach to visual question-answering, for example, is to first describe an image by a set of captions, and then to find the nearest caption in response to a question (Agrawal et al. (2017) ; Zitnick et al. ( 2016)). In the case of image synthesis from text, one approach is to invert the mapping from a joint visual-semantic embedding to the image space (Reed et al. (2016b; a) ).

Here we focus on visual-semantic embeddings for the generic task of cross-modal retrieval; i.e. the retrieval of images given captions, or of captions from a query image. As is common in information retrieval, we measure performance by R@K, i.e., recall at K -the fraction of queries for which the correct item is retrieved in the closest K points to the query in the embedding space (K is usually a small integer, often 1). More generally, retrieval is a natural way to assess the quality of joint embeddings for image and language data for use in subsequent tasks (Hodosh et al. (2013) ).

To this end, the problem is one of ranking, for which the correct target(s) should be closer to the query than other items in the corpus, not unlike learning to rank problems (e.g., Li (2014) ), and max-margin structured prediction Chapelle et al. (2007) ; Le & Smola (2007) . The formulation and model architecture in this paper are most closely related to those of Kiros et al. (2014) , learned with a triplet ranking loss. In contrast to that work, we advocate a novel loss, the use of augmented data, and fine-tuning, that together produce a significant increase in caption retrieval performance over the baseline ranking loss on well-known benchmark datasets. We outperform the best reported result on MS-COCO by almost 9%. We also demonstrate that the benefit from a more powerful image encoder, and fine-tuning the image encoder, is amplified with the use of our stronger loss function. To ensure reproducibility, our code will be made publicly available. We refer to our model as VSE++.

Finally, we note that our formulation complements other recent articles that propose new model architectures or similarity functions for this problem. Wang et al. (2017) propose an embedding network to fully replace the similarity function used for the ranking loss. An attention mechanism on both image and caption is used by Nam et al. (2016) , where the authors sequentially and selectively focus on a subset of words and image regions to compute the similarity. In Huang et al. (2016) , the authors use a multi-modal context-modulated attention mechanism to compute the similarity between an image and a caption. Our proposed loss function and triplet sampling could be extended and applied to other such approaches.

2. LEARNING VISUAL-SEMANTIC EMBEDDINGS:



2.1. IMAGE-CAPTION RETRIEVAL:

For image-caption retrieval the query is a caption and the task is to retrieve the most relevant image(s) from a database. Or the query may be an image and one retrieves relevant captions. The goal is to maximize recall at K (R@K), the fraction of queries for which the most relevant item is ranked among the top K items returned.

Let S = {(i n , c n )} N

n=1 be a training set of image-caption pairs. We refer to (i n , c n ) as positive pairs and (i n , c m =n ) as negative pairs; i.e., the most relevant caption to the image i n is c n and for caption c n , it is the image i n . We define a similarity function s(i, c) ∈ R that should, ideally, give higher similarity scores to positive pairs than negatives. In caption retrieval, the query is an image and we rank a database of captions based on the similarity function; i.e., R@K is the percentage of queries for which the positive caption is ranked among the top K captions using s(i, c). And likewise for image retrieval. In what follows the similarity function is defined on the joint embedding space. This approach differs from others, such as Wang et al. (2017) , which use a similarity network to directly classify an image-caption pair as matching or non-matching. Similarly, let ψ(c; θ ψ ) ∈ R D ψ be a representation of a caption c in a caption embedding space (e.g. a GRU-based text encoder). Here, θ φ and θ ψ denote the model parameters used for the respective mappings to obtain the initial image and caption representations.

The mappings into the joint embedding space are then defined in terms of linear projections; i.e.,

EQUATION

EQUATION

where W f ∈ R D φ ×D and W g ∈ R D ψ ×D . We further normalize f (i; W f , θ φ ), and g(c; W g , θ ψ ), to lie on the unit hypersphere. The similarity function in the embedding space is then defined as an inner product:

EQUATION

Let θ = {W f , W g , θ ψ } be the model parameters. If we also fine-tune the image encoder, then we would also include θ φ in θ.

Training entails the minimization of empirical loss with respect to θ, i.e., the cumulative loss over training data

EQUATION

where Figure 1 : An illustration of typical positive pairs and the nearest negative samples. Here assume similarity score is the negative distance. Filled circles show a positive pair (i, c), while empty circles are negative samples for the query i. The dashed circles on the two sides are drawn at the same radii. Notice that the hardest negative sample c is closer to i in (a). Assuming a zero margin, (b) has a higher loss with the SH loss compared to (a). The MH loss assigns a higher loss to (a). its use in image retrieval (Frome et 

(i n , c n ) is a suitable

EQUATION

where [x] + ≡ max(x, 0). This hinge loss comprises two symmetric terms, with i and c being queries. The first sum is taken over all negative captions ĉ given query i. The second negative images î given caption c. Each term is proportional to the expected loss (or violation) over sets of negative samples. If i and c are closer to one another in the joint embedding space than to any negatives pairs, by the margin α, the hinge loss is zero. In practice, for computational efficiency, rather than summing over all possible negatives in the training set, it is common to only sum over (or randomly sample) the negatives within a mini-batch of stochastic gradient descent (e. Of course there are other loss functions that one might consider. One approach is a pairwise hinge loss in which elements of positive pairs are encouraged to be within a radius ρ 1 in the joint embedding space, while negative pairs should be no closer than ρ 2 > ρ 1 . This is problematic as it constrains the structure of the latent space more than does the ranking loss, and it entails the use of two hyper-parameters which can be very difficult to set. Another possible approach is to use Canonical Correlation Analysis to learn W f and W g , thereby trying to preserve correlation between the text and images in the joint embedding (e.g., Klein et al. (2015) ; Eisenschtat & Wolf (2016)). By comparison, when measuring performance as R@K, for small K, a correlation-based loss will not give sufficient influence to the embedding of negative items in the local vicinity of positive pairs, which is critical for R@K. 2010)), we focus on hard negatives for training, i.e., the negatives closest to each training query. This is particularly relevant for retrieval since it is the hardest negative that determines success or failure as measured by R@1.

2.3. EMPHASIS:

Given a positive pair (i, c), the hardest negatives are given by i = arg max j =i s(j, c) and c = arg max d =c s(i, d). To emphasize hard negatives we therefore define our loss as

EQUATION

Like Eq. 5, the loss comprises two terms, one with i and one with c as queries. Unlike Eq. 5, this loss is specified in terms of the hardest negatives, c and i . Hereafter, we refer to the loss in Eq. 6 as Max of Hinges (MH) loss, and the loss function in Eq. 5 as Sum of Hinges (SH) loss.

An example of where the MH loss is superior to SH is when multiple negatives with relatively small violations combine to dominate the SH loss. For example, in Fig. 1 , a positive pair is depicted together with two sets of negative samples. In Fig. 1 (a), there exists a single negative sample that is too close to the query. Essentially, moving such a hard negative, might require a significant change to the mapping. However, any training step that pushes the hard negative away, can bring back many small violating negative samples, as in Fig. 1(b ). Using the SH loss, these 'new' negative samples may dominate the loss, so the model is pushed back to the first example in Fig. 1 (a). This may create local minima in the SH loss that may not be as problematic for the MH loss as it focuses solely on the hardest negative.

For computational efficiency, instead of finding the hardest negatives in the whole training set, we find them in a mini-batch. With random sampling of the mini-batches, this approximate yields other advantages. One is that there is a high probability of getting hard negatives that are harder than at least 90% of the entire training set. Moreover, the loss is potentially robust to label errors in the training data because the probability of sampling the hardest negative over the entire training set is somewhat low. In Appendix A, we analyze the probability of sampling hard negatives further.

3. EXPERIMENTS:

We first perform experiments with our approach, VSE++, and compare it to a baseline formulation with SH loss, referred to as VSE0, and other state-of-the-art approaches. Essentially, the baseline formulation, VSE0, is the same used by Kiros et with 10 crops as 10C. We also consider using random crops, denoted by RC. For RC, we have the full VGG19 model and extract features over a single randomly chosen cropped patch on the fly as opposed to pre-computing the image features once and reusing them.

For the caption encoder, we use a GRU similar to the one used in Kiros et al. (2014) . We set the dimensionality of the GRU, D ψ , and the joint embedding space, D, to 1024. The dimensionality of the word embeddings that are input to the GRU is set to 300.

We further note that in Kiros et al. (2014) , the caption embedding is normalized, while the image embedding is not. Normalization of both vectors means that the similarity function is cosine similarity.

In VSE++ we normalize both vectors. Not normalizing the image embedding changes the importance of samples. In our experiments, not normalizing the image embedding helped the baseline, VSE0, to find a better solution. However, VSE++ is not significantly affected by this normalization.

3.1. DATASETS:

We evaluate our method on the Microsoft COCO dataset (Lin et The effect of data augmentation and fine-tuning. We copy the relevant results for VSE++ from Table 1 to enable an easier comparison. Notice that applying all the modifications with the exception the VSE0 model reaches 56.0% for R@1, while VSE++ achieves 64.6%.

epochs with a learning rate of 0.00002. We set the margin to 0.2 for most of the experiments. We use a mini-batch size of 128 in all our experiments. Notice that since the size of the training set for different models is different, the actual number of iterations in each epoch can vary. For evaluation on the test set, we tackle over-fitting by choosing the snapshot of the model that performs best on the validation set. The best snapshot is selected based on the sum of the recalls on the validation set.

3.3. RESULTS ON MS-COCO:

The results on the MS-COCO dataset are presented in Table 1 . To understand the effect of training and algorithmic variations we report ablation studies for the baseline VSE0 (see Table 2 ). Our best result with VSE++ is achieved by using ResNet152 and fine-tuning the image encoder (row 1.11), where we see 21.2% improvement in R@1 for caption retrieval and 21% improvement in R@1 for image retrieval compared to UVS (rows 1.1 and 1.11). Notice that using ResNet152 and fine-tuning can only lead to 12.6% improvement using the VSE0 formulation (rows 2.6 and 1.1), while our MH loss function brings a significant gain of 8.6% (rows 1.11 and 2.6).

Comparing VSE++ (ResNet152, fine-tuned) to the current state-of-the-art on MS-COCO, 2WayNet (row 1.11 and row 1.5), we see 8.8% improvement in R@1 for caption retrieval and compared to sm-LSTM (row 1.11 and row 1.4), 11.3% improvement in image retrieval. We also report results on the full 5K test set of MS-COCO in rows 1.13 and 1.14.

Effect of the training set. We compare VSE0 and VSE++ by incrementally improving the training data. Comparing the models trained on 1C (rows 1.1 and 1.6), we only see 2.7% improvement in R@1 for image retrieval but no improvement in caption retrieval performance. However, when we train using RC (rows improvement of 5.9% and 5.1%, respectively, in R@1 for caption retrieval compared to VSE0. This shows that VSE++ can better exploit the additional data.

Effect of a better image encoding. We also investigate the effect of a better image encoder on the models. Row 1.9 and row 2.4 show the effect of fine-tuning the VGG19 image encoder. We see that the gap between VSE0 and VSE++ increases to 6.1%. If we use ResNet152 instead of VGG19 (row 1.10 and row 2.5), the gap is 5.6%. As for our best result, if we use ResNet152 and also fine-tune the image encoder (row 1.11 and row 2.6) the gap becomes 8.6%. The increase in the performance gap shows that the improved loss of VSE++ can better guide the optimization when a more powerful image encoder is used.

3.4. RESULTS ON FLICKR30K:

Tables 3 summarizes the performance on Flickr30K. We obtain 23.1% improvement in R@1 for caption retrieval and 17.6% improvement in R@1 for image retrieval (rows 3.1 and 3.14). We observed that VSE++ over-fits when trained with the pre-computed features of 1C. The reason is potentially the limited size of the Flickr30K training set. As explained in Sec. 3.2, we select a snapshot of the model before over-fitting occurs, based on performance with the validation set. Over-fitting does not occur when the model is trained using the RC training data. Our results show the improvements incurred by our MH loss persist across datasets, as well as across models.

3.5. BEHAVIOR OF LOSS FUNCTIONS:

We have observed that the MH loss can take a few epochs to 'warm-up' during training. Fig. 2(a) depicts such behavior on the Flickr30K dataset using RC. One can see that the SH loss starts off faster, but after approximately 5 epochs MH loss surpasses SH loss. To explain this, the MH loss depends on a smaller set of triplets compared to the SH loss. At the beginning of the training, there is so much that the model has to learn. However, the gradient of the MH loss, may only be influenced by a small set of triples. As such, it can take longer to train a model with the MH loss. We explored a simple form of curriculum learning (Bengio et al. (2009) ) to speed-up the training. We start training with the SH loss for a few epochs, then switch to the MH loss for the rest of the training. However, it did not perform better than training solely with the MH loss.

3.6. EFFECT OF NEGATIVE SET SIZE ON MH LOSS:

In practice, our MH loss searches for the hardest negative only within each mini-batch at each iteration. To explore the impact of this approximation we examined how performance depends on the effective sample size over which we searched for negatives (while keeping the mini-batch size fixed at 128). In the extreme case, when the negative set is the training set, we get the hardest negatives in the entire training set. As discussed in Sec. 2.3, sampling a negative set smaller than the training set can potentially be more robust to label errors. 3 , row 3.9 and row 3.11). Notice that, in the first 5 epochs the SH loss achieves a better performance, however, from there-on the MH loss leads to much higher recall rates. shows the effect of the negative sample size on the MH Loss function. We compare the caption retrieval performance for different negative set sizes varied from 2 to 512. In practice, for negative set sizes smaller than the mini-batch size, 128, we randomly sample the negative set from the mini-batch. In other cases where the mini-batch size is smaller than the negative set, we randomly sample the mini-batch from the negative set. We observe that on this dataset, the optimal negative set size is around 128. Interestingly, for negative sets as small as 2, R@1 is slightly below VSE0. To understand this, note that the SH loss is still over a large sample size which has a relatively high probability of containing hard negatives. For large negative sets, the model takes longer to train for the first epochs. Using the negative set size 512, the performance dropped. This can be due to the small size of the dataset and the increase in the probability of sampling the hardest negative and outliers. Even though the performance drops with larger mini-batch sizes, it still performs better than the SH loss.

3.7. IMPROVING ORDER EMBEDDINGS:

Given the simplicity of our approach, our proposed loss function can complement the recent approaches that use more sophisticated model architectures or similarity functions. Here we demonstrate the benefits of the MH loss by applying it to another approach to joint embeddings called order-embeddings Vendrov et al. (2015) . The main difference with the formulation above is the use of an asymmetric similarity function, i.e., s(i, c) = -max(0, g(c; W g , θ ψ ) -f (i; W f , θ φ )) 2 . Again, we simply replace their use of the SH loss by our MH loss. VSE++: [3] Two young men are outside skateboarding together.

GT: a brown cake with white icing and some walnut toppings VSE0: [6] A large slice of angel food cake sitting on top of a plate.

VSE++: [16] A baked loaf of bread is shown still in the pan.

GT: A woman holding a child and standing near a bull.

VSE0: [1] A woman holding a child and standing near a bull.

VSE++: [1] A woman holding a child looking at a cow.

GT: A woman in a short pink skirt holding a tennis racquet.

VSE0: [6] A man playing tennis and holding back his racket to hit the ball.

VSE++: [1] A woman is standing while holding a tennis racket. Table 4 reports the results when the SH loss is replaced by the MH loss. We replicate their results using our Order0 formulation and get slightly better results (row 4.1 and row 4.3). We observe 4.5% improvement from Order0 to Order++ in R@1 for caption retrieval (row 4.3 and row 4.5). Compared to the improvement from VSE0 to VSE++, where the improvement on the 10C+rV training set is 1.8%, we gain an even higher improvement here. This shows that the MH loss can potentially improve numerous similar loss functions used in retrieval and ranking tasks. ). We performed experiments on the MS-COCO and Flickr30K datasets and showed that our proposed loss significntly improves performance on these datasets. We observed that the improved loss can better guide a more powerful image encoder, ResNet152, and also guide better when fine-tuning an image encoder. With all modifications, our VSE++ model achieves state-of-the-art performance on the MS-COCO dataset, and is slightly below the best recent model on the Flickr30K dataset. Our proposed loss function can be used to train more sophisticated models that have been using a similar ranking loss for training.

:



Figures and Tables:

VISUAL-SEMANTIC EMBEDDING Let φ(i; θ φ ) ∈ R D φ be a feature-based representation computed from the image (e.g. the representation before logits in VGG19 (Simonyan & Zisserman (2014)) or ResNet152 (He et al. (2016))).

loss function for a single training exemplar. Recent approaches to joint visual-semantic embeddings have used a form of triplet ranking loss (Kiros et al. (2014); Karpathy & Fei-Fei (2015); Zhu et al. (2015); Socher et al. (2014)), inspired

g., see Kiros et al. (2014); Socher et al. (2014); Karpathy & Fei-Fei (2015)).

ON HARD NEGATIVES Inspired by common loss functions used in structured prediction (Tsochantaridis et al. (2005); Yu & Joachims (2009); Felzenszwalb et al. (

Figure 2: Analysis of the behavior of the MH loss on the Flickr30K dataset training with RC. Fig. (a) compares the SH loss to the MH loss (Table3, row 3.9 and row 3.11). Notice that, in the first 5 epochs the SH loss achieves a better performance, however, from there-on the MH loss leads to much higher recall rates. Fig.(b)shows the effect of the negative set size on the R@1 performance.

Figure 2: Analysis of the behavior of the MH loss on the Flickr30K dataset training with RC. Fig. (a) compares the SH loss to the MH loss (Table3, row 3.9 and row 3.11). Notice that, in the first 5 epochs the SH loss achieves a better performance, however, from there-on the MH loss leads to much higher recall rates. Fig.(b)shows the effect of the negative set size on the R@1 performance.

Fig.2(b)shows the effect of the negative sample size on the MH Loss function. We compare the caption retrieval performance for different negative set sizes varied from 2 to 512. In practice, for negative set sizes smaller than the mini-batch size, 128, we randomly sample the negative set from the mini-batch. In other cases where the mini-batch size is smaller than the negative set, we randomly sample the mini-batch from the negative set. We observe that on this dataset, the optimal negative set size is around 128. Interestingly, for negative sets as small as 2, R@1 is slightly below VSE0. To understand this, note that the SH loss is still over a large sample size which has a relatively high probability of containing hard negatives. For large negative sets, the model takes longer to train for the first epochs. Using the negative set size 512, the performance dropped. This can be due to the small size of the dataset and the increase in the probability of sampling the hardest negative and outliers. Even though the performance drops with larger mini-batch sizes, it still performs better than the SH loss.

Two elephants are standing by the trees in the wild. VSE0: [9] Three elephants kick up dust as they walk through the flat by the bushes. VSE++: [1] A couple elephants walking by a tree after sunset. GT: A large multi layered cake with candles sticking out of it. VSE0: [1] A party decoration containing flowers, flags, and candles. VSE++: [1] A party decoration containing flowers, flags, and candles. GT: The man is walking down the street with no shirt on. VSE0: [24] A person standing on a skate board in an alley. VSE++: [10] Two young men are skateboarding on the street. GT: A row of motorcycles parked in front of a building. VSE0: [2] a parking area for motorcycles and bicycles along a street VSE++: [1] A number of motorbikes parked on an alley GT: some skateboarders doing tricks and people watching them VSE0: [39] Young skateboarder displaying skills on sidewalk near field.

Figure 3: Examples of test images and the top 1 retrieved captions for VSE0 and VSE++ (ResNet)finetune. The value in brackets is the rank of the highest ranked ground-truth caption. GT is a sample from the ground-truth captions.

This paper focused on learning visual-semantic embeddings for cross-modal, image-caption retrieval. Inspired by structured prediction, we proposed a new loss based on violations incurred by relatively hard negatives compared to current methods that used expected errors(Kiros et al. (2014);Vendrov et al. (2015)

al. (2014)) and the Flickr30K dataset(Young et al. (2014)). Flickr30K has a standard 30, 000 images for training. FollowingKarpathy & Fei-Fei (2015), we use 1000 images for validation and 1000 images for testing. We also use the splits ofKarpathy & Fei-Fei (2015) for MS-COCO. In this split, the training set contains 82, 783 images, 5000 validation and 5000 test images. However, there are also 30, 504 images that were originally in the validation set of MS-COCO but have been left out in this split. We refer to this set as rV. Some papers use rV for training (113, 287 training images in total) to further improve accuracy. We report results using both training sets. Each image comes with 5 captions. The results are reported by either averaging over 5 folds of 1K test images or testing on the full 5K test images. Results of experiments on MS-COCO.

Table content: <table><tr><td>#</td><td>Model</td><td colspan="2">Trainset</td><td/><td colspan="3">Caption Retrieval</td><td colspan="2">Image Retrieval</td></tr><tr><td/><td/><td/><td/><td colspan="4">R@1 R@10 Med r</td><td colspan="2">R@1 R@10 Med r</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="3">1K Test Images</td></tr><tr><td>1.1</td><td>UVS (Kiros et al. (2014), GitHub)</td><td colspan="3">1C (1 fold) 43.4</td><td/><td>85.8</td><td>2</td><td>31.0</td><td>79.9</td><td>3</td></tr><tr><td>1.2</td><td>Order (Vendrov et al. (2015))</td><td colspan="2">10C+rV</td><td>46.7</td><td/><td>88.9</td><td>2.0</td><td>37.9</td><td>85.9</td><td>2.0</td></tr><tr><td colspan="3">1.3 Embedding Network (Wang et al. (2017))</td><td>?</td><td>50.4</td><td/><td>69.4</td><td>-</td><td>39.8</td><td>86.6</td><td>-</td></tr><tr><td>1.4</td><td>sm-LSTM (Huang et al. (2016))</td><td/><td>?</td><td>53.2</td><td/><td>91.5</td><td>1</td><td>40.7</td><td>87.4</td><td>2</td></tr><tr><td>1.5</td><td colspan="2">2WayNet (Eisenschtat &amp; Wolf (2016))</td><td>?</td><td>55.8</td><td/><td>-</td><td>-</td><td>39.7</td><td>-</td><td>-</td></tr><tr><td>1.6</td><td>VSE++</td><td colspan="3">1C (1 fold) 43.6</td><td/><td>84.6</td><td>2.0</td><td>33.7</td><td>81.0</td><td>3.0</td></tr><tr><td>1.7</td><td>VSE++</td><td colspan="2">RC</td><td>49.0</td><td/><td>88.4</td><td>1.8</td><td>37.1</td><td>83.8</td><td>2.0</td></tr><tr><td>1.8</td><td>VSE++</td><td colspan="2">RC+rV</td><td>51.9</td><td/><td>90.4</td><td>1.0</td><td>39.5</td><td>85.6</td><td>2.0</td></tr><tr><td>1.9</td><td>VSE++ (fine-tuned)</td><td colspan="2">RC+rV</td><td>57.2</td><td/><td>93.3</td><td>1.0</td><td>45.9</td><td>89.1</td><td>2.0</td></tr><tr><td>1.10</td><td>VSE++ (ResNet152)</td><td colspan="2">RC+rV</td><td>58.3</td><td/><td>93.3</td><td>1.0</td><td>43.6</td><td>87.8</td><td>2.0</td></tr><tr><td>1.11</td><td>VSE++ (ResNet152, fine-tuned)</td><td colspan="2">RC+rV</td><td colspan="2">64.6</td><td>95.7</td><td>1.0</td><td>52.0</td><td>92.0</td><td>1.0</td></tr><tr><td/><td/><td/><td/><td/><td/><td colspan="3">5K Test Images</td></tr><tr><td>1.12</td><td>Order (Vendrov et al. (2015))</td><td colspan="2">10C+rV</td><td>23.3</td><td/><td>65.0</td><td>5.0</td><td>18.0</td><td>57.6</td><td>7.0</td></tr><tr><td>1.13</td><td>VSE++ (fine-tuned)</td><td colspan="2">RC+rV</td><td>32.9</td><td/><td>74.7</td><td>3.0</td><td>24.1</td><td>66.2</td><td>5.0</td></tr><tr><td>1.14</td><td>VSE++ (ResNet152, fine-tuned)</td><td colspan="2">RC+rV</td><td colspan="2">41.3</td><td>81.2</td><td>2.0</td><td>30.3</td><td>72.4</td><td>4.0</td></tr><tr><td>#</td><td>Model</td><td>Trainset</td><td/><td colspan="4">Caption Retrieval</td><td colspan="2">Image Retrieval</td></tr><tr><td/><td/><td/><td colspan="5">R@1 R@10 Med r</td><td colspan="2">R@1 R@10 Med r</td></tr><tr><td>2.1</td><td>VSE0</td><td colspan="3">1C (1 fold) 43.2</td><td/><td>85.0</td><td>2.0</td><td>33.0</td><td>80.7</td><td>3.0</td></tr><tr><td>1.6</td><td>VSE++</td><td colspan="3">1C (1 fold) 43.6</td><td/><td>84.6</td><td>2.0</td><td>33.7</td><td>81.0</td><td>3.0</td></tr><tr><td>2.2</td><td>VSE0</td><td>RC</td><td colspan="2">43.1</td><td/><td>87.1</td><td>2.0</td><td>32.5</td><td>82.1</td><td>3.0</td></tr><tr><td>1.7</td><td>VSE++</td><td>RC</td><td colspan="2">49.0</td><td/><td>88.4</td><td>1.8</td><td>37.1</td><td>83.8</td><td>2.0</td></tr><tr><td>2.3</td><td>VSE0</td><td>RC+rV</td><td colspan="2">46.8</td><td/><td>89.0</td><td>1.8</td><td>34.2</td><td>83.6</td><td>2.6</td></tr><tr><td>1.8</td><td>VSE++</td><td>RC+rV</td><td colspan="2">51.9</td><td/><td>90.4</td><td>1.0</td><td>39.5</td><td>85.6</td><td>2.0</td></tr><tr><td>2.4</td><td>VSE0 (fine-tuned)</td><td>RC+rV</td><td colspan="2">50.1</td><td/><td>90.5</td><td>1.6</td><td>39.7</td><td>87.2</td><td>2.0</td></tr><tr><td>1.9</td><td>VSE++ (fine-tuned)</td><td>RC+rV</td><td colspan="2">57.2</td><td/><td>93.3</td><td>1.0</td><td>45.9</td><td>89.1</td><td>2.0</td></tr><tr><td>2.5</td><td>VSE0 (ResNet152)</td><td>RC+rV</td><td colspan="2">52.7</td><td/><td>91.8</td><td>1.0</td><td>36.0</td><td>85.5</td><td>2.2</td></tr><tr><td>1.10</td><td>VSE++ (ResNet152)</td><td>RC+rV</td><td colspan="2">58.3</td><td/><td>93.3</td><td>1.0</td><td>43.6</td><td>87.8</td><td>2.0</td></tr><tr><td>2.6</td><td>VSE0 (ResNet152, fine-tuned)</td><td>RC+rV</td><td colspan="2">56.0</td><td/><td>93.5</td><td>1.0</td><td>43.7</td><td>89.7</td><td>2.0</td></tr><tr><td colspan="2">1.11 VSE++ (ResNet152, fine-tuned)</td><td>RC+rV</td><td colspan="2">64.6</td><td colspan="2">95.7</td><td>1.0</td><td>52.0</td><td>92.0</td><td>1.0</td></tr></table>



Table content: <table/>

Results on the Flickr30K dataset.

Table content: <table><tr><td>#</td><td>Model</td><td>Trainset</td><td colspan="3">Caption Retrieval</td><td colspan="3">Image Retrieval</td></tr><tr><td/><td/><td/><td colspan="3">R@1 R@10 Med r</td><td colspan="3">R@1 R@10 Med r</td></tr><tr><td>3.1</td><td>UVS (Kiros et al. (2014))</td><td>1C</td><td>23.0</td><td>62.9</td><td>5</td><td>16.8</td><td>56.5</td><td>8</td></tr><tr><td>3.2</td><td>UVS (GitHub)</td><td>1C</td><td>29.8</td><td>70.5</td><td>4</td><td>22.0</td><td>59.3</td><td>6</td></tr><tr><td>3.3</td><td>Embedding Network (Wang et al. (2017))</td><td>?</td><td>40.7</td><td>79.2</td><td>-</td><td>29.2</td><td>71.7</td><td>-</td></tr><tr><td>3.4</td><td>DAN (Nam et al. (2016))</td><td>?</td><td>41.4</td><td>82.5</td><td>2</td><td>31.8</td><td>72.5</td><td>3</td></tr><tr><td>3.5</td><td>sm-LSTM (Huang et al. (2016))</td><td>?</td><td>42.5</td><td>81.5</td><td>2</td><td>30.2</td><td>72.3</td><td>3</td></tr><tr><td>3.6</td><td>2WayNet (Eisenschtat &amp; Wolf (2016))</td><td>?</td><td>49.8</td><td>-</td><td>-</td><td>36.0</td><td>-</td><td>-</td></tr><tr><td>3.7</td><td>DAN (ResNet152) (Nam et al. (2016))</td><td>?</td><td>55.0</td><td>89.0</td><td>1</td><td>39.4</td><td>79.1</td><td>2</td></tr><tr><td>3.8</td><td>VSE0</td><td>1C</td><td>29.8</td><td>71.9</td><td>3.0</td><td>23.0</td><td>61.0</td><td>6.0</td></tr><tr><td>3.9</td><td>VSE0</td><td>RC</td><td>31.6</td><td>71.7</td><td>4.0</td><td>21.6</td><td>63.8</td><td>5.0</td></tr><tr><td>3.10</td><td>VSE++</td><td>1C</td><td>31.9</td><td>68.0</td><td>4.0</td><td>23.1</td><td>60.7</td><td>6.0</td></tr><tr><td>3.11</td><td>VSE++</td><td>RC</td><td>38.6</td><td>74.6</td><td>2.0</td><td>26.8</td><td>66.8</td><td>4.0</td></tr><tr><td>3.12</td><td>VSE++ (fine-tuned)</td><td>RC</td><td>41.3</td><td>77.9</td><td>2.0</td><td>31.4</td><td>71.2</td><td>3.0</td></tr><tr><td>3.13</td><td>VSE++ (ResNet152)</td><td>RC</td><td>43.7</td><td>82.1</td><td>2.0</td><td>32.3</td><td>72.1</td><td>3.0</td></tr><tr><td>3.14</td><td>VSE++ (ResNet152, fine-tuned)</td><td>RC</td><td>52.9</td><td>87.2</td><td>1.0</td><td>39.6</td><td>79.5</td><td>2.0</td></tr></table>

Comparison on MS-COCO. Training set for all the rows is 10C+rV.

Table content: <table><tr><td>#</td><td>Model</td><td/><td colspan="2">Caption Retrieval</td><td/><td colspan="2">Image Retrieval</td></tr><tr><td/><td/><td>R@1</td><td>R@10</td><td>Med r</td><td>R@1</td><td>R@10</td><td>Med r</td></tr><tr><td/><td/><td/><td/><td colspan="2">1K Test Images</td><td/><td/></tr><tr><td>4.1</td><td>Order (Vendrov et al. (2015))</td><td>46.7</td><td>88.9</td><td>2.0</td><td>37.9</td><td>85.9</td><td>2.0</td></tr><tr><td>4.2</td><td>VSE0</td><td>49.5</td><td>90.0</td><td>1.8</td><td>38.1</td><td>85.1</td><td>2.0</td></tr><tr><td>4.3</td><td>Order0</td><td>48.5</td><td>90.3</td><td>1.8</td><td>39.6</td><td>86.7</td><td>2.0</td></tr><tr><td>4.4</td><td>VSE++</td><td>51.3</td><td>91.0</td><td>1.2</td><td>40.1</td><td>86.1</td><td>2.0</td></tr><tr><td>4.5</td><td>Order++</td><td>53.0</td><td>91.9</td><td>1.0</td><td>42.3</td><td>88.1</td><td>2.0</td></tr></table>

Peer review (iclr2018):
 The paper describes learning joint embedding of sentences and images. The main point is in using a triplet loss that is applied to hardest-negatives, instead of averaging over all triplets. This led to improvement over SOTA in a task of caption ranking on MS-COCO, nd good performance on flickr30K. The main issue with this paper is novelty. Using hard negatives is routinely used in many embedding tasks, and has been discussed in many publications. For instance recently Wu et all in ICCV2017, bu also many other papers. When used in practice wit real-world datasets, taking the max (hardest negative) tends to be very sensitive to label noise, since the hardest negative is sometime just a positive sample with incorrect label. In these cases focusing on the hardest negative reduces performance. While it is good to know that using hard negatives improves recall measures on coco, it is not clear that this paper provides enough novel insight to be interesting enough for the ICLR audience. It may be a better fit in a conference that stresses empirical performance, like in machine vision conferences.