Transformer-based Cross-Modal Recipe Embeddings
with Large Batch Training
Jing Yang1[0000−0002−2103−6825] , Junwen Chen1[0000−0003−1355−4128], and Keiji Yanai1[0000−0002−0431−183X]
The University of Electro-Communications, Tokyo, Japan{yang-j,chen-j,yanai}@mm.inf.uec.ac.jp

Abstract. Cross-modal recipe retrieval aims to exploit the relationships and accomplish mutual retrieval between recipe images and texts, which is clear forhuman but arduous to formulate. Although many previous works endeavored tosolve this problem, most works did not efficiently exploit the cross-modal information among recipe data. In this paper, we present a frustratingly straightforward cross-modal recipe retrieval framework, Transformer-based Network forLarge Batch Training (TNLBT) achieving high performance on both recipe retrieval and image generation tasks, which is designed to efficiently exploit therich cross-modal information. In our proposed framework, Transformer-based encoders are applied for both image and text encoding for cross-modal embedding
learning. We also adopt several loss functions like self-supervised learning loss
on recipe text to encourage the model to further promote the cross-modal embedding learning. Since contrastive learning could benefit from a larger batch size
according to the recent literature on self-supervised learning, we adopt a large
batch size during training and have validated its effectiveness. The experimental
results showed that TNLBT significantly outperformed the current state-of-theart frameworks in both cross-modal recipe retrieval and image generation tasks
on the benchmark Recipe1M by a huge margin. We also found that CLIP-ViT
performs better than ViT-B as the image encoder backbone. This is the first work
which confirmed the effectiveness of large batch training on cross-modal recipe
embedding learning.
Keywords: cross-modal recipe retrieval, transformer, vision transformer, image
generation

1

INTRODUCTION

Cross-modal recipe retrieval investigates the relation between recipe texts and recipe
images to enable mutual retrieval between them. Recipe1M dataset [20, 16] is frequently used as a benchmark to evaluate the performance of cross-modal recipe retrieval
frameworks. One challenge of the recipe retrieval task is that food images usually contain many non-food parts like plates and different backgrounds as noise. The recipe
texts are perplexing, making them difficult to encode, since there are three components
in the recipe text: title, ingredients, and instructions, the last two of which are long andstructured texts. Most existing works focus just on recipe or image embedding learning,

2

Y. Jing, J. Chen and K. Yanai

though we believe that a framework that focuses on both recipe and image embedding
learning is necessary.
As one of the early works Salvador et al. [20] proposed Joint Embedding (JE), projecting embeddings from both modalities into a common shared space and minimizing
the cosine similarity between them to enable cross-modal recipe retrieval. GAN [7] wasintroduced in some recent works [27, 24, 22] for food image synthesis from embeddings
to obtain more reliable cross-modal embeddings. By confirming if the same or similar
embeddings can be extracted from the food images generated from cross-modal embeddings, the retrieval performance were improved. Furthermore, modality adversarial
loss was introduced in ACME [24] which can narrow the modality gap between image
and text embedding. Inspired by the success of GAN-based image synthesis in recipe
retrieval, we also adopt GAN-based image synthesis in this work. However, regarding recipe text embedding, LSTM [8] was simply applied for encoding recipe texts inthese works [27, 24, 22], which sometimes limits leveraging the information in long andstructured recipe texts. In order to address this issue, some other methods [6, 26, 19] focused on improving recipe embedding learning. Authors of MCEN [6] applied attention
mechanism, and Zan et al. [26] applied BERT [4] for recipe encoding. With the trendof Transformer [23] in natural language processing, X-MRS [9] and H-T [19] were recently proposed to adopt this technique in recipe retrieval. Authors of H-T in particular
introduced a simple but effective framework with a Transformer-based structure recipe
encoder and self-supervised learning, allowing the model to explore complementary information among recipe texts. Inspired by H-T, we adopt a Transformer-based recipe
encoder and self-supervised learning for recipe embedding learning.
In this paper, we propose a frustratingly straightforward Transformer-based framework that uses all the current state-of-the-art techniques. That is, we use a hierarchicaltransformer architecture recipe encoder for recipe embedding learning, and an adversarial network to investigate the complementary information between recipe and image
embedding. In the experiments, we adopt ViT [5] and CLIP-ViT [18] as the image
encoder backbone to validate the effectiveness the proposed framework. Since contrastive learning benefits from larger batch sizes according to the recent work on selfsupervised learning [2] and self-supervised learning and triplet losses are used in the
proposed framework, we adopt large batch training in our experiments. Furthermore,
we conducted extensive experiments and ablation studies to further validate the effectiveness of our proposed framework. We discovered that large batch training was surprisingly effective for Transformer-based cross-modal recipe embedding. The results
showed that our proposed framework outperformed the state-of-the-arts both in recipe
retrieval (medR 1.0, R1 56.5 in 10k test set size) and image generation tasks (FID score16.5) by a large margin. More specifically, we summarize the contributions of this work
as follows:
1. We proposed a frustratingly straightforward Transformer-based cross-modal recipe
retrieval framework, Transformer-based Network for Large Batch Training (TNLBT),
which achieves the state-of-the-art performance on both recipe retrieval and image
generation tasks.
2. We conducted experiments with large batch inspired by the effectiveness of large
batch training on contrastive learning [2].

Transformer-based Cross-Modal Recipe Embeddings with Large Batch Training3

3. Through the comprehensive experiments, we confirmed that the proposed framework outperforms the current state-of-the-arts with a large margin, especially in thecase of a large batch size, 768. This is the first work which confirmed the effectiveness of large batch training on cross-modal recipe retrieval as far as we know.

2

RELATED WORK

2.1

Cross-Modal Recipe Retrieval

Cross-modal retrieval aims to enable mutual retrieval between two different modalities,
often image and text. The common idea of cross-modal retrieval is to embed featuresfrom two different modalities into a shared common space while keeping the distribution of corresponding embedded features close to enable mutual retrieval. A substantial
issue in this task is how to narrow the gap between the various modalities [11].
Salvador et al., who was the first to propose the cross-modal recipe retrieval task
and the Recipe1M dataset, proposed joint embedding [20] to enable cross-modal recipe
retrieval. This method was modified in AdaMine [1] by using a triplet loss [21] to improve the retrieval accuracy. In order to further exploit the information in the Recipe1M
dataset, some state-of-the-art techniques like Transformer [23] and Generative Adversarial Networks (GAN) [7] are used in recipe retrieval. GAN, enables image generation
conditioned on recipe embedding in recipe retrieval framework. GAN was introduced
in several previous works [27, 24, 22], which improves retrieval accuracy while enabling image generation conditioned on recipe text. The issues of these previous works
are, the retrieval accuracy and quality of generated images are very limited since they
only adopted simple LSTM [8] for text embedding learning. However, in our proposed
method we obtain generated images of better quality using Transformer.
Rather than incorporating complex networks like GAN, some research [6, 26, 19]
focus on recipe embedding learning to enhance the performance of retrieval tasks. Authors of MCEN[6] introduced cross-modal attention and consistency and Zan et al. [26]introduced BERT [4] as a recipe encoder to enable cross-modal retrieval. Authors of XMRS [9] introduced a Transformer [23] encoder to gain recipe embedding, further proposed the use of imperfect multilingual translations, and achieved state-of-the-art performances on retrieval tasks. Salvador et al. proposed a simply but effective framework
H-T [19], to facilitate the power of Transformer. However, authors of these works [6,26, 19] just simply applied bi-directional triplet loss on image and recipe features without image synthesis and reconstruction, which somewhat limited the cross-modal information learning between recipe and image features. In order to address this issue,
we introduce GAN-based architecture to enhance the cross-modal embedding learning in our architecture in addition to adopting Transformer-based recipe encoders andself-supervised learning on recipe-only samples in the Recipe1M.
2.2

Food Image Synthesis

GAN [7] has been introduced and proven its effectiveness for improving recipe retrieval
performance in some recent works [27, 24, 22, 9]. Authors of R2GAN [27], ACME [24]4

Y. Jing, J. Chen and K. Yanai

Fig. 1: The architecture of proposed network TNLBT. Training process is controlled
by the four loss functions Lrec , Lret , LM A , Ltrans .

and X-MRS [9] applied text-conditioned image synthesis which generates recipe images from recipe embeddings, while authors of RDE-GAN [22] proposed to disentangle image features into dish shape features, which contained only non-recipe information, and recipe embeddings and to integrate both of them to generate recipe images.
Similar to ACME and RDE-GAN, we also leverage GAN in our proposed framework
to improve retrieval accuracy while enabling image generation conditioned on recipe
text. The biggest difference between our work and the previous works is, we applied
Transformer-based encoders to further improve the quality of generated images compared to the previous works [24, 22].

3

Method3.1

Overview

We propose a frustratingly straightforward Transformer-based framework TNLBT forcross-modal recipe retrieval which has the hierarchical Transformers for text encoding, and Vision Transformer (ViT) [5] as an image encoder. Furthermore, we propose to adopt large batch training which is confirmed to be beneficial for contrastive
training and Transformer-based networks. Figure 1 shows the architecture of the proposed framework. Similar to H-T [19], our proposed framework applies hierarchicalTransformer encoders and a triplet loss Lrec with self-supervised learning to explore
complementary information in the recipe text. ViT-B [5] or CLIP-ViT [18] is usedas the backbone of image encoder. The biggest difference between our work and HT [19] is, we introduce a more sophisticated adversarial network to leverage the learned
Transformer-based embeddings. Rather than just a bi-directional triplet loss is adopted

Transformer-based Cross-Modal Recipe Embeddings with Large Batch Training

5

in H-T for the distance learning of image and recipe embeddings, we further introduce
several loss functions to enhance the cross-modal embedding learning and enable image generation conditioned on recipe text. Inspired by the success of triplet loss [21], a
triplet loss Lret for distance learning in image-recipe retrieval is applied in the retrieval
space. In order to mitigate the modality gap problem, we use the modality alignment
loss function LM A as in ACME [24]. Furthermore, we introduce image-to-recipe andrecipe-to-image information recover, and a translation consistency loss function Ltransto keep modality-specific information. We expect that the embeddings learned in theabove process correctly retains the original information and enables image generationfrom recipe embeddings. Finally the overall loss function can be formulated as follows:
Ltotal = λ1 Lrec + λ2 LM A + λ3 Ltrans + Lret ,(1)

where λ1 , λ2 , λ3 are hyperparameters to control the loss balance.
3.2

Recipe Encoder and Self-Supervised Learning

The purpose of the recipe encoder, ER (even though we adopt several encoders here,
we note these as a whole recipe encoder ER ), is to encode recipe text, t, to recipe embeddings, R. In order to better explore and make use of the huge information amongrecipe text with components of title, ingredients, and instructions, we adopt the hierarchical Transformers as the recipe encoder similar to H-T [19]. Note that we introduced
an adversarial network for image synthesis and modality alignment better to exploit thecorrelation in image-recipe pairs, while H-T simply applied a bi-directional triplet loss
function to image-recipe pairs.
For the three components (title, ingredients, instructions) in the recipe texts, the proposed framework TNLBT encodes them separately using hierarchical Transformer encoders (Ettl , Eing , Eins ), with 2 layers and 4 attention heads, to obtain embeddings oftitle, ingredients and instructions (ettl , eing , eins ) initially. Next, we apply self-supervised
recipe loss Lrec as in H-T, on ettl , eing , eins to explore the complementary information
among them. The introduction of this loss allows us to use recipe-only image-recipe
pairs to further leverage the complementary information among the three componentsof the recipe text. Finally, we apply a merging encoder Emrg to project three components embeddings (ettl , eing , eins ) to an unified recipe embedding, R.
3.3

Image Encoder

The purpose of the image encoder EV is to encode recipe image i to image embedding
V. We adopt the base size model of Vision Transformer (ViT-B) [5] pre-trained on
ImageNet-21k [3] as the image encoder backbone. In order to better leverage the benefitof large batch training, we also adopt CLIP-ViT [18] as image encoder backbone where
the proposed framework achieved superior performance on retrieval task.
3.4

Modality Alignment Loss

For the purpose of narrowing the gap between recipe and image embeddings, which
often results in bad generalization or slow convergence, we applied modality alignment

6

Y. Jing, J. Chen and K. Yanai

loss LM A as in ACME [24]. We adopt a discriminator DM here to achieve this goal,
which aligns the distribution of recipe embeddings R = ER (t) and image embeddings
V = EV (i) during training. An adversarial loss is applied by controlling the recipe
and image embeddings aligned to each other resulting in the discriminator DM cannot
distinguish the source of the given embeddings empirically. Following ACME, we also
adopt WGAN-GP [10] here and the loss function is formulated as follows:
LM A = Ei∼p(i) [log(1 − DM (EV (i)))] + Et∼p(t) [log(1 − DM (ER (t)))] (2)3.5Retrieval Loss

Following the success of triplet loss [21] in the recent works [1, 24, 22, 19], we adopt a
triplet loss for distance learning. Here we obtain the recipe and image embeddings R =ER (t), V = EV (i). We obtain an anchor recipe embedding Ra and an anchor image
embedding Va , and obtain a negative sample with subscript n and a positive sample
with subscript p to process this distance learning. The loss function is formulated asfollows:
Lret =

X[d(Va , Rp ) − d(Va , Rn ) + α]+ +V

X[d(Ra , Vp ) − d(Ra , Vn ) + α]+ , (3)R

where margin α = 0.3, d(.) is the Euclidean distance, [x]+ = max(x, 0). In addition,
we adopt a hard sample mining strategy [12] to further facilitate the distance learning.
3.6

Translation Consistency Loss

The learned embeddings are useful for training but sometimes lost the modality-specificinformation which is meaningful and important. To alleviate this information loss, we
adopt translation consistency loss following ACME [24], which ensures the learned
embeddings preserve the original information across modalities. We accomplish thisgoal by forcing the recipe and image embeddings to recover the information in the othermodalities: recipe image generation and ingredients prediction conditioned on recipe
embedding and image embedding respectively. Hence, the total translation consistency
loss is composed of the losses for the recipe and the image as follows:
Ltrans = Ltransr + Ltransi(4)

Image Generation from Recipe Embeddings In order to preserve the modalityspecific information in recipe embeddings, we aim to recover the information in image
modality to ensure this property, and we set two losses as for two-fold goal: (1) we expect the generated image is as realistic as possible, (2) we expect the generated realistic
image matches the target recipe. To accomplish these two goals, we adopt GAN [7] togenerate images from recipe embeddings and introduce a loss Lr2i for goal (1) and aTransformer-based Cross-Modal Recipe Embeddings with Large Batch Training

7

loss Lclsr2i for goal (2) following ACME [24]. During training, the discriminator Dr2i
is used to distinguish the generated image and real image, and the generator G is used
to generate food images which is conditioned on the recipe embedding F C(ER (r)).
The loss Lr2i for ensuring the generated images realistic is formulated as follows:
Lr2i = Ei∼p(i) [log(1 − Dr2i (i))] + Et∼p(t) [log(1 − Dr2i (G(FC(ER (t)))))] (5)While ensuring the generated image is as realistic as possible, we introduce loss Lclsr2i
to ensure that the generated image matches the target recipe. During training, a classifier clsr2i is used to encourage the generator to generate a food image with the corresponding food category to recipe embedding. Lclsr2i is simply a cross-entropy loss.
Combining these two loss functions for two goals separately, the translation consistency
loss for the recipe is formulated as follows:
Ltransr = Lr2i + Lclsr2i(6)Classification and Prediction from Image Embeddings We adopt ingredients prediction and title classification on image embeddings here for ensuring the translationconsistency of images. We leverage a multi-label network here to predict the ingredients from image embeddings using a 4,102-d one-hot vector representing the existenceof 4,102 different ingredients. We denote this multi-label objective as Li2r .
The same as ensuring the generated image is with the correct food category in defining Lclsr2i , we propose to ensure the predicted ingredients are from the food with thecorrect category, which means the image embeddings can be classified into the correct food category. We use Lclsi2r here to make sure the image embeddings can beclassified into one correct food category of 1,047 recipe categories, where Lclsi2r is across-entropy loss. Combining these two loss functions, the translation consistency loss
for the image is formulated as follows:
Ltransi = Li2r + Lclsi2r

4(7)

EXPERIMENTS

In this section, we present the experiments to validate the effectiveness of our proposed
framework TNLBT both in recipe retrieval and image generation tasks, including ablation studies and comparison with the previous works. Extensive experiments are also
performed to further validate the effectiveness of TNLBT.
4.1

Implementation Details

Data set. We evaluated the performance of our proposed method on Recipe1M [20,16] following the previous works. Following the official dataset splits, we used 238,999
image-recipe pairs for training, 51,119 pairs for validation, and 51,303 pairs for testing.
Evaluation Metrics. Following previous works, we evaluated retrieval performancewith the median rank (medR) and R@{1,5,10} on two different test set sizes 1k and8

Y. Jing, J. Chen and K. Yanai

Table 1: Comparison with the existing works on retrieval performance. The performance of recipe retrieval is evaluated on the criteria of medR(↓) and R@{1,5,10}(↑).
Especially all the R@K metrics were all improved by around 10.0% in TNLBT-C compared to TNLBT-V.
1k
10k
Image-to-Recipe
Recipe-to-Image
Image-to-Recipe
Recipe-to-Image
medR R@1 R@5 R@10 medR R@1 R@5 R@10 medR R@1 R@5 R@10 medR R@1 R@5 R@10
JE [20]5.2
R2GAN [27]2MCEN [6]2
ACME [24]1
SCAN [25]1
IMHF [15]1
RDEGAN [22] 1
1
H-T [19]
X-MRS [9]124.0 51.0
39.1 71.0
48.2 75.8
51.8 80.2
54.0 81.7
53.2 80.7
59.4 81.0
60.0 87.6
64.0 88.3

65.0
81.7
83.6
87.5
88.8
87.6
87.4
92.9
92.6

5.1
2
1.9
1
1
1
1
1
1

25.0 52.0
40.6 72.6
48.4 76.1
52.8 80.2
54.9 81.9
54.1 82.4
61.2 81.0
60.3 87.6
63.9 87.6

65.0
83.3
83.7
87.6
89.0
88.2
87.2
93.2
92.6

41.9
13.9
7.2
6.7
5.9
6.2
3.5
4
3

13.5 33.5
20.3 43.3
22.9 46.8
23.7 49.3
23.4 48.2
36.0 56.1
27.9 56.4
32.9 60.6

44.9
54.4
57.9
60.6
58.4
64.4
68.1
71.2

39.2
12.6
6.6
6
5.1
5.8
3
4
3

14.2 35.0
21.4 44.3
24.4 47.9
25.3 50.6
24.9 48.3
38.2 57.7
28.3 56.5
33.0 60.4

TNLBT-V
TNLBT-C

75.1 92.3 95.3
81.0 95.2 97.4

1
1

75.2 92.5 95.4
80.3 95.2 97.4

2
1

48.0 73.7 81.5
56.5 80.7 87.1

2
1

48.5 73.7 81.5
55.9 80.1 86.8

1
1

46.8
55.2
59.0
61.6
59.4
65.8
68.1
70.7

Table 2: Comparison with the existing
works on image generation. The perfor- Table 4: Comparison on image-to-recipe
mance of image generation is evaluated on retrieval performance with different batch
sizes adopted in TNLBT. Results are rethe criteria of FID(↓).
ported on rankings of size 10k.
MethodFID
ACME [24]CHEF [17]X-MRS [9]

30.7
23.0
28.6

TNLBT-V
TNLBT-C

17.9
16.5

TNLBT-V
TNLBT-C#batch medR R@1 R@5 R@10 medR R@1 R@5 R@10

Table 3: Evaluation of the importance ofdifferent components on image-to-recipe
retrieval in TNLBT. * means training without recipe-only data.
Applied Components

medR R@1 R@5 R@10

Lret
Lret +LM A
Lret +LM A +Lrec
Lret +LM A +Lrec +Ltransi
Lret +LM A +Lrec +Ltransr
Lret +Lrec +Ltransr +Ltransi
Lret +LM A +Lrec +Ltransr +Ltransi *Lret +LM A +Lrec +Ltransr +Ltransi
TNLBT-V (large batch training)
TNLBT-C (CLIP-ViT applied)

2.0
2.0
2.0
2.0
2.0
2.0
2.0
2.0
2.0
1.0

43.4 70.3
41.1 68.0
43.2 70.0
43.5 70.3
44.5 71.3
43.5 70.7
43.1 69.8
44.3 70.9
48.0 73.7
56.5 80.7

79.4
77.3
79.0
79.0
80.2
79.5
78.7
79.7
81.5
87.1

64
128
256
512
768
1024

3.0
2.0
2.0
2.0
2.0
2.0

36.6 64.3
40.9 68.3
44.3 70.9
47.1 73.4
48.0 73.7
47.5 73.3

74.3
77.6
79.7
81.6
81.5
81.2

2.0
1.4
1.0
1.0
1.0
1.0

48.0 75.4
50.1 77.1
53.5 79.1
55.9 80.4
56.5 80.7
56.0 79.8

83.9
84.9
86.3
86.9
87.1
86.5

Table 5: Evaluation on image generationin different batch sizes on the criteria of
FID(↓).
#batch
64
128
256
512
768

FID (TNLBT-V)
17.9
19.3
27.1
34.4
46.4

FID (TNLBT-C)
16.5
21.9
30.1
48.6
69.7

10k. We reported the average metrics of 10 groups which are randomly chosen fromthe test set. We also evaluated the performance of image generation conditioned onrecipe embeddings using Fréchet Inception Distance (FID) [13] score following previ-

Transformer-based Cross-Modal Recipe Embeddings with Large Batch Training

9

Fig. 2: Generated images conditioned on different recipe embeddings. The first row arerecipes used to generate images, the second row are the ground-truth images which
recipe embeddings aim to recover. The third row are the generated images.
ous works [22, 9] measuring the similarity of the distribution between generated images
and ground-truth images.
Training Details. The objective is to minimize the total loss function Eq. 1. We empirically decided the hyperparameters in Eq. 1 as follows: λ1 = 0.05, λ2 = 0.005, λ3 =0.002. For ViT-B [5] image encoder backbone framework, We trained the model for 50
epochs using Adam [14] with a learning rate of 10−4 . For CLIP-ViT [18] image encoder
backbone framework, we first trained the model for 20 epochs while freezing the image
encoder with a learning rate of 10−4 and then we trained the model for another 100
epochs with a learning rate of 10−6 where the image encoder were not frozen. Since
contrastive learning benefits from larger batch sizes and more training steps comparedto supervised learning [2], batch size was set to 768 during training.
4.2

Cross-Modal Recipe Retrieval

We evaluated and compared the performance of the proposed framework TNLBT withthe previous works in Table 1. TNLBT with ViT-B [5] image encoder backbone (TNLBTV) outperforms all the existing works across all metrics with large margins. We also
tested the performance of our proposed framework with CLIP-ViT [18] image encoder
backbone (TNLBT-C), which outperforms all the existing works across metrics witheven much larger margins. When the image encoder backbone was changed from ViTB to CLIP-ViT, all the R@K metrics were all improved by around 10.0%, and medR on
10k test set size was improved to 1.0, which has never been achieved before.
4.3

Image Generation

The proposed model generates images of 128 × 128 resolutions conditioned on recipe
embeddings. We computed the FID [13] score to evaluate the image generation performance of our proposed method1 . The results are reported in Table 2, where TNLBT
outperformed the previous works [24, 17, 9] with a large margin on image generation2 .
1
2

We used the open source code on https://github.com/mseitzer/pytorch-fid
We borrow the FID scores of CHEF [17] and ACME [24] reported in X-MRS [9].

10

Y. Jing, J. Chen and K. Yanai

Figure 2 shows qualitative results on image generation conditioned on recipe embeddings, showing that TNLBT could generate appropriate recipe images corresponding to
the given ingredients in recipes.
4.4

Ablation Studies

We also performed ablation studies to evaluate the importance of the applied components in Table 3. When LM A was applied, the accuracy decreased incrementally
while with more loss functions applied the accuracy increased again. However, when
only LM A was taken, the retrieval accuracy decreased compared to the combinationof all loss functions. Hence we empirically think there is a complementary relationship among LM A and the other loss terms except Lret , contributing to the retrieval
performance. The loss combination without Ltransi applied outperforms all the othercombinations by an incremental margin. Finally, when the proposed large batch training
strategy and CLIP-ViT [18] applied, the retrieval accuracy was further improved.
4.5

Batch Size in Cross-Modal Embedding Learning

Since contrastive learning benefits from large batch size was reported in [2], we performed extensive experiments here to validate the influence of batch size on both recipe
retrieval and image generation, where we are the first to validate this influence.
Recipe Retrieval. Table 4 shows the image-to-recipe retrieval performance of TNLBTwith different batch sizes. Retrieval performance was improved substantially when thebatch size increased from 64 to 512, indicating the importance of adopting large batch
during training. The improvement by increasing batch size is limited afterwards and theperformance was hurt incrementally when batch size increased to 1024.
Image Generation. We also investigated the influence of batch size on image generation performance in Table 5, where the performance of image generation hurt withbatch size increasing, while the performance of recipe retrieval improved according toTable 4. Hence, we emperically believe that batch size serves as a trade-off here.

5

CONCLUSIONSIn this research, we proposed a frustratingly straightforward Transformer-based Network for Large Batch Training (TNLBT) using the Hierarchical Transformer-based
recipe encoder, the Vision Transformer-based image encoder, and a sophisticated adversarial network for cross-modal recipe embedding learning. We further adopted selfsupervised learning to investigate the complementary information in recipe texts. Throughthe experiments, it was confirmed that TNLBT outperformed the state-of-the-arts on
both cross-modal recipe retrieval and food image generation tasks by large margins.
We also found that CLIP-ViT [18] achieved much better performance than ViT-B [5]as an image encoder backbone in TNLBT. With the extensive experiments, we foundthat the retrieval performance could benefit from a large batch size while the performance of image generation conditioned on recipe embeddings sometimes got hurt by alarge batch size, where we were the first to validate the influence of batch size in recipe
retrieval and image generation tasks.

Transformer-based Cross-Modal Recipe Embeddings with Large Batch Training

11

References
1. Carvalho, M., Cadène, R., Picard, D., Soulier, L., Thome, N., Cord, M.: Cross-modal retrieval in the cooking context: Learning semantic text-image embeddings. In: Proc.of ACM
SIGIR Conference on Research and Development in Information Retrieval. pp. 35–44 (2018)
2. Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learningof visual representations. In: Proc. of International Conference on Machine Learning (2020)
3. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A large-scale hierarchical image database. In: Proc.of IEEE Computer Vision and Pattern Recognition (2009)
4. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectionaltransformers for language understanding. In: Proc. of the Conference of the North American
Chapter of the Association for Computational Linguistics. pp. 4171–4186 (2019)
5. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image isworth 16x16 words: Transformers for image recognition at scale. arXiv:2010.11929 (2020)
6. Fu, H., Wu, R., Liu, C., Sun, J.: MCEN: Bridging cross-modal gap between cooking recipes
and dish images with latent variable model. In: Proc.of IEEE Computer Vision and Pattern
Recognition (2020)
7. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S.,
Courville, A.C., Bengio, Y.: Generative adversarial networks. Communication of ACM
63(11), 139–144 (2020)
8. Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional LSTM
and other neural network architectures. Neural networks 18(5-6), 602–610 (2005)
9. Guerrero, R., Xuan, H.P., Vladimir, P.: Cross-modal retrieval and synthesis (x-mrs): Closing
the modality gap in shared representation learning. In: Proc.of ACM International Conference Multimedia (2021)
10. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training ofwasserstein gans. In: Advances in Neural Information Processing Systems. pp. 5767–5777(2017)
11. Guo, W., Wang, J., Wang, S.: Deep multimodal representation learning: A survey. IEEE
Access 7, 63373–63394 (2019)12. Hermans, A., Beyer, L., Leibe, B.: In defense of the triplet loss for person re-identification.
arXiv preprint arXiv:1703.07737 (2017)
13. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a
two time-scale update rule converge to a local nash equilibrium. In: Proceedings of the 31st
International Conference on Neural Information Processing Systems. p. 6629–6640 (2017)
14. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Proc. of International
Conference on Learning Representation (2015)
15. Li, J., Sun, J., Xu, X., Yu, W., Shen, F.: Cross-modal image-recipe retrieval via intra- andinter-modality hybrid fusion. In: Proc. of ACM International Conference on Multimedia Retrieval. p. 173–182 (2021). https://doi.org/10.1145/3460426.3463618
16. Marin, J., Biswas, A., Ofli, F., Hynes, N., Salvador, A., Aytar, Y., Weber, I., Torralba, A.:
Recipe1m+: A dataset for learning cross-modal embeddings for cooking recipes and foodimages. IEEE Transactions on Pattern Analysis and Machine Intelligence 43(1), 187–203(2019)
17. Pham, H.X., Guerrero, R., Pavlovic, V., Li, J.: CHEF: Cross-modal hierarchical embeddings
for food domain retrieval. In: Proc. of the AAAI Conference on Artificial Intelligence. pp.
2423–2430 (2021)
18. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A.,
Mishkin, P., Clark, J., Krueger, G., Sutskever, I.: Learning transferable visual models from12

Y. Jing, J. Chen and K. Yanai

natural language supervision. In: Proc. of International Conference on Machine Learning.
vol. 139, pp. 8748–8763 (2021)
19. Salvador, A., Gundogdu, E., Bazzani, L., Donoser, M.: Revamping cross-modal recipe retrieval with hierarchical transformers and self-supervised learning. In: Proc.of IEEE Computer Vision and Pattern Recognition (2021)
20. Salvador, A., Hynes, N., Aytar, Y., Marin, J., Ofli, F., Weber, I., Torralba, A.: Learning crossmodal embeddings for cooking recipes and food images. In: Proc.of IEEE Computer Vision
and Pattern Recognition (2017)21. Schroff, F., Kalenichenko, D., Philbin, J.: FaceNet: A unified embedding for face recognitionand clustering. In: Proc.of IEEE Computer Vision and Pattern Recognition (2015)22. Sugiyama, Y., Yanai, K.: Cross-modal recipe embeddings by disentangling recipe contents
and dish styles. In: Proc.of ACM International Conference Multimedia (2021)
23. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł.,
Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing
Systems. pp. 5998–6008 (2017)
24. Wang, H., Sahoo, D., Liu, C., Lim, E.p., Hoi, S.C.: Learning cross-modal embeddings withadversarial networks for cooking recipes and food images. In: Proc.of IEEE Computer Vision
and Pattern Recognition. pp. 11572–11581 (2019)
25. Wang, H., Sahoo, D., Liu, C., Shu, K., Achananuparp, P., Lim, E., Hoi, S.C.H.: Cross-modal
food retrieval: Learning a joint embedding of food images and recipes with semantic consistency and attention mechanism. arXiv:2003.03955 (2020)
26. Zan, Z., Li, L., Liu, J., Zhou, D.: Sentence-based and noise-robust cross-modal retrieval on
cooking recipes and food images. In: Proc. of the International Conference on Multimedia
Retrieval. p. 117–125 (2020)
27. Zhu, B., Ngo, C.W., Chen, J., Hao, Y.: R2GAN: Cross-modal recipe retrieval with generative
adversarial network. In: Proc.of IEEE Computer Vision and Pattern Recognition (2019)