Learning to Taste: A Multimodal Wine Dataset
Authors: Thoranna Bender
Simon Moe Sørensen
Alireza Kashani
Kristjan Eldjarn Hjorleifsson
Grethe Hyldig
Søren Hauberg
Serge Belongie
Frederik Rahbæk Warburg
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Crowd annotations
Multi-modal
Concept embeddings

Abstract: We present WineSensed, a large multimodal wine dataset for studying the relations between visual perception, language, and flavor. The dataset encompasses 897k images of wine labels and 824k reviews of wines curated from the Vivino platform. It has over 350k unique bottlings, annotated with year, region, rating, alcohol percentage, price, and grape composition. We obtained fine-grained flavor annotations on a subset by conducting a wine-tasting experiment with 256 participants who were asked to rank wines based on their similarity in flavor, resulting in more than 5k pairwise flavor distances. We propose a low-dimensional concept embedding algorithm that combines human experience with automatic machine similarity kernels. We demonstrate that this shared concept embedding space improves upon separate embedding spaces for coarse flavor classification  (alcohol percentage, country, grape, price, rating) and representing human perception of flavor.

19 As a first step towards modeling flavor. we focus on wine since (1) wines have been studied for Learning to Taste
A Multimodal Wine Dataset
Thoranna Bender
Simon Moe Sørensen
Alireza Kashani
K. Eldjarn Hjorleifsson
Grethe Hyldig
Søren Hauberg
Serge Belongie a Frederik Warburg 4
Technical University of Denmark Vivino
California Institute of Technology
University of Copenhagen
Abstract
We present WineSensed, a large multimodal wine dataset for studying the relations between visual perception, language, and flavor. The dataset encompasses 897k im- ages of wine labels and 824k reviews of wines curated from the Vivino platform. It has over 350k unique bottlings, annotated with year, region, rating, alcohol percent- age, price, and grape composition. We obtained fine-grained flavor annotations on a subset by conducting a wine-tasting experiment with 256 participants who were asked to rank wines based on their similarity in flavor, resulting in more than 5k pairwise flavor distances. We propose a now-dimensional concept embedding algo- rithm that combines human experience with automatic machine similarity kernels. We demonstrate that this shared concept embedding space improves upon separate embedding spaces for coarse flavor classification (alcohol percentage, country, grape, price, rating) and aligns with the intricate human perception of flavor.
10
Introduction
13
14 Vision, language, audio, touch, smell, and taste are sensory inputs that ground humans in shared representation, which enables to interact, converse, and create. Recent advances in multimodal learning have shown that combining diverse modalities in shared representation leads to useful and better-ground grounded models [Girdhar et al., 2023, Chen et al., 2023]. Inspired by recent progress, we 18 propose to add flavor to the list of modalities used to learn shared representations.
17
20 centuries, (2) their flavors have been carefully categorized, and (3) classification systems exist to ensure that flavor near-consistent across bottles of the same unique bottling.
22 We bridge the gap between the machine learning and food science communities by presenting
23 WineSensed, multimodal wine dataset that consists of images, user reviews, and flavor annotations.
24 Our motivation twofold. On one hand, internet photos and user reviews are scalable source of
25 data, offering abundant, diverse, and easily accessible insights into wine qualities. On the other hand, human flavor annotations, while not scalable, provide more direct and granular understanding of the wines' flavor profile. By combining these resources, we aim to capture the best of both worlds, yielding richer, more intricate dataset.
28
29 We organized a large sensory study to obtain human- -anotttdd flavor profiles of the wines. The study
30 applies the "Napping" methodology [Pagès, 2005], which is commonly used conduct consumer 31 surveys [imm et al., 2013, Ribeiro et al., 2020]. In this study, 256 participants annotated their perceived taste similarities of various wines. In Fig. the "human kernef" illustrates how participants
Submitted to 37th Conference on Neural Information Processing Systems (NcurIPS 2023) Track on Datascts and Benchmarks. Do not distribute.

---

Machine kernel
***
Shared latent space
images
reviews
attributes
Human kernel
Flavor as an additional data modality. The WineSensed dataset consists of a large collection of images, user reviews, and metadata about unique bottlings (upper left). In a large user study, we collected flavor annotations of over 100 wines using the "Napping" method [Pagès, 2005], where participants were asked to place wines on sheet of paper based on their perceived taste similarity (lower left). We propose an algorithm to combine these data modalities into shared representation (right) and find that using taste annotations as an additional modality improves performance in downstream tasks.
Figure 1:
were instructed to place wines on sheet of paper based on how similar they perceived their flavor to 34 be. The Napping method enabled us to annotate wine flavors with a high level of detail and harness
33
35 the perception of a broad spectrum of individuals. It scales well, as asking a participant to annotate five wines yields 10 pairwise annotations. All participants combined annotated more than Sk flavor distances.
36
37
To complement these annotations, curate images of wine labels, user reviews, and wine attributes (country of origin, alcohol percentage, price, and grape composition) from the Vivino platform, a popular online social network for wine enthusiasts." WineSensed, therefore, represents a large, multimodal dataset that merges seerrgeneatted content with sensory assessments, bridging the gap between subjective consumer perception and objective flavor profiles.
39
40
41
42
Along with the dataset, we propose Flavor Embeddings from Annotated Similarity & Text-Image (FEAST) that leverages recent developments in large multimodal models to embed user reviews and images of wine labels into now-dimensional, latent representation that contains semantic and structural information that correlates with taste. Our model aligns this representation with the flavor annotations from our user study. We find that this combined representation yields a "flavor space" that models coarse flavor concepts like alcohol percentage, country, grape, and the year of production, while also being aligned with more intricate human perception of flavor.
45
46
47
48
49
50
Experimentally, we find (1) that using the pairwise distances (rather than ordering) of the annotated 51 wines improves the flavor representation, which confirms the established methodology in food science,
52 and validates our annotation process. (2) We discover that using multiple data modalities (images,
53 text, and flavor annotations) boosts the flavor representations, highlighting usefulness of our
54 multimodal dataset. (3) Finally, we show that the proposed multimodal model produces flavor space
55 with high alignment with humans' perception of flavor.
https://vvvino.com

---

5e 2 Background and related work
57
Multimodal representations. Learning a shared representation between modalities can reveal useful representations that generalize well and appear grounded in reality. Pioneering work [de Sa, 1994] proposes to learn the correlation between vision and audio. A number of deep learning methods propose to use large collections of weakly annotated data to learn shared vision-language representa- tions [Joulin et al., 2016, Desai and Johnson, 2021, Radford et al., 2021b, Mahajan et al., 2018), shared audio-text representations [Agostinelli et al., 2023], shared vision-audio representations [Ngiam et al., 2011, Owens et al., 2016, Arandjelovic and Zisserman, 2017, Narasimhan et al., 2022, Hu et al., 2022], shared vision-touch [Yang et al., 2022] representations, or shared sound and Inertial Measurement Unit (IMU) representations [Chen et al., 2023]. Recently, ImageBind [Girdhar et al., 2023] showed that images can bind multiple modalities (images, text, audio, depth, thermal, IMU) into a shared representation. While recent advances in other areas of multimodal learning have been fueled by large datasets, the difficulty of quantifying and collecting high-quality flavor data has made it challenging for the machine learning community to develop similar representations for flavor.
58
60
61
62
63
64
65
68
69
Quantifying flavor. Understanding and engineering flaver is a central part of food science and essential in the quest towards healthy and sustainable food production [Savage. 2012], but the use of machine learning methods this end is still in its infancy. Fuentes al. [2019] found correlation between seasonal weather characteristics, and wine quality and aroma profiles, thereby verifying what wine producers have long held to be true. Similarly, Gupta [2018] found that sulfur dioxide, pH,
70
71
73
74
75 and alcohol levels are useful for predicting wine quality. Due to the difficulty of gathering quality perception data, much work focuses on how 'low-level' chemical aspects related to 'high-level' taste properties, e.g. in assessing the quality of chocolate and beer [Gunaratne et al., 2019, Gonzalez Viejo al., 2018].
76
77
78
Analyzing person's perception of wine is challenging due to the complex nature of flavor, which remains ill-understood, and the difficulty in obtaining consistent verbal descriptions of taste across individuals. Napping [Pagès, 2005] is the de facto method to analyze perceived taste in consumer 82 surveys. Participants receive taste samples and are instructed to place them on a sheet of paper based on how similar they perceive their taste to be, with closer meaning more similar. Such experiments
80
81
83
84 are usually conducted with 10-25 participants and less than 20 variants of a product [Giacalone et al., 2013, Pagès et al., 2010, Mayhew al., 2016). In this study, we scale this data collection process to 86 256 participants and 108 unique bottlings of red wine, resulting in over 400 napping papers collected
87 and more than 5k annotated flavor distances. In contrast to previous works [Giacalone et al., 2013,
88 Pagès et al., 2010, Mayhew et al., 2016] our objective is to incorporate taste as one of the modalities that contribute to the shared representations for improved grounding of machine learning models.
89
Human kernel learning. Annotating flavor with Napping [Pagès, 2005] does not provide image- flavor or fextorlavor correspondences but rather relative flavor similarities between sampled products. According to [Miller, 2019] humans are better at describing abstract concepts such as taste with contrastive questions, such as "does wine X taste more similar to wine Y or Z?" For this reason, the machine learning community has used contrastive questions in multiple settings, e.g., for understand- ing how humans perceive light reflection from surfaces by presenting annotators with image triplets depicting the Stanford Bunny with varying material properties [Agarwal al., 2007], produce a genre embedding of musical artists [Van Der Maaten and Weinberger, 2012], and for discovering underlying narratives in online discussions [Christensen et al., 2022]. Most relevant to work is SNaCK [Wilber et al., 2015), which presents annotators with image triplets depicting foods and asked which two of them taste more similar, to obtain flavor triplets. They proposed to combine this ligel human flavor understanding with low-level image statistics to learn food concepts, e.g., that even though guacamole and wasabi look similar, their taste is not. Having humans annotate image triplets of foods works well for coarse concepts, but does not encompass nuanced differences in taste.
91
93
94
95
96
97
99
100
101
102
103
104
In this work, we focus on the much finer-grained taste difference found in wines. These nuances and the complex nature of wine tasting, which involves taste and smell, are not easily conveyed through text images.
3

---

Images
User reviews
Attributes
Ciasyy Sangloveso
Comples velvety berress,
witte Fescer yet
Country Iulyy
Iguorica pespary
Grape: Sargiveeee
spico.. gots bettor with
Region Abruzam
ADISTONT
Rating: 4.3
Dark fruity notes, taste loov
fruily cherlies bomree ourrent Well balanced tamiins wod.
Price: $8
AB
More of fond wire... Unbalaanced with mush emphasis the tuutt and lazking in acidity, Too rich Es own!
Great who really round and sot in the mouth. chocolate. plum. vani lo, leather in the end
Country:
Zinlandd
SHOWDORN
SHOWD
Reglon:
Henry wine but round and soft lanriis, Great with heavy autumin stews de
A rice Zitandel, and I'm not a fant Good ba ance acitlty and sweetteess medium body, alightly dry.
Rating
$23
Cheery: kauor, spices. stoel
intersee notes bodied
Country: Fall
Grape: Agfianio
Reglon: Ipino Taraaii
wine!!! Utitthnt
Atcik: 15 Rating:
Very
Tharics Daniel Uibwttt!!! il
taste hour later the swoutrees really cornes out but
$20.90
Figure Examples from WineSensed. The dataset consists of images of wine labels, user-generated reviews, ver-wine attributes (country, grape, region, alcohol percentage, rating, price), and flavor annotations. Here are examples of the images, reviews, and attributes.
Flavor datasets. The machine learning community has produced numerous food datasets for classifying which meal is in an image [Bossard et al., 2014, Min et al., 2020], retrieving a recipe given an image [Salvador et al., 2017, Li et al., 2022], or predicting the origin wines [Dua and Graff, 2017]. While is possible to extract coarse information about taste from such datasets [Wilber al., 2015], they do not encompass higher resolution details of taste, such as the differences between Cabernet Sauvignon and Pinot Noir.
108
109
110
111
112
113
Similarly. the food science community has developed many datasets for understanding and predicting food flavors, nutrient content, and chemistry. Flavornet [Arn and Acree, 1998], a dataset on human- perceived aroma compounds, explores partly how smells relate to perceived bitterness or fruitiness in a wine. However, its limitation its lack of context linking these odors to specific wine varieties and its limited focus on flavor aspects. FoodDB [Harrington et al., 2019] offers comprehensive information wide variety of food, its nutrient contents, potential health effects, and macro and micro constituents. However, it lacks user-generated reviews and sensory data, which are crucial for understanding the subjective human perception of food and wine. The Wine Data Set [Dua and Graff, 2017] focuses on wines, but only contains wines originating from one region in Italy, limiting the dataset's ability to capture the broader diversity of flavor profiles of wines from various regions worldwide. Furthermore, Dua and Graff [2017] solely incorporate the chemical compounds present in each wine, without annotations of flavors and information associating specific wines with each chemical compound. In contrast previous work, we present multimodal dataset that contains flavor similarities.
114
115
116
117
120
121
122
123
124
125
large corpus of images and reviews, well
The WineSensed dataset
127
128
We present WineSensed, a large, multimodal wine dataset that combines human flavor annotations, images, and reviews. In this section, we provide overview of the curation process for each of these modalities.

---

GALNIO
Sorger
CAYAUS
CAYMUS
1ATA FALLO
CAYMUS
CAYMUS
PLNE RIDE
RIDGI
VINEISERS
MASCOTA
KIENEE
JUSTIN
Figure Examples of images. The viewpoint, lighting, and composition vary across images.
131
Annotated flavors. The flavor data consists of over 5k human-annotann pairwise similarities between
132
108 unique bottlings. Each annotated pair is annotated at least five times reduce noise.
133
These annotations are collected through a series of wine-tasting events attended by total of 256 non-expert wine drinkers. Most participants were between 21-25 years old, and more than half of them were from Denmark. Each participant volunteered their time, dedicating a maximum of two hours to complete the annotations. The experiment was conducted in accordance with the "De Videnskabsetiske Komiteer" (e. the Danish ethics committee for science) (see Appendix I).
134
135
136
137
138
We randomly selected 5 wines for the participants to taste. The participants did not have access to any information regarding the individual wines. The wine was poured into non-transparent shot glasses and the labels of the wines were covered during the entire experiment. The participants were insttucte to put colored stickers (representing each of the five wines) on sheet of paper based on their taste similarity, closer meaning more similar. The participants could repeat the process up to three times, ensuring they did not consume more than 225 ml of wine. The average participant repeated the experiment two times.
139
140
141
142
148
145
We automatically digitized the participants' annotations by taking photo of each filled-out sheet. We used the Harris corner detector [Harris et al., 1988] to find the corners of the paper and a homographic projection to obtain an aligned top-down view of the paper. The images were mapped into HSV color space and threshold filter applied find the different colored stickers that the participant used represent the wines. Having identified the location, we computed the Euclidean pixel-wise distance between all pairs of points, resulting in distance matrix of wine similarities. A more detailed description of the collection and digitization of the napping papers can be found in D.
146
147
148
150
151
152
User-ss. We curated 824k text reviews from the Vivino platform. The reviews were filtered to contain least 10 characters to avoid non-informative reviews such as 'good' and 'bad.' Fig. 2 shows examples of user-reviews. The reviews are free text and can contain special tokens such as emojis. The reviews tend to describe price, pairing, and general terms of wine. Some also describe which flavors the reviewer tastes. These reviews are subjective and can vary based on personal factors and context, leading to inconsistent flavor profiles. Moreover, they only contain coarse flavor descriptions
153
154
156
157
158
and focus more on aspects like preference, price, occasion, and so forth. Fig. 4 shows the distribution of word count per review, number of reviews per unique bottling, and the most common keywords.
159
Images.
The dataset has 897k images of wine labels. Wine labels are known to play major role in consumer's 's decision to purchase particular wine, so it is reasonable to believe that label design
5

---

2.5
2
105
Log Froquency
Prequency
2.0
10a
1.5
1.5
1.0
101
0.5
0.5
10-1
0.25
1.75
0.2
1.0
fmil
Image
X10*
Count
(a) Images per unique bottling
(b) Words per review
(c) Most common keywords
Figure 4: Summary statistics of user reviews and images. Most unique bottlings have less than 10 images. The average review length is 16 words. Common keywords in the reviews include 'fruit', "dry", and 'smooth' revealing coarse semantic information about the flavor of the wines while other keywords such as 'good' and 'great' do not reveal flavor information.
3,5x114
X10*
2.00 X10*
3,0
1.75
1.75
2.5
1.50
1,50
haps Valie
2.0
1.25
1.25
Edue ou Daera
Suars Aftice
1.00
1.00
1.5
1.0
0.50
0.50
0.25
0.25
0.0
0.00
0.00
(a) Country
(b) Grape
(c) Region
x00*
xtel
1aX10!
20
25
1.5
Dopass
2.
1.0
&
1.
09
60 1624 30400
06336
6677 8038
133
140
1485
15.0
16.0
an
39
n
2013 2013
2017
200 2001
Price
(c) Alcohol content
(f) Rating
(g) Year
Figure 5: Wine attributes. WineSensed contains attributes about the geolocation of production (country, region) and the grape composition of each wine. Furthermore, the dataset includes informa- tion on the average price of the wine, alcohol percentage, average rating the Vivino platform, and the year of production. The histograms show the distribution of these attributes.
162
carries information regarding the taste of the wine [Talbot, 2019]. Fig. shows examples of images from the dataset. The images vary in their viewing angle, illumination, and image composition.
163
Attributes. Each wine is associated with the geographical location of the vineyard (both country and region), grape varietal composition, vintage, alcohol content, pricing, and average user rating. Fig. 5 shows the distribution of these attributes. Most wines originate from Italy, with Sangiovese being the most commonly used grape. The wines occupy the lower range of the price spectrum, with the most expensive ones priced at around 40 USD. The attributes are available for 5% of the dataset entries.
166
187
188
4 Flavor Embeddings from Annotated Similarity & Text-Image (FEAST)
169
170
The embeddings recent large image and text networks contain structural and semantic information, however, they do model the intricacies of human flavor. We propose FEAST, method to align these embeddings to the human perception of flavor using small set of human-annotan similarities. FEAST takes text and/or images input, as well as coman-annotated flavor similarities. outputs a unified embedding that aligns with human sensory perception. Fig. 6 provides an overview 175 of the proposed method.
flavor
173
174
178 We first embed the text and/or images into latent space with CLIP [Radford al., 2021a], We use CLIP because of its large training corpus and its image-text aligned latent space, however, highlights that other pretrained networks can be used. We use t-SNE [Van der Maaten and Hinton, 2008] to

---

Imagea
Image Encoder
WT
Dimensionality
Machine Kernel
CUP
Shared Latent
Tust
19NE
FLA
Space
Combiner
ALBERT PLAN-TS PEGASUS
ICP
Procustrs SNaCK
Very interse
CCA
bieckcumentt perfect.
OLIP
Dimensionality
Pairwise distances
NMDS
Figure 6: Model overview. FEAST takes text and/or images input well numan-annotern flavor similarities. The text and/or images are embedded into a latent representation with CLIP. We NMDS embed the flavor similarities. The two representations a latent space that uses the structural information CLIP embeddings and the intricacies human annotations. The bolded methods in the orange, blue, and green boxes indicate choices for our best model, and their remaining combinations serve as an overview of the evaluated baselines.
aligned with CCA to produce
179
reduce the dimensionality of the latent space to 2, which simplifies and constrain the later alignment with the pairwise flavor annotations.
180
The pairwise distances embedded into 2D representation using Non-metric multidimensional scaling (NMDS) with the SMACOF strategy [de Leeuw and Mair, 2009]. NMDS allows us to preserve the original flavor distances provided by humans in a shared space, where each unique bottling is represented with point location, rather than pairwise distances. MDS is commonly used in food science to analyze sensory annotations from Napping studies [Pineau et al., 2022, Varela and Ares, 2012, Nestrud and Lawless, 2010].
183
184
185
186
We then align these two 2D representations to get joint representation that benefits from the structural and semantic information of the image and/or text representations, scales to unobserved unique bottlings, and is aligned with the human perception of flavor. We use Canonical Correlation Analysis (CCA) [Harold, 1936] to align the two representations. CCA identifies and connects common patterns between these representation spaces, ensuring that the final representation consistent across all input modalities.
188
189
190
191
192
Experiments
193
194
We conduct two experiments on the WineSensed dataset. First, we explore how well recent large pretrained language and image models explain wine attributes that correlate with the flavor of wine. Second, explore multimodal models' capabilities to represent more intricate flavors.
195
196
197
Experimental setup. We explore several configurations of human kernels, machine kernels, and "combiners" that align the two representations. Fig. 6 provides an overview of our baselines. The human kernel is formed with t-STE [Van Der Maaten and Weinberger, 2012], a low dimensional graph representation reduced with t-SNE or NMDS, where the notable difference is that t-STE discards the flavor distances, and solely optimizes for triplet orderings. The machine kernel consists pretrained model to embed text and/or images into low dimensional space, (2) which is then compressed into two-dimensional space. For (1), we explore DistilBert
198
199
200
of two steps: (1)

---

Table 1: Ablation of machine ker-
Table 2: Ablation of Modal- ities. Accuracy of single and combined modalities. Using multiple modalities improves performance. We find that combining image, text, and flavor yields much better ac- curacy than modeling each modality separately.
Table 3: Ablation of hu- man kernels, reducers, and combiners.
nels. Accuracy of machine kernels across image and text modalities. Im- age models perform worse than text models. ALBERT, BART and CLIIP perform the best, all models perform better than random using at least one classification method.
Reducer Human Kemel
Ace f
SVM
NN
Random
0.11
0.11
L-STE I-SNE NMDS
0.13
0.10
Acc
0.15
0.13
Machine kemel
Modality
SVM
NN
0.16
0.13
Random
0.11
0.11
Reducer
Acc f
VIT
Image
0.09
0.13
Machine
SVM
DeiT
Image
0.14
0.15
T5
CLIP
ResNET
Text
Image
Image
0.15
0.11
0.15
0.16
0.15
0.16
Modality
SVM
Ace
NN
1-SNE
UMAP
0.15
0.20
0.22
0.21
0.18
0.25
ALBERT
Text
0.15
0.18
Flavor
0.16
0.11
Acc f
BART
Text
0.16
Image
0.11
0.15
Combiner
SVM
NN
DistiBERT
Text
0.15
0.17
Text
0.16
0.18
CLIP
Text
0.16
0.18
Text+Flavor
0,23
ICP
0.21
0.24
FLAN-TS
Text
0.15
Image+Text Text
0.22
0.25
Procrustes
0.19
0.23
PEGASUS
Text
0.13
0.13
Flavor
0.23
0.18
SNaCK
0.23
0.24
BART
Text
0.11
0.15
Image+ Flav+
0.28
0.26
CCA
0.28
0.26
204
[Sanh et al.. 2019], T5 [Raffel et al., 2020], ALBERT [Lan et al., 2019], BART [Lewis et al., 2019]. PEGASUS [Zhang al., 2020], FLAN-T5 [Chung al., 2022] and CLIP for embedding text and VIT (Dosovitskiy et al., 2020], ResNet [He et al., 2016]. DeiT [TTourron et al., 2021], and CLIP for embedding images. For (2), we explore t-SNE, UMAP [Mclnnes et al., 2018], and PCA [Pearson,
207
208
1901]. For the combiners, we experiment with CCA, Iterative Closest Point (ICP) [Chen and Medioni,
209
1992], Procrustes [Gower, 1975] and SNaCK. For more detailed description of the implementation and software packages used, please refer to E the Appendix.
210
211
5.1
Coarse flavor predictions
212
We first explore how well pretrained language and vision models explain wine attributes that correlate with flavor. We then investigate using FEAST to align the machine and human kernels improves the representation.
213
214
Implementation details. We use a balanced SVM classifier with an RBF kernel well as a Multi- layer Perceptron [] neural network to predict wine attributes of the flavor embeddings. We predict price, alcohol percentage, rating, region, country, and grape variety as these attributes are known to correlate with the perceived wine flavor. We mitigate imbalanced class distributions with class weight balancing and oversampling of the minority classes. report the accuracy averaged over the seven attributes computed through 5-fold cross-validation. The accuracy measures how coherent the embeddings are with the flavor attributes. A more detailed description of the implementation can be
216
217
218
219
220
found in J.2.
223
Results. Tables to 3 ablates our proposed method and summarizes our main conclusions. Please see Appendix J.2 for per attribute classification accuracy for all combinations machine kernels, human kernels, modalities, reduces, and combiners.
224
225
Table 1 shows that most pretrained image and text models yield slightly higher performance than the random baseline. The text encoders are slightly better than the image encoders. BART and CLIP perform the best. All encoders in the table use t-SNE to reduce the embedding to 2D. Table 3 (middle) shows t-SNE yields better accuracy than UMAP and PCA when using a CLIP encoder.
227
228
229
230
Table (top) shows that NMDS performs better than t-STE. NMDS uses the relative distances between annotations, whereas t-STE discretizes the annotations and considers only the ordering within each triplct. The results suggest that the pairwise distances useful to model the flavor space.

---

Table 4: Fine-grained flavor predictions. Triplet Agreement Ratio (TAR) between text, image, and multi-modal encoders and human annotated flavor similarities. A higher TAR indicates that the model's s representation space more aligned with humans' perception of flavor.
Machine Kernel
Human Kernel
Combiner
Modality
TAR
Random
0.5
CLIP t-SNE
Text
0.82
CLIP t-SNE
Image
0.82
CLIP + t-SNE
Image Text
0.81
CLIP + I-SNE
Image Flavor
0.89
CLIP + t-SNE
Text Flavor
0.88
CLIP t-SNE
NMDS
CCA
Image Text Flavor
0.91
233
Table 3 (bottom) shows that using CCA align the two representations yields higher accuracy than SNaCK ICP.
235
Table 2 shows that including flavor as a modality increases the accuracy, egg. using flavor to align the image or text embeddings lead to higher accuracy. Using CLIP followed by t-SNE, NMDS, and CCA to combine language, vision, and flavor into a single representation leads to the best configurations, illustrating that the human annotations are useful for learning a flavor representation. Maybe most surprisingly, we show that each modality by itself is par with the random baseline, but their combination produces latent space that much better describes the flavor attributes.
236
237
238
241
5.2
Fine-grained flavor predictions
242
We now proceed to evaluate more intricate flavor predictions by using human-annotated flavor similarities as ground truth.
243
Implementation details. To evaluate our representation, we measure the Triplet Agreement Ratio (TAR) [van der Maaten and Weinberger, 2012] between predicted flavor embeddings and the flavors. TAR measures the agreement between triplet derived from the latent space and the ground truth triplets from the flavor annotations. Higher TAR means that the ordering of distances in the latent space corresponds to the human perception of flavor. This measure indicates how aligned the two representations are, and provides higher granularity flavor prediction than flavor attributes. more detailed description of the implementation can be found in F.
247
248
249
250
Table 4 ablates FEAST and shows that for the higher gramularity predictions both the pretrained text and image encoders improve upon the random baseline. We show that including the human kernel with NMDS further improves the TAR scores. This highlights the usefulness of the flavor distances recorded by the human annotators. In Appendix F, we show results from all configurations of human kernels, machine kernels, reducers, and combiners. We find that NMDS consistently yields better performance than t-STE, and that combining human and machine kernels improves the TAR scores across multiple model configurations.
Results.
252
253
254
255
256
257
Discussion & Conclusion
259
In this paper, we introduce WineSensed, an extensive multimodal dataset curated for flavor modeling. The dataset comprises over 897k images and 824k reviews, and has over 5k human-annotated pairwise flavor similarities, obtained via sensory study involving 256 participants. We propose a simple algorithm, FEAST, to align semantic information from machine kernels with flavor similarities from human annotators in shared flavor representation. We find that combining these modalities improves both coarse and fine-grained flavor predictions.
260
262
263
264
285
WineSensed further strengthens the collaboration between the food science and machine learning communities. introduces flavor modality in multimodal models, and serves entry point for the development of machine learning models for flavor analysis and potentially deepening

---

288
comprehension of wine flavors. The dataset and the proposed procedures open many interesting possibilities, such as using flavor to ground foundation models or extending the dataset with other modalities, such as chemical composition, or other food categories.
269
270
Constraints and considerations. The dataset annotated flavor in the array of modalities in multimodal models. Its current scope is constrained to selected group of red wines, predominantly Italian ones. While this enables a more nuanced understanding of flavors within Italian wines, it may not represent the broader spectrum of red wines globally. Furthermore, the dataset's emphasis on wines prevalent in Western cultures highlights a geo- cultural bias. Expanding the dataset to encompass more diverse drink types from different cultures could provide more comprehensive understanding of global flavor perception. Lastly, the Napping methodology is not immune the influences of participants' backgrounds and experiences. Individual perceptions, shaped by personal histories, can introduce nuances in the data. Though leveraging non-expert wine drinkers for flavor annotations introduces subjectivity, this approach, inspired by common sensory study practices, broadens taste perspectives, enhances study accessibility, and offers commercial value, with multiple annotations per entry mitigating individual biases. Exploring a broader range of foods and beverages remains a valuable direction for future work.
novel first step including human-
278
274
275
276
277
281
282
283
Ac Aenooweeemmnnts. This work supported by the Pioneer Centre for AI, DNRF grant number Pl. and by research grant (42062) from VILLUM FONDEN. This project received funding from the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (grant agreement 757360), as well as the Danish Data Science Academy (DDSA).
286
287
288
10


Reviews:
[{"review_id": "hZ8Fvx0rFf", "reviewer": "Reviewer_DdVt", "summary": "", "questions": "", "limitations": "- The paper does not mention whether the data source biases (most wines being Italian e.g.) can affect the analysis.\n\n- Can the experience / background of the participants (line 134) affect the taste embeddings, or is Napping robust to that?", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Extensive dataset\n- Introduces FEAST, a way to augment food metadata with subjective taste rankings by aligning the embedding spaces\n- Includes mutliple ablations to understand the effect of the different modalities, and finds that the taste embeddings are useful\n- The data is very well documented\n- The code to reproduce the study is well documented and the study can be extended to other food datasets.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "fjoQH0pCOz", "reviewer": "Reviewer_YEVG", "summary": "", "questions": "", "limitations": "I find important to extend the work to other foods and beverages, however the authors didn’t reflect enough on what are the limitations and future directions of the solution, providing an unsatisfactory discussion. I suggest improving the coarse flavor predictions by replacing the SVM classifier with a neural network, so that the pretrained models can be fine-tuned on the classification task. I also suggest trying larger language models (e.g. BART-large) and better performing ones (e.g. FLAN-T5, PEGASUS).\n", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "The authors explore an uncharted direction for multimodal representations by combining images, text and human perception of flavor. This novelty opens the possibility for new studies by designing original experiments leveraging the peculiar nature of the dataset. I particularly appreciated how the authors proved that the ordering of distances in the latent space corresponds to the human perception of flavor.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "HMiCWgWUMV", "reviewer": "Reviewer_1rM1", "summary": "", "questions": "", "limitations": "When release the dataset, whether the license of the multi-modal data source is obeyed?", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The work is novel for the flavor modality.\n2. The work is easy to follow.\n3. The experiments are comprehensive.", "weaknesses": "", "quality": null, "clarity": 1, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "FJKx2NqCv7", "reviewer": "Reviewer_PNh1", "summary": "", "questions": "", "limitations": "Authors should take a look at the following comments and try to answer to it:\n\n1. No Graph Models: Despite the inherent interconnectedness and potential for graphical representation in the data (e.g., wines as nodes, similarity in taste as edges), the study does not make use of graph models. Graph models could potentially provide a more accurate representation of the data's complex, non-linear relationships.\n2. Age and Cultural Bias: The majority of participants in the wine-tasting events were between 21-25 years old and more than half were from Denmark. This could introduce a bias in the flavor annotations, which may not represent a wider range of cultural or age-related taste preferences.\n3. Reliance on Human Input: While the use of human-annotated flavor profiles provides a unique perspective, it also means that the dataset is susceptible to the subjective biases of the annotators.\n4. Subjectivity: The use of non-expert wine drinkers for annotating flavors introduces an element of subjectivity into the data. While this approach makes the study more accessible, it may compromise the accuracy or consistency of the flavor annotations.\n5. Dependence on Pretrained Networks: The methodology relies heavily on pretrained networks like CLIP. This dependency could be viewed as a limitation, especially in situations where these pretrained models may not be applicable or available.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Novelty: The study introduces a unique, large, and multimodal dataset that includes human flavor annotations. This is a significant contribution as there are few datasets currently available that incorporate the sense of taste, especially in the field of wine tasting.\n2. Multimodal Approach: The use of both text and images as input is an innovative approach that allows the model to make sense of more complex, multidimensional data, providing a richer and more detailed understanding of the wines.\n3. Human-annotated Flavor Profiles: The use of human-annotated flavor profiles gives the model a more nuanced understanding of wine flavors. These human inputs provide a unique perspective that cannot be captured through traditional computational methods.\n4. FEAST: The development of FEAST, which aligns the image and/or text embeddings with the human perception of flavor, represents an exciting advancement in the field of artificial intelligence, showing potential for applications in other areas beyond wine tasting.\n5. Integration of Machine Learning and Food Science: By bridging the gap between machine learning and food science, the study opens up new avenues for interdisciplinary research, promoting a more comprehensive understanding of sensory inputs.\n6. Extensive Dataset: With over 350k unique vintages, 897k images of wine labels, and 824k reviews of wines, the dataset is notably extensive. This depth and breadth of data give researchers ample material to train and test models, potentially leading to more accurate and reliable results.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "iGkPCj8zMy", "reviewer": "Reviewer_N3w8", "summary": "", "questions": "", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The dataset is unique and highly relevant, as flavor is an important but (extremely) understudied modality for multimodal deep learning. Modeling flavor perception could enable many applications in food science, consumer products, etc.\n\n- The flavor annotation methodology using Napping is novel for machine learning and provides more nuanced human flavor similarity judgments than prior approaches like image triplets. The large scale of the study (256 people, 5K annotations) is impressive.\n\n- Multimodal modeling with vision, language, and human flavor annotations is compelling and well-motivated. The proposed FEAST method for aligning representations is simple but effective.\n\n- The comprehensive experiments cover many encoder, dimensionality reduction, and fusion options. The gains from multimodal learning validate the utility of the dataset.\n\n- The dataset website is designed to look more fancy than the paper writing.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper has received the scores of 5, 6, 6, 6, and 10, where all reviewers unanimously voted for accepting the paper after reading the authors’ rebuttal.

Most of the reviewers appreciated the importance of such an innovative task and benchmark, but there still remains a concerns about the baseline models’ performances. 
In this respect, I recommend accepting the paper as a poster.
Decision: Accept (Poster)