Can Transformers Smell Like Humans?
Authors: Farzaneh Taleb
Miguel Vasco
Antonio H. Ribeiro
Mårten Björkman
Danica Kragic
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Representational Alignment
Olfactory
Transformers
Representation Learning
Neuroscience
Primary area: neuroscience_and_cognitive_science
TL;DR: In this work, we evaluate representational alignment between human olfactory perception and representations extracted from pre-trained transformers.

Abstract: The human brain encodes stimuli from the environment into representations that form a sensory perception of the world. Despite recent advances in understanding visual and auditory perception, olfactory perception remains an under-explored topic in the machine learning community due to the lack of large-scale datasets annotated with labels of human olfactory perception. In this work, we ask the question of whether pre-trained transformer models of chemical structures encode representations that are aligned with human olfactory perception, i.e., can transformers smell like humans? We demonstrate that representations encoded from transformers pre-trained on general chemical structures are highly aligned with human olfactory perception.  We use multiple datasets and different types of perceptual representations to show that the representations encoded by transformer models are able to predict: (i) labels associated with odorants‌‌ provided by experts; (ii) continuous ratings provided by human participants with respect to pre-defined descriptors; and (iii) similarity ratings between odorants provided by human participants. Finally, we evaluate the extent to which this alignment is associated with physicochemical features of odorants known to be relevant for olfactory decoding.

Can Transformers Smell Like Humans?
Farzaneh Taleb
Miguel Vasco
Department of Intelligent Systems KTH Royal Institute of Technology Stockholm, Sweden farzantn@kth.c
Department Intelligent Systems KTTH Royal Institute of Technology Stockholm, Sweden miguelsv@kth.se se
António H. Ribeiro
Mårten Björkman
Department of Information Technology
Department of Intelligent Systems KTH Royal Institute of Technology
Uppsala University Uppsala, Sweden
Stockholm, Sweden celle@kth.se
antonio. ribeirofit.u..s
Danica Kragic
Department of Intelligent Systems
KTH Royal Institute of Technology
Stockholm. Sweden
dani@kth.se
Abstract
The human brain encodes stimuli from the environment into representations that form sensory perception of the world. Despite recent advances in understanding visual and auditory perception, olfactory perception remains an under-cxplored topic in the machine learning community due to the lack of large-scale datasets annotated with labels of human olfactory perception. In this work, we ask the question of whether pre-trained transformer models of chemical structures en- code representations that are aligned with human olfactory perception, i.e., can transformers smell like humans? We demonstrate that representations encoded from transformers pre-trained on general chemical structures are highly aligned with human olfactory perception. We multiple datasets and different types of perceptual representations to show that the representations encoded by transformer models are able to predict: (i) labels associated with odorants provided by experts; (ii) continuous ratings provided by human participants with respect to pre-defined descriptors; and (iii) similarity ratings between odorants provided by human partic- ipants. Finally, we evaluate the extent to which this alignment is associated with physicochemical features of odorants known be relevant for olfactory decoding.
Introduction
The human brain receives sensory input from the environment and encodes it into high-dimensional representation space, forming a perception of the world [1]. Recent studies have significantly improved understanding of the underlying mechanisms of visual, linguistic, and auditory percep- tion [2-4]. Indeed, there is a significant level of alignment berween human response (from neuron to behavior) and activations of deep neural networks when provided with the same stimuli [5-14].
Despite these recent advances, human olfactory perception remains an under-explored topic. There is no single organizing principle that determines the dimensions of odor space, making the characteriza-
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

CC CC
MoLFormer
Alignment
Figure Evaluating representational alignment between human and pre-trained transformers. Human participants are stimulated with two odorant substances and asked to rate the perceptual similarity between them (Left). We encode representations of the same pair of odorants using MoLFormer and compute the similarity between pairs of representations (Right). Finally, we measure alignment between the two systems.
tion of odor perception and its relation to chemical compounds an open and complex problem [15]. A lack of universally accepted methods describe odorants either quantitatively qualitatively makes this problem even more challenging. There are very few studies that have explored the mapping of chemical structures olfactory perception [16-19]. addition, processing chemical olfactory stimuli using deep neural networks has not been extensively investigated. Nevertheless, training the existing supervised models usually requires an extensive effort by experts to label data.
Transformer-basee models [20, 21] are a breakthrough in machine learning, surpassing the need for extensive labeling by utilizing implicit supervision without the necessity for direct labels. These models have demonstrated impressive performance in various tasks such as image [22], video [23], and natural language processing [24]. More recently, they have also shown promising results in encoding chemical structures [25].
In this work, ask the question of whether representations of odorant chemical structures extracted from transformers pre-trained on chemical structures align with human olfactory perception or, in other words, can transformers smell like humans? We employ MoLformer [25], state-of-ther t-thh-rrtt transformer, which is pre-trained on chemical structures and we show that representations odorants extracted from this model:
predict labels assigned to odorants by experts (Section 4.1);
can predict continuous perceptual ratings provided by human participants (Section 4.2);
present a high correlation index with human perceptual similarity ratings (Section 4.3);
present high correlation index with physicochemical descriptors known be relevant for olfactory perception (Section 4.4).
Surprisingly these results hold for models that were not explicitly trained the purpose of predicting the human olfactory experience. To the best our knowledge, provide the first empirical study on evaluating the alignment between odorant representations encoded by transformers and human olfactory perception.
2
Related Work
The availability of larger datasets, together with advances in predictive methods, has led to increasing interest in the prediction of olfactory perception from molecular structures.
Olfactory perception prediction. Learning predictive models of olfactory from molecular structures has been addressed mostly by the neuroscience community. Several works used standard chemoinfor- matic representations of molecules to model olfactory perception [17, 19, 18]. Specifically. Snitz et al. [18] proposed computational framework and algorithm based on structural features molecules to predict perceptual similarities between odorant pairs. This algorithm leverages feature engineering to identify the most relevant subset of features among 1433 physicochemical descriptors to predict pair odorant perceptual similarities.

---

Later, Ravia et al. [17] extended this model to also include the perceived intensity of molecular components. They employed 21 physicochemical descriptors discovered in previous works and proposed weighting approach for multicomponent odorants (MC-odorants) based on their perceived intensity. They reported a higher correlation when employing the weighting approach compared to using the same model without it. However, the representation and generalization capabilities of these models are quite limited and unexplored.
Deep neural networks for odorants. Recently, Lee et al. [16] proposed a novel representation learning model of odorants, based on a message-passingg graph neural network [26], which they refer to as Principal Odor Map (POM). To train this model, they curated and merged data from Leffingwell [27] and GoodScent [28] databases to compile dataset of about 5000 molecules with 138 expert-labeled odor descriptors. This model outperforms the baselines in multiple odor prediction tasks and shows relatively high alignment with human ratings in describing odorants. Nevertheless, training this model requires labeled data, relying on subjective evaluations of numerous odorants by experts. Besides being time-consuming and laborious, this process can introduce subjective biases into the model, concern magnified by our incomplete understanding of the foundational factors off olfactory perception.
Large-scale molecular models. Large-scale pre-trained models, often known as foundation mod- els [21], have been recently explored to perform diverse tasks by leveraging large amounts of unlabeled data. MoLFormer [25] model has been proposed in the context of chemical prediction tasks, able to extract rich representations from chemical structures. MoLFormer consists of a transformer-bic based architecture, with linear attention and relative positional encodings. This model is trained using self-supervised approach, on multiple datasets (e.g., the PubChem [29] and ZINC [30] datasets) on a masked token prediction loss.
3 Method
In this section, we provide a detailed description of the datasets utilized in this study and outline the methodology for extracting both odorant (machine) and perceptual (human) representations. Addition- ally, we present the main model and baseline methods employed, along with the evaluation metrics used to assess their performance. Our experiments do not require significant computational resources: we mostly train linear models that do not involve GPU usage or models that can be trained single commercially available GPU under one hour. All computational code to reproduce our results is available at https: : //ggttttt...tt///////// com cow/Farzaneh--11 Tatrraannn-aaalb///e
al gnment
3.1 Datasets
We use the publicly available version of the following datasets provided by Pyrfume repository [31].
Geoooset (GS-LF) 127, 28]. We employ a curated and merged version of the Goodscents [28] and Leffingwell [27] datasets, provided by [32], following the procedure intro- duced by Lee et al. [16]. This dataset contains 4983 molecules with 138 expert-labeled descriptors (e.g. creamy, grassy), where each odorant may be linked to multiple descriptors.
Sagar [33]. This dataset contains the rating of 160 odorants by 3 human participants, with respect to 15+3 perceptual descriptors. In addition 15 common descriptors among participants, there are more descriptors that vary among them. We excluded these variable descriptors and focused solely on the common descriptors among the participants. The provided ratings were already normalized within the range of [-1, I] and the mean ratings across all the subjects are computed for subsequent analysis.
Keller [34]. This dataset contains ratings of 480 structurally and perceptually diverse molecules by 55 human participants, evaluating 23 descriptors. Participants were instructed to adjust slider to rate odorants according to individual descriptors, with the slider position subsequently translated into scale ranging from 0 to 100. Ratings were then averaged across all participants for further analysis.
Ravia [17]. This dataset contains similarity ratings of 195 unique pairs of MC-odorants and mono- molecules by 94 participants. The similarity values were averaged across all the participants. In this work, we disregarded the factor of odorant intensity and averaged similarity ratings based on the unique pairs of odorants.
3

---

Snitz [18]. This dataset includes similarity ratings from 139 participants and 359 unique pairs of odorants, In each trial, participants were presented with two distinct odorants and asked rate the degree of similarity in their smells. These ratings were then averaged across all participants.
3.2 Odorant representations
Odorants can be described as single molecule or mixture of molecules, which we denote as multicomponent odorants (MC-odorants). In this section we describe the method to extract odorant representations from the main pre-trained model (Mol.Formen and our baseline models (DAM and Open-POM).
MoLFormer. We employ MoLFormer [25] to encode SMILES strings associated with a single molecule and extract 768-dimensional vector from the last layer of the model. SMILES (simplified molecular-input line-entry system) is string-based representation that encodes relevant chemical information such as the type of atoms, their bonds, and the substructures present in the molecule. For MC-odorants, we average the extracted representations across all available mono-molecule components within that MC-odorant. The odorant representation for each dataset matrix of X; 6 R" 768
768 where is the number of unique odorants.
Open-POM. The principal odor map (POM) is supervised-learning model, based on message- passing graph neural network [26], which trained on the GS-LF datasets to predict odorant percep- tual labels. We employ publicly available version of this model, which denote by Open-POM [32]. We train Open-POM for 150 epochs, using 30 different train-test splits, and we extract representations from the penultimate layer of this model. The odorant representation for each dataset matrix of X, E R where nt is the number of unique odorants. For MC-odorants, we average the representations extracted for each individual molecule within the mixture.
Distance Angle Model (DAM). Snitz et al. [18] proposed a distance angle model (DAM) that uses 21 physicochemical descriptors predict the similarities between pairs of odorants. We extract these 21 descriptors for each odorant using AlvaDesc [35] and discard 6 of them due to NaN values produced by this software. We use the remaining subset of 15 physicochemical descriptors out of 21 to measure similarity between odorants or train a linear mapping from them to the perceptual representation space. The odorant representation for each dataset is matrix of X; R=x15 where is the number of unique odorants. As suggested by Ravia et al. [17], we average the representations extracted for each individual molecule within the mixture compute representations for MC-odorants.
3.3 Perceptual representations
Perceptual representations of odorants are provided by human participants when exposed odorant stimuli. Perceptual olfactory data were collected in one of the following ways:
Experts label the odorants, where each odorant may be linked to multiple labels (e.g., [27, 28)). The perceptual representation is a matrix of yı E {0, 1 jaxd where n. is the number of unique odorants and d is the number of classes.
2. Non-expert participants provided ratings with regards to a set of predefined descriptors (e.g., [33, 34].) In this case, the averaged perceptual representations over participants and replicas form a matrix of E [a, bjaxd, where n is the number of unique odorants, d is the number of descriptors, and and b denote the minimum and maximum values participants can use to describe the odorants with respect to these descriptors.
3. Participants evaluated the perceived similarity between pairs of odorants (e.g.. [18, 17]). In this case, the averaged perceptual representations over participants and replicas are a vector of И E [a, bjnx1 where is the number of unique "pair of odorants" and and b indicate the range values participants can use to rate the odorants' similarity with respect to the descriptors.
3.4 Alignment between perceptual and odorants representations
We measure the similarity between two representation spaces directly when it is possible (Section 4.3), otherwise we train linear model to predict perceptual representations for each odorant (Section
4.1, 4.2). We use nested 5-fold cross- validaaai to tune the hyper-parameters of the linear models

---

True Positive Rate
True Positive Rate
True Positive Rate
0.5
0.5
0.5
0
0.5 False Positive Rate
1
0.5 False Positive Rate
0.5
1
False Positive Rate
RDC 0.90 0.00
Mean (AUC- 0.97 a 0.000
Mean (AUC- 0.00)
(a) MoLFormer
(b) Open-POM
(c) DAM
Figure 2: ROC curve for linear classifiers trained on GS-LF representations extracted from three different models. Each curve corresponds to separate test split, with the thicker curve representing the average performance across all splits. We highlight that MoLFormer outperforms DAM, despite not being trained to predict perceptual labels but does not achieve the performance level of Open- POM, which demonstrates the highest performance. The chance level is shown with red dashed line.
and assess evaluation metrics on the test set that was held out during the training phase using an 80%-20% train-test split. This process is repeated 30 times using 30 different train-test splits.
3.5 Evaluation metrics
In this section, we introduce the main evaluation metrics to measure the alignment in this paper.
Micro-averaged ROC-AUC score. The micro-averaged ROC-AUC score was computed assess the performance of each model for the multi-label classification task. The micro-averages ROC-AUC score is computed by aggregating true positive, false positive, true negative, and false negative values across all classes.
Normalized Root Mean Squared Error (NRMSE). The root mean squared error (RMSE) is the difference between the observed values and predicted ones for the regression task. Here, we normalize it by the range of true observations - i.e., NRMSE
(MSSE/(maxy)) min(y)).
Pearson Correlation Coefficient (CC). We report the Pearson correlation coefficient between predicted results and real values. It measures the linear correlation between two sets of data and is the ratio between the covariance of two variables and the product of their standard deviations.
4 Results
In this section, we evaluate whether the representations encoded by pre-trained models of chemical data can predict the human olfactory experience despite not being explicitly trained for this purpose. First, we focus on a subset of experiments aimed at predicting expert-assigned labels from odorants through linear mapping from representations to perceptual descriptors (Section 4.1). Subsequently, we aim to predict continuous scores provided by human participants (Section 4.2). Finally, we seek to predict the direct similarity scores from the representations extracted from odorants (Section 4.3). Additionally, we provide insights into the potential reasons underlying the observed alignments (Section 4.4).
4.1 Expert-assigned labels classification
To assess the performance of MoLFormer in predicting expert-assigned labels for odorants, we implemented linear mapping from the representations extracted by MoLFormer to the odorant rep- resentations extracted from GS-LF dataset. First, the dimensionality of the extracted representations is reduced to 20 using PCA, followed by z-scoring of each feature. Then, we train individual logistic regression models for each descriptor. This process was repeated 30 times, each with different train-test split, to quantify the uncertainty of the results.
5

---

mesty
ethereal
meaty
savory
cognac
savory
beefy
fermented
beefy
Principal Component 2
roasted
Principal Component 2
alcoholic
roasted
ethereal
muguet
floral
cognac
lavender
muguet
fermented
jasmin
lavender
alcoholic
jasmin
Principal Component
Principal Component
(a) MoLFormer
(b) PPennPOM
Figure Visualization of odorant representations encoded by different models on the GS-LF dataset using the figure layout suggested by [16]. We plot the first and second principal components (PCs) of the representation spaces. Areas dense with molecules that have broad category labels (floral, meaty, or ethereal) are shaded, while areas dense with narrow category labels outlined. MoLFormer captures the perceptual relationship between different odorants in its representation space, despite not being explicitly trained for this purpose.
We apply the same procedure without dimensionality reduction to DAM model representations, For the Open-POM model, which is already trained end-to-end and supervised the same dataset, directly extracted the predictions for the test set without retraining the model. As shown in Figure the MoLFormer model achieves high ROC-AUC scores in odorant classification, outperforming the DAM model, which is trained using 15 physicochemical descriptors. However, the performance of MoLFormer lower than that of OOM,-POM, which is trained end-to-end with supervision the same dataset.
An additional experiment is conducted to understand the degree of perceptual details captured in the odorant representation space MoLFormer by comparing odorant representations encoded by this model with the representations encoded by Open-POM. In Figure 3 depict the first two principal components of the representations. We highlight the similarity between the representations encoded by both Open-POM and MoLFormer and observe that the latter is able to capture the perceptual relationship between different odorants despite not using any perceptual labels during training (unlike the supervised Open-POM Model).
4.2 Continues perceptual rating prediction
To evaluate the capabilities of the MoL.ormer model to predict continuous rating scores with respect to pre-defined descriptors, provided by human participants, we train separate linear regression models with regularization applied using the Lasso penalty for each descriptor. Once again, the dimensionality of the extracted representations is reduced to 20 using PCA (for MoLFormer and Open-POM), followed by z-scoring cach feature. This procedure is repeated using 30 different train-test splits.
The results of these experiments are shown in Table and Figure 4. Table shows the average Pearson correlation coefficient and NRMSE across all descriptors, while Figure 4 presents the results for each individual descriptor. As shown in Table 1, overall, none of the models exhibit a high correlation. Nevertheless, Moo_rrrrr slightly underperforms compared to Open-POM in both datasets. However, performs better than DAM for the Keller dataset but worse than DAM for the Sagar dataset, where DAM even outperforms OPM.

---

Table 1: Performance of the models to predict continuous ratings averaged across all perceptual descriptors. We compute the average Pearson correlation coefficient (CC) and normalized root mean squared error (NRMSE) across all descriptors. MoLFormer shows slightly worse performance than Open-POM but better than DAM for the Keller dataset and worse than DAM for the Sagar dataset, where DAM outperforms Open-POM.
Keller
Sagar Open-POM
MoLFormer
Open-POM
DAM
MoLFormer
DAM
CC (t)
0.20 = 0.00
0.22 = 0.01
0.17 = 0.00
0.25 0.01
0.29 + 0.01 0.18
0.01
0.35 = 0.01
NRMSE (L)
0.15 = 0.00
0.15 = 0.00
0.15 = 0.00
0.19 = 0.00
0.17
Conrebtioon Coeffident
Correlation Coeffident
0.
04
0.3
& 93
0.2
84
Acia
Ammonia
Bakery
Burne
Chamical
Colc
Decayen
Familiarity
Rish
Flower
Fruit
Garlic
Gress
Intersity
Musky
Pleasantress
Sour
Spices
Swesty
Warm
Intersity
Plaasantness
Sour
Decayed
Musky
Swesty
Cool
Floral
Sweet
Warm
Bakery
Spicy
0.2
0.2
0.15
NRMSE
NRMSE
1.15
0.1
0.05
0.05
Acid
Ammonia
Bakery
Burne
Chamical
Cole
Decayed
Familliatty
Fish
Fruit
Garl
Gress
Intensityy
Musiky
Pleasentreess
Sour
Spices
Sweety
Sweet
Warm
Wood
Intensity
Plaasintmes
Sour
Decayed
Musky
Fruity
Sweety
Cool
Floral
Sweet
Warm
Bakery
Splcy
Descriptor
Descriptor
MoLFormer
Open-POM
Open-POM
(a) Keller
(b) Sagar
Figure 4: Performance of the models predict continuous ratings per descriptor. We computed Correlation and NRMSE between predicted and actual ratings per perceptual descriptor. Despite being trained predict human olfactory labels, the MoLFormer model performs on par with the Open-POM and DAM models.
According to Figure MoLFormer model performs on par with the Open-POM and DAM models, which are trained with supervision in predicting the rating for each descriptor. In summary, although, on average, MoLFormer performs slightly worse than Open-POM, it still demonstrates similar degree of alignment, especially despite the absence of supervision in its training process.
4.3 Representational similarity analysis
In order to evaluate the direct alignment between the odorant similarities encoded by MoLFormer and those obtained from human participants, separately encode each odorant by MoLFormer (and the baseline models) and compute the cosine similarity between the extracted representations. Subsequently, compute the Pearson correlation between the similarity scores computed by the models and those provided by human participants in the Ravia and Snitz datasets. The results are presented in Figure 5a.
These results show that the MoLForrer is able to extract representations that encode information related to the human olfactory perception, despite not having access to that information during model training. We highlight a significant high correlation between perceptual and odorant representation for the Snitz (r 0.64, p < 0.0001) and Ravia datasets (r 0.66, 0.0001).

---

Correlation Coefficient
0.7
Correlation Coefficient
0.7
0.5
0.6
0.3
0.1
0.5
Ravia 2020
Snitz 2013
1
3
5
Dataset
Model Layer
MoLFormer
Open-POM
DAM
Snitz 2013
Ravia 2020
(b)
Figure 5: Representational similarity analysis for Snitz and Ravia datasets: a) Correlation coeffi- cients between similarity scores provided by human participants and computed using representations encoded by the different models b) Correlation coefficients considering odorant representations extracted from different layers of the MoLFormer model.
The comparison with the baseline models indicates that it performs on par with the Open-POM model and significantly outperforms the DAM model. These results suggest that, despite being trained with some form of supervision, these models may struggle to effectively extract similarities between odorants. Additionally, the findings demonstrate that MoLFormer is more proficient at identifying similarities between pairs of odorants than mapping them to set of predefined descriptors. This superior performance may be due to the model's s ability capture measure of similarity, as perceived by humans, rather than introducing subjective language bias associated with pre-defined descriptors.
Finally, we aim to evaluate whether the depth of the layer in the MoLFormer model, from which we extract the odorant representations, affects the representational alignment. To assess this, we repeat the described procedure this section for each layer separately. As shown in Figure 5b, representational alignment improves with increasing layer depth, indicating that deeper layers of the transformer are more aligned with high-level perceptual representations.
4.4 Decoding relevant physicochemical features from pre-trained representations
To evaluate whether MoLformer effectively extracts features from chemical structures relevant to olfactory perception, we evaluate the alignment of MoLFormer with physicochemical descriptors that are used in the DAM model. To do so, we train 15 linear regression models, each one predict single physicochemical descriptor from the extracted representations of the MoLFormer. We subsequently evaluate the correlation between the predicted and true values. As shown in Figure Mol demonstrates high degree of alignment in predicting these values. Out of the 15 physiochemical descriptors, MoLformer successfully predicts the values for 13 descriptors as well as or better than the Open-POM model.
Next, we evaluate whether this alignment changes across the layers of MoLformer. Therefore, we repeat the same procedure for each layer separately. As illustrated in Figure 7, the alignment with the identified chemical features decreases with increasing layer depth. However, as demonstrated in Figure 5b, the alignment with perception improves. These results collectively are consistent with well-known principles vision models, where the lower layers typically capture low-level, localized features like edges and textures, while deeper layers gradually shift toward higher-level, abstract representations, such as shapes and objects[36). Nonetheless, additional investigation required to fully reveal and comprehend this potential hierarchical structure.

---

Correlation Coeficient
0.8
0.6
0.4
0.2
nCIR
ZM1
GNar
S1K
piPC08
MATSIv
MATS7V
GATS1V
Elg05 AEA(bo)
SM02 AEA(bo)
SM03 AEA(dm)
SM10 AEA(dm)
SM13 AEA(dm)
SpMin3 Bh(v)
nRCOSR
0.2
0.15
NRMSE
0.05
nCIR
ZM1
GNar
S1K
piPC08
MATSiv
MATS7V
GATS1V
Elg05 AEA((b))
SM02_AR AEA(bo)
SM03 AEA(dm)
SM10 AEA(dm)
SM13 AEA(dm))
SpMin3 Bh(())
nRCOSR
Descriptor
FoLmer
Open-POM
Figure 6: Performance of the models to predict relevant physicochemical descriptors. We com- puted Correlation and NRMSE between the predicted and actual values of descriptors. MoLFormer is able to predict 13 out of 15 physicochemical descriptors related to smell as well as or better than the Open-POM model, demonstrating high alignment with physicochemical descriptors.
0.6
Correlation Coefficient
0.2
EigbS_AEAAbo)
SMO2_AEAbo)
2
0.6
0.2
AEAldm)
SM10 AEAldm)
AEAldm)
SpMin3, Bhiv)
0.6
0.2
81012
81012
4 81012
2 81012
2 81012
Layer
Figure 7: Correlation between the actual and predicted value of physicochemical descriptors diminishes the layer depth increases.
5 Discussion
In this study, we investigated the alignment between odorant representations encoded by the MoL- Former, a self-supervised transformer model pre-trained on chemical structures, and human olfactory perception. We evaluated the alignment between these representations by analyzing the similarity between them or finding linear mapping between the representations. Additionally, we offered insights into the potential reasons behind the observed alignments by exploring relevant chemical features extracted by the model.

---

Perceptual prediction from pretrained models. We demonstrate for the first time that representa- tions extracted from pre-trained large models, solely trained on chemical structures, align closely with the perceptual representations of odorants. This finding suggests that odorant perception can be accurately predicted from chemical structures. Furthermore, we show that this model can predict a subset of physicochemical descriptors known to be relevant to offactory perception. Together, these results offer valuable predictions for chemists and neuroscientists to explore in future research.
Evaluating alignment across multiple datasets. To evaluate alignment from various perspectives, we designed three different experiments. First, we leveraged dataset with expert-provides labels for odorants. assessing the model's s ability to independently predict multi-targe binary labels for each odorant. This task did not involve variability from human participants or continuous odorant ratings. MoLFormer exhibited relatively high performance in predicting these binarized labels. Second, we used datasets containing average continuous ratings from human participants, which inherently present more challenges due to variability among non-expert participants' ratings. Our evaluation revealed that while all models performed poorly on this task, MoL.Frrmer performed comparably to supervised models. Lastly, we evaluated direct similarity scores between odorants from two datasets, examining the alignment between human- -rooiide similarity scores and those computed from the representations encoded by models. MoLFormer showed high alignment, highlighting its ability to predict similarity between odorants rather than relying on auman-made descriptors. This suggests that pre-defined descriptors for describing odorants may need to be more carefully chosen, and models trained with these descriptors might not accurately reflect the true similarity between odorants.
Reduction in alignment with physicochemical descriptors across layers of the models. We conducted a complementary analysis to identify potential reasons underlying the observed perceptual alignment. Our focus was on the subset of features previously identified as significant for decoding olfactory perception from chemical structures. Our findings indicate that MoLFormer representations exhibit a high degree of alignment with these features. While most features show strong alignment, a few demonstrate less alignment (such as nRCOSR). These results collectively suggest that while these features are important, their significance varies. Additionally, our analysis of the predictability of these features across the different layers of the model shows that as we go through the layers, we observe a decrease in alignment with physicochemical descriptors despite with perception. This observation aligns well with established principles in vision models, where lower layers have been shown capture lowelevel, local features such edges and textures, while progressively transitioning to align with higher-level, abstract representations, such as shapes and objects, in deeper layers [36]. However, further exploration is needed to fully uncover and understand this potential hierarchy.
increase in alignment
Limitations. Our work is perhaps best understood in the context of its limitations. We do not directly take into consideration the intensity or concentration of each individual molecule within a mixture during the encoding of odorants. Incorporating these intensity factors in future work could potentially improve the alignment. Additionally, our research was constrained by the available datasets, which typically lack sufficient variations in different odorants, particularly for continuous rating regression tasks. Furthermore, we only considered the average rating scores and did not evaluate the alignment on per-subject basis.
Future Work. We aim to leverage these findings to develop improved models of olfactory perception. Specifically, we plan to utilize unsupervised models trained exclusively on chemical structures to identify which chemical features are crucial for predicting perception, thereby avoiding the introduction of biases from human subjective perception. Additionally, we intend to investigate the mechanisms underlying olfactory perceptions decoded from chemical features. The observed aligmment trends across different layers of the model may provide key insights into this process. Finally, evaluating representational alignment between the extracted representations from transformers trained on chemical structures and fMRI data from the brain can provide deeper insights into the underlying mechanisms of olfactory perception.
Acknowledgement
This work was supported by the Knut and Alice Wallenberg Foundation, Swedish Research Council, ERC AdV BIRD 88480, and ERC Syn D2Smell 18977 The authors would like to thank Johan Lundström for helpful discussions.
10

---

References
[1] Bruno A Olshausen and David Field. Sparse coding of sensory inputs. Current opinion in neurobiology, 14(4):481-45) 2004.
[2] Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C Love, Erin Grant, Jascha Achterberg, Joshua B Tenenbaum, et al. Getting aligned on representational alignment. arXiv preprint arXiv:2310.13013 2023.
[3] Giorgio Ganis, William L Thompson, and Stephen M Kosslyn. Brain areas underlying visual mental imagery and visual perception: an fmri study. Cognitive Brain Research, 20(2):22--411, 2004.
[4] Angela D) Friederici. The cortical language circuit: from auditory perception sentence comprehension. Trends in cognitive sciences, 16(5):262-268, 2012.
[5] Subba Reddy Oota, Manish Gupta, Raju S Bapi. Gael Jobard, Frédéric Alexandre, and Xavier Hinaut. Deep neural networks and brain alignment: Brain encoding and decoding (survey). arXiv preprint 2XXi:23377110445 2023.
[6] Jerry Tang, Meng Du, Vy Vo, Vasudev Lal, and Alexander G Huth. Brain encoding models based on multimodal transformers can transfer across language and vision. arXiv preprint arXiv:2305. 12248, 2023.
[7] Dota Tianai Dong and Mariya Toneva. Interpreting multimodal video transformers using brain recordings, In ICLR 2023 Workshop on Multimodal Representation Learning: Perks and Pitfalls, 2023.
[8] Charles F Cadieu, Ha Hong. Daniel LK Yamins, Nicolas Pinto.
Diego Ardila, Ethan A Solomon, Najib J Majaj, and James DiCarlo. Deep neural networks rival the representation of primate cortex for core visual object recognition. PLoS computational biology, 10(12):11003633, 2014.
[9] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy. Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition most brain-like? BioRxiv, page 407007, 2018.
[10] Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature human behaviour, 7(3):430-441, 2023.
[11] Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Deep language algorithms predict semantic comprehension from brain activity. Scientific reports, 12(1):16327, 2022.
[12] Daniel LK Yamins and James DiCarlo. Using goal-driven deep learning models understand sensory cortex. Nature neuroscience, 19(3):356-36 365,, 2016.
[13] Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in machines) with natural lunguage-processin (in the brain). Advances in neural information processing systems, 32, 2019.
[14] Juliette Millet, Charlotte Caucheteux, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, Jean-Remi King, al. Toward realistic model of speech processing in the brain with self-supervised learning. Advances in Neural Information Processing Systems, 35:33428-33443, 2022.
[15] Mario Pannunzi and Thomas Nowotny. Odor stimuli: not just chemical identity. Frontiers in physiology, 10: 14:1428. 2019.
[16] Brian Lee, Emily J Mayhew, Benjamin Sanchez-Lengeling, Jennifer Wei, Wesley Qian, Kelsie Little, Matthew Andres, Britney B Nguyen, Theresa Moloy, Jane K Parker, et al. A principal odor map unifies diverse tasks in human olfactory perception. BioRxiv, pages 2022-09, 2022.
11

---

[17] Aharon Ravia, Kobi Snitz, Danielle Honigstein, Maya Finkel, Rotem Zirler, Ofer Perl, Lavi Secundo, Christophe Laudamiel, David Harel, and Noam Sobel. measure of smell enables the creation of olfactory metamers. Nature, 588(7836): 11803666:111133333 2020.
[18] Kobi Snitz, Adi Yablonka, Tali Weiss, Idan Frumin, Rehan M Khan, and Noam Sobel. Predicting odor perceptual similarity from odor structure. PLoS computational biology, 90)):e1033144, 2013.
[19] Andreas Keller, Richard C Gerkin, Yuanfang Guan, Amit Dhurandhar, Gabor Turu, Bence Szalai, Joel D Mainland, Yusuke Ihara, Chung Wen Yu, Russ Wolfinger, et al. Predicting human olfactory perception from chemical features of odor molecules. Science, 355(6327):220-826, 2017.
[20] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[21] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 07258, 2021.
[22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 2020.
[23] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for supersuue video pre-training. Advances in neural information processing systems, 35:10078- 2022.
[24] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[25] Jerret Ross, Brian Belgodere, Vijil Chenthamarakshan, Inkit Padhi, Youssef Mroueh, and Payel Das. Large-scale chemical language representations capture molecular structure and properties. Nature Machine Intelligence, ((112)::26-1222 2022.
[26] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. International conference on machine learning. pages 1263-1272. PMLR, 2017.
[27] Leffingwell and Associates. Database of perfumery materials and performance, 2001.
[28] GoodScent. The good scents company.
[29] Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, et al. Pubchem 2019 update: improved access chemical data. Nucleic acids research, 470D1):D1102-D1100 2019.
[30] John J Irwin and Brian K Shoichet. Zinc- free database of commercially available compounds for virtual screening. Journal of chemical information and modeling, 45(1):177-182, 2005.
[31] Jason B Castro, Travis J Gould, Robert Pellegrino, Zhiwei Liang, Liyah A Coleman, Famesh Patel, Derek S Wallace, Tanushri Bhatnagar, Joel Mainland, and Richard Gerkin. Pyrfume: A window to the world's olfactory data. bioRxiv, pages 2022-09 29, 2022.
[32] Aryan Amit Barsainyan, Ritesh Kumar, Pinaki Saha, and Michael Schmuker. Openpom, 2023.
[33] Vivek Sagar, Laura K Shanahan, Christina M Zelano, Jay A Gottfried, and Thorsten Kahnt. High-precision mapping reveals the structure of odor coding in the human brain. Nature neuroscience, pages 1-8, 2023.
[34] Andreas Keller and Leslie Vosshall. Olfactory perception of chemically diverse molecules. BMC neuroscience, 17:1-17, 2016.
12

---

[35] Andrea Mauri. alvadese: A tool to calculate and analyze molecular descriptors and fingerprints. Ecotoxicological QSARs, pages 801-820, 2020.
[36] MD Zeiler. Visualizing and understanding convolutional networks. In European conference on computer vision/arXiv, volume 1311, 2014.
13

---

A Noise Ceiling
In order evaluate the quality data and the upper limit of the models, we computed the noise ceiling (Equateion 1, 2) for the Sagar and Keller datasets as these are the only ones that have multiple evaluators for each odorant, and those are publicly available. The results show that we have the average noise ceilings of 0.28 + 0.1 for the Keller dataset and 0.7 = 0.05 for the Sagar dataset (Table S.I, S.2). The results show that the data of the Sagar dataset is less noisy, and there still room for the models to increase the alignment. However, the Keller dataset alignment results are relatively close to the noise ceiling value.
rj = Reepoonss from Participant j. Mean Response across participants)
(1)
1 N
Noise Ceiling
T3
Table S.1: Noice ceiling per descriptor for Sagar dataset
Descriptor
Bakery
Burnt
Cool
Decayed
Fishy
Floral
Fruity 0.79
Intensity 0.75
Noise Ceiling
0.68
0.70
0.68
0.72
0.75
0.73
Descriptor
Musky
Pleasantness
Sour
Spicy 0.66
Sweaty 0.62
Sweet 0.72
Warm 0.61
Noise Ceiling
0.71
0.74
0.66
Table S.2: Noice ceiling per descriptor for Keller dataset
Descriptor
Acid
Ammonia
Bakery
Burnt
Chemical
Cold 0.17
Decayed
Noise Ceiling
0.21
0.21
0.32
0.27
0.27
0.29
Descriptor
Familiarity
Fish
Flower
Fruit
Garlic
Grass 0.25
Intensity
Noise Ceiling
0.33
0.21
0.26
0.37
0.31
0.53
Descriptor
Musky
Pleasantness
Sour
Spices 0.24
Sweaty 0.24
Sweet 0.41
Warm 0.17
Noise Ceiling
0.22
0.52
0.23
B Representational Similarity Matrix (RSM)
In order to better visualize how the models and humans represent different odors, we visualized representational similarity matrices for humans participants, Open-Pom and MoLFormer across pairs of odorants for Ravia dataset in Figure S.1. The white cells show the pair of odorants for which no similarity score is available. MC-odorants corresponding to each mixture are provided in Table S.3
C t-SNE Visualizations
We also reduce representations of GS-LF dataset extracted from MoLFormer and Open-POM datasets using t-SNE (Figure S.2).
D Fine-tuned MolFormer
We fine-tuned MoLFormer using GS-LF dataset which is large and inclusive dataset of odorants.
Then we extracted representations for the datasets and tasks. Figure S.3 shows ROC-AUC curve for GS-LF dataset. Figure S.4 demonstrates the results for the continuous rating prediction tasks for Keller and Sagar datasets, and FigureS.5 shows the results for RSA for Ravia and Sni tz datasets.
14

---

100
1.00
1.0
80
0.75
0.9
0.50
0.8
60
0.25
0.7
40
0.00
0.6
20
0.25
0.5
0.50
0.4
(a) Human
(b) MoLFereer
(c) Open-POM
Figure S.I: RSM for different pairs odorants for Ravia dataset.
meaty
savory
beefy
meaty
ethereal cognac
savory
beefy
Principal Component 2
termented alcoholic
Principal Component
roasted
floral
muguet
lavender
floral
Jasmin
muguet
lavender
jasmin
ethereal
cognac
termented
alcoholic
Principal Component 1
Principal Component 1
(a) MoLFormer
(b) Open-POM
Figure S.2: t-SNE Visualization of odorant representations encoded by different models on the GS-LF dataset using the figure layout suggested by [16]. reduced dimensionality using t-SNE. Areas dense with molecules that have broad category labels (floral, meaty, ethereal) are shaded, while areas dense with narrow category labels are outlined. Formrr captures the perceptual relationship between different odorants in its representation space, despite not being explicitly trained for this purpose.
E Decoding chemical features
In this section, we present the results for predicting physicochemical descriptors from odorants for each dataset separately (Figure S.6-S.10). We observe that MoLFormer can better predict physicochemical descriptors in most cases.
F Decoding chemical features across layers of MoL.ormer
We also show how the alignment between physicochemical descriptors changes across layers of the model for each dataset separately. As it is shown, alignments decrease across layers. This might be because the first layers extract more low-level information while deeper layers, extract more high-level features.
15

---

Table S.3: MC-odorants corresponding indices in RSM presented Figure S.1
Index
MC-Odorant
0
1 15500777774444000777770994444443
2 224455590001100:077::555111666
3
4
24125755117711778822773333333333
5
512776:81487777688887777771
7
526:26331;1140:110002
8
9
5562233889400226611988622999
11
55694077601111186:6166777
12
13
65511155000000777777776677777814
62551,5656690111995555333
15
64433399777253314405555311122222
16
3544:228337111978833314444118
17
55449900884100000077770770991133
18
7194:3202666111770677777666666
19
719::994006660111122226677766666
20
7410:240:93009:8633
21
7410:26627366024::770..4444
22
23
75513314..313222300077:111100000
24
25
26
27
16

---

True Positive Rate
0.5
0
0.5 False Positive Rate
(AUC 0.00)
Figure S.3: ROC curve for the linear classifier trained on GS-LF representations extracted from the fine-tuned MoLFormes Each curve corresponds to separate test split, with the thicker curve representing the average performance across all splits.
Corelatton Coefitient
Correlation Coeffcient
04
0.2
61
Aclo
Ammonis
Salery
burn
Chemical
Cold
Decayed
Fumillarty
Fa
Flower
Garlc
Intersity
Musly
Plcasntnee
Sour
Sweet
Swoet
Wam
Wood
Intersitt
Plesrantaess
Burn
Dacayed
Musky
Fruty
Sweety
Hora
Warm
Bakery
0.2
0.15
MRMSE
NRMSE
0.1
0.05
Ammmono
Burrt
Chemical
Cald
Decayed
Familltrtty
Fish
Flowor
Garlic
Immersity
Mosky
Wecasmttne tnees
Sour
Sweett
Warm
Wond
Internity
Plearantness
Burn
Denaay ynd
Musky
Fruitty
Sweety
Floral
Swave
Warm
Bakery
Descriptor
Descriptor
MoLFormer
Open-POM
CAM
Fine-tuned MoLFormer
MotFormer
Open-POM
DAM
Fine-tuned MoLFormer
Keller
(b) Sagar
Figure S.4: Performance of the models to predict continuous ratings per descriptor. We computed Correlation and NRMSE between predicted and actual ratings per perceptual descriptor. Fine-tuned MoLFormer shows slightly worse performance in predicting continues ratings.
Correlation Coefficient
0.7
0.5
0.3
Snitz 2013
Ravia 2020
Dataset
MoLFormer
DAM
Open-POM
Finetuned MoLFormer
Figure S.5: Representational similarity analysis for Snitz and Ravia datasets. Fine-tuned MoL- Former performs on par with MoLFormer.
17

---

0.8
0.6
Correlation
0.4
0.2
noIR
ZM1
GNar
S1K
piPCO8
MATS1v
MATS7V
GATS1v
Eig05 AEA((o)
SM02 AEA((b)))))
SM03 AEA(dm)
SM10 AEA(dm)
SM13 AEA(dm)
SpMin3 Bh(v)
nRCOSR
0.2
NRMSE
S..0
0.1
0.05
8005
ncR
ZM1
GNar
S1K
piPC08
MAT51v
MAT57V
GATS1v
Eig05 AEA(bo)
SMO2 SM02_AEAN AEA(bo))
SM03 AEA(dm)
SM10 AEA(dm)
SM13 AEA(dm)
SpMin3 Bh(v)
nRCOSR
Descriptor
MoLFormer
Open-POM
Figure S.6: Performance of the models to predict relevant physicochemical descriptors for
Keller dataset. MolFormer performs slightly better than Open-POM in predicting descriptors.
Correlation Coefficient
0.8
0.6
0.4
0.2
nCIR
ZM1
GNar
S1K
piPCO8
MATS1v
MAT57V
GATS1v
Eig05 AEA(bo)
SM02 AEA(bo)
SM03 AEA(dm)
SM10 AEA(dm)
SM13 AEA((mm)))
SpMin3 Bh(v)
nRCOSR
NRMSE
0.2
0.1
0.05
nnR
ZM1
GNar
S1K
piPC08
MATS1v
MAT57v
GATS1v
Eig05 AEA(bo)
SMO2 SM02_AEARE AEA(bo)
SM03 AEA(dm)
SM10 AEA(dm)
SM13 AEA(dm)
SpMin3 Bh(v)
nRCOSR
Descriptor
MoLFormer
Open-POM
Figure S.7: Performance of the models to predict relevant physicochemical descriptors for Sagar
dataset. MolFormer performs slightly better than Open-POM in predicting descriptors.
18

---

Correlation Coefficient
0.8
0.6
0.4
0.2
noIR
ZM1
GNar
S1K
piPCO8
MATS1v
MATS7V
GATS1v
Eig05 AEA(bo)
SM02 AEA((b))))
SM03 AEA(dm)
SM10 AEA(dm)
SM13 AEA(dm)
SpMin3 Bh(v)
nRCOSR
0.2
0.15
NRMSE
0.1
0.05
0
ncR
ZM1
GNar
S1K
piPC08
MAT51v
MAT57v
GATS1v
Eig05 AEA(bo)
SMO2 SM02_AEAN AEA(bo))
SM03 AEA(dm)
SM10 AEA(dm)
SM13 AEA(dm)
SpMin3 Bh(v)
nRCOSR
Descriptor
MoLFormer
Open-POM
Figure S.8: Performance of the models to predict relevant physicochemical descriptors for Ravia
dataset. MolFormer performs slightly better than Open-POM in predicting descriptors.
Correlation Coefficient
0.8
0.6
0.4
0.2
nOIR
GNar
S1K
piPCO8
MATS1v
MAT57V
GATS1v
Eig05 AEA(bo)
SM02 AEA(bo)
SM03 AEA(dm)
SM10 AEA(dm)
SM13 AEA(dm))
SpMin3 Bh(v)
nRCOSR
0.2
0.15
NRMSE
0.1
0.05
nnR
ZM1
GNar
S1K
piPC08
MATS1v
MAT57v
GATS1v
Eig05 AEA(bo)
SM02 SM02_AEARE AEA(bo)
SM03 AEA(dm)
SM10 AEA(dm)
SM13 AEA(dm)
SpMin3 Bh(v)
nRCOSR
Descriptor
MoLFormer
Open-POM
Figure S.9: Performance of the models to predict relevant physicochemical descriptors for Snitz
dataset. MolFormer performs slightly better than Open-POM in predicting descriptors.
19

---

Correlation Coefficient
0.8
0.6
0.4
0.2
nCIR
ZM1
GNar
S1K
piPC08
MATS1v
MATS7v
GATS1V
Eig05 AEA(bb)
SMO2 AEA(bb))
SM03, AEA(dm)
SM10 AEA(dm)
SM13 (..AEA((m))))
SpMin3 Bh(v)
nRCOSR
0.2
0.15
NRMSE
0.1
0.05
0
ncIR
ZM1
GNar
piPC08
MATS1v
MATS7V
GATS1v
Eig05 AEA(bo)
SM02 AEA(bo)
SM03 AEA(dm)
SM10 AEA(dm)
SM13 AEA(dm)
SpMin3 Bh(v)
nRCOSR
Descriptor
MoLFormer
Open-POM
Figure S.10: Performance of the models predict relevant physicochemical descriptors for
GS-LF dataset. MolFormer performs slightly better than Open-POM in predicting descriptors.
nCIR
ZM1
GNar
S1K
piPC08
0.6
0.2
Correlation Coefficient
MATS1V
MATS7V
GATS1V
(gg05_AEA(bo)
SM02_AEA(bo)
1
0.6
0.2
SM03 (EA{dm)
(M10_AEA(dm)
(EA(d))
SpMin3_Bhil
nRCOSR
1
0.6
0.2
2 4 8 1012
2 4 6 8 1012
2 6 1012
2 4 6 81022
6 8 1012
Layer
Figure S.11: Correlation between the actual and predicted value of physicochemical descriptors Keller dataset diminishes as the layer depth increases.
20

---

nCIR
ZM1
GNar
S1K
piPC08
1
0.6
Correlation Coefficient
0.2
MATS1v
MATS7v
GATS1V
[ig05_AEA(bo)
SM02_AEA(bo)
1
0.6
0.2
SM03 (
(M10_AEA(dm)
SM13 (113_AAEA(d))
SpMin3_Bhi Bh(v)
nRCOSR
1
0.6
0.2
2
8 1012
2 4 6 8 1012
2 6 8 1012
2 4 8112
6 8012
Layer
Correlation between the actual and predicted value of physicochemical descriptors in Sagar dataset diminishes as the layer depth increases.
Figure S.12:
nCIR
ZM1
GNar
S1K
piPC08
0.6
Correlation Coefficient
0.2
MATS1V
MATS7V
GATS1V
Eig05_AEA(bo)
SM02_ AEA(bo)(()))
1
0.6
0.2
SM03 30A(dmm
SM10 AEA(d)
SM13 AMA3))
SpMin3_Bhrs Bh(v)
nRCOSR
1
0.6
0.2
2
1012
2 4 6 8 1012
2 1012
2 4 8012
6 8 1012
Layer
Figure S.13: Correlation between the actual and predicted value of physicochemical descriptors Ravia ia dataset diminishes as the layer depth increases.
21

---

nCIR
ZM1
GNar
S1K
piPC08
1
0.6
Correlation Coefficient
0.2
MATS1v
MATS7V
GATSIV
Big05_AEA(bo)
SMO2_AEA(bo)
1
0.6
0.2
SM03 (MM0_AAA(dm)
SM13 (M)
SpMin3_BMh [B([v)
nRCOSR
1
0.6
0.2
2
1012
4 6 1012
2 1012
2 4 81012
6 8 1012
Layer
Figure S.14:
Correlation between the actual and predicted value of physicochemical descriptors in Snitz dataset diminishes as the layer depth increases..
[t]0.75
nCIR
ZM1
GNar
S1K
piPC08
1
0.6
Correlation Coefficient
0.2
MATS1v
MATS7V
GATS1V
Sig05_AEA(bo)
SMO2 SM02 AEA(bo)
1
0.6
0.2
SM03 AEA(dm)
SM10_AEA(cr AEA(dm)
SM13 AEA(dm)
SpMin3 Bh(v)
nRCOSR
1
0.6
0.2
2
1012
4 6 1012
1012 Layer
2 4 81012
6 8 8012
Figure S.15: Correlation between the actual and predicted value of physicochemical descriptors GS-LK dataset diminishes as the layer depth increases.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the limitations of the current work in the penultimate paragraph of the discussion section.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We fully disclose all the information needed to reproduce the main experimen- tal results of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The parts of main experiment is uploaded and the whole code is available through the anonymized link provided in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce
cemiionooliyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / /ttpps.ccpppliiiiimm///tttp/nttc wwtppccppcc////ccccccccccccccclt http//uudeeeeeeeee https//ooSSSuuiiseeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars suitably and correctly defined.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe sufficient information on the computer resources needed to produce the experiments in the supplementary material section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /ttps://neurip. cc/ppblicc/thicchuiiee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original owners of assets (e.g., code, data, models), used in the paper, properly credited.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing. Although we have used datasets that involve human subjects, we did not collect those data and used the datasets that are already approved and their details are publicly available in the original work.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Although have used datasets that involve human subjects, we did not collect those data and used the datasets that are already approved and publicly available.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "bUNxSTevCv", "reviewer": "Reviewer_Y4G5", "summary": "This paper investigates the ability of MOLformer, a model trained on SMILES strings in a BERT-like fashion, to predict the human assessment of odors. The assessments were based on natural language descriptors of odors by human experts, ratings on different NL descriptors by naive subjects, and finally, on similarity ratings between odors.  It compares MOLformer, which was trained in an unsupervised manner, to an engineered model (DAM) and a supervised model Open-POM. Despite being trained in an unsupervised manner to represent molecules, MOLformer holds its own against the supervised models. In addition, MOLFormer shines when evaluated using an RSA technique, suggesting that MOLformer captures the relationships between molecular odors better than the other models.", "questions": "See my questions above:\n1. variance captured by PCA\n2. How are the physicochemical descriptors derived?\n3. Try UMAP or t-SNE", "limitations": "These are discussed.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "+ Assessing models representations of odors is an understudied area. I believe this is the first work to do this.\n+ quite a few datasets were used, providing binary, rating scale, and similarity judgment data. These might be the only ones available\n+ The results are relatively convincing", "weaknesses": "+ Despite a rather systematic reporting of models and datasets, some things were left out of the descriptions. For example:\n\t- Apparently the GS-LF datasets features are binary. I assume this means that that particular descriptor was used by at least 1 judge. So is it the case that if multiple judges used the same descriptor, the results were still recorded as binary?\n\t- how dimensionality was reduced. I assume PCA. How much variance was captured in 20 PCs?\n\t- There were apparently descriptors in the similarity dataset as well (line 123). These weren’t described.\n\n+ the references in the introduction were weird. See below.\n\nTypos, comments:\n\nThe references you use in the introduction seem rather bizarre. Two Damasio references for encoding sensory input into a high-dimensional space?\nReference #4 seems out of place in this context.\nReferences 8-10 - you have no primary references here; for example, most would reference DiCarlo’s lab here.\nReference #12 is weird. Why not reference the original transformer model? And while it’s true that for language, they are self-supervised, the original use of them for vision was supervised. It is only later that self-supervised vision models (e.g., the Masked Autoencoder) came about. \n\nline 53: Olfactory is mis-spelled! \n\nFigure 2: You might try using UMAP or t-SNE here instead of (or in addition to) PCA.\n\nIn your results section, you never say what the train-test split was (i.e., 80/20?)\n\nI was thinking - they should use RSA - and then you did! This is a very interesting result because, rather than just being able to predict human results, the model is better at both of the others in capturing the *relationships* between the odors, just from their chemical descriptions.\n\t\n- Figure 6: Explain why didn’t you evaluate DAM here.\n- You didn’t describe how the physicochemical descriptors were obtained. \n\nLine 284: In section 4.4, it wasn’t clear that for this analysis, you were trying to discover the reasons for the alignment. You should say something about that when introducing that analysis.\n\nlines 289-292: It makes sense to me that the low-level features of the molecules would be best encoded in earlier layers and not in later layers, where more abstract features would be encoded. This seems to fit well with the way vision models work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and for the time spent reviewing our paper. We address the questions (Q), weaknesses (W), and comments (C) raised by the reviewer as follows:\n\n\n* **(W1) Description of GS-LF dataset**: GS-LF dataset is a multi-label binary dataset, where each data point (the “odorant”) has a set of labels (the “descriptors”) evaluated by a single expert evaluator (the “judge”). As such, no odorant has ratings from multiple judges, but the single judge was asked to assign labels to each odorant chosen among 138 descriptors. We clarified this point in the updated version of the paper.\n\n* **(W2 and Q1) Dimensionality reduction method and variance captured**:  In this work, we use PCA to reduce the dimensionality of the representations. We report the amount of variance explained by the first 20 PCs in the following table. The results show that our 20 PCs capture the majority of the variance of the data.\n\n| Dataset / Method | Keller | Sagar | GS-LF|\n| -------------|:-------------:|:-------------:|:-------------:|\n|MoLFormer|${0.62 \\pm{ 0.002}}$ |${0.67\\pm{0.004}}$|${0.59\\pm{0.000}}$|\n|Open-Pom|${0.70 \\pm{ 0.002}}$|${0.74\\pm{0.003}}$|- (trained end-to-end)|\n|Fine-tuned MoLFormer|${0.75\\pm{ 0.001}}$|${0.78 \\pm{ 0.003}}$|${0.75\\pm{0.03}}$|\n\n\n\n* **(W3) Descriptors in the similarity dataset (line 123)**: You are correct. Line 123 is not accurate. Similarity datasets do not include descriptors, and in this case, $y_i \\in [a, b]^{n  \\times 1}$. We have corrected this line in the updated version of the paper.\n* **(W4) References in the introduction**: We thank the reviewer for this comment. As pointed out as well by reviewer UQYS, we have replaced Damasio references by the work of DiCarlo & Cox, (2007) and Olshausen & Fields, (2004). We agree that reference [4] is out of place: in line 34 it should read “(...) impressive performance in various tasks such as control [4], image [13], (...)”. We have also added the reference to the work of Vaswani et al. (2017) as the original reference for the transformer. Moreover, we agree that [13] uses a supervised objective to train the transformer, and we also added the reference to He et al. (2022) on MAEs to the discussion on self-supervised image transformers.\n* **(C1, Q3) UMAP and T-SNE visualizations**: We add the T-SNE figures to the attached rebuttal PDF. We also ran experiments with UMAP and will add it to the appendix of the paper (we don't include in the attached PDF due to lack of space). Overall, we found these figures to be harder to interpret than the PCA figure, which was already suggested in [20].\n* **(C2) Train-test split**: In all evaluations in our work, we use a 80-20% train-test split, with 30 different randomly-seeded partitions. Moreover, we used nested cross-validation to estimate the hyper-parameters of the linear models. We have added this information to the beginning of Section 4 of the updated version of the paper. \n* **(C3, Q2) Physio-chemical descriptors and Figure 6**: We extract the physio-chemical descriptors using the AlvaDesc software. These physio-chemical descriptors are those that are used as input features of the DAM model proposed by Snitz et al. (2013). As such, in Figure 6, we don’t evaluate the prediction capabilities of the DAM model, as the targets would be the same as its input features. We have clarified this point in Section 4.4 of the updated version of the paper.\n* **(C4) Motivation of section 4.4 (line 284)**: We have clarified the motivation for the additional analysis in the updated version of the paper.\n* **(C5) Alignment with low-level physio-chemical descriptors**: We thank the reviewer for the insightful comment on the results regarding the alignment of the model with physio-chemical descriptions, which we fully agree with. The reviewer pointed out correctly that in vision models, low-level features are extracted in the early layers of the model, and the same behavior can be observed here. We added this insightful interpretation in our final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RgzILsscZW", "reviewer": "Reviewer_UQYS", "summary": "This paper took a transformer model that was pre-trained on general chemical structures and tested whether the resulting model representations aligned with human olfactory perception. Specifically, the authors used a transformer for chemical structures called MoL-Former. MoL-Former was trained via masked token prediction loss. The representations from MoL-Former can predict odor labels from experts and odor ratings from non-experts (although the correlations between the model and rating scores is quite low). The representations are also highly correlated with physiochemical descriptors that are related to human olfactory perception.\n\nThe authors also compare the MoL-Former self-supervised transformer model to other baseline models (Open-POM, which is supervised with odor labels; and DAM, which predicts similarities between pairs of odors). MoL-Former generally performs comparably to these models, even though the baseline models are supervised and MoL-Former is self-supervised.", "questions": "Does the input to MoL-Former (and the baseline comparison models) correspond to olfactory receptors? If the input data from olfactory receptors does not match the input data to MoL-Former, then it's not actually completing the same task, and it becomes much harder to argue that the model can \"smell like humans.\"", "limitations": "See weaknesses above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Transformer models have been revolutionary in modeling sensory modalities of vision and audition, but olfaction is comparatively understudied. If olfaction can also be explained with transformer models, it indicates that the domain-general learning mechanisms of transformers can explain sensory processing in the brain more broadly. Put more simply, the results support a universal, domain-general learning mechanism for sensory processing across modalities. This, in and of itself, is an interest finding worthy of publication.\n\nAnother strength of the paper is that it relies on self-supervision to pre-train the transformer. This is important both practically and theoretically. From a practical standpoint, acquiring labelled datasets is challenging, whereas acquiring large-scale unlabelled datasets is becoming increasing more feasible. From a theoretical standpoint, this provides a better comparison to human olfaction than supervised models. Most human olfactory learning (and sensory learning more generally) is unsupervised. We are rarely given labels for odors, and animals are able to learn about odors without any verbal labels.\n\nFinally, the inclusion of baseline models for comparison to the unsupervised transformer is important for interpreting their results. By comparing to OpenPOM and DAM, the author demonstrate that a self-supervised transformer can perform comparably to previous supervised models.", "weaknesses": "Major\n- Section 4.1 Expert labels classification was difficult to follow. How was dimensionality reduced to 20, through PCA perhaps? How was the same procedure applied to DAM, given that DAM already has less than 20 features? Why wasn't the visualization in Figure 3 also performed on DAM?\n- Sections 4.2 & 4.3 need noise ceilings in order to be interpretable. The authors analyzed each model's predicted average rating with the actual average rating from participants. However, it is unreasonable to expect a model to be more correlated with average ratings than a typical individual human participant. The noise ceiling computes the average correlation between the human participants and mean performance to estimate a reasonable \"ceiling\" performance that could be expected from models. When the data are very noisy, the noise ceiling will be low (and the human benchmark will generally be less useful because of its low precision), and when the data are not noisy, the noise ceiling will approach 1.\n\nMinor\n- The first sentence of the paper references Damasio, 1989 and Meyer & Damasio, 2009. These are both fantastic papers, but they don't really fit the sentence where they are referenced. References like DiCarlo & Cox, 2007 and Olshausen & Fields, 2004 strike me as more relevant.\n- A representational similarity matrix (RSM) would be a great visualization for Section 4.3. It would be really interesting to visualize how the patterns of similarity between different odorants compared across models and humans. You don't need to add it, but it would be helpful for readers to visualize how the models and humans are representing the different odors.\n- The description of SMILES is a bit hard to follow. Could you provide an example and explain how (or if) the inputs to Open-POM, DAM, and MoL-Former differ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and for the time spent reviewing our paper. We now address the questions (Q) and weaknesses (W) raised by the reviewer:\n* **(W1.1) Dimensionality Reduction**: In this work, we use PCA to reduce the dimensionality of the representations for MoLFormer and Open-POM. You are correct that DAM does not require PCA. We have clarified this in the updated version of the document.\n* **(W1.2) DAM on Figure 3**: We thank the reviewer for the comment. We have added the t-SNE visualization of the DAM model **(Figure II. c)** in the general rebuttal document. We also added such visualizations using UMAP and PCA in the updated version of the paper. (We didn’t include them in the general rebuttal document due to the lack of space.)\n* **(W2) Noise ceilings**: We thank the reviewer for the suggestion to use noise ceilings in Section 4.2 and Section 4.3. We have computed the noise ceilings for the \"Sagar\" and \"Keller\" datasets (as these are the only ones that have multiple evaluators for each odorant, and those are publicly available.), as shown in the following tables. The results show that we have noise ceilings of **$0.28\\pm{0.1}$** for the Keller dataset and of **$0.7\\pm{0.05}$** for the Sagar dataset. The results show that the data of the Sagar dataset is less noisy, and there is still room for the models to increase the alignment. However, the Keller dataset alignment results are relatively close to the noise ceiling value. We will also add the results divided by noise ceiling value in the appendix of the paper.\n\n**Sagar dataset**:\n\n| Bakery | Burnt  | Cool | Decayed | Fishy|Floral | Fruity | Intensity | Musky | Pleasantness | Sour| Spicy | Sweaty | Sweet | Warm |\n|:-------------:|:-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|-------------:|\n| 0.68  | 0.70 | 0.68| 0.72| 0.75 |0.73 | 0.79| 0.75| 0.71   |  0.74| 0.66|  0.66  | 0.62|0.72| 0.61|\n\n\n\n\n\n\n\n\n\n\n\n**Keller dataset**:\n|Acid|Ammonia|Bakery|Burnt|Chemical|Cold|Decayed|Familiarity|Fish|Flower|Fruit|Garlic|Grass|Intensity|Musky|Pleasantness|Sour|Spices|Sweaty|Sweet|Warm|\n|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|\n|0.21|0.21|0.32|0.27|0.27|0.17|0.29|0.33|0.21|0.26|0.37|0.31|0.25|0.53|0.22|0.52|0.23|0.24|0.24|0.41|0.17|\n\n\n\n* **(W3) Introduction references**: We do agree that DiCarlo & Cox, 2007 and Olshausen & Fields, 2004, are more suitable for the purpose of this sentence and, as such, we have followed the reviewer’s suggestion and replaced them in the updated version of the paper.\n\n* **(W4) RSM analysis**: We thank the reviewer for the analysis suggestions. We provided the RSM figure only for the Ravia dataset and for MoLFormer, OpenPom and Human Perception in **Figure I** of the general rebuttal document, and we will add that for both the Ravia and Snitz datasets and also for the DAM model in the updated version of the paper (we don't include Snitz dataset and DAM model in the pdf due to the lack of space). We also added a table showing the correspondence of labels on axes with the real CIDs in the paper appendix. \n* **(W5) SMILES/Input for the different models**: The SMILES representation encodes the three-dimensional structure of a chemical (e.g., a molecule) into a short ASCII string. It does so by first performing a depth-first tree search over the chemical graph, adding the elements, bonds, and rings, and subsequently removing hydrogen atoms and breaking cycles (e.g., water is represented by “O” in the SMILES notation). We use SMILES representations of odorants as input to the MoLFormer. For example, toluene is written as a string like \"Cc1ccccc1\". For the Open-POM, each odorant is represented as a graph, where each atom is represented as a node feature vector (e.g., valence, hydrogen count, atomic number…), and each bond is represented as a edge feature vector (e.g., degree, aromaticity, whether it is a ring or not). Finally, for the DAM model, we use a set of predefined physio-chemical features, which we extract using the AlvaDesc software. We have clarified these representations in the updated version of the document.\n\n* **(Q1) Correspondences of Inputs**: We thank the reviewer for raising this point. In our work, “smell like a human” corresponds to whether representations of odorant chemical structures extracted from learning-based models are aligned with human olfactory perception (line 37). The underlying assumption to this question is that the \"stimuli\" provided to human participants are similar to the ones provided to the models using different types of representations. We agree that there are limitations that are not captured by our representation, such as the concentration and intensity of different molecules in an odorant, and we will try to discuss this better in the updated paper. But in our study, input to the olfactory receptors are the molecules of a specific odorant, and we used the representations of the exact same molecules as the input of the model.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bRob31kDK5", "reviewer": "Reviewer_6yPX", "summary": "This paper mainly focus on the question \"Can transformers smell like humans\". By using a chemical structure transformer called MoLFormer to encode the odorant molecules, this paper proposes that the MoLFormer pre-trained representation can classify the odors of variety of molecules without careful fine-tuning. The method builds a relationship between odorant chemical strutures and their odors, and the odors are labeled by human-beings.", "questions": "1. How many models (beside MoLFormer) does the research group consider to generate odorant representations? \n2. Do the authors consider to fine-tune the MoLFormer for better performance?\n3. Besides transformer, do the authers consider more kinds of models such CNN and graph network?", "limitations": "1. It is better to do more analyses about the fine-tuning of the network.\n2. It is better to consider to include some innovation in the existing models.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper tries to use pre-trained transformer to build the relationship between chemical structures and odors, and to some extent, shows that transformer has ability to do odorant classification. \n2. This paper applies multiple kinds of databases to support the proposed argumentation, and the databases cover variety of labeling methods to label the odors. Also, experiments are included for every applied labeling methods.", "weaknesses": "1. The most important part of the odors classification system is the network, unfortunately, the paper only directly apply a well-designed transformer (MoLFormer) without any modification. \n2. It looks like this paper dose not finetune the MoLFormer using the applied databases. The paper argues that the model can achieve good performance without fine-tuning, however, it is better to include experiments to support this argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and for the time spent reviewing our paper We now address the questions (Q), weaknesses (W), and limitations (L) raised by the reviewer:\n\n* **(W2, Q2,  L1) Fine-tuning results**: We thank the reviewer for the suggestion. We present the results of fine-tuning the complete MoLFormer model in **Figure III** of the general rebuttal document and will add these results to the paper. The change compared to MoLFormer is not significant in many cases. This raises interesting questions regarding the feasibility of fine-tuning pre-trained large-scale chemical models for human olfaction (and the method to do so).\n* **(W1, L2) Model Architecture**:  We agree that the development of novel network architectures for this task is interesting but, as it stands, we believe our work responds to a different, but also very relevant question. We explore whether representations extracted from pre-trained large-scale models on chemical data are aligned with human perception. Our finding suggests that pre-trained models are highly aligned with human perception of odorants **without being explicitly trained** on it, even when compared to supervised learning methods **explicitly trained** on human olfactory perceptual data. \n* **(Q1, Q3) Models employed in our work**: In our work, we consider three types of models to generate odorant representations: (i) the MoLFormer, a self-supervised, *transformer-based*, large-scale pre-trained model of chemical data, which is not trained on human assessments of olfactory stimuli; (ii) the Open-POM, a supervised *graph neural network* model that is trained explicitly using human assessments of olfactory stimuli; (iii) the DAM model, a *feature-engineered* representation model that was specifically proposed to be used in prediction tasks of the human olfactory experience. To the best of our knowledge, Open-POM is the SOTA deep-learning model trained for prediction tasks of the human olfactory experience, and the DAM model is the SOTA feature-engineered model for prediction tasks of the human olfactory experience.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for the constructive and interesting questions and also for the positive and encouraging feedback. We did the following extra experiments, analysis, and visualizations to answer the questions raised by the reviewers.

1. We visualized representational similarity matrices (RSM) for Ravia and Snitz datasets. **Figure I** shows results for the Ravia dataset  
2. We applied two additional methods for reducing dimensionality: t-SNE and UMAP. **Figure II** shows the results obtained for t-SNE.  
3. We fine-tuned the model and visualized the results of fine-tuning in the general rebuttal document, see **Figure III**.

You can find these results in the attached document.

Meta-review: The work described in this paper demonstrates that a transformer model that was pre-trained on general chemical structures (self-supervised) forms representations that align well with human olfactory perception.

Olfaction is still understudy (compared to e.g., vision or audition) and our understanding of the neural computational processes and
representations underlying human olfactiory perception is still very limited. Thus this paper makes a welcomed contribution. The reviewers
liked the model approach, the careful comparison with mutliple datasets of human psychophysical assessments of odors, and the results. The work suggests that the domain-general learning mechanisms of transformers may explain sensory processing in the brain more broadly, which is an interesting scientific hypothesis with broad implications. Thus, the work is of substantial interest to the NeurIPS community.
Decision: Accept (spotlight)