Can Transformers Smell Like Humans?

Authors: farzaneh  taleb
miguel  vasco
antônio h ribeiro
mårten  björkman
danica  kragic
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 13:30:29

Abstract: The human brain encodes stimuli from the environment into representations that form a sensory perception of the world. Despite recent advances in understanding visual and auditory perception, olfactory perception remains an under-explored topic in the machine learning community due to the lack of large-scale datasets annotated with labels of human olfactory perception. In this work, we ask the question of whether pre-trained transformer models of chemical structures encode representations that are aligned with human olfactory perception, i.e., can transformers smell like humans? We demonstrate that representations encoded from transformers pre-trained on general chemical structures are highly aligned with human olfactory perception.  We use multiple datasets and different types of perceptual representations to show that the representations encoded by transformer models are able to predict: (i) labels associated with odorants‌‌ provided by experts; (ii) continuous ratings provided by human participants with respect to pre-defined descriptors; and (iii) similarity ratings between odorants provided by human participants. Finally, we evaluate the extent to which this alignment is associated with physicochemical features of odorants known to be relevant for olfactory decoding.

Main paper: 
 introduction: 
 The human brain receives sensory input from the environment and encodes it into a high-dimensional representation space, forming a perception of the world [1] . Recent studies have significantly improved our understanding of the underlying mechanisms of visual, linguistic, and auditory perception [2] [3] [4] . Indeed, there is a significant level of alignment between human response (from neuron to behavior) and activations of deep neural networks when provided with the same stimuli [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] .Despite these recent advances, human olfactory perception remains an under-explored topic. There is no single organizing principle that determines the dimensions of odor space, making the characteriza-Figure 1 : Evaluating representational alignment between human and pre-trained transformers. Human participants are stimulated with two odorant substances and asked to rate the perceptual similarity between them (Left). We encode representations of the same pair of odorants using MoLFormer and compute the similarity between pairs of representations (Right). Finally, we measure the alignment between the two systems. tion of odor perception and its relation to chemical compounds an open and complex problem [15] . A lack of universally accepted methods to describe odorants either quantitatively or qualitatively makes this problem even more challenging. There are very few studies that have explored the mapping of chemical structures to olfactory perception [16] [17] [18] [19] . In addition, processing chemical olfactory stimuli using deep neural networks has not been extensively investigated. Nevertheless, training the existing supervised models usually requires an extensive effort by experts to label data.Transformer-based models [20, 21] are a breakthrough in machine learning, surpassing the need for extensive labeling by utilizing implicit supervision without the necessity for direct labels. These models have demonstrated impressive performance in various tasks such as image [22] , video [23] , and natural language processing [24] . More recently, they have also shown promising results in encoding chemical structures [25] .In this work, we ask the question of whether representations of odorant chemical structures extracted from transformers pre-trained on chemical structures align with human olfactory perception or, in other words, can transformers smell like humans? We employ MoLformer [25] , a state-of-the-art transformer, which is pre-trained on chemical structures and we show that representations of odorants extracted from this model:• can predict labels assigned to odorants by experts (Section 4.1);• can predict continuous perceptual ratings provided by human participants (Section 4.2);• present a high correlation index with human perceptual similarity ratings (Section 4.3);• present a high correlation index with physicochemical descriptors known to be relevant for olfactory perception (Section 4.4).Surprisingly these results hold for models that were not explicitly trained for the purpose of predicting the human olfactory experience. To the best of our knowledge, we provide the first empirical study on evaluating the alignment between odorant representations encoded by transformers and human olfactory perception. 
related work: 
 The availability of larger datasets, together with advances in predictive methods, has led to an increasing interest in the prediction of olfactory perception from molecular structures.Olfactory perception prediction. Learning predictive models of olfactory from molecular structures has been addressed mostly by the neuroscience community. Several works used standard chemoinformatic representations of molecules to model olfactory perception [17, 19, 18] . Specifically, Snitz et al. [18] proposed a computational framework and algorithm based on structural features of molecules to predict perceptual similarities between odorant pairs. This algorithm leverages feature engineering to identify the most relevant subset of features among 1433 physicochemical descriptors to predict pair-wise odorant perceptual similarities.Later, Ravia et al. [17] extended this model to also include the perceived intensity of molecular components. They employed 21 physicochemical descriptors discovered in previous works and proposed a weighting approach for multicomponent odorants (MC-odorants) based on their perceived intensity. They reported a higher correlation when employing the weighting approach compared to using the same model without it. However, the representation and generalization capabilities of these models are quite limited and unexplored.Deep neural networks for odorants. Recently, Lee et al. [16] proposed a novel representation learning model of odorants, based on a message-passing graph neural network [26] , which they refer to as Principal Odor Map (POM). To train this model, they curated and merged data from Leffingwell [27] and GoodScent [28] databases to compile a dataset of about 5000 molecules with 138 expert-labeled odor descriptors. This model outperforms the baselines in multiple odor prediction tasks and shows a relatively high alignment with human ratings in describing odorants. Nevertheless, training this model requires labeled data, relying on subjective evaluations of numerous odorants by experts. Besides being time-consuming and laborious, this process can introduce subjective biases into the model, a concern magnified by our incomplete understanding of the foundational factors of olfactory perception.Large-scale molecular models. Large-scale pre-trained models, often known as foundation models [21] , have been recently explored to perform diverse tasks by leveraging large amounts of unlabeled data. MoLFormer [25] model has been proposed in the context of chemical prediction tasks, able to extract rich representations from chemical structures. MoLFormer consists of a transformer-based architecture, with linear attention and relative positional encodings. This model is trained using a self-supervised approach, on multiple datasets (e.g., the PubChem [29] and ZINC [30] datasets) on a masked token prediction loss.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discuss the limitations of the current work in the penultimate paragraph of the discussion section.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we provide a detailed description of the datasets utilized in this study and outline the methodology for extracting both odorant (machine) and perceptual (human) representations. Additionally, we present the main model and baseline methods employed, along with the evaluation metrics used to assess their performance. Our experiments do not require significant computational resources: we mostly train linear models that do not involve GPU usage or models that can be trained on a single commercially available GPU under one hour. All computational code to reproduce our results is available at https://github.com/Farzaneh-Taleb/transformer-olfactory-alignmentWe use the publicly available version of the following datasets provided by Pyrfume repository [31] .Leffingwell-Goodscent (GS-LF) [27, 28] . We employ a curated and merged version of the Goodscents [28] and Leffingwell [27] datasets, provided by [32] , following the procedure introduced by Lee et al. [16] . This dataset contains 4983 molecules with 138 expert-labeled descriptors (e.g. creamy, grassy), where each odorant may be linked to multiple descriptors.Sagar [33] . This dataset contains the rating of 160 odorants by 3 human participants, with respect to 15+3 perceptual descriptors. In addition to 15 common descriptors among participants, there are 3 more descriptors that vary among them. We excluded these variable descriptors and focused solely on the common descriptors among the participants. The provided ratings were already normalized within the range of [-1, 1] and the mean ratings across all the subjects are computed for subsequent analysis.Keller [34] . This dataset contains ratings of 480 structurally and perceptually diverse molecules by 55 human participants, evaluating 23 descriptors. Participants were instructed to adjust a slider to rate odorants according to individual descriptors, with the slider position subsequently translated into a scale ranging from 0 to 100. Ratings were then averaged across all participants for further analysis.Ravia [17] . This dataset contains similarity ratings of 195 unique pairs of MC-odorants and monomolecules by 94 participants. The similarity values were averaged across all the participants. In this work, we disregarded the factor of odorant intensity and averaged similarity ratings based on the unique pairs of odorants.Snitz [18] . This dataset includes similarity ratings from 139 participants and 359 unique pairs of odorants. In each trial, participants were presented with two distinct odorants and asked to rate the degree of similarity in their smells. These ratings were then averaged across all participants.Odorants can be described as a single molecule or a mixture of molecules, which we denote as multicomponent odorants (MC-odorants). In this section we describe the method to extract odorant representations from the main pre-trained model (MoLFormer) and our baseline models (DAM and Open-POM).MoLFormer. We employ MoLFormer [25] to encode SMILES strings associated with a single molecule and extract a 768-dimensional vector from the last layer of the model. SMILES (simplified molecular-input line-entry system) is a string-based representation that encodes relevant chemical information such as the type of atoms, their bonds, and the substructures present in the molecule.For MC-odorants, we average the extracted representations across all available mono-molecule components within that MC-odorant. The odorant representation for each dataset is a matrix ofX i ∈ R n×768where n is the number of unique odorants.Open-POM. The principal odor map (POM) is a supervised-learning model, based on a messagepassing graph neural network [26] , which is trained on the GS-LF datasets to predict odorant perceptual labels. We employ a publicly available version of this model, which we denote by Open-POM [32] . We train Open-POM for 150 epochs, using 30 different train-test splits, and we extract representations from the penultimate layer of this model. The odorant representation for each dataset is a matrix of X i ∈ R n×256 where n is the number of unique odorants. For MC-odorants, we average the representations extracted for each individual molecule within the mixture.Distance Angle Model (DAM). Snitz et al. [18] proposed a distance angle model (DAM) that uses 21 physicochemical descriptors to predict the similarities between pairs of odorants. We extract these 21 descriptors for each odorant using AlvaDesc [35] and discard 6 of them due to NaN values produced by this software. We use the remaining subset of 15 physicochemical descriptors out of 21 to measure similarity between odorants or train a linear mapping from them to the perceptual representation space. The odorant representation for each dataset is a matrix of X i ∈ R n×15 where n is the number of unique odorants. As suggested by Ravia et al. [17] , we average the representations extracted for each individual molecule within the mixture to compute representations for MC-odorants.Perceptual representations of odorants are provided by human participants when exposed to odorant stimuli. Perceptual olfactory data were collected in one of the following ways:1. Experts label the odorants, where each odorant may be linked to multiple labels (e.g., [27, 28] ). The perceptual representation is a matrix of y i ∈ {0, 1} n×d where n is the number of unique odorants and d is the number of classes.2. Non-expert participants provided ratings with regards to a set of predefined descriptors (e.g., [33, 34] .) In this case, the averaged perceptual representations over participants and replicas form a matrix of y i ∈ [a, b] n×d , where n is the number of unique odorants, d is the number of descriptors, and a and b denote the minimum and maximum values participants can use to describe the odorants with respect to these descriptors.3. Participants evaluated the perceived similarity between pairs of odorants (e.g., [18, 17] ). In this case, the averaged perceptual representations over participants and replicas are a vector ofy i ∈ [a, b] n×1where n is the number of unique "pair of odorants" and a and b indicate the range of values participants can use to rate the odorants' similarity with respect to the descriptors.We measure the similarity between two representation spaces directly when it is possible (Section 4.3), otherwise we train a linear model to predict perceptual representations for each odorant (Section 4.1, 4.2). We use nested 5-fold cross-validation to tune the hyper-parameters of the linear models 2 : ROC curve for linear classifiers trained on GS-LF representations extracted from three different models. Each curve corresponds to a separate test split, with the thicker curve representing the average performance across all splits. We highlight that MoLFormer outperforms DAM, despite not being trained to predict perceptual labels but does not achieve the performance level of Open-POM, which demonstrates the highest performance. The chance level is shown with red dashed line. and assess evaluation metrics on the test set that was held out during the training phase using an 80%-20% train-test split. This process is repeated 30 times using 30 different train-test splits.In this section, we introduce the main evaluation metrics to measure the alignment in this paper.Micro-averaged ROC-AUC score. The micro-averaged ROC-AUC score was computed to assess the performance of each model for the multi-label classification task. The micro-averaged ROC-AUC score is computed by aggregating true positive, false positive, true negative, and false negative values across all classes.Normalized Root Mean Squared Error (NRMSE). The root mean squared error (RMSE) is the difference between the observed values and predicted ones for the regression task. Here, we normalize it by the range of true observations -i.e., NRMSE = RMSE/(max(y)min(y)).Pearson Correlation Coefficient (CC). We report the Pearson correlation coefficient between predicted results and real values. It measures the linear correlation between two sets of data and is the ratio between the covariance of two variables and the product of their standard deviations.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The parts of main experiment is uploaded and the whole code is available through the anonymized link provided in the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
results: 
 In this section, we evaluate whether the representations encoded by pre-trained models of chemical data can predict the human olfactory experience despite not being explicitly trained for this purpose. First, we focus on a subset of experiments aimed at predicting expert-assigned labels from odorants through linear mapping from representations to perceptual descriptors (Section 4.1). Subsequently, we aim to predict continuous scores provided by human participants (Section 4.2). Finally, we seek to predict the direct similarity scores from the representations extracted from odorants (Section 4.3). Additionally, we provide insights into the potential reasons underlying the observed alignments (Section 4.4).To assess the performance of MoLFormer in predicting expert-assigned labels for odorants, we implemented a linear mapping from the representations extracted by MoLFormer to the odorant representations extracted from GS-LF dataset. First, the dimensionality of the extracted representations is reduced to 20 using PCA, followed by z-scoring of each feature. Then, we train individual logistic regression models for each descriptor. This process was repeated 30 times, each with a different train-test split, to quantify the uncertainty of the results. [16] . We plot the first and second principal components (PCs) of the representation spaces. Areas dense with molecules that have broad category labels (floral, meaty, or ethereal) are shaded, while areas dense with narrow category labels are outlined. MoLFormer captures the perceptual relationship between different odorants in its representation space, despite not being explicitly trained for this purpose.We apply the same procedure without dimensionality reduction to DAM model representations. For the Open-POM model, which is already trained end-to-end and supervised on the same dataset, we directly extracted the predictions for the test set without retraining the model. As shown in Figure 2 the MoLFormer model achieves high ROC-AUC scores in odorant classification, outperforming the DAM model, which is trained using 15 physicochemical descriptors. However, the performance of MoLFormer is lower than that of Open-POM, which is trained end-to-end with supervision on the same dataset.An additional experiment is conducted to understand the degree of perceptual details captured in the odorant representation space of MoLFormer by comparing odorant representations encoded by this model with the representations encoded by Open-POM. In Figure 3 we depict the first two principal components of the representations. We highlight the similarity between the representations encoded by both Open-POM and MoLFormer and observe that the latter is able to capture the perceptual relationship between different odorants despite not using any perceptual labels during training (unlike the supervised Open-POM Model).To evaluate the capabilities of the MoLFormer model to predict continuous rating scores with respect to pre-defined descriptors, provided by human participants, we train separate linear regression models with regularization applied using the Lasso penalty for each descriptor. Once again, the dimensionality of the extracted representations is reduced to 20 using PCA (for MoLFormer and Open-POM), followed by z-scoring of each feature. This procedure is repeated using 30 different train-test splits.The results of these experiments are shown in Table 1 and Figure 4 . Table 1 shows the average Pearson correlation coefficient and NRMSE across all descriptors, while Figure 4 presents the results for each individual descriptor. As shown in Table 1 , overall, none of the models exhibit a high correlation. Nevertheless, MoLFormer slightly underperforms compared to Open-POM in both datasets. However, it performs better than DAM for the Keller dataset but worse than DAM for the Sagar dataset, where DAM even outperforms Open-POM.Table 1 : Performance of the models to predict continuous ratings averaged across all perceptual descriptors. We compute the average Pearson correlation coefficient (CC) and normalized root mean squared error (NRMSE) across all descriptors. MoLFormer shows slightly worse performance than Open-POM but better than DAM for the Keller dataset and worse than DAM for the Sagar dataset, where DAM outperforms Open-POM.MoLFormer Open-POM DAM MoLFormer Open-POM DAM CC (↑) 0.20 ± 0.00 0.22 ± 0.01 0.17 ± 0.00 0.25 ± 0.01 0.29 ± 0.01 0.35 ± 0.01 NRMSE (↓) 0.15 ± 0.00 0.15 ± 0.00 0.15 ± 0.00 0.19 ± 0.00 0.18 ± 0.00 0.17 ± 0.00 Figure 4 : Performance of the models to predict continuous ratings per descriptor. We computed Correlation and NRMSE between predicted and actual ratings per perceptual descriptor. Despite not being trained to predict human olfactory labels, the MoLFormer model performs on par with the Open-POM and DAM models.According to Figure 4 MoLFormer model performs on par with the Open-POM and DAM models, which are trained with supervision in predicting the rating for each descriptor. In summary, although, on average, MoLFormer performs slightly worse than Open-POM, it still demonstrates a similar degree of alignment, especially despite the absence of supervision in its training process.In order to evaluate the direct alignment between the odorant similarities encoded by MoLFormer and those obtained from human participants, we separately encode each odorant by MoLFormer (and the baseline models) and compute the cosine similarity between the extracted representations. Subsequently, we compute the Pearson correlation between the similarity scores computed by the models and those provided by human participants in the Ravia and Snitz datasets. The results are presented in Figure 5a .These results show that the MoLFormer is able to extract representations that encode information related to the human olfactory perception, despite not having access to that information during model training. We highlight a significant high correlation between perceptual and odorant representation for the Snitz (r = 0.64, p < 0.0001) and Ravia datasets (r = 0.66, p < 0.0001). The comparison with the baseline models indicates that it performs on par with the Open-POM model and significantly outperforms the DAM model. These results suggest that, despite being trained with some form of supervision, these models may struggle to effectively extract similarities between odorants. Additionally, the findings demonstrate that MoLFormer is more proficient at identifying similarities between pairs of odorants than mapping them to a set of predefined descriptors. This superior performance may be due to the model's ability to capture a measure of similarity, as perceived by humans, rather than introducing subjective language bias associated with pre-defined descriptors.Finally, we aim to evaluate whether the depth of the layer in the MoLFormer model, from which we extract the odorant representations, affects the representational alignment. To assess this, we repeat the described procedure in this section for each layer separately. As shown in Figure 5b , representational alignment improves with increasing layer depth, indicating that deeper layers of the transformer are more aligned with high-level perceptual representations.To evaluate whether MoLformer effectively extracts features from chemical structures relevant to olfactory perception, we evaluate the alignment of MoLFormer with physicochemical descriptors that are used in the DAM model. To do so, we train 15 linear regression models, each one to predict a single physicochemical descriptor from the extracted representations of the MoLFormer. We subsequently evaluate the correlation between the predicted and true values. As shown in Figure 6 , MoLformer demonstrates a high degree of alignment in predicting these values. Out of the 15 physiochemical descriptors, MoLformer successfully predicts the values for 13 descriptors as well as or better than the Open-POM model.Next, we evaluate whether this alignment changes across the layers of MoLformer. Therefore, we repeat the same procedure for each layer separately. As illustrated in Figure 7 , the alignment with the identified chemical features decreases with increasing layer depth. However, as demonstrated in Figure 5b , the alignment with perception improves. These results collectively are consistent with well-known principles in vision models, where the lower layers typically capture low-level, localized features like edges and textures, while deeper layers gradually shift toward higher-level, abstract representations, such as shapes and objects [36] . Nonetheless, additional investigation is required to fully reveal and comprehend this potential hierarchical structure.  
discussion: 
 In this study, we investigated the alignment between odorant representations encoded by the MoL-Former, a self-supervised transformer model pre-trained on chemical structures, and human olfactory perception. We evaluated the alignment between these representations by analyzing the similarity between them or finding a linear mapping between the representations. Additionally, we offered insights into the potential reasons behind the observed alignments by exploring relevant chemical features extracted by the model.Perceptual prediction from pretrained models. We demonstrate for the first time that representations extracted from pre-trained large models, solely trained on chemical structures, align closely with the perceptual representations of odorants. This finding suggests that odorant perception can be accurately predicted from chemical structures. Furthermore, we show that this model can predict a subset of physicochemical descriptors known to be relevant to olfactory perception. Together, these results offer valuable predictions for chemists and neuroscientists to explore in future research.Evaluating alignment across multiple datasets. To evaluate alignment from various perspectives, we designed three different experiments. First, we leveraged a dataset with expert-provided labels for odorants, assessing the model's ability to independently predict multi-target binary labels for each odorant. This task did not involve variability from human participants or continuous odorant ratings. MoLFormer exhibited relatively high performance in predicting these binarized labels. Second, we used datasets containing average continuous ratings from human participants, which inherently present more challenges due to variability among non-expert participants' ratings. Our evaluation revealed that while all models performed poorly on this task, MoLFormer performed comparably to supervised models. Lastly, we evaluated direct similarity scores between odorants from two datasets, examining the alignment between human-provided similarity scores and those computed from the representations encoded by models. MoLFormer showed a high alignment, highlighting its ability to predict similarity between odorants rather than relying on human-made descriptors. This suggests that pre-defined descriptors for describing odorants may need to be more carefully chosen, and models trained with these descriptors might not accurately reflect the true similarity between odorants.Reduction in alignment with physicochemical descriptors across layers of the models. We conducted a complementary analysis to identify potential reasons underlying the observed perceptual alignment. Our focus was on the subset of features previously identified as significant for decoding olfactory perception from chemical structures. Our findings indicate that MoLFormer representations exhibit a high degree of alignment with these features. While most features show strong alignment, a few demonstrate less alignment (such as nRCOSR). These results collectively suggest that while these features are important, their significance varies. Additionally, our analysis of the predictability of these features across the different layers of the model shows that as we go through the layers, we observe a decrease in alignment with physicochemical descriptors despite an increase in alignment with perception. This observation aligns well with established principles in vision models, where lower layers have been shown to capture low-level, local features such as edges and textures, while progressively transitioning to align with higher-level, abstract representations, such as shapes and objects, in deeper layers [36] . However, further exploration is needed to fully uncover and understand this potential hierarchy.Limitations. Our work is perhaps best understood in the context of its limitations. We do not directly take into consideration the intensity or concentration of each individual molecule within a mixture during the encoding of odorants. Incorporating these intensity factors in future work could potentially improve the alignment. Additionally, our research was constrained by the available datasets, which typically lack sufficient variations in different odorants, particularly for continuous rating regression tasks. Furthermore, we only considered the average rating scores and did not evaluate the alignment on a per-subject basis.Future Work. We aim to leverage these findings to develop improved models of olfactory perception. Specifically, we plan to utilize unsupervised models trained exclusively on chemical structures to identify which chemical features are crucial for predicting perception, thereby avoiding the introduction of biases from human subjective perception. Additionally, we intend to investigate the mechanisms underlying olfactory perceptions decoded from chemical features. The observed alignment trends across different layers of the model may provide key insights into this process. Finally, evaluating representational alignment between the extracted representations from transformers trained on chemical structures and fMRI data from the brain can provide deeper insights into the underlying mechanisms of olfactory perception.In order to evaluate the quality of data and the upper limit of the models, we computed the noise ceiling (Equateion 1, 2) for the Sagar and Keller datasets as these are the only ones that have multiple evaluators for each odorant, and those are publicly available. The results show that we have the average noise ceilings of 0.28 ± 0.1 for the Keller dataset and 0.7 ± 0.05 for the Sagar dataset (Table S .1, S.2). The results show that the data of the Sagar dataset is less noisy, and there is still room for the models to increase the alignment. However, the Keller dataset alignment results are relatively close to the noise ceiling value. We also reduce representations of GS-LF dataset extracted from MoLFormer and Open-POM datasets using t-SNE (Figure S.2).We fine-tuned MoLFormer using GS-LF dataset which is a large and inclusive dataset of odorants. Then we extracted representations for all the datasets and tasks. [16] . We reduced dimensionality using t-SNE. Areas dense with molecules that have broad category labels (floral, meaty, or ethereal) are shaded, while areas dense with narrow category labels are outlined. MoLFormer captures the perceptual relationship between different odorants in its representation space, despite not being explicitly trained for this purpose.In this section, we present the results for predicting physicochemical descriptors from odorants for each dataset separately (Figure S.6-S.10). We observe that MoLFormer can better predict physicochemical descriptors in most cases.We also show how the alignment between physicochemical descriptors changes across layers of the model for each dataset separately. As it is shown, alignments decrease across layers. This might be because the first layers extract more low-level information while deeper layers, extract more high-level features. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report error bars suitably and correctly defined.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The original owners of assets (e.g., code, data, models), used in the paper, are properly credited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: the paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing. Although we have used datasets that involve human subjects, we did not collect those data and used the datasets that are already approved and their details are publicly available in the original work.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: Although we have used datasets that involve human subjects, we did not collect those data and used the datasets that are already approved and publicly available.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix:  


Decision: The work described in this paper demonstrates that a transformer model that was pre-trained on general chemical structures (self-supervised) forms representations that align well with human olfactory perception.

Olfaction is still understudy (compared to e.g., vision or audition) and our understanding of the neural computational processes and
representations underlying human olfactiory perception is still very limited. Thus this paper makes a welcomed contribution. The reviewers
liked the model approach, the careful comparison with mutliple datasets of human psychophysical assessments of odors, and the results. The work suggests that the domain-general learning mechanisms of transformers may explain sensory processing in the brain more broadly, which is an interesting scientific hypothesis with broad implications. Thus, the work is of substantial interest to the NeurIPS community.

Reviews: review: paper_summary: paper_summary: This paper investigates the ability of MOLformer, a model trained on SMILES strings in a BERT-like fashion, to predict the human assessment of odors. The assessments were based on natural language descriptors of odors by human experts, ratings on different NL descriptors by naive subjects, and finally, on similarity ratings between odors.  It compares MOLformer, which was trained in an unsupervised manner, to an engineered model (DAM) and a supervised model Open-POM. Despite being trained in an unsupervised manner to represent molecules, MOLformer holds its own against the supervised models. In addition, MOLFormer shines when evaluated using an RSA technique, suggesting that MOLformer captures the relationships between molecular odors better than the other models.
strength_weakness: strength_weakness: + Despite a rather systematic reporting of models and datasets, some things were left out of the descriptions. For example:
	- Apparently the GS-LF datasets features are binary. I assume this means that that particular descriptor was used by at least 1 judge. So is it the case that if multiple judges used the same descriptor, the results were still recorded as binary?
	- how dimensionality was reduced. I assume PCA. How much variance was captured in 20 PCs?
	- There were apparently descriptors in the similarity dataset as well (line 123). These weren’t described.

+ the references in the introduction were weird. See below.

Typos, comments:

The references you use in the introduction seem rather bizarre. Two Damasio references for encoding sensory input into a high-dimensional space?
Reference #4 seems out of place in this context.
References 8-10 - you have no primary references here; for example, most would reference DiCarlo’s lab here.
Reference #12 is weird. Why not reference the original transformer model? And while it’s true that for language, they are self-supervised, the original use of them for vision was supervised. It is only later that self-supervised vision models (e.g., the Masked Autoencoder) came about. 

line 53: Olfactory is mis-spelled! 

Figure 2: You might try using UMAP or t-SNE here instead of (or in addition to) PCA.

In your results section, you never say what the train-test split was (i.e., 80/20?)

I was thinking - they should use RSA - and then you did! This is a very interesting result because, rather than just being able to predict human results, the model is better at both of the others in capturing the *relationships* between the odors, just from their chemical descriptions.
	
- Figure 6: Explain why didn’t you evaluate DAM here.
- You didn’t describe how the physicochemical descriptors were obtained. 

Line 284: In section 4.4, it wasn’t clear that for this analysis, you were trying to discover the reasons for the alignment. You should say something about that when introducing that analysis.

lines 289-292: It makes sense to me that the low-level features of the molecules would be best encoded in earlier layers and not in later layers, where more abstract features would be encoded. This seems to fit well with the way vision models work.
questions: questions: See my questions above:
1. variance captured by PCA
2. How are the physicochemical descriptors derived?
3. Try UMAP or t-SNE
limitations: limitations: These are discussed.
paper_summary: paper_summary: This paper took a transformer model that was pre-trained on general chemical structures and tested whether the resulting model representations aligned with human olfactory perception. Specifically, the authors used a transformer for chemical structures called MoL-Former. MoL-Former was trained via masked token prediction loss. The representations from MoL-Former can predict odor labels from experts and odor ratings from non-experts (although the correlations between the model and rating scores is quite low). The representations are also highly correlated with physiochemical descriptors that are related to human olfactory perception.

The authors also compare the MoL-Former self-supervised transformer model to other baseline models (Open-POM, which is supervised with odor labels; and DAM, which predicts similarities between pairs of odors). MoL-Former generally performs comparably to these models, even though the baseline models are supervised and MoL-Former is self-supervised.
strength_weakness: strength_weakness: Major
- Section 4.1 Expert labels classification was difficult to follow. How was dimensionality reduced to 20, through PCA perhaps? How was the same procedure applied to DAM, given that DAM already has less than 20 features? Why wasn't the visualization in Figure 3 also performed on DAM?
- Sections 4.2 & 4.3 need noise ceilings in order to be interpretable. The authors analyzed each model's predicted average rating with the actual average rating from participants. However, it is unreasonable to expect a model to be more correlated with average ratings than a typical individual human participant. The noise ceiling computes the average correlation between the human participants and mean performance to estimate a reasonable "ceiling" performance that could be expected from models. When the data are very noisy, the noise ceiling will be low (and the human benchmark will generally be less useful because of its low precision), and when the data are not noisy, the noise ceiling will approach 1.

Minor
- The first sentence of the paper references Damasio, 1989 and Meyer & Damasio, 2009. These are both fantastic papers, but they don't really fit the sentence where they are referenced. References like DiCarlo & Cox, 2007 and Olshausen & Fields, 2004 strike me as more relevant.
- A representational similarity matrix (RSM) would be a great visualization for Section 4.3. It would be really interesting to visualize how the patterns of similarity between different odorants compared across models and humans. You don't need to add it, but it would be helpful for readers to visualize how the models and humans are representing the different odors.
- The description of SMILES is a bit hard to follow. Could you provide an example and explain how (or if) the inputs to Open-POM, DAM, and MoL-Former differ?
questions: questions: Does the input to MoL-Former (and the baseline comparison models) correspond to olfactory receptors? If the input data from olfactory receptors does not match the input data to MoL-Former, then it's not actually completing the same task, and it becomes much harder to argue that the model can "smell like humans."
limitations: limitations: See weaknesses above.
paper_summary: paper_summary: This paper mainly focus on the question "Can transformers smell like humans". By using a chemical structure transformer called MoLFormer to encode the odorant molecules, this paper proposes that the MoLFormer pre-trained representation can classify the odors of variety of molecules without careful fine-tuning. The method builds a relationship between odorant chemical strutures and their odors, and the odors are labeled by human-beings.
strength_weakness: strength_weakness: 1. The most important part of the odors classification system is the network, unfortunately, the paper only directly apply a well-designed transformer (MoLFormer) without any modification. 
2. It looks like this paper dose not finetune the MoLFormer using the applied databases. The paper argues that the model can achieve good performance without fine-tuning, however, it is better to include experiments to support this argument.
questions: questions: 1. How many models (beside MoLFormer) does the research group consider to generate odorant representations? 
2. Do the authors consider to fine-tune the MoLFormer for better performance?
3. Besides transformer, do the authers consider more kinds of models such CNN and graph network?
limitations: limitations: 1. It is better to do more analyses about the fine-tuning of the network.
2. It is better to consider to include some innovation in the existing models.

Comments: title: thanks for the response
comment: Thank you for the discussion and for recognizing our efforts with the score raise.
Thank you for your comments and feedback
We will add the definition of noise ceilings in the Appendix. Nice suggestion!

Interesting question about the RSM visualizations. Your hypothesis is an interest one. Indeed, we believe that the main directions used in PCA and the different ways the features are combined in t-SNE and UMAP are quite important in making these groups clearly distinct.

Finally, we thank the reviewer for the discussion and feedback. Also, thank you for recognizing our efforts with the score raise.
Thank you for providing these helpful explanations and revisions!

Any idea why there isn't much visible grouping by scent category (e.g., meaty, floral, ethereal) for the RSM visualization? I suppose the grouping from PCA, t-SNE, and UMAP must rely on how different features are weighted and combined.

Regardless of that question, I am very enthusiastic that the Sagar noise ceiling is reasonably high. You mention that you will "add the results divided by noise ceiling value in the appendix of the paper," and I ask that you please also include what the noise ceilings are for reference. This gives readers important information about the quality of each benchmark dataset.

Based on the authors' rebuttal PDF and responses to my review, I have decided to raise my score. I will modify my review to raise my rating from a 6 to a 7.
Thanks for your response to my criticisms. Since I already gave the paper what I consider a relatively high score, I will leave it at that. I think it's a very interesting paper, showing that a self-supervised transformer trained on molecular structures (without anything specific to odors), can reasonably well predict odor ratings by humans. Good job, and I hope the paper is accepted.
I thank the authors for answering my comments.

I believe that the authors' responses can somehow answer my questions, even though I still cannot be convinced completely. I would like to re-rate the paper to Borderline accept.