Multi-scale Sinusoidal Embeddings Enable Learning on High Resolution Mass Spectrometry Data
Authors: Gennady Voronov
Venue: Submitted to ICLR 2023 (2023)
Keywords: Cheminformatics for Life Sciences
Tandem Mass Spectrometry
Transformers

Abstract: Small molecules in biological samples are studied to provide information about disease states, environmental toxins, natural product drug discovery, and many other applications. The primary window into the composition of small molecule mixtures is tandem mass spectrometry (MS2), which produces data that are of high sensitivity and part per million resolution. We adopt multi-scale sinusoidal embeddings of the mass data in MS2 designed to meet the challenge of learning from the full resolution of MS2 data. Using these embeddings, we provide a new state of the art model for spectral library search, the  standard task for initial evaluation of MS2 data. We also investigate the task of chemical property prediction from MS2 data, that has natural applications in  high-throughput MS2 experiments and show that an average $R^2$ of 80\% for novel compounds can be achieved across 10 chemical properties prioritized by medicinal chemists. We use dimensionality reduction techniques and experiments with different floating point resolutions to show the essential role multi-scale sinusoidal embeddings play in learning from MS2 data.

Under review as a conference paper at ICLR 2023
MULTI-SCALE SINUSOIDAL EMBEDDINGS ENABLE LEARNING ON HIGH RESOLUTION MASS SPECTROME- TRY DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Small molecules in biological samples are studied to provide information about disease states, environmental toxins, natural product drug discovery, and many other applications. The primary window into the composition of small molecule mixtures is tandem mass spectrometry (MS2), which produces high sensitivity and part per million resolution data. We adopt multi-scale sinusoidal embeddings the mass data in MS2 designed to meet the challenge of learning from the full resolution of MS2 data. Using these embeddings, we provide a new state of the art model for spectral library search, the standard task for initial evaluation of MS2 data. We also investigate the task of chemical property prediction from MS2 data, that has natural applications in high-throughput MS2 experiments and show that an average !t2 of 80% for novel compounds can be achieved across 10 chemical properties prioritized by medicinal chemists. We vary the resolution of the input spectra directly by using different floating point representations of the MS2 data, and show that the resulting sinusoidal embeddings are able learn from high resolution portion of the input MS2 data. We apply dimensionality reduction to the embeddings that result from different resolution input masses to show the essential role multi-scale sinusoidal embeddings play in learning from MS2 data.
INTRODUCTION
Metabolomies is the study of the small molecule ($ 1,000 Daltons) contents of complex biological samples. Tandem Mass Spectrometry (MS/MS), in conjunction with chromatography, is one of the most commonly used tools in metabolomics. Tandem Mass Spectrometry works by measuring with very high resolution the masses of molecules and their constituent fragments. While MS/MS techniques are highly sensitive and precise, inferring the identity of the molecules and their properties from the resulting mass spectra is commonly regarded as of metabolomics' primary bottlenecks (Dnnn et al.] (013). Improved tools for these tasks will impact applications across many areas of science including disease diagnostics, characterization of disease pathways, development of new agrochemicals, improved forensics analysis, and the discovery of new drugs (Zhang et al.) 2020).
Profiling unknown molecules with mass spectrometry consists of several steps. First, molecules of interest are ionized and separated by their mass to charge ratio (m/z), resulting in the MS1 spectrum. Then, individual "precursor" ions fragmented, and the m/z of the fragments are recorded in the same manner. The resulting spectrum contains the m/ and intensities (together, the "peaks") of all resulting fragments, and called the MS2 spectrum. See Glish & Vachet (2003).
In recent years, several machine learning methods have been developed to identify the structures and properties of small molecules from their mass spectra. These approaches (Huber et al., 2021a b Kutuzova et al. 2021 Litsa et al.] 2021; [Shrivastava et al.) 2021] van Der Hooft et al. 2016 historically discretize m/z (via tokenization or binning). However, the m/zz values obtained in modern mass spectrometry experiments are collected with parts per million levels of resolution. There are three critical reasons to expect that modeling m. values as numeric quantities, in contrast to discretized values, is the appropriate technique. First, we know that relevant chemical information is present at the millidalton level (Jones et al.) Pourshahan Limbach] 2008) and discretization schemes typically strip this information away. Additionally, mass differences between peaks represent

---

Under review as a conference paper at ICLR 2023
fragmentation pattems, and are therefore relevant understanding the molecule. Learning utilize this information from tokens is exceedingly difficult. Finally, discretization is susceptible to edge effects, where slight mass difference can map to different bins if the masses close to the edge of a bin. In light of these considerations, we model m/z as a numerical value using sinusoidal embeddings, which we hypothesize will enable us capture information across many orders of magnitude.
In this work, we apply numerical representation of m/z that uses sinusoidal embeddings across multiple scales to retain the information content of MS2 data across its entire mass resolution. To demonstrate the ability of these embeddings enable learning at the high resolution of MS2 experimental data, we apply them to a search task and 10 regression tasks.
Our first test task spectral library search (Stein & Scott 1994). In spectral library search, spectra from unknown compounds are compared to spectra in a database to find matches using similarity function over pairs of spectra. This task the primary method used in standard metabolomics analyses, but is challenging because spectra for compound vary widely with experimental conditions. We find that a similarity function based on sinusoidal embeddings achieves state of the art both for finding the exactly correct compound, and also for finding close structural analogs, which is useful for compounds not contained in spectral databases.
We further investigate predicting chemical properties relevant to drug discovery from MS2 data and apply the same modeling approach to this task. We achieve 80% average R²² for out of sample molecules across 10 properties, which is high enough enable first-pass filtering and selection of candidate drug molecules in high-throughput experiments based solely on spectral data.
In each task, using sinusoidal embeddings results in new state of the art. The confirmation of across-task improvement provides evidence that the embeddings are a general improvement, rather than task specific. To determine whether these results due to learning from the high resolution portion of the data, we experiment with inputting MS2 data cast to half precision floating point numbers, and show that the performance noticeably degrades relative to double precision. Finally, we visualize embeddings generated with varied floating point precision MS2 inputs using UMAP (Melnne al. 2018) projections and show that non-trivial high dimensional structure only emerges with sufficiently high precision input. Taken together, these results are the first clear evidence that sinusoidal embeddings enable effective learning from high mass resolution MS2 data across tasks in metabolomics.
1.1
RELATED WORK
Modeling numerical data in terms of sinusoidal functions has a long history in many scientific fields. In machine learning, sinusoidal embeddings are most commonly used to encode the discrete positions of natural language token inputs in transformer models, which are otherwise pesition-agnostio (Vaswani et al. 2017). Other work has used sinusoidal embeddings for multi-dimensional positional encoding in image recognition (Li et al. 2021a). In mass spectrometry, sinusoidal embeddings have been used in proteomics for inferring protein sequences from mass values, either with (Qiao et al.) 2019) or without (Yilmaz al. (022) an initial mass binning step. but without exploring their role in model performance or comparing to alternative approaches. These techniques have never been applied in the domain of metabolomics. Metabolomics differs from proteomics in that the molecules of of interest are 2 - 3 orders of magnitude lower mass and are graph structured rather than sequential as in proteins. Consequently, the tasks and challenges of modeling of small molecules are sharply divergent from those in proteomics.
For modeling m/z values in the field of metabolomics, previous machine learning models have relied primarily on discretization of the continuous mass inputs. This is usually accomplished by binning m/s values into fixed length vectors with peak intensity as the value for each element. Various authors have used binned representations of spectra for spectral library search (Huber et al.] (0211)) unsupervised topic modeling (van Der Hooft et al., 2016). and molecule identification (Kutuzova et al. 2021 Litsa al. 2021). Alternatively, masses have been tokenized via rounding for tasks such as unsupervised spectral simllaity (Huber et al. 2021a and molecule prediction from synthetic data (Shrivastava et al. 2021
2

---

Under review as a conference paper at ICLR 2023
Other approaches rely on tokenization of m/2 values by assigning molecular formula each peak, and taking the formula as a token. Even when this method is able to uniquely identify molecular formulas for every fragment ion, it still faces problem endogenous to tokenization: it's very difficult to reason about the relationships between discrete tokens except by pattern recognition over a very large number of training examples. [Böcker & Rasche (2008) addressed this difficulty by modeling the fragmentation process with a tree structure, and Dührkop et al. (2015) used these fragmentation inputs molecular fingerprint prediction model that is used assign molecules spectra from molecular structure databases. The tree- tree-based approaches are prohibitively computationally expensive for large volumes of data Cao et al. (2021).
MODEL
2.
BASE MODEL ARCHITECTURE
An MS2 spectrum,
S (m/ precursor (m/z, 2,, fragment, 1.... (
(m/z, Iragmeet
is composed of precursor m/2 and a set of N fragment peaks at various m/z's and intensities (T). Various other data are typically collected, including precursor abundance, charge, collision energy, etc, but these are not used in this work. Transformer encoders (Vaswani et al. (017) without the positional encoding are explicitly fully-symmetric functions and hence are ideally suited to model set of fragmentation peaks. We therefore take our base model, SpectrumEncoder (S). to be transformer encoder, whose inputs are a set of (m/2,1) I) pairs that includes the precursor along with all of the fragment peaks. We normalize the intensities an intensity of 2 to the precursor. Finally, we take as output the embedding vector from the final transformer layer corresponding to the precursor input. See Figure 2 Note, the use of transformers study synthetic MS2 spectra has been explored in Shrivastava et al. (2021).
maximum of 1 for fragments and assign
This flexible approach enables experiment with and compare various representations of MS2 data. We pursue two approaches: tokenization of m/z and modeling m/z n as numerical values via sinusoidal embeddings. Before describing these, we need the following definition for simple two layer feed forward MLP,
FF (x) W2ReLu (W1x by) + by-
Note that each occurrence of FF we employ below is separate instance with distinct weights.
2.1.1 TOKENIZED m/z PEAK EMBEDDING
Our first approach to discretize m/2 by rounding to 0.1 precision and treating these objects as tokens (as an NLP context). These tokens are embedded in dense vector space in Huber et al.) (2021a); Mikolov et al. (2013). via an embedding function TE, We then construct the token peak embedding.
PEtoken (m/2, I) FF (TE (m/2) I), where || denotes concatenation. A diagram of P PE is shown Figure
(3)
1.2 SINGSOIDAL m/= PEAK EMBEDDING
We now consider numerical peak embedding that utilizes the full m/2 precision. We employ sinusoidal embedding as follows:
25/(d-2) 2)
Amax
SE (m/z, 2i:
sin
2T
Auin
m/z
(4)
Amin
21/(d-2)
Amax
SE (m/2,222 1;d) = cos
2T
Amin
m/z
(5)
Aunin
The frequencies are chosen so that the wavelengths are log-spaced from Amin
10 2.5 Daltons
to Amax 3.3.3 Daltons, corresponding the mass scales wish to resolve. This embedding
3

---

Under review as a conference paper at ICLR 2023
nVss embbekling
stoldy me?2)
toenbee arbedding
Pask
costay
:
(a) Architecture of PEtoken.
(b) Architecture of PEein-
Figure Architectures used to embed MS2 peaks (composed m/2 and an intensity I). Both
approaches produce an embedding of m/2 z of dimension d. The m/2 embedding in Figure la (left) is produced by a learned embedding layer. The m/z embedding Figure Ib (right) is produced by sinusoidal embedding followed by two layer MIP. In both cases, an intensity value is appended to make a vector of dimension + This is then fed into simple forward network to produce peak embedding of dimension d. We use blue highlight the network components involved in producing an my embedding and purple to highlight the full peak embedding after intensity is incorporated.
is inspired by Li al. (2021a); Vaswani et al. (2017). Similarly to Equation 3 construct the sinusoidal peak embedding.
PEgi (m/2, = FF (FF (SE(m/2)) I)
(6)
We also experiment with casting the m/2 inputs as half, single, and double precision floating point values. Equations 5 are computed at the precision specified by m/2 and the results are then cast to
4
the precision of the model weights used for training. A diagram of PEsin is shown in Figure lb
2.1.3 TRANSFORMER
The set of fragment peaks and the precursor are passed through either of the two peak embeddings specified above. The result is a sequence of embedding vectors which is then passed through transformer encoder:
SpectrumEncoder (8)
PE (m/z, precursor ....,PE (m/2, fregmentx) )
TransformerEncodder
Here PE stands in for either PFtsken PEsin- Transformer has embedding dimension = 512 and six layers, each with 32 attention heads and an inner hidden dimension of d. Note that all hidden and embedding layers in our peak embeddings above have dimension d as well. As mass spectra have no intrinsic ordering, we opt to not include a positional encoding.
In order get single embedding vector as output, the final transformer layer query only attends to the first embedding, corresponding to the position of the precursor m/z. /. A diagram of our full model architecture is shown Figure
2.2 MODEL TRAINING
We train the base models discussed here via two independent tasks aimed at applications within metabolomics and medicinal chemistry. These tasks are described in detail below. We build these models using PyTorch (Falcon 2019 Paszke et al. 2019) and train them with the Adam (Kingma & Ba 2014) optimization algorithm with parameters 3 0.9, 3 0.999, and learning rate a= 5.0 x 10 We also use a weight decay (Krogh & Hertz 1991) parameter of 0.1, a gradient clipping parameter of 0.5, and dropout (Srivastava et al. 2014) parameter of 0.1. All models were trained using between 25 and 50 epochs. Finally, all model weights were trained using half precision floating point values, regardless of the precision of the m/z inputs.
4

---

Under review as a conference paper at ICLR 2023
Nigm
unbeedrgg
unteddrrg
Transforeer
FEER
Transformer
Query
Query
iravidiyy
Vilu
Vilao
Figure 2: Full model architecture that highlights how we featurize and embed an MS2 spectrum in a dense vector space.
2.2.1 SPECTRAL SIMILARITY WITH SIAMESE NETWORKS
A common workflow for the analysis of mass spectrometry data is to compute some notion of spectral similarity for pairs of spectra that is intended to correlate with the molecular similarity of the underlying compounds (Yilmaz et al. 2017). This can then be used for tasks such as spectral library search (Stein & Scott, 1994) and molecular networking (Quinn al. [017] Watrous al.) 2012). Historically, most spectral similarity metrics used in the field have been heuristic-based (Li et al.) 2021b Yilmaz et al. 2017). More recently, spectral similarity methods that make use of techniques from deep learning have been developed (Huber et al. 2021ap b). The latter of these, MS2Deepscore, is a Siamese network that trains a simple MLP (by binning m 2) to predict a measure of molecular similarity from pairs of mass spectra. As is common practice in this field, we similarity given by a Tanimoto similarity computed from RDKit Landrum et al. (2013) topological fingerprints of the corresponding molecular structures (Bajusz al. (015). The loss function the mean square error (MSE) between the ground truth molecular similarity and the cosine similarity evaluated on the dense embedding vectors generated by the base model.
molecular
Molecular similarity scores range from 0 to 1. The scores of randomly sampled pairs from our labeled dataset are not uniformly distributed and strong similarity matches are rare. Huber al. (2021b). describe procedure to sample pairs of spectra such that the molecular similarities will be uniformly distributed. We use this procedure to sample pairs for training and evaluation tasks.
Following Huber et al. (222bb) we also train a Siamese network, but we do so using the base model transformer architectures outlined in Section 2. instead of with the binned mass vectors used by Huber. We report performance results for models trained using PEtoken and PEsin below in Section
2.2.2 PROPERTY PREDICTION
We also train the base model architecture predict 10 chemical properties relevant medicinal chemistry directly from an MS2 spectrum. We do so by training the following network,
Propertics FF (SpectrumEncoder (S))
where the final layer has dimension equal to the number of properties we wish to predict. We also scale all of our training labels to zero mean and unit variance. For inference, we run predicted properties through an inverse-scaler. As in Section 2.2.1 we train multiple versions of this property prediction model. Due to the lack of existing methods for comparison, Equation [ on binned spectrum representations for a simple baseline. We replace SpectrumEncoder with the MS2Deepscore feed forward architecture found in Huber et al. (2021b) for this baseline.
also train version of
5

---

Under review as a conference paper at ICLR 2023
The properties predict are standard indicators of druglikeness and bioavailability in the field of medicinal chemistry. These include the properties that make up the Quantitative Estimate of Druglikeness (QED) along with several others (Bickerton et al. 2012 [ et al. 2012 Veber et al., 2002). For the complete list of properties, see Table 2 All properties are computed deterministically from chemical structure using RDKit (Landrum et al. 2013), there is additional dependency on experimental data.
3 RESULTS
3.1 DATA
We construct set of labeled MS2 spectra by combining a number of standard public and commercial
datasets and one small dataset 0.1% of spectra) that requires permission to access because
contains psychoactive substances (Wang et al. 2016 Mehta 2020 Sawada et al. [012 Horai et al.
2010, Mikaia et al., 2014, [mith et al.] 2005| Mardal et all]|2019] See our Reproducibility Statement for additional preprocessing details. Chemical properties and fingerprints are then computed from the cleaned molecules using RDKit (Landrum et al., 2013). Our resulting dataset has 1,251,830 spectra corresponding to 45,351 distinct structures.
In metabolomics experiments, biological samples contain compounds that have previously been profiled in MS2 libraries ("known" compounds) and compounds that have not been profiled ("novel compounds"). Identifying the structure and properties of known molecules is an important task on its own (e.g. "dereplication") We therefore split our data that the test and development sets partially disjoint from train at the level of structures, and fully disjoint from train at the level of spectra. This enables us evaluate model performance on both "known" and "novel" compounds. To implement this split, we partition our set of labeled spectra into a training set, development set (i.e. validation set), and test set as follows. We randomly select set of 1002 molecules from our data and place them and all of their associated spectra into the development set. From the remaining molecules, we follow the same procedure and place 1002 molecules into the test set. This creates a test and development set containing 1002 molecules each that are fully disjoint from each other. The remaining molecules are assigned the training set. We then remove 998 spectra that correspond to 941 molecules represented in the training set to the development set, and 998 to the test set, corresponding to 953 structures from the training set. The final training, development, and test sets have 1,214,812 18,750, and 18,268 spectra respectively. The training set contains 43,347 structures. We use the development set for hyperparameter optimization and only access the test set to compute reported metrics.
3.2 SPECTRAL SIMILARITY
To benchmark spectral library search, we apply two criteria for accurate molecular identification: retrieving the exact molecule and retrieving molecule which is an approximate match, i.e. has molecular similarity (defined in section [.2.1] greater than 0.6. To account for widely varying numbers of spectra for each molecule, report the accuracies macro averages over molecular structures.
We train number of spectral similarity models that report state of the art performance are in standard usage as described in Section 2.2. We refer to the sinusoidal Siamese transformer and tokenized Siamese transformer models as the sinusoidal and tokenized models respectively. To test performance against reported state of the art models, we train both a Spec2Vec (Huber et al. 2021a) and an MS2Deepscore (Huber et al., (021b) model on our data. We also show results modified cosine spectral similarity, which is a standard, unlearned, similarity function common in applications (Watrous et al. (0022). In Table T we report spectral library search accuracies for both known and novel compounds obtained from number of spectral similarity models. We omit exact match accuracies for novel compounds since these have been sampled so that no exact matches can be found. For models where it is applicable, the same table reports the M SE loss between predicted and actual molecular similarity on spectrum pairs drawn from train, known, and novel spectra sets.
We find that learned approaches improve modified-cosine spectral library search. Of the learned approaches, we find that Spec2Vec has strong, only surpassed by the double precision sinusoidal model, spectral library search performance on novel, but only beats MS2Deepscore known.

---

Under review as a conference paper at ICLR 2023
Table 1: Spectral library search performance
MSE
Spectral library search accuracy
Spectral Set Match
train
known
novel
known
novel
Exact
Approx.
Approx.
Modified Cosine
0.61
0.672
0.351
Spec2 Vec
0.903
0.954
0.414
MS2Deepscore
0.026
0.029
0.047
0.852
0.918
0.387
Tokenized m/z
0.017
0.019
0.030
0.933
0.961
0.382
Sinusoidal
0.019
0.019
0.03
0.913
0.955
0.401
m/z (float16)
Sinusoidal
0.013
0.015
0.024
0.937
0.97
0.435
m/z 64)
MS2Deepscore only outperforms the tokenized model on novel and underperforms everything on known. Moreover, MS2Deepscore is substantially less accurate at predicting the specific pair molecular similarity on known and novel spectra when compared to all transformer approaches. The double precision sinusoidal model produced the best performance on all evaluation tasks. This model produces state of the art known spectral library search accuracies of 0.937 and 0.97 for exact and approximate matching respectively. Numerical representation of MS2 m/2 via sinusoidal embeddings consistently outperforms all the discretization based approaches we benchmarked.
To assess whether the multi-scale embeddings are learning from high resolution information, we also train and evaluate versions of our sinusoidal model where m/ z's are cast to half precision floating point values. Half precision floating point can only resolve up approximately parts per O 000), far below the parts per million precision of experimental mass spectrometry data. The performance of the half precision model, reported in Table I drops to be on par with the tokenized model (worse on known and better novel). We do not report results with single precision m/2 since these were indistinguishable from double precision m/2. On evaluation metrics, using m/z inputs of precision greater than 16 bits dramatically improves performance of the sinusoidal model. This indicates that sinusoidal embedding of m/z values enables learning from the the high resolution portion of mass spectrometry data.
3.2.1 QUALITATIVE EMBEDDING ANALYSTS
To further characterize the respective properties of token and multi-scale sinusoidal embeddings, we inspect UMAP (MeInnes et al., 2018) projections of our m/z embeddings in Figure 3 For siamese transformer models described in Section 2.2. For our sinusoidal models, we embed 50,000 m/z values between and 1,000 Daltons using FF (SE (m/2)). We do not use the full peak embedding function as we are not interested in the embedding of intensity information. Because our tokenization procedure involves rounding m/ /2 values to decimal place, for the tokenized model we only embed 10,000 m/2 values between and 1,000 Daltons.
this analysis,
While m/2 measurements are continuous, the space they represent is inherently discrete. Two molecular fragments differing in atomic composition could nevertheless have a very similar m/2's. However, the large separation between molecular mass scales and the mass scale at which nuclear forces manifest themselves means that the relative atomic composition may be deduced from the fractional m/z (Jones et al. 2004, Pourshahian & Limbach, 2008), defined by (m/2) m/z [m/z]. Therefore, we expect a quality embedding of m/ z to embed fragments similar in both m/z and (m/2} close to one another, while paying greater attention to the latter.
As is seen in Figure 3 embeddings from tokenized and low precision sinusoidal models are able to capture general trends in m/z. However, they fail to preserve distances in m/2 and show little to no structure in fm/2). In contrast, the high precision sinusoidal embeddings allow our models to represent important information in (m/z), while preserving distance in m/2.
7

---

Under review as a conference paper at ICLR 2023
Tokenized
1000
Half Precision Sinusoidal
Double Precision Sinusoide!
LN
400
400
200
200
200
a
Sinusoloal
Precision Sinuseidal
0.8
0.8
06
0.6
0.5
0.4
0.4
(b)
Figure 3: (3a) UMAP projections of m/z emheddings colored by m/2 value. (3b) UMAP projections of m/z embeddings colored by {m/z]. High resolution sinusoidal embeddings imbue model latent space with additional structure not found in other baseline models.
3.3 PROPERTY PREDICTION
Many applications metabolomics can be unblocked with knowledge just limited of chemical properties, without need for identification of molecular structure. Towards this end, we investigate the parallel task of chemical property prediction from MS2 spectra. We evaluate sinusoidal vs. tokenized m/2 embedding performance on list of properties that have compelling applications in drug discovery and are easily computable from our molecule labels. For this experiment, we train sinusoidal model and several baseline models as described in Section 2.2.2
In Table 2 property prediction models are evaluated on known and novel structures using R². All transformer models outperform our feed forward baseline. Half precision sinusoidal models produced an R² 0.746 (0.936) on known (novel) molecules averaged across all 10 properties. These results competitive with tokenization but substantially underperform when compared to higher precision sinusoidal embeddings. When m/2 values are specified at double precision, sinusoidal embeddings show improvements over tokenization each property, resulting in 11% improvement over tokenization on R² for novel molecules averaged across all properties.
This is further evidence that the multi-scale information is captured by sinusoidal embeddings and allows improved generalization across tasks. In addition, the performance is strong enough to enable medicinal chemistry applications, such hit prioritization based on druglikeness. This has major implications in the drug discovery context, as inspecting the properties of unknown molecules in complex mixtures was previously not possible without difficult isolation steps and individual experimentation.
4 CONCLUSIONS
The results presented here the first demonstration that the sinusoidal representation of m/2s measured tandem mass spectrometry experiments enables deep learning models to learn from high resolution mass data. These results include state of the art performance spectral library search and
8

---

Under review as a conference paper at ICLR 2023
Table 2: R² for predicted chemical properties
Feed Forward
Tokenized m/z
Sinusoidal m/z
Property
Known
Novel
Known
Novel
Known
Novel
all
0.824
0.604
0.943
0.72
0.976
0.8
atomic log P
0.768
0.357
0.919
0.437
0.968
0.622
number of hydrogen bond acceptors
0.835
0.762
0.969
0.887
0.987
0.941
number of hydrogen bond donors
0,834
0.572
0.924
0.683
0.968
0.69
polar surface area number of rotatable bonds number of aromatic rings number of aliphatic rings number of heteroatoms fraction sp3 carbons quantitative estimate of druglikeness
0.851
0,716
0.965
0.862
0.986
0.907
0.801
0.669
0.939
0.792
0.976
0.838
0.838
0.372
0.934
0.474
0.981
0.655
0.839
0.638
0.943
0.762
0.974
0.821
0,823
0.719
0,968
0.902
0.988
0.946
0.846
0.618
0.951
0.723
0.977
0.821
0.806
0.617
0.912
0.681
0.952
0.755
property prediction tasks in metabolomics. The property prediction results of are sufficient quality to functionally inform and advance drug-discovery efforts.
Moreover, present results that compare across numerical floating point precisions for the mass inputs, and show that sinusoidal embeddings perform better when higher mass resolution data is used. Finally, we visualize high-dimensional structure of the mass embeddings that emerges only when high precision mass values are utilized. We expect that sinusoidal embeddings of m/z will be useful component of further machine learning applications with MS2 data.
REPRODUCIBILITY STATEMENT
We train model combination of free public datasets (Wang al. 2016 Mehta 2020
Sawada et al. 2012 Horai et al. 2010), commercially available datasets (Mikaia et al. 2014 Smith 2005), and small dataset (0.1% % of all spectra) available if user-group of qualified researchers joined (Mardal et al. 2019). The necessary preprocessing steps are outlined in Section 3.1 Additionally, we allow spectra collected across instrument types and experimental parameters including positive and negative ion mode, collision energies, etc. We constrain that spectra in our dataset have least 5 peaks and least decimal places of m/z resolution. We strip all stereochemistry from our molecular structure labels, which is common step taken in MS2 modeling and allows for better molecule-disjonnt splitting.
The model architecture is explained in detail in Section and all parameters used in model training are specified Section 2.2 We include our source code the supplementary materials.
REFERENCES
Dávid Bajusz, Anita Rácz, and Károly Héberger. Why tanimoto index an appropriate choice for fingerprint-based similarity calculations? Journal of cheminformatics, 7(1):1-13, 2015.
G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90-98, 2012.
Sebastian Böcker and Florian Rasche. Towards de novo identification of metabolites by analyzing tandem mass spectra. Bioinformaties, 24(16)i14-i55, 2008
Liu Cao, Mustafa Guler, Azat Tagirdzhanov, Yu-Y Lee, Alexey Gurevich, and Hosein Mohi- mani. Moldiscovery: learning mass spectrometry fragmentation small molecules. Nature Communications, 12(1):1-1g 2021.
Kai Dührkop. Huibin Shen, Marvin Meusel, Juho Rousu, and Sebastian Böcker. Searching molecular structure databases with tandem mass spectra using esi: Fingerid. Proceedings of the National Academy of Sciences, 1 12804)) 22500-22588 12585, 2015.

---

Under review as a conference paper at ICLR 2023
Warwick B Dunn, Alexander Erban, Ralf JM Weber, Darren J Creek, Marie Brown, Rainer Breitling, Thomas Hankemeier, Royston Goodacre, Steffen Neumann, Joachim Kopka, et al. Mass appeal: metabolite identification in mass speetrometry-fooceee untargeted metabolomies. Metabolomics, (1):44666, 2013.
William Falcon. The pytorch lightning team. Pytorch lightning, 3:6, 2019.
Gary L. Glish and Richard W Vachet. The basics of mass spectrometry in the wentty-frss' century. Nature reviews drug discovery, 2(2):140-1500 2003.
Hisayuki Horai, Masanori Arita, Shigehiko Kanaya, Yoshito Nihei, Tasuku Ikeda, Kazuhiro Suwa, Yuya Ojima, Kenichi Tanaka, Satoshi Tanaka, Ken Aoshima, et al. Massbank: public repository for sharing mass spectral data for life sciences. Journal of mass spectrometry, 45(7):703-714, 2010.
Florian Huber, Lars Ridder, Stefan Verhoeven, Jurriaan H Spaaks, Faruk Diblen, Simon Rogers, and Justin JJ Van Der Hooft. Spec2vec: Improved mass spectral similarity scoring through learning of structural relationships. PLoS computational biology, 17(2): 1008724. 2021a.
Florian Huber, Sven van der Burg, Justin JJ van der Hooft, and Lars Ridder. Ms2deepscore: a novel deep learning similarity measure to compare tandem mass spectra. Journal of cheminformatics, 13 (1):1-14, 2021b.
Jeffrey Jones, Michael J Stump, Richard C Fleming, Jackson Lay, and Charles L Wilkins. Strategies and data analysis techniques for lipid and phospholipid chemistry clucidation by intact cell maldi-ftms. Journal of the American Society for Mass Spectrometry, 15(11):1655-167, 2004.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
Svetlana Kutuzova, Christian Igel, Mads Nielsen, and Douglas McCloskey. Bi-modal variational autoencoders for metabolite identification using tandem mass spectrometry. bioRxiv, 2021.
Greg Landrum et al. Rdkit: software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 2013.
Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. Learnable fourier features for multi- dimensional spatial positional encoding. Advances in Neural Information Processing Systems, 34: 15816-15829, 2021a.
Yuanyue Li. Tobias Kind, Jacob Folz, Arpana Vaniya, Sajjan Singh Mehta, and Oliver Fiehn. Spectral entropy outperforms ms/ms dot product similarity for mmall-molecule compound identification. Nature Methods, 18(12):552--153) 2021b.
Christopher A Lipinski, Franco Lombardo, Beryl W Dominy, and Paul J Feeney. Experimental and computational approaches to estimate solubility and permeability in drug discovery and development settings. Advanced drug delivery reviews, 64:4-17, 2012.
Eleni Litsa, Vijil Chenthamarakshan, Payel Das, and Lydia Kavraki. Spec2mol: An end-to-end deep learning framework for translating ms/ms spectra de-novo molecules. 2021.
Marie Mardal, Mette Findal Andreasen, Christian Brinch Mollerup. Peter Stockham, Rasmus Telving, Nikolaos Thomaidis, Konstantina S Diamanti, Kristian Linnet, and Petur Weihe Dalsgaard. Highresnps. com: an online crowd-sourced hr-ms database for suspect and non-targeted screening of new psychoactive substances. Journal of Analytical Toxicology. 2019.
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 1rXiv:1802.03422 2018.
SS Mehta. Massbank of north america (mona): An open-access, auto-curating mass spectral database for compound identification in metabolomics presentation, 2020.
10

---

Under review as a conference paper at ICLR 2023
Anzor Mikaia, Principal Edward White V EI, Vladimir Zaikin EI, Damo Zhu EI, O David Sparkman EI, Pedatsur Neta, Igor Zenkevich RI, Peter Linstrom, Yuri Mirokhin, Dmitrii Tchekhovskoi, et al. Nist standard reference database la. Standard Reference Data, NIST, Gaithersburg, MD, USA 2014.
tions in vector space. arXiv preprint weee://itttttnnuddddeeeeeeeeeeee
https://ww:: nist.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation word representa- 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Soheil Pourshahian and Patrick A Limbach. Application of fractional mass for the identification of eeptide--ligonucleotte cross-links by mass spectrometry. Journal of mass spectrometry, 43(8): 1081-1088, 2008.
Rui Qiao, Ngoc Hieu Tran, Lei Xin, Baozhen Shan, Ming Li, and Ali Ghodsi. Deepnovov2: Better de novo peptide sequencing with deep learning, arXiv preprint 1xXiv:194.00855 2019.
Robert Quinn, Louis-Felit Nothias, Oliver Vining. Michael Meehan, Eduardo Esquenazi, and Pieter C Dorrestein. Molecular networking as a drug discovery, drug metabolism, and precision medicine strategy. Trends in pharmacological sciences, 38(2); 14--5:44,155, 2017.
Yuji Sawada, Ryo Nakabayashi, Yutaka Yamada, Makoto Suzuki, Muneo Sato, Akane Sakata, Kenji Akiyama, Tetsuya Sakurai, Fumio Matsuda, Toshio Aoki, et al. Riken tandem mass spectral database (respect) for phytochemicals: plant-specific ms/ms-based data resource and database. Phytochemistry, 82:38 45, 2012.
Aditya Divyakant Shrivastava, Neil Swainston, Soumitra Samanta, Ivayla Roberts, Marina Wright Muelas, and Douglas B Kell. Massgenie: A transformer-bast based deep learning method for identifying small molecules from their mass spectra. Biomolecules, 11(1)):1933, 2021.
Colin A Smith, Grace O' O'Maille, Elizabeth J Want, Chuan Qin, Sunia Trauger, Theodore Brandon, Darlene E Custodio, Ruben Abagyan, and Gary Siuzdak. Metlin: a metabolite mass spectral database. Therapeutic drug monitoring. 7766)7777775, 2005.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 19(1):1929-199 1958, 2014.
Stephen E Stein and Donald R Scott. Optimization and testing of mass spectral library search algorithms for compound identification. Journal of the American Society for Mass Spectrometry, (9):859-866, 1994.
Justin Johan Jozias van Der Hooft, Joe Wandy, Michael Barrett, Karl EV Burgess, and Simon Rogers. Topic modeling for untargeted substructure exploration in metabolomics. Proceedings of the National Academy of Sciences, (33448)117338111 13743, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Daniel F Veber, Stephen R Johnson, Hung- Yuan Cheng, Brian R Smith, Keith W Ward, and Kenneth D Kopple. Molecular properties that influence the oral bioavailability of drug candidates. Journal of medicinal chemistry, 45(12):2615-2623 2002.
Mingxun Wang. Jeremy Carver, Vanessa V Phelan, Laura M Sanchez, Neha Garg, Yao Peng, Don Duy Nguyen, Jeramie Watrous, Clifford A Kapono, Tal Luzzatto- Luzzatto-knann, al. Sharing and community curation of mass spectrometry data with global natural products social molecular networking. Nature biotechnology, 34(8):828 837, 2016.

---

Under review as a conference paper at ICLR 2023
Jeramie Watrous, Patrick Roach, Theodore Alexandrov, Brandi S Heath, Jane Y Yang, Roland D Kersten, Menno van der Voort, Kit Pogliano, Harald Gross, Jos M Raaimmakers, et al. Mass spectral molecular networking of living microbial colonies. Proceedings of the National Academy of Sciences, 109126):E1773--11122 2012.
Melih Yilmaz, William Fondrie, Wout Bittremieux, Sewoong Oh, and William S Noble. De novo mass speytrometry peptide sequencing with a transformer model. In International Conference on Machine Learning, pp. 25514-25522. PMLR, 2022.
Şule Yilmaz, Elien Vandermarliere, and Lennart Martens. Methods to calculate spectrum similarity. In Proteome bioinformatics, pp. 75- Springer, 2017.
Xi-wu Zhang, Qiu-han Li, Jin-jin Dou, et al. Mass spectrometry- metabolomics in health and medical science: A systematic review. RSC advances, 10(6):3022-3104, 2020.
12


Reviews:
[{"review_id": "xFcBB6UJ-Mf", "reviewer": "Reviewer_EA9S", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper applied sinusoidal embeddings for spectral library search in mass spectrometry in metabolomics. ", "summary_of_the_review": "The paper empirically demonstrates the advantage of using sinusoidal embeddings for spectral library search in metabolomics. The idea of using sinusoidal embeddings to avoid binning of ms spectrum is not novel and was recently published. The authors take this idea one step further and demonstrate that it is superior that currently existing simple benchmarks for spectral library search in the field of metabolomics. The biggest contribution of this paper is the empirical demonstration of their results, but this section is not transparent at all.", "strength_and_weaknesses": "The authors propose to use sinusoidal embedding to obtain a representation of the measured spectrum, allowing the comparison of similarities between measured compounds. The advantage of using sinusoidal embeddings in mass spectrometry is significant and was already demonstrated by Yilmaz et al. Here the authors additionally confirmed it with experiments in metabolomics. \n\nMinor comments:\n* What is a “development” set? What happened to the “validation” set? Why do the authors call \n“validation” set “development” set?\n* From the description in section 3.1 it is unclear whether test and validation sets can have an overlapping set of molecules. It is clear that they don’t overlap with training, but not clear if they overlap with each other. The whole section is not very clear. The part about known and not known molecules is not very clear either. Also stratification by spectrum alone and not by molecules is also unusual. I have concerns that the model can overfit individual molecules. The network proposed by the authors is much larger than the benchmarks and it is not clear to me how the authors demonstrated overfitting if the test set contains molecules present at training.\n* Section 3.2 is not well written. The metrics are not well-defined. The thresholds seem arbitrary. It is not clear at all how these numbers were estimated. Was FDR used to find these cutoffs? Maybe these metrics are commonly used in metabolomics, but they should be clearly described for a machine learning conference.\n* The authors demonstrate the improved numbers in their tables, but since the metrics and the whole evaluation are not transparent, it is hard to assess what is the actual significance of these numbers.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rPrf_k4k35T", "reviewer": "Reviewer_oTst", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use sinusoidal embeddings to represent m/z and intensity values when modeling metabolomics mass spectrometry data using deep learning models.  The idea is sensible though not particularly novel.  Such embeddings have been used previously for mass spectrometry data (as referenced in the related work section), just not for data generated from small molecules.  Three sets of results are given.  The first set I found confusing (see details below), but the upshot there seems to be that the proposed method works better than state-of-the-art methods.  The second set of results involve interpreting UMAPs.  I didn't find this section compelling, and I think it should be eliminated.  The third set of results involves predicting physical properties of molecules directly from spectra. This seemed like an interesting and potentially impactful application, and I was surprised that no one apparently has worked on this problem previously.\n", "summary_of_the_review": "This paper addresses an important problem but uses a fairly straightforward technique that has been used previously in very similar settings.  The paper is at times hard to follow, and of the three sets of results only the third is very compelling.\n", "strength_and_weaknesses": "Overall, the paper makes a modest contribution in terms of novelty, and with some reorganization could tell a reasonably compelling story.\n\nOne problem is that the paper is not written in a way that is accessible to an ICLR audience.  I will try to outline below some places where I got confused, but keep in mind that I have a lot more experience working with mass spec data than your typical ICLR reader.  I'm a bit worried that you will find my list of questions below frustrating to read, because probably many of the answers to my questions are embedded in the text.  My goal is to show you how a reasonably well-informed reader got confused by the paper as written.\n\nIn the paragraph that spans pp. 1-2, I don't understand why discretization techniques should be \"particularly susceptible to measurement errors.\"  Also, it seems strange to me that this discussion doesn't mention edge effects, which is another clear drawback of the discretization approach (separate from the loss of precision).\n\nThe final paragraph of the introduction does not do a good job of making clear whether the proposed method represents an improvement relative to the current state of the art.\n\nThere is something important that I simply don't understand about the spectral library search application.  My understanding is that the state of the art method is to use a sort of generalized scalar product (or cosine similarity) that is carried out directly on the non-discretized list of peaks.  I think this is the method used in several widely used library search tools, e.g., from Pieter Dorrenstein's group.  I think the paper should explain how methods like the ones cited in Section 1.1 (e.g., Huber 2021b) improve on this simple approach that does not require discretization.  Results from a simple method like this, as well as perhaps cosine similarity carried out on discretized vectors, should be included in Table 1.\n\nAlso in Section 1.1, please explain what it means to do discretization by \"direct molecular formula assignment.\"  Without a definition, it's impossible to understand the subsequent points made in this paragraph.\n\nThe paper claims that \"the peaks have no intrinsic ordering,\" but I disagree.\nIndeed, it seems quite strange to me that the peaks in the spectrum are not sorted by m/z for input to the model.\n\nFor the comparison to tokenized peak embedding (Section 2.1.1), it seems like a key parameter is the bin size of 0.1.  To make the comparison informative, you should vary this parameter to find an optimal setting for a given dataset.\n\nThe paper uses the term \"molecular similarity,\" but I don't really know what this means.  For example, we are told that \"Molecular similarity scores range from 0 to 1.\"  Which specific scores are referred to here, and how are they computed.  This seems critical because later there are thresholds (0.60 and 0.95) that get applied to these scores.\n\nI find it highly problematic to report results based on an \"internal, proprietary dataset,\" since this renders all of the results irreproducible.  The experiments should be performed only using publicly available data, and that data, including the splits, should be made available along with the paper.\n\nSection 3.1 uses a lot of jargon that needs to be explained for an ICLR audience.  E.g., \"positive and negative ion mode\" and \"collison energies.\"  What is involved in stripping stereochemistry from molecular structure labels, and why does this step allow for better molecule-disjoint splitting?  What are \"chemical fingerprints\"?\n\nI don't understand what the phrase \"lookup reference\" refers to.  Why don't you just say that you query it against the training set?  What is being \"looked up\" here?  Later, you refer to a \"reference set.\" Is this just the same as the training set?  Please be consistent in your terminology.\n\nIn the evaluation framework, why it it sensible to use arbitrary score thresholds (0.60 and 0.95)? Why not just use a rank-based approach akin to ROC?  It seems to me that if you really are interested in doing spectral library search, what matters is whether the top-ranked match for each query is in fact the correct molecule.  This gives you a ranked list of spectrum-spectrum matches, where each has a binary label of correct or incorrect.\n\nWhat does it mean to \"macro average\" something?\n\nI got lost in the paragraph that begins \"Because novel molecules ...\"  Where do the numbers 0.31 and 0.89 come from? Is this an empirical result, and if so, how is it computed.  I am not convinced that the random baseline is particularly interesting.  What does seem interesting is how the generalized scalar product, which is trivial to compute and requires no discretization, would perform.\n\nI did not find the UMAP analysis particularly informative or compelling.\n\nThe property prediction task seems quite interesting, and I am surprised that no one has done this before.  \n\nMinor:\n\nOn p. 1, modify the sentence that talks about \"modeling m/z values as numeric quantities\" to clarify that you are contrasting this with the discretization approach.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Hfy4WGy6QO1", "reviewer": "Reviewer_4K3S", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed multi-scale sinusoidal embeddings for spectral library search.  Then the proposed sinusoidal m/z peak embeddings are fed into a transformer to get a single embedding vector as an output, which is then fed into a feedforward neural network to predict chemical properties.", "summary_of_the_review": "Overall, because of the niche topic that is less relevant to ICLR and the closed-source dataset and code, I don't think this paper is appropriate for ICLR acceptance. ", "strength_and_weaknesses": "Strength: The idea of sinusoidal embeddings seems to be well-motivated for this special task: MS analysis. Also, the authors explain the idea quite clearly and the paper is well-written.\n\nWeakness: Firstly, I think the analysis of tandem mass spectrometry is a very niche problem in metabolomics, which seems to be less relevant to the focus of the ICLR conference. I was wondering if this paper could submit to a more specialized conference and journal. \nSecond, it seems to the reviewer that neither the code nor the dataset can be made publicly available because of intellectual property reasons, which are against the spirit of ICLR and the reviewer doubts its potential impact on the scientific community.   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject