MS$^2$-Transformer: An End-to-End Model for MS/MS-assisted Molecule Identification
Authors: Mengji Zhang
Yingce Xia
Nian Wu
Kun Qian
Jianyang Zeng
Venue: ICLR 2022 Submitted (2022)

Abstract: Mass spectrometry (MS) acts as an important technique for measuring the mass-to-charge ratios of ions and identifying the chemical structures of unknown metabolites.  Practically, tandem mass spectrometry (MS/MS), which couples multiple standard MS in series and outputs fine-grained spectrum with fragmental information, has been popularly used.  Manually interpreting the MS/MS spectrum into the molecules (i.e., the simplified molecular-input line-entry system, SMILES) is often costly and cumbersome, mainly due to the synthesis and labeling of isotopes and the requirement of expert knowledge.  In this work, we regard molecule identification as a spectrum-to-sequence conversion problem and propose an end-to-end model, called MS$^2$-Transformer, to address this task.  The chemical knowledge, defined through a fragmentation tree from the MS/MS spectrum, is incorporated into MS$^2$-Transformer.  Our method achieves state-of-the-art results on two widely used benchmarks in molecule identification. To our best knowledge, MS$^2$-Transformer is the first machine learning model that can accurately identify the structures (e.g., molecular graph) from experimental MS/MS rather than chemical formula/categories only (e.g., C$_6$H$_{12}$O$_6$/organic compound), demonstrating it the great application potential in biomedical studies.

Under review as a conference paper at ICLR 2022
MS TRANSFORRME AN END-TO-END MODEL FOR MS/MS- MOLECULE IDENTIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Mass spectrometry (MS) acts an important technique for measuring the mass- to-charge ratios of ions and identifying the chemical structures of unknown metabolites. Practically, tandem mass spectrometry (MS/MS), which couples multiple standard MS in series and outputs fine-grained spectrum with fragmental information, has been popularly used. Manually interpreting the MS/MS spec- trum into the molecules (i.e., the simplified molecular-inuu line-entry system, SMILES) is often costly and cumbersome, mainly due to the synthesis and la- beling of isotopes and the requirement of expert knowledge. In this work, we regard molecule identification as spectrum-to-sequenee conversion problem and propose end-to- model, called Transfoanfrr,,r to address this task. The chemical knowledge, defined through a fragmentation tree from the MS/MS spee- trum, is incorporated into MS MS*-Transformer Our method achieves state-of-the-art results on two widely used benchmarks in molecule identification. To our best knowledge, MS²-Transformee is the first machine learning model that can accu- rately identify the structures (e.g., molecular graph) from experimental MS/MS rather than chemical formula/categories only (e.g., CoHssoo/orgaanic compound), demonstrating the great application potential in biomedical studies
INTRODUCTION
Mass spectrometry (MS)-based metabolomics plays critical role in life science (Klünemann et al.) 2021 Banh al. 2021), which assists in discovering molecular biomarkers/drugs for diagno- sis/therapy. MS measures the mass-to-charge (m/z) ratios of adduct ions and outputs mass spec- trum where intensity denotes the ion abundance at the corresponding m/z. In real-world applications, tandem mass spectrometry (MS/MS) is more used, where two standard MS are sequentially coupled for a more fine-grained spectrum than using one MS only (denoted as MS1). In MS/MS, the sec- ond MS (denoted as MS2) records the intensities of the fragments which are broken down from the adduct ions in MS1. As a result, MS/MS affords the capability of separating and identifying ions with close m/z-ratios in MSI based on their fragments in MS2. An example is shown Figure Ta), where multiple peaks exist in different m/z ratios. In MS/MS spectrum, the correlations of peaks determine the strengths of chemical bonds, and the locations peaks indicate the masses of potential functional groups.
To identify the molecules based on MS/MS, conventionally, there are two types of approaches. The first one is accurate mass analysis. in which an experimentally measured mass determines the elemental composition by mass matching. However, such an approach has high false-positive rates, as shown in (Kind & Fiehn 2006; Shen et al.) 2019). The other approach is database searching (DS), in which a matching score between the experimentally obtained spectrum and the database made up of standard spectra is calculated. However, DS does not bring significant improvement in molecule identification due to the limitation of the database. For example, about 90% of known metabolites don't 't have reference tandem mass spectra in METLIN (https://melin.scctt)) wwt.pps..cuuu and HMDB (http://www.hhr www.hmdh.ca/ databases (Shen et al.] 2019 Vinaixa et al. 2016). To overcome this difficulty, researchers turn more careful chemical designs. For example, people use CaClllbelled isotopes to determine the number of carbons (Tsugawa et al. 2019), but the cost is too high (about $2000 per gram).
Cooee available http:///ghhhi...oomommmmm/mmmomh

---

Under review as a conference paper at ICLR 2022
Correspondingly, identifying metabolites with machine learning methods is trend and progress has been achieved in metabolite classification by neural networks recently (Dührkop et al. 2021). However, rough classification of chemical categories, like lipids or benzenoids, still cannot solve the acute problem of identifying the specific structure for profound medical research (e.g.. the accurate structure recognition in antihody-antige interaction), leaving great space for MS/MS identification.
In this work, we regard the task as ppectrum-to-seqeene task, where the input is the MS/MS spec- trum, and the output is the molecule structure, which is represented by the simplified molecular-input line-entry system (SMILES). Considering that chemists often leverage fragmentation tree (Figure [T) for molecule identification, which describes the relations the peaks in the spectrum, we corporate it into our model. In fragmentation tree, the nodes represent the peaks, and the edges represent the fragmentation reactions between peaks. The detailed calculation process is described in Figure Db mainly including three steps: initialization of fragmentation graph, scoring of frag- mentation graph, and ranking of fragmentation trees. A fragmentation tree can be obtained by the SIRIUS chemical tool (Dührkop et al. 2019).
a
MS1
[Adduct ion]"
Initialize fragmentation graph
Iniiaaize every node by enumerating all possible molecular formulse matching the moncisolopic
+
Score fragmentation graph
Calzulate the log odds diferentiate fragments and for nodes;
Collision
Surface Photon
common
edges.
MS2
[Fraga]*
[Fragu]'
[Adduct ion]
[Fragz]
[FragJ]
Rank the fragmentation tree
[Frag,]'
Enumerral all the possible fragmentation trees and choose the with nighee score.
Figure I: Workflow of MS/MS and fragmentation tree. (a) Illustration for adduct ions and fragmen- tation of metabolites. Metabolites firstly adducted by selected cations to form [Adduct ion]* in MS1. Then, [Adduct ion]* is fragmented to form fragments in an inert gas atmosphere, and the resulted fragments are recorded MS2. (b) Calculation process of fragmentation tree. Firstly, ini- tialization of fragmentation graph is performed by enumerating all possible molecular formulas for every node by matching the monoisotopic mass. Secondly, scoring is calculated separately for nodes and edges. For nodes, calculate the mass difference between the measured peak and the molecular formula, then compute log odds differentiate the fragment and the background noise. The princi- ple behind this is that mass differences for fragments are assumed to be Normal distributed, while exponentially distributed for noises. For edges, calculate the common neutral losses, which are typ- ically described as the loss of small molecules (e.g., water and ammonia), not ions. Thirdly, ranking of fragmentation trees is performed by choosing the one with highest score among the possible fragmentation trees.
We propose an end-to-end model, MS Transformer, for MS/MS assisted molecule identification. follows the encoder-decoder framework, and it takes both the spectrum and the fragmentation tree as input. Our model consists of three modules: (I) Peak embedding module, where we map the (m/z ratio, intensity) pairs to high-dimension vectors. The embeddings are used for each peak in the spectrum and each node in the tree. (2) Fragmentation aggregation module, where the encoder of Transffasfrrmer consists of stacks of blocks. Each block is made up of a self-attention layer (used to process the spectrum), graph encoding layer (used to process the fragmentation tree), and feed-forward layer (used to fuse the features extracted by the above two
2

---

Under review as a conference paper at ICLR 2022
layers). In this way, the global information in the spectrum and the local information in the tree fully leveraged. (3) SMILES reconstruction module, where the decoder is the standard Transformer decoder, which outputs the SMILES based on the output of the encoder. We conduct experiments on widely used datasets for MS/MS, MassBank and CASMI, and have achieved state-of-the-ard results.
The main contributions of our work are follows:
We propose MSS--Tranffomms machine learning-basee model to identify the molecule with structural information based on Fragmentation Aggregation (FA) module and Trans- former. Experiments on benchmark datasets validate that MM2--ransfoome achieves state- of-the-art performance.
We interpret the representation of MS/MS data as information aggregation on chemical bonds globally and locally. The global aggregation on chemical bonds performed among all peaks, referred as fully-connected graph. The local aggregation on bonds is performed among partial peaks, which are estimated by the precursor-product relationship via frag- mentation tree, referred locally-connected graph.
We introduce self-attention mechanism and message passing neural network in the aggre- gation on fully- and locally connected graphs for learning the representation of bonds' strengths from MS/MS data.
2 PROBLEM FORMULATION AND NOTATIONS
Let X' and Y' denote source domain and target domain, which are collections of MS/MS spec- trum and molecules, respectively. We assume that the molecules are represented by the canonical SMILES, which provides unique representation string for one molecule. Our task is to learn mapping f X + V. that can identify the molecule from the input spectrum. We focus on the supervised setting, where the dataset consists of M spectrum molecule I; E X. y: y.
ppeltrum-molccule pairs D {(2+ 9:)
Each x; can be further decomposed into z, = {(my, Lm,) ) where m, denotes the m/z ratio, Img represents the corresponding ion intensity, and N, denotes the number of peaks in the spectrum. Note that the m/z ratio is discrete, while the intensity is continuous.
Distinct yet limited patterns can be summarized from MS/MS spectra, providing assistance for the chemists to infer the molecular candidates from bio-fluids. The patterns carrying prior knowledge can be divided into monoisotopic mass and fragmentation patterns.
Monoisotopic mass calculates the molecular mass by summing up the atoms' masses with their most abundant naturally occurring stable isotopes, and it can provide assistance in enumerating all the potential chemical formulas. For example, as shown in Figure (2h the recorded m/z ratio of 1030 Da with a negligible shift of 1.0949 mDa from its monoisotopie mass, could help infer the potential candidate with a chemical formula of CgH13NO2-
The fragmentation pattern is derived from the correlations among peaks, and it can help find potential functional groups that constitute the molecule. For example, as illustrated in Figure 2p, Am/z of 18 Da may denote the dehydration process of losing an H2O, indicating that there exists hydroxyl in the target molecule's 's chemical structure.
Thus, based on the monoisotopic masses and the fragmentation patterns, researchers can enumer- ate all the possible chemical formulas and functional groups, making it feasible to calculate the hypothetical fragmentation tree by maximum likelihood from the MS/MS spectrum, as shown in Fp. The nodes refer to the peaks in MS/MS, and edges refer to the fragmentation correlations between peaks (Böcker Dührkop 2016 Dührkop et al. 2019).
3
OUR MODEL
We introduce our proposed model in this section. Our model consists of three modules, the peak embedding module, the fragmentation aggregation module, and the SMILES reconstruction module. We also introduce training technique used in our method. The overall framework is in Figure 3
3

---

Under review as a conference paper at ICLR 2022
a
b
10000
Intensity
17 Da
Da
Am/z 18
5000
36 Da
Da
46 Da
0
70
80
100 m/z
110
120
130
Figure 2: Demonstration of experimental MS/MS. (a) Typical experimental MS/MS of aminocaproic acid, in which the monoisotopic mass is marked with red star and the fragmentation pattern of dehydration is indicated by Am/z of 18 Da. (b) The fragmentation tree of aminocaproic acid calculated from (a).
3.1 PEAK EMBEDDING MODULE
We use an m/z embedding Em/= E R2 and an intensity embedding Ejat E Boxd to encode the input. Em/z is retrieved from the embedding EM E RVxd, V is the number of distinct m/z ratios in dataset, and v number of m/z ratios in one spectrum. Mathematically, the input m/z vector M (binned with size of Da) and the intensity vector I is processed follows:
h W.concat (Pint: Ern/a): Pint Want Em/z (M
where concat means that the inputs are concatenated along the last dimension, Wo E Rdxd and Went R are the parameters be learned. The ion intensity Ij is firstly projected into d- dimension vector, then concatenated with the embedding of its corresponding m/z. In this way, the h.).
input m/z vector M and intensity vector are encoded as h (N), 12
3.2 FRAGMENTATION AGGREGATION MODULE
Given MS/MS spectrum Ti, the corresponding fragmentation tree Tj be obtained from as with the external chemical tools (Dührkop et al. 2019). Note that the number of nodes in Tú is the same as the number of element in x4, i.c., N₁. We regard as an undirected graph, and use to denote the edges in T: For each node in T; denote the neighbors as N()) {k (k, E & The fragmentation aggregation module consists of blocks, where each block consists of local aggregation module (i.e., a graph layer), global aggregation module (i.e., a self-attention layer) and module to merge the two representations.
Local aggregation: We performed the local information aggregation on the fragmentation tree T; with MPNN (Gilmer et al. 2017). Let hy denote the features of node output by the 1-th block. We set {h]} as the embeddings output by the peak embedding module. Mathematically, the 1-th block of local aggregation works as follows:
Mical, W.Ml-1
Σ JEN(()
W,
[iii
where W, and W,, are network parameters. That is, we aggregate local features on the tree using GraphSage (Hamilton et al. 2018), and obtain V
Global aggregation: In parallel, use self-attention layer to globally process all b 1 A self- attention layer is defined as follows:
N aayWoh
g
(3)
Σk=1 exp(h: -W,
1=1
where W, and W,, are parameters to be learned.
4

---

Under review as a conference paper at ICLR 2022
After that, utilize both local and global features of the 1-th block and obtain
A h1-1 (M h hgghtall.):
(4)
where LN denotes layer normalization (Ba et al. 2016).
After that, h is fed into a feed-forward layer and hy is obtained. Mathematically,
h W2ReLu(Wi, + b1) + b2.
(5)
where W1, W2. b, by are be learned.
We can eventually obtain H = (4f /2 .. k from the last block of the encoder.
3.3 SMILES RECONSTRUCTION MODULE
We use the standard Transformer decoder (Vaswani al. 2017> to generate the molecule following the teacher forcing training strategy with the ground truth from a prior time step as input. The decoder consists self-attention layer, encoder-decoder attention layer, and feed-forward layer. The H- will be used in the encoder-decoder attention, and the SMILES will be generated token by token.
intmm inten, inbro]
Positional encoding
Linear
Local aggregation
Concal &
Add
Feed
Add nomi
[ms, mo]
miz embedding
HP
Global
morm
Linear
aggregation
(a)
Peak embedding module
(b)
Fragmentation aggregation module
Positional encoding
SMILES (shifted right)
Character embedding
+
Masked multi-head attention
Add
Multi-head attention
Add
Feed forward
Add
Linear
SMILES
norm
norm
nom
(c) SMILES reconstruction module
N
Figure 3: The architecture of UIS--rrasformmr. In peak embedding module, the m/z's are processed by an embedding layer with positional encoding, and the intensities are first projected into the same dimension as the m/z's embedding, then concatenated with m/z embedding for the construction of peak embedding by Linear layer. In the fragmentation aggregation module, the peak embedding was updated by local aggregation on the fragmentation tree, and global aggregation on the fully connected graph, where the nodes denote the peaks in the spectrum and the edges denote the frag- mentation correlations. In the SMILES reconstruction module, the learned peak embedding was used for SMILES reconstruction by Transformer.
3.4 DROPPING MODULE
To relieve the overfitting effects, inspired by (Larsson et al. 2017 Zhu et al. 2020), we design an aggregation switch to control the choice of local and global fragmentation aggregation. Mathemati- cally, during training, Eqn.(4) is refined as followed:
LN (Nooaact)))
p < Plocali
hEll h/-1 +
Plocal S p < Plocal + Pglobal
(6)
LN(Naaal,ss + sg)
Plocal Pglobal S p S 1;
where p is uniformly sampled from [0, 1], M and giohaat defined in Eqn. (2) and Eqn.
respectively. With probability Plecats we only the local aggregation; with probability Polohals
5

---

Under review as a conference paper at ICLR 2022
Table Summary of benchmark datasets.
Dataset
#Training pairs
#lesting pairs
#Molecules
MassBank
15,784 380
3,945
3,675
CASMI
94
393
use the global aggregation; otherwise, we use both of them. At inference time, both of the two branches are used Eqn.04).
In this work, we focus on spectrum-to-SMIIL. The proposed fragmentation aggregation method is general and can be easily applied to other MS/MS-based applications like metaholite classification.
EXPERIMENTS
4.1 DATASETS AND BASELINE MODELS
We performed the experiments on the benchmark datasets of electrospray ionization (ESI) MS/MS spectra of the positive ionization mode from MassBank (Horai et al. 2010) and the MS/MS spectra of the Critical Assessment of Small Molecule Identification (CASMI) Contest (Nikolic et al. 2017 Schymanski et al.) 2017). We split the dataset randomly by spectrum, because the cosine similarity score (CSC) for intra-moles mole-lel was low, with mean CSC of 0.38.
The MassBank dataset consists of 19,729 spectrum- SMILLS pairs and 3,675 distinct molccules in Figure 4 It covers wide range of chemical compounds, including benzenoids, alkaloids, organic acids, and lipids, etc. The length of SMILES varies from to 100, the number of edges in the frag- mentation trees fluctuates from 59, and the number of peaks in MS/MS covers the range from 2 to 189, The CASMI dataset consists of 474 spectrum-SMILES pairs and 393 distinct molecules in Figure The length of SMILES varies from
93. The number of edges in the fragmentation trees fluctuates from to 58, the number of peaks in MS/MS covers the range from 2 to 185, and the precursor mass of both MassBank and CASMI is below 500 Da.
ecges (MMnssiank)
lengthe
2500
1000
500
2000
19.0
Court
900
$1.0
Count
400
10.0
1000
400
200
500
200
100
10
20
30
50
60
50
100
150
20
40
50
100
Edges of
A Peaks MBMS
Longi's of SMILES
d
The number of ecges in (CASM)
The number of peoks MSMS (CASMI)
The lengths SMILES (CASM)
25
30
36.3
70.0
20
56.0
35
10
0
Peaks MSNS
Figure 4: The data statistics of the benchmark datasets. The distribution for the number of (a) edges in T and (b) peaks in MS/MS, and (c) the lengths of SMILES in MassBank dataset. The distribution for the number of (d) edges in T and (e) peaks in MS/MS, and (f) the lengths of SMILES in CASMI dataset. The red line indicated the 95% percentile.

---

Under review as a conference paper at ICLR 2022
The reconstruction quality of molecule identification was tested based validity of valency straints (denoted as V), reconstruction accuracy with valid valency (denoted as R). top-1 and -5 reconstruction accuracy (denoted Topl and Top5), Formula score (denoted Fl score), mean rank of the correct prediction (denoted as mean rank), mean relative ranking position (denoted as mean RRP). F1 score is similar to the scheme in F1 racing based on the rank of the correct prediction. R. V, and top 1/tp/ accuracy are widely used in molecule reconstruction problem (Shi et al.) 2020) Jin et 2019), while Fl score, mean rank, and mean RRP are widely used in the performance eval- uation of CASMI (http:/ www.cmassiiii..c..cc.........m coniiiicoottsttt.. org 2017 001007770 index. shtml). ). FI score, mean rank, and mean RRP are calculated with the top- 10 candidates. Above these measurements, state- of-art models perform better with larger R. V, topl/top5, Fl score, and mean RRP, and smaller mean rank. The top-k candidates were generated by beam-search.
We chose the following models as the baselines: MetFrag (Ruttkies et al. 2016) represented the typical conventional database-searching model for MS/MS-assisted molecule identificaiton. Mettra first generated in silico fragmentation spectra of molecules from different databases, then matched against m/z ratios. MetFrag was downloaded from https:/Apb-baalee-eee www.e..iihuu..... io/ MetFrag pro ject met wwliaaggcc///vi//ia/ear/// 2.4.5). The searching databases for MetFrag in- cluded PubChem and KEGG and the identification was performed with a mass acuuracy of 0.1 Da. RNN first encoded the spectrum into hidden representations, then generated the SMILES in seq2seq architecture with Luong attention mechanism and LSTT operators. We implemented the chatbot]
model using the code shared at Intpp:: /github...... com/ Transformer represented the classic NLP model, and it first encoded the hidden representa- tions with self-attention mechanism then generated the SMILES with self- and cross-attention mechanisms. We implemented the model using the code shared at https: http::///gttube com/ [adoree0122//tttenttoo----------
4.2 TRAINING DETAILS
As view MS/MS-assistted molecule identification the translation from tandem spectrum to SMILES string, which describes the structure of chemical molecules using short ASCII strings, the loss function defined by multi-class cross entropy loss as follows:
L Thot log Pk:
where K represents the number of label classes, Ps represents the predicted probabilities of ASCII characters, and Yhot represents the original one-hot encoding on ASCII characters.
In order improve generalization and model calibration, we estimate the network's parameters with label smoothing by minimizing the following loss function:
K
V/& = (1 -) )) * Yhot + eu(k); =
Σ k=1
A log lkg pk.
(8)
where u(k) represents the prior distribution of ASCII characters.
During training, we used the Adam optimizer (Kingma & Ba 2017) with varied learning rate [Vaswani et al. 2017) as follows:
Ir --= 0..55 min(step.num" 0.5 step_num warmup_steps -1.5).
(9)
where the learning rate was firstly increased during the warm steps and then decreased. We trained our models on one machine with four NVIDIA GeForee RTX 2070 Super GPUs with batch size of 64 and a training epoch of 500.
4.3 EVALUATION RESULTS ON MOLECULE IDENTIFICATION
MassBank: Table 2 shows the performance of the models in the MassBank dataset. Compared to the conventional database-seardee metrics.
model, all deep learning models have superior performances all - Transformer has achieved competitive performances on the seven metrics. As a model
introducing the prior chemical knowledge, 2S2 Traansasffrrerr has improved the top-1 accuracy and

---

Under review as a conference paper at ICLR 2022
Intensity
Intensity
150
200
250
300
350
400
450
100
150
200
250
300
350
m22
miz
-5.54E-04
-2.70E-01
-1.30E+00
-2.09E005
-5.12E-01
-8.04E-01
-1.41+000
-1,42+000
-1,45E+00
-1.04E+00
-1.12E400
-1,20E+00
-1.46E+00
-1.54E+00
-1.62E+00
-1.39E+00
-1,45E+00
-1.49E+00
Figure 5: The demonstration of generated SMILES (a) MassBank and (b) CASMI dataset. The above spectrum denotes the MS/MS spectrum the input of USe-Trrasformer, the below molec- ular graphs denote the top-9 candidates generated with beam search, the number below molecular graph indicates the searching score, and the circled molecule denotes the accurate prediction of true molccule.
Table 2: Comparison of different models' performance on MassBank dataset.
Model
V%
R%
Top1%
Top5%
Fl score
Mean rank
Mean RRP
MedFrag-PubChem
0.10
0.48
328
9.96
0.04
MeiFrag-KEGG
1.14
1.55
1,405
9.85
0.02
RNN
77.4
59.0
45.7
47.0
45,953
5.32
0.47
Transformer
97.7
58.0
56.7
70.1
64,980
3.11
0.69
- Transformer
97.1
63.2
61.4
73.7
68,736
2.77
0.72
reconstruction accuracy by 5%, compared with Transformer. Also, -ra-Transformmr improved the performances evaluated by F1 score, mean rank, and mean RRP.
CASMI: Table 3 shows the performance of the models in the CASMI dataset. Compared to the conventional database-scarching model, deep learning models have superior performances on all metrics. Due to insufficient data pairs in the CASMI dataset, the top-5 identification accuracy only achieves 52.1% 1%. MS²-Transfore Transforrrr still shows superior performances to Transformer, with an improvement of 8%/163 in reconstruction accuracy/F1 score, respectively. Importantly, there were 73 distinct molecules which hadn't 't been seen in the training dataset, and the top-1/5 accuracy achieved 40.79% 55.6%% for these independent molecules, respectively.
By the way, 2 Transformer has obtained the above performances in both MassBank and CASMI only with additional 0.026 M parameters in light-cost way, compared the conventional trans- former.
5 CONCLUSIONS AND FUTURE WORK
In this paper, we propose the 2 Transformer for reconstructing the right chemical structures from MS/MS spectrum based the aggregation of prior chemical knowledge. Our method identifies the
8

---

Under review as a conference paper at ICLR 2022
Table 3: Comparison of different models' performance CASMI dataset.
Model
V%
R%
Top1%
Top5%
score
Mean rank
Mean RRP
MetFrag-PubChem
0.0
3.2
61
9.65
0.04
MedGrag-KEGG
3.19
5.32
111
9,49
0.05
RNN
78.7
40.5
31.9
31.9
750
6.81
0.32
Transformer
95.7
34.4
33.0
46.8
971
5.57
0.44
MS3-Transformer
95.7
42.2
40.4
52.1
1,134
4.91
0.51
right chemical structures of molecules and exploits the fragmentation inferences for potential frag- ment candidates. Experimental results showed that USs-TTrasformer outperformed Transformer in the benchmark datasets of molecule identification with an improvement 5%-7%% in top-1 accuracy.
For further improvement, plan extend the MS ASs-Transformrr problems involving multiple ionization modes (i.e., both positive and negative ionization modes) to investigate the fragmentation aggregation mechanisms in different experimental settings. We also plan extend our model to encoding the MS/MS spectrum of mixtures, providing help in the clinical research of molecule identificatinn from complex bio-fluids, not only from the pure substances.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
Robert S. Banh, Esther S. Kim, Quentin Spillier, Douglas E. Biancur, Keisuke Yamamoto, Albert S. W. Sohn, Guangbin Shi, Drew R. Jones, Alec C. Kimmelman, and Michael E. Pacold. The polar oxy-metabllome reveals the h-hdrooyymandelate coq10 synthesis pathway. Nature, 597: 420 420-425, 2021.
Sebastian Böcker and Kai Dührkop. Fragmentation trees reloaded. Journal of Cheminformatics, 8, 2016.
Kai Dühhkop, Marcus Ludwig Markus Fleischauer, Alexander A. Aksenov, Alexey V. Melnik, Mar- vin Meusel, Pieter C. Dorrestein, Juho Rousu, and Sebastian Böcker. SIRIUS 4: rapid tool for turning tandem mass spectra into metabolite structure information. Nature Methods, 16(4):
299-302, 2019.
Kai Dührkop, Louis-Félix Nothias, Markus Fleischauer, Raphael Reher, Mareus Ludwig, Martin A. Hoffmann, Daniel Petras, William H. Gerwick, Juho Rousu, Pieter C. Dorrestein, and Sebastian Böcker. Systematic classification of unknown metabolites using high-resolution fragmentation mass spectra. Nature Biotechnology, 39, 2021.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry, 2017.
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs, 2018
Hisayuki Horai, Masanori Arita, Shigehiko Kanaya, Yoshito Nihei, Tasuku Ikeda, Kazuhiro Suwa, Yuya Ojima, Kenichi Tanaka, Satoshi Tanaka, Ken Aoshima, Yoshiya Oda, Yuji Kakazu, Miyako Kusano, Takayuki Tohge, Fumio Matsuda, Yuji Sawada. Masami Yokota Hirai, Hiroki Nakanishi, Kazutaka Ikeda, Naoshige Akimoto, Takashi Maoka, Hiroki Takahashi, Takeshi Ara. Nozomu Sakurai, Hideyuki Suzuki, Daisuke Shibata, Steffen Neumann, Takashi lida, Ken Tanaka, Kimito Funatsu, Fumito Matsuura, Tomoyoshi Soga, Ryo Taguchi, Kazuki Saito, and Takaaki Nishioka. Massbank: a public repository for sharing mass spectral data for life sciences. Journal of Mass Spectrometry, 45(7):703-714, 2010.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation, 2019.

---

Under review as a conference paper at ICLR 2022
Tobias Kind and Oliver Fiehn. Metabolomic database annotations via query of elemental composi- tions: Mass accuracy is insufficient even at less than ppm. Bioinformatics, 7, 2006.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Martina Klünemann, Sergej Andrejev, Sonja Blasche. Andre Mateus, Prasad Phapalc, Saravanan Devendran, Johanna Vappiani, Bernd Simon, Timothy A. Scott, Eleni Kafkia, Dimitrios Kon- stantinidis, Katharina Zirngibl, Eleonora Mastrorilli, Manuel Banzhaf, Marie-Theress Mackmull, Felix Hövelmann, Leo Nesme, Ana Rita Brochado, Lisa Maier, Thomas Bock, Vinita Periwal, Manjeet Kumar, Yongkyu Kim, Melanie Tramontano, Carsten Schultz, Martin Beck, Janosch Hennig, Michael Zimmermann, Daniel C. Sévin, Filipe Cabreiro, Mikhail M. Savitski, Peer Bork, Athanasios Typas, and Kiran R. Patil. Bioaccumulation of therapeutic drugs by human gut bacte- ria. Nature, 597:533-538, 2021.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural net- works without residuals. In ICLR, 2017.
Dejan Nikolic, Martin Jones, Lloyd Sumner, and Warwick Dunn. Casmi 2014: Challenges, solutions and results. Current Metabolomics, 5:5 17, 2017.
2:171,,0017.
Christoph Ruttkies, Emma L. Schymanski, Sebastian Wolf, Juliane Hollender, and Steffen Neu- mann. Metfrag relaunched: incorporating strategies beyond in silico fragmentation. Journal of Cheminformatics, 8, 2016.
Emma I.. Schymanski, Christoph Ruttkies, Martin Krauss, Céline Brouard, Tobias Kind, Kai Dührkop, Felicity Allen, Arpana Vaniya, Dries Verdegem. Sebastian Böcker, Juho Rousu, Huibin Shen, Hiroshi Tsugawa, Tanvir Sajed, Oliver Fichn, Bart Ghesquière, and Steffen Neumann. Crit- ical assessment of small molecule identification 2016: automated methods. Journal of Chemin- formatics, 9, 2017.
Xiaotao Shen, Ruohong Wang, Xin Xiong, Yandong Yin, Yuping Cai, Zaijun Ma, Nan Liu, and Zheng-Jiang Zhu. Metabolic reaction network-based recursive metabolite annotation for untar- geted metabolomics. Nature Communications, 10, 2019.
Chence Shi, Minkai Xu, Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graphaf: a flow-based autoregressive model for molecular graph generation, 2020.
Hiroshi Tsugawa, Ryo Nakabayashi, Tetsuya Mori, Yutaka Yamada, Mikiko Takahashi, Amit Rai, Ryosuke Sugiyama, Hiroyuki Yamamoto, Taiki Nakaya, Mami Yamazaki, Rik Kooke, Johanna A. Maoonaa,, Nihal Oztolan-Frol. Joost J. B. Keurentjes, Masanori Arita, and Kazuki Saito. A cheminformatics approach to characterize metabolomes in stable-isotope-labelee organisms. Nature Methods, 16:295-298, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
Maria Vinaixa, Emma Schymanski. Steffen Neumann, Miriam Navarro, Reza M. Salek, and Oscar Yanes. Mass spectral databases for lc/ms- and ge/ms- metabolomics: State of the field and future prospects. TrAC Trends in Analytical Chemistry. 78:23-35, 2016.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tieyan Liu. Incorporating bert into neural machine translation. In International Conference on Learning Representations, 2020. URL https: http:/:::////eeeeveeeee net wwwp:nnetettenttnieeeeeeeee http://wwww.ttoroooooooooooooooo
10


Reviews:
[{"review_id": "pDTViK_xEE2", "reviewer": "Reviewer_g6pz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses an important problem using a convincingly good model.  The design choices overall seem sensible, and the results appear to be strong.\nHowever, I have two main critiques, and a number of minor comments as well.\n\nFirst, this is a well-studied problem, but the the paper fails completely to situate the performance relative to the state of the art.  For example, we are told (p. 6) which performance measures were used to evaluate the system, but not whether these are the same performance measures employed in other studies (e.g., CAMSI).  It's important to use commonly accepted performance measures or to justify using other measures.  More importantly, the method is compared to the author's own implementations of several competing machine learning models. This is mildly interesting, but not nearly as interesting as would be a comparison to existing, state-of-the-art methods on these same benchmark datasets.  Without such comparisons, it's impossible to evaluate the practical impact of this work.\n\nSecond, the description of the peak embedding module (Section 3.1) is not clear at all.  I get that E_{m/z} is a matrix of dimension V \\times d, but then I don't understand what E_i is. I think maybe it's a vector of length d, but the sentence is written as if there is a matrix of size E_i \\times d. The next sentence says that V is the number of distinct m/z ratios.  I think this means that V is the number of peaks, but then it's also the number of intensities.  I don't understand why V only applies to the m/z ratios (since every m/z ratio comes paired with an intensity). The real problem is that we are told that m_i is encoded using a one-hot representation.  This is completely mysterious to me, and is related to the incorrect claim (p. 3) that m/z is discrete. The m/z ratio is a real value, measured with varying amounts of precision depending on the instrument type.  I do not understand in what sense this measurement can be considered discrete, nor how to encode it as a one-hot representation.  I am guessing that it's a vector where each entry corresponds to a mass bin, but then why aren't we told what bin size is used?  This seems like a critical parameter, and for realistic values it means that the vector will be really long (e.g., hundreds of thousands).  Given all the problems in the first 1.5 sentences of this subsection, I really can't make sense of Equation (1) nor the subsequent description of the encoder.\n\nThe claim (p. 1) that \"about 90% [of] spectrum molecules is not covered by the standard database\" is problematic.  No citation is given to support this claim. Plus, it surely cannot be true in general, since the coverage will depend strgonly on the type of sample being analyzed and the choice of database.\n\nThe fragmentation tree shown in Figure 1B is not described well enough.  More detail needs to be provided somewhere in the paper about how the initial graph is constructed and exactly what the edges mean,\n\nAt first I thought that Table 1 and Figure 4 seemed redundant and that the table should be eliminated. But then I noticed that the mean values marked in Figure 4 do not agree with the mean values reported in Table 1. So I think I must be misunderstanding something here, which is worrisome.\n\nWe are told on p. 8 that the model achieves its performance \"in a light-cost [sic] way\" using only \"an additional 0.26 M parameters.\" What is this relative to?\n\nThe fragmentation inference task in Section 4.4 is not well described. I do not really understand what exactly the problem is here. More importantly, no baseline method is included for this task, so we do not know whether the proposed model improves on the state of the art.\n\nThe use of English is problematic throughout.  Careful editing is needed to fix grammatical errors.  Here are a few:\n\nAbstract: \"demonstrating it the great\"\n\np. 1: \"progresses have been achieved\"\n\np. 5: \"uniformed sampled\"\n\nIn Figure 1A, the fragmentation is not always accomplished via a laser.\n\n", "summary_of_the_paper": "This paper proposed to a use a deep learning model with a transformer architectur to translate between mass spectra and chemical structures of unknown metabolites.", "summary_of_the_review": "I think this might be very good work, but the explanation of the model is problematic, and the empirical results fail to explain how the model's performance compares to the state of the art.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kGPznmEn0T", "reviewer": "Reviewer_ZGQp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is tackle a very interesting biochemistry problem --  tandem mass spectrometry (MS/MS).  The paper provides clear description for the audience to understand the data and the problem (e.g. Fig 4 and Fig5). The three modules are interesting (e.g. encode the peak embedding and incorporate fragmentation tree structure). \n\nSome questions: \n(1) Although the three modules framework is interestingly designed, it seems to me that the contribution of machine learning novelty seems not very clear. \n(2) Table 1 shows the MS/MS number of pairs seem limited, especially CASMI. Are transformer-based or deep learning models that require a lot of training data the most appropriate methods, compared with non-deep learning based methods?  \n(3) MS/MS is a well established technique for many years with mature data analysis pipeline (from spectrum to molecule identification), can the author provide evaluation against currently working data analysis pipeline to Table 2 and Table 3 so that the readers can understand the performance of deep learning model against conventional working model?\n(4) Fig 5 a,  why the true predictions in the circles have different scores? \n(5) Fig 3 SMILES-reconstruction module, why the input is smiles (shifted right). Is the smiles string the target, not the input? \n", "summary_of_the_paper": "\nThis paper provides an end-to-end model, denoted as MS2 -Transformer, using MS/MS spectrum data to specify molecule identity. The application is very important to biochemical studies.  This model consists of three modules, the peak embedding module, the fragmentation aggregation module, and the SMILES reconstruction module. The MS2 - Transformer incorporates chemical knowledge through a fragmentation tree from the MS/MS spectrum. \n \n", "summary_of_the_review": "This paper proposes an interesting method to solve an important biochemical problem, data analysis of tandem mass spectrometry (MS/MS). The paper may has a larger scientific impact if it submit to a corresponding high impact journals, which are likely have more appropriate audience. However, if the paper is interested in machine learning conference, some more contribution of machine learning novelty and more quantification of the methods can strengthen the paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "NHv1nTVpyYt", "reviewer": "Reviewer_PCDA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting paper that exploits sequence to sequence machine learning models to solve mass spectrometry problems. \nThe strength of this paper is that they incorporated extra chemical knowledge of the spectrum via its fragmentation tree to enhance the input feature. \nThe weakness of this paper mainly comes from its experimental settings: \n(1) The paper doesn't explain how many different molecules/compounds are in the MassBank dataset and the CASMI dataset. \n(2) How are your training, validation, and testing sets split? The MassBank dataset contains spectrum records of the same molecule but at different experimental conditions (e.g. different collision energy). On average there are around 10 records per compound. Did you split your training, validation, and testing sets randomly or by compounds? This makes a huge difference in terms of performance. The former would be an easy task due to the high overlapping of known compounds in the training set, which makes it more like an overfitting problem. The latter should be the right way to test the model's performance.\n(3) How is your reconstruction loss defined? \n(4) What is your label in equation (7)? \n(5) The same molecular structure could have many different SMILES representations. How does your model address this issue (non-deterministic mapping)?\n(6) They author doesn't provide a code to reproduce their experiments.", "summary_of_the_paper": "This paper proposed a transformer model for molecule structure identification using the experimental mass spectra. \nThis model takes the experimental mass spectrum and the corresponding fragmentation tree structure as the input for the transformer to predict the SMILES of the molecule in an end-to-end manner. \nIn their experiments, they show their model improves a vanilla transformer by 5% top-1 accuracy.", "summary_of_the_review": "This paper proposed a customized transformer to solve the molecular structure identification problem via MS/MS spectrum. The application itself is interesting but the authors need to justify their performance with a clear experimental setting and a better illustration of the evaluation process. \nWith the current paper, I this it is marginally below the acceptance threshold. If you address my questions properly, I may adjust my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Finally, all reviewers leaned towards rejection. The main concerns were missing methodological depth and questions regarding the experimental evaluation (unclear link between experimental outcomes and methodological details). The rebuttal was not perceived as being fully convincing, and finally nobody wanted to champion this paper. I think that this work has some potential, but in its present form, it does not seem to be ready for publication.
Decision: Reject