SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Annotating spectra. We emphasize that SCARF can be trained with any product formula annotations, which can be labeled [35] or inferred with varied computational strategies [13]. Herein, we utilize the MAGMa algorithm [40]. In brief, for a given molecule-spectrum pair in the training dataset, the molecule is combinatorially fragmented at each atom up to a depth of 3 breakages to create subfragments. This creates a bank of possible molecular formulae, and each peak in the spectrum is assigned to its nearest possible formula within a mass difference of 20 parts-per-million.
4.2 Spectra prediction
Predicting product formulae (SCARF-Thread). SCARF-Thread is trained and used to reconstruct prefix trees and evaluated by its ability to recover the ground truth product formula set. The set of generated product formulae is rank-ordered by the probability of each product formula and filtered to the top  $k$  predicted product formulae. The fraction of ground truth formulae (22.29 peaks on average in NIST20) contained in the top  $k$  set is computed as coverage.
We compare coverage achieved by SCARF-Thread to several baselines: (i) CFM-ID [2], a fragmentation based approach (\$A.3.1); (ii) a random baseline that samples product formulae from a uniform distribution; (iii) a frequency baseline, which ranks product formulae by the frequency the product formula candidate (or product formula difference) appears in the training set; (iv) an LSTM autoregressive neural network baseline (\$A.3.2) that is trained to predict molecular formula vectors in sequence from highest to lowest intensity; and two model ablations, (v) SCARF-Thread-D and (vi) SCARF-Thread-F, which only make uni-directional elements difference or forward predictions of element counts respectively (i.e.,  $\alpha$  in Equation 3 is fixed to 0 for (v) and 1 for (vi)).
In general, SCARF-Thread starkly outperforms all baselines tested (Table I). By generating 300 peaks, SCARF-Thread is able to cover on average  $91\%$  and  $72\%$  of the true formulae in the ground truth test set for NIST20 and NPLIB1 respectively. Our difference- and forward-only directional prediction ablations demonstrate the benefits of modeling both the atom counts for each element and the differences in counts from the original molecule.
Predicting mass spectra. We next evaluate the strength of SCARF-Weave for intensity prediction on the same test dataset. We compare against five baselines: a fragmentation-based approach, CFM-ID
Table 2: Spectra prediction in terms of cosine similarity, coverage (proportion of ground-truth peaks that are covered by the top 100 non-zero predictions), validity (the fraction of predicted peaks for which a chemically plausible explanation is possible), and time. Best value in each column is typeset in bold (higher is better for all metrics but time). Results are averaged across 3 random seeds on a single data split for all retrainable models (i.e., not CFM-ID).
Dataset	NIST20	NPLIB1	Time (s)
Metric	Cosine sim.	Coverage	Valid	Cosine sim.	Coverage	Valid
CFM-ID	0.412	0.278	1.000	0.377	0.235	1.000	1114.7
3DMolMS	0.510	0.734	0.945	0.394	0.507	0.919	3.5
FixedVocab	0.704	0.788	0.997	0.568	0.563	0.998	5.5
NEIMS (FFN)	0.617	0.746	0.948	0.491	0.524	0.949	3.9
NEIMS (GNN)	0.694	0.780	0.947	0.521	0.547	0.943	4.9
SCARF	0.726	0.807	1.000	0.536	0.552	1.000	21.1
[2]; two NEIMS binned prediction models ([50]; §A.3.3), using either feed forward network modules (FFNs), as in the original work, or graph neural network modules (GNNs) as in SCARF-Weave and described by Zhu et al. [58]; a retrained variant of 3DMolMS ([23]; §A.3.4), a binned spectrum predictor that utilizes a point cloud neural network over a single molecular conformer input generated by RDKit [39]; and FixedVocab, a formula prediction model that predicts intensities at a fixed library of formulae and formulae differences inspired by GRAFF-MS ([34]; §A.3.5).
To enable fair comparison across models, we predict test spectra at 15k bins (0.1 bin resolution between 0 and 1500) with a maximum of 100 peaks for each predicted molecule. With the exception of CFM-ID, all models are hyperparameter optimized ( $\S \text{A.5.6}$ ), retrained completely, and conditioned on the same covariate inputs as SCARF; such steps lead to large performance boosts to the prior NEIMS method in particular. We evaluate the quality of our predictions based upon four core criteria reflecting our original desiderata of accuracy, physical-sensibleness, and speed:
1. Cosine sim.: Cosine similarity between the ground truth and predicted spectra, indicating spectrum prediction accuracy.
2. Coverage: The fraction of ground truth spectrum peaks covered by the predicted spectrum.
3. Valid: The fraction of predicted peaks that can be explained by a subformula (that obeys basic ring-double bond equivalent heuristics [37]) of the predicted molecule.
4. Time (s): The wall time it takes (using a single CPU and no batched calculations) to load the model and predict spectra for 100 randomly selected molecules.
SCARF is more accurate than all other approaches on NIST20, improving cosine similarity over a GNN binned prediction approach by over 0.02 points in NIST20 and 0.01 in NPLIB1 (Table ②). Further, our method is more physically grounded insofar as all predicted peaks are guaranteed to be valid subformulae, unlike the unconstrained binned approaches, where nearly  $5\%$  of peak predictions cannot be explained by a valid molecular formula. Importantly, SCARF still operates 2 orders of magnitude faster than CFM-ID (Table ②).
The heterogeneity, reduced dataset size, and increased average molecular weight (Figure A3) of NPLIB1 leads to substantially worse absolute performance across all models. Interestingly, in this setting, the FixedVocab approach [34] performs better, perhaps because the strict priors of formula constraints are more helpful with fewer and more challenging training examples. We further stratify results by molecule size in Figure A2 showing that all models are generally more accurate on smaller compounds. We additionally validate that cosine similarity is not merely measuring a model's ability to predict the parent mass peak by computing a modified cosine similarity with the original molecule's mass masked (Table A7).
A key application for forward spectrum prediction is to use predicted spectra to determine the most plausible molecular structure assignment. We posit forward spectrum prediction models should be
Figure 5: SCARF enables more accurate retrieval of ground truth molecules within the NIST20 dataset. A. Average retrieval accuracy of SCARF at various top k thresholds. Retrieval is conducted on the same test split, and retrieval accuracy is averaged across models trained for three separate random seeds. B-C. Example spectrum predictions made by SCARF (top) compared to the ground truth spectrum (bottom). Up to 5 predicted peaks with the highest intensity are annotated with their molecular formula explanation as predicted by SCARF. The full molecule is shown inset. Further examples are in the Appendix (Figure A1).
particularly helpful in differentiating structurally similar molecules and design a retrieval task to showcase such potential. For each test set molecule, we extract 49 potential "decoy" options based upon the most structurally similar isomers (i.e., compounds with the same precursor formula) within PubChem [27] as judged by Tanimoto similarity using Morgan fingerprints. While retrieval could be conducted on the entirety of PubChem or other similarly large molecular databases, we believe this subset retrieval setting is more practical and better mirrors a real-world setting (see §A.4 for justification). We predict the spectra for all molecules and rank them according to their similarity to the ground truth spectrum, computing the accuracy for retrieval. Herein, we specifically emphasize models and retrieval on the NIST20 dataset, as it is a much larger and higher quality dataset.
SCARF reaches a top 1 and top 5 retrieval accuracy in this task of  $18.7\%$  and  $54.1\%$  respectively, representing an improvement over the methods with the second best top 1 accuracy of  $17.5\%$  (NEIMS (GNN)) and top 5 accuracy of  $52.2\%$  (Fixed Vocabulary) (Figure 5A). We highlight two example predictions from SCARF (Figure 5B-C), with additional randomly sampled test set predictions shown in Figure A.1. We repeat similar experiments within NPLIB1, but find that cosine similarity performance is uncorrelated with relative ranking performance; feed forward fingerprint based approaches are better at retrieval, despite relatively weak cosine similarity ( $\S \text{A.1}$ ). FixedVocab [34] performs especially well on NPLIB1, again likely due to the helpful biases imparted by constraining the formula vocabulary.
This result underscores previous observations regarding how database and model biases can skew retrieval results under certain settings [22]. That is, models may be more or less robust for certain classes of molecules, so the composition of these classes in the retrieval library may affect the retrieval accuracy accordingly. The observed discrepancy between cosine similarity and retrieval performance can further be explained by the dataset shift required for computing retrieval accuracy; cosine similarity is evaluated on "in-domain" data, whereas retrieval relies also on accuracy on unlabeled data that may be "out-of-domain."
Forward vs backward models. Computational tools to identify mass spectra are often divided into two categories: forward and backward models. Forward models, i.e., spectrum predictors, such as SCARF or the methods discussed in Section 2, operate in the causal direction and try to predict the spectrum given the molecule. Backward models start from the spectrum and predict features or even full molecule structures. Early backward models used heuristics, expert rules, and even neural networks [8, 10, 42]. Such approaches have more recently been augmented with kernel methods and more modern, deep representation learning techniques [12, 16, 19, 47]. These models are complementary to spectrum predictors.
Mass spectra for proteomics. Although this paper has focused on small molecules, similar trends of deep representation learning for mass spectra are also emerging in the adjacent field of proteomics [54], with Shouman et al. [41] recently proposing a benchmark challenge in this domain. While small molecule and protein spectra are similar, proteomics spectra tend to be more easily predicted as fragments are often formed at peptide bonds. We believe adapting SCARF to this task would be an interesting direction for future work.
Neural set generation. Our work is also related to methods for modeling sets and multisets. SCARF-Thread generates a set as output, which has been studied elsewhere in the context of n-gram generation [45], object detection [32, 57], and point cloud generation [28]. The product formulae sets we generate, however, are different to those considered in these other task; in our setting, each member of the set (i.e., individual product formula) represents a multiset of atom types (i.e., multiple carbons, multiple hydrogens, etc.) and is constrained physically by the precursor formula.
In this paper we introduced SCARF, an approach utilizing prefix tree data structures to efficiently decode mass spectra from molecules. By first predicting product formulae and then assigning these formulae intensities, we are able to combine the advantages of previous neural and fragment based approaches, providing fast and physically grounded predictions. We show how these resulting predictions are both more accurate in predicting experimentally-observed spectra and yield improvements in identifying a molecule's structure from its respective mass spectrum, as tested on a widely used dataset.
In term of limitations, our model is data dependent, as indicated by the relative performance across the NIST20 and NPLIB1 datasets. SCARF is also highly reliant upon the quality of product formula label assignment. The current commercial status of mass spectrometry training data poses a barrier to entry, and identifying high quality public domain data is critical for future studies. A key contribution of this work is to retrain and optimize the hyperparameters of competing methods on a publicly available dataset under equivalent conditions to allow for future extensions. Directly training on a ranking-based loss or learning a model specific spectra distance function may be one way to improve upon our model's performance in the retrieval setting, and we outline additional potential ideas more explicitly in §A.6
Future directions will involve further developing SCARF for real world use cases such as unknown metabolite elucidation in clinical samples. Specific directions will include more carefully modeling covariates (e.g., collision energies and MS/MS instrument types), grounding product formulae in molecular graph substructures, and utilizing such models to augment inverse spectrum-to-molecule annotation tools.
We thank members of the Coley Research Group, as well as Michael Murphy, for helpful discussions and comments. S.G. thanks the MIT-Takeda Program for financial support. J.B., J.X., and C.W.C. thank the Machine Learning for Pharmaceutical Discovery and Synthesis consortium for additional support.
[1] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2623-2631, 2019.
[2] Felicity Allen, Russ Greiner, and David Wishart. Competitive fragmentation modeling of ESI-MS/MS spectra for putative metabolite identification. Metabolomics, 11(1):98-110, 2015.
[3] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner,
[51] David S. Wishart. Metabolomics for investigating physiological and pathophysiological processes. Physiological Reviews, 99(4):1819-1875, 2019. doi: 10.1152/physrev.00035.2018.
[52] Sebastian Wolf, Stephan Schmidt, Matthias Müller-Hannemann, and Steffen Neumann. In silico fragmentation for computer assisted identification of metabolite mass spectra. BMC Bioinformatics, 11(1):148, 2010. doi: 10.1186/1471-2105-11-148.
[53] Shipei Xing, Sam Shen, Banghua Xu, and Tao Huan. Molecular formula discovery via bottom-up MS/MS interrogation. bioRxiv, 2022. doi: 10.1101/2022.08.03.502704. URL https://www.biorxiv.org/content/early/2022/08/05/2022.08.03.502704
[54] Melih Yilmaz, William Fondrie, Wout Bittremieux, Sewoong Oh, and William S. Noble. De novo mass spectrometry peptide sequencing with a transformer model. In Proceedings of the 39th International Conference on Machine Learning, pages 25514-25522, 2022.
[55] Adamo Young, Bo Wang, and Hannes Rost. MassFormer: Tandem Mass Spectrum Prediction with Graph Transformers. arXiv preprint arXiv:2111.04824, 2021.
[56] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets. In Advances in Neural Information Processing Systems 30, pages 3391-3401, 2017.
[57] Yan Zhang, Jonathon Hare, and Adam Prugel-Bennett. Deep set prediction networks. In Advances in Neural Information Processing Systems 32, 2019.
[58] Hao Zhu, Liping Liu, and Soha Hassoun. Using Graph Neural Networks for Mass Spectrometry Prediction. In Machine Learning for Molecules Workshop at NeurIPS 2020, 2020.
[59] Richard Licheng Zhu and Eric Jonas. Rapid approximate subset-based spectra prediction for electron ionization-mass spectrometry. Analytical Chemistry, 2023.
A.1 Extended results
We benchmark models in terms of retrieval accuracy as described (§ 4.3) for both the NIST20 and NPLIB1 datasets (Table A1, A2). We recreate retrieval experiments using the full PubChem retrieval library in Table A3. We reproduce main text results with standard error values included in Tables A4, A5, and A6. We showcase additional spectra predictions from our model trained on NIST20 in Figure A1.
Table A1: NIST20 spectra prediction retrieval top k accuracy for different values of k. All values represent the mean across three separate random seeds  $\pm$  the standard error of the mean for a single test set.
top k	1	2	3	4	5	8	10
Random	0.026 ± 0.000	0.052 ± 0.001	0.076 ± 0.001	0.098 ± 0.001	0.120 ± 0.000	0.189 ± 0.001	0.233 ± 0.002
3DMolMS	0.055 ± 0.002	0.105 ± 0.000	0.146 ± 0.002	0.185 ± 0.004	0.225 ± 0.004	0.332 ± 0.003	0.394 ± 0.004
FixedVocab	0.172 ± 0.002	0.304 ± 0.002	0.399 ± 0.001	0.466 ± 0.004	0.522 ± 0.006	0.638 ± 0.005	0.688 ± 0.003
NEIMS (FFN)	0.105 ± 0.002	0.243 ± 0.006	0.324 ± 0.006	0.387 ± 0.006	0.440 ± 0.007	0.549 ± 0.005	0.607 ± 0.002
NEIMS (GNN)	0.175 ± 0.003	0.305 ± 0.001	0.398 ± 0.001	0.462 ± 0.002	0.515 ± 0.003	0.632 ± 0.003	0.687 ± 0.003
SCARF	0.187 ± 0.004	0.321 ± 0.006	0.417 ± 0.004	0.486 ± 0.004	0.541 ± 0.005	0.652 ± 0.004	0.708 ± 0.005
Table A2: NPLIB1 spectra prediction retrieval top k accuracy for different values of k. All values represent the mean across three separate random seeds  $\pm$  the standard error of the mean for a single test set.
top k	1	2	3	4	5	8	10
Random	0.033 ± 0.001	0.061 ± 0.005	0.092 ± 0.003	0.118 ± 0.003	0.141 ± 0.006	0.216 ± 0.006	0.258 ± 0.006
3DMolMS	0.087 ± 0.001	0.159 ± 0.010	0.218 ± 0.004	0.268 ± 0.006	0.317 ± 0.006	0.427 ± 0.008	0.488 ± 0.005
FixedVocab	0.193 ± 0.003	0.314 ± 0.004	0.390 ± 0.003	0.448 ± 0.005	0.492 ± 0.001	0.587 ± 0.005	0.635 ± 0.006
NEIMS (FFN)	0.195 ± 0.003	0.313 ± 0.002	0.388 ± 0.003	0.447 ± 0.006	0.488 ± 0.002	0.585 ± 0.007	0.624 ± 0.010
NEIMS (GNN)	0.174 ± 0.007	0.285 ± 0.004	0.362 ± 0.002	0.422 ± 0.001	0.471 ± 0.002	0.586 ± 0.007	0.640 ± 0.005
SCARF	0.135 ± 0.007	0.242 ± 0.001	0.320 ± 0.001	0.389 ± 0.004	0.444 ± 0.002	0.569 ± 0.001	0.630 ± 0.008
Table A3: Retrieval accuracy on NIST20 for a single 500 molecule subset of the test set using a library of 49 decoys or all decoys contained in PubChem ("None"). Results were repeated for 3 random training seeds of the model and are shown  $\pm$  the standard error of the mean. The top value is shown in bold.
PubChem limit Top k	50 1	2	3	None 1	2	3
FixedVocab	0.168 ± 0.003	0.308 ± 0.003	0.410 ± 0.005	0.145 ± 0.004	0.258 ± 0.004	0.323 ± 0.001
NEIMS (FFN)	0.102 ± 0.002	0.237 ± 0.003	0.315 ± 0.004	0.087 ± 0.003	0.183 ± 0.008	0.236 ± 0.007
NEIMS (GNN)	0.169 ± 0.003	0.300 ± 0.004	0.402 ± 0.005	0.138 ± 0.004	0.239 ± 0.005	0.312 ± 0.008
SCARF	0.204 ± 0.009	0.326 ± 0.005	0.432 ± 0.009	0.167 ± 0.003	0.258 ± 0.001	0.336 ± 0.003
A.2 Dataset preparation
NIST20 [35] is prepared by extracting all positive-mode experimental spectra collected in higher-energy collision-induced dissociation (HCD) mode (i.e., collected on Orbitrap mass spectrometers). Spectra are filtered, so that we keep only those for which the associated molecule (M) has (i) a mass under 1,500 Da, (ii) contains only elements from a predefined set (i.e., "C", "N", "P", "O", "S", "Si", "T", "H", "Cl", "F", "Br", "B", "Se", "Fe", "Co", "As", "Na", "K"), and (iii) is charged with common adduct types (i.e., "[M+H]+", "[M+Na]+", "[M+K]+", "[M-H2O]+", "[M+NH3+H]+", and "[M-2H2O+H]+"). Because non-standard empirical spectra databases [48] often do not include the measured collision energies, we pool all collision energies for each compound-adduct pairing to create a single spectrum. We refer the reader to Young et al. [55] for detailed instructions for purchasing and extracting the NIST20 dataset.
All spectrum intensities are square-root transformed to provide higher weighting to lower intensity peaks, normalized to a maximum intensity of 1 (i.e., through dividing by the maximum intensity),
Figure A1: Example spectra predictions from the NIST20 dataset for 10 randomly selected test molecules. The ground truth spectra are shown underneath in black, with predictions above in teal. Molecules are shown inset.
Table A4: Model coverage of true peak formulae as determined by MAGMa at various max formula cutoffs for the NPLIB1 dataset. Results are calculated for a single held out test split, shown  $\pm$  the standard error of the mean across three random seeds for all models that were retrained. The best value (i.e., highest) is typeset in bold for each column.
Coverage @	10	30	300	1000
Random	0.004	0.014	0.126	0.336
Frequency	0.090	0.151	0.466	0.688
CFM-ID	0.170	0.267	-	-
Autoregressive	0.072 ± 0.001	0.082 ± 0.002	0.095 ± 0.001	0.099 ± 0.000
SCARF-D	0.158 ± 0.001	0.284 ± 0.003	0.681 ± 0.002	0.856 ± 0.002
SCARF-F	0.155 ± 0.002	0.306 ± 0.003	0.708 ± 0.003	0.859 ± 0.001
SCARF	0.164 ± 0.009	0.309 ± 0.014	0.724 ± 0.013	0.879 ± 0.004
Table A5: Model coverage of true peak formulae as determined by MAGMa at various max formula cutoffs for the NIST20 dataset. Results are calculated for a single held out test split, shown  $\pm$  the standard error of the mean across three random seeds for all models that were retrained. The best value (i.e., highest) is typeset in bold for each column.
Coverage @	10	30	300	1000
Random	0.009	0.026	0.232	0.532
Frequency	0.173	0.275	0.659	0.830
CFM-ID	0.197	0.282	-	-
Autoregressive	0.204 ± 0.001	0.262 ± 0.002	0.309 ± 0.005	0.317 ± 0.006
SCARF-D	0.248 ± 0.001	0.425 ± 0.002	0.839 ± 0.002	0.941 ± 0.001
SCARF-F	0.249 ± 0.001	0.476 ± 0.002	0.855 ± 0.000	0.943 ± 0.001
SCARF	0.308 ± 0.002	0.552 ± 0.001	0.907 ± 0.002	0.968 ± 0.001
filtered to exclude any noise peaks with normalized intensity under 0.003, and subsetted to only the top 50 highest intensity peaks. All peaks are mass-shifted by the weight of the parent adduct (i.e., if the spectrum is  $[M + H] + "$ , the weight of a proton is subtracted from each child peak).
A.2.1 Product formulae assignments
Because the precursor ion and adduct species are known for the training dataset, we subtract the precursor adduct mass from every peak in the training set, and attempt to annotate each peak with a plausible product formula (i.e., a subset of the true precursor formula).
We opt to constrain the training product formulae to be subsets of contiguous heavy atoms of the parent molecule as derived with the MAGMa algorithm [40].
We note two important limitations of these heuristics. First, by using molecular substructures to annotate product formulae, our model is less prone to correctly identifying complex rearrangements. Second, it is also possible for adduct switching to occur. Namely, if the precursor ion has a sodium adduct (" $[M + Na] +$ ", some of the product formulae may actually switch and acquire a hydrogen adduct instead. We assume no adduct switching in our formulation, instead focusing on the novelty of the prefix tree decoding approach, as these represent data labeling challenges, rather than modeling challenges.
In addition, any predictive models of product formulae distributions will more closely predict spectra that would be produced on instrumentation similar to the training sets utilized [9, 11]. Given this, we encourage users of such models to treat these predictions as putative, rather than experimentally valid.
Table A6: Spectra prediction in terms of cosine similarity, coverage (proportion of ground-truth peaks that are covered by the top 100 non-zero predictions), validity (the fraction of predicted peaks for which a chemically plausible explanation is possible), and time. Best value in each column is typeset in bold (higher is better for all metrics but time). Values are shown  $\pm$  the standard error of the mean computed across three random seeds on a single test set for all models that could be retrained (i.e., not CFM-ID).
Dataset	NIST20	NPLIB1	Time (s)
Cosine sim.	Coverage	Valid	Cosine sim.	Coverage	Valid
CFM-ID	0.412 ± 0.000	0.278 ± 0.000	1.00 ± 0.000	0.377 ± 0.000	0.235 ± 0.000	1.00 ± 0.000	1114.7
3DMolMS	0.510 ± 0.000	0.734 ± 0.001	0.94 ± 0.001	0.394 ± 0.002	0.507 ± 0.001	0.92 ± 0.000	3.5
FixedVocab	0.704 ± 0.000	0.788 ± 0.000	1.00 ± 0.000	0.568 ± 0.002	0.563 ± 0.001	1.00 ± 0.000	5.5
NEIMS (FFN)	0.617 ± 0.000	0.746 ± 0.001	0.95 ± 0.001	0.491 ± 0.002	0.524 ± 0.001	0.95 ± 0.000	3.9
NEIMS (GNN)	0.694 ± 0.000	0.780 ± 0.000	0.95 ± 0.001	0.521 ± 0.002	0.547 ± 0.003	0.94 ± 0.001	4.9
SCARF	0.726 ± 0.001	0.807 ± 0.000	1.00 ± 0.000	0.536 ± 0.007	0.552 ± 0.008	1.00 ± 0.000	21.1