Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper applied sinusoidal embeddings for spectral library search in mass spectrometry in metabolomics. 
strength_and_weaknesses: The authors propose to use sinusoidal embedding to obtain a representation of the measured spectrum, allowing the comparison of similarities between measured compounds. The advantage of using sinusoidal embeddings in mass spectrometry is significant and was already demonstrated by Yilmaz et al. Here the authors additionally confirmed it with experiments in metabolomics. 

Minor comments:
* What is a “development” set? What happened to the “validation” set? Why do the authors call 
“validation” set “development” set?
* From the description in section 3.1 it is unclear whether test and validation sets can have an overlapping set of molecules. It is clear that they don’t overlap with training, but not clear if they overlap with each other. The whole section is not very clear. The part about known and not known molecules is not very clear either. Also stratification by spectrum alone and not by molecules is also unusual. I have concerns that the model can overfit individual molecules. The network proposed by the authors is much larger than the benchmarks and it is not clear to me how the authors demonstrated overfitting if the test set contains molecules present at training.
* Section 3.2 is not well written. The metrics are not well-defined. The thresholds seem arbitrary. It is not clear at all how these numbers were estimated. Was FDR used to find these cutoffs? Maybe these metrics are commonly used in metabolomics, but they should be clearly described for a machine learning conference.
* The authors demonstrate the improved numbers in their tables, but since the metrics and the whole evaluation are not transparent, it is hard to assess what is the actual significance of these numbers.


clarity,_quality,_novelty_and_reproducibility: I don’t see any technical novelty in the paper. Sinusoidal embeddings were already proposed for the field of mass spectrometry (Yilmaz et al.) as noted by the authors. The idea of using neural networks for spectral library search is not novel either as confirmed by the authors. The results section is not easy to read if you are not very familiar with particular evaluation benchmarks in mass spectrometry in metabolomics. 

summary_of_the_review: The paper empirically demonstrates the advantage of using sinusoidal embeddings for spectral library search in metabolomics. The idea of using sinusoidal embeddings to avoid binning of ms spectrum is not novel and was recently published. The authors take this idea one step further and demonstrate that it is superior that currently existing simple benchmarks for spectral library search in the field of metabolomics. The biggest contribution of this paper is the empirical demonstration of their results, but this section is not transparent at all.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use sinusoidal embeddings to represent m/z and intensity values when modeling metabolomics mass spectrometry data using deep learning models.  The idea is sensible though not particularly novel.  Such embeddings have been used previously for mass spectrometry data (as referenced in the related work section), just not for data generated from small molecules.  Three sets of results are given.  The first set I found confusing (see details below), but the upshot there seems to be that the proposed method works better than state-of-the-art methods.  The second set of results involve interpreting UMAPs.  I didn't find this section compelling, and I think it should be eliminated.  The third set of results involves predicting physical properties of molecules directly from spectra. This seemed like an interesting and potentially impactful application, and I was surprised that no one apparently has worked on this problem previously.

strength_and_weaknesses: Overall, the paper makes a modest contribution in terms of novelty, and with some reorganization could tell a reasonably compelling story.

One problem is that the paper is not written in a way that is accessible to an ICLR audience.  I will try to outline below some places where I got confused, but keep in mind that I have a lot more experience working with mass spec data than your typical ICLR reader.  I'm a bit worried that you will find my list of questions below frustrating to read, because probably many of the answers to my questions are embedded in the text.  My goal is to show you how a reasonably well-informed reader got confused by the paper as written.

In the paragraph that spans pp. 1-2, I don't understand why discretization techniques should be "particularly susceptible to measurement errors."  Also, it seems strange to me that this discussion doesn't mention edge effects, which is another clear drawback of the discretization approach (separate from the loss of precision).

The final paragraph of the introduction does not do a good job of making clear whether the proposed method represents an improvement relative to the current state of the art.

There is something important that I simply don't understand about the spectral library search application.  My understanding is that the state of the art method is to use a sort of generalized scalar product (or cosine similarity) that is carried out directly on the non-discretized list of peaks.  I think this is the method used in several widely used library search tools, e.g., from Pieter Dorrenstein's group.  I think the paper should explain how methods like the ones cited in Section 1.1 (e.g., Huber 2021b) improve on this simple approach that does not require discretization.  Results from a simple method like this, as well as perhaps cosine similarity carried out on discretized vectors, should be included in Table 1.

Also in Section 1.1, please explain what it means to do discretization by "direct molecular formula assignment."  Without a definition, it's impossible to understand the subsequent points made in this paragraph.

The paper claims that "the peaks have no intrinsic ordering," but I disagree.
Indeed, it seems quite strange to me that the peaks in the spectrum are not sorted by m/z for input to the model.

For the comparison to tokenized peak embedding (Section 2.1.1), it seems like a key parameter is the bin size of 0.1.  To make the comparison informative, you should vary this parameter to find an optimal setting for a given dataset.

The paper uses the term "molecular similarity," but I don't really know what this means.  For example, we are told that "Molecular similarity scores range from 0 to 1."  Which specific scores are referred to here, and how are they computed.  This seems critical because later there are thresholds (0.60 and 0.95) that get applied to these scores.

I find it highly problematic to report results based on an "internal, proprietary dataset," since this renders all of the results irreproducible.  The experiments should be performed only using publicly available data, and that data, including the splits, should be made available along with the paper.

Section 3.1 uses a lot of jargon that needs to be explained for an ICLR audience.  E.g., "positive and negative ion mode" and "collison energies."  What is involved in stripping stereochemistry from molecular structure labels, and why does this step allow for better molecule-disjoint splitting?  What are "chemical fingerprints"?

I don't understand what the phrase "lookup reference" refers to.  Why don't you just say that you query it against the training set?  What is being "looked up" here?  Later, you refer to a "reference set." Is this just the same as the training set?  Please be consistent in your terminology.

In the evaluation framework, why it it sensible to use arbitrary score thresholds (0.60 and 0.95)? Why not just use a rank-based approach akin to ROC?  It seems to me that if you really are interested in doing spectral library search, what matters is whether the top-ranked match for each query is in fact the correct molecule.  This gives you a ranked list of spectrum-spectrum matches, where each has a binary label of correct or incorrect.

What does it mean to "macro average" something?

I got lost in the paragraph that begins "Because novel molecules ..."  Where do the numbers 0.31 and 0.89 come from? Is this an empirical result, and if so, how is it computed.  I am not convinced that the random baseline is particularly interesting.  What does seem interesting is how the generalized scalar product, which is trivial to compute and requires no discretization, would perform.

I did not find the UMAP analysis particularly informative or compelling.

The property prediction task seems quite interesting, and I am surprised that no one has done this before.  

Minor:

On p. 1, modify the sentence that talks about "modeling m/z values as numeric quantities" to clarify that you are contrasting this with the discretization approach.

clarity,_quality,_novelty_and_reproducibility: I covered these points above.

summary_of_the_review: This paper addresses an important problem but uses a fairly straightforward technique that has been used previously in very similar settings.  The paper is at times hard to follow, and of the three sets of results only the third is very compelling.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed multi-scale sinusoidal embeddings for spectral library search.  Then the proposed sinusoidal m/z peak embeddings are fed into a transformer to get a single embedding vector as an output, which is then fed into a feedforward neural network to predict chemical properties.
strength_and_weaknesses: Strength: The idea of sinusoidal embeddings seems to be well-motivated for this special task: MS analysis. Also, the authors explain the idea quite clearly and the paper is well-written.

Weakness: Firstly, I think the analysis of tandem mass spectrometry is a very niche problem in metabolomics, which seems to be less relevant to the focus of the ICLR conference. I was wondering if this paper could submit to a more specialized conference and journal. 
Second, it seems to the reviewer that neither the code nor the dataset can be made publicly available because of intellectual property reasons, which are against the spirit of ICLR and the reviewer doubts its potential impact on the scientific community.   
clarity,_quality,_novelty_and_reproducibility: I think the lack of reproducibility in both the dataset and implementation aspects hinders the impact and quality of this paper.
summary_of_the_review: Overall, because of the niche topic that is less relevant to ICLR and the closed-source dataset and code, I don't think this paper is appropriate for ICLR acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes multi-scale sinusoidal embeddings for spectral library search. 

Strength of the paper: 
1. The idea of sinusoidal embeddings seems to be well-motivated for this special task: Mass Spectrometry analysis. 

Weakness of the paper: 
1. The current writing is inaccessible to ICLR audience with a lot of confusing expressions. 


justification_for_why_not_higher_score: The topic is a bit distant from ICLR and the writing is unclear.
Decision: Reject