Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work uses sequence-to-seqeunce transformers to align multiple sequences which is an important problem in computational biology.  The approach mainly proposes a scheme to convert multiple seqeunces into one long input sequence and the desired output aligned sequence as the output sequence in the seq2seq setup. The training data for this setup is generated via simulation. To account for long sequences, a heuristic scheme for splitting inputs/outputs into multiple shorter seqeunces is described. An ensemble of transformers is trained where each transformer is trained on the data from a different simulation and this ensemble is used for prediction. Finally, the proposed approach is compared against other commonly used multiple sequence aligners on datasets of neucleotide and protein sequences.
strength_and_weaknesses: I am not well-versed in computational biology, so I am unable to judge the quality of empirical analysis because I am not sure how effectively the chosen datasets reflect progress in this area, and I am also unaware of the technical details of the baselines and models that are used in comparison. Hence, I will mostly reserve my detailed comments on the proposed technique and the results that are described in the paper.

Strengths:

-- The approach is fairly straightforward to implement, and seems reasonable to try.

-- The results show that the proposed approach is in general better than the baselines when considering the performance across various numbers of sequences to be aligned. Although MUSCLE, and PRANK consistently perform similarly to the proposed approach.

-- The "model misspecification" experiment is interesting because it aims to characterize robustness to distributional variaitons in training/test data. The proposed approach does seem robust to the tested misspecification.

Weaknesses:

-- The approach involves several heuristics and is a fairly common instantiation of generic transformer-based sequence to sequence paradigm. Hence, it is low on technical novelty. However, the application domain of computational biology is unusual and I haven't seen the use of this paradigm in the proposed domain.

-- The main contribution of the paper is to develop schemes for flattening multiple sequences into one input sequence and use the aligned sequence as the output sequence. Although, two such schemes are compared, the choice of the scheme seems arbitrary and inadequate analysis/ theoretical justification is provided for the choice.

-- I might be mistaken but it seems like the test data is also synthetically generated. While this would still be fine for comparing approaches, performance on naturally occurring data would strengthen the paper. Also, why not generate test data under all the proposed schemes in Table 2 and measure performance on different kinds of datasets? 

-- The paper would benefit from a detailed description of the baselines in the main text so that it is clear how the proposed approach differs from existing approaches.

-- Similar to the point above, more analysis to show how the proposed approach is doing better compared to the baselines would help. Concretely, characterizing the errors various approaches differ in would significantly strengthen the paper. Do the baselines have hyperparameters that could be tuned? Were they tuned for different number of sequences etc.

-- MUSCLE and PRANK seem fairly close to the proposed approach. Interestingly, while the results show them to be competitive/better for neucleotides and worse for protein, the misspecification experiment flips this order. This is interesting and should be studied in detail.

-- Minor point: Instead of using "words", please use "token" while talking about sequences.
clarity,_quality,_novelty_and_reproducibility: Please see above. Overall, I think the paper is short on technical novelty but is applied to a novel domain. The results are interesting and supportive of the approach but the paper needs more analysis.
summary_of_the_review: The proposed approach is reasonable and the empirical comparison shows its superiority over other baselines. However, more analysis needs to be done and the baselines and experiments should be more thoroughly explained.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to treat multiple sequence alignment task as a machine translation task by processing 'concats' into 'spaces'. It is an interesting idea, and the authors resolve the main obstacle of transformer sequence length limitation with segments.
strength_and_weaknesses: ## Major concern

1. Due to the nature of the transformer, there is no guarantee that all input(e.g. ATCG) will be mapped to the output. The MSA will be meaningless if BetaAlign cannot guarantee that input tokens are not corrupted in the resulting alignment.
2. The experiments are conducted on simulated datasets with few sequences in each MSA. It seems that BetaAlign is more time-consuming than traditional methods since each segment needs to pass the transformer once. The alignment comparison between BetaAlign and traditional ones shows that it is comparable, but it remains a question that why we need deep learning here to achieve similar results but consume more time.
3. The authors should mention HMM-based methods[1] as well as other ones incorporating profiles.
4. Datasets based on structural alignment should be used as the benchmark dataset, like BAliBASE. Using simulated data similar to the training set is not fair for traditional ones.

## Minor comments

1. The construction of the training & testing set is ambiguous. Also, the parameters of traditional methods for benchmarking should be stated.
2. The evaluating metric **SC-error** is confusing. Why not use the original CS (column score) or SPS (Sum-of-pairs score)[2]? I think CS is already normalized.
3. Latest downstream tasks of MSA like AlphaFold2 for 3D structure prediction use MSA with around one hundred sequences for promising performance. Figure 4 should show the comparison of a more diverse number of input sequences.

[1]  S. R. Eddy, Accelerated profile HMM searches. *PLoS Comp. Biol.*, 7:e1002195, 2011.

[2] Julie D. Thompson, Frédéric Plewniak, Olivier Poch, A comprehensive comparison of multiple sequence alignment programs, *Nucleic Acids Research*, Volume 27, Issue 13, 1 July 1999
clarity,_quality,_novelty_and_reproducibility: The idea is kind of interesting. However, the experimental design and results contain flaws. A main category of methods is missed. And the comparison is unfair.
summary_of_the_review: The idea is kind of interesting. However, the motivation is not very clear. Also, the comparison between the proposed method and the traditional MSA methods is not fair. HMM-based methods are missing. The advancement of the method compared to the previous methods is unclear to me.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes using sequence-to-sequence translation from NLP as the core of a Multiple Sequence Alignment. They train their model on phylogenetic simulators and compare against standard alignment methods. 
strength_and_weaknesses: Strengths:
- The paper likely improves the SOTA for some applications of multiple sequence alignments.
- The paper develops an interesting splitting strategy to apply this method for longer sequences.
- The leveraging of simulators allows the method to more closely tailor the alignment to the phylogenetic process.

Weaknesses:
- While model misspecification is addressed in simulator parameter space, it isn't addressed in the mismatch between simulators and real data. An experiment that applies to the "gold standard" real alignment data while the model is trained on simulator data would be a vital experiment to convince the reader that the effects won't decay in the sim2real setting as often occurs in genomics, robotics, and vision.
- Compute can be prohibitive(only applied to a moderate number of sequences) even with the splitting strategy as the number of sequences increases or the number of alignments that need to be performed. Further analysis of compute time(including training) is likely one of the crucial pragmatic aspects of a bioinformatic piece of software that is noticeably missing.
- The splitting strategy likely will not work multiple sequence local alignments given how to split the sequences will likely be unclear.
- More discussion around inferring the correct simulator from data would greatly improve the method's viability.

Minor Comment:
- Providing a more substantial Related Work section in the Main Text would improve its readability and context for this paper.
- Connection to the likelihood-free inference literature for related works (Beaumont et al 2002[1] where ABC was presented and Chan et al 2018 [2] which combines simulators + deep NNs + sets of sequences). 

[1] Beaumont et al (2002). Approximate Bayesian Computation in Population Genetics.
[2] Chan et al (2018). A Likelihood-Free Inference Framework for Population Genetic Data using Exchangeable Neural Networks.
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is generally clear but is missing some key elements such as a Related Works section (some of it is relegated to the Appendix). Additional qualification of the settings in which this method would work well vs poorly as well as additional humility("This pioneering paper") would make extremely clear what problem this alignment method is tackling. For example, this method does not seem well-suited to aligning a large number of sequences (or at the least it wasn't tested) or aligning a large number of sets of sequences. The compute cost can be prohibitive in some settings and the splitting mechanism is not general to any form of multiple sequence alignment and qualifying the exact type of problem this is trying to tackle in the introduction and abstract would likely be greatly appreciated by the reader.

Quality: The paper develops a good idea and executes it over a set of thorough experiments from a ML perspective. However, a few additional experiments are likely necessary to demonstrate its qualities as a viable piece of the bioinformatic workflow.

Novelty: The paper adopts some interesting ideas from NLP and likelihood-free inference and applies it to the MSA problem.

Reproducibility: In order for this paper to be accepted, code should be released given the nature of the problem that is being tackled and its place in the bioinformatics workflow.
summary_of_the_review: The paper provides an interesting combination of ideas and applies them to the MSA problem. On technical and empirical merit, this is interesting and a good paper. The key weaknesses for the methods usability are somewhat uncovered primarily (1) extension to real data (2)  addressing of compute challenges and (3) how to infer the correct simulator parameters. Furthermore, the exposition is a bit grandiose that sets the reader up for high expectations that detracts from the overall work. More humility in word choice, providing reader context of related work, and defining clearly the set of MSA tasks the method excels in and struggles at would greatly improve the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a novel solution to the sequence-to-sequence alignment problem based on transformer. The sequence-to-sequence alignment problem is an important problem in computational biology and is often solved with dynamic programming approaches. The paper proses a creative solution to this problem by formulating the sequence-to-sequence alignment problem as a problem of translating from one to another language, which in turn is addressed by transformers.

I find this is a very interesting idea, and enjoyed reading the paper very much. The idea to formulate the sequence-to-sequence alignment problem as a translation problem is very creative. This is also the first approach that I have seen that uses deep networks for sequence alignment, and for this it yields very competitive results. Therefore I think this paper is a strong addition to the literature on `learning algorithms'. 

However, all reviewer raised the point that the network is trained on simulated data, and application on real data would be preferable. The authors raised convincing arguments for their choice of using simulated data, and I don't see training and evaluating  as a major weakness. However, I recommend to explain the reasons for using simulated data well in the final paper, since this will be a question many readers from the machine learning community will have. 

A more significant issue raised by R2 is that the training and test distributions are equal or are very similar. Thus, we don't get a good idea on how the method works on data that differs significantly from the training distribution, and this would be very important to know. I therefore recommend that the authors add such simulation in their final paper, and discuss this in a limitations section.

Another issue raised by R1 and R2 is that an aligned output sequence could be different from the input sequence. This failure case can easily be detected, but this is an important point, and I suggest the paper makes this very clear in the final paper that this is a potential limitation. 



justification_for_why_not_higher_score: There are too many small issues like a lack of a more comprehensive out-of-distribution analysis, lacking error analysis, and a lack of highlighting some of the weaknesses for the paper to be highlighted at the conference.
justification_for_why_not_lower_score: see meta review.
Decision: Accept: poster