How Effective are State Space Models for Machine Translation?
Authors: Hugo Pitorro, Pavlo Vasylenko, Marcos Vinicius Treviso, Andre Martins


Reviewer paper summaries: This paper presents a comparison of Transformer models and linear recurrent models (specifically RetNet and Mamba) for machine translation. The authors examine the effectiveness of these models on sentence-level and paragraph-level datasets and explore the impact of integrating attention mechanisms into Mamba.

This work investigates how State Space Models (SSM) perform on machine translation (MT) tasks. Specifically, the authors compare SSM-based models with Transformer-based models and hybrid models of SSM and Transformer. Moreover, pretrain and non-pretrain settings are both explored, and the authors also augment an existing dataset to investigate the impact of sequence lengths in training. Overall, I feel the paper is well-organized and brings interesting findings to the community.
Summary of strengths: The paper provides an experimental comparison between transformers and linear recurrent models for machine translation, providing useful information to the community. The paper directly addresses the scalability issues of transformer attention layers with long contexts, offering an investigation into different options.

1. The paper is well-written and mostly very clear.
2. The experimental designs are thorough, including: 1) pretrained models vs. training from scratch, 2) comparing the length of training samples, 3) Transformer vs. SSM vs. Hybrid of Transformer and SSM.
3. I feel the findings of this work are important to the MT community.
Summary of weaknesses: The paper primarily focuses on evaluating existing models rather than introducing fundamentally new techniques or architectures.

1. I think the WMT14 dataset is a better choice for the NER experiments as the gaps in BLEU and COMET are generally smaller (except for RetNet, which can be seen as an outlier). 
2. The capability of the learning/generalization of Transformer and SSM models may be affected by both the model sizes and dataset sizes.  However, there is a lack of a clear picture from this work of how these two factors play a role in applying these models in MT tasks (in both training from scratch and finetuning settings). 
3. The paragraph-level translation setting is interesting. It may connect to the length extrapolation issue, as we see a clear length discrepancy among the training and testing sets.  However, it is not very clear from the author's analysis why Mamba and Transformer ++ (which is enhanced with Rotary embedding) are extremely bad while training with WMT-6M (which does not have augmentation in length).
Comments, suggestions and typos: The paper is overall clearly written.

1. L21: "show" -> "shows"
2. L150: "relevant and irrelevant inputs" is not clear
3. L385: do you mean in this "regard"?
4. In the NER experiment, the "unseen" category is not clear.
5. L408, "concatenate N random training samples," is not clear. Do you mean grouping 50% of the training data N by N? If so, why does it "consistently maintain a 50% ratio between .... **samples**"?
6. Did the authors perform repeated experiments with different seeds? I feel RetNet's performance on WMT14 is too low.
Overall assessment scores: 3

3.5