Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper reports a set of experimental results based on protein sequence masked language modeling using a convolutional architecture, which they name "CARP", comparing to ESM-1b, a previously published transformer based model. The authors show that
  * CARP performs similarly to ESM-1b in terms of MLM loss, for different orders of magnitudes of parameter counts,
  * CARP is sometimes better than ESM-1b on some downstream tasks, specifically when the evaluation set is out of domain, but it's worse on tasks that involve structure prediction of proteins,
  * CARP might generalize better to sequence lengths longer than the ones seen in training.

The authors make some progress towards disentangling the effect of pretraining data and transformer architecture on protein sequence pretraining applications. They show that in some cases pretraining more actually hurts performance on downstream tasks.
strength_and_weaknesses: Strengths:
* The paper gives an interesting account of current work on protein sequence language modeling. This might be useful for researchers unfamiliar with these applications.
* The observation that the authors’ CNN (CARP) obtains nearly the same loss during pretraining as a transformer (ESM) is quite interesting and surprising. (Figure 2) 
* The paper makes the interesting observation that model size and pre-training don’t always improve downstream tasks.
* The out-of-domain results seem to strongly favor CARP. This could be interesting, since it shows that for some tasks CNNs might be generalizing better in this application space.
 
Weaknesses:
* The paper says “The **primary** drawback of transformers is that the compute and memory required by the attention layers scale quadratically with input sequence length“. This is too strong a statement. Quadratic dependency on the length is, at best, **one** of the drawbacks, but probably not the primary one. There are many versions of transformers that don’t have quadratic dependence on the length of the sequence and can outperform convolution approaches, e.g. Big Bird (Zaheer et al, 2020, cited in this paper), Performer (Choromanski et al, 2020, also cited in this paper).
* The paper claims that “[positional] encodings can be difficult to extend past the maximum length seen during training”. Where are the limits coming however on the input length during training? If the length during training is capped artificially because of memory requirements, then this limitation can be removed again by using an architecture like Big Bird, and by simply further training Big Bird on longer input sequences after the initial pre-training.
* Figure 2b is potentially misleading, since it’s comparing MLM loss on different test sets (March 2018 UniRef50 for ESM-1b vs March 2020 UniRef50 for CARP). Why didn’t the authors evaluate ESM-1b on the March 2020 UniRef50 test set? The MLM loss for higher sequence lengths is also very noisy, indicating that there are perhaps very few sequences of high lengths?
For example the CNN results in Table 1 are markedly lower than the transformer based ESM-1b baseline. The authors show this table to show that CNNs are not at too large a disadvantage compared to transformers, but the gap still seems large. ESM-1b also wins on zero-shot evaluations, although CNNs seem very close in this case.
* The paper draws a distinction between convolutional nets and transformers, but it should be conceptually simple to create a hybrid architecture, by adding an attention mechanism and positional embeddings in the convolution. It could be interesting to examine this in future work.

clarity,_quality,_novelty_and_reproducibility: The paper is clear and well written. The finding that convolutional nets can outperform transformers on out-of-domain evaluations seems surprising and perhaps novel.
summary_of_the_review: The paper experiments with a convolutional architecture for protein sequence pretraining and shows that in some downstream tasks it can outpeform a recently published transformer model. The improvements seem to mostly be on out-of-domain tasks (Table 2).

It's a bit unclear how significant these out-of-domain improvements are, since they are only compared to the ESM-1b baseline, which seems to do terribly on some of these tasks.

Still the discussion is quite interesting to read and it could conceivably lead to a better architecture in future work, by combining the ESM and CARP architectures perhaps, or by finding a way to improve the transformer architecture to do better in out-of-domain evaluations.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Protein sequence language models are mostly based on the Transformer architecture; while the complexity of it increases exponentially. Thus, this paper studies the use of CNNs instead of Transformer for masked language model pre-training and finetuning. Authors proposed the CARP model for protein sequence pre-training. CNN exhibits the linear complexity according to the length of sequences. More importantly, in some downstream evaluations for estimating the structure prediction and zero-shot mutation effect prediction and for OOD generalization, CNN performs better than the Transformer.

strength_and_weaknesses: Authors proposed the alternative approaches to the Transformer using the CNN architecture. Standard dilated convolutions are involved to further boost the speed. 

Compared to the ESM-1b paper, the results are not good;sometimes showing inferior results. In some fields, the proposed CNN-based model is showing better accuracy in some cases; at the same time it is showing inferior results for other cases. I am not clearly convinced with the authors’ arguments saying that CNNs are alternative to the Transformers. 
Also, there is no result for longer sequences; while this was the initial question for this draft.

clarity,_quality,_novelty_and_reproducibility: The explanations are good enough for reproducing the results. 
summary_of_the_review: The motivation for this paper is good; while I do not think the current draft is effectively offering the evidence for better speed and accuracy. Thus, I recommend reject for this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: the submission compared convolution neural networks of the same building block and the transformer models on the impact of masked pre-training on downstream tasks including secondary structure predictions and protein design tasks. Results show that there isn't a significant difference between convolutional neural networks and transformer models on downstream tasks. Based on the empirical evidence, the submission argues that the effect of pre-training should be decoupled from the architercture.
strength_and_weaknesses: *Strength*

1. The presentation is clear, and the decoupling claim is reasonable.

2. Empirical results are comprehensive, and downstream tasks contain testing various aspects of a pre-trained model.


*Weaknesses*

1. there are many ways to handle longer sequences during inference for transformers, for example, from the perspective of positional embeddings, one can use a set of fixed embeddings or recently proposed rotary embeddings.

2. it wasn't immediately clear to me whether positional embeddings were used for convolutional neural networks. Since convolution kernels are homogenous across locations, I imagine that positional embeddings would be helpful.

3. it always puzzles me whether long-distance attention is learnt in transformers, and I understand that people have designed downstream tasks that require long-distance attention to perform well, but what if the realised tasks themselves could be solved with a decent accuracy using only short-distance attention? I was wondering if there exists work on checking the biases in the downstream tasks? 
clarity,_quality,_novelty_and_reproducibility: the presentation is clear. 

although the model and the experiments are not very original, the argument that the effect of pre-training should be decoupled from the choice of model architecture is reasonable.
summary_of_the_review: I would vote for weak acceptance of this paper so that researchers in the field are aware of the decent performance of convolutional neural networks and their argument on decoupling.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The main claim of the paper is that despite the Transformer is used as a dominate solution for sequence modeling in protein settings, CNNs are also a viable solution as well and CNNs enjoy a linear complexity in computation which is very suitable for long sequence modeling tasks in bioinformatics. The reviewers and the authors had a good discussion about the paper and the authors are very responsive to the questions that the reviewers posted. After summarizing the reviews and the revisions, I believe the paper does not meet the acceptance standard now. The main reason I think are two folded. First, I believe mentioning that using CNNs is a viable solution is a good contribution, however, it’s a somewhat obvious thing to the research community. In NLP, people using CNNs as the replacement for LSTMs in machine translation and many other applications as well. It isn’t a particular surprising conclusion or a very novel usage. Second, perhaps more importantly, the comparison in experiments may point out CNNs’ comparable results, but it does not offer quite a lot of insights of why it’s doing better when comparing with transforms, especially in the protein modeling setting. As a paper which holds such claims, giving more insights I believe is a very important thing. I therefore recommend not to accept the paper.
Decision: Reject