Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This work performs a neural architecture search to find an alternative to Transformer language models for efficient training and inference. Primer is a resulting architecture and its two main modifications are (1) squaring ReLU activations and (2) adding depthwise convolution to attention multi-head projections. Primer and Primer-EZ (applied these two modifications only) achieve better efficiency-accuracy trade-off while also following a scaling law like Transformer. 
main_review: The paper is well-organized and clearly written. Extensive experimental results and analysis on them support the effectiveness of the proposed method. Although the scope of this work is for autoregressive language modeling, the impact of this work could be quite significant in that Primer can be used for the alternative to Transformer which is currently a defacto standard architecture for NLP and many other ML problems. Primer is found by the automatic architecture search rather than designed by human intuition 

Strong initialization with Transformer what the authors called conceptual initialization is a neat idea for the efficient search. However, it reduces the scope of possible architecture significantly, limiting a chance to find a largely different novel architecture. It would be nice if there is a way to alleviate this hard limit with some kind of relaxation.

It would be interesting to check whether the search space contains previous works on modifying a Transformer architecture in near search steps from the initialization after an exhaustive literature survey on those works or fix the search strategy inspired by those methods.

It is questionable that the most training efficient architecture (having the best training accuracy with a fixed amount of budget) is also the most accurate model having the best test accuracy because generalization ability might be sacrificed.

Is there any way to include efficiency metrics directly to a fitness score during the search?

In Section 3, other modifications of Primer except two major modifications are removed by future post-processing. Is there a way to prevent these marginal or not generalizable modifications as a result of the search in advance without requiring human intervention?

Minor: the term of DNA is used without mentioning the unabbreviated version (maybe Differential Neural Architecture) first.
limitations_and_societal_impact: The paper honesty reveals its limitations and suggests meaningful future directions. This work focuses on autoregressive language modeling (decoder-only). Nevertheless, Appendix A.11 includes results on encoder-decoder masked language modeling. Results on encoder-only language models like BERT would be also interesting. Since Transformer is universally used for encoder-only, decoder-only, and encoder-decoder, I am curious whether Primer can take a similar role.

I am wondering Primer can perform well even on a GPT-3 scale as the authors mentioned in the limitation section and other modalities.

Does the proposed method generalize to other deep learning libraries other than TensorFlow like PyTorch?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper defines a new neural architecture search framework based on generating TensorFlow-based neural network programs. The paper discusses the search space and the evolutionary search-based searching method. The authors perform the search algorithm on a language modeling benchmark and show two specific modifications to the original Transformer model in the resulting model generalizes well to other tasks: squaring ReLU activation and adding a spatial depth-wise convolution after the QKV transformation of the multi-head attention layer.


main_review: I lean towards accepting the paper. The paper performs neural architecture search on a much larger searching space compared to previous works and provides insights from search results that would be useful for a much wider set of language modeling tasks. Detailed comments:

Strengths:
- This paper works on a much larger search space compared to previous works.
- The distilled insights (i.e. two techniques in Primer-EZ) would be greatly beneficial for the community. I believe this actually will ease the reproduction of the paper because it can now be verified by the evaluation of the effectiveness of these two techniques.
- The evaluation in the paper is comprehensive and the paper is clear and easy to understand.

Weaknesses:
- It would be better to include deeper discussions on why the resulting modification performs well.
- The search result is only a slight modification of the original transformer structure. This seems to contradict the starting point of the paper, which is to introduce a larger search space compared to previous works.

Minor issues:
- Line 59 “a evolutionary search DNA” -> “an evolutionary”

limitations_and_societal_impact: Please refer to the main review section.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper mainly investigates how to perform architecture search to obtain the efficient transformer for auto-regressive language modeling. Compared with previous works, this searched model introduces two simple and effective modifications: squaring ReLU and depthwise convolution. Experimental results also demonstrate that the proposed model can achieve comparable results with fewer computations.
main_review: Strengths:

1. This paper applies neural architecture search, which adopts regularized evolution and conceptual initialization, to obtain the efficient transformer model for language modeling tasks.
2. Experimental results have validated the effectiveness and efficiency of the searched model.

Weaknesses:

1. The contribution of this paper is incremental. First, the searching algorithm is based on the existing works (Evolutionary search). The main difference is this paper aims to obtain a trade-off between the training time and sample efficiency, but what is the optimal trade-off has not been discussed. Second, as a NAS method, the searching algorithm is more important than the searched model. However, the authors cost many columns for introducing the searched model. 
2. The part of the introduction section is like a program instruction. I am not sure whether it is appropriate for a research paper.  
3. Authors highlight that the multi-DConv-Head attention is important, but no ablation study has been conducted for analyzing the impact of this module.

Questions:

1. Is Tensorflow Program necessary? If the core contributions of this paper are the searching algorithm and the searched models, I think it should not be sensitive to the code platform (like PyTorch or MXNet). In other words, I think it is unnecessary to highlight Tensorflow usage in a research paper.
2. This paper argues the improvements are mainly from Squared ReLU and Multi-DConv-Head Attention. So these modules are determined by neural architecture search, or just hand-crafted design?

Comments:

1. Line 115, "search search" => "search"

=======================

After read authors' response, I have raised my score to 5. But I still lean negative opinion on this paper. Here are my responses:
1.	Just as aforementioned, your paper focuses more attention on the model (named Primer) rather than the searching algorithm. Therefore, I think your title needs to be changed, like "Primer: xxx" rather than "searching for efficient transformer". Otherwise, it is unfair for other NAS papers, let alone you have not compared with any other NAS-based methods. And your paper is mainly focused on the autoregressive language model, called "for language modeling", which is also a little over-claim.
2.	Searching efficient transformer has been surveyed in many previous works [a][b][c]. The difference between these works is that they only focus on lightweight compressed transformers due to the limitation of resources.  Considering no one has searched such a large-scale model previously, it can be viewed as a contribution and I have raised my score to 5 based on this merit. However, a question is if under the same computation, is it possible to obtain a better model than other NAS methods? 
3.	About Squared ReLU and Separable Conv plus multi-head attention, I have privately conduct translation experiments on the original Transformer via replacing ReLU with Squared ReLU, and do not observe any improvement in bleu (maybe it is more suitable for the searched architecture or requires different specific hyper-parameters). And for Separable Conv plus multi-head attention has been used in many previous works, including NAS-based methods [a][b][c] and some computer vision works [d][e] also point out that Separable Conv plus multi-head attention can bring additional benefits. Therefore, it is unnecessary to emphasize the improvement is from Squared ReLU and Separable Conv plus multi-head attention in the search space. And more importantly, a convincing approach to validate the effectiveness of Squared ReLU and Separable Conv plus multi-head attention is using a search space with the constraints that we can not use Squared ReLU and Separable Conv plus multi-head attention. In other words,  under these constraints, maybe we can get other advanced architectures, rather than Squared ReLU and Separable Conv plus multi-head attention. However, the authors only tried to replace the corresponding module on the original Transformer and Primer in the ablation study, rather than replace the search space and re-search architecture. 
4.	Will the searching code be released? 
 
 
[a] AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search.

[b] NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search

[c] AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models.

[d] CvT: Introducing Convolutions to Vision Transformers

[e] Co-Scale Conv-Attentional Image Transformers
 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies how to improve the transformer architecture for the task of _autoregressive_ language modeling. To do so, they start from the transformer architecture and do low-level architecture search. They find an architecture, Primer, which provides better training efficiency / scaling laws over the original transformer. Most interestingly, they show that a simple variant of the transformer Primer-EZ, which simply adds a convolution layer and a squaring operation to the ReLU activations, can provide similar or better improvements than the full Primer. They test their implementation of Primer-EZ in multiple codebases and datasets, and show that the gains transfer.
main_review: Strengths:
* I really like the idea of Primer-EZ, where they strip down all the complexity induced by the architecture search and focus on the smallest number of modifications that they can make while still getting good results. One of the downsides of typical NAS is that the resulting architecture is a mess and impossible to implement from scratch---this is fortunately not true for Primer-EZ.
* I like that they tested across numerous codebases and datasets. My takeaway from this part was that Primer-EZ actually works, as opposed to other transformer modifications which arent reproducible across different codebases https://arxiv.org/abs/2102.11972.

Weaknesses:
* The only thing that would make this a “home run” paper for me would be experiments that show gains on masked language model pretraining + finetuning. Right now it seems that Transformer++ is a strong baseline in this setting (see the Appendix). However, I agree with the authors that this is out of scope and that focusing on autoregressive language modeling itself is a fine goal. 
* I would like to see more discussion and comparison with the Evolved Transformer architecture https://arxiv.org/abs/1901.11117. 
limitations_and_societal_impact: Yes, this work can improve training efficiency and reduce costs across the field.
review_title: null

Meta-review: The submission makes two main contributions: defining neural architecture search using lower level primitives than previous work, and using this search to introduce a Transformer variant they call Primer. Results show that Primer can outperform vanilla Transformers on various language modeling tasks.

While three of the reviewers support acceptance, Reviewer dssT raises several concerns, arguing that a new search space is not enough of a contribution. The reviewer also would like to see a search space that does not include the squared ReLU and Separable conv plus multi-head attention that are used in the final solution. Other reviewers appreciated the larger, low level search space. I think that the authors convincingly rebut the need to exclude modules from the search space, as for example, excluding squared ReLU would require excluding multiplication. There are also concerns that it is only evaluated on language modeling, but the authors are up front about this limitation, and I agree with most of the reviewers that this is an important and general enough task to focus on.

Overall, the consensus is that this is a good paper that should be accepted.
Decision: Accept (Poster)