Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a transformer architectural variant. The motivation of this variant is that it contains a sliding language modeling objective for sentence scoring, that enables it to look at all the tokens in a sentence using bidirectional context in a single pass. This solves the issue of using standard causal language modeling or masked language models, which respectively have limitations of using only unidirectional context and requiring multiple forward passes. 
strengths_and_weaknesses: ## Strengths
- The structure of the paper is well-motivated. It is clear what the limitations of causal LM and MLM are for sentence scoring, and the paper proposes an architecture that addresses those two issues. 
- The writing of the paper is quite clear. There is a natural progression from motivation to method to results, and overall the paper is pretty understandable.
- The significance of the paper is very high *in principle*. As causal LM and MLM are the standard objectives that are very commonly used in NLP, a new paradigm that outperforms these two types of objectives would be very impactful.
- The fact that SLM needs 3 times the compute of similar sized models is a limitation, but the authors explicitly explored this Section 5.

## Weaknesses
The promise of a better sentence scorer is potentially very impactful. The authors chose to evaluate sentence re-ranking on multiple generated candidates for NMT and for ASR. I have several concerns about these experiments that made them not particularly convincing for me. 
- First, the experiments were, in my view, are a bit thin. For instance, the gain from SLM over BERT is not very large, and there is still a large gap between the oracle. Furthermore, while several baselines were provided, there have been many architectures recently proposed, and probably more comparisons are needed to fully justify the method. As some examples, it could have been good to compare with T5, BART, XLNet, etc.
- I am also not sure about the setup for the ablations. Why were the ablations not performed on all the datasets? I think this could have been addressed by putting a final row in Table 2, with SLM with one-third of the compute so that the compute is matched. It is fine if it does not win in all settings under the same compute, but in this case it would be good to be transparent about how much compute is needed to get the same performance, and whether there are performance gains in multiple different compute-matched settings.
- Finally, I am not sure why NMT and ASR were chosen as the experimental settings in the first place. It could have been stronger if the paper had chosen a language understanding benchmark like GLUE, which could be a better fit for pretrained language models.

I have voted for rejection primarily because the experiments are not particularly convincing, but if enough of the questions/concerns that I had are addressed in the rebuttal I will be happy to change my score.
questions: - The gains from SLM in the experiments required tuning a hyperparameter on a dev set and selecting the best value to use on the test set. Was the same hyperparameter value usable across all datasets? In what hyperparameter range was the model performing better than the baselines? What happens if we don't tune this hyperparameter?
- Why was NMT and ASR chosen instead of general language understanding, which how LMs such as BERT and GPT are mostly used today? If transcormer does better in those settings, then the impact of this work will be much greater. For instance, sentiment analysis can be formulated as scoring two sentences: "the sentiment of <review> is positive" and "the sentiment of <review> is negative". This is called "rank classification", and this is how GPT-3 is evaluated. While it is not necessary to re-train a GPT-3 size model, even on the small scale some experiments on language understanding tasks like these could improve the experimental validation of the paper immensely.
- I might have missed it but will the code be made publicly available to reproduce the experiments? This might be important to gain adoption for this method.
- Since there is a high-cost of re-training from scratch, is it possible for existing language models to be adapted for this new objective? This might enable more compute-efficient experiments. 
limitations: The authors were up-front about the additional computational cost of their method, though additional experiments would be needed for the empirical gains in compute-matched settings to be convincing for me.

The societal impact part seems fine.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work presents a Transformer-based model for sentence scoring, which aims to overcome the limitations of GPT-style causal language model pretraining (only the unidirectional context is used) and BERT-style masked language model pretraining (multiple forward passes are required when scores for each token are needed). The authors propose an XLNet-style "multi-stream" self-attention, which employs a forward stream (to capture the LTR context), a backward stream (for the RTL context), and a query stream which captures information from both.

This method is experimentally validated on two tasks, neural machine translation and reranking for ASR. The proposed method generally outperforms CLM and MLM approaches. The authors also perform further experiments showing that the proposed method is still able to outperform CLM when the size of the models is adjusted such that the amount of inference-time computation is equal; finally, they also attempt to make MLM more efficient by masking multiple tokens at a time (thereby reducing the amount of required computation), but the proposed method still comes out ahead.
strengths_and_weaknesses: The main idea seems like fairly straightforward variant of the XLNet architecture, not too dissimilar from other past proposals to modify its base architecture by changing the nature of the "streams".

Aside from originality, overall the paper feels solid. Both the architecture and the experiments are clearly described. The tasks chosen by the authors for the evaluation are realistic, and at least for the NMT experiments the baselines are sensible and their performance is in the right ballpark.

Given that one of the selling points of the proposed method is computational efficiency, the experiments of sections 5.1 and 5.2 are useful. The results are promising, showing that under comparable amounts of computation the authors' proposal still comes ahead.

A couple of minor points:
* It would be useful to have dataset and split sizes for the experiments of section 4 directly in the paper.
* IWSLT14 might be a small dataset, but I don't think German, Spanish, Italian, etc. should be described as "low-resource". 
questions: Could you clarify what the "oracle" line in Tabels 2 and 3 represents exactly?
limitations: I do not see any potential negative societal impacts of this work. Limitations are sufficiently addressed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a modified Transformer encoder, the Transcormer, which gives per-position log (pseudo-)probabilities in a single pass (unlike masked LMs) while still capturing token-token dependencies at all layers. These probabilities sum to give a score that can be used to rescore NMT and ASR hypotheses. This works by having the key-value representations be concatenated from a forward (inclusive) and backward (inclusive) attention stream before taking the outer product with the query, which never sees the content of its own position. Experiments on IWSLT, WMT, and LibriSpeech suggest this scheme outperforms MLM and CLM for rescoring.

**Post-rebuttal: I raise soundness to 4/4 and my score from 4/10 to 6/10.** The work is thorough and introduces an architectural innovation to more richly model $P(y_s|y_{\backslash s})$ in one pass. Unlike the paper originally claimed, it is not the first such model (T-TA and Electric); also, unsupervised sentence scoring remains niche (MLM scoring is more for inspection of pre-existing models, and for downstream tasks most works use discriminative ranking). However, the authors have done many experiments, it performs better than MLMs and previous models, and may have applications elsewhere (analogous to XLNet, also published at NeurIPS). There are still many grammar errors; please work with a copyeditor or native English speaker on any camera-ready.
strengths_and_weaknesses: The authors succeed in proposing a scheme that models token probabilities conditioned on all other tokens (Eq. 1), in a single pass, while preserving "deep bidirectionality". The method appears sound, and while the mechanism is complicated, Figures 1 and 2 do well to explain it (though Figure 2b should have labels saying which axis is layer $l-1$ and which is layer $l$).

However, I have concerns re: the clarity and significance of this work.

Firstly, there are two uncited works that have proposed one-pass solutions to token-wise probabilities:
- T-TA: J Shin, Y Lee, S Yoon, K Jung, “Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning”, ACL 2020
- Electric: K Clark, M-T Luong, Q Le, C Manning, "Pre-Training Transformers as Energy-Based Cloze Models", EMNLP 2020

In particular, the first work's proposed T-TA has many similarities:
- a query initialized with position embeddings only
- a query that is updated through to the end and is selectively blocked on the self position

while having some benefits (no 3x in cost?) and disadvantages (the keys, values do not transform over layers).

The authors should also acknowledge [18] in section 5.3, where an almost identical analysis is performed, even down to the choice of length and plot (Figure 3 here is largely [18]'s Figures 3+4).

While the authors compare on some tasks (NMT, ASR) as prior work, it would be good to compare on the same datasets or even n-best lists. For example, [18], [20], and the above two works compare on a shared set of n-best LibriSpeech lists while this work creates new ones. This is mitigated by the authors releasing model code (thank you!), but would help convince readers -- especially as SLM models are not off-the-shelf but trained from scratch (and hopefully released).

It is also unclear e.g., what L20's "quality of a sentence" or what L23-24's "precise sentence scoring" mean. Only in Section 2.1 (page 3) do we get two definitions:
- via CLM, which by the chain rule gives the log-likelihood score $\log P(y_1, ..., y_{|y|})$
- via MLM, which gives the pseudo-log-likelihood score $\sum_{i=1}^{|y|} \log P(y_i \mid y_{\backslash y_i})$

SLM models the latter. It would be better to make this clear early, and *then* mention how/why these quantities are empirically (or even theoretically?) good quantities to rescore with (so before in 26-38), with the second being better (which is why(?) SLM models it). Discriminative rescoring should also be discussed/contrasted early, or at least in the main text (not the appendix).

I appreciate some of the analyses which complement past work, in particular A.3.1's discussion comparing sentence vs. fixed-length pre-training, and 5.3 showing SLM and MLM's similar scores. Section 5.1 and 5.2 are unsatisfying though, as increasing the number of masked tokens would be expected to be poor given the mismatch with pretraining (in BERT, masking 15%).

Minor notes:
- The word "sliding" evokes sliding windows (i.e., only a contiguous, fixed length of inputs are considered at any time), giving the wrong impression
- L195-198: The structure of bi-LMs should be explained more
- L163: should clarify that $p_i$ refer to position _embeddings_

Writing should be proofread, e.g.,
- L18 and elsewhere: "language modelings" --> "language modeling approaches" or "language modeling schemes" or even "language models"
- L85: "in a chain-style rule" --> "via the chain rule"
- Figure 2: "reuse" --> "reuses"; "can not" --> "cannot"
- Table 5, Appendix Table 5: "test-clear" --> "test-clean"
- L233, 236, Appendix Table 5: "LibrSpeech" --> "LibriSpeech"
questions: 
- Could you compare and contrast your approach with other one-pass models (T-TA, Electric)?

- Given only the sum of probabilities is ever used and since one must pretrain anyways, have you compared with the stochastic estimation or distillation methods (L96-L97)? Alternately, do you have an application where token-wise probabilities are essential?

- Will you release your pretrained SLMs?

- L199-200 (SLM for scoring, CLM for NLG, MLM for NLU) is not really explained -- are there experiments for this? esp. since MLM and SLM model the same quantity.

- Have you considered ways of matching MLM and SLM costs that avoid MLM pretraining mismatch (and how do you formally define cost), e.g., a shallower MLM, lower attention dimensions? Alternately, what if the MLM were pretrained with masked sets?

limitations: While SLM gives token-wise probabilities out of the box, it does require pretraining from scratch and is rather specialized (whereas MLM and CLM have other uses). Hopefully more uses can be explored to justify this cost; see works above for examples, like acceptability (BLiMP) or text similarity (STS).
review_title: null

Meta-review: metareview: This paper proposes a transformer architectural variant. The motivation of this variant is that it contains a sliding language modeling objective for sentence scoring, that enables it to look at all the tokens in a sentence using bidirectional context in a single pass. This solves the issue of using standard causal language modeling or masked language models, which respectively have limitations of using only unidirectional context and requiring multiple forward passes.
The empirical gains in terms of quality are modest, but the speed-ups are quite impressive. It would have been nice to see evaluations on a few more tasks (like SuperGLUE). It is a bit unclear why such results are not presented, but on balance the paper is probably still above the cut-off.
confidence: Less certain
Decision: Accept