Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling
Authors: Kaitao Song
Yichong Leng
Xu Tan
Yicheng Zou
Tao Qin
Dongsheng Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: Natural Language Processing
Language Modeling
Reranking

Abstract: Sentence scoring aims at measuring the likelihood score of a sentence and is widely used in many natural language processing scenarios, like reranking, which is to select the best sentence from multiple candidates. Previous works on sentence scoring mainly adopted either causal language modeling (CLM) like GPT or masked language modeling (MLM) like BERT, which have some limitations: 1) CLM only utilizes unidirectional information for the probability estimation of a sentence without considering bidirectional context, which affects the scoring quality; 2) MLM can only estimate the probability of partial tokens at a time and thus requires multiple forward passes to estimate the probability of the whole sentence, which incurs large computation and time cost. In this paper, we propose \textit{Transcormer} -- a Transformer model with a novel \textit{sliding language modeling} (SLM) for sentence scoring. Specifically, our SLM adopts a triple-stream self-attention mechanism to estimate the probability of all tokens in a sentence with bidirectional context and only requires a single forward pass. SLM can avoid the limitations of CLM (only unidirectional context) and MLM (multiple forward passes) and inherit their advantages, and thus achieve high effectiveness and efficiency in scoring. Experimental results on multiple tasks demonstrate that our method achieves better performance than other language modelings. 

Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling
Kaitao Song" Yichong Leng".,, Xu Tan', Yicheng Zou?, Tao Qin', Dongsheng Li! Microsoft Research Asiat, University of Science and Technology of China", Fudan University (kaitaosong, xuta, taoqin, dongsoi]mmicroooff croooft........m com, 1yc123go@mail ustc.. edu. en, yczou180fudan. edu.
Abstract
Sentence scoring aims measuring the likelihood score of sentence and is widely used in natural language processing scenarios, like reranking, which is to select the best sentence from multiple candidates. Previous works on sentence scoring mainly adopted either causal language modeling (CLM) like GPT or masked language modeling (MLM) like BERT, which have some limitations: CLM only utilizes unidirectional information for the probability estimation of sentence without considering bidirectional context, which affects the scoring quality; 2) MLM can only estimate the probability of partial tokens at a time and thus requires multiple forward passes estimate the probability of the whole sentence, which incurs large computation and time cost. In this paper, we propose Transcormer a Transformer model with a novel sliding language modeling (SLM) for sentence scoring. Specifically, our SLM adopts triple-stream self-attention mechanism to estimate the probability all tokens sentence with bidirectional context and only requires single forward pass. SLM avoid the limitations CLM (only unidirectional context) and MLM (multiple forward passes) and inherit their advantages, and thus achieve high effectiveness and efficiency in scoring. Experimental results on multiple tasks demonstrate that our method achieves better performance than other language models. Our code and pre-trained models will be released at: https: //tgtgubt.t/.t//////////gttebhtt com/ wwwo/mmiceee wow.///iorrroooto/////r.o.ooooo/ Transcormeri
Introduction
Sentence scoring is measure the log-likelihood score of sentence via language model, that can be used to represent the relative likeliness of a sentence. Specifically, good sentence should have relatively lower log-like ligeeiild score, which means more linguistically acceptable for the sentence [I] Due to such nature, sentence scoring has been widely used in many natural language processing (NLP) scenarios. For instance, it be used to rerank candidates in neural machine translation (NMT) or automatic speech recognition (ASR) tasks [2] 3 or evaluate sentences in linguistic acceptability [4]. Therefore, how to design effective language models calculate sentence scores efficiently is very important.
Recently, neural network based language models (LM) 5 8 have been considered as the most widely used technique for sentence scoring, since they can produce density estimation of the whole sentence by computing the probability of each token and summing up their values as the sentence score. Specifically, causal language modeling (CLM) g and masked language modeling (MLM) [T]] the most representative LMs. For given sentence {21,"" "* mxx) where I; is the i-th
'The first two authors are equal contributions.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

token of x. CLM can predict next token conditioned on unidirectional context and its objective is to
optimize Di log
x P P(ay si))). Hence, CLM is usually used for solving natural language generation (NLG) tasks [9 [12. While for MLM, it replaces a subset of tokens x5 in x as special symbol [MASK] and then predicts the masked tokens based on the corrupted sequence x,ss. The objective
of MLM is to optimize E
i- log P(æs, (xss)), so it is able to learn bidirectional context and can be used for solving natural language understanding (NLU) tasks [O
[5]
Since CLM and MLM
can be learned in an unsupervised fashion, many works [10 13
19 20 have pre-trained these LMs on large-scale corpus to extract powerful linguistic representations, and these models can be directly used out of the box to predict the probabilities of tokens. Inspired by the
14
9
16
17
1 15
successes of LMs, some works [21 [2 23 24 25 have tried to use pre-trained CLMM or MLM to generate sentence scores NMT ASR reranking tasks and achieved some promising results.
However, we notice that both CLM and ML. still suffer from some deficiencies in calculating sentence scores. For instance, some works (25 utilized GPT-style model for ASR reranking within single inference, yet GPT model can only extract unidirectional information due to the limitation of CLM, without considering the whole sentence semantics, and thus affect the sentence score. To utilize bidirectional context, some works [22] 23 24 applied BERT model for rescoring. However, the nature of MLM is mask some tokens in the sentence for prediction, which means it requires the BERT model to forward multiple times and each forward pass only masks one token for prediction. As result, it is tmme-consuming that for calculating sentence scores, CLMM needs one-pass inference but only uses unidirectional information and MLM is costly in computing sentence score although it can use bidirectional context. Therefore, a natural question arises: is it possible to design a language model to use bidirectional context for sentence scoring with only one inference pass.
directly adopt MLM for scoring sentence. In summary, we find
In this paper, we introduce Transcormer, a Transformer model designed for sentence scoring. More specifically, our Transcormer leverages novel language modeling scheme, named sliding lan- guage modeling (SLM), that produces the probability of all tokens within single inference pass and simultaneously utilizes bidirectional context. To fulfill this target, innovatively design triple-stream self-attention mechanism, which consists of two content streams (a forward stream and backward stream) and one query stream. By employing specifically -eeieeee mask strategies on the attention matrix, our method allows each token in the query stream to leverage all token information except itself (i.e., the tokens before and after it) for estimating its probability to avoid any information leakage. To the best of our knowledge, SLM is the first language modeling tailored for sentence scoring. We pre-train our SLM on large-scale corpus, and then evaluate it on multiple datasets. Experimental results demonstrate that Transcormer outperforms the baselines by up to + NMT tasks, ~0% relative improvements on ASR tasks.
0.8/0.6 BLEU score on small/large-sce
The main contributions of this work are summarized as follows:
We analyze the pros and cons of CLM and MLM when using them for scoring sentences, and propose Transcormer with a new sliding language modeling, which uses bidirectional context for probability estimation within a single pass.
We introduce a novel triple-stream self-attention mechanism in SLM, which has two content streams to collect forward/backwaat semantics, and a query stream to estimate the probability of each token in a sentence.
Experimental results on multiple datasets demonstrate the effectiveness and efficiency of our SLM for sentence scoring.
2 Background
2.1 Sentence Scoring
Sentence scoring has long history in NLP applications, especially in reranking tasks (e.g., reranking
for machine translation [26 27 28
29] or speech recognition (00). Generally, given n-best candi-
dates generated by text generation models, reranking aims at scoring each candidate to select the
mainly used statistical LM or combined it with RNN-based LM to calculate the sentence scores. Recently, end-to-end neural network based LM
best answer. Early works (26] 27
28 29
31
32
has became the de facto approach for scoring and has been applied in many NLP tasks 66 5 33

---

Specifically, causal language modeling (CLM) and masked language modeling (MLM) are the most representative language modeling methods, among which GPT I and BERT TO are the most famous examples, respectively. As aforementioned, CLM conditions on the previous states to predict next token so that it can obtain the probability of each token of the sentence in a single pass, but can only capture unidirectional information. For MIM, it is able to use bidirectional context for prediction but the masked-pree peeeed-preiittion mechanism limits MLM to producing the probability of all tokens within one forward pass (since MLM can only provide the probability of masked tokens). To calculate the sentence score, a kind solutions 22 23
is to forward multiple times and only mask one token each time. Therefore, the MLM for sentence scoring is formulated as: Σ |x| log (a;)) We can find that the cost of MLM for scoring needs |x] inference passes, which is too time-consuming.
35 have been proposed to alleviate these issues in MLM. Wang et al. [22) and Salazar et al. attempted to use stochastic estimation or distillation to avoid N-passes problem to approximately estimate the probability of each toke produced by MLM. with a sacrifice of performance. Clark et al. 34 adopted a two-cloze tower [36] with noise-contrastive estimation to provide sentence probability, and Shin et al. [35 only considered word embedding the inputs of key and value in transformer without any interaction. Besides, some works [37 38] adopted discriminative language modeling to approximately estimate sentence scores based on the paired data, but this paradigm must require labeled data from the downstream tasks and cannot utilize unlabled data for pre-training. More discussions about related works can be found in Appendix A.2 Therefore, how to calculate sentence scores efficiently with bidirectional context is the main challenge in MLM.
Some recent works 22 34
Overall, CLM only needs single forward pass to estimate the probability of all tokens but cannot extract bidirectional context, while MLM leverages bidirectional information but needs multiple inference passes. Consequently, we raise a natural question: is it possible to design pre-trained lan- guage model to support all token prediction in a single pass and simultaneously leverage bidirectional information? This is exactly the motivation of our method.
2.2 Multiple-Stream Self-Attention
The pioneer of multiple-stream self-attention is XL.Ne [T6. which introduces two-streams self- attention to incorporate autoregressive pre-training for language understanding, which pre-trains Transformer [39] via using a content stream and a query stream. In details, for the t-th step, the content stream is able capture the dependency from the tokens before the t-th step and itself (i.e.,
æzz)), while the query stream is only allowed to view the tokens before the t-th step (i.e., x211) )) to avoid information leakage. Besides, there are some other variants of two-stream self-attention [17 40 4
which are designed for solving NLU tasks. For example. MPNet [77 used two-stream self- attention to build masked and permuted pre-training. ProphetNet 40 designed multiple query streams to predict N-gram future steps for sequence-to-sequene tasks, and ERNIF-GAN [41] proposed multi- flow generation model, which includes two query streams for span and word prediction. We observe that these works mainly used single content stream and then used one many query streams to predict more information. Different from these works, we introduce a triple-stream self-attention mechanism, which enables query stream to leverage two content streams for prediction, and thus enjoys the benefits of additional bidirectional context for estimating token probability.
3 Transcormer
To inherent the advantages of CLM and MI.M for sentence scoring and avoid their limitations, we propose Transcormer - a Transformer model with novel sliding language modeling for sentence scoring. First, we summarize that an ideal language modeling for sentence scoring should satisfy two requirements: model should be able to use bidirectional context for effective probability estimation of each token; 2) model should produce the probability of all tokens in a sentence within a single inference pass for efficiency. Therefore, to fulfill these two requirements, we formulate a new language modeling, named as sliding language modeling (SLM), and describe it in Section 3.1 In SLM, we propose a Triple-Stream Self-Attention mechanism based on Transformer (please see Section
3.2 for details) to use bidirectional context for each token prediction and avoid information leakage. We also discuss the differences between SLM and other LMs Section 3. presents the pipeline of our Transcormer for sentence scoring with SLM.
Figure
3

---

X1
X2
X3
X
X
Transcormer
P
2
P 3
P 4
S
P 1
2
3
P4
5
P
3
+
+
+
1
2
X3
X4
X5
X
X
X
X
X
forward stream
query stream
backward stream
Figure I: Transcormer with sliding language modeling. The left and the right (in blue) are forward and backward streams, respectively, and the middle (in green) is query stream. For query stream, the inputs are only the positional information. We use gray and red line to represent the allowed attended positions in the content and query streams.
3.1 Sliding Language Modeling
Considering the pros and cons of CLM and MLM for scoring, we notice: 1) CLM can produce probability of all tokens within one forward pass, and thus obtain the unidirectional information of the whole sentence; 2) MLM for sentence scoring needs multiple inference passes, so that many context has been repeatedly calculated and cause a waste of computation. So, it possible reuse token information to build bidirectional context for token prediction?
Therefore, we propose sliding language modeling (SLM) address the inherent flaws in previous LMs (i.e, CLM and MLM) for sentence scoring. Specifically, we first maintain two individual streams collect forward (left-to-right) context and backward (right-to-left) context. And for each token prediction, we decompose the sentence information as the past tokens (the tokens before it) and future tokens (the tokens after it) respectively. As result, our SLM enforces each token to only capture the dependency from its past tokens and its future tokens concurrently, so that each token can utilize the whole sentence information (except itself) to estimate token probability. The objective function of SLM is formulated as:
X
log P(r,(xxcixxx;;))
where X and x respectively correspond to the tokens before the i-th token and after the i-th token, and 0 represents the parameters of SLM. Thanks to such design, our SLM can utilize bidirectional context to produce the probability of each token within one forward pass, and thus satisfy the above requirements for sentence scoring. However, previous experiences [16] 42 pointed out that the states with bidirectional information will cause information leakage when propagating to the next layer Therefore, how implement SL. to avoid information leakage and maintain different states together still a troublesome problem.
Triple-Stream Self-Attention
Based on the Eqn of our proposed SLM, model needs to maintain two states to collect forward and backward contexts for prediction, and we call these two states as the forward stream and backward
Foor example, assume the sequence So each position 4). hi and h3 should collect the information from positions 32, {1, 3). (1,2). However. when i and l/3 are delivered to l at the second layer, it will cause a cyclic leakage as h9 should not obtain information from position 2.
tokens, and the hidden states of i-th token at first layer as A..

---

forward stream
backward stream
Input Sequence
Attention
X2
X
X
X2
X
X
P
P
3
4
ss
forward stream
%
X2
X22
+
+
+
+
X
X2
X
X4
X5
KV
X
X
11
P2
P3
P4
backward stream
%
S
+
+
+
+
1
X
X3
X4
X5
KV
P2
P1
2
Pg
P4
5
Query stream
P4
+
+
Ps query stream
KW KV
(a)
(b)
Figure 2: (a) The structure of our triple-stream self-attention used in our sliding language modeling. The query stream reuses the hidden states from both forward and backward (content) stream the key and value in attention.
(b) The attention mask matrix used in our triple-stream self-attention. The above row is the attention matrix for the forward and backward stream and the below row is the attention matrix for the query stream. The cell in gray color means this position cannot be attended.
stream respectively. To avoid information leakage, we additionally maintain an individual state for prediction and control it to only capture the dependency from the forward and backward streams. We name this state as the query stream. Therefore, we propose a novel Triple-Stream Self-Attention to update each stream, and the detailed design is described as following.
To fulfill our target, we choose Transformer [39 as our basic model, due to its fiexibility in cap- turing global dependency. Assume the input sequence as {W1, W2, , wn} and its positions as {PI.P2. }. where wy and p; represents the embedding of i-th token and its position in the sentence, and n is the token number. For the query stream, we only use the positional embeddings {PI.2.. iP..P..... ... pul as the input. For the forward and backward streams, we also maintain two individual states and both of them use the tokens plus its position (i.e., {wi + Pl, W2 + P2, Wy + Pr, }) as the input. For the /-th layer calculation, we denote the forward and backward streams of the position
i as R. and hts and they are updated as:
M Attention(Q = 1i411 KV R : 0)2 N
(2)
M Attention(Q Q -1,-1 KV
F1-1.00 0),
where Attention(.-) refers the self-attention [39 in Transformer and Q, K, V denote the query, key and value in self-attention. Hence, NV collects the information before the position i and itself, and i collects the information after the position and itself. For the query stream, we denote of as its hidden states, and concatenate forward stream li and backward stream hi of the current layer to be the key/value of query stream. So of is updated as:
df Attention Q = q. ..V
:))
(4)
Here we find that of is required to only capture the dependency from the forward stream before the position and the backward stream after the position i, rather than itself. Due such design, the query stream is able to capture bidirectional context for estimating token probability and avoid information leakage, which is more effective than CLM in using context. More importantly, our triple-stream self-attention enables model to predict the probability of all tokens in a sentence within a single forward pass, which demonstrates more efficiency than MLM. presents the detailed design of our triple-stream self-attention. We can find that in the query stream, the masked matrix like a sliding window to control each token to view its previous states in forward stream and its future states in backward stream. And that is why name our model as sliding language modeling.
5

---

LM
Model
Cost
Context
Scenario
CLM
GPT [
x1
forward
NLG
MLM
BERT [10]
Xn
bidirectional
NLU
Bi-LM
ELMO 43
x2
forward + backward
NLU
SLM
Transcormer
x3
bidirectional
Scoring
Table 1: Comparisons between SLM and other LMs. We assume all LMs adopt the same architectures (e.g., Transformer). The "Cost" column means the relative computations compared with CLM when calculating a sentence with n tokens. The "Context" column means the contextual information usage for prediction.
3.3 Discussion
To better understand our SLMM, we analyze the advantages of our SLM over other L.M.. The comparisons are listed in Table We select three representative LMs for comparisons, which are CLM (BERT), MLM (GPT) and bidirectional LM (Bi-LM, used in ELMO 43 respectively, From Table we have the following observations:
When compared with CLM, our SLM requires 3 x computations. However, SLM can fully use the whole sentence information for prediction while CLM can only use unidirectional information. Even scaling CLM also demonstrates the effectiveness of our SLM in using context.
parameters, still can not use bidirectional context for prediction. This
2. MLM is powerful at extracting bidirectional context but it needs nx inferences to calculate the whole sentence information limited its masked prediction. Our SLM just needs a single inference and uses bidirectional information for prediction with only 3x computations. Especially, our SLM shows higher efficiency compared with MLM when is large.
3. Bi-LM can also extract forward and backward contextual information, but just simply concate- nates the forward and backward features for the final prediction, without any interactions. Instead, our SLM can iteratively fuse the bidirectional information thanks to our triple-stream self-attention mechanism.
Overall, the design of SLM is dedicated for sentence scoring, while CLM prefers NLG tasks and MLM/Bi-LM prefer NLU tasks.
4
Experiments
In this section, we describe our experimental setup, and the results on NMT and ASR datasets.
4.1 Experimental Setup
We adopt Transformer (39 as the backbone network. Following previous works 10, we adopt base setting and small setting for our model as TranscormeTbase with 110M parameters and Transcormer,maat with 34M parameters, that consists of 12/6 transformer layers and each layer has 768/512 hidden size and 12/8 attention heads. During the pre-training. we use wikipedia plus bookcorpus (16GB) as the training corpus, to be consistent with previous works [TO]. Our model trained the sentence-level (i.e., one sentence per sample). We choose Adam [45] the default optimizer with learning rate of 5e 4. 31 = 0.9, 32 0.98 and 1e 6, and weight decay is set as 0.01. The learning rate warms up over the first 10,000 steps and then linearly decays. We set the batch size as 8192 tokens per batch, and the training step is 125,000 steps. We use 32 NVIDIA Tesla 32GB GPUs, with FP16 speedup. The total training needs 5.5 days for TranscormeTbase- For more experimental settings (e.g., dataset and its split sizes), please refer to the Appendix. The code and pre-trained models will be released at: https: :tttss//giitee Transcormeri
com http://www...oo/iiiooooo////////
'ELMO trainsins left-to-right and tight-to-left LSTM [4 and concatenates the outputs of each last unidirectional LSTM layer for prediction.

---

IWSLT
WMT
Model
De
It
NI
Ro
Tr
De-En
Oracle
41.80
48.69
41.89
44.38
27.90
46.01
29.60
27.25
39.17
Baseline
34.77
41.20
34.95
37.73
22.67
38.73
24.21
21.65
32.54
CLM (GPT)
34.96
41.39
35.14
38.08
22.91
39.03
24.62
22.14
32.88
MLM (BERT)
35.14
41.54
35.54
38.14
23.00
39.21
24.65
22.36
33.07
Bi-LM (ELMO)
35.10
41.52
35.21
38.03
23.09
39.07
24.53
21.91
32.90
SLM (Transcormers..ee)
35.24
41.86
35.52
38.45
23.29
39.34
24.69
22.41
33.10
SLM (trans)ormerrmattt)
35.05
41.58
35.15
38.06
22.98
39.08
24.52
22.06
32.94
Table 2:
Reranking results on IWSLT and WMT tasks, and all LMs have the same model architecture as Transcormer. The translation direction of all IWSLT tasks is English and all results are reported in BLEU score. All LMs are pre-trained in the wikipedia + bookcorpus (16GB) with the same optimization. The last row is the oracle score from the generated candidates.
4.2 Experiments on Neural Machine Translation
We choose IWSLT14 dataset [46]. which includes multiple small-scale translation tasks from dif- ferent languages to English, and WMT14 English-German dataset for evaluation. We adopt Transformer 39] as the machine translation model with 6-6 transformer layers to generate multiple candidates for reranking, with beam size of 10. The hidden size and attention head are set as 512/1024 and 8/16 for IWSLT and WMT tasks respectively. During the reranking, we combine the original score produced by the machine translation model and the LM score with hyper-yarament parameter X, by following previous experiences [7 21. The hyper-parament of [0.0, 2.0). and then select the best À to evaluate the test set. The results are reported in Table 2 in terms of BLEU [47]. Each task will be evaluated by five times based on different pre-trained checkpoints, and report the mean value, with variance of 0.05. From Table 2 have the following observations:
À tuned on dev set with a range
Our Transcormerhase can obtain better performance than CLM and Bi-LM 5 in both small-scale IWSLT and large-scale WMT tasks, which indicates the importance of bidirectional context for sentence scoring, and further validate the ability of our SLM utilizing bidirectional information.
When compared with MLM, our Transcormerhase also achieves comparable results. Considering that the computation MI. for scoring linear the input length and needs inference passes, our SL.M show higher efficiency with only 3x computations in single pass to maintain three streams.
Overall, all of the results reveal that our model is more effective in using contextual information for probability estimation and more efficient with only single forward pass, especially for long sentences.
Model
dev-clean
dev-other
test-clean
test-other
Baseline
2.80
6.90
3.06
7.05
CLM (GPT)
2.47
6.13
2.73
6.33
MLM (BERT)
2.30
5,65
2.59
5.90
Bi-LM (ELMO)
2.41
5.92
2.63
6.12
SLM (Transcormerfasse)
2.23
5.54
2.49
5.72
SLM (Transcormer,.o. )ma)))
2.48
5.95
2.62
6.20
Oracle
1.45
4.23
1.59
4.19
Table 3: Reranking results LibrSpeech dataset. All results reported in WER.
Here we only evaluate German Bermman-Fnnglise direction as our model is trained on English domain. Here, replace LSTM Transformer in Bi-LM keep the consistence in architecture.
7

---

Overall
ANA. AGR ARG. STR
BINDING
CTRL. RAIS. D-N AGR ELLIPSIS
FILLER GAP
IRREGULAR
ISLAND
QUANTIFIERS
S-V AGR
NPI
Model
GPT-2 (345M)
82.6
99.4
83.4
77.8
83.0
96.3 86.3 81.3
94.9
71.7
74.7
74.1
88.3
BERT (base)
84.2
97.0
80.0
82.3
79.6
97.6 89.4 83.1 96.5
73.6
84.7
71.2
92.4
BERT (large)
84.8
97.2
80.7
82.0
82.7 97.6 86.4 84.3 92.8 77.0
83.4
72.8 91.9
RoBERTa (base)
85.4
97.3
83.5
77.8
81.9 97.0 91.4 90.1 96.2 80.7
81.0
69.8
91.9
RoBERTa (large)
86.5
97.8
84.6
79.1
84.1
96.8
90.8 88.9 96.8 83.4
85.5
70.2 91.4
Transcormer (base)
84.6
98.1
80.7
83.2
80.2
96.0
90.7
84.1 95.5
74.3
85.7
73.4 91.3
+ BERT Init, 20K steps
85.0
98.1
80.0 84.8 79.2 96.2 89.3 84.6 96.1 76.6 87.8 75.0 91.7
Table Results on BLiMP. The results on GPT-2, BERT and RoBERTa are taken from 21]. The "+ BERT Init, 20K steps" means the Transcormer model uses BERT model for initialization and then trains in SLM with 20K steps (nearly 5 epochs).
4.3
Experiments on Automatic Speech Recognition
We choose LibrSpeech 48 to evaluate the performance of our model for reranking on ASR task. We train a Conformer model 49 on LibrSpeech, which has 12 encoder layers and 6 decoder layers with 512 hidden size and 8 attention heads, and the beam size is set as 10. In addition, we use SpecAug 50 as a data augmentation technology to further improve the accuracy of ASR system. We use word error rate (WER) to evaluate the performance of ASR tasks. We follow the same tuning technique used in NMT tasks for hyper-parameter X, but with a larger range as [0.0, 5.0]. The results are reported in Table From 3 we find that our model can give nearly 20% relative improvements over the baseline and also outperform other LMs, including CLM, MLM and Bi-LM. The results on ASR task further demonstrates the generalization and effectiveness of our SLM in sentence scoring.
4.4 Experiments on Linguistic Acceptability
Following previous experiences 2T. we also conduct experiments on Benchmark of Linguistic Minimal Pairs (BLiMP) 4 which includes 67K minimal pairs that contrast in grammatical accept- ability and isolate specific phenomena in syntax, morphology or semantics. BLiMP provides an unsupervised setting that uses language models to evaluate sentences and the acceptable sentence can be assigned by a lower log-likelihood score. We compare our model with GPT-2, BERT and RoBERTa, and the results reported in Table 4 We find that our Transcormer can easily beat GPT model in BLiMP, which further validates the effectiveness of our SLM in using bidirectional context for sentence scoring. And when compared with BERT model, our Transcormer can also match or outperform BERT performance slightly. Considering that our model only needs a single pass produce the probability of all tokens, which also manifests the efficiency of our SLM in sentence scoring. Due to the limitation of resources, our model currently cannot beat RoBERTa since RoBERTa costs more computations than BERT and thus extract better semantics for prediction. But we believe our Transcormer can achieve the better performance by training our model with the same computations.
Analyses
In this section, we conduct some method analyses on our proposed SLM and CLM/MLM. Besides, we also provide more analyses in Appendix.
5.1 Latency Comparisons between MLM and SLM
To better manifest the efficiency of our SLM in rescoring when compared with MLM, we further test the inference latency between SLM and MLM at both GPU and CPU under different lengths of the input sequence. Specifically, we measure the inference latency of each model at a batch size of to validate the efficiency, and all results are reported in Table 5 From Table we can find that MLM (BERT) produces very high latency when compared with SLM (Transcormer), especially in CPU devices. Even using smaller BERT model, it still cannot avoid the inherent issues (i.e., requires
8

---

п' computations) in MLM for scoring, which needs 20x and 166x additional computations over Transcormer (SLM) in GPU and CPU. These results also demonstrate the efficiency of our model in using bidirectional context for calculating sentence scores.
Sent 10
Sent
100
# Sent = 500
Model
#Params
GPU
CPU
GPU
CPU
GPU
CPU
BERT (small)
34M
53ms
12.87s
502ms
317s
3658ms
1,650s
BERT (base)
110M
76ms
27.06s
750ms
703s
7890ms
3,210s
BERT (large)
340M
135ms
59.91s
1390ms
1676s
19770ms
7,433s
Transcormer (small)
34M
59ms
3.45s
7l
7.64s
183ms
9.90s
Transcormer (base)
110M
67ms
6.29s
97ms
17.85s
246ms
20.03s
Table 5: Inference latency between BERT (MLM) and Transcormer (SLM) at different sequence lengths. "# Sent" means the length of input sequence for evaluation. GPU is evaluated at NVIDIA Tesla V 100-SXMM-166GH and CPU Intel (R) Xeon (R) Platinum 8168 CPU @ 2. 2.70GHz. The units of numbers in "GPU" and "CPU" columns are at millisecond (ms) and second (s).
5.2 Comparable Computation between CLM and SLM
As aforementioned, our SLM needs 3x computations when compared with CLM. To make a fair com- parison, also pre-train Transcormer with 34M parameters in total, which consists 6 trans- former layers and each layer has 512 hidden size and 8 attention heads. Hence, our Transcormer has approximately 1 parameters We conduct experiments on three NMT tasks and an ASR task (LibriSpeech dataset) for comparisons and the results are listed in Table and (i.e. "Transcormer row"). We can find that even under the same computation, our model still outperforms CLM and this result further validates the necessity of using bidirectional context for sentence scoring. Besides, considering that Transcormer,man has fewer parameters, our model is also friendly to the device deployments (e.g., CPU).
TranscormeToase- and has the similar computations as baze.
Model
Cost
PPL
dev-clean
dev-other
test-clean
test-other
Baseline
2.80
6.90
3.06
7.05
MLM (A: 1)
Xn
4.26
2.30
5.65
2.59
5.90
MLM (k: 2)
X [/22]
8.41
2.41
5.87
2.70
6.20
MIM (A: 3)
X n//3
11.58
2.60
5.95
2.87
6.41
MLM (A: =
x3
2.75
6.71
2.98
6.93
MLM (A: )
x2
2.80
6.80
3.01
6.99
SLM
3.85
2.23
5.54
2.49
5.72
Table 6: Comparisons of sampling different tokens for prediction on MLM. We choose ASR reranking task on LibriSpeech dataset to evaluate the results and also report PPL on subset of sentences with same length (n 20).
5.3 Varying Numbers of Forward Passes in MIM
As mentioned above, MLM needs to forward n passes as each time only mask one token. So what will happen if we allow each pass to mask more tokens? Therefore, we design experiments that enforces MLM forward tokens for prediction so that only needs [n/k] passes, and investigate the effect of different k. For certain k; we randomly split the sentence as [n/k] sets, and each time masks one subset for prediction. The comparisons are listed in Table We find that using larger k will severely harm the performance, even if it can reduce the number of inference passes. When is set as 5. which is equal to the cost our SLM, it can hardly give any improvements over baseline. We think that masking more tokens in the sentence will make it more difficult to estimate the token probability at a time. These comparisons also highlight the efficiency and effectiveness of our SLM for sentence scoring.

---

5.4 Sentence Scoring Quality at Each Position
Following previous experiences 21 we count the cross- entropy loss of each position in sentences for each LM, better analyze the effectiveness of bidirectional context to estimate token probability. Specifically, we sample subset of sentences as S, and each sentence has the same token number (here is 20). For each position i, we count the average cross-entropy over the sampled subset S based on the output probability of each LMM The results are displayed in Figure 3 We can find that: 1) For CLM, the cross-entropy loss higher the first several posi- tions and gradually decreases for the subsequent positions but is still higher than MLM and SLM, which indicates that only using undirectional information is not enough to measure the sentence score precisely. SLM almost obtain similar loss as MLM at each position. Consider- ing SLM just needs single pass while MLM needs n passes, this phenomenon further validates the superiority and efficiency of SLM in scoring sentences.
CLM
MLM
SLM
Croco strpy
3: Fosttion
Figure 3: The average cross-entropy loss of each LM at each position. MLM uses passes to predict each position.
6 Conclusion
In this paper, we propose Transcormer, a Transformer with a novel sliding language modeling for sentence scoring. Specifically, our SLM able to produce the probability of each token over the whole sentence within a single forward pass, and utilizes bidirectional context for prediction, and thus inherents the advantages of CLM and MLM and avoids their deficiencies. To the best of our knowledge, the proposed Transcormer is the first pre-trained language model tailored for sentence scoring. Experimental results on multiple datasets demonstrate the effectiveness of our Transcormer in computing sentence score for reranking tasks.
Besides, we summarize some potential directions of our Transcormer and SLM the future works:
Currently our Transcormer is only conducted on the English domain under the base setting, due to limited computation. We expect to develop large-scale Transcormer and use different language domains or multilingual data for training in the future.
We design sliding language modeling for sentence scoring, and our experiments are mainly on reranking task. However, based on the characteristics of our SLM, we believe our model can also be used for other scenarios (e.g., error correction [51] 52 53 data selection), and we will explore the specific fine-tuning techniques when applying our SLM on different downstream tasks.
Besides, our Transcormer mainly pre-trains SLM on an encoder framework. However, our SLM is not limited to model structure. For example, SLM can be easily extended to encoder-decoder framework [88 based on paired data. Therefore, we also expect to explore the possibility of using SLM on different frameworks.
Although our paper mainly focuses on text data, we want to highlight that SLM can also be extended to other different modalities with sequential characteristic (e.g. image, speech [54] and time series data). Consequently, how to apply SLM to other modalities also a valuable topic in the future.
References
[1] Jey Han Lau, Alexander Clark, and Shalom Lappin. Grammaticality. acceptability, and proba- bility: A probabilistic view linguistic knowledge. Cogn. Sci., 41(5): 2017.
[2] Çaglar Gülçchre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huci-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On using monolingual corpora in neural machine translation. CoRR, abs/1503 503.03535, 2015.
[3] William Chan, Navdeep Jaitly, Quoe Le, and Oriol Vinyals. Listen, attend and spell: neural network for large vocabulary conversational speech recognition. In 2016 IEEE International
10

---

Conference on Acoustics, Speech and Signal Processing, ICASSP 2016, Shanghai. China, March 20-25, 2016, pages 4960-4964. IEEE, 2016.
[4] Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang. and Samuel R. Bowman. Blimp: The benchmark of linguistic minimal pairs for english. Trans. Assoc. Comput. Linguistics, 8:377-392, 2020.
[5] Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook fair's WMT19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation, WMT 2019, Florence, Italy, August 1-2, 2019 Volume 2: Shared Task Papers, Day 1, pages 314-319, Association for Computational Linguistics, 2019.
[6] Yingee Xia, Xu Tan, Fei Tian, Fei Gao, Di He, Weicong Chen, Yang Fan, Linyuan Gong, Yichong Leng, Renqian Luo, Yiren Wang, Lijun Wu, Jinhua Zhu, Tao Qin, and Tie- Liu. Microsoft research asia's systems for WMT19. In Proceedings of the Fourth Conference Machine Translation, WMT 2019, Florence, Italy, August 1-2, 2019 - Volume 2: Shared Task Papers, Day 1, pages 424 433. Association for Computational Linguisties, 2019.
[7] Kyra Yee, Yann N. Dauphin, and Michael Auli. Simple and effective noisy channel modeling for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 5695-5700. Association for Computational Linguistics. 2019.
[8] Yuchen Liu, Long Zhou, Yining Wang, Yang Zhao, Jiajun Zhang, and Chengqing Zong. comparable study on model averaging, ensembling and reranking in NMT. In Natural Language Processing and Chinese Computing 7th CCF International Conference, NLPCC 2018, Hohhot, China, August 26-30, 2018, Proceedings, Part II, volume 11109 of Lecture Notes in Computer Science, pages 299-308. Springer, 2018
[9] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre- traininin 2018.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACI, pages 4171-4186, 2019.
[11] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
[12] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, pages 1-16, virtual, 2020. Curran Associates, Inc.
[13] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/ 1907.19 1 1692, 2019.
[14] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, 2020. OpenReview.net.
[15] Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language un- derstanding and generation. In NewrIPS, pages 13042-13054, Vancouver Convention Centre, Vancouver, Canada, 2019, Curran Associates, Inc.

---

[16] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NeurIPS, volume 32, pages 5753-5763, Vancouver Convention Centre, Vancouver, Canada, 2019.
[17] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Van Liu. Mpnet: Masked and permuted pre-training for language understanding. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, virtual, 2020. Curran Associates Inc.
[18] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: Masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 5926-5936, Long Beach Convention Center, Long Beach, 2019. PMLR.
[19] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: denoising sequence-to-sequeee pre-training for natural language generation, translation, and comprehension. In ACL, pages 7871-7880, Online, 2020. Association for Computational Linguistics.
[20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint 1910v: 10683, 2019.
[21] Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020, Online, July 5-10, 2020, pages 2699-2712. Association for Computational Linguistics, 2020.
[22] Alex Wang and Kyunghyun Cho. BERT has a mouth, and it must speak: BERT as a markov random field language model. CoRR, abbs/1902.4444 04094, 2019.
[23] Joonbo Shin, Yoonhyung Lee, and Kyomin Jung. Effective sentence scoring method using BERT for speech recognition. In Proceedings of The 11th Asian Conference on Machine Learning, ACML 2019, 17-19 November 2019, Nagoya, Japan, volume 101 of Proceedings of Machine Learning Research, pages 1081-1093. PMLR, 2019.
[24] Shih-Hsuan Chiu and Berlin Chen. Innovative bert-based reranking language models for speech recognition. In IEEE Spoken Language Technology Workshop, SLT 2021, Shenzhen, China, January 12-22, 2021, pages 266 271. IEEE, 2021.
[25] Yue Weng, Sai Sumanth Miryala, Chandra Khatri, Runze Wang, Huaixiu Zheng, Piero Molino, Mahdi Namazifar, Alexandros Papangelis, Hugh Williams, Franziska Bell, and Gökhan Tür. Joint contextual modeling for ASR correction and language understanding. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pages 6349-6355 IEEE, 2020.
[26] Shankar Kumar and William J. Byrne. Minimum bayes-risk decoding for statistical machine translation. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HL.T-NACLL 2004, Boston, Massachusetts, USA, May 2-7, 2004, pages 169-176, 2004.
[27] Raphael Shu and Hideki Nakayama. Later-stage minimum bayes-risk decoding for neural machine translation. CoRR, abs/ Abs//704..3116) 169, 2017.
[28] Libin Shen, Anoop Sarkar, and Franz Josef Och. Discriminative reranking for machine trans- lation. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2004, Boston, Massachusetts, USA, May 2-7, 2004, pages 177-184, 2004.
[29] Franz Josef Och. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, 7-12 July 2003, Sapporo Convention Center, Sapporo, Japan, pages 160-167, 2003.
12

---

[30] Yukun Ma, Erik Cambria, and Benjamin Bigot. ASR hypothesis reranking using prior- informed restricted boltzmann machine. In Computational Linguistics and Intelligent Text Processing 18th International Conference, CICLing 2017, Budapest, Hungary, April 17-23, 2017, Revised Selected Papers, Part I, volume 10761, pages 503-514, Springer, 2017.
[31] Michael Auli and Jianfeng Gao. Decoder integration and expected BLEU training for recurrent neural network language models. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA. Volume 2: Short Papers. pages 136-142 The Association for Computer Linguistics, 2014.
[32] Terumasa Ehara. In Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation: 5th Workshop on Asian Translation, @AT@ PACLIC 2018, Hong Kong, December 1-3, 2018. Association for Computational Linguistics, 2018.
[33] Yuguang Wang, Shanbo Cheng. Liyang Jiang. Jiajun Yang, Wei Chen, Muze Li, Lin Shi, Yanfeng Wang, and Hongtao Yang. Sogou neural machine translation systems for WMT17. In Proceedings of the Second Conference on Machine Translation, WMT 2017, Copenhagen, Denmark, September 7-8, 2017, pages 410-415, Association for Computational Linguisties, 2017.
[34] Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Pre-training transformers emeryy-based cloze models. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 285-294. Association for Computational Linguistics, 2020.
[35] Joongbo Shin, Yoonhyung Lee, Seunghyun Yoon, and Kyomin Jung, Fast and accurate deep bidirectional language representations for unsupervised learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 823-835. Association for Computational Linguistics, 2020.
[36] Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. Cloze-driven pretraining of self-attention networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-JJCNLP 2019, Hong Kong. China, November 3-7, 2019, pages 5359-53300 5368. Association for Computational Linguistics, 2019.
[37] Sumanta Bhattacharyya, Amirnohammad Rooshenas, Subhajit Naskar, Simeng Sun, Mohit Iyyer, and Andrew McCallum. evergy-baseee reranking: Improving neural machine translation using energy-based models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1thh International Joint Conference on Natural Language Processing, ACI/IJCNI.P 2021, ( (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4528-4537. Association for Computational Linguisties, 2021.
[38] Ann Lee, Michael Auli, and Marc' Aurelio Ranzato. Discriminative reranking for neural machine translation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume 1: Long Pupers), Virtual Event, August 1-6, 2021, 1, pages 7250- 7264. Association for Computational Linguistics, 2021.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998-6008, Long Beach Convention Center, Long Beach, 2017. Curran Associates, Inc.
[40] Weizhen Qi, Yu Yan, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. Prophetnet: Predicting future n-gram for Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 2401-2410. Association for Computational Linguistics, 2020.
pre-training. In
[41] Dongling Xiao, Han Zhang, Yu-Kun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE- GEN: an enhanced multi-flow pre-training and fine-tuning framework for natural language generation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 3997-4003. ijcai.org. 2020.
13

---

[42] Jungo Kasai, James Cross, Marjan Ghazvininejad, and Jiatao Gu. Non-autoregressive machine translation with disentangled context transformer. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 51 41555.55. PMIR, 2020.
[43] Matthew E. Peters, Mark Neumann, Mohit lyyer. Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume I (Long Papers), pages
Association for Computational Linguistics,
2018.
[44] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory, Neural Comput., 98)):1335- 1780, 1997.
[45] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[46] Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th IWSLT evaluation campaign. In Proceedings of the / 11th International Workshop on Spoken Language Translation: Evaluation Campaign, pages 2 17, Lake Tahoe, California, dec 2014.
[47] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311-318. ACL,
2002.
[48] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015, pages
IEEE, 2015.
[49] Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmentee transformer for speech recognition. In INTERSPEECH, 2020.
[50] Daniel S Park, William Chan, Yu Zhang, Chung-Chern Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. In INTERSPEECH, 2019.
[51] Yichong Leng, Xu Tan, Linchen Zhu, Jin Xu, Renqian Luo, Linquan Liu, Tao Qin, Xiangyang Li, Edward Lin, and Yan Liu. Fastcorrect: Fast error correction with edit alignment for automatic speech recognition. Advances in Neural Information Processing Systems, 34, 2021.
[52] Yichong Leng. Xu Tan, Rui Wang, Linchen Zhu, Jin Xu, Wenjie Liu, Linquan Liu, Xiang-Yan -Yng Li, Tao Qin, Edward Lin, et al. Fastcorrect 2: Fast error correction on multiple candidates for automatic speech recognition. In Findings of the Association for Computational Linguistics: 2021.
EMNIP 2021, pages
[53] Kaitao Song, Xu Tan, and Jianfeng Lu. Neural machine translation with error correction. In Proceedings of the Twenty-Ning International Conference on International Joint Conferences on Artificial Intelligence, pages 3891-3897, 2021.
[54] Yichong Leng, Xu Tan, Sheng Zhao, Frank Soong, Xiang Yang Li, and Tao Qin. MBNet: MOS prediction for synthesized speech with mean- bias network. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 391 1-395, IEEE, 2021.
[55] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1-31-, oct 2013.
14

---

[56] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. CoRR, abs/2107.13586, 2021.
15


Reviews:
[{"review_id": "uDpp4-pxXgD", "reviewer": "Reviewer_UGD7", "summary": "This paper proposes a transformer architectural variant. The motivation of this variant is that it contains a sliding language modeling objective for sentence scoring, that enables it to look at all the tokens in a sentence using bidirectional context in a single pass. This solves the issue of using standard causal language modeling or masked language models, which respectively have limitations of using only unidirectional context and requiring multiple forward passes. ", "questions": "- The gains from SLM in the experiments required tuning a hyperparameter on a dev set and selecting the best value to use on the test set. Was the same hyperparameter value usable across all datasets? In what hyperparameter range was the model performing better than the baselines? What happens if we don't tune this hyperparameter?\n- Why was NMT and ASR chosen instead of general language understanding, which how LMs such as BERT and GPT are mostly used today? If transcormer does better in those settings, then the impact of this work will be much greater. For instance, sentiment analysis can be formulated as scoring two sentences: \"the sentiment of <review> is positive\" and \"the sentiment of <review> is negative\". This is called \"rank classification\", and this is how GPT-3 is evaluated. While it is not necessary to re-train a GPT-3 size model, even on the small scale some experiments on language understanding tasks like these could improve the experimental validation of the paper immensely.\n- I might have missed it but will the code be made publicly available to reproduce the experiments? This might be important to gain adoption for this method.\n- Since there is a high-cost of re-training from scratch, is it possible for existing language models to be adapted for this new objective? This might enable more compute-efficient experiments. ", "limitations": "The authors were up-front about the additional computational cost of their method, though additional experiments would be needed for the empirical gains in compute-matched settings to be convincing for me.\n\nThe societal impact part seems fine.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The structure of the paper is well-motivated. It is clear what the limitations of causal LM and MLM are for sentence scoring, and the paper proposes an architecture that addresses those two issues. \n- The writing of the paper is quite clear. There is a natural progression from motivation to method to results, and overall the paper is pretty understandable.\n- The significance of the paper is very high *in principle*. As causal LM and MLM are the standard objectives that are very commonly used in NLP, a new paradigm that outperforms these two types of objectives would be very impactful.\n- The fact that SLM needs 3 times the compute of similar sized models is a limitation, but the authors explicitly explored this Section 5.\n\n## Weaknesses\nThe promise of a better sentence scorer is potentially very impactful. The authors chose to evaluate sentence re-ranking on multiple generated candidates for NMT and for ASR. I have several concerns about these experiments that made them not particularly convincing for me. \n- First, the experiments were, in my view, are a bit thin. For instance, the gain from SLM over BERT is not very large, and there is still a large gap between the oracle. Furthermore, while several baselines were provided, there have been many architectures recently proposed, and probably more comparisons are needed to fully justify the method. As some examples, it could have been good to compare with T5, BART, XLNet, etc.\n- I am also not sure about the setup for the ablations. Why were the ablations not performed on all the datasets? I think this could have been addressed by putting a final row in Table 2, with SLM with one-third of the compute so that the compute is matched. It is fine if it does not win in all settings under the same compute, but in this case it would be good to be transparent about how much compute is needed to get the same performance, and whether there are performance gains in multiple different compute-matched settings.\n- Finally, I am not sure why NMT and ASR were chosen as the experimental settings in the first place. It could have been stronger if the paper had chosen a language understanding benchmark like GLUE, which could be a better fit for pretrained language models.\n\nI have voted for rejection primarily because the experiments are not particularly convincing, but if enough of the questions/concerns that I had are addressed in the rebuttal I will be happy to change my score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qjs5QCoba1v", "reviewer": "Reviewer_ayVD", "summary": "This work proposes a modified Transformer encoder, the Transcormer, which gives per-position log (pseudo-)probabilities in a single pass (unlike masked LMs) while still capturing token-token dependencies at all layers. These probabilities sum to give a score that can be used to rescore NMT and ASR hypotheses. This works by having the key-value representations be concatenated from a forward (inclusive) and backward (inclusive) attention stream before taking the outer product with the query, which never sees the content of its own position. Experiments on IWSLT, WMT, and LibriSpeech suggest this scheme outperforms MLM and CLM for rescoring.\n\n**Post-rebuttal: I raise soundness to 4/4 and my score from 4/10 to 6/10.** The work is thorough and introduces an architectural innovation to more richly model $P(y_s|y_{\\backslash s})$ in one pass. Unlike the paper originally claimed, it is not the first such model (T-TA and Electric); also, unsupervised sentence scoring remains niche (MLM scoring is more for inspection of pre-existing models, and for downstream tasks most works use discriminative ranking). However, the authors have done many experiments, it performs better than MLMs and previous models, and may have applications elsewhere (analogous to XLNet, also published at NeurIPS). There are still many grammar errors; please work with a copyeditor or native English speaker on any camera-ready.", "questions": "\n- Could you compare and contrast your approach with other one-pass models (T-TA, Electric)?\n\n- Given only the sum of probabilities is ever used and since one must pretrain anyways, have you compared with the stochastic estimation or distillation methods (L96-L97)? Alternately, do you have an application where token-wise probabilities are essential?\n\n- Will you release your pretrained SLMs?\n\n- L199-200 (SLM for scoring, CLM for NLG, MLM for NLU) is not really explained -- are there experiments for this? esp. since MLM and SLM model the same quantity.\n\n- Have you considered ways of matching MLM and SLM costs that avoid MLM pretraining mismatch (and how do you formally define cost), e.g., a shallower MLM, lower attention dimensions? Alternately, what if the MLM were pretrained with masked sets?\n", "limitations": "While SLM gives token-wise probabilities out of the box, it does require pretraining from scratch and is rather specialized (whereas MLM and CLM have other uses). Hopefully more uses can be explored to justify this cost; see works above for examples, like acceptability (BLiMP) or text similarity (STS).", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors succeed in proposing a scheme that models token probabilities conditioned on all other tokens (Eq. 1), in a single pass, while preserving \"deep bidirectionality\". The method appears sound, and while the mechanism is complicated, Figures 1 and 2 do well to explain it (though Figure 2b should have labels saying which axis is layer $l-1$ and which is layer $l$).\n\nHowever, I have concerns re: the clarity and significance of this work.\n\nFirstly, there are two uncited works that have proposed one-pass solutions to token-wise probabilities:\n- T-TA: J Shin, Y Lee, S Yoon, K Jung, “Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning”, ACL 2020\n- Electric: K Clark, M-T Luong, Q Le, C Manning, \"Pre-Training Transformers as Energy-Based Cloze Models\", EMNLP 2020\n\nIn particular, the first work's proposed T-TA has many similarities:\n- a query initialized with position embeddings only\n- a query that is updated through to the end and is selectively blocked on the self position\n\nwhile having some benefits (no 3x in cost?) and disadvantages (the keys, values do not transform over layers).\n\nThe authors should also acknowledge [18] in section 5.3, where an almost identical analysis is performed, even down to the choice of length and plot (Figure 3 here is largely [18]'s Figures 3+4).\n\nWhile the authors compare on some tasks (NMT, ASR) as prior work, it would be good to compare on the same datasets or even n-best lists. For example, [18], [20], and the above two works compare on a shared set of n-best LibriSpeech lists while this work creates new ones. This is mitigated by the authors releasing model code (thank you!), but would help convince readers -- especially as SLM models are not off-the-shelf but trained from scratch (and hopefully released).\n\nIt is also unclear e.g., what L20's \"quality of a sentence\" or what L23-24's \"precise sentence scoring\" mean. Only in Section 2.1 (page 3) do we get two definitions:\n- via CLM, which by the chain rule gives the log-likelihood score $\\log P(y_1, ..., y_{|y|})$\n- via MLM, which gives the pseudo-log-likelihood score $\\sum_{i=1}^{|y|} \\log P(y_i \\mid y_{\\backslash y_i})$\n\nSLM models the latter. It would be better to make this clear early, and *then* mention how/why these quantities are empirically (or even theoretically?) good quantities to rescore with (so before in 26-38), with the second being better (which is why(?) SLM models it). Discriminative rescoring should also be discussed/contrasted early, or at least in the main text (not the appendix).\n\nI appreciate some of the analyses which complement past work, in particular A.3.1's discussion comparing sentence vs. fixed-length pre-training, and 5.3 showing SLM and MLM's similar scores. Section 5.1 and 5.2 are unsatisfying though, as increasing the number of masked tokens would be expected to be poor given the mismatch with pretraining (in BERT, masking 15%).\n\nMinor notes:\n- The word \"sliding\" evokes sliding windows (i.e., only a contiguous, fixed length of inputs are considered at any time), giving the wrong impression\n- L195-198: The structure of bi-LMs should be explained more\n- L163: should clarify that $p_i$ refer to position _embeddings_\n\nWriting should be proofread, e.g.,\n- L18 and elsewhere: \"language modelings\" --> \"language modeling approaches\" or \"language modeling schemes\" or even \"language models\"\n- L85: \"in a chain-style rule\" --> \"via the chain rule\"\n- Figure 2: \"reuse\" --> \"reuses\"; \"can not\" --> \"cannot\"\n- Table 5, Appendix Table 5: \"test-clear\" --> \"test-clean\"\n- L233, 236, Appendix Table 5: \"LibrSpeech\" --> \"LibriSpeech\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RpVLlXV3cjP", "reviewer": "Reviewer_2vXa", "summary": "This work presents a Transformer-based model for sentence scoring, which aims to overcome the limitations of GPT-style causal language model pretraining (only the unidirectional context is used) and BERT-style masked language model pretraining (multiple forward passes are required when scores for each token are needed). The authors propose an XLNet-style \"multi-stream\" self-attention, which employs a forward stream (to capture the LTR context), a backward stream (for the RTL context), and a query stream which captures information from both.\n\nThis method is experimentally validated on two tasks, neural machine translation and reranking for ASR. The proposed method generally outperforms CLM and MLM approaches. The authors also perform further experiments showing that the proposed method is still able to outperform CLM when the size of the models is adjusted such that the amount of inference-time computation is equal; finally, they also attempt to make MLM more efficient by masking multiple tokens at a time (thereby reducing the amount of required computation), but the proposed method still comes out ahead.", "questions": "Could you clarify what the \"oracle\" line in Tabels 2 and 3 represents exactly?", "limitations": "I do not see any potential negative societal impacts of this work. Limitations are sufficiently addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main idea seems like fairly straightforward variant of the XLNet architecture, not too dissimilar from other past proposals to modify its base architecture by changing the nature of the \"streams\".\n\nAside from originality, overall the paper feels solid. Both the architecture and the experiments are clearly described. The tasks chosen by the authors for the evaluation are realistic, and at least for the NMT experiments the baselines are sensible and their performance is in the right ballpark.\n\nGiven that one of the selling points of the proposed method is computational efficiency, the experiments of sections 5.1 and 5.2 are useful. The results are promising, showing that under comparable amounts of computation the authors' proposal still comes ahead.\n\nA couple of minor points:\n* It would be useful to have dataset and split sizes for the experiments of section 4 directly in the paper.\n* IWSLT14 might be a small dataset, but I don't think German, Spanish, Italian, etc. should be described as \"low-resource\". ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept