Meet in the Middle: A New Pre-training Paradigm
Authors: Anh Tuan Nguyen
Nikos Karampatziakis
Weizhu Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: language modeling
pre-training
deep learning
NLP
TL;DR: We propose a new pre-training objective for autoregressive LMs and show that models trained this way are better in perplexity and many other metrics

Abstract: Most language models (LMs) are trained and applied in an autoregressive left-to-right fashion, predicting the next token from the preceding ones. However, this ignores that the full sequence is available during training. 
In this paper, we introduce ``Meet in the Middle'' (MIM) a new pre-training paradigm that improves data 
efficiency by training in two directions, left-to-right and right-to-left, and encouraging the respective models
to agree on their token distribution for each position. While the primary outcome is an improved left-to-right LM,
we also obtain secondary benefits in the infilling task. There, we leverage the two pre-trained directions to propose an infilling procedure that builds the completion simultaneously from both sides. We conduct extensive experiments on both programming and natural languages and show that MIM significantly surpasses existing pre-training paradigms, in both left-to-right generation as well as infilling.
Code and models available at https://github.com/microsoft/Meet-in-the-Middle

Meet in the Middle: A New Pre-training Paradigm
Anh Nguyen" Nikos Karampatziakis"
Weizhu Chen
Microsoft Azure AI
Abstract
Most language models (LMs) are trained and applied in an autoregressive left- to-right fashion, predicting the next token from the preceding ones. However, this ignores that the full sequence is available during training. In this paper, we introduce "Meet in the Middle" (MIM) new pre-training paradigm that improves data efficiency by training in two directions, left-to-right and right-to-left, and encouraging the respective models to agree on their token distribution for each po- sition. While the primary outcome is an improved left-to-right LM, we also obtain secondary benefits in the infilling task. There, we leverage the two pre- pre-trained di- rections to propose an infilling procedure that builds the completion simultaneously from both sides. We conduct extensive experiments on both programming and natural languages and show that MIM significantly surpasses existing pre-training paradigms, in left-to-right generation as well as infilling.
Introduction
Language models (LMs) are becoming essential for various assisted authoring tasks, including text summarization and code completion. In order to be usable in many different applications, most LMs must be able to generate the next token from the sequence of previous tokens. Thus, pre-training has focused optimizing the model's ability to predict the next token given the previous tokens (the prefix), as measured by perplexity. However, this neglects that at pre-traue pre-training time we also have access the subsequent tokens (the suffix). While the suffix cannot be used as an input the model, there are other ways to incorporate it into pre-training which have not received much attention in the literature. In this work we propose a more data-efficient approach, preserving the autoregressive nature of LMs while fully utilizing pre-training data.
Why should we be interested in alternative pre-training procedures? After all, the main artifact produced during pre-training is an autoregressive left-to-right LM and the pre-training objective closely matches both how the LM applied and the desire predict the next token well. Nonetheless, there are still compelling reasons to consider alternative training objectives. First, data efficiency: current LMs are trained with sparse supervision, only learning from the actual next token. By exploring denser supervision during training, we aim to improve performance with less data. Second, related tasks: in real-world scenarios, users may need infill or modify existing text. Current left-to-right LMs are limited in this regard, as they cannot use both sides of the insertion position, often leading to suboptimal results.
In this work, we introduce a novel pre-training paradigm named "Meet in the Middle" (MIM). The MIM approach is founded on central principles. Primarily, it introduces an additional right-to-left LM to complement the standard left-to-right model, enabling both models to co-regularize and benefit from each other's context. This improves data efficiency and consistency, with models metaphorically meeting in the middle by adjusting their output probabilities to agree with one another. Secondarily,
Equual Contribution
"Code and models available
the-Middleeeee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

MIM enables an efficient infilling inference procedure that capitalizes on the agreement between the two LMs developed during the pre-training phase. For infilling, the models literally meet in the middle, constructing the completion from both ends. This aspect of MIM, while secondary to the pre-training objective, offers crucial benefits. Co-regularization not only reinforces consistency between the two LMs but also aids in early termination of the generation process during infilling.
To train MIM, we employ two decoding flows within single shared decoder-only architec- ture [BMR+20 (CND The forward and backward LMs generate tokens in opposite directions, predicting the next token and previous token, respectively. We pre-train both models jointly on large text corpus using combination of standard LM loss and an agreement regularizer. Upon completion, the forward model serves as drop-in replacement for existing autoregressive LMs, while the backward model can be discarded or utilized for tasks such as infilling.
In our experiments, we assess MIM's effectiveness for pre-training LMs across various domains and tasks. Using public code and language data, we pre-train LMs of different sizes and evaluate their performance based on perplexity and code completion tasks. We compare MIM with FIM [2TT*221 and other baselines, demonstrating its superior performance in terms of perplexity and task-specific evaluation metries. Furthermore, we conduct ablation studies to validate the effectiveness of our primary proposals during training and inference. In summary, our main contributions include:
Introducing a novel pre-training paradigm for LMs that efficiently utilizes training data by incorporating both prefix and suffix context while retaining the autoregressive nature of LMs. This achieved by training forward and backward models and promoting their agreement.
Proposing simple, efficient infilling inference procedure that leverages context from both sides and the tendency of forward and backward models agree. Our approach employs parallelism more effectively than existing procedures, resulting in superior quality and latency compared to the state of the art.
Pre-training LMs of various sizes on public code and language data using MIM, evaluating their performance on both human and programming languages, and demonstrating that MIM outperforms numerous baselines in standard evaluation metrics. Additionally, we make some models and code publicly available.
2 Preliminaries
We introduce notation used throughout the paper. For token sequence I1, F1,,42.... .2.2...,,NN. we denote T<i as the prefix æ1, 22. 24-1 and æ>; as the suffix T4+1, Z442, ... TN. Definitions for zgi and T2i are analogous. We suppress model dependencies on learnable parameters. Arrows distinguish the two models and their outputs; for example, h represents the forward model and P the backward model, while H and H are hidden representations from the respective models.
2.1 The Infilling task
Given a token sequence 3i 22, ZN, an insertion position i, the infilling task involves generating plausible M-token sequence ,i,..., Wir to fill the gap between rei and -zi-1- In real-world applications, M is unknown. The infilling task is harder than the left-to-right generation task as the latter is special case of infilling with empty suffix. Since generating the best M-token left-to-right generation is a search problem that is tackled by heuristics such as greedy search or beam search, we will also follow heuristic approaches for the strictly harder infilling problem.
technique for infilling that utilizes context from both sides is "Fill in the Middle" (FIM) [BJT+22
FIM forms the context by concatenating the suffix and prefix (in that order), ensuring coherence near the completion point. FIM easily applied to pre-trained LMs with minimal modification and computationally efficient. However, FIM's 's drawbacks include training on unnatural concatenations of prefix and suffix, giving the prefix more influence on the completion over the suffix, and an ad-hoc training procedure that only considers a few of the ((N2) possible splits in an N-token document.

---

2.2 Bidirectional Language Modeling
Bidirectional language modeling has been used in the literature two different ways: The first is to describe non-autoreereessive LMs with objectives like Masked Language Modeling, which produce better representations but face difficulties in-context learning PLR 22).
Earlier work [HR18] PNI 18 used the term bidirectional language modeling to describe models with left-to-right and right-to-lett autoregressive flows. Our approach falls in this category, remaining autoregressive and using future tokens only for regularization during training.
3 Meet in the Middle
Our proposal involves pre-training two models to predict their next token from different imput views (prefix vs. suffix), while maintaining agreement on the probability distributions for these tokens. This provides dense supervision signal compared simply predicting the next token. also encourages the models to "meet in the middle" i.e. reach a compromise prediction informed by different views.
3.1 Pre-training
We train two decoder-only LMs, forward model and backward model T, with shared pa- rameters. d' predicts next tokens in the forward direction, while 5 predicts previous tokens in the backward direction. To improve data efficiency, we use a co-regularivation term that encourages the models to agree the predicted probability distribution over the vocabulary for each token. We use total variation distance to measure disagreement between the models. Define the disagreement on I;
DT (>11))= = Σ - - zEV
where is the vocabulary. The agreement regularizer is the sum of these terms over all sentences and tokens. provides the models with denser supervision signal which improves data efficiency and helps train better autoregressive LM. This is because two probability distributions over all tokens are compared while in the traditional LM objective only the probability of the observed token matters. Moreover it encourages the models agree their predictions which helps with the efficiency of infilling as we will see later. Then, the full training loss for dataset S of sequences is given by:
log (7 (I:((2i) - log (5 (2,(25))) 3 DTY (7115).
(I)
FES
The hyperparameter 0.1 our experiments. After pre-training, d can be used
left-to-right autoregressive LMM
3.2 Infilling
3.2.1 Inference
Our inference procedure for infilling aims to be efficient and low-latency. Our approach is shown in Figure [ At a high level, the two models start building a completion each from their own side and they try to literally meet in the middle. Initially, the prefix and suffix are consumed by 7 and y respectively. Then p and 'p generate tokens synchronously one at a time. For each generated token from 7 (resp. p check whether it matches generated token from y (resp. from F¹). If there is a match, we have a "meet in the middle" candidate position for joining the two generated sequences.
In an ideal scenario, F and F generate identical sequences, albeit in reverse order, resulting in "meeting the middle" after half of the tokens are generated by each model. The agreement regularizer is thus crucial: if P and "p yield disparate sequences, our method's speed is inferior to FIM since early termination isn't possible. However, if there's complete agreement, with sufficient parallelism, our method can outperform FIM in generation latency, as each model only needs to generate half the completion autoregressively. Our experiments demonstrate our method's superior latency to FIM, indicating that in most instances, sequences meet close to the middle.

---

Prefix
Suffix
The
quick
dog
MIM
Verification
brown
fox
jumped
over
P(x+)
4 (11))
jumps
over
the
lazy
Figure 1: Inference procedure infilling. Given the prefix "The quick" and the suffix "dog", the models P and 5 generate tokens until candidate peee-in-the-midee token is detected (shaded area). use single token for illustration purposes, although the method can use more tokens. Given the candidate MIM token, the T (respectively P) model can in parallel verify that the tokens generated by y (resp. 7) are acceptable completions (only the F verification is shown to reduce clutter). The numbers in the top right of each box show the order of operations. Two boxes with the same number can be executed in parallel. Similarly, thick lines show that information (embeddings, tokens) can flow in parallel, while thin lines denote sequential steps.
However, the procedure may yield false positives. For instance, matching a token such as "the" from both sides may not guarantee coherent infilling upon joining. To mitigate this, we employ 4-gram matching in our infilling experiments. Additionally, we run a parallel verification procedure to ensure that the resultant sequence could have been generated by either d or running to completion.
Our parallel verification procedure is adapted from 22 Concretely, assume that the latest token generated from matches with token in position generated from P. is the case in Figure for the token "over". Then we can use the tokens that were generated by P before s as inputs to p in parallel. In the context of Figure 1 we provide the tokens "over", "the", "lazy" as inputs to 7 in parallel. each output of Φ matches the corresponding input, we have verified that p would have autoregressively generated the tokens it copied from p 's output. Grounding the discussion back to Figure p' generates (in parallel) "the" and "lazy" we have verified that y would have match, the generation from
autoregressively generated the same tokens. In cases of partial either fort-dorwards to the first disagreement point or resumes autoregressive generation from d and T. While our implementation position, we could adopt less stringent acceptance criteria, like a sufficiently high probability of the candidate token in the previous output. If 7) and y finish generating without meeting passing verification, we return the higher-pron higher-prohabillty sequence according to its generating model.
greedy top-1 sampling and comparison of input and output each
3.2.2 Infilling Specific Enhancements
Our model's infilling performance can be boosted by adopting the Synchronous Bidirectional Atten- tion mechanism 77Z19]. Though this reduces compatibility with autoregressive language models (LMs), it enhances generation through bidirectional conditioning. The Synchronous Bidirectional Attention layer modifies the regular attention layer activations as follows: Let H and स be the outputs of an attention layer in p and T respectively. For a hyperparameter X, the fused attention hidden representation becomes H
H IY and H H + AH. Instead of H and H, the models 7 and fp use H, and H7 f- If X = 0, 7 and y become classic autoregressive transformers. However, careful training is necessary avoid information leakage with this layer. mitigate this, adopt two-stage training process, [ZZZ19] for details.

---

Experiments
4.1 Data and Models
Our code models are pre-trained on a large and diverse corpus of public code with permissive licenses, covering multiple programming languages. The dominant languages in corpus are Python, Java, and C++. After filtering and deduplication, the corpus contains about 300B tokens. Details are Our models train on single pass over this data. This is about six times larger than the dataset used to train the Incoder model [FAL 22), which included 50B tokens comprising code and Stack Overflow data.
provided in Table 9
We train our natural language models on data from the following datasets: CC-News, OpenWeb WebText, CC-Stories, and CC-100 to assess the models language modeling capability, In particular:
CC-News: 63M English news articles crawled from Sept 2016 to Feb 2019 (76GB).
An open source recreation of the GPT-2 WebText dataset (38GB).
CC-Stories: subset of CommonCrawl data filtered to match the story-like style of Winograd schemas (31GB).
CC-100: A dataset extracted from CommonCrawl snapshots between Jan 2018 and Dec 2018, filtered to match the style of Wikipedia (292GB).
Apart from the natural language data utilized in our language modeling experiments, we also train our models on filtered version of the Falcon RefinedWeb corpus [PMH 23] used in the previous work of [LBE+ 23), that has a total 88B tokens. We then present the performance results on common
natural language benchmarks in section 4.3
We use a decoder-only transformer LM VSP+ 17
BMR 20) for pre-training. The model trained to predict the next token in both left-to-right and right-to-left directions, using the same model and parameters. To explore the impact of model size, we pre-train models with different capacities: 350M, 1.3B, 2.7B. For the hyperparameters and training setup of each model size, please refer to the Appendix.
As baselines, we use well-known published results but also pre-train three FIM models with context- level FIM [BJ 22]. Character-level transformations are applied, with a FIM-rate of 0.5 and SPM+PSM joint training. Both the MIM and FIM models are pre-trained using the Megatron-LM framework (SP 19]. See the Appendix for model configurations and architectural details.
4.2
Benchmarks and metrics
4.2.1 Code generation and infilling
We assess MIM's performance both in autoregressive left-to-right generation and infilling tasks. For the autoregressive generation, the model generates the code body from a function signature, docstring. and test cases. Evaluation datasets include HumanEval [CT]+21 21J. MBPP [AON 21], and APPS [HBK-21] for Python problems. We use the multilingual HumanEval-X dataset for multiple programming languages. For this task, the left-to-right model alone is used for inference without any provided suffix.
In the infilling task. the model fills blank lines in incomplete program. This is evaluated using the HumanEval Infilling [2JT-22] and MBXP AAW= datasets. MIM is compared with Incoder, three FIM variants we trained the same data MIM, and code-davinci-002, as these datasets provide suffixes that not all models can manage effectively.
We use the pass @ k metrics [CT]*21 to measure the percentage of generated codes that pass all tests within the top-k candidates. We report pass@1 I, pass 10, and pass@ 100. For generation, top-p sampling with p 0.95 is employed with temperatures of 0.2 for pass I and 0.8 for pass@ 10 and @ 100. We also include single-line exact match (EM) metrics to indicate the percentage of completed lines matching the reference solution's 's masked lines (22
5

---

4.2.2 Language Modeling
In addition to code generation and infilling, we evaluate MIM's language modeling capabilities, particularly its next-token prediction ability measured by perplexity. Perplexity evaluated both in-domain and out-of-domain settings. The former involves a held-out subset of the combined training data, while the latter uses the Pile dataset |BB 22],. a multi-domain public language modeling dataset. Average perplexity across all Pile subsets is reported for our models and baselines.
4.2.3 Natural Language Benchmarks
Finally, we evaluate MIM on established benchmarks for natural language, encompassing both commonsense reasoning and language understanding. For commonsense reasoning, we have tested our models on Winogrande, ARC-Easy, ARC-Challenge, BoolQ, and SIQA. In terms of knowledge and language understanding, we have assessed MIM's performance on PIQA, HellaSwag, and OpenbookQA. We compare against phi-1.5-webonll [LBE*23] and present zero-shot results for these tasks using the LM-Eval Harness, with the exception of MMLU where we use 2-shot performance.
4.3
Main results
4.3.1
Code generation and infilling
First we focus on the superior performance of our from-scratch pre-trained FIM model over various baselines, in Table For instance, our 2.7B model yields a 28.5% pass@ in HumanEval, signifi- cantly outperforming the 15.2% pass @1 of Incoder's 6.7B model [JT FIM-2.7B also beats other strong baselines such as Codex 2.5B and ModeGen-Multti IB, achieving 67.8% in pass 100 compared their respective 44.9% and 59.5%. This performance is attributable to larger, cleaner training data and implementation details (discussed the Appendix).
Having established a strong baseline, we now benchmark MIM's autoregressive generation in a prefix-only setting using both the HumanEval and MBPP datasets. While FIM vs. MIM is the apples- to-apples comparison, we include many other models for reference and note that all these could have leveraged our MIM objective during their training. Table T shows MIM consistently outperforming FIM across all metrics for all model sizes (350M, 1.3B, 2.7B). For example, MIM-2.77 7B enhances HumanEval pass @1 to 30.7%, 2.2% gain over FIM-2.7B. The improvement on MBPP is similar, in terms of pass @ 1. We note that the improvements increase with the model size. This is evidence that the larger models benefit more from the agreement regularizer. We further evaluate this setting on APPS dataset, which has three different difficulty levels. We report the results in Table 3 The trend and improvement are similar to the other two datasets, in which MIM consistently outperforms FIM across all the metrics and difficulty levels. In the multilingual setting, we compare MIM with FIM baselines in the HumanEval-X dataset. We observe consistent improvement across all metrics and all the programming languages that we evaluated on. Results are reported in Table 2
These findings illustrate MIM's s superiority over FIM for left-to-right autoregressive generation. The main claim in 22) is that FIM does not harm the original left-to-right generative capability and can be learned for free. We contend that MIM's pre-training, enhanced by the agreement regularizer, results in superior generative model, thereby proposing new pre-training paradigm.
MIM in infilling setting, motivated by real-world applications like GitHub Copilot, where developers frequently edit in the middle of source files. MIM outperforms FIM across all metrics and model sizes on both the HumanEval Infilling and the MBXP Infilling benchmarks (Table D. Notably, the MIM-2.7H 7B model achieves pass @ of 26.3% and an exact match metric of 57.8%, improvements of 3.5% and 6.1% respectively, over the FIM-2.7B 7B model. This underscores the consistent improvement that MIM offers across different model sizes and tasks.
Finally,
4.3.2 Language Modeling
Here we compare FIM and MIM models pre-trained on language modeling tasks. We use perplexity different settings, namely in-domain and out-of-domain. our evaluation metric and look
The perplexity results of both settings are summarized in Table Across all experiments, MIM consistently outperform FIM baselines in terms of perplexity. The MIM-2.7B model has the best

---

Methods
HumanEval
MBPP
HE Infilling
MBXP
10
100
10
100
EM
Incoder-1.33 1.3B
8.9
16.7
25.6
11.3
26.8
42.7
8.6
31
9.2
Incoder-6.7B
15.2
27.8
47
19.4
46.5
66.2
14.5
44.1
20.8
CodeGen-Multi Munnn--uBtBBBBBB
18.2
28.7
44.9
CodeGen-Multi ButGGen-Mllti16BB
18.32
32.07
50.80
Codex-2.5B
21.36
35.42
59.5
Codex-12B
28.81
46.81
72.31
code-davinci-00)
39
60.6
84.1
51.8
72.8
84.1
code-davinci-002
47
74.9
94.1
58.1
76.7
84.5
51.3
57.6
Ma-B8
3.6
18.7
5.0
Ma-22B
15.9
46.3
21.4
PaLM-540B
26.2
76.2
36.8
LLaMA-7B
10.5
36.5
17.7
LLaMA-13B
15.8
52.5
22.0
LLaMA-33B
21.7
70.7
30.2
LLaMA-65B
23.7
79.3
37.7
FIM-350M
12.8
16.7 7
27.8
14.8
30.2
44.5
11.8
37
14.3
FIM-1.3B
20.8
39.4
51.7
25.9
45.6
62.5
15.7
42.2
22.1
FIM-2.7B
28.5
45.6
67.8
38.2
61.2
76.1
22.8
51.7
30.4
MIM-350M
13.7
17.2
28.5
16.5
33.7
47.4
41.7
16.4
MIM-1.3B
22.4
41.7
53.8
26.8
47.6
65.1
17.4
47.6
24.5
MIM-2.7B
30.7
48.2
69.6
42.2
64.8
79.3
26.3
57.8
35.7
Table 1: pass @ @ (%) on the HumanEval, MBPP and HumanEval Infilling and MBXP benchmarks.
Additionally, Exact Match (EM) metric [FAL+22 is reported for HumanEval Infilling. FIM the baseline Fill in the Middle" (FIM). MIM is proposed method with the enhancements from section 3.2.2 For reference, we report evaluation numbers other models, namely Incoder-1.3B and Incoder-6.77 6.7B [FAL*22], Codex- 12B [CTJ*21), 1ode-davinci-001 and 002. Note that, Codex- 12B, davinci-001, PaLM (CND*22 and LLaMA [TLI*23 were trained only with left-to-right autoregressive objective, thus, cannot perform infilling.
Methods
C++
Java
Go
1
10
100
100
10
100
Incoder-6B
10.0
20.0
35.0
9.0
19.0
40.0
8.0
14.0
29.0
CodeGen-6B
12.0
20.0
36.0
15.0
18.0
40.0
9.0
22.0
40.0
CodeGen- 168
18.0
30.0
50.0
15.0
38.0
13.0
25.0
47.0
CodeGeeX 13B
20.0
31.0
50.0
16.0
38.0
58.0
15.0
25.0
49.0
FIM-350M
8.5
18.3
24.3
11.3
21.5
27.6
10.2
16.8
31.4
FIM-1.3B
16.7
31.5
43.2
15.4
25.2
32.6
12.5
23.7
39.6
FIM-2.7B
24.5
38.6
51.3
18.3
28.6
38.7
15.2
30.4
50.2
MIM-350M
10.2
19.6
26.3
11.1
22.4
28.2
10.8
17.5
31.8
MIM-1.3B
19.3
36.5
45.7
17.6
27.4
34.7
13.6
25.1
41.7
MIM-2.7B
27.4
41.3
54.1
21.6
32.7
53.6
Table 2: pass@k (%) results HumanEval-X in the zero-shot settings for the baselines FIM and
our MIM approach. Results of 1, 10 and 100 are reported across all categories.
perplexity across all datasets in both settings. For example, MIM-2.7B model obtains perplexity of 9.54 in Open WebText dataset, relative 19.9% reduction in perplexity over FIM-2.7B model, which obtains perplexity of 11.92. In the Pile dataset (GBB+21) which represents the out-of-domain setting, MIM-2.7B also outperforms FIM-2.7B by a relative reduction of 15.3% perplexity (9.24 vs 10.92), which further reinforces the advantages of MIM pre-training for natural languages.
4.3.3 Natural Language Benchmarks
We trained MIM model with total of 1.3B parameters, which shares architectural details with 23), on filtered version of the Falcon RefinedWeb corpus, comprising a total of 88B tokens. This model was subsequently compared with the baseline model, phi-ol--web-only in [LBE 23).
7

---

Methods
Introductory
Interview
Competition
k
100
10
100
10
100
FIM-350M
3.6
7.8
11.5
0.0
0.3
1.2
0.0
0.04
0.9
FIM-1.3B
8.2
11.6
17.4
0.12
0.59
1.9
0.01
0.07
1.7
FIM-2.7B
12.4
15.7
20.8
0.27
0.72
2.4
0.03
0.095
2.4
MIM-350M
4.7
9.2
14.3
0.2
0.51
2.3
0.02
0.06
1.4
MIM-1.3B
10.6
14.2
21.2
0.36
0.76
3.6
0.043
0.09
2.2
MIM-2.7B
14.3
18.2
24.6
0.52
1.4
5.2
0.067
0.18
3.3
Table 3: pass@k (%) results on the APPS benchmarks in the zero-shot settings for the baselines FIM
and our MIM approach. Results of k 1, 10 and 100 are reported across all categories.
Methods
Datasets
MODELS
CC-NEWS
OPENWEBTEXT
CC-STORIES
CC-100
THE PILE
FIM-350M
21.16
17.91
20.89
14.23
15.14
FIM-1.3B
18.78
13.28
17.52
11.34
12.48
FIM-2.7B
13.45
11.92
13.43
9.43
10.92
MIM-350M
19.43
17.23
18.75
13.45
13.97
MIM-1.3B
16.04
12.23
14.63
11.16
10.79
MIM-2.7B
11.17
9.54
11.35
8.76
9.24
Table 4: Perplexity on all datasets in in-domain setting (CC-News, WebTex,, CC-Stories, & CC-100 held-out datasets), and perplexity results in out-of-domain setting (The Pile (GBB+21)
which was trained on identical data using an autoregressive left-to-right objective. Tables and summarize the results. On commonsense reasoning MIM-1.3B-web-om outperforms by 0.42% on average while on knowledge understanding outperforms by 1.2% on average. Overall, (combining the two tables) MIM improves by 0.76% on average.
4.4 Ablation Study
4.4.1 Effect of Infilling Specific Enhancements
In this section, we perform an ablation study to qualitatively assess the effect the enhancements we proposed in section 3.2.2 for the infilling task. We compare the purely autoregressive MIM model (A (A=0) 0) where the LMS are not allowed to observe generated tokens from the opposite side, and only utilize context from their side during generation. We contrast this with using the Synchronous Bidirectional Attention layer with 0.3 that conditions on previously generated tokens from both sides. We used perplexity the validation data to select the value X. We reuse the same value of X during infilling to avoid any potential mismatch between training and inference.
We conduct an experiment HumanEval Infilling [2 and results are summarized in Table We notice that models that directly incorporate bidirectional context always outperform models that only utilize unidirectional context across all model sizes. As always, this is at the expense of the forward model no longer being a a drop-in replacement for standard autoregressive LMs.
4.4.2 Effect of Agreement Regularizer
While MIM performs better than FIM it's still possible that this difference is simply due to the parameters being trained on two tasks (left-to-right and right-to-left). To preclude this, we perform an ablation study where we remove the token-level agreement regularizer of Section [.1 during training while keeping the infilling enhancements (A 0.3). We show that encouraging agreement during training helps improve the infilling performance of our models in the HumanEval Infilling benchmark [2] as summarized in Table Comparing with Table 7 we see that models trained without token-level agreement regularization in general perform worse than models that do not utilize bidirectional context (A = 0), which further emphasizes the importance of our agreement regularizer in making the predictions consistent between forward and backward directions.

---

Methods
Datasets
MODELS
WINOGRANDE
ARC-EASY
ARC-CHALLENGE
BooLQ
SIQA
phi-1.5 hai-L.S-web-only
0.604
0.666
0.329
0.632
0.414
MIM-1.3B-BH -.B--bbboyy
0.591
0.687
0.337
0.628
0.423
Table Results of MIM in standard commonsense reasoning benchmarks, including WinoGrande,
Arc-Easy, Are- Are-Challenge, BoolQ and SIQA, vs, the baseline phi-1.5-web-only in [LBE 23)
Methods
Datasets
MODELS
PIQA
HELLASWAG
MMLU
OPENROOKQA
-hi-L-S-web-only
0.743
0.478
0.309
0.274
MIM-1.3B -Bnbb.ll-
0.736
0.491
0.327
0.298
Table 6: Results of MIM in standard knowledge understanding benchmarks, including PIQA,
HellaSwag, MMLU and OpenbookQA. vs. the baseline phi-1.5-web-only in [LBE 23).
Methods
HE Infilling
MBXP
Methods
HE Infilling
MBXP
MODELS
pass@1
EM
pass
MODELS
Regularizer
pass@1 I
EM
pass @
MIM-350M
0.0
12.5
38.6
14.1
MIM-350M
no reg
11.8
35.9
13.5
0.3
14.6
41.7
16.4
reg
14.6
41.7
16.4
MIM-1.3B
0.0
15.6
45.2
21.7
MIM-1.3B
no reg
13.8
42.7
22.7
0.3
17.4
47.6
24.5
reg
17.4
47.6
24.5
MIM-2.7B
0.0
24.7
54.3
32.4
MIM-2.7B
no reg
23.2
52.5
30.6
0.3
26.3
57.8
35.7
reg
26.3
57.8
35.7
Table 7: pass @1 (%) results of MIM with (A
Table 8: pass@1 (%) results of MIM with and
0.3) and without (A = 0.0) bidirectional context on HvaanEval Infilling and MBXP. Exact match results on HumanEval Infilling are also reported.
without agreement regularizer on HumanEval In-
filling and MBXP. Exact match results on Hu- manEval Infilling are also reported.
4.5 Efficiency
4.5.1 Training Efficiency
At first glance, one might hypothesize that our proposed methodology could potentially necessitate twice the computational resources in comparison to the conventional left-to-right autoregressive training, with most of the performance gains attributed to this additional computation. Contrary to this supposition, empirical evidence gathered from our experiments suggests different narrative. Our Meet-in-the-
(MIM) strategy exhibits performance metrics that are not only commensurate but often superior to those of the left-to-right autoregressive training. This assertion is corroborated by comparative analysis of the validation log perplexity MIM models and left-to-right autoregressive models across different model sizes. All models were trained on the filtered Falcon Ref corpus [LEE* 23]. As illustrated in Figures 3 and in the Appendix, when assessed under the same computational budget (either FLOP count wall clock time), MIM models consistently outperform left-to-fe to-rigght autoregressive models across relevant computational budgets. Besides, when comparing MIM models and left-to-right autoregressive models on a time- basis, still observe major speedups over the baselines.
4.5.2 Inference Efficiency
Finally we assess the efficiency of our infilling procedure, contrasting it with FIM through inference latency across various batch sizes. Figure shows that MIM achieves a speedup over FIM baselines in both single and half precision, using 1.3B model, across all HumanEval Infilling examples. MIM-1.3B specifically performs 4% to 6% faster than FIM-1.3B in single precision, and 3% to 5% faster in half precision. This is due to MIM leveraging available parallelism better than FIM.

---

Related work
There is an extensive body of work on bidirectional language modeling. Early work such ULMFIT [HR18) and ELMo [PNI+18) 18) kept the autoregressive nature of the LM, while later work such as BERT DCLTI9) T5 IRSR 20), and SpanBERT [JCL + 20] focused on representation learning and was -uoorerrreeg.ssivee.... XLNET LYDY+ 19), on the other hand, utilizes bidirectional context during training by the permutation language modeling objective, which maximizes the likelihood over all factorization orderings of the training sequences. However, because these later models typically focus on representation learning, in-context learning via prompting can be difficult [PLR 22).
Two works that train neural models using similar ideas are |SKS 18) and 19] In the former the authors train RNNs for real-time acoustic modeling. This constraint is very similar our desire of having a forward model that can generate the next token from the previous ones. However, they propose regularize the forward and backward RNNs by requiring their representations to be close in Euclidean distance. We suspect that this constraint may be unnecessarily stringent, and trading it off against the LM's perplexity during training could hurt overall performance.
On the other hand, 19 encourage agreement in probability space, However, they are interested only in neural machine translation and only encourage agreement of probabilities at the output sequence level. In contrast, we encourage agreement on every token.
Sharing all parameters between different factorizations of the sequence was first proposed in XLNET (YDY+ 19 but has also been used with only forward and backward models in the context of image captioning ZZHL+ These works motivated us to share all parameters between our two LMs.
Using LMs for infilling was first proposed in [DLL20] where the authors tackled the more challenging setup with multiple blanks. (BJT+22) applies "Fill in the Middle" (FIM) to the training data by randomly splitting each training instance into tuple of (prefix, middle, suffix) and concatenate these sections into single example together with their sentinel tokens after tokenization. [BJT [2] also contains thorough section on research related to infilling. We refer the reader there for more details.
Furthermore, [HH 222 and FAL propose an extension of FIM. namely "Causal Masked Language Modeling" (CM3) and explore multi-region infilling problem. Similar to us, (BIT+222 and 22 leverage FIM to pre-train decoder-only LMs on code data and evaluate their models on zero-shot code completion benchmark created from HumanEval dataset 2C]
In SLB17 the authors propose bidirectional beam search that also employs forward and backward LMs. However their focus is in improving accuracy at the expense of latency. Their procedure performs multiple passes over the completion, fixing tokens one-by-one. Our approach focuses on achieving better infilling accuracy than FIM while also reducing latency.
6 Conclusion
This paper presented "Meet-in-the dhedMid a novel method devised to address two significant challenges in large LMs: pre- pre-training data efficiency and effective context handling in infilling tasks. By using both forward and backward LMs that share parameters and are trained for mutual agreement and next token prediction, the method significantly enhances the performance of existing autoregressive LMs.
Acknowledgments
We would like to thank Dejian Yang, Jian-Guang Lou, Yuanzhi Li at Microsoft Research for helping us with training data preparation. We also sincerely thank Daniel Fried and Armen Aghajanyan at Meta Al for helpful discussion regarding the training details of FIM basclines.
References
[AGW+22] Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati,
10

---

Baishakhi Ray. Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. Multi- lingual evaluation of code generation models. CoRR, abs/2210..14865 2022.
[AHR 222 Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettle- moyer. CM3: A causal masked multimodal model of the internet. CoRR, abs/2201.075222 2022.
[AON+21] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le. and Charles Sutton. Program synthesis with large language models. CoRR, abs/2 100/2200.,7732, 2021.
[BJT+22] Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey. Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. CoRR, abs/2207. 14255, 2022.
[BMR 200] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark. Christopher Berner, Sam McCandlish, Alec Radford. llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- -ien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
[CND+22] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov. Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellsterm, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR, 0tbs//204.003111 2022.
[CT]+21] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri. Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry. Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, An- drew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR, abs/2107.03374, 2021.
[DCLT19]
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HIT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 4171 1171-4186. Association for Computational Linguistics, 2019.
11

---

[DFE 22] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficien exact attention with io-awareness. CoRR, 0bbb/2205.14115. 2022.
[DLL20] Chris Donahue, Mina Lee, and Percy Liang. Enabling language models to fill in the blanks. arXiv preprint arXiv:2005.05530 05533, 2020.
[2] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Fric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zeutlemoyer, and Mike Lewis. Incoder: generative model for code infilling and synthesis. CoRR, abs/2204.00599, 2022.
[GBB+21] Leo Gao, Stella Biderman, Sid Black, Laurence Golding. Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leaby. The pile: An 800gb dataset of diverse text for language modeling. CoRR, Abs/2101.00027, 2021.
[2] Tao Ge, Heming Xia, Xin Sun, Si-Qing Chen, and Furu Wei. Lossless acceleration for seq2seq generation with aggressive decoding. CoRR, abs/2205.103500 2022.
[HBK*21] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song. et Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.099::: 2021.
[HR18] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text 2018.
classification. arXiv preprint
[JCL*20] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Trans. Assoc. Comput. Linguistics, 8:6477, 2020.
[KB15] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[33 Yuan-Fang Li, Sébastien Bubeck, Ronen Eldan, Allison Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. ArXiv, abs/2300.05466. 2023.
[MNA+ 18] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David Gareía, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In 6th International Conference on Learn- ing Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
[PLR+22] Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Ballison-Burch, Bidirectional language models are also few-shot learners. arXiv preprint 2tti::222291440500 2022.
[3MH+23] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra-Aimeé Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: Outperforming curated corpora with web data, and web data only. ArXiv, abs/2306.01 116, 2023.
[PNI+18] Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Marilyn A. Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HIL 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Long Pupers), pages 2227-2237, Association for Computational Linguistics, 2018
12

---

[RSR + 20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1: 14:::-140::6, 2020.
[Sha19] Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR,
abs/1911.00 1.0150,, 2019.
1K]+18] Dmitriy Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, Chris Pal, and Yoshua Bengio. Twin networks: Matching the future for sequence generation. In International Conference on Learning Reprexentations, 2018.
[SLB17] Qing Sun, Stefan Lee, and Dhruv Batra. Bidirectional beam search: Forward-backward inference in neural sequence models for fill-i-tthe-blank image captioning. In Pro- ceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 6961-6969, 2017.
[SPP+19] 19] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/ 000000, 2019.
[TL]+ Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023.
[VSP+ 17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Process- ing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998 6008, 2017.
[YDY+19] 19] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoe V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Aché-Buuc Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Process- ing Systems 32: Annual Conference Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764, 2019.
[2] Yuanen Zhou, Zhenzhen Hu, Daqing Liu, Huixia Ben, and Meng Wang. Compact bidirectional transformer for image captioning. arXiv preprint wxXiv:2201001944 2022.
19] Zhirui Zhang, Shuangzhi Wu, Shujie Liu, Mu Li, Ming Zhou, and Tong Xu. Regularizing neural machine translation by target- binertbbdireciinnall agreement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 443 453, 2019.
[77719]] Long Zhou, Jiajun Zhang, and Chengqing Zong. Synchronous bidirectional neural machine translation. Trans. Assoc. Comput. Linguistics, 7:91-105, 2019.
13


Reviews:
[{"review_id": "c6FVqe4MRx", "reviewer": "Reviewer_M9cS", "summary": "This paper presents a new pre-training approach called MIM for language models, which promotes agreement between left-to-right and right-to-left models regarding their token distribution for each position. The authors have developed a straightforward and effective infilling inference method that utilizes context from both directions and the tendency of forward and backward models to align. The paper pre-trains language models of different sizes on public code and language data using MIM, and the results of various experiments demonstrate the superiority of MIM over existing baselines.", "questions": "1. Have you experimented with MIM using larger model scales, such as 7b and 13b? Have you identified any scaling laws for this approach?\n2. The predicted probability distribution from different directions may not the identical due to the different contexts. For instance, predicting the first token from a left-to-right perspective may be challenging, but it may be more accurate when using a right-to-left approach. What is your perspective on this? Could you provide more information on the probability differences observed in various positions?\n3. Some of the methods previously used in the NMT field may be directly used for language models. Have you evaluated the performance of these methods in your experiments?\n", "limitations": "n/a", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This paper investigates the bidirectional agreement approach for language models, and the results of various experiments demonstrate the superiority of MIM over existing baselines.\n- The methodology employed in the paper is robust, and the results are statistically significant. The ablation studies conducted in the paper confirm the effectiveness of the primary proposals during both training and inference.", "weaknesses": "- The proposed method is not very novel, as similar ideas have been proposed in the NMT field (treated as conditional language models). However, it is still valuable to investigate the impact of these methods on pure language models.\n- The bidirectional agreement approach leads to more memory cost and training time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to provide a comprehensive review of our work. We appreciate your thoughtful feedback and constructive comments. Below, we address the concerns and questions you raised.\n\nWeaknesses:\n\nMore Memory Cost and Training Time: Our method indeed requires more memory and training time. However, the memory consumption can be effectively mitigated by using a smaller batch size during training and we observed that the smaller batch size does not harm the training convergence. Furthermore, since our two models share parameters, the overall memory footprint is smaller than having two separate models. We believe that the benefits and robustness of the proposed method make these additional computational costs justifiable and we will include a discussion of these costs in the camera ready version.\n\nQuestions:\n\nLarger Model Scales: Unfortunately, due to limited resources, we have not pre-trained 7B and 13B parameter models. We will try to perform scaling law experiments on small scale models which can suggest how perplexity will extrapolate to these larger models by the camera ready deadline.\n\nProbability Distribution Differences: You astutely noted the potential for differences in the predicted probability distribution due to varying contexts. We concur that in many cases, \n$p(x_i | x_{<i})$ should not be equal to $p(x_i | x_{>i})$. Our regularizer's design indeed does not place a special emphasis on the actual token $x_i$ but aims to minimize the average difference across all words in the vocabulary and across all positions. The regularizer does not need to be universally correct, it just needs to have a positive contribution which is demonstrated by our empirical results.\n\nMethods from the NMT Field: We agree that integrating ideas from neural machine translation could lead to intriguing advancements in pre-training. Our work takes inspiration from some bidirectional approaches in the NMT field that have led to substantial improvements. However, exploring further techniques in this paper is beyond our scope."}, {"review_id": "XbWMxLodYm", "reviewer": "Reviewer_5x9n", "summary": "The paper proposes a new pretraining objective called meet in the middle. This is a bidirectional objective consisting of forward plus backward logprob and an agreement term measured by total variation distance between the two. The forward and backward models share parameters and optionally attention.\n\nInference is done by generating in two directions until they meet in the middle with agreement, potentially introducting a 2x factor in parallelism and reduced latency.\n\nThe model are tested on both code generation accuracy and perplexity on several training set and the pile. They compared to both published baselines training on different data as well as FIM trained on the same data, showing strong results.", "questions": "given that FIM actually conditions on both the prefix and suffix whereas MIM only conditions on one at a time with weak interactions, any discussion on how MIM outperforms despite that?\n\nnot very clear what is the backup when the model fails to meet in the middle and how false positives are dealt with, especially for long context generations.", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "bidirectional agreement objective that is pretty simple and intuitive in some ways, giving strong results compared to FIM. should be easy enough for others to adopt\n\nresults and comparisons are strong\n\nsome helpful ablations are included", "weaknesses": "\"New paradigm\" is perhaps exaggerated for introducing bidirectional model via sharing and agreement \n\nstill more complexity compared to FIM, and some source of improvements are not very clear. For instance, there are several claims of improved data efficiency due to bidirectionally, but the experiments do not seem to test this directly. If this is the main claim, then some direct left to right comparisons would make more sense.\n\nthe optional improvements seems pretty important and yielded half the improvement over FIM, so perhaps should not be considered optional. For instance, HE infilling results for MIM, MIM-lambda=0, and FIM are respectively 26.3, 24.7 and  22.8. The caption of table 1 says the 26.3 was without the enhancement from 3.2.2 whereas the ablations in table 5 suggest it was.\n\nno comparisons with more recent models like santacoder, starcoder\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and insightful comments on our paper. Your observations have guided us to reflect on certain aspects of our work, and we appreciate the opportunity to clarify and expand on those points.\n\nWeaknesses:\n\nTesting Improved Data Efficiency Directly: Your observation concerning the lack of direct evidence for improved data efficiency is well-noted.  We believe that the results in Table 4 provide indirect support for our claim, particularly considering the empirical support of the claim that \"FIM is for free/cheap\" in the original FIM paper (Bavarian  et. al. 2022) as well as in other papers including Santacoder and Starcoder that you mentioned. However, we acknowledge that a direct comparison with a non-FIM model might have been more illustrative. We will include this comparison in the camera ready version.\n\nWording on Optional Improvements: Indeed these improvements are only optional for left-to-right generation. We will change the wording to \"infilling improvements\" to be more precise. We are grateful to you for highlighting the error in Table 1's caption. We will ensure to correct it. To clarify, the columns in Table 1 that summarize the infilling tasks use the model with infilling improvements.\n\nComparisons with Santacoder and Starcoder: We appreciate your interest in comparisons with these recent models. Both of these models were trained with the FIM approach (Bavarian et. al. 2022). MIM is a general technique whose purpose is to improve upon FIM. Therefore the proper comparison is whether Santacoder/Starcoder trained with MIM outperforms Santacoder/Starcoder trained with FIM. We will explain this point in more detail in the camera ready version. \n\nQuestions:\n\nHow MIM Outperforms FIM: We speculate that MIM beats FIM because of context placement. In FIM, either the prefix or the suffix must be placed further from the completions, while in MIM, the relevant context is always nearby. This placement likely provides MIM with an advantage in attending to relevant information and achieving better performance.\n\nBackup When the Model Fails to Meet in the Middle: In such cases, we resort to using the completion with the best probability according to the model that generated it. This is explained in line 140 of the paper.\n"}, {"review_id": "1y830GUk47", "reviewer": "Reviewer_SDkH", "summary": "This paper proposes an extension of a paper  \"Fill in the middle\" (FIM)  (Bavarian et al 2022). The idea is to  define different objective function by considering two models for both directions and to \"Meet in the middle\" (MIM). One goal is to be more data efficient and the targeted application is in-filling for code generation. \n", "questions": "The first sentence of the abstract sounds really strange: the masked LM objective is used for instance with BERT, and the \"B\" means bidirectionnal. Beyond this lazy remark, a real discussion in the paper on this bidirectional point could be insightful for the reader. In the related work section the authors say that there is large body of work on bidirectional training. What are the real motivation of this new criterion ? What are the differences with BERT MLM, or ELMO ? What is expected in terms of performance ? \n\n- The experimental part focuses on the comparison between FIM and MIM (in the text). The differences with other  models are not really commented ?\n- The models used could be better described: \"a decoder only transformer LM\". Could you precise the number of layers, heads, ... (maybe in an appendix) ?\n", "limitations": "Not really. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper proposes an extension of a paper  \"Fill in the middle\" (FIM)  (Bavarian et al 2022). The idea is to  define different objective function by considering two models for both directions and to \"Meet in the middle\" (MIM). One goal is to be more data efficient and the targeted application is in-filling for code generation. \n", "weaknesses": "The method is simple, but the motivations remain unclear. For\ninstance, the paragraph on FIM should be improved. If a reader does\nnot already know about FIM, it is difficult to understand.  You could\nprovide some equations. Moreover a discussion, on why MIM is simpler\n(and could be better) is necessary.  Moreover my main concern is about\nthe results: at the end why MIM is really more data-efficient that FIM\nor than a BERT-like trained from scratch of the same task ?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to thoroughly review our paper. Your comments have highlighted some important areas for clarification and improvement, and we are grateful for the opportunity to address your concerns.\n\nWeaknesses:\n\nUnclear Motivation: The motivation for our work stems from addressing the limitation of FIM and pure left-to-right training. We discussed some of these in the first three sections of the paper but space limitations constrained a more comprehensive discussion. We refer readers to the papers we cite in the related work section in particular the Fill in the Middle paper and the Twin Networks paper, for additional insights. \n\nComparison with FIM and BERT on Data Efficiency: Your concern about why MIM is more data-efficient than FIM or a BERT-like model trained from scratch for the same task is well-noted. Many tasks we are targeting are challenging for BERT due to its non-autoregressive nature. Concerning FIM, MIM's advantage stems from keeping the context close to the generation, which we believe leads to its outperformance over FIM. We also explained this in our response to reviewer 5x9n.\n\nQuestions:\n\nDiscussion on Bidirectional and Comparison with BERT and ELMO: We appreciate your suggestion to discuss the bidirectional aspect more thoroughly. \nThe real motivation behind our work is rooted in the explosive adoption of generative language models for a wide array of use cases, including chatbots and assistive co-authoring. Unlike BERT and ELMO, where generation is non-trivial, FIM and MIM excel in text or code infilling. Our work shows that for such generative tasks, bidirectional modeling during training can still help.\n\nComparison with Other Models besides FIM: We concentrated on FIM as it was our strongest baseline and it was trained with exactly the same data as MIM. This helps isolate the effect of our proposal, while a comparison with other models would be confounded by other choices. We will include some more recent models in the camera ready version but the apples-to-apples comparison will remain FIM vs. MIM.\n\nModel Details: We appreciate your call for a more precise description of the models used, including the number of layers, heads, etc. We agree that this information is valuable, and we will include it in the appendix.\n\nThank you again for your thoughtful comments. We hope your concerns have been addressed by this response. If so, please consider raising your rating."}, {"review_id": "mcqhmgW5C3", "reviewer": "Reviewer_zFcY", "summary": "The paper proposes a new pre-training paradigm for language models that jointly improves the training data efficiency and the capabilities of the LMs in the infilling task. The first technique is a training objective that aligns the predictions of a left-to-right LM with those of a right-to-left LM, trained on the same data but in reverse order. The second technique is a bidirectional inference procedure that enables both LMs to meet in the middle.\n\nThe paper reports extensive experiments on both programming and natural language models, outperforming strong baselines.", "questions": "Quantify how much additional computational cost there is compared to regular left-to-right pretraining. Ablation of using such additional computation to train the left-to-right longer, and compare results to understand whether the gain reported in this paper comes from computational advantages.", "limitations": "N/A", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A nice idea to pretrain from both directions to improve data efficiency through consistency regularizer. The result models are naturally fit for infilling tasks.\n2. Strong results compared to FIM baselines with 3+% gain on most tasks evaluated.\n3. Clear ablations to understand where the gain comes from by turning off consistency regularizer.", "weaknesses": "1. Computational cost: training requires to train two models (forward and backward), which comes with additional computational cost.\n2. Evaluation is restricted to coding. While there is perplexity eval on natural languages, it is unclear whether lower perplexity translates to gains in model capabilities such as few-shot learning.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your comprehensive review and for recognizing the contributions and results of our paper. Your thoughtful comments have provided us with valuable insights, particularly concerning the computational costs and evaluation limitations. We'd like to address these points and clarify our intentions.\n\nWeaknesses:\n\nComputational Cost: Indeed training two models (forward and backward) incurs additional computational expenses. We acknowledge this and appreciate your interest in quantifying this aspect. We will include an analysis of the computational cost in comparison to regular left-to-right pretraining in the camera ready version of our paper. This will help clarify whether the gains we reported are attributed to computational advantages or inherent model efficiencies. Note that reviewer M9cS has raised a similar question so please check our response to them as well.\n\nEvaluation Limitation in Coding: While our focus was primarily on coding, your point about extending the evaluation to natural languages and examining downstream task performance is insightful. However such comparisons can be confounded by the exact in-context learning strategy and the amount of effort spent in prompt engineering. Our scope is pre-training and, similar to other works with the same scope, we mostly focus on perplexity and coding task performance.\n\nQuestions:\n\nAblation and Comparison with Extended Left-to-Right Training: Your suggestion to perform an ablation study using the additional computation to train the left-to-right model longer is particularly insightful. This will allow us to delineate whether the improvements stem from the bidirectional approach itself or merely from the extended training. We commit to including this analysis in the camera ready version as it will provide a more nuanced understanding of the strengths and potential limitations of our method.\n\nYour review has highlighted essential areas where we can further refine our work, and we appreciate your positive evaluation and constructive feedback. We are committed to incorporating these insights to enhance the clarity and robustness of our research."}, {"review_id": "YV90uOGoNG", "reviewer": "Reviewer_V8eN", "summary": "The paper proposes a method named \"Meet in the Middle\" (MIM), an novel inference approach for language models. The uniqueness of MIM lies in its bidirectional training—left-to-right and right-to-left—that not only augments data efficiency but also aligns the token distribution for each position. The method improves the performance on infilling task by employing a dual-direction infilling procedure. Empirical evidence gathered from extensive testing on programming and natural languages substantiates MIM's superior performance over traditional pre-training methodologies in both, left-to-right generation and infilling.", "questions": "N/A", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The novelty of the \"Meet in the Middle\" (MIM) stands out, particularly with respect to the inference phase in text infilling.\n\n2. MIM successfully addresses the important task of general text infilling, encompassing both natural and programming languages. Its universality as an ensemble approach heightens the performance of language models, agnostic of their underlying neural network architecture.\n\n3. The empirical validation of this research is robust. The efficacy of the proposed method is rigorously assessed against multiple recognized benchmarks and contrasted with strong, widely accepted baselines.", "weaknesses": "1. One potential drawback of the proposed MIM model is its increased computational demand and memory usage, attributable to the incorporation of two models, as compared to conventional left-to-right LMs. It would enhance the comprehensiveness of the comparison if the MIM model is benchmarked against an ensemble of two left-to-right LMs pretrained with different random seeds. The experiment is useful even on a smaller scale.\n\n2. The visual presentation of the paper could benefit from improvements. The majority of the tables and figures (except figure 1), are inadequately sized for legibility, particularly when printed. This could impede the reader's understanding and appreciation of the presented research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful review and for recognizing the novelty and efficacy of our proposed method. We would like to address the weaknesses you identified and clarify our approach to addressing them.\n\nWeaknesses:\n\nIncreased Computational Demand and Comparison with Ensemble of Two Left-to-Right LMs: We acknowledge the concern regarding the increased computational demand due to the bidirectional nature of our model. Your suggestion to compare MIM with an ensemble of two left-to-right LMs pretrained with different random seeds is insightful but may not offer a direct comparison. Such an ensemble would possess twice the number of parameters, unlike our MIM model, which utilizes one set of shared parameters. While we can explore including this experiment, we are unsure about its capacity to elucidate further the reasons behind MIM's superior performance. We will, however, commit to clearly detailing the computational requirements in the paper and provide a more nuanced discussion of how MIM compares with other approaches.\n\nVisual Presentation of Tables and Figures:  We value your feedback regarding the legibility of the tables and figures in our paper. We recognize that the visual presentation is crucial for a clear understanding of our findings. We will include comprehensive tables in full-size in the appendix and in the main paper we will provide more focused comparisons with only the most relevant baselines."}]

Author rebuttal:

Meta-review: The paper introduces "Meet in the Middle" (MIM), a new pre-training paradigm for autoregressive language models that trains in both left-to-right and right-to-left directions to improve data efficiency and performance in tasks like infilling.

Reviewers generally agree that the MIM approach is robust, with positive results outperforming existing baselines, specifically Fill in the Middle. The methodology is considered simple, intuitive, and easily adoptable (M9cS, 5x9n). MIM's applicability to both natural and programming languages is praised, as are the clear ablation studies that help understand its effectiveness (SDkH, zFcY, V8eN).

Reviewers, however, raised concerns about the increased computational cost and memory usage due to the bidirectional aspect (M9cS, zFcY, V8eN). Some reviewers question the novelty, pointing out that similar ideas exist in Neural Machine Translation (M9cS, 5x9n). There's also a call for clearer motivation and justification, especially about MIM's data efficiency compared to existing methods like FIM and BERT (SDkH, 5x9n).
Decision: Accept (poster)