Title: Meet in the Middle: A New Pre-training Paradigm
Venue: NeurIPS 2023
Accepted: yes
Keywords: language modeling

pre-training

deep learning

NLP
Abstract: Most language models (LMs) are trained and applied in an autoregressive left-to-right fashion, predicting the next token from the preceding ones. However, this ignores that the full sequence is available during training. 
In this paper, we introduce ``Meet in the Middle'' (MIM) a new pre-training paradigm that improves data 
efficiency by training in two directions, left-to-right and right-to-left, and encouraging the respective models
to agree on their token distribution for each position. While the primary outcome is an improved left-to-right LM,
we also obtain secondary benefits in the infilling task. There, we leverage the two pre-trained directions to propose an infilling procedure that builds the completion simultaneously from both sides. We conduct extensive experiments on both programming and natural languages and show that MIM significantly surpasses existing pre-training paradigms, in both left-to-right generation as well as infilling.
Code and models available at https://github.com/microsoft/Meet-in-the-Middle

Meet in the Middle: A New Pre-training Paradigm

Abstract

Most language models (LMs) are trained and applied in an autoregressive leftto-right fashion, predicting the next token from the preceding ones. However, this ignores that the full sequence is available during training. In this paper, we introduce 'Meet in the Middle' (MIM) a new pre-training paradigm that improves data efficiency by training in two directions, left-to-right and right-to-left, and encouraging the respective models to agree on their token distribution for each position. While the primary outcome is an improved left-to-right LM, we also obtain secondary benefits in the infilling task. There, we leverage the two pre-trained directions to propose an infilling procedure that builds the completion simultaneously from both sides. We conduct extensive experiments on both programming and natural languages and show that MIM significantly surpasses existing pre-training paradigms, in left-to-right generation as well as infilling. 2

1 Introduction

Language models (LMs) are becoming essential for various assisted authoring tasks, including text summarization and code completion. In order to be usable in many different applications, most LMs must be able to generate the next token from the sequence of previous tokens. Thus, pre-training has focused on optimizing the model's ability to predict the next token given the previous tokens (the prefix), as measured by perplexity. However, this neglects that at pre-training time we also have access to the subsequent tokens (the suffix). While the suffix cannot be used as an input to the model, there are other ways to incorporate it into pre-training which have not received much attention in the literature. In this work we propose a more data-efficient approach, preserving the autoregressive nature of LMs while fully utilizing pre-training data.

Why should we be interested in alternative pre-training procedures? After all, the main artifact produced during pre-training is an autoregressive left-to-right LM and the pre-training objective closely matches both how the LM is applied and the desire to predict the next token well. Nonetheless, there are still compelling reasons to consider alternative training objectives. First, data efficiency: current LMs are trained with sparse supervision, only learning from the actual next token. By exploring denser supervision during training, we aim to improve performance with less data. Second, related tasks: in real-world scenarios, users may need to infill or modify existing text. Current left-to-right LMs are limited in this regard, as they cannot use both sides of the insertion position, often leading to suboptimal results.

In this work, we introduce a novel pre-training paradigm named 'Meet in the Middle' (MIM). The MIM approach is founded on two central principles. Primarily, it introduces an additional right-to-left LMto complement the standard left-to-right model, enabling both models to co-regularize and benefit from each other's context. This improves data efficiency and consistency, with models metaphorically meeting in the middle by adjusting their output probabilities to agree with one another. Secondarily,

MIM enables an efficient infilling inference procedure that capitalizes on the agreement between the two LMs developed during the pre-training phase. For infilling, the models literally meet in the middle, constructing the completion from both ends. This aspect of MIM, while secondary to the pre-training objective, offers crucial benefits. Co-regularization not only reinforces consistency between the two LMs but also aids in early termination of the generation process during infilling.

To train MIM, we employ two decoding flows within a single shared decoder-only architecture [BMR + 20], [CND + 22]. The forward and backward LMs generate tokens in opposite directions, predicting the next token and previous token, respectively. We pre-train both models jointly on a large text corpus using a combination of standard LM loss and an agreement regularizer. Upon completion, the forward model serves as a drop-in replacement for existing autoregressive LMs, while the backward model can be discarded or utilized for tasks such as infilling.

In our experiments, we assess MIM's effectiveness for pre-training LMs across various domains and tasks. Using public code and language data, we pre-train LMs of different sizes and evaluate their performance based on perplexity and code completion tasks. We compare MIM with FIM [BJT + 22] and other baselines, demonstrating its superior performance in terms of perplexity and task-specific evaluation metrics. Furthermore, we conduct ablation studies to validate the effectiveness of our primary proposals during training and inference. In summary, our main contributions include:

Introducing a novel pre-training paradigm for LMs that efficiently utilizes training data by incorporating both prefix and suffix context while retaining the autoregressive nature of LMs. This is achieved by training forward and backward models and promoting their agreement.

Proposing a simple, efficient infilling inference procedure that leverages context from both sides and the tendency of forward and backward models to agree. Our approach employs parallelism more effectively than existing procedures, resulting in superior quality and latency compared to the state of the art.

Pre-training LMs of various sizes on public code and language data using MIM, evaluating their performance on both human and programming languages, and demonstrating that MIM outperforms numerous baselines in standard evaluation metrics. Additionally, we make some models and code publicly available.

2 Preliminaries

We introduce notation used throughout the paper. For a token sequence x 1 , x 2 , . . . , x N , we denote x <i as the prefix x 1 , x 2 , . . . x i -1 and x >i as the suffix x i +1 , x i +2 , . . . x N . Definitions for x ≤ i and x ≥ i are analogous. We suppress model dependencies on learnable parameters. Arrows distinguish the two models and their outputs; for example, - → p represents the forward model and ← -p the backward model, while - → H and ← -H are hidden representations from the respective models.

2.1 The Infilling task

Given a token sequence x 1 , x 2 , . . . , x N , an insertion position i , the infilling task involves generating a plausible M -token sequence y 1 , . . . , y M to fill the gap between x ≤ i and x ≥ i +1 . In real-world applications, M is unknown. The infilling task is harder than the left-to-right generation task as the latter is a special case of infilling with an empty suffix. Since generating the best M -token left-to-right generation is a search problem that is tackled by heuristics such as greedy search or beam search, we will also follow heuristic approaches for the strictly harder infilling problem.

A technique for infilling that utilizes context from both sides is 'Fill in the Middle' (FIM) [BJT + 22]. FIM forms the context by concatenating the suffix and prefix (in that order), ensuring coherence near the completion point. FIM is easily applied to pre-trained LMs with minimal modification and is computationally efficient. However, FIM's drawbacks include training on unnatural concatenations of prefix and suffix, giving the prefix more influence on the completion over the suffix, and an ad-hoc training procedure that only considers a few of the O ( N 2 ) possible splits in an N -token document.

2.2 Bidirectional Language Modeling

Bidirectional language modeling has been used in the literature in two different ways: The first is to describe non-autoregressive LMs with objectives like Masked Language Modeling, which produce better representations but face difficulties in in-context learning [PLR + 22].

Earlier work [HR18, PNI + 18] used the term bidirectional language modeling to describe models with left-to-right and right-to-left autoregressive flows. Our approach falls in this category, remaining autoregressive and using future tokens only for regularization during training.

3 Meet in the Middle

Our proposal involves pre-training two models to predict their next token from different input views (prefix vs. suffix), while maintaining agreement on the probability distributions for these tokens. This provides a dense supervision signal compared to simply predicting the next token. It also encourages the models to 'meet in the middle' i.e. reach a compromise prediction informed by different views.

3.1 Pre-training

We train two decoder-only LMs, a forward model - → p and a backward model ← -p , with shared parameters. - → p predicts next tokens in the forward direction, while ← -p predicts previous tokens in the backward direction. To improve data efficiency, we use a co-regularization term that encourages the models to agree on the predicted probability distribution over the vocabulary for each token. We use total variation distance to measure disagreement between the models. Define the disagreement on x i

where V is the vocabulary. The agreement regularizer is the sum of these terms over all sentences and tokens. It provides the models with a denser supervision signal which improves data efficiency and helps us train a better autoregressive LM. This is because two probability distributions over all tokens are compared while in the traditional LM objective only the probability of the observed token matters. Moreover it encourages the models to agree on their predictions which helps with the efficiency of infilling as we will see later. Then, the full training loss for dataset S of sequences is given by:

The hyperparameter β is set to 0 . 1 in our experiments. After pre-training, - → p can be used as a left-to-right autoregressive LM.

3.2 Infilling

3.2.1 Inference

Our inference procedure for infilling aims to be efficient and low-latency. Our approach is shown in Figure 1. At a high level, the two models start building a completion each from their own side and they try to literally meet in the middle. Initially, the prefix and suffix are consumed by - → p and ← -p respectively. Then - → p and ← -p generate tokens synchronously one at a time. For each generated token from - → p (resp. ← -p ) we check whether it matches a generated token from ← -p (resp. from - → p ). If there is a match, we have a 'meet in the middle' candidate position for joining the two generated sequences.

In an ideal scenario, - → p and ← -p generate identical sequences, albeit in reverse order, resulting in 'meeting in the middle' after half of the tokens are generated by each model. The agreement regularizer is thus crucial; if - → p and ← -p yield disparate sequences, our method's speed is inferior to FIM since early termination isn't possible. However, if there's complete agreement, with sufficient parallelism, our method can outperform FIM in generation latency, as each model only needs to generate half the completion autoregressively. Our experiments demonstrate our method's superior latency to FIM, indicating that in most instances, sequences meet close to the middle.

Prefix

The

quick

dog

fox

jumped

over

over

the

lazy

glyph[check]

- →

p

(

x

t

)

← -

p

(

x

t

)

MIM

Verification

glyph[check]

0

0

0

1

1

2

2

3

3

4

4

5

5

Suffix

However, the procedure may yield false positives. For instance, matching a token such as 'the' from both sides may not guarantee a coherent infilling upon joining. To mitigate this, we employ 4-gram matching in our infilling experiments. Additionally, we run a parallel verification procedure to ensure that the resultant sequence could have been generated by either - → p or ← -p running to completion.

Our parallel verification procedure is adapted from [GXS + 22]. Concretely, assume that the latest token generated from - → p matches with a token in position s generated from ← -p , as is the case in Figure 1 for the token 'over'. Then we can use the tokens that were generated by ← -p before s as inputs to - → p in parallel . In the context of Figure 1 we provide the tokens 'over', 'the', 'lazy' as inputs to - → p in parallel. If each output of - → p matches the corresponding input, we have verified that - → p would have autoregressively generated the tokens it copied from ← -p 's output. Grounding the discussion back to Figure 1, if - → p generates (in parallel) 'the' and 'lazy' we have verified that - → p would have autoregressively generated the same tokens. In cases of partial or no match, the generation from - → p either fast-forwards to the first disagreement point or resumes autoregressive generation from - → p and ← -p . While our implementation use greedy top-1 sampling and comparison of input and output at each position, we could adopt less stringent acceptance criteria, like a sufficiently high probability of the candidate token in the previous output. If - → p and ← -p finish generating without meeting or passing verification, we return the higher-probability sequence according to its generating model.

3.2.2 Infilling Specific Enhancements

Our model's infilling performance can be boosted by adopting the Synchronous Bidirectional Attention mechanism [ZZZ19]. Though this reduces compatibility with autoregressive language models (LMs), it enhances generation through bidirectional conditioning. The Synchronous Bidirectional Attention layer modifies the regular attention layer activations as follows: Let - → H and ← -H be the outputs of an attention layer in - → p and ← -p respectively. For a hyperparameter λ , the fused attention hidden representation becomes - → H f = - → H + λ ← -H and ← -H f = ← -H + λ - → H . Instead of - → H and ← -H , the models - → p and ← -p use - → H f and ← -H f . If λ = 0 , - → p and ← -p become classic autoregressive transformers. However, careful training is necessary to avoid information leakage with this layer. To mitigate this, we adopt a two-stage training process, see [ZZZ19] for details.

4 Experiments

4.1 Data and Models

Our code models are pre-trained on a large and diverse corpus of public code with permissive licenses, covering multiple programming languages. The dominant languages in our corpus are Python, Java, and C++. After filtering and deduplication, the corpus contains about 300B tokens. Details are provided in Table 9. Our models train on a single pass over this data. This is about six times larger than the dataset used to train the Incoder model [FAL + 22], which included 50B tokens comprising code and Stack Overflow data.

We train our natural language models on data from the following datasets: CC-News , OpenWebText , CC-Stories , and CC-100 to assess the models language modeling capability. In particular:

CC-News : 63M English news articles crawled from Sept 2016 to Feb 2019 (76GB).

OpenWebText : An open source recreation of the GPT-2 WebText dataset (38GB).

CC-Stories : A subset of CommonCrawl data filtered to match the story-like style of Winograd schemas (31GB).

CC-100 : A dataset extracted from CommonCrawl snapshots between Jan 2018 and Dec 2018, filtered to match the style of Wikipedia (292GB).

Apart from the natural language data utilized in our language modeling experiments, we also train our models on a filtered version of the Falcon RefinedWeb corpus [PMH + 23] used in the previous work of [LBE + 23], that has a total of 88B tokens. We then present the performance results on common natural language benchmarks in section 4.3.3

We use a decoder-only transformer LM [VSP + 17, BMR + 20] for pre-training. The model is trained to predict the next token in both left-to-right and right-to-left directions, using the same model and parameters. To explore the impact of model size, we pre-train models with different capacities: 350M, 1.3B, 2.7B. For the hyperparameters and training setup of each model size, please refer to the Appendix.

As baselines, we use well-known published results but also pre-train three FIM models with contextlevel FIM [BJT + 22]. Character-level transformations are applied, with a FIM-rate of 0 . 5 and SPM+PSM joint training. Both the MIM and FIM models are pre-trained using the Megatron-LM framework [SPP + 19]. See the Appendix for model configurations and architectural details.

4.2 Benchmarks and metrics

4.2.1 Code generation and infilling

We assess MIM's performance both in autoregressive left-to-right generation and infilling tasks. For the autoregressive generation, the model generates the code body from a function signature, docstring, and test cases. Evaluation datasets include HumanEval [CTJ + 21], MBPP [AON + 21], and APPS [HBK + 21] for Python problems. We use the multilingual HumanEval-X dataset for multiple programming languages. For this task, the left-to-right model alone is used for inference without any provided suffix.

In the infilling task, the model fills blank lines in an incomplete program. This is evaluated using the HumanEval Infilling [BJT + 22] and MBXP [AGW + 22] datasets. MIM is compared with Incoder, three FIM variants we trained on the same data as MIM, and code-davinci-002, as these datasets provide suffixes that not all models can manage effectively.

We use the pass@k metrics [CTJ + 21] to measure the percentage of generated codes that pass all tests within the top-k candidates. We report pass@1, pass@10, and pass@100. For generation, top-p sampling with p = 0 . 95 is employed with temperatures of 0 . 2 for pass@1 and 0 . 8 for pass@10 and pass@100. We also include single-line exact match (EM) metrics to indicate the percentage of completed lines matching the reference solution's masked lines [FAL + 22, BJT + 22].

4.2.2 Language Modeling

In addition to code generation and infilling, we evaluate MIM's language modeling capabilities, particularly its next-token prediction ability measured by perplexity. Perplexity is evaluated in both in-domain and out-of-domain settings. The former involves a held-out subset of the combined training data, while the latter uses the Pile dataset [GBB + 21], a multi-domain public language modeling dataset. Average perplexity across all Pile subsets is reported for our models and baselines.

4.2.3 Natural Language Benchmarks

Finally, we evaluate MIM on established benchmarks for natural language, encompassing both commonsense reasoning and language understanding. For commonsense reasoning, we have tested our models on Winogrande, ARC-Easy, ARC-Challenge, BoolQ, and SIQA. In terms of knowledge and language understanding, we have assessed MIM's performance on PIQA, HellaSwag, and OpenbookQA. We compare against phi-1.5-web-only [LBE + 23] and present zero-shot results for these tasks using the LM-Eval Harness, with the exception of MMLU where we use 2-shot performance.

4.3 Main results

4.3.1 Code generation and infilling

First we focus on the superior performance of our from-scratch pre-trained FIM model over various baselines, in Table 1. For instance, our 2.7B model yields a 28.5% pass@1 in HumanEval, significantly outperforming the 15.2% pass@1 of Incoder's 6.7B model [BJT + 22]. FIM-2.7B also beats other strong baselines such as Codex 2.5B and CodeGen-Multi-6.1B, achieving 67.8% in pass@100 compared to their respective 44.9% and 59.5%. This performance is attributable to larger, cleaner training data and implementation details (discussed in the Appendix).

Having established a strong baseline, we now benchmark MIM's autoregressive generation in a prefix-only setting using both the HumanEval and MBPP datasets. While FIM vs. MIM is the applesto-apples comparison, we include many other models for reference and note that all these could have leveraged our MIM objective during their training. Table 1 shows MIM consistently outperforming FIM across all metrics for all model sizes (350M, 1.3B, 2.7B). For example, MIM-2.7B enhances HumanEval pass@1 to 30.7%, a 2.2% gain over FIM-2.7B. The improvement on MBPP is similar, in terms of pass@1. We note that the improvements increase with the model size. This is evidence that the larger models benefit more from the agreement regularizer. We further evaluate this setting on the APPS dataset, which has three different difficulty levels. We report the results in Table 3. The trend and improvement are similar to the other two datasets, in which MIM consistently outperforms FIM across all the metrics and difficulty levels. In the multilingual setting, we compare MIM with FIM baselines in the HumanEval-X dataset. We observe consistent improvement across all metrics and all the programming languages that we evaluated on. Results are reported in Table 2.

These findings illustrate MIM's superiority over FIM for left-to-right autoregressive generation. The main claim in [BJT + 22] is that FIM does not harm the original left-to-right generative capability and can be learned for free. We contend that MIM's pre-training, enhanced by the agreement regularizer, results in a superior generative model, thereby proposing a new pre-training paradigm.

Finally, we test MIM in an infilling setting, as motivated by real-world applications like GitHub Copilot, where developers frequently edit in the middle of source files. MIM outperforms FIM across all metrics and model sizes on both the HumanEval Infilling and the MBXP Infilling benchmarks (Table 1). Notably, the MIM-2.7B model achieves pass@1 of 26.3% and an exact match metric of 57.8%, improvements of 3.5% and 6.1%, respectively, over the FIM-2.7B model. This underscores the consistent improvement that MIM offers across different model sizes and tasks.

4.3.2 Language Modeling

Here we compare FIM and MIM models pre-trained on language modeling tasks. We use perplexity as our evaluation metric and look at two different settings, namely in-domain and out-of-domain .

The perplexity results of both settings are summarized in Table 4. Across all experiments, MIM consistently outperform FIM baselines in terms of perplexity. The MIM-2.7B model has the best

perplexity across all datasets in both settings. For example, MIM-2.7B model obtains a perplexity of 9.54 in OpenWebText dataset, a relative 19 . 9% reduction in perplexity over FIM-2.7B model, which obtains a perplexity of 11.92. In the Pile dataset [GBB + 21], which represents the out-of-domain setting, MIM-2.7B also outperforms FIM-2.7B by a relative reduction of 15 . 3% in perplexity (9.24 vs 10.92), which further reinforces the advantages of MIM pre-training for natural languages.

4.3.3 Natural Language Benchmarks

We trained a MIM model with a total of 1.3B parameters, which shares architectural details with [LBE + 23], on a filtered version of the Falcon RefinedWeb corpus, comprising a total of 88B tokens. This model was subsequently compared with the baseline model, phi-1.5-web-only in [LBE + 23],

which was trained on identical data using an autoregressive left-to-right objective. Tables 5 and 6 summarize the results. On commonsense reasoning MIM-1.3B-web-only outperforms by 0.42% on average while on knowledge understanding it outperforms by 1.2% on average. Overall, (combining the two tables) MIM improves by 0.76% on average.

4.4 Ablation Study

4.4.1 Effect of Infilling Specific Enhancements

In this section, we perform an ablation study to qualitatively assess the effect of the enhancements we proposed in section 3.2.2 for the infilling task. We compare the purely autoregressive MIM model ( λ = 0) where the LMs are not allowed to observe generated tokens from the opposite side, and only utilize context from their own side during generation. We contrast this with using the Synchronous Bidirectional Attention layer with λ = 0 . 3 that conditions on previously generated tokens from both sides. We used perplexity on the validation data to select the value λ . We reuse the same value of λ during infilling to avoid any potential mismatch between training and inference.

We conduct an experiment on HumanEval Infilling [BJT + 22] and results are summarized in Table 7. We notice that models that directly incorporate bidirectional context always outperform models that only utilize unidirectional context across all model sizes. As always, this is at the expense of the forward model no longer being a a drop-in replacement for standard autoregressive LMs.

4.4.2 Effect of Agreement Regularizer

While MIM performs better than FIM it's still possible that this difference is simply due to the parameters being trained on two tasks (left-to-right and right-to-left). To preclude this, we perform an ablation study where we remove the token-level agreement regularizer of Section 3.1 during training while keeping the infilling enhancements ( λ = 0 . 3 ). We show that encouraging agreement during training helps improve the infilling performance of our models in the HumanEval Infilling benchmark [BJT + 22], as summarized in Table 8. Comparing with Table 7, we see that models trained without token-level agreement regularization in general perform worse than models that do not utilize bidirectional context ( λ = 0) , which further emphasizes the importance of our agreement regularizer in making the predictions consistent between forward and backward directions.

Table 7: pass@1 ( % ) results of MIM with ( λ = 0 . 3) and without ( λ = 0 . 0) bidirectional context on HumanEval Infilling and MBXP. Exact match results on HumanEval Infilling are also reported.

4.5 Efficiency

4.5.1 Training Efficiency

At first glance, one might hypothesize that our proposed methodology could potentially necessitate twice the computational resources in comparison to the conventional left-to-right autoregressive training, with most of the performance gains attributed to this additional computation. Contrary to this supposition, empirical evidence gathered from our experiments suggests a different narrative. Our Meet-in-the-Middle (MIM) strategy exhibits performance metrics that are not only commensurate but often superior to those of the left-to-right autoregressive training. This assertion is corroborated by a comparative analysis of the validation log perplexity of MIM models and left-to-right autoregressive models across different model sizes. All models were trained on the filtered Falcon RefinedWeb corpus [LBE + 23]. As illustrated in Figures 3 and 4 in the Appendix, when assessed under the same computational budget (either FLOP count or wall clock time), MIM models consistently outperform left-to-right autoregressive models across all relevant computational budgets. Besides, when comparing MIM models and left-to-right autoregressive models on a time-basis, we still observe major speedups over the baselines.

4.5.2 Inference Efficiency

Finally we assess the efficiency of our infilling procedure, contrasting it with FIM through inference latency across various batch sizes. Figure 2 shows that MIM achieves a speedup over FIM baselines in both single and half precision, using a 1.3B model, across all HumanEval Infilling examples. MIM-1.3B specifically performs 4% to 6% faster than FIM-1.3B in single precision, and 3% to 5% faster in half precision. This is due to MIM leveraging available parallelism better than FIM.

Table 8: pass@1 ( % ) results of MIM with and without agreement regularizer on HumanEval Infilling and MBXP. Exact match results on HumanEval Infilling are also reported.

5 Related work

There is an extensive body of work on bidirectional language modeling. Early work such as ULMFiT [HR18], and ELMo [PNI + 18] kept the autoregressive nature of the LM, while later work such as BERT [DCLT19], T5 [RSR + 20], and SpanBERT [JCL + 20] focused on representation learning and was non-autoregressive. XLNET [YDY + 19], on the other hand, utilizes bidirectional context during training by the permutation language modeling objective, which maximizes the likelihood over all factorization orderings of the training sequences. However, because these later models typically focus on representation learning, in-context learning via prompting can be difficult [PLR + 22].

Two works that train neural models using similar ideas are [SKS + 18] and [ZWL + 19]. In the former the authors train RNNs for real-time acoustic modeling. This constraint is very similar to our desire of having a forward model that can generate the next token from the previous ones. However, they propose to regularize the forward and backward RNNs by requiring their representations to be close in Euclidean distance. We suspect that this constraint may be unnecessarily stringent, and trading it off against the LM's perplexity during training could hurt overall performance.

On the other hand, [ZWL + 19] encourage agreement in probability space. However, they are interested only in neural machine translation and only encourage agreement of probabilities at the output sequence level. In contrast, we encourage agreement on every token.

Sharing all parameters between different factorizations of the sequence was first proposed in XLNET [YDY + 19] but has also been used with only forward and backward models in the context of image captioning [ZHL + 22]. These works motivated us to share all parameters between our two LMs.

Using LMs for infilling was first proposed in [DLL20] where the authors tackled the more challenging setup with multiple blanks. [BJT + 22] applies 'Fill in the Middle' (FIM) to the training data by randomly splitting each training instance into a tuple of (prefix, middle, suffix) and concatenate these sections into a single example together with their sentinel tokens after tokenization. [BJT + 22] also contains a thorough section on research related to infilling. We refer the reader there for more details.

Furthermore, [AHR + 22] and [FAL + 22] propose an extension of FIM, namely 'Causal Masked Language Modeling' (CM3) and explore multi-region infilling problem. Similar to us, [BJT + 22] and [FAL + 22] leverage FIM to pre-train decoder-only LMs on code data and evaluate their models on zero-shot code completion benchmark created from HumanEval dataset [CTJ + 21].

In [SLB17] the authors propose bidirectional beam search that also employs forward and backward LMs. However their focus is in improving accuracy at the expense of latency. Their procedure performs multiple passes over the completion, fixing tokens one-by-one. Our approach focuses on achieving better infilling accuracy than FIM while also reducing latency.

6 Conclusion

This paper presented 'Meet-in-the-Middle' a novel method devised to address two significant challenges in large LMs: pre-training data efficiency and effective context handling in infilling tasks. By using both forward and backward LMs that share parameters and are trained for mutual agreement and next token prediction, the method significantly enhances the performance of existing autoregressive LMs.

Acknowledgments

We would like to thank Dejian Yang, Jian-Guang Lou, Yuanzhi Li at Microsoft Research for helping us with training data preparation. We also sincerely thank Daniel Fried and Armen Aghajanyan at Meta AI for helpful discussion regarding the training details of FIM baselines.

References

[AGW + 22] Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati,

Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang. Multilingual evaluation of code generation models. CoRR , abs/2210.14868, 2022.

[AHR + 22] Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, and Luke Zettlemoyer. CM3: A causal masked multimodal model of the internet. CoRR , abs/2201.07520, 2022.

[AON + 21] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. CoRR , abs/2108.07732, 2021.

[BJT + 22] Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. CoRR , abs/2207.14255, 2022.

[BMR + 20] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual , 2020.

[CND + 22] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR , abs/2204.02311, 2022.

[CTJ + 21] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel HerbertVoss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. CoRR , abs/2107.03374, 2021.

[DCLT19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pretraining of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers) , pages 4171-4186. Association for Computational Linguistics, 2019.

[DFE + 22] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. CoRR , abs/2205.14135, 2022.

[DLL20] Chris Donahue, Mina Lee, and Percy Liang. Enabling language models to fill in the blanks. arXiv preprint arXiv:2005.05339 , 2020.

[FAL + 22] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. CoRR , abs/2204.05999, 2022.

[GBB + 21] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR , abs/2101.00027, 2021.

[GXS + 22] Tao Ge, Heming Xia, Xin Sun, Si-Qing Chen, and Furu Wei. Lossless acceleration for seq2seq generation with aggressive decoding. CoRR , abs/2205.10350, 2022.

[HBK + 21] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938 , 2021.

[HR18] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146 , 2018.

[JCL + 20] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Trans. Assoc. Comput. Linguistics , 8:64-77, 2020.

[KB15] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

[LBE + 23] Yuan-Fang Li, Sébastien Bubeck, Ronen Eldan, Allison Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report. ArXiv , abs/2309.05463, 2023.

[MNA + 18] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings . OpenReview.net, 2018.

[PLR + 22] Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison-Burch. Bidirectional language models are also few-shot learners. arXiv preprint arXiv:2209.14500 , 2022.

[PMH + 23] Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra-Aimée Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only. ArXiv , abs/2306.01116, 2023.

[PNI + 18] Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Marilyn A. Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers) , pages 2227-2237. Association for Computational Linguistics, 2018.

[RSR + 20] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21:140:1-140:67, 2020.

[Sha19] Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR , abs/1911.02150, 2019.

[SKS + 18] Dmitriy Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, Chris Pal, and Yoshua Bengio. Twin networks: Matching the future for sequence generation. In International Conference on Learning Representations , 2018.

[SLB17] Qing Sun, Stefan Lee, and Dhruv Batra. Bidirectional beam search: Forward-backward inference in neural sequence models for fill-in-the-blank image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6961-6969, 2017.

[SPP + 19] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR , abs/1909.08053, 2019.

[TLI + 23] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR , abs/2302.13971, 2023.

[VSP + 17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017.

[YDY + 19] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 5754-5764, 2019.

[ZHL + 22] Yuanen Zhou, Zhenzhen Hu, Daqing Liu, Huixia Ben, and Meng Wang. Compact bidirectional transformer for image captioning. arXiv preprint arXiv:2201.01984 , 2022.

[ZWL + 19] Zhirui Zhang, Shuangzhi Wu, Shujie Liu, Mu Li, Ming Zhou, and Tong Xu. Regularizing neural machine translation by target-bidirectional agreement. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 443-450, 2019.

[ZZZ19] Long Zhou, Jiajun Zhang, and Chengqing Zong. Synchronous bidirectional neural machine translation. Trans. Assoc. Comput. Linguistics , 7:91-105, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method named \"Meet in the Middle\" (MIM), an novel inference approach for language models. The uniqueness of MIM lies in its bidirectional training—left-to-right and right-to-left—that not only augments data efficiency but also aligns the token distribution for each position. The method improves the performance on infilling task by employing a dual-direction infilling procedure. Empirical evidence gathered from extensive testing on programming and natural languages substantiates MIM's superior performance over traditional pre-training methodologies in both, left-to-right generation and infilling.\n\nSTRENGTHS:\n1. The novelty of the \"Meet in the Middle\" (MIM) stands out, particularly with respect to the inference phase in text infilling.\n\n2. MIM successfully addresses the important task of general text infilling, encompassing both natural and programming languages. Its universality as an ensemble approach heightens the performance of language models, agnostic of their underlying neural network architecture.\n\n3. The empirical validation of this research is robust. The efficacy of the proposed method is rigorously assessed against multiple recognized benchmarks and contrasted with strong, widely accepted baselines.\n\nWEAKNESSES:\n1. One potential drawback of the proposed MIM model is its increased computational demand and memory usage, attributable to the incorporation of two models, as compared to conventional left-to-right LMs. It would enhance the comprehensiveness of the comparison if the MIM model is benchmarked against an ensemble of two left-to-right LMs pretrained with different random seeds. The experiment is useful even on a smaller scale.\n\n2. The visual presentation of the paper could benefit from improvements. The majority of the tables and figures (except figure 1), are inadequately sized for legibility, particularly when printed. This could impede the reader's understanding and appreciation of the presented research.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-11"}

comments: SUMMARY:
The paper proposes a new pre-training paradigm for language models that jointly improves the training data efficiency and the capabilities of the LMs in the infilling task. The first technique is a training objective that aligns the predictions of a left-to-right LM with those of a right-to-left LM, trained on the same data but in reverse order. The second technique is a bidirectional inference procedure that enables both LMs to meet in the middle.

The paper reports extensive experiments on both programming and natural language models, outperforming strong baselines.

STRENGTHS:
1. A nice idea to pretrain from both directions to improve data efficiency through consistency regularizer. The result models are naturally fit for infilling tasks.
2. Strong results compared to FIM baselines with 3+% gain on most tasks evaluated.
3. Clear ablations to understand where the gain comes from by turning off consistency regularizer.

WEAKNESSES:
1. Computational cost: training requires to train two models (forward and backward), which comes with additional computational cost.
2. Evaluation is restricted to coding. While there is perplexity eval on natural languages, it is unclear whether lower perplexity translates to gains in model capabilities such as few-shot learning.

QUESTIONS:
Quantify how much additional computational cost there is compared to regular left-to-right pretraining. Ablation of using such additional computation to train the left-to-right longer, and compare results to understand whether the gain reported in this paper comes from computational advantages.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an extension of a paper  \"Fill in the middle\" (FIM)  (Bavarian et al 2022). The idea is to  define different objective function by considering two models for both directions and to \"Meet in the middle\" (MIM). One goal is to be more data efficient and the targeted application is in-filling for code generation.\n\nSTRENGTHS:\nThis paper proposes an extension of a paper  \"Fill in the middle\" (FIM)  (Bavarian et al 2022). The idea is to  define different objective function by considering two models for both directions and to \"Meet in the middle\" (MIM). One goal is to be more data efficient and the targeted application is in-filling for code generation.\n\nWEAKNESSES:\nThe method is simple, but the motivations remain unclear. For\ninstance, the paragraph on FIM should be improved. If a reader does\nnot already know about FIM, it is difficult to understand.  You could\nprovide some equations. Moreover a discussion, on why MIM is simpler\n(and could be better) is necessary.  Moreover my main concern is about\nthe results: at the end why MIM is really more data-efficient that FIM\nor than a BERT-like trained from scratch of the same task ?\n\nQUESTIONS:\nThe first sentence of the abstract sounds really strange: the masked LM objective is used for instance with BERT, and the \"B\" means bidirectionnal. Beyond this lazy remark, a real discussion in the paper on this bidirectional point could be insightful for the reader. In the related work section the authors say that there is large body of work on bidirectional training. What are the real motivation of this new criterion ? What are the differences with BERT MLM, or ELMO ? What is expected in terms of performance ? \n\n- The experimental part focuses on the comparison between FIM and MIM (in the text). The differences with other  models are not really commented ?\n- The models used could be better described: \"a decoder only transformer LM\". Could you precise the number of layers, heads, ... (maybe in an appendix) ?\n\nLIMITATIONS:\nNot really.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new pretraining objective called meet in the middle. This is a bidirectional objective consisting of forward plus backward logprob and an agreement term measured by total variation distance between the two. The forward and backward models share parameters and optionally attention.\n\nInference is done by generating in two directions until they meet in the middle with agreement, potentially introducting a 2x factor in parallelism and reduced latency.\n\nThe model are tested on both code generation accuracy and perplexity on several training set and the pile. They compared to both published baselines training on different data as well as FIM trained on the same data, showing strong results.\n\nSTRENGTHS:\nbidirectional agreement objective that is pretty simple and intuitive in some ways, giving strong results compared to FIM. should be easy enough for others to adopt\n\nresults and comparisons are strong\n\nsome helpful ablations are included\n\nWEAKNESSES:\n\"New paradigm\" is perhaps exaggerated for introducing bidirectional model via sharing and agreement \n\nstill more complexity compared to FIM, and some source of improvements are not very clear. For instance, there are several claims of improved data efficiency due to bidirectionally, but the experiments do not seem to test this directly. If this is the main claim, then some direct left to right comparisons would make more sense.\n\nthe optional improvements seems pretty important and yielded half the improvement over FIM, so perhaps should not be considered optional. For instance, HE infilling results for MIM, MIM-lambda=0, and FIM are respectively 26.3, 24.7 and  22.8. The caption of table 1 says the 26.3 was without the enhancement from 3.2.2 whereas the ablations in table 5 suggest it was.\n\nno comparisons with more recent models like santacoder, starcoder\n\nQUESTIONS:\ngiven that FIM actually conditions on both the prefix and suffix whereas MIM only conditions on one at a time with weak interactions, any discussion on how MIM outperforms despite that?\n\nnot very clear what is the backup when the model fails to meet in the middle and how false positives are dealt with, especially for long context generations.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new pre-training approach called MIM for language models, which promotes agreement between left-to-right and right-to-left models regarding their token distribution for each position. The authors have developed a straightforward and effective infilling inference method that utilizes context from both directions and the tendency of forward and backward models to align. The paper pre-trains language models of different sizes on public code and language data using MIM, and the results of various experiments demonstrate the superiority of MIM over existing baselines.\n\nSTRENGTHS:\n- This paper investigates the bidirectional agreement approach for language models, and the results of various experiments demonstrate the superiority of MIM over existing baselines.\n- The methodology employed in the paper is robust, and the results are statistically significant. The ablation studies conducted in the paper confirm the effectiveness of the primary proposals during both training and inference.\n\nWEAKNESSES:\n- The proposed method is not very novel, as similar ideas have been proposed in the NMT field (treated as conditional language models). However, it is still valuable to investigate the impact of these methods on pure language models.\n- The bidirectional agreement approach leads to more memory cost and training time.\n\nQUESTIONS:\n1. Have you experimented with MIM using larger model scales, such as 7b and 13b? Have you identified any scaling laws for this approach?\n2. The predicted probability distribution from different directions may not the identical due to the different contexts. For instance, predicting the first token from a left-to-right perspective may be challenging, but it may be more accurate when using a right-to-left approach. What is your perspective on this? Could you provide more information on the probability differences observed in various positions?\n3. Some of the methods previously used in the NMT field may be directly used for language models. Have you evaluated the performance of these methods in your experiments?\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}