SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Methods	HumanEval	MBPP	HE Infilling	MBXP
k	1	10	100	1	10	100	1	EM	1
Incoder-1.3B	8.9	16.7	25.6	11.3	26.8	42.7	8.6	31	9.2
Incoder-6.7B	15.2	27.8	47	19.4	46.5	66.2	14.5	44.1	20.8
CodeGen-Multi-6B	18.2	28.7	44.9	-	-	-	-	-	-
CodeGen-Multi-16B	18.32	32.07	50.80	-	-	-	-	-	-
Codex-2.5B	21.36	35.42	59.5	-	-	-	-	-	-
Codex-12B	28.81	46.81	72.31	-	-	-	-	-	-
code-davinci-001	39	60.6	84.1	51.8	72.8	84.1	-	-	-
code-davinci-002	47	74.9	94.1	58.1	76.7	84.5	51.3	74	57.6
PaLM-8B	3.6	-	18.7	5.0	-	-	-	-	-
PaLM-62B	15.9	-	46.3	21.4	-	-	-	-	-
PaLM-540B	26.2	-	76.2	36.8	-	-	-	-	-
LLaMA-7B	10.5	-	36.5	17.7	-	-	-	-	-
LLaMA-13B	15.8	-	52.5	22.0	-	-	-	-	-
LLaMA-33B	21.7	-	70.7	30.2	-	-	-	-	-
LLaMA-65B	23.7	-	79.3	37.7	-	-	-	-	-
FIM-350M	12.8	16.7	27.8	14.8	30.2	44.5	11.8	37.6	14.3
FIM-1.3B	20.8	39.4	51.7	25.9	45.6	62.5	15.7	42.2	22.1
FIM-2.7B	28.5	45.6	67.8	38.2	61.2	76.1	22.8	51.7	30.4
MIM-350M	13.7	17.2	28.5	16.5	33.7	47.4	14.6	41.7	16.4
MIM-1.3B	22.4	41.7	53.8	26.8	47.6	65.1	17.4	47.6	24.5
MIM-2.7B	30.7	48.2	69.6	42.2	64.8	79.3	26.3	57.8	35.7
Table 1: pass@k (%) on the HumanEval, MBPP and HumanEval Infilling and MBXP benchmarks. Additionally, Exact Match (EM) metric  $\left[\mathrm{FAL}^{+}22\right]$  is reported for HumanEval Infilling. FIM is the baseline 'Fill in the Middle' (FIM). MIM is our proposed method with the enhancements from section 3.2.2. For reference, we report evaluation numbers of other models, namely Incoder-1.3B and Incoder-6.7B  $\left[\mathrm{FAL}^{+}22\right]$ , Codex-12B  $\left[\mathrm{CTJ}^{+}21\right]$ , code-davinci-001 and 002. Note that, Codex-12B, davinci-001, PaLM  $\left[\mathrm{CND}^{+}22\right]$  and LLaMA  $\left[\mathrm{TLI}^{+}23\right]$  were trained only with left-to-right autoregressive objective, thus, cannot perform infilling.
Methods	C++	Java	Go
k	1	10	100	1	10	100	1	10	100
Incoder-6B	10.0	20.0	35.0	9.0	19.0	40.0	8.0	14.0	29.0
CodeGen-6B	12.0	20.0	36.0	15.0	18.0	40.0	9.0	22.0	40.0
CodeGen-16B	18.0	30.0	50.0	15.0	38.0	60	13.0	25.0	47.0
CodeGeeX-13B	20.0	31.0	50.0	16.0	38.0	58.0	15.0	25.0	49.0
FIM-350M	8.5	18.3	24.3	11.3	21.5	27.6	10.2	16.8	31.4
FIM-1.3B	16.7	31.5	43.2	15.4	25.2	32.6	12.5	23.7	39.6
FIM-2.7B	24.5	38.6	51.3	18.3	28.6	38.7	15.2	30.4	50.2
MIM-350M	10.2	19.6	26.3	11.1	22.4	28.2	10.8	17.5	31.8
MIM-1.3B	19.3	36.5	45.7	17.6	27.4	34.7	13.6	25.1	41.7
MIM-2.7B	27.4	41.3	54.1	21.6	30.8	39.1	17.4	32.7	53.6
Table 2: pass@k (%) results on HumanEval-X in the zero-shot settings for the baselines FIM and our MIM approach. Results of  $k = 1$ , 10 and 100 are reported across all categories.
perplexity across all datasets in both settings. For example, MIM-2.7B model obtains a perplexity of 9.54 in OpenText dataset, a relative  $19.9\%$  reduction in perplexity over FIM-2.7B model, which obtains a perplexity of 11.92. In the Pile dataset  $\left[\mathrm{GBB}^{+}21\right]$ , which represents the out-of-domain setting, MIM-2.7B also outperforms FIM-2.7B by a relative reduction of  $15.3\%$  in perplexity (9.24 vs 10.92), which further reinforces the advantages of MIM pre-training for natural languages.
4.3.3 Natural Language Benchmarks
We trained a MIM model with a total of 1.3B parameters, which shares architectural details with  $\mathrm{[LBE^{+}23]}$ , on a filtered version of the Falcon RefinedWeb corpus, comprising a total of 88B tokens. This model was subsequently compared with the baseline model, phi-1.5-web-only in  $\mathrm{[LBE^{+}23]}$
Methods	Introductory	Interview	Competition
k	1	10	100	1	10	100	1	10	100
FIM-350M	3.6	7.8	11.5	0.0	0.3	1.2	0.0	0.04	0.9
FIM-1.3B	8.2	11.6	17.4	0.12	0.59	1.9	0.01	0.07	1.7
FIM-2.7B	12.4	15.7	20.8	0.27	0.72	2.4	0.03	0.095	2.4
MIM-350M	4.7	9.2	14.3	0.2	0.51	2.3	0.02	0.06	1.4
MIM-1.3B	10.6	14.2	21.2	0.36	0.76	3.6	0.043	0.09	2.2
MIM-2.7B	14.3	18.2	24.6	0.52	1.4	5.2	0.067	0.18	3.3
Table 3:  $\text{pass}@\mathbf{k}$  (%) results on the APPS benchmarks in the zero-shot settings for the baselines FIM and our MIM approach. Results of  $k = 1, 10$  and 100 are reported across all categories.
Methods	Datasets
MODELS	CC-NEWS	OPENWEBTEXT	CC-STORIES	CC-100	THE PILE
FIM-350M	21.16	17.91	20.89	14.23	15.14
FIM-1.3B	18.78	13.28	17.52	11.34	12.48
FIM-2.7B	13.45	11.92	13.43	9.43	10.92
MIM-350M	19.43	17.23	18.75	13.45	13.97
MIM-1.3B	16.04	12.23	14.63	11.16	10.79
MIM-2.7B	11.17	9.54	11.35	8.76	9.24
Table 4: Perplexity on all datasets in in-domain setting (CC-News, OpenWebText, CC-Stories, & CC-100 held-out datasets), and perplexity results in out-of-domain setting (The Pile [GBB+21])
which was trained on identical data using an autoregressive left-to-right objective. Tables 5 and 6 summarize the results. On commonsense reasoning MIM-1.3B-web-only outperforms by  $0.42\%$  on average while on knowledge understanding it outperforms by  $1.2\%$  on average. Overall, (combining the two tables) MIM improves by  $0.76\%$  on average.
4.4.1 Effect of Infilling Specific Enhancements
In this section, we perform an ablation study to qualitatively assess the effect of the enhancements we proposed in section 3.2.2 for the infilling task. We compare the purely autoregressive MIM model  $(\lambda = 0)$  where the LMs are not allowed to observe generated tokens from the opposite side, and only utilize context from their own side during generation. We contrast this with using the Synchronous Bidirectional Attention layer with  $\lambda = 0.3$  that conditions on previously generated tokens from both sides. We used perplexity on the validation data to select the value  $\lambda$ . We reuse the same value of  $\lambda$  during infilling to avoid any potential mismatch between training and inference.
We conduct an experiment on HumanEval Infilling  $\left[\mathrm{BJT}^{+}22\right]$  and results are summarized in Table 7. We notice that models that directly incorporate bidirectional context always outperform models that only utilize unidirectional context across all model sizes. As always, this is at the expense of the forward model no longer being a drop-in replacement for standard autoregressive LMs.
4.4.2 Effect of Agreement Regularizer
While MIM performs better than FIM it's still possible that this difference is simply due to the parameters being trained on two tasks (left-to-right and right-to-left). To preclude this, we perform an ablation study where we remove the token-level agreement regularizer of Section 3.1 during training while keeping the infilling enhancements  $(\lambda = 0.3)$ . We show that encouraging agreement during training helps improve the infilling performance of our models in the HumanEval Infilling benchmark  $\mathrm{[BJT^{+}22]}$ , as summarized in Table 8. Comparing with Table 7, we see that models trained without token-level agreement regularization in general perform worse than models that do not utilize bidirectional context  $(\lambda = 0)$ , which further emphasizes the importance of our agreement regularizer in making the predictions consistent between forward and backward directions.
Methods	Datasets
MODELS	WINOGRANDE	ARC-EASY	ARC-CHALLENGE	BOOLQ	SIQA
phi-1.5-web-only	0.604	0.666	0.329	0.632	0.414
MIM-1.3B-web-only	0.591	0.687	0.337	0.628	0.423
Table 5: Results of MIM in standard commonsense reasoning benchmarks, including WinoGrande, Arc-Easy, Arc-Challenge, BoolQ and SIQA, vs. the baseline phi-1.5-web-only in  $\left[\mathrm{LBE}^{+}23\right]$ .
Methods	Datasets
MODELS	PIQA	HELLASWAG	MMLU	OPENBOOKQA
phi-1.5-web-only	0.743	0.478	0.309	0.274
MIM-1.3B-web-only	0.736	0.491	0.327	0.298
Table 6: Results of MIM in standard knowledge understanding benchmarks, including PIQA, HellaSwag, MMLU and OpenbookQA, vs. the baseline phi-1.5-web-only in  $\left[\mathrm{LBE}^{+}23\right]$ .
Methods	HE Infilling	MBXP
MODELS	λ	pass@1	EM	pass@1
MIM-350M	0.0	12.5	38.6	14.1
0.3	14.6	41.7	16.4
MIM-1.3B	0.0	15.6	45.2	21.7
0.3	17.4	47.6	24.5
MIM-2.7B	0.0	24.7	54.3	32.4
0.3	26.3	57.8	35.7
Table 7: pass@1 (%) results of MIM with  $(\lambda = 0.3)$  and without  $(\lambda = 0.0)$  bidirectional context on HumanEval Infilling and MBXP. Exact match results on HumanEval Infilling are also reported.
Methods	HE Infilling	MBXP
MODELS	Regularizer	pass@1	EM	pass@1
MIM-350M	no reg	11.8	35.9	13.5
+ reg	14.6	41.7	16.4
MIM-1.3B	no reg	13.8	42.7	22.7
+ reg	17.4	47.6	24.5
MIM-2.7B	no reg	23.2	52.5	30.6
+ reg	26.3	57.8	35.7
Table 8: pass@1  $(\%)$  results of MIM with and without agreement regularizer on HumanEval Infilling and MBXP. Exact match results on HumanEval Infilling are also reported.
4.5 Efficiency
4.5.1 Training Efficiency
At first glance, one might hypothesize that our proposed methodology could potentially necessitate twice the computational resources in comparison to the conventional left-to-right autoregressive training, with most of the performance gains attributed to this additional computation. Contrary to this supposition, empirical evidence gathered from our experiments suggests a different narrative. Our Meet-in-the-Middle (MIM) strategy exhibits performance metrics that are not only commensurate but often superior to those of the left-to-right autoregressive training. This assertion is corroborated by a comparative analysis of the validation log perplexity of MIM models and left-to-right autoregressive models across different model sizes. All models were trained on the filtered Falcon RefinedWeb corpus  $\left[\mathrm{LBE}^{+}23\right]$ . As illustrated in Figures 3 and 4 in the Appendix, when assessed under the same computational budget (either FLOP count or wall clock time), MIM models consistently outperform left-to-right autoregressive models across all relevant computational budgets. Besides, when comparing MIM models and left-to-right autoregressive models on a time-basis, we still observe major speedups over the baselines.
4.5.2 Inference Efficiency
Finally we assess the efficiency of our infilling procedure, contrasting it with FIM through inference latency across various batch sizes. Figure 2 shows that MIM achieves a speedup over FIM baselines in both single and half precision, using a 1.3B model, across all HumanEval Infilling examples. MIM-1.3B specifically performs  $4\%$  to  $6\%$  faster than FIM-1.3B in single precision, and  $3\%$  to  $5\%$  faster in half precision. This is due to MIM leveraging available parallelism better than FIM.
There is an extensive body of work on bidirectional language modeling. Early work such as ULMFiT [HR18], and ELMo  $\left[\mathrm{PNI}^{+}18\right]$  kept the autoregressive nature of the LM, while later work such as BERT [DCLT19], T5  $\left[\mathrm{RSR}^{+}20\right]$ , and SpanBERT  $\left[\mathrm{JCL}^{+}20\right]$  focused on representation learning and was non-autoregressive. XLNET  $\left[\mathrm{YDY}^{+}19\right]$ , on the other hand, utilizes bidirectional context during training by the permutation language modeling objective, which maximizes the likelihood over all factorization orderings of the training sequences. However, because these later models typically focus on representation learning, in-context learning via prompting can be difficult  $\left[\mathrm{PLR}^{+}22\right]$ .
Two works that train neural models using similar ideas are  $\mathrm{[SKS^{+}18]}$  and  $\mathrm{[ZWL^{+}19]}$ . In the former the authors train RNNs for real-time acoustic modeling. This constraint is very similar to our desire of having a forward model that can generate the next token from the previous ones. However, they propose to regularize the forward and backward RNNs by requiring their representations to be close in Euclidean distance. We suspect that this constraint may be unnecessarily stringent, and trading it off against the LM's perplexity during training could hurt overall performance.
On the other hand,  $\left[\mathrm{ZWL}^{+}19\right]$  encourage agreement in probability space. However, they are interested only in neural machine translation and only encourage agreement of probabilities at the output sequence level. In contrast, we encourage agreement on every token.
Sharing all parameters between different factorizations of the sequence was first proposed in XLNET [YDY  $^{+}19]$  but has also been used with only forward and backward models in the context of image captioning  $\mathrm{[ZHL^{+}22]}$ . These works motivated us to share all parameters between our two LMs.
Using LMs for infilling was first proposed in [DLL20] where the authors tackled the more challenging setup with multiple blanks.  $\left[\mathrm{BJT}^{+}22\right]$  applies "Fill in the Middle" (FIM) to the training data by randomly splitting each training instance into a tuple of (prefix, middle, suffix) and concatenate these sections into a single example together with their sentinel tokens after tokenization.  $\left[\mathrm{BJT}^{+}22\right]$  also contains a thorough section on research related to infilling. We refer the reader there for more details.
Furthermore,  $\mathrm{[AHR^{+}22]}$  and  $\mathrm{[FAL^{+}22]}$  propose an extension of FIM, namely "Causal Masked Language Modeling" (CM3) and explore multi-region infilling problem. Similar to us,  $\mathrm{[BJT^{+}22]}$  and  $\mathrm{[FAL^{+}22]}$  leverage FIM to pre-train decoder-only LMs on code data and evaluate their models on zero-shot code completion benchmark created from HumanEval dataset  $\mathrm{[CTJ^{+}21]}$ .
In [SLB17] the authors propose bidirectional beam search that also employs forward and backward LMs. However their focus is in improving accuracy at the expense of latency. Their procedure performs multiple passes over the completion, fixing tokens one-by-one. Our approach focuses on achieving better infilling accuracy than FIM while also reducing latency.
This paper presented "Meet-in-the-Middle" a novel method devised to address two significant challenges in large LMs: pre-training data efficiency and effective context handling in infilling tasks. By using both forward and backward LMs that share parameters and are trained for mutual agreement and next token prediction, the method significantly enhances the performance of existing autoregressive LMs.
We would like to thank Dejian Yang, Jian-Guang Lou, Yanzhi Li at Microsoft Research for helping us with training data preparation. We also sincerely thank Daniel Fried and Armen Aghajanyan at Meta AI for helpful discussion regarding the training details of FIM baselines.
[AGW+22] Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati,
$\left[\mathrm{RSR}^{+}20\right]$  Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-140:67, 2020.
[Sha19] Noam Shazeer. Fast transformer decoding: One write-head is all you need. CoRR, abs/1911.02150, 2019.
$\left[\mathrm{SKS}^{+}18\right]$  Dmitriy Serdyuk, Nan Rosemary Ke, Alessandro Sordoni, Adam Trischler, Chris Pal, and Yoshua Bengio. Twin networks: Matching the future for sequence generation. In International Conference on Learning Representations, 2018.
[SLB17] Qing Sun, Stefan Lee, and Dhruv Batra. Bidirectional beam search: Forward-backward inference in neural sequence models for fill-in-the-blank image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6961-6969, 2017.
$\left[\mathrm{SPP}^{+}19\right]$  Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. CoRR, abs/1909.08053, 2019.
$\left[\mathrm{TLI}^{+}23\right]$  Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023.
[VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[  $\mathrm{YD}\mathrm{Y}^{+}19]$  Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764, 2019.
$\left[\mathrm{ZHL}^{+}22\right]$  Yuanen Zhou, Zhenzhen Hu, Daqing Liu, Huixia Ben, and Meng Wang. Compact bidirectional transformer for image captioning. arXiv preprint arXiv:2201.01984, 2022.
$\left[\mathrm{ZWL}^{+}19\right]$  Zhirui Zhang, Shuangzhi Wu, Shujie Liu, Mu Li, Ming Zhou, and Tong Xu. Regularizing neural machine translation by target-bidirectional agreement. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 443-450, 2019.
[ZZZ19] Long Zhou, Jiajun Zhang, and Chengqing Zong. Synchronous bidirectional neural machine translation. Trans. Assoc. Comput. Linguistics, 7:91-105, 2019.