Textbooks Are All You Need
Authors: name: Suriya Gunasekar; name: Yi Zhang; name: Jyoti Aneja; name: Caio César Teodoro Mendes; name: Allie Del Giorno; name: Sivakanth Gopi; name: Mojan Javaheripi; name: Piero Kauffmann; name: Gustavo de Rosa; name: Olli Saarikivi; name: Adil Salim; name: Shital Shah; name: Harkirat Singh Behl; name: Xin Wang; name: Sébastien Bubeck; name: Ronen Eldan; name: Adam Tauman Kalai; name: Yin Tat Lee; name: Yuanzhi Li
Published: 2023-06-20 16:14:25

Abstract: We introduce phi-1, a new large language model for code, with significantly
smaller size than competing models: phi-1 is a Transformer-based model with
1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook
quality" data from the web (6B tokens) and synthetically generated textbooks
and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains
pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays
surprising emergent properties compared to phi-1-base, our model before our
finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller
model with 350M parameters trained with the same pipeline as phi-1 that still
achieves 45% on HumanEval.