Orca: Progressive Learning from Complex Explanation Traces of GPT-4
Authors: name: Subhabrata Mukherjee, name: Arindam Mitra, name: Ganesh Jawahar, name: Sahaj Agarwal, name: Hamid Palangi, name: Ahmed Awadallah

Recent research has focused on enhancing the capability of smaller models
through imitation learning, drawing on the outputs generated by large
foundation models (LFMs). A number of issues impact the quality of these
models, ranging from limited imitation signals from shallow LFM outputs; small
scale homogeneous training data; and most notably a lack of rigorous evaluation
resulting in overestimating the small model's capability as they tend to learn
to imitate the style, but not the reasoning process of LFMs. To address these
challenges, we develop Orca (We are working with our legal team to publicly
release a diff of the model weights in accordance with LLaMA's release policy
to be published at https://aka.ms/orca-lm), a 13-billion parameter model that
learns to imitate the reasoning process of LFMs. Orca learns from rich signals
from GPT-4 including explanation traces; step-by-step thought processes; and
other complex instructions, guided by teacher assistance from ChatGPT. To
promote this progressive learning, we tap into large-scale and diverse
imitation data with judicious sampling and selection. Orca surpasses
conventional state-of-the-art instruction-tuned models such as Vicuna-13B by
more than 100% in complex zero-shot reasoning benchmarks like Big-Bench Hard
(BBH) and 42% on AGIEval. Moreover, Orca reaches parity with ChatGPT on the BBH
benchmark and shows competitive performance (4 pts gap with optimized system
message) in professional and academic examinations like the SAT, LSAT, GRE, and
GMAT, both in zero-shot settings without CoT; while trailing behind GPT-4. Our
research indicates that learning from step-by-step explanations, whether these
are generated by humans or more advanced AI models, is a promising direction to
improve model capabilities and skills.