PanGu-Coder2: Boosting Large Language Models for Code with Ranking
  Feedback
Authors: name: Bo Shen; name: Jiaxin Zhang; name: Taihong Chen; name: Daoguang Zan; name: Bing Geng; name: An Fu; name: Muhan Zeng; name: Ailun Yu; name: Jichuan Ji; name: Jingyang Zhao; name: Yuenan Guo; name: Qianxiang Wang
Published: 2023-07-27 15:28:29

Abstract: Large Language Models for Code (Code LLM) are flourishing. New and powerful
models are released on a weekly basis, demonstrating remarkable performance on
the code generation task. Various approaches have been proposed to boost the
code generation performance of pre-trained Code LLMs, such as supervised
fine-tuning, instruction tuning, reinforcement learning, etc. In this paper, we
propose a novel RRTF (Rank Responses to align Test&Teacher Feedback) framework,
which can effectively and efficiently boost pre-trained large language models
for code generation. Under this framework, we present PanGu-Coder2, which
achieves 62.20% pass@1 on the OpenAI HumanEval benchmark. Furthermore, through
an extensive evaluation on CoderEval and LeetCode benchmarks, we show that
PanGu-Coder2 consistently outperforms all previous Code LLMs.