PanGu-Coder2: Boosting Large Language Models for Code with Ranking
  Feedback
Authors: name: Bo Shen, name: Jiaxin Zhang, name: Taihong Chen, name: Daoguang Zan, name: Bing Geng, name: An Fu, name: Muhan Zeng, name: Ailun Yu, name: Jichuan Ji, name: Jingyang Zhao, name: Yuenan Guo, name: Qianxiang Wang

Large Language Models for Code (Code LLM) are flourishing. New and powerful
models are released on a weekly basis, demonstrating remarkable performance on
the code generation task. Various approaches have been proposed to boost the
code generation performance of pre-trained Code LLMs, such as supervised
fine-tuning, instruction tuning, reinforcement learning, etc. In this paper, we
propose a novel RRTF (Rank Responses to align Test&Teacher Feedback) framework,
which can effectively and efficiently boost pre-trained large language models
for code generation. Under this framework, we present PanGu-Coder2, which
achieves 62.20% pass@1 on the OpenAI HumanEval benchmark. Furthermore, through
an extensive evaluation on CoderEval and LeetCode benchmarks, we show that
PanGu-Coder2 consistently outperforms all previous Code LLMs.