ICE-GRT: Instruction Context Enhancement by Generative Reinforcement
  based Transformers
Authors: name: Chen Zheng, name: Ke Sun, name: Da Tang, name: Yukun Ma, name: Yuyu Zhang, name: Chenguang Xi, name: Xun Zhou

The emergence of Large Language Models (LLMs) such as ChatGPT and LLaMA
encounter limitations in domain-specific tasks, with these models often lacking
depth and accuracy in specialized areas, and exhibiting a decrease in general
capabilities when fine-tuned, particularly analysis ability in small sized
models. To address these gaps, we introduce ICE-GRT, utilizing Reinforcement
Learning from Human Feedback (RLHF) grounded in Proximal Policy Optimization
(PPO), demonstrating remarkable ability in in-domain scenarios without
compromising general task performance. Our exploration of ICE-GRT highlights
its understanding and reasoning ability to not only generate robust answers but
also to provide detailed analyses of the reasons behind the answer. This
capability marks a significant progression beyond the scope of Supervised
Fine-Tuning models. The success of ICE-GRT is dependent on several crucial
factors, including Appropriate Data, Reward Size Scaling, KL-Control, Advantage
Normalization, etc. The ICE-GRT model exhibits state-of-the-art performance in
domain-specific tasks and across 12 general Language tasks against equivalent
size and even larger size LLMs, highlighting the effectiveness of our approach.
We provide a comprehensive analysis of the ICE-GRT, underscoring the
significant advancements it brings to the field of LLM.