REFINE-AF: A Task-Agnostic Framework to Align Language Models via
  Self-Generated Instructions using Reinforcement Learning from Automated
  Feedback
Authors: name: Aniruddha Roy; name: Pretam Ray; name: Abhilash Nandy; name: Somak Aditya; name: Pawan Goyal
Published: 2025-05-10 07:23:19

Abstract: Instruction-based Large Language Models (LLMs) have proven effective in
numerous few-shot or zero-shot Natural Language Processing (NLP) tasks.
However, creating human-annotated instruction data is time-consuming,
expensive, and often limited in quantity and task diversity. Previous research
endeavors have attempted to address this challenge by proposing frameworks
capable of generating instructions in a semi-automated and task-agnostic manner
directly from the model itself. Many of these efforts have relied on large
API-only parameter-based models such as GPT-3.5 (175B), which are expensive,
and subject to limits on a number of queries. This paper explores the
performance of three open-source small LLMs such as LLaMA 2-7B, LLama 2-13B,
and Mistral 7B, using a semi-automated framework, thereby reducing human
intervention, effort, and cost required to generate an instruction dataset for
fine-tuning LLMs. Furthermore, we demonstrate that incorporating a
Reinforcement Learning (RL) based training algorithm into this LLMs-based
framework leads to further enhancements. Our evaluation of the dataset reveals
that these RL-based frameworks achieve a substantial improvements in 63-66% of
the tasks compared to previous approaches.