How to Leverage Demonstration Data in Alignment for  Large Language Model? A Self-Imitation Learning Perspective
Authors: Teng Xiao, Mingxiao Li, Yige Yuan, Huaisheng Zhu, Chao Cui, Vasant G Honavar


Reviewer paper summaries: This paper introduces a novel framework called Generalized Self-Imitation Learning (GSIL), an effective and efficient method for aligning Large Language Models (LLMs). GSIL optimizes the model using offline demonstration data to enhance its performance on specific tasks.

The authors argue that while LLMs acquire extensive knowledge through pre-training, they may not meet the requirements of specific tasks. Therefore, post-training processes (alignment or fine-tuning) are necessary to improve the model's performance on these tasks. Current alignment methods mainly include Supervised Fine-Tuning (SFT) based on demonstration data and Preference Fine-Tuning based on preference data.

The authors propose a new imitation learning framework, GSIL, which uses an alternative objective with density ratio estimation to leverage self-generated data, simplifying the optimization of the imitation learning objective and eliminating the need for complex adversarial training required in standard imitation learning. The GSIL framework can learn effective strategies from human demonstration data and achieve fast and simple fine-tuning similar to SFT. Additionally, GSIL offers a unified perspective for handling alignment with demonstration data and can be implemented with a class of general convex functions.

Experiments on several challenging benchmarks, such as coding, mathematical reasoning, and instruction following, show that GSIL consistently and significantly outperforms baseline methods. The authors also study the performance of GSIL under different hyperparameters and explore the impact of various loss functions on the model's performance.
Summary of strengths: 1. In the Method section, the theoretical derivation of the proposed GSIL method is clear and easy to understand. GSIL has three main advantages over existing methods: GSIL achieves lightweight and efficient model fine-tuning by avoiding complex adversarial training, making the alignment process simpler and faster; GSIL can use different density ratio estimation loss functions, such as Logistic, Hinge, Brier, etc., providing flexibility for model alignment and allowing it to adapt to various tasks and requirements; GSIL offers a unified perspective for handling alignment with demonstration data, linking imitation learning with density ratio estimation, enabling the use of simple classification methods to address complex alignment problems.

2. In the Experiment section, GSIL significantly outperforms traditional Supervised Fine-Tuning (SFT) and Self-Imitation Learning (SPIN) methods, demonstrating its effectiveness in tasks such as coding, mathematical reasoning, and instruction following. Additionally, GSIL performs excellently in generating safe and harmless responses, which is crucial for applications requiring the generation of high-quality and safe content.
Summary of weaknesses: 1. GSIL introduces additional hyperparameters (such as β and γ) that need to be manually adjusted to achieve optimal performance, which may increase the complexity and time cost of experiments.

2. Although the authors illustrate in Figure 1 the differences between GSIL's core Reverse KL and the conventional Forward KL used by other methods, stating that "forward KL encourages all responses in datasets to have equal probability, leading to an overestimation of the long tail of the target distribution, whereas reverse KL sharpens the probability mass on certain high-quality regions" (line 258), there are no explicit comparative experiments in the subsequent Experiment section to substantiate this point. The classification results for different types of problems shown in Figure 3 do not distinctly highlight the unique advantages of Reverse KL.

3. According to the main experimental results in Table 2, while GSIL achieves good results with certain density ratio estimation loss functions, there are instances where the results are even inferior to the baseline SFT (for example, in the ARC task, SFT achieves 58.28, surpassing many GSIL settings). The authors lack an in-depth explanation for this phenomenon.
Comments, suggestions and typos: 1. For some typos: line 426 "Figure 1" does not include a link to Figure 1; line 511 I think you might be missing a "use" in the phrase "we the".

2. I am very interested in Figure 2. Can this figure be interpreted as: the GSIL W/ Logistic method allows rewards and real part to continuously benefit model training, whereas the SPIN method does not? Could the authors provide more interpretation on this?
Overall assessment scores: 3.5