Unlock the Correlation between Supervised Fine-Tuning and Reinforcement Learning in Training Code Large Language Models
Authors: Jie chen, Xintian Han, Yu Ma, Xun Zhou, Liang Xiang


Reviewer paper summaries: To study the effect of supervised finetuning (SFT) and reinforcement learning (RL) in post-training of code LLMs, they manually create a dataset of 100 Python functions (called atom functions) and also multiple datasets of composite functions by composition of the atom functions. With such datasets, they are able to study the effect of SFT and RL on mixtures of these datasets, and the generalization capability by properly splitting the train/validation sets constructed from different splits of atom functions. Some interesting findings are: 1) SFT with a mixture of both atom functions and compositions datasets is required to perform well on an out-of-distribution compositions dataset. 2) Continuing to do RL can further improve the generalization performance. 3) SFT + RL perform the best on the target domain with the out-of-distribution compositions dataset constructed in this paper but doesn't perform the best on very out-of-distribution datasets like HumanEval and MBPP.
Summary of strengths: * By manually creating the dataset of atom functions and composite functions, they are able to study the effect of SFT and RL on the mixture of such datasets and the generalization capability of the model, and are able to draw some interesting findings.
* The findings are interesting and potentially useful for the community to understand the effect of SFT and RL in post-training of code LLMs.
Summary of weaknesses: * The finetune dataset used in the paper may not be large enough compared to finetune datasets used in practice.
* The paper studies a specific kind of scenario where the dataset is constructed by composition of atom functions; it would be interesting to see how the results generalize to other kinds of more noisy and diverse datasets.
Comments, suggestions and typos: * What are the results for the untuned models on the composite_c dataset?
* The results with various settings of fine-tuned models are tested on composite_c. Is there a result of fine-tuning in-distribution, i.e., fine-tuning on a set of data generated with the same setting as composite_c to understand the performance of fine-tuned models in-distribution?
* The plots shown in Fig 6 and Fig 7 both have the same legend showing the mixture of composite_A and atom_base, but Fig 7 is referred to as training on composite_b and atom_base. Is this intended?
* In Fig 2, the result with composite_A(1%) and Atom_base(600%) after training for 400 steps does not seem to saturate yet. Is it possible that the performance of the model will continue to improve if trained longer and achieve similar performance as the SFT+RL model?
Overall assessment scores: 3