Does Instruction Tuning Reduce Diversity? A Case Study Using Code Generation

Authors: Alexander Shypula, Shuo Li, Botong Zhang, Vishakh Padmakumar, Kayo Yin, Osbert Bastani
Primary area: interpretability and explainable AI
Keywords: LLM, Diversity, RLHF, DPO, SFT, Program Synthesis, Evaluation

Abstract:
Large Language Models (LLMs) should ideally generate diverse content for open-ended prompts (e.g., variety in cooking recipes). Anecdotal evidence has suggested that preference-tuned language models struggle to generate diverse content, which would have important implications for how we align models. However, research on this question has been limited by the difficulty of measuring diversity, which naively would require costly human evaluation. We propose to leverage code as a means to study semantic diversity, since code has executable semantics. To this end, we create an open-ended program synthesis task, enabling us to cheaply evaluate the diversity of hundreds of thousands of generations. Using our methodology, we find that while instruction-tuning reduces syntactic and lexical diversity, it can actually increase semantic diversity. We also study the effect of model size and prompting technique on diversity. Finally, we find that neural diversity metrics correlate poorly with our semantic diversity metrics, highlighting the need for more rigorous methodologies for evaluating diversity.

Decision: reject
Reviewer scores: 3, 5, 6, 1

Reviews and discussion:
<thread>
creation date:
1730143859969

thread_type:
official_review

<thread_object>
creator: reviewer_aset

<content>
summary:
The authors conduct a case study on how instruction tuning affects the diversity of generated programs. Through the study, the authors find that instruction tuning reduces the lexical and syntactic diversity of generations but increases the semantic diversity of generations.

strengths:
There are several strengths of the paper:
- The overall structure is clear and easy to follow.
- The authors conduct experiments on several Llama models.
- The authors evaluate the diversity from multiple dimensions.

weaknesses:
The contributions and the setups are very limited, specifically:
- There is no evaluation of correctness metrics. Representative benchmarks like HumanEval [1] and BigCodeBench [2] should be included.
- Neural diversity like CodeBERTScore is not very appropriate here. Prior works like ICE-Score [3] show that CodeBERTScore is very weak compared to the evaluators based on LLMs as Judges. For example, the authors should consider using ICE-Score instead of CodeBERTScore.
- The conclusion that "Program semantics may be harder to model than natural language." is a bit wrong, as BERTScore itself is not robust [4-6]. There is no reason why CodeBERTScore should accurately reflect the program's correctness.
- The open models included in the evaluation are not diverse. The authors should use other Code LLMs, such as StarCoder2 [7] and CodeQwen [8].
- The training setups are not documented, and the results could be questionable.
- The experiments are Python-only. The authors should conduct evaluations on more programming languages.
- There is no related work on Code LLMs, code generation, and coding benchmarks.


[1] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ... & Zaremba, W. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.

[2] Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., ... & Von Werra, L. (2024). Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.


[3] Zhuo, T. Y. (2024, March). ICE-Score: Instructing Large Language Models to Evaluate Code. In Findings of the Association for Computational Linguistics: EACL 2024 (pp. 2232-2242).

[4] Hanna, M., & Bojar, O. (2021, November). A fine-grained analysis of BERTScore. In Proceedings of the Sixth Conference on Machine Translation (pp. 507-517).

[5] Sun, T., He, J., Qiu, X., & Huang, X. J. (2022, December). BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 3726-3739).

[6] Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., ... & Zhou, J. (2023, December). Is ChatGPT a Good NLG Evaluator? A Preliminary Study. In Proceedings of the 4th New Frontiers in Summarization Workshop (pp. 1-11).

[7] Lozhkov, A., Li, R., Allal, L. B., Cassano, F., Lamy-Poirier, J., Tazi, N., ... & de Vries, H. (2024). Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173.

[8] Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., ... & Zhu, T. (2023). Qwen technical report. arXiv preprint arXiv:2309.16609.

questions:
- What's the column N in the tables?
- There is a lack of motivation to conduct investigations on code generation over text generation. The authors should provide sufficient explanations.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730260956400

thread_type:
official_review

<thread_object>
creator: reviewer_liwb

<content>
summary:
The authors proposed a novel strategy for studying semantic diversity by focusing on code generation, illustrating some interesting properties of the output diversity for LLM open-ended generation.

strengths:
1. The semantic diversity and other diversity measurement is crucial for openended tasks to evaluate the generation of models in terms of creativity, randomness, etc. 
2. The metric devised by the authors makes for measuring model output diversities. 
3. The most interesting part is the insights coming from the experiment results, offering fresh arguments for openended code generation diversity.

weaknesses:
The major concerns are the meaningfulness of studying diversity and the generalizability of conclusions to general-domain openended generations.

1. Why is it meaningful to study diversity for LLM open-ended generation? From my perspective, it may contribute to more effective solution searching if we scale up the inference computation (like o1). But this is not discussed in this work, so whether only studying the diversity is  meaningful remains uncertain.
2. Code generation is a special case in open-ended generation: it usually has a "correct" output. As the semantic diversity is measured by considering the model output, then why it matters when it generates diverse outputs while only one of them is correct? Isn't it true that we only care about whether it covers the correct output?
3. Whether the conclusions obtained generalizable to the general-domain openended generation? I think it's hard to evaluate, and at the same time, the answer might be no.

questions:
Can you elaborate more on the benefit of higher response diversity other than some applications like creative writing?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730396101983

thread_type:
official_review

<thread_object>
creator: reviewer_7jxp

<content>
summary:
This work explores the impact of instruction tuning on the ability of Large Language Models (LLMs) to create diverse outputs. For their empirical analysis, they focused on code generation tasks, which allow for automatic verification of correctness via code execution. To measure code diversity, they introduced a holistic evaluation that encompasses semantic, lexical, syntactic, and neural diversity. The key contribution of this study is a methodology for evaluating LLMs' capacity to generate diverse code outputs. Additionally, they investigated the relationship between model size and diversity metrics.

strengths:
1. The research focuses on a relevant matter, as there are few established evaluation frameworks in place to assess the capabilities of large language models (LLMs) in terms of diversity.

2. The study is not only theoretical but also includes several empirical demonstrations, providing tangible evidence.

weaknesses:
1. The dataset construction could benefit from more detailed explanations.
2. The overall presentation of the study requires improvement to enhance clarity.
3. The study's findings regarding the research question, "Does instruction tuning reduce diversity?", are not conclusive. 
4. The contribution appears insufficient for a comprehensive paper, but it could form the basis of a shorter piece, such as a research agenda exploring this topic further.

questions:
### Comments and suggestions on Soundness
The authors utilized 21 handcrafted abstractions derived from CodeNet, along with corresponding test cases sourced from AlphacaCode, as the foundation for the dataset. Unfortunately, further specifics are inadequately detailed in the appendix, making it challenging to evaluate the integrity of the data collection procedure.

### Comments and suggestions about Presentation
1. The overall paper outline needs improvements. For example, as the study requires of multiple concepts and terminology to set the background (e.g., approaches to measure LLMs diversity, types of LLMs, instruction-tuning types, NLP techniques), I suggest including a stronger Background/Related Work section in the main paper. As a second example, I suggest creating a flow chart figure that clearly paints the big picture followed to create the dataset. 
2. Grammar and Narrative. The paper needs general proofreading to improve quality. These are a few typos that I took note of: ‘the the’ (line 389), ‘We’ (line 296).

### Questions
1. Why are you presenting this work as a case study?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698200074

thread_type:
official_review

<thread_object>
creator: reviewer_kwb6

<content>
summary:
This paper examines the impact of instruction tuning (via supervised finetuning (SFT) or preference-tuning (PT)) on output diversity, using code as a medium for semantic diversity through test case execution. Despite limitations in experimental scope (e.g., limited problem set, comparison using different baselines for SFT and PT), the study presents three notable findings: (1) optimal temperature values for achieving semantic diversity in code, (2) both SFT and PT increase semantic diversity, and (3) PT achieves greater semantic diversity than SFT.

strengths:
- Figure 1’s temperature plot reveals potential improvement over CodeT's [1] default T=0.8, suggesting that T=0.9 or T=1.0 may yield higher diversity. This deserves further elaboration.

[1] Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, & Weizhu Chen (2023). CodeT: Code Generation with Generated Tests. In The Eleventh International Conference on Learning Representations.

weaknesses:
- The experimental results are limited to 21 competitive programming problems, raising concerns about generalizability.
- The comparison between CodeLLama (based on LLama2) and LLama3(.1) for SFT vs. PT may be problematic, as the use of distinct model versions could affect the observed diversity differences. A controlled experiment using LLama3(.1)-based SFT would improve reliability.
- Instruction-tuned LLM diversity could be influenced by the training set’s diversity. It would be beneficial to include a controlled comparison of LLMs without instruction tuning and LLMs with tuning that vary in training set or trajectory diversity.
- The finding that execution-based diversity differs from lexical/syntactic diversity is unsurprising. Prior work (e.g., APPS [2]) has documented similar discrepancies between lexical-based (BLEU) and execution-based metrics. Additionally, Section 4.2 reiterates results seen in counterfactual code augmentation research [3,4]. Further, the statement on lines 327-338 about the difficulty of semantic differentiation in code contrasts with the use of code to study semantic diversity.
- CodeBERTScore is outdated. More current options include UniXCoder [5] or CodeExecutor [6]. Another approach could leverage LLM-based evaluation, such as in CHASE-SQL [7], which employs a fine-tuned LLM as a code-pair verifier.

[2] Hendrycks, D., Basart, S., Kadavath, S., Mazeika, M., Arora, A., Guo, E., Burns, C., Puranik, S., He, H., Song, D., & Steinhardt, J. (2021). Measuring Coding Challenge Competence With APPS. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks.

[3] Cito, J., Dillig, I., Murali, V., & Chandra, S. (2022). Counterfactual explanations for models of code. In Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice (pp. 125–134). Association for Computing Machinery.

[4] Hojae Han, Minsoo Kim, Seung-won Hwang, Nan Duan, and Shuai Lu. 2023. Intervention-Based Alignment of Code Search with Execution Feedback. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2241–2263, Singapore. Association for Computational Linguistics. 

[5] Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7212–7225, Dublin, Ireland. Association for Computational Linguistics.

[6] Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. 2023. Code Execution with Pre-trained Language Models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4984–4999, Toronto, Canada. Association for Computational Linguistics.

[7] Pourreza, Mohammadreza, et al. "CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL." arXiv preprint arXiv:2410.01943 (2024).

questions:
- Could you elaborate more regarding the best semantic diversity in Figure 1?
- Could the number of problems be increased to enhance generalizability?
- Could SFT and PT be compared on identical base models, for instance, by adding an SFT baseline trained on LLama3.1-base?
- Could a controlled comparison be added for LLMs with and without instruction tuning, varying training set or trajectory diversity?
- Could CodeExecutor be adopted for neural diversity evaluation?
- Could an LLM-based neural diversity metric, such as in CHASE-SQL, be added?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733293737192

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
As the review period ends, we would like to thank all the reviewers for their thoughtful feedback and time. 

During our discussion period, multiple reviewers recognized the significant contributions of our work:

**Reviewer 7Jxp noted both the critical need and empirical rigor of our work:** 
- *"The research focuses on a relevant matter, as there are few established evaluation frameworks in place to assess the capabilities of large language models (LLMs) in terms of diversity"* 
-  *"The study is not only theoretical but also includes several empirical demonstrations, providing tangible evidence."*

**Reviewer LiwB emphasized both the methodological and empirical value:** 
- *"The semantic diversity and other diversity measurement is crucial for openended tasks to evaluate the generation of models in terms of creativity, randomness, etc."* 
-  *"The most interesting part is the insights coming from the experiment results, offering fresh arguments for openended code generation diversity."*

## *We have significantly strengthened the technical depth and rigor of our work through multiple substantial additions:*

### 1. Enhanced Evaluation Framework: 

- We’ve added ICE-Score evaluations using GPT-4o as suggested by Reviewers kwb6 and ASEt. This adds more robustness to and further reinforces our findings on the nature of semantic diversity measurement vs. neural and lexical diversity metrics

### 2. Expanded Model Coverage: 
- Incorporated QWen-Coder 2.5 (7B Base, 7B Instruct, 34B Base, 34B Instruct) providing additional validation of our key findings
- New SFT Experiments: Conducted new fine-tuning experiments on Llama3, Llama3.1, and CodeLlama-7b-Python using Magicoder-OSS-Instruct-75K
 - Comprehensive Correctness Metrics: Added extensive HumanEval and MBPP scores for models evaluated 

### 3. Rigorous Documentation: 

- Added 2-pages of detailed dataset construction documentation with flowcharts and thorough process explanations

These additions strengthen our key finding that challenges the dominant narrative about preference tuning (i.e. RLHF or DPO) reducing diversity:

- The QWen-Coder results align with our findings from LLama3.1, providing critical validation across model families
ICE-Score evaluations demonstrate that even advanced LLM-based metrics struggle to capture semantic diversity, reinforcing the value of our execution-based methodology

- Our new SFT experiments provide controlled comparisons and add further robustness by controlling for base-model comparisons

## *To our knowledge, we have systematically addressed major reviewer concerns that were brought up:*

- Shared concern (kwb6, ASEt) about neural metrics: Added ICE-Score evaluation using GPT-4o
- Reviewer kwb6's request for controlled SFT experiments: Completed with three new model variants
- Reviewer ASEt's concern about correctness metrics and model coverage: Added comprehensive HumanEval/MBPP results and QWen-Coder-2.5 results
- Reviewer 7Jxp's request for documentation: Added two pages of detailed methodology including flowcharts
- Reviewer LiwB's questions about motivation: Expanded background section with concrete applications and demonstrations about importance

Given the additional experiments requested, we intend to increase sample size per problem to 200 and naturally to ensure our langauge is polished. In our work, we challenge the dominant narrative that methods like RLHF or DPO universally reduce diversity. Our expanded experiments and analysis provide evidence that traditional diversity metrics may lead to incomplete conclusions. Like in prior work, we find that PPO/DPO reduces lexical diversity; however, we find that it may actually increase semantic diversity (diversity of meaningful content, and not just superficial form). These findings, supported by our novel execution-based methodology and comprehensive evaluations across multiple model families, represent an important contribution to our understanding of how instruction tuning affects model behavior.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates how instruction tuning affects diversity in code generation outputs from large language models (LLMs). The main findings show that preference tuning can increase semantic diversity while reducing lexical/syntactic diversity, challenging existing beliefs about instruction tuning's impact on generation diversity. While the methodology of using code execution for measuring semantic diversity is novel, there are several important limitations: (1) experiments are restricted to only Python language and 21 problems, raising concerns about generalizability, (2) lack of comprehensive model coverage in initial submission, and (3) insufficient explanation of dataset construction process. These limitations, combined with presentation issues and questions about broader impact beyond code generation, suggest the work would benefit from substantial improvements before publication.