Kati8/ICLR_2023_2025 huggingface dataset:
authors: Shunyu Yao
authors: Howard Chen
authors: Austin W. Hanjie
authors: Runzhe Yang
authors: Karthik R Narasimhan
reviews: 
Comments: The authors propose COLLIE, a grammar-based framework for probing the performances of LLMs on constrained text generation tasks. COLLIE realizes the constraints on different levels (word, paragraph, passage) and imposes requirements on various aspects, e.g., character count, word count, and positions of a specific text. Experiments are conducted on instructions built from three sources and on various strong LLMs (GPTs, PaLMs, and etc.). The results suggest that GPT-4 generally outperforms other compared models but still performs poorly on a subset of the tasks. 

Overall, while the concept of this task is simple, the challenges posed to existing LLMs are non-trivial, clearly pointing out the incapabilities of these models. I believe COLLIE has the potential to drive further developments of existing models on more nuanced tasks.
Strengths: * The paper is well-written and easy-to-understand.
* COLLIE is a conceptually simple but scientifically non-trivial method, and is beneficial to further development of LLMs.
* The experiments and analyses are comprehensive, including evaluations on current state-of-the-art models.
* The authors also provide complementary code which helps verify their approach.
Weaknesses: I have one question regarding to the instructions obtained from different sources. Since the grammar is translated into natural language instruction to prompt the model, how do the instructions from different source datasets differ from one another?
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper presents a framework, COLLIE, that allows researchers to build constrained text generation benchmark using different combinations of generation levels and modeling challenges (tasks), and a benchmark, COLLIE-v1, which is constructed using that framework and consists of 20,80 data instances comprising 13 structures.
The value of this paper lies in that it provides a method which allows future work to construct data of their interest in a scalable manner, and the analyses that the this paper conducts provide insights to researchers who are focusing on developing LLMs with better logical, reasoning, and compositional capacities.
Strengths: 1. This paper provides a method which allows future work to construct data of their interest in a scalable manner.
2. The analyses that the this paper conducts provide insights to researchers who are focusing on developing LLMs with better logical, reasoning, and compositional capacities.
3. This paper is generally well-written and easy to follow.
Weaknesses: There is no great weaknesses that I can find - but there is a minor one:

Although I understand that the authors are focusing on more "basic" units, such as tokens, sentence, etc, so this paper can be more practical and useful for downstream applications, such as pretraining and evaluating LLMs, most of current work on constrained text generation seem to focus on text summarization, including controllable text summarization (e.g., MACSum Zhang et al, 2023). However this paper does not mention any of this, and does not discuss any potential that the proposed method can be applied to those constrained text generation (summarization) tasks. I personally suggest that the authors provide some discussion or explain on this issue.

Zhang, Yusen, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, and Rui Zhang. "Macsum: Controllable summarization with mixed attributes." Transactions of the Association for Computational Linguistics 11 (2023): 787-803.
Recommendation: 8: accept, good paper
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: The authors propose a framework to automatically extract samples from large unlabeled text corpora for constrained text generation. Specifically, they manually craft rules/constraints and find satisfying texts as references. For example, a constraint can be "the third last word being mankind". They then evaluate off-the-shelf LLMs on this extracted dataset.
Strengths: 1. The authors evaluated different off-the-shelf LLMs and showed that they don't fully solve this task.
Weaknesses: 1. The idea is not very novel, and similar ideas have already been explored. For example, [1] also constructed a dataset using similar constraints for instruction fine-tuning.
2. The dataset may not be very useful. Specifically, because the rules are too vague/arbitrary, the extracted ground truth is not useful for the evaluation process: the authors only use them for comparing fluency. In addition, since the rules can be arbitrarily designed, this compiled dataset does not hold much value, because many similar datasets can be compiled with different engineering details.

[1] Controlled text generation with natural language instructions. https://proceedings.mlr.press/v202/zhou23g/zhou23g.pdf
Recommendation: 6: marginally above the acceptance threshold
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 3 good
Clarity: 3 good
reviews: 
Comments: This paper introduces a grammar based framework called COLLIE for constrained text generation at varying levels of specifications. Additionally it proposes a development tool for automated extraction of task instances given a constraint structure and text corpus. Furthermore, the paper constructs a dataset from three sources - Wikipedia, CCNews, and Project Gutenberg using the previously mentioned framework and calls it COLLIE-v1. The resultant dataset COLLIE-v1 is constructed using manually crafted constraints and is used to analyze LLM performances as well as highlight their shortcomings. The paper focuses on five of the most prominent LLMs, namely GPT-3.5,GPT-4,PaLM, Vicuna-7B and Alpaca-7B.

Interestingly, the COLLIE framework enables flexible, extensible and dynamic constraint construction that can co-evolve with the upcoming LLMs and help in understanding their shortcomings to better solve them.

COLLIE can thus help in not only evaluating and benchmarking LLMs but also help in constraint text generation independently.
Strengths: 1. Well written paper with evaluation on competitive LLM baselines.
    
2. Combination of rule based and neural based generation enables NLP grounded generations
    
3. Open-sourcing of code and the related dataset for promoting further research.
    
4. Comprehensive analysis to highlight the shortcoming of current LLMs that needs to be addressed.
Weaknesses: 1. Some important details for instruction rendering should be moved to the main paper.
    
2. The paper mentions that the technique can be used for constraining words, word blacklisting, however a qualitative analysis is missing for the same in the current version.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: This work presents COLLIE, a grammar-based system for controlled text generation across multiple constraint specifications. The authors also develop an automated tool for extracting task instances from constraint schemas and text sources. Using this framework, they assemble COLLIE-v1, a large-scale benchmark derived from Wikipedia, CCNews, and Project Gutenberg with manually defined constraints. This resource is leveraged to evaluate major LLMs including GPT-3.5, GPT-4, PaLM, Vicuna-7B, and Alpaca-7B, revealing important limitations in their performance. A notable contribution is COLLIE's adaptability and modularity—the framework supports constraint evolution as new models emerge, facilitating ongoing evaluation and understanding of LLM weaknesses. Beyond benchmarking purposes, the framework proves valuable for constrained generation tasks in general applications.
Strengths: 1. The paper is clearly written and includes evaluation against strong baseline LLMs. 2. Integration of symbolic and neural approaches provides grounded NLP generation capabilities. 3. Public release of both the implementation and accompanying dataset promotes reproducibility and future work. 4. Thorough empirical investigation identifying gaps and limitations that current state-of-the-art models must overcome.
Weaknesses: 1. Several critical implementation details concerning instruction processing would benefit from inclusion in the main text rather than supplementary materials. 2. While the paper discusses applicability to word-level constraints and word filtering mechanisms, the current version lacks qualitative examples demonstrating effectiveness on these specific tasks.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: This work presents COLLIE, a grammar-driven methodology for generating constrained text across multiple specification granularities. The authors also develop an automated tool for extracting task examples from constraint specifications and textual corpora. Using this framework, they assemble COLLIE-v1, a benchmark constructed from Wikipedia, CCNews, and Project Gutenberg with hand-designed constraints. This benchmark is leveraged to evaluate and identify limitations in five state-of-the-art LLMs: GPT-3.5, GPT-4, PaLM, Vicuna-7B, and Alpaca-7B. A key contribution is COLLIE's capacity to support adaptable, scalable, and configurable constraint specification that can adjust alongside emerging LLM capabilities, facilitating deeper investigation of model weaknesses and informing future improvements. The framework proves valuable for both LLM evaluation and benchmarking purposes, as well as for independent constrained text generation applications.
Strengths: 1. Clearly articulated manuscript with rigorous testing against leading state-of-the-art language models. 2. Integration of symbolic and deep learning approaches yields linguistically grounded output generation. 3. Public release of implementation code and associated benchmark dataset to facilitate community-driven advances. 4. Thorough empirical investigation identifying critical performance gaps and deficiencies in contemporary LLMs requiring resolution.
Weaknesses: 1. Several critical technical specifics pertaining to instruction interpretation warrant relocation from supplementary sections to the primary narrative. 2. The framework's applicability to word-level constraints and vocabulary filtering is mentioned but lacks corresponding empirical examples and qualitative discussion in the present version.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent