Package: collie-bench
COLLIE: Systematic Construction of Constrained Text Generation Tasks (Website)We propose the COLLIE framework for easy constraint structure specification, example extraction, instruction rendering, and model evaluation.InstallWe recommand using Python 3.9 (3.10 as of now might have incompatabilty of certain dependencies).To install in development mode (in cloned project directory):pipinstall-e.After installation you can access the functionalities throughimport collie.We will add COLLIE to PyPI soon.OverviewThere are two main ways to use COLLIE:Use thedataset we constructedto compare performance of your prompting/modeling methods to the ones reported in the paperWrite your own constraints; make it harder, more compositional, etc to explore the limits of models and probe failure cases by following the steps inCOLLIE frameworkDatasetThe dataset used in the paper is atdata/all_data.dilland can be loaded bywithopen("data/all_data.dill","rb")asf:all_data=dill.load(f)all_datawill be a dictionary with keys as the data source and constraint type, and values as a list of constraints. For example,all_data['wiki_c07'][0]is{'example':'Black market prices for weapons and ammunition in the Palestinian Authority-controlled areas have been rising, necessitating outside funding for the operation.','targets':['have','rising','the'],'constraint':...,'prompt':"Please generate a sentence containing the word 'have', 'rising', 'the'.",...}Reproducing the results reported in the paper:Our model results can be found inlogs/folderTo plot the figures/tables in the paper, check outscripts/analysis.ipynbTo run the models to reproduce the results, runpython scripts/run_api_models.pyandpython scripts/run_gpu_models.pyThe COLLIE FrameworkThe framework follows a 4-step process:Constraint SpecificationExtractionRenderingEvaluationStep 1: Constraint Specification (Complete Guide)To specify a constraint, you need the following concepts defined as classes incollie/constraints.py:Level: deriving classesInputLevel(the basic unit of the input) andTargetLevel(the level for comparing to the target value); levels include'character','word','sentence', etcTransformation: defines how the input text is modified into values comparable against the provided target value; it derives classes likeCount,Position,ForEach, etcLogic:And,Or,Allthat can be used to combine constraintsRelation: relation such as'=='or'in'for compariing against the target valueReduction: when the target has multiple values, you need to specify how the transformed values from the input is reduced such as'all','any','at least'Constraint: the main class for combining all the above specificationsTo specify a constraint, you need to provide at least theTargetLevel,Transformation, andRelation.
They are going to be wrapped in thec = Constraint(...)initialization. Once the constraint is specified, you can usec.check(input_text, target_value)to verify any given text and target tuple.Below is an example of specifying a "counting the number of word constraint".>>>fromcollie.constraintsimportConstraint,TargetLevel,Count,Relation# A very simple "number of word" constraint.>>>c=Constraint(>>>target_level=TargetLevel('word'),>>>transformation=Count(),>>>relation=Relation('=='),>>>)>>>print(c)Constraint(InputLevel(None),TargetLevel(word),Transformation('Count()'),Relation(==),Reduction(None))Check out theguideto explore more examples.Step 2: Extraction (Complete Guide)Once the constraints are defined, you can now extract examples from the datasources (e.g., Gutenberg, Wikipedia) that satisfy the specified constraints.To download necessary data files including theGutenberg, dammitcorpus to thedatafolder, run from the root project dir:bash download.shRun extraction:python -m collie.examples.extractThis will sweep over all constraints and data sources defined incollie/examples/. To add additional examples, you can add them to the appropriate python files.
Extracted examples can be found in the foldersample_data. The files are named as:{source}_{level}.dill. Thedata/all_data.dillfile is simply a concatenation of all these source-level dill files.Step 3: RenderingTo render a constraint, simply run:>>>fromcollie.constraint_rendererimportConstraintRenderer>>>renderer=ConstraintRenderer(>>>constraint=c,# Defined in step one>>>constraint_value=5>>>)>>>print(renderer.prompt)Pleasegenerateasentencewithexactly5words.Step 4: EvaluationTo check constraint satisfication, simply run:>>>text='This is a good sentence.'>>>print(c.check(text,5))True>>>print(c.check(text,4))FalseCitationlease cite our paper if you use SimCSE in your work:@inproceedings{yao2023collie,title={COLLIE: Systematic Construction of Constrained Text Generation Tasks},author={Yao, Shunyu and Chen, Howard and Wang, Austin and Yang, Runzhe and Narasimhan, Karthik},booktitle={ArXiv},year={2023},html={}}LicenseMIT. Note that this is the license for our code, but each data source retains their own respective licenses.