Title: IaC-Eval: A Code Generation Benchmark for Cloud Infrastructure-as-Code Programs
Benchmark: IaC-Eval
Summary: Evaluating LLMs ability to generate Infrastructure-as-Code (IaC) code (part of cloud computing)
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: IaC code generation
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Not defined
Task definition: The LLM generates an Infrastructure-as-Code (IaC) program based on a set of instructions.
Task item definition: The LLM must generate a program given (i) a natural language prompt describing the problem (ii) user intent specifications written in Rego, and (iii) an example of a correct configuration in Terraform HCL. 
Task definition detail: All examples are based on AWS services (though there are many different AWS services)
Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions)
Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: All examples created for the benchmark rather than being based on real-world problems.
Dataset size: 458
Dataset metadata available: Yes
Dataset metadata detail: Difficulty level
Sampling method: Unknown
Split reviewed: Test
Response format: Structured response (e.g. valid JSON, API call alone)
Metric definition: Functional correctness checks. Evaluated by (1) producing a dependency graph from the code (2) using an IaC policy engine to check whether the instruction specification are in the program.
Metric aggregation: Simple Mean
Metric statistics: Mean, 
Metric subscores: Yes
Metric subscores detail: Difficulty level
Metric metascoring: pass@k (any correct answer in k trials)
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Discuss the limitation that it only uses AWS services and ignores e.g. Azure.
Taxonomy: Code Generation / 
New taxonomy: General Application / Code Generation