DOI: 10.5281/zenodo.17194685
title: CoReEval: A Large-Scale Dataset for Evaluating Code Readability with Human and Machine Judgments
title: From Metrics to Models: Large Language Models as Human-Aligned Code Readability Evaluators

description: Dataset Description 

CoReEval (Code Readability Evaluation) is a large-scale evaluation dataset designed to compare code readability judgments produced by humans, Large Language Models (LLMs), and a traditional deep learning model (Scalabrino et al.).The dataset was created as part of the study presented in the anonymous paper:“From Metrics to Models: Large Language Models as Human-Aligned Code Readability Evaluators.”

Contents

The dataset is provided in three complementary formats:








File Name
Format
Approx. Size




readability_results.csv
CSV
789 MB


readability_results.json
JSON
1.44 GB


readability_results_part*.xlsx (x9)
Excel
~23 MB per file






Purpose

This dataset enables in-depth analysis of how well LLMs can replicate human judgments of code readability. It covers:





Multiple programming languages: Java, Python, and CUDA




Two code types: Production code (code) and unit tests (test)




Three established open-source datasets: Buse, Dorn, and Scalabrino




Two developer profiles: Junior and Senior




Two prompt families:





templates (generic prompts)




templates-2 (developer-guided, human-centric prompts)





Dataset Structure

Each row in the dataset represents the readability evaluation of a code snippet by a specific model or method. The fields include:








Field
Description




group
One of templates, templates-2, or scalabrino_direct


persona
JUNIOR or SENIOR (for LLM-generated results)


dataset
Dataset origin: Buse, Dorn, or Scalabrino


language
Programming language: java, python, or cuda


code_type
code (production code) or test (unit test)


model_name
Name of the LLM or SCALABRINO (for the deep learning baseline)


prompt_type
Prompt strategy: ZSL, FSL, CoT, ToT, or None (for Scalabrino)


top_p
Decoding parameter top_p (LLMs only)


temperature
Decoding temperature (LLMs only)


iteration
Iteration number (1 to 3)


snippet_id
Unique identifier of the evaluated snippet


score
Raw score (1–20) predicted by the model


normalized_score
Normalized score (1–20)


likert_category
Readability category: readable, neutral, or unreadable (Likert 1–5)


groundtruth_score
Corresponding human Likert score


reasoning
Justification text (LLMs only; None for Scalabrino baseline)






Large Language Models Used

The dataset includes results from a wide range of API-based and locally hosted Large Language Models (LLMs), covering diverse architectures and providers.








Model
Provider
Size
Access




Claude 3 Sonnet
Anthropic
Unknown
API


Deepseek Chat
Deepseek
~64B
API


GPT-4o
OpenAI
Unknown
API


Mistral Large (v2407)
Mistral
~65B (Mixture-of-Experts)
API


Deepseek LLM 7B Chat
Deepseek (HF)
7B
Local


Falcon 3 7B Instruct
TII (HF)
7B
Local


Gemma 7B IT
Google (HF)
7B
Local


Mistral 7B Instruct
Mistral (HF)
7B
Local


Phi-3 Mini
Microsoft (HF)
4.2B
Local


Qwen2.5 7B Instruct
Alibaba (HF)
7B
Local






Decoding Parameter Grid

Each LLM was evaluated using a full grid of decoding parameters to assess generation variability and robustness.








Temperature ↓ / Top_p →
0.5
0.75
1.0




0.1
Yes
Yes
Yes


0.5
Yes
Yes
Yes


1.0
Yes
Yes
Yes






This results in 9 decoding settings per prompt, per model, per snippet, and per iteration. All combinations were used consistently to ensure coverage of deterministic to more creative generation settings.

Replicability

Each configuration (model_name, prompt_type, top_p, temperature) was run across three different iterations to ensure robustness and capture variability.In total, the dataset contains over 1.4 million readability judgments.

Anonymity

All information related to the authors, institutions, or affiliations has been removed to comply with double-blind reviewing requirements.

Usage Scenarios

This dataset can be used to:





Evaluate the ability of LLMs to replicate human readability judgments




Study the impact of prompt engineering on model outputs




Analyze correlations between different models, profiles, and settings




Explore the structure and quality of reasoning generated by LLMs