p-doom/crowd-code-dataset-0.2 huggingface dataset:
[tab] gdr/judgement_prompts.py (python)
judgement_prompt_v1 = """
You are an expert AI Problem Analyst. Your task is to classify the intrinsic difficulty of a mathematical or logical problem.

Analyze the problem based on the cognitive steps required for a solution.
-   **Easy:** Requires a single, straightforward operation or direct recall of a common fact (e.g., simple arithmetic, a single formula).
-   **Medium:** Requires multiple sequential steps or the application of a standard process that combines a few concepts (e.g., a multi-step word problem, basic algebra).
-   **Hard:** Requires complex, multi-step reasoning, abstract thinking, setting up a system of equations, or logical inference where the solution path is not immediately obvious.

You will be given a problem. Respond with ONLY one word: Easy, Medium, or Hard. Do not provide any explanation or other text.

**Problem:**
{problem}
"""
[tab] pyproject.toml (plaintext)
[project]
name = "adaptive-compute"
version = "0.1.0"
description = "Adaptive compute for reasoning tasks"
requires-python = ">=3.13"
dependencies = [
    "datasets>=4.1.0",
    "google-genai>=1.45.0",
    "python-dotenv>=1.1.1",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["gdr"]
[tab] evaluation/check_problem_difficulty.py (python)
#!/usr/bin/env python3
"""
Script to process JSONL files and print file names and entries.
"""

import json
import sys
import argparse
from google import genai
import os
gemini = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))

def get_problem_difficulty(problem, response):
    """ask gemini to determine the difficulty of the problem"""
    prompt = f"""
    You are a helpful assistant that determines the difficulty of a math problem.
    The problem is: {problem}
    I tried to solve the problem and got the following response: {response}
    Return the difficulty of the problem on a scale of 1 to 10, where 1 is the easiest and 10 is the hardest.
    """
    response = gemini.models.generate_content(model="gemini-2.5-flash", contents=[prompt])
    return response.text

def process_jsonl_file(file_path):
    """Process a single JSONL file and print its entries."""
    print(f"
{'='*60}")
    print(f"Processing file: {file_path}")
    print(f"{'='*60}
")
    
    try:
        with open(file_path, 'r', encoding='utf-8') as f:
            for line_num, line in enumerate(f, start=1):
                line = line.strip()
                if not line:  # Skip empty lines
                    continue
                
                try:
                    entry = json.loads(line)
                    print(f"Entry {line_num}:")
                    print(f"Problem:
 {entry['input'][0]['content']}")
                    difficulty = get_problem_difficulty(entry['input'][0]['content'], entry['output'])
                    print(f"Gemini Difficulty verdict: {difficulty}")
                    print("-" * 60)
                except json.JSONDecodeError as e:
                    print(f"Error parsing entry on line {line_num}: {e}")
                    print(f"Line content: {line}")
                    print("-" * 60)
    except FileNotFoundError:
        print(f"Error: File '{file_path}' not found.")
    except Exception as e:
        print(f"Error processing file '{file_path}': {e}")


def main():
    parser = argparse.ArgumentParser(
        description='Process JSONL files and print file names and entries.'
    )
    parser.add_argument(
        'files',
        nargs='+',
        help='One or more JSONL files to process'
    )
    
    args = parser.parse_args()
    
    for file_path in args.files:
        process_jsonl_file(file_path)


if __name__ == '__main__':
    main()
[terminal_command] TERMINAL ()
python evaluation/check_problem_difficulty.py evaluation/hard_problems.jsonl > evaluation/gpt-5-verfict-hard.out && \
python evaluation/check_problem_difficulty.py evaluation/easy_problems.jsonl > evaluation/gpt-5-verfict-easy.out
[terminal_command] TERMINAL ()
python evaluation/check_problem_difficulty.py evaluation/hard_problems.jsonl > evaluation/gpt-5-verfict-hard-no-answer.out && python evaluation/check_problem_difficulty.py evaluation/easy_problems.jsonl > evaluation/gpt-5-verfict-easy-no-answer.out
[content] evaluation/easy_problems.jsonl (jsonl)
{"input": [{"role": "user", "content": "Solve the following math problem. Make sure to put the answer (and only answer) inside \\boxed{}.

What is the total volume in cubic feet of three boxes if each box is a cube with edge length 4 feet?"}], "output": "<think>
Okay, so I need to find the total volume of three boxes, and each box is a cube with an edge length of 4 feet. Hmm, let's start by recalling the formula for the volume of a cube. I think the volume of a cube is calculated by edge length cubed, right? So if each edge is 4 feet, then the volume of one box would be 4 multiplied by 4 multiplied by 4. Let me check that again.

Yes, volume of a cube is side³. So, 4 feet * 4 feet * 4 feet. Let me compute that. First, 4*4 is 16, and then 16*4 is 64. So each cube has a volume of 64 cubic feet. Now, since there are three boxes, I need to multiply the volume of one box by 3. That should give the total volume. 

So, 64 cubic feet per box times 3 boxes. Let's do 64*3. Breaking it down: 60*3 is 180, and 4*3 is 12. Adding them together, 180 +12 is 192. So, the total volume for all three boxes should be 192 cubic feet. 

Wait, let me make sure I didn't make a mistake in the multiplication. 64*3... 64+64=128, and 128+64=192. Yep, that's correct. So the individual volumes are 64 each, three of them add up to 192. I think that's it. No, wait, maybe I should check the edge length again. The problem says each box is a cube with edge length 4 feet. So the calculation is right. Edge length is 4, so 4^3 is 64. Three boxes, so 3*64=192. Yep, that seems right. I don't think I made any errors here.
</think>

The volume of one cube is calculated as edge length cubed: \\(4^3 = 64\\) cubic feet. For three such boxes, the total volume is \\(3 \	imes 64 = 192\\) cubic feet.

\\boxed{192}", "category": "math", "license": "cc-by-4.0", "reasoning": "on", "generator": "Qwen-2.5-32B-Instruct, DeepSeek-R1", "used_in_training": "Ultra, Super, Nano", "version": "v1.1", "system_prompt": "detailed thinking on"}
[tab] evaluation/gpt-5-verdict-hard-thinking.out (plaintext)

============================================================
Processing file: evaluation/hard_problems_thinking.jsonl
============================================================

Entry 1:
Problem:
 Solve the following math problem. Make sure to put the answer (and only answer) inside \boxed{}.

Let \( X \) be a set of 11 members. Let \( P \) be the collection of subsets of \( X \) which have three or more members. Is it possible to arrange all the members of \( P \) in a sequence such that each member of the sequence is either its predecessor minus one member or its predecessor plus one member?
OpenAI GPT-5 Difficulty verdict: 5 — It hinges on noticing the bipartite (parity) structure and computing even/odd subset counts via binomial sums to spot a 45-element imbalance, a standard but nontrivial combinatorial insight.
------------------------------------------------------------
Entry 2:
Problem:
 Solve the following math problem. Make sure to put the answer (and only answer) inside \boxed{}.

The four complex roots of
\[2z^4 + 8iz^3 + (-9 + 9i)z^2 + (-18 - 2i)z + (3 - 12i) = 0,\]when plotted in the complex plane, form a rhombus.  Find the area of the rhombus.
OpenAI GPT-5 Difficulty verdict: 6 — It needs a clever shift (z → w−i), recognition of the even quartic structure, and a Vieta-based geometric insight to relate the constant term’s magnitude to the rhombus area, but the algebra is straightforward once seen.
------------------------------------------------------------
Entry 3:
Problem:
 Solve the following math problem. Make sure to put the answer (and only answer) inside \boxed{}.

A jar has 10 red candies and 10 blue candies. Terry picks two candies at random, then Mary picks two of the remaining candies at random. Given that the probability that they get the same color combination, irrespective of order, is \( \frac{m}{n} \), where \( m \) and \( n \) are relatively prime positive integers, find \( m+n \).
OpenAI GPT-5 Difficulty verdict: Difficulty: 4/10 — It requires basic combinatorial counting and simple conditional probability with straightforward casework and symmetry, no advanced techniques.
------------------------------------------------------------
[tab] evaluation/evaluate_rewrites.py (python)
"""
Script to evaluate the quality of the rewrites of the thought traces.

For now we have three different evaluation methods:
1. Rewriting correctness: Does rewriting make sense → prompt powerful model to check rewritten traces
2. Rewriting length/difficulty correlation → are easy problems being rewritten shorter etc → Check token length
3. Check answer correctness → Regex
"""

import re


def extract_boxed_answer(s):
    """
    Extracts the content within \boxed{} from a string.
    Returns None if no boxed answer is found.
    """
    match = re.search(r'\\boxed\s*{([^}]*)}', s)
    if match:
        return match.group(1).strip()
    return None

def check_boxed_answer_correctness(original, rewritten):
    """
    Checks if both original and rewritten answers contain a boxed answer,
    and returns True if they are the same, False if they differ,
    and None if a boxed answer is not found in either.
    """
    orig_boxed = extract_boxed_answer(original)
    rew_boxed = extract_boxed_answer(rewritten)
    if orig_boxed is not None and rew_boxed is not None:
        return orig_boxed == rew_boxed
    elif orig_boxed is None and rew_boxed is None:
        return False
    else:
        return None


if __name__ == "__main__":
    original = "The answer is \\boxed{10}."
    rewritten = "The answer is \\boxed{10}."
    print(check_boxed_answer_correctness(original, rewritten))
[content] evaluation/evaluate_rewrites.py (python)
def check_rewriting_length_difficulty_correlation(original, rewritten):
    """
    Checks if the rewriting length is correlated with the difficulty of the problem.
    """
    pass

# ========= Check answer correctness =========
[content] evaluation/evaluate_rewrites.py (python)

        return len(original) < len(rewritten)
    elif difficulty == "Medium":
        return len(original) == len(rewritten)
    elif difficulty == "Hard":
        return len(original) > len(rewritten)
    else:
        return None
[content] evaluation/evaluate_rewrites.py (python)
istant that checks if the rewriting makes sense.
    The original thought trace is: {original}
    The rewritten thought trace is: {rewritten}
    Return True if the rewriting makes sense, False otherwise.
    """
    response = openai.chat.completions.create(
        model="gpt-5",
        messages=[{"role": "system", "content": "You are a helpful assistant that checks if the rewriting makes sense."}, {"role": "user", "content": prompt}]
    )
[content] evaluation/evaluate_rewrites.py (python)

        return True
    elif "Final Verdict: False" in response.choices[0].message.content:
        return False
    else:
        raise ValueError(f"Invalid response: {response.choices[0].message.content}")
[content] evaluation/evaluate_rewrites.py (python)
evaluate_rewrites(original, rewritten, difficulty):
    """
    Evaluates the quality of the rewrites.
    """
    return {
        "rewriting_correctness": check_rewriting_correctness(original, rewritten),
        "rewriting_length_difficulty_correlation": check_rewriting_length_difficulty_correlation(original, rewritten, difficulty),
        "boxed_answer_correctness": check_boxed_answer_correctness(original, rewritten)
    }
[content] evaluation/evaluate_rewrites.py (python)
        rewrite_correctness_list.append(check_rewriting_correctness(original, rewritten))
        rewrite_length_difficulty_correlation_list.append(check_rewriting_length_difficulty_correlation(original, rewritten, difficulty))
        boxed_answer_correctness_list.append(check_boxed_answer_correctness(original, rewritten))

    return rewrite_correctness_list, rewrite_length_difficulty_correlation_list, boxed_answer_correctness_list
[content] evaluation/evaluate_rewrites.py (python)

    rewrite_length_difficulty_correlation_score = sum(rewrite_length_difficulty_correlation_list) / len(rewrite_length_difficulty_correlation_list)
    boxed_answer_correctness_score = sum(boxed_answer_correctness_list) / len(boxed_answer_correctness_list)
[content] evaluation/evaluate_rewrites.py (python)

    print(f"Rewrite correctness score: {rewrite_correctness_score}")
    print(f"Rewrite length/difficulty correlation score: {rewrite_length_difficulty_correlation_score}")
    print(f"Boxed answer correctness score: {boxed_answer_correctness_score}")