p-doom/crowd-code-dataset-0.2 huggingface dataset:
[tab] data/handcrafted_traces/hello_world.md (markdown)
# Assistant
``` bash
ls -la
```

# User
<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
</stdout>

# Assistant
``` bash
cat <<'EOF' > hello_world.py
print("Hello, World!")
EOF && cat -n hello_world.py
```

# User
<stdout>
     1  print("Hello, World!")
</stdout>

# Assistant
```bash
python hello_world.py
```

# User
<stdout>
Hello, World!
</stdout>

# Assistant
```bash
echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
```
[tab] data/handcrafted_traces/hello_world_insert.md (markdown)
# Assistant
``` bash
ls -la
```

# User
<stdout>
[alfred.nguyen@hai-login1.haicore.berlin:~] $ ls -la
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
-rw-rw---- 1 alfred.nguyen alfred.nguyen     0 Nov 13 18:05 hello_world.py
[alfred.nguyen@hai-login1.haicore.berlin:~] $ 
</stdout>

# Assistant
``` bash
cat -n hello_world.py
```

# User
<stdout>
     1  print("")
</stdout>

# Assistant
```bash
sed -i 's/^print("")/print("Hello, World!")/' hello_world.py && cat -n hello_world.py
```

# User
<stdout>
     1  print("Hello, World!")
</stdout>

# Assistant
```bash
python hello_world.py
```

# User
<stdout>
Hello, World!
</stdout>

# Assistant
```bash
echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
```
[tab] src/evaluation/evaluate_client.py (python)
import openai
import json
import os
import tyro
from dataclasses import dataclass
from tqdm import tqdm


@dataclass
class Args:
    name: str = "hello_world_insert"
    input_path: str = "data/incremental_test_cases/"
    output_path: str = "data/evaluation_results/"
    system_prompt_file: str = "data/system_prompts/minimal_v1.md"
    api_base_url: str = "http://hai002:30000/v1"


def load_dataset(filepath):
    """Yields each test case from the jsonl file."""
    with open(filepath, "r") as f:
        for line in f:
            yield json.loads(line)


def format_prompt_for_model(context, system_prompt):
    """Formats the context into a single string for the LLM."""
    prompt_str = f"SYSTEM: {system_prompt}
"
    for message in context:
        role = message["role"].upper()
        content = message["content"]
        prompt_str += f"{role}: {content}
"
    prompt_str += "ASSISTANT:"
    return prompt_str


def run_evaluation_client(args: Args):
    client = openai.OpenAI(base_url=args.api_base_url, api_key="EMPTY")

    test_cases = list(load_dataset(args.input_path + args.name + ".jsonl"))
    system_prompt = open(args.system_prompt_file, "r").read()

    num_correct = 0
    results = []

    print(f"Running inference on {len(test_cases)} test cases...")
    for test_case in tqdm(test_cases):
        formatted_messages = [{"role": "system", "content": system_prompt}]

        for turn in test_case["context"]:
            formatted_messages.append(turn)

        breakpoint()
        try:
            response = client.chat.completions.create(
                model="default",
                messages=formatted_messages,
                max_tokens=256,
                temperature=0.0,
                stop=["
```", "ASSISTANT:", "USER:"],
            )
            generated_text = response.choices[0].message.content
        except Exception as e:
            print(f"API Error on task {test_case['task_id']}: {e}")
            generated_text = ""  # Mark as incorrect on error

        generated_command = generated_text
        expected_command = test_case["expected_final_response"]
        is_correct = 1 if generated_command == expected_command else 0
        num_correct += is_correct
        results.append(
            {
                "task_id": test_case["task_id"],
                "generated": generated_command,
                "expected": expected_command,
                "score": is_correct,
            }
        )

    os.makedirs(args.output_path, exist_ok=True)
    json.dump(results, open(args.output_path + args.name + ".json", "w"))
    # 6. Report final score
    pass_rate = (num_correct / len(test_cases)) * 100
    print("
--- Evaluation Complete ---")
    print(f"Total Test Cases: {len(test_cases)}")
    print(f"Correct: {num_correct}")
    print(f"Pass Rate (Exact Match): {pass_rate:.2f}%")


if __name__ == "__main__":
    args = tyro.cli(Args)
    run_evaluation_client(args)
[content] src/evaluation/evaluate_client.py (python)
    api_base_url: str = "http://hai005:30000/v1"


def load_dataset(filepath):
    """Yields each test case from the jsonl file."""
    with open(filepath, "r") as f:
        for line in f:
            yield json.loads(line)


def format_prompt_for_model(context, system_prompt):
    """Formats the context into a single string for the LLM."""
    prompt_str = f"SYSTEM: {system_prompt}
"
    for message in context:
        role = message["role"].upper()
        content = message["content"]
        prompt_str += f"{role}: {content}
"
    prompt_str += "ASSISTANT:"
    return prompt_str


def run_evaluation_client(args: Args):
    client = openai.OpenAI(base_url=args.api_base_url, api_key="EMPTY")

    test_cases = list(load_dataset(args.input_path + args.name + ".jsonl"))
    system_prompt = open(args.system_prompt_file, "r").read()

    num_correct = 0
    results = []

    print(f"Running inference on {len(test_cases)} test cases...")
    for test_case in tqdm(test_cases):
        formatted_messages = [{"role": "system", "content": system_prompt}]

        for turn in test_case["context"]:
            formatted_messages.append(turn)
[tab] data/system_prompts/crowd_code_v1.md (markdown)
You are a helpful assistant that can interact multiple times with a computer shell to solve programming tasks.
Your response must contain exactly ONE bash code block with ONE command (or commands connected with && or ||).

Format your response as shown in <format_example>.
<format_example>
```bash
your_command_here
```
</format_example>
Failure to follow these rules will cause your response to be rejected.
[tab] data/incremental_test_cases/hello_world_insert.jsonl (jsonl)
{"task_id": "hello_world_insert/0", "context": [{"role": "assistant", "content": "``` bash
ls -la```"}, {"role": "user", "content": "<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
-rw-rw---- 1 alfred.nguyen alfred.nguyen     0 Nov 13 18:05 hello_world.py
</stdout>"}], "expected_final_response": "``` bash
cat -n hello_world.py```"}
{"task_id": "hello_world_insert/1", "context": [{"role": "assistant", "content": "``` bash
ls -la```"}, {"role": "user", "content": "<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
-rw-rw---- 1 alfred.nguyen alfred.nguyen     0 Nov 13 18:05 hello_world.py
</stdout>"}, {"role": "assistant", "content": "``` bash
cat -n hello_world.py```"}, {"role": "user", "content": "<stdout>
     1  print(\"\")
</stdout>"}], "expected_final_response": "```bash
sed -i 's/^print(\"\")/print(\"Hello, World!\")/' hello_world.py && cat -n hello_world.py```"}
{"task_id": "hello_world_insert/2", "context": [{"role": "assistant", "content": "``` bash
ls -la```"}, {"role": "user", "content": "<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
-rw-rw---- 1 alfred.nguyen alfred.nguyen     0 Nov 13 18:05 hello_world.py
</stdout>"}, {"role": "assistant", "content": "``` bash
cat -n hello_world.py```"}, {"role": "user", "content": "<stdout>
     1  print(\"\")
</stdout>"}, {"role": "assistant", "content": "```bash
sed -i 's/^print(\"\")/print(\"Hello, World!\")/' hello_world.py && cat -n hello_world.py```"}, {"role": "user", "content": "<stdout>
     1  print(\"Hello, World!\")
</stdout>"}], "expected_final_response": "```bash
python hello_world.py```"}
{"task_id": "hello_world_insert/3", "context": [{"role": "assistant", "content": "``` bash
ls -la```"}, {"role": "user", "content": "<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
-rw-rw---- 1 alfred.nguyen alfred.nguyen     0 Nov 13 18:05 hello_world.py
</stdout>"}, {"role": "assistant", "content": "``` bash
cat -n hello_world.py```"}, {"role": "user", "content": "<stdout>
     1  print(\"\")
</stdout>"}, {"role": "assistant", "content": "```bash
sed -i 's/^print(\"\")/print(\"Hello, World!\")/' hello_world.py && cat -n hello_world.py```"}, {"role": "user", "content": "<stdout>
     1  print(\"Hello, World!\")
</stdout>"}, {"role": "assistant", "content": "```bash
python hello_world.py```"}, {"role": "user", "content": "<stdout>
Hello, World!
</stdout>"}], "expected_final_response": "```bash
echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT```"}
[tab] data/handcrafted_traces/hello_world_insert.md (markdown)
# Assistant
``` bash
ls -la
```

# User
<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
-rw-rw---- 1 alfred.nguyen alfred.nguyen     0 Nov 13 18:05 hello_world.py
</stdout>

# Assistant
``` bash
cat -n hello_world.py
```

# User
<stdout>
     1  print("")
</stdout>

# Assistant
```bash
sed -i 's/^print("")/print("Hello, World!")/' hello_world.py && cat -n hello_world.py
```

# User
<stdout>
     1  print("Hello, World!")
</stdout>

# Assistant
```bash
python hello_world.py
```

# User
<stdout>
Hello, World!
</stdout>

# Assistant
```bash
echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
```
[content] data/handcrafted_traces/hello_world.md (markdown)
[alfred.nguyen@hai-login1.haicore.berlin:~] $ ls -la
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
[alfred.nguyen@hai-login1.haicore.berlin:~] $
[tab] src/serving/launch_sglang.sh (shellscript)
module load CUDA/12.8

source .venv/bin/activate

# model_id="qwen/qwen2.5-0.5b-instruct"
model_id="Qwen/Qwen3-Coder-30B-A3B-Instruct"
# model_id="Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8"

python3 -m sglang.launch_server --model-path $model_id --host 0.0.0.0
[tab] src/serving/launch_sglang.sh (shellscript)
module load CUDA/12.8

source .venv/bin/activate

# model_id="qwen/qwen2.5-0.5b-instruct"
# model_id="Qwen/Qwen3-Coder-30B-A3B-Instruct"
model_id="/fast/project/HFMI_SynergyUnit/tab_model/data/outputs/hf/qwen3_600m_full_finetune_crowd_code_dataset_seqlen_16k_token_batch_size_128k/2025-11-24_18-17-40"
# model_id="Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8"
# model_id="Qwen/Qwen3-0.6B"

# context_length=128000
context_length=40960

python3 -m sglang.launch_server \
    --model-path $model_id \
    --host 0.0.0.0 \
    --context-length $context_length \
    --port 30000 \
    --mem-fraction-static 0.95
[tab] src/serving/launch_sglang_qwen30b.sh (shellscript)
module load CUDA/12.8

source .venv/bin/activate

# model_id="qwen/qwen2.5-0.5b-instruct"
model_id="Qwen/Qwen3-Coder-30B-A3B-Instruct"
# model_id="/fast/project/HFMI_SynergyUnit/tab_model/data/outputs/hf/qwen3_600m_full_finetune_crowd_code_dataset_seqlen_16k_token_batch_size_128k/2025-11-24_18-17-40"
# model_id="Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8"
# model_id="Qwen/Qwen3-0.6B"

# context_length=128000
context_length=40960

python3 -m sglang.launch_server \
    --model-path $model_id \
    --host 0.0.0.0 \
    --context-length $context_length \
    --port 30000 \
    --mem-fraction-static 0.95
[tab] data/eval/handcrafted/hello_world_insert.md (markdown)
# Assistant
``` bash
ls -la
```

# User
<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
-rw-rw---- 1 alfred.nguyen alfred.nguyen     0 Nov 13 18:05 hello_world.py
</stdout>

# Assistant
``` bash
cat -n hello_world.py
```

# User
<stdout>
     1  print("")
</stdout>

# Assistant
```bash
sed -i 's/^print("")/print("Hello, World!")/' hello_world.py && cat -n hello_world.py
```

# User
<stdout>
     1  print("Hello, World!")
</stdout>

# Assistant
```bash
python hello_world.py
```

# User
<stdout>
Hello, World!
</stdout>

# Assistant
```bash
echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
```
[tab] data/eval/handcrafted/fibonacci_cached.md (markdown)
# Assistant
``` bash
ls -la
```

# User
<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
</stdout>

# Assistant
``` bash
cat <<'EOF' > fibonnacci.py && cat -n fibonnacci.py
import argparse

def fibonnacci(n):
    if n <= 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonnacci(n-1) + fibonnacci(n-2)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--n", type=int, default=10)
    args = parser.parse_args()
    print(fibonnacci(args.n))
EOF
```

# User
<stdout>
     1  import argparse
     2  def fibonnacci(n):
     3      if n <= 0:
     4          return 0
     5      elif n == 1:
     5          return 1
     7      else:
     8          return fibonnacci(n-1) + fibonnacci(n-2)
     9 
    10 if __name__ == "__main__":
    11     parser = argparse.ArgumentParser()
    12     parser.add_argument("--n", type=int, default=10)
    13     args = parser.parse_args()
    14     print(fibonnacci(args.n))
</stdout>

# Assistant
``` bash
python fibonnacci.py --n 15
```

# User
<stdout>
55
</stdout>
The output is very slow. Can we make it faster?

# Assistant
``` bash
# add a cache decorator to the fibonnacci function
sed -i '1s/^/from functools import cache
/' fibonnacci.py && cat -n fibonnacci.py
```

# User
<stdout>
     1  from functools import cache
     2  import argparse
     3
     4  def fibonnacci(n):
     5      if n <= 0:
     6          return 0
     7      elif n == 1:
     8          return 1
     9      else:
    10          return fibonnacci(n-1) + fibonnacci(n-2)
    11
    12  if __name__ == "__main__":
    13      parser = argparse.ArgumentParser()
    14      parser.add_argument("--n", type=int, default=10)
    15      args = parser.parse_args()
    16      print(fibonnacci(args.n))
</stdout>

# Assistant
``` bash
sed -i '4s/^/@cache
/' fibonnacci.py && cat -n fibonnacci.py
```

# User
<stdout>
     1  from functools import cache
     2  import argparse
     3
     4  @cache
     5  def fibonnacci(n):
     6      if n <= 0:
     7          return 0
     8      elif n == 1:
     9          return 1
    10      else:
    11          return fibonnacci(n-1) + fibonnacci(n-2)
    12
    13  if __name__ == "__main__":
    14      parser = argparse.ArgumentParser()
    15      parser.add_argument("--n", type=int, default=10)
    16      args = parser.parse_args()
    17      print(fibonnacci(args.n))
</stdout>

# Assistant
``` bash
python fibonnacci.py --n 15
```

# User
<stdout>
55
</stdout>
Better!

# Assistant
``` bash
echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
```
[tab] data/eval/handcrafted/hello_world.md (markdown)
# Assistant
``` bash
ls -la
```

# User
<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
</stdout>

# Assistant
``` bash
cat <<'EOF' > hello_world.py
print("Hello, World!")
EOF && cat -n hello_world.py
```

# User
<stdout>
     1  print("Hello, World!")
</stdout>

# Assistant
```bash
python hello_world.py
```

# User
<stdout>
Hello, World!
</stdout>

# Assistant
```bash
echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
```
[tab] data/eval/handcrafted/fibonacci.md (markdown)
# Assistant
``` bash
ls -la
```

# User
<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
</stdout>

# Assistant
``` bash
cat <<'EOF' > fibonnacci.py
import argparse

def fibonnacci(n):
    if n <= 0:
        return 0
    elif n == 1:
        return 1
    else:
        return fibonnacci(n-1) + fibonnacci(n-2)

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--n", type=int, default=10)
    args = parser.parse_args()
    print(fibonnacci(args.n))
EOF && cat -n fibonnacci.py
```

# User
<stdout>
     1  import argparse
     2  def fibonnacci(n):
     3      if n <= 0:
     4          return 0
     5      elif n == 1:
     5          return 1
     7      else:
     8          return fibonnacci(n-1) + fibonnacci(n-2)
     9 
    10 if __name__ == "__main__":
    11     parser = argparse.ArgumentParser()
    12     parser.add_argument("--n", type=int, default=10)
    13     args = parser.parse_args()
    14     print(fibonnacci(args.n))
</stdout>

# Assistant
``` bash
python fibonnacci.py --n 15
```

# User
<stdout>
55
</stdout>

# Assistant
```
echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
```
[tab] data/eval/handcrafted/palindrome.md (markdown)
# Assistant
``` bash
ls -la
```

# User
<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
</stdout>

# Assistant
``` bash
cat <<'EOF' > palindrome_checker.py
def is_palindrome(text):
    text = text.lower().replace(" ", "")
    return text == text[::-1]

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser(description="Check if text is a palindrome")
    parser.add_argument("text", help="Text to check")
    args = parser.parse_args()
    if is_palindrome(args.text):
        print(f'"{args.text}" is a palindrome')
    else:
        print(f'"{args.text}" is not a palindrome')
EOF && cat -n palindrome_checker.py
```

# User
<stdout>
     1  def is_palindrome(text):
     2      text = text.lower().replace(" ", "")
     3      return text == text[::-1]
     4 
     5  if __name__ == "__main__":
     6      import argparse
     7      parser = argparse.ArgumentParser(description="Check if text is a palindrome")
     8      parser.add_argument("text", help="Text to check")
     9      args = parser.parse_args()
    10      if is_palindrome(args.text):
    11          print(f'"{args.text}" is a palindrome')
    12      else:
    13          print(f'"{args.text}" is not a palindrome')
</stdout>

# Assistant
``` bash
python palindrome_checker.py racecar
```

# User
<stdout>
"racecar" is a palindrome
</stdout>

# Assistant
``` bash
python palindrome_checker.py hello
```

# User
<stdout>
"hello" is not a palindrome
</stdout>

# Assistant
``` bash
echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
```
[tab] data/eval/handcrafted/simple_calc_correct_error.md (markdown)

# Assistant
``` bash
ls -la
```

# User
<stdout>
total 336
drwxrwx--- 3 alfred.nguyen alfred.nguyen   318 Nov 13 18:05 .
drwxr-xr-x 9 alfred.nguyen alfred.nguyen   171 Nov 12 15:29 ..
-rw-rw---- 1 alfred.nguyen alfred.nguyen     0 Nov 13 18:05 calculator.py
</stdout>


# Assistant
``` bash
cat -n calculator.py
```

# User
<stdout>
     1  def add(a, b):
     2      return a + b
     3
     4  def subtract(a, b):
     5      return a - b
     6
     7  def multiply(a, b):
     8      return a * b
     9
     10  def divide(a, b):
     11      if b == 0:
     12          return "Error: Division by zero"
     13      return a // b
     14
     15  if __name__ == "__main__":
     16      import argparse
     17      parser = argparse.ArgumentParser(description="Simple calculator")
     18      parser.add_argument("operation", choices=["add", "subtract", "multiply", "divide"])
     19      parser.add_argument("a", type=float)
     20      parser.add_argument("b", type=float)
     21      args = parser.parse_args()
     22
     23      if args.operation == "add":
     24          result = add(args.a, args.b)
     25      elif args.operation == "subtract":
     26          result = subtract(args.a, args.b)
     27      elif args.operation == "multiply":
     28          result = multiply(args.a, args.b)
     29      else:
     30          result = divide(args.a, args.b)
     31
     32      print(result)
</stdout>

# Assistant
``` bash
python calculator.py add 5 3
```

# User
<stdout>
8.0
</stdout>

# Assistant
``` bash
python calculator.py multiply 4 7
```

# User
<stdout>
28.0
</stdout>

# Assistant
``` bash
python calculator.py divide 5 2
```

# User
<stdout>
2
</stdout>

# Assistant
``` bash
sed -i 's/^    return a \/\/ b/    return a \/ b/' calculator.py && cat -n calculator.py
```

# User
<stdout>
     1  def add(a, b):
     2      return a + b
     3
     4  def subtract(a, b):
     5      return a - b
     6
     7  def multiply(a, b):
     8      return a * b
     9
    10  def divide(a, b):
    11      if b == 0:
    12          return "Error: Division by zero"
    13      return a / b
    14
    15  if __name__ == "__main__":
    16      import argparse
    17      parser = argparse.ArgumentParser(description="Simple calculator")
    18      parser.add_argument("operation", choices=["add", "subtract", "multiply", "divide"])
    19      parser.add_argument("a", type=float)
    20      parser.add_argument("b", type=float)
    21      args = parser.parse_args()
    22
    23      if args.operation == "add":
    24          result = add(args.a, args.b)
    25      elif args.operation == "subtract":
    26          result = subtract(args.a, args.b)
    27      elif args.operation == "multiply":
    28          result = multiply(args.a, args.b)
    29      else:
    30          result = divide(args.a, args.b)
    31
    32      print(result)
</stdout>

# Assistant
``` bash
python calculator.py divide 5 2
```

# User
<stdout>
2.5
</stdout>

# Assistant
``` bash
echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
```
[tab] data/eval/handcrafted/simple_calc.md (markdown)
# User 
Create a simple calculator program that can add, subtract, multiply, and divide two numbers.

# Assistant
``` bash
cat <<'EOF' > calculator.py
def add(a, b):
    return a + b

def subtract(a, b):
    return a - b

def multiply(a, b):
    return a * b

def divide(a, b):
    if b == 0:
        return "Error: Division by zero"
    return a / b

if __name__ == "__main__":
    import argparse
    parser = argparse.ArgumentParser(description="Simple calculator")
    parser.add_argument("operation", choices=["add", "subtract", "multiply", "divide"])
    parser.add_argument("a", type=float)
    parser.add_argument("b", type=float)
    args = parser.parse_args()

    if args.operation == "add":
        result = add(args.a, args.b)
    elif args.operation == "subtract":
        result = subtract(args.a, args.b)
    elif args.operation == "multiply":
        result = multiply(args.a, args.b)
    else:
        result = divide(args.a, args.b)

    print(result)
EOF && cat -n calculator.py
```

# User
<stdout>
     1  def add(a, b):
     2      return a + b
     3
     4  def subtract(a, b):
     5      return a - b
     6
     7  def multiply(a, b):
     8      return a * b
     9
     10  def divide(a, b):
     11      if b == 0:
     12          return "Error: Division by zero"
     13      return a / b
     14
     15  if __name__ == "__main__":
     16      import argparse
     17      parser = argparse.ArgumentParser(description="Simple calculator")
     18      parser.add_argument("operation", choices=["add", "subtract", "multiply", "divide"])
     19      parser.add_argument("a", type=float)
     20      parser.add_argument("b", type=float)
     21      args = parser.parse_args()
     22
     23      if args.operation == "add":
     24          result = add(args.a, args.b)
     25      elif args.operation == "subtract":
     26          result = subtract(args.a, args.b)
     27      elif args.operation == "multiply":
     28          result = multiply(args.a, args.b)
     29      else:
     30          result = divide(args.a, args.b)
     31
     32      print(result)
</stdout>

# Assistant
``` bash
python calculator.py add 5 3
```

# User
<stdout>
8
</stdout>

# Assistant
``` bash
python calculator.py subtract 9 4
```

# User
<stdout>
5
</stdout>

# Assistant
``` bash
python calculator.py multiply 4 7
```

# User
<stdout>
28
</stdout>

# Assistant
``` bash
python calculator.py divide 5 2
```

# User
<stdout>
2.5
</stdout>

# Assistant
``` bash
echo COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT
```
[tab] src/input_pipeline/trace_to_testcases.py (python)
import json
import re
import tyro
import os
from dataclasses import dataclass


@dataclass
class Args:
    input_dir: str = "data/eval/handcrafted"
    output_file: str = "data/eval/handcrafted_test_cases/handcrafted_test_cases.jsonl"


def parse_md_file(input_file):
    with open(input_file, "r") as file:
        content = file.read()

    sections = re.split(r"# (User|Assistant)", content)
    sections = [s.strip() for s in sections if s.strip()]

    parsed_data = []
    current_dialogue = None

    for index, section in enumerate(sections):
        if section.startswith("User") or section.startswith("Assistant"):
            if current_dialogue is not None:
                parsed_data.append(current_dialogue)
            current_dialogue = {
                "role": section.lower(),
                "content": sections[index + 1].strip(),
            }

    if current_dialogue is not None:
        parsed_data.append(current_dialogue)

    return parsed_data


def create_incremental_test_cases(parsed_data, task_name):
    test_cases = []

    task_number = 0
    for i in range(1, len(parsed_data) - 1, 2):  # Ensure context ends with user content
        context = parsed_data[: i + 1]
        expected_response = (
            parsed_data[i + 1]["content"]
            if parsed_data[i + 1]["role"] == "assistant"
            else ""
        )

        test_case = {
            "task_id": f"{task_name}/{task_number}",
            "context": context,
            "expected_final_response": expected_response,
        }
        test_cases.append(test_case)
        task_number += 1

    return test_cases


def write_jsonl(test_cases, output_file):
    with open(output_file, "w") as file:
        for test_case in test_cases:
            file.write(json.dumps(test_case) + "
")


# Main execution
if __name__ == "__main__":
    args = tyro.cli(Args)

    os.makedirs(os.path.dirname(args.output_file), exist_ok=True)

    incremental_test_cases_total = []
    # iterate over all md files in the data/eval/handcrafted directory
    for file in os.listdir(args.input_dir):
        if file.endswith(".md"):
            task_name = file.split("/")[-1].split(".")[0]
            parsed_data = parse_md_file(os.path.join(args.input_dir, file))
            incremental_test_cases = create_incremental_test_cases(
                parsed_data, task_name
            )
            incremental_test_cases_total.extend(incremental_test_cases)

    write_jsonl(incremental_test_cases_total, args.output_file)
    print(
        f"Created {len(incremental_test_cases_total)} incremental test cases and saved to {args.output_file}"
    )
[tab] data/prompts/crowd_code_v1.md (markdown)
You are a helpful assistant that can interact multiple times with a computer shell to solve programming tasks.
Your response must contain exactly ONE bash code block with ONE command (or commands connected with && or ||).

Format your response as shown in <format_example>.
<format_example>
```bash
your_command_here
```
</format_example>
Failure to follow these rules will cause your response to be rejected.
[tab] data/prompts/judge_v2.md (markdown)
You are an expert command equivalence evaluator.
Your role is to analyze and judge the semantic equivalence of two Bash commands in the context of a conversational assistant that generates shell commands.

Follow these principles:

Understand Context:
Read the conversation history carefully to understand what the user intends to do.

Compare Commands:
Analyze both the expected and generated commands.

Focus on functional equivalence, not exact syntax.

Consider whether both commands achieve the same goal, even if they differ in structure, flags, or order.

Be Precise but Flexible:
Minor variations (e.g., using ls -l vs. ls --long) should not heavily penalize the score.
Major logical or behavioral differences (e.g., deleting files vs. listing them) should yield low scores.

Output Format:
Always respond in strict JSON format:

```json
{
  "score": <float between 0 and 1>,
  "reasoning": "<brief, clear explanation of key differences or similarities>"
}
```


Scoring Criteria:

1.0 → Commands are identical or produce exactly the same result

0.7–0.9 → Minor differences but same goal

0.4–0.6 → Related, but with meaningful differences (e.g., missing filters, flags, or output changes)

0.0–0.3 → Incorrect, dangerous, or completely different commands

Tone:
Be concise, neutral, and objective. No extra commentary outside the JSON.