p-doom/crowd-code-dataset-0.2 huggingface dataset:
[content] tab-model-eval/src/evaluation/sglang_eval.py (python)
        num_generation_samples = len(samples)
        num_generated_command_empty = 0
        for sample_idx, sample in enumerate(samples):
            is_generated_command_empty = sample["generated_command"] == ""
            if is_generated_command_empty:
                print(
                    f"Returning failure object for task {test_case['task_id']} due to empty generated command"
                )
                sample_results.append(
                    {
                        "task_id": test_case["task_id"],
                        "error": f"Empty generated command for task {test_case['task_id']}",
                        "equivalent": 0,
                        "generated_command_empty": 1,
                    }
                )
                num_generated_command_empty += 1
                continue
[tab] tab-model-eval/data/prompts/judge_prompt_v3_with_context.md (markdown)
# Semantic Equivalence Evaluation

Determine if the **Generated Command** achieves the same outcome as the **Expected Command**.

## Definitions

**Semantically Equivalent** — Commands produce the same result when executed. Differences that DO NOT affect equivalence:
- Flag order (`ls -la` ≡ `ls -al`)
- Short vs long flags (`-h` ≡ `--help`)
- Equivalent syntax (`cat file | grep x` ≡ `grep x file`)
- Whitespace or quoting style
- Line insertion position when order doesn't matter (e.g., import statements in most languages)
- Minor formatting differences in generated files that don't affect program behavior

**NOT Equivalent** — Commands produce different results:
- Different files, directories, or arguments
- Missing or extra flags that change behavior
- Different operations entirely
- Malformed or incomplete commands
- Order differences only when order semantically matters (e.g., command pipelines, ordered data insertions)

## Conversation History

{context}

## Commands

**Expected:**
{expected}

**Generated:**
{generated}

## Task

Compare the commands and respond with this exact JSON:

{{
  "reason": "<brief reasoning>",
  "equivalent": <1 or 0>
}}

Where:
- `1` = Same outcome (semantically equivalent)
- `0` = Different outcome (not equivalent)
[tab] tab-model-eval/data/prompts/system_prompt_v2.md (markdown)
You are a helpful assistant that interacts with a computer shell to solve programming tasks.
Your goal is to predict the next bash command a developer would most likely execute, given their editing and navigation history.

=== CONVERSATION FORMAT ===
The conversation history alternates between:
- Assistant messages: bash commands in fenced code blocks
- User messages: command output wrapped in <stdout>...</stdout> tags

After each edit, you should show the resulting file contents using `cat -n FILE | sed -n 'START,ENDp'`, which produces 6-character right-aligned line numbers followed by a tab, e.g.:
     1	first line
     2	second line

The chained cat command should show {viewport_lines} lines around the edited region.

=== RESPONSE FORMAT ===
Your response must contain exactly ONE bash code block with one command or two commands connected with &&.

<format_example>
```bash
your_command_here
```
</format_example>

Failure to follow these rules will cause your response to be rejected.

=== EDIT COMMAND FORMAT (IMPORTANT) ===
When you want to EDIT a file, you MUST encode the edit using line-based sed commands in ONE of the following forms, and you MUST NOT use substitution commands like "Ns/old/new/g".

Assume all line numbers are 1-based and paths are absolute.
Allowed edit encodings (choose exactly one per response):

1) Replace a contiguous block of lines:
   sed -i 'START,ENDc\
NEW_LINE_1\
NEW_LINE_2\
...
' /abs/path/to/file && cat -n /abs/path/to/file | sed -n 'VSTART,VENDp'

2) Delete a contiguous block of lines:
   sed -i 'START,ENDd' /abs/path/to/file && cat -n /abs/path/to/file | sed -n 'VSTART,VENDp'

3) Insert new lines BEFORE a given line:
   sed -i 'STARTi\
NEW_LINE_1\
NEW_LINE_2\
...
' /abs/path/to/file && cat -n /abs/path/to/file | sed -n 'VSTART,VENDp'

4) Append new lines at the END of the file:
   sed -i '$a\
NEW_LINE_1\
NEW_LINE_2\
...
' /abs/path/to/file && cat -n /abs/path/to/file | sed -n 'VSTART,VENDp'

Where VSTART and VEND specify a small viewport around the edited region.

Do NOT emit commands like "3s/print/print()/g" or any other "s/old/new/" style sed substitution; instead, always rewrite the affected lines using one of the line-based forms above.

When you are NOT editing files (e.g., running tests, git commands, tools, etc.), you may emit arbitrary bash commands.
[tab] slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_anta_claus.sh (shellscript)
# LAUNCH ANTA CLAUS EVAL JOBS

# 1.7B
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen1.7B_no_lora_bs_8_13032805
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen1.7B_no_lora_bs_16_13032803
# sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen1.7B_lora_bs_8_13032802
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen1.7B_lora_bs_16_13032800
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen1.7B_no_lora_bs_32_13032804

# 4B
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen4B_no_lora_bs_8_13032814
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen4B_lora_bs_8_13032811
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen4B_lora_bs_16_13032809
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen4B_no_lora_bs_16_13032812
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen4B_no_lora_bs_32_13032813
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen4B_lora_bs_32_13032810

# 8B
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen8B_no_lora_bs_32_13034357
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen8B_lora_bs_16_13032815
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen8B_lora_bs_8_13032817
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen8B_no_lora_bs_16_13032818
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen8B_lora_bs_32_13032816
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen8B_no_lora_bs_8_13032821

# 32B
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen32B_no_lora_bs_32_13032806
sbatch /p/home/jusers/mahajan1/juwels/projects/envcomp/miles/slurm/jobs/mihir/juelich/crowd-pilot/eval/eval_miles_run_offline.sh qwen32B_lora_bs_8_13032807
[content] slurm/dev/mihir/berlin/crowd_pilot/evaluate/handcrafted/eval_miles_run.sh (shellscript)
RUN_NAME=$1

# Extract RUN_ID (last underscore-separated field)
RUN_ID="${RUN_NAME##*_}"

# Extract model size from RUN_NAME and map to Qwen3 model name
# Pattern: qwen{size}_lora_... -> Qwen3-{size}
model_prefix="${RUN_NAME%%_*}"  # e.g., "qwen1.7B"
model_size="${model_prefix#qwen}"  # e.g., "1.7B"

# Map to official Qwen3 model names
case "$model_size" in
    0.6B|0.6b)   BASE_MODEL="Qwen3-0.6B" ;;
    1.7B|1.7b)   BASE_MODEL="Qwen3-1.7B" ;;
    4B|4b)       BASE_MODEL="Qwen3-4B" ;;
    8B|8b)       BASE_MODEL="Qwen3-8B" ;;
    14B|14b)     BASE_MODEL="Qwen3-14B" ;;
    32B|32b)     BASE_MODEL="Qwen3-32B" ;;
    30B-A3B|30b-a3b)     BASE_MODEL="Qwen3-30B-A3B" ;;
    235B-A22B|235b-a22b) BASE_MODEL="Qwen3-235B-A22B" ;;
    *)
        echo "Error: Unknown model size '$model_size' in RUN_NAME '$RUN_NAME'"
        exit 1
        ;;
esac

echo "Extracted from RUN_NAME: RUN_ID=$RUN_ID, BASE_MODEL=$BASE_MODEL"
[content] slurm/dev/mihir/berlin/crowd_pilot/evaluate/handcrafted/eval_miles_run_batch.sh (shellscript)
echo ""
echo "=============================================="
echo "Evaluation complete. Local logs saved to:"
echo "  $local_log_dir/eval_$RUN_ID/"
echo ""
echo "To sync to wandb from the login node, run:"
echo "  python src/evaluation/sync_local_logs_to_wandb.py --log_dir $local_log_dir/eval_$RUN_ID"
echo "=============================================="
[content] slurm/dev/mihir/berlin/crowd_pilot/evaluate/handcrafted/eval_miles_run_batch.sh (shellscript)
    # Append to batch lists (comma-separated)
    GENERATIONS_FILES="${GENERATIONS_FILES},${generations_file}"
    EVALUATIONS_FILES="${EVALUATIONS_FILES},${evaluations_file}"
    EVAL_STEPS="${EVAL_STEPS},${step}"
done

# =============================================================================
# PHASE 2: Batch evaluation with single judge model load
# =============================================================================
echo ""
echo "=============================================="
echo "PHASE 2: Batch evaluation (single judge load)"
echo "=============================================="

echo "Running batch evaluation with:"
echo "  Generations files: $GENERATIONS_FILES"
echo "  Evaluations files: $EVALUATIONS_FILES"
echo "  Eval steps: $EVAL_STEPS"
[content] slurm/dev/mihir/berlin/crowd_pilot/evaluate/handcrafted/eval_miles_run_batch.sh (shellscript)
# =============================================================================
# PHASE 1: Generate for all checkpoints
# =============================================================================
echo ""
echo "=============================================="
echo "PHASE 1: Generating outputs for all checkpoints"
echo "=============================================="

# Arrays to collect files for batch evaluation
GENERATIONS_FILES=""
EVALUATIONS_FILES=""
EVAL_STEPS=""

# First, generate for base model (step 0)
[content] slurm/dev/mihir/berlin/crowd_pilot/evaluate/handcrafted/eval_miles_run_batch.sh (shellscript)
# =============================================================================
# eval_miles_run_batch.sh
# Batch evaluation script - generates for all checkpoints first, then evaluates
# all in one go to avoid loading/unloading the judge model repeatedly.
# =============================================================================