p-doom/crowd-code-dataset-0.2 huggingface dataset:
[tab] tab-model-eval/data/prompts/judge_system_prompt_v3.md (markdown)
You are an expert command-line evaluator. Determine whether two bash commands are semantically equivalent — meaning a user would consider both valid solutions to the same task. Be lenient: if the commands accomplish the same goal with minor differences in approach, they are equivalent. Only mark as not equivalent when the commands clearly do different things or one would fail. Respond only in valid JSON format.
[tab] tab-model-eval/data/prompts/judge_prompt_v3.md (markdown)
# Semantic Equivalence Evaluation

Determine if the **Generated Command** achieves the same outcome as the **Expected Command**.

## Definitions

**Semantically Equivalent** — Commands produce the same result when executed. Differences that DO NOT affect equivalence:
- Flag order (`ls -la` ≡ `ls -al`)
- Short vs long flags (`-h` ≡ `--help`)
- Equivalent syntax (`cat file | grep x` ≡ `grep x file`)
- Whitespace or quoting style
- Line insertion position when order doesn't matter (e.g., import statements in most languages)
- Minor formatting differences in generated files that don't affect program behavior

**NOT Equivalent** — Commands produce different results:
- Different files, directories, or arguments
- Missing or extra flags that change behavior
- Different operations entirely
- Malformed or incomplete commands
- Empty commands
- Order differences only when order semantically matters (e.g., command pipelines, ordered data insertions)

## Commands

**Expected:**
{expected}

**Generated:**
{generated}

## Task

Compare the commands and respond with this exact JSON:

{{
  "equivalent": <1 or 0>,
  "reason": "<brief explanation>"
}}

Where:
- `1` = Same outcome (semantically equivalent)
- `0` = Different outcome (not equivalent)
[content] tab-model-eval/src/evaluation/sglang_eval.py (python)
Ensure the model reasons before making the decision by swapping JSON fields in instructions.
    # We do this here to enforce the behavior regardless of the state of the .md files on disk.
    if '"equivalent":' in prompt_template and '"reason":' in prompt_template:
        # If 'equivalent' appears before 'reason', swap them
        if prompt_template.find('"equivalent":') < prompt_template.find('"reason":'):
            # Replace common variations of the JSON block
            prompt_template = prompt_template.replace(
                '"equivalent": <1 or 0>,
  "reason": "<brief explanation>"',
                '"reason": "<detailed reasoning>",
  "equivalent": <1 or 0>',
            ).replace(
                '"equivalent": <1 or 0>,
  "reason": "<detailed explanation>"',
                '"reason": "<detailed reasoning>",
  "equivalent": <1 or 0>',
            )

    #
[tab] tab-model-eval/data/prompts/judge_prompt_v3_with_context.md (markdown)
# Semantic Equivalence Evaluation

Determine if the **Generated Command** achieves the same outcome as the **Expected Command**.

## Definitions

**Semantically Equivalent** — Commands produce the same result when executed. Differences that DO NOT affect equivalence:
- Flag order (`ls -la` ≡ `ls -al`)
- Short vs long flags (`-h` ≡ `--help`)
- Equivalent syntax (`cat file | grep x` ≡ `grep x file`)
- Whitespace or quoting style
- Line insertion position when order doesn't matter (e.g., import statements in most languages)
- Minor formatting differences in generated files that don't affect program behavior

**NOT Equivalent** — Commands produce different results:
- Different files, directories, or arguments
- Missing or extra flags that change behavior
- Different operations entirely
- Malformed or incomplete commands
- Order differences only when order semantically matters (e.g., command pipelines, ordered data insertions)

## Conversation History

{context}

## Commands

**Expected:**
{expected}

**Generated:**
{generated}

## Task

Compare the commands and respond with this exact JSON:

{{
  "reason": "<detailed reasoning>",
  "equivalent": <1 or 0>
}}

Where:
- `1` = Same outcome (semantically equivalent)
- `0` = Different outcome (not equivalent)
[tab] tab-model-eval/data/prompts/judge_prompt_v2.md (markdown)
# Semantic Equivalence Evaluation

Determine if the **Generated Command** achieves the same outcome as the **Expected Command**.

## Definitions

**Semantically Equivalent** — Commands produce the same result when executed. Differences that DO NOT affect equivalence:
- Flag order (`ls -la` ≡ `ls -al`)
- Short vs long flags (`-h` ≡ `--help`)
- Equivalent syntax (`cat file | grep x` ≡ `grep x file`)
- Whitespace or quoting style

**NOT Equivalent** — Commands produce different results:
- Different files, directories, or arguments
- Missing or extra flags that change behavior
- Different operations entirely
- Malformed or incomplete commands

## Commands

**Expected:**
{expected}

**Generated:**
{generated}

## Task

Compare the commands and respond with this exact JSON:

{{
  "reason": "<detailed reasoning>",
  "equivalent": <1 or 0>
}}

Where:
- `1` = Same outcome (semantically equivalent)
- `0` = Different outcome (not equivalent)
[tab] tab-model-eval/data/prompts/judge_prompt_v2_with_context.md (markdown)
# Semantic Equivalence Evaluation

Determine if the **Generated Command** achieves the same outcome as the **Expected Command**. Consider the **Conversation History** when evaluating.

## Definitions

**Semantically Equivalent** — Commands produce the same result when executed. Differences that DO NOT affect equivalence:
- Flag order (`ls -la` ≡ `ls -al`)
- Short vs long flags (`-h` ≡ `--help`)
- Equivalent syntax (`cat file | grep x` ≡ `grep x file`)
- Whitespace or quoting style

**NOT Equivalent** — Commands produce different results:
- Different files, directories, or arguments
- Missing or extra flags that change behavior
- Different operations entirely
- Malformed or incomplete commands

## Conversation History

{context}

## Commands

**Expected:**
{expected}

**Generated:**
{generated}

## Task

Compare the commands and respond with this exact JSON:

{{
  "reason": "<detailed reasoning>",
  "equivalent": <1 or 0>
}}

Where:
- `1` = Same outcome (semantically equivalent)
- `0` = Different outcome (not equivalent)