p-doom/crowd-code-dataset-0.2 huggingface dataset:
[content] tab-model-eval/src/evaluation/sglang_eval.py (python)
    # Ensure the model reasons before making the decision by swapping JSON fields in instructions.
    # We do this here to enforce the behavior regardless of the state of the .md files on disk.
    if '"equivalent":' in prompt_template and '"reason":' in prompt_template:
        # If 'equivalent' appears before 'reason', swap them
        if prompt_template.find('"equivalent":') < prompt_template.find('"reason":'):
            # Replace common variations of the JSON block
            prompt_template = prompt_template.replace(
                '"equivalent": <1 or 0>,
  "reason": "<brief explanation>"',
                '"reason": "<brief reasoning>",
  "equivalent": <1 or 0>',
            ).replace(
                '"equivalent": <1 or 0>,
  "reason": "<detailed explanation>"',
                '"reason": "<brief reasoning>",
  "equivalent": <1 or 0>',
            )
[terminal_command] TERMINAL ()
git commit -m "feat: Add support for multiple judge samples per rollout

- Add num_samples parameter to Args class
- Add n parameter to judge API calls to enable multiple samples
- Add num_judge_samples_per_rollout metric to evaluation results
- Update wandb logging to include judge sample count

This allows running multiple judge evaluations per generation
sample to get more robust evaluation results."