Yi30/deepseek-v4-swebench-trajectories

DeepSeek V4 Flash — SWE-bench Verified Trajectories

20 agent trajectories from DeepSeek V4 Flash solving SWE-bench Verified instances using mini-swe-agent.

Dataset Structure

data/
  no_think/          # 10 trajectories — thinking disabled
    astropy__astropy-12907.traj.json
    astropy__astropy-13033.traj.json
    ...
  think_high/        # 10 trajectories — thinking enabled (reasoning_effort=high)
    astropy__astropy-12907.traj.json
    astropy__astropy-13033.traj.json
    ...

Each .traj.json is a mini-swe-agent trajectory file containing the full agent conversation: system prompt, user task, every assistant tool call, every command output, token usage, cost, and exit status.

Setup

# Server (2 GPUs, vLLM 0.23.1+)
CUDA_HOME=/usr/local/cuda-13 CUDA_VISIBLE_DEVICES=2,3 \
/home/yiliu7/workspace/venvs/vllm/bin/vllm serve \
  /storage/yiliu7/deepseek-ai/DeepSeek-V4-Flash \
  --trust-remote-code --kv-cache-dtype fp8 --block-size 256 \
  --tensor-parallel-size 2 --max-model-len 131072 \
  --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 \
  --enable-auto-tool-choice --reasoning-parser deepseek_v4 --port 8000

Config

No Thinking

model:
  model_name: "hosted_vllm//storage/yiliu7/deepseek-ai/DeepSeek-V4-Flash"
  model_kwargs:
    api_base: "http://localhost:8000/v1"
    max_tokens: 49152
    temperature: 1.0
    top_p: 0.95
    timeout: 900000

Think High

model:
  model_name: "hosted_vllm//storage/yiliu7/deepseek-ai/DeepSeek-V4-Flash"
  model_kwargs:
    api_base: "http://localhost:8000/v1"
    max_tokens: 49152
    temperature: 1.0
    top_p: 0.95
    timeout: 900000
    extra_body:
      chat_template_kwargs:
        thinking: true
        reasoning_effort: high

Important: Thinking is chat_template_kwargs.thinking — NOT thinking: {type: enabled} (that format is silently ignored).

How to Inspect

Quick stats for a single trajectory

import json

traj = json.load(open("data/no_think/astropy__astropy-12907.traj.json"))
info = traj["info"]
msgs = traj["messages"]

# Basic stats
print(f"Exit status: {info['exit_status']}")
print(f"API calls: {info['model_stats']['api_calls']}")
print(f"Messages: {len(msgs)}")

# Token usage
prompt = sum(m["extra"]["response"]["usage"]["prompt_tokens"] 
             for m in msgs if m.get("role") == "assistant" and "extra" in m)
compl = sum(m["extra"]["response"]["usage"]["completion_tokens"] 
            for m in msgs if m.get("role") == "assistant" and "extra" in m)
print(f"Prompt tokens: {prompt:,}")
print(f"Completion tokens: {compl:,}")

# Tool calls
actions = [m for m in msgs if m.get("role") == "assistant"]
print(f"Tool calls: {len(actions)}")
for i, a in enumerate(actions):
    tc = a.get("tool_calls", [])
    if tc:
        fn = tc[0]["function"]
        print(f"  [{i}] {fn['name']}({fn['arguments'][:100]})")

Full trajectory browser

pip install mini-swe-agent
mini-extra inspect data/no_think/

Compare two trajectories side-by-side

import json

def load_stats(path):
    t = json.load(open(path))
    msgs = t["messages"]
    api = t["info"]["model_stats"]["api_calls"]
    prompt = sum(m["extra"]["response"]["usage"]["prompt_tokens"] 
                 for m in msgs if m.get("role")=="assistant" and "extra" in m)
    compl = sum(m["extra"]["response"]["usage"]["completion_tokens"] 
                for m in msgs if m.get("role")=="assistant" and "extra" in m)
    return api, prompt, compl

for iid in ["astropy__astropy-12907", "astropy__astropy-13033", ...]:
    no_api, no_p, no_c = load_stats(f"data/no_think/{iid}.traj.json")
    th_api, th_p, th_c = load_stats(f"data/think_high/{iid}.traj.json")
    print(f"{iid}: API {no_api}→{th_api}, tokens {no_p+no_c:,}→{th_p+th_c:,}")

Results Summary

Overall

Metric	No Thinking	Think High
Submission rate	100%	100%
Resolved	6/10 (60%)	6/10 (60%)
Format errors	0	0
Total API calls	350	317
Total tokens	6,118,046	5,637,575
Avg API calls/instance	35.0	31.7
Avg tokens/instance	611,805	563,758

Per-Instance

instance	API no→think	Tokens no→think	ΔTokens	Resolved
astropy-12907	15→16	125K→154K	+23%	✓
astropy-13033	22→25	177K→273K	+55%	✗
astropy-13236	48→53	496K→735K	+48%	✓
astropy-13398	37→36	1.07M→909K	-15%	✗
astropy-13453	36→26	513K→349K	-32%	✓
astropy-13579	28→36	594K→1.21M	+104%	✓
astropy-13977	41→31	607K→471K	-22%	✗
astropy-14096	51→46	1.34M→806K	-40%	✓
astropy-14182	54→27	1.05M→423K	-60%	✗
astropy-14309	18→21	138K→309K	+125%	✓
AVG	35→32	612K→564K	-8%

Resolved Instances (both modes)

• astropy__astropy-12907 — separability_matrix bug fix
• astropy__astropy-13236 — unit conversion fix
• astropy__astropy-13453 — WCS transformation fix
• astropy__astropy-13579 — table indexing fix
• astropy__astropy-14096 — modeling compound model fix
• astropy__astropy-14309 — fitting outlier rejection fix

Unresolved Instances (both modes)

• astropy__astropy-13033 — WCS celestial transform
• astropy__astropy-13398 — modeling separable bug
• astropy__astropy-13977 — table operations fix
• astropy__astropy-14182 — visualization coordinate fix

Key Findings

1. 100% submission rate — DeepSeek V4's native deepseek_v4 tool parser eliminates format errors entirely. Every API call produces a valid tool call.

2. Thinking doesn't improve accuracy — Both modes resolve the exact same 6 instances. The bottleneck is the bash-only agent scaffold (no file-edit tools), not model reasoning.

3. Thinking improves efficiency — 9% fewer API calls, 8% fewer tokens overall. The model reasons more per call (+11% completion tokens) but makes better decisions that reduce total calls.

4. Wide token variance — 125K–1.34M tokens per instance. Resolved instances average 510K tokens; unresolved 519K — token usage does not predict success.

5. Model comparison with Qwen3 35B:

Metric	Qwen3 Best	DeepSeek V4
Submission	90%	100%
Resolved	50%	60%
Format errors	1	0
Avg actions	49	35

Citation

@misc{deepseek-v4-swebench-trajectories,
  author = {mini-swe-agent},
  title = {DeepSeek V4 Flash SWE-bench Verified Trajectories},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/...}}
}

License

MIT — trajectories generated by mini-swe-agent (MIT licensed) using DeepSeek V4 Flash.